Low-Latency LLM Web Search Caching on Commodity CPUs

Low-Latency LLM Web Search Caching on Commodity CPUs

If you’ve been building (or just tinkering with) AI search, you’ve probably noticed a nasty pattern: the expensive part isn’t always the LLM. It’s the chaos around it—session memory that vanishes, repeated work when users rephrase the same question, and redundant embeddings for the same URLs across thousands of sessions.

New research from the authors of the OreoLook paper lays out a very practical answer engine design called OreoLook (formerly lixSearch). It’s an open-source, provider-routed LLM synthesis system paired with automated browser agents for web discovery—running its local search, caching, session management, and embedding stack on commodity CPU hardware. The main contribution isn’t “yet another search agent.” It’s a three-layer caching architecture that keeps costs and latency under control as usage scales.

In this post, I’ll walk through the problems the team hit in production—lost conversational context, semantic rephrasing duplication, and repeated embedding computation—and how their three caching layers fix them with a surprisingly lightweight setup centered around Redis plus disk overflow. The best part? It’s designed so you can integrate it without rebuilding your whole pipeline.

Why This Matters: Caching Is the Real “LLM Product Feature” Right Now

Right now, AI search is moving from “cool demo” to “paying workload.” That means two things happen fast: (1) query volume grows, and (2) user behavior becomes messy in predictable ways. People don’t ask exactly once; they chat, they rephrase, and they bounce back to older conversations. Without careful state management, your system quietly turns into a cost amplifier.

This research is significant because it treats “multi-turn web search” as an infrastructure problem, not a model problem. A lot of earlier AI work focused on retrieval quality (RAG), better prompting, or smarter ranking. Those matter—but once you’ve got an answer grounded in the live web, the next bottleneck is usually how many times you re-do the same work.

A scenario you can apply today: imagine a customer support bot that answers “How do I reset my router?” and then follows up with “Reset to factory settings” or “Steps for TP-Link model X.” If your system doesn’t deduplicate semantically, it will repeatedly run browser agents, refetch pages, and pay for remote LLM synthesis. Meanwhile, if your system doesn’t persist the last few turns, the assistant will keep “rediscovering” context. And if you don’t cache embeddings for popular help-center URLs, you’ll compute vector embeddings again and again across sessions. OreoLook’s approach is built exactly to prevent that death by a thousand redundant operations.

The authors also build on the earlier ecosystem (like LangChain-style memory modules and GPTCache-style semantic response caching), but their key move is unified engineering: session context, semantic query deduplication, and URL embedding reuse become one lightweight package with consistent integration points. That’s what makes it production-friendly rather than a collection of half-solutions.

The “Three Failures” That Break LLM Search Cost Control in Production

OreoLook started simple: automated browser agents fetch web content, then a provider-routed LLM synthesizes an answer. No retrieval database. No cache layer. The initial version worked—and cut per-query costs versus proprietary search APIs because it avoided per-call web search fees.

But as usage increased, three problems showed up clearly:

1. “What did we just talk about?”
   Multi-turn conversations require context. The team observed that as sessions grew and the system scaled, it lost context. Storing entire conversation histories in memory didn’t scale well across thousands of concurrent sessions.

2. “Didn’t we already answer this?”
   Users rephrase. Without semantic deduplication, “weather Tokyo” followed by “Tokyo weather forecast” triggered a full pipeline rerun—search agent work, page fetching, and LLM synthesis—despite the near-identical intent.

3. “We already embedded this URL.”
   When RAG-like improvements were added, popular URLs (Wikipedia, news sites, docs) were repeatedly embedded across sessions. They measured that embedding computation cost isn’t trivial: local embedding runs on CPU, and each repeated URL embedding was wasted compute.

A quick cost framing from the paper makes the stakes real. Commercial AI search pricing scales linearly with volume and doesn’t offer meaningful economies of scale. For example, OpenAI’s search-related call pattern is described as $10–30 per 1,000 search calls plus injected web context tokens, with observed practical query costs around $0.03–$0.10 depending on model and context tier. Perplexity’s Sonar API is priced $5/1K for base search (up to $18/1K for Sonar Pro). These details matter because they explain why developers care so much about shaving even “small” per-query costs.

OreoLook’s three-layer caching architecture is designed to attack those exact failure points.

How OreoLook Routes Work: From User Query to Cached or Synthesized Answers

The architecture lives in front of the pipeline. Before the system pays for “expensive” steps (like provider LLM synthesis, browser agent page fetching, and embedding computation), it checks whether it can short-circuit.

At a high level, each incoming user message runs through a caching coordinator that can:
- restore session context (so follow-ups make sense),
- detect semantically equivalent queries (so it can reuse an earlier LLM answer),
- reuse embeddings for previously seen URLs (so it doesn’t recompute vectors).

A single Redis instance, but three separate logical partitions

Instead of putting everything under one key prefix soup, the team used three separate Redis logical databases inside one Redis instance:
- DB 0: semantic query cache
- DB 1: URL embedding cache
- DB 2: session context window

This design decision isn’t just tidy—it supports different TTLs, different scopes (per-session vs global), and independent failure modes. That’s a big deal operationally: you can flush or expire one layer without trashing the others.

The coordinator abstraction: “one object, three caching jobs”

The integration surface is intentionally small. The paper describes a wrapper with one per-session object and a unified API (conceptually something like):
- add_message_to_context
- get_semantic_response
- get_url_embedding
- get_stats

Under the hood, each call routes to the correct cache layer. That means you can add this caching without redesigning the whole assistant.

And yes—the original paper (https://arxiv.org/abs/2609.05463) goes deeper into implementation details and measurements, but the core concept is straightforward: make repeated work expensive to do and cheap to avoid.

Layer 1: Session Context Window With Hot Memory and Compressed Disk Overflow

Let’s start with the most “human” part: remembering the conversation.

Most teams start by keeping the last N messages. That works until you scale or your users return later and ask: “What was that article you found earlier?” Then your context is gone.

OreoLook uses a rolling window in Redis for the most recent messages, with disk archival for overflow.

The hot window (Redis) stays bounded

For each session, it keeps the last k messages (default k=20). Implementation-wise, messages are stored as individual Redis keys with TTL, and an ordered Redis list tracks insertion order.

When a new message arrives:
1. it’s pushed into the head of the Redis list,
2. if the list exceeds k, the oldest message is popped,
3. that popped message is serialized and appended to a Huffman-compressed disk archive,
4. Redis keys for evicted messages are deleted.

So Redis memory usage stays O(k) per session, even if conversations last a long time.

Disk overflow prevents “context amnesia” after eviction

If Redis is empty for that session (because messages were evicted or the hot window was flushed), the system transparently re-hydrates context:
- it loads the most recent k messages from disk back into Redis,
- then the assistant can continue as if nothing happened.

This is a really important production detail: it means the assistant doesn’t “forget” permanently—Redis is treated like fast cache memory, not the source of truth.

Why Huffman compression instead of gzip/lz4?

The paper’s compression choice is surprisingly opinionated.

They tried common compressors first (e.g., zlib and compare against lz4). Huffman coding won’t usually beat a dictionary-based compressor on large payloads, but their archives are typically small (often 1–100 KB, and especially <<5 KB).

Their reasoning:
- Small payloads: dictionary overhead in zlib can dominate.
- Byte frequency skew: English text has heavy skew (spaces ~18% of bytes, e ~13%, while z ~0.07%).
- Operational simplicity: canonical Huffman coding in pure Python means zero native dependencies.

They also report measured compression behavior: for small production archives, Huffman gets roughly 65–69% compression ratio (with synthetic text around 54%). They compare it directly against zlib and lz4 in the evaluation and conclude Huffman’s edge here is practical, not magical.

Latency reality: Redis is fast, disk is “rare but acceptable”

The evaluation reports that Redis reads are about two orders of magnitude faster than disk reads. Even for the largest archive (133 turns), disk re-hydration completes in about 107 ms, which is acceptable because this happens infrequently (only when returning to a session or after eviction).

Layer 2: Semantic Query Cache That Catches Rephrasings Before You Pay for LLMs

Now to the second failure: rephrasing.

Users don’t repeat exact strings; they change wording. If you only cache by exact match, you miss most opportunities.

OreoLook’s Semantic Query Cache uses embeddings to detect when two queries are “close enough” to reuse an earlier LLM answer.

The mechanism: embed the query, compare cosine similarity, short-circuit if matched

For every incoming query:
1. compute a query embedding q using sentence-transformers/all-MiniLM-L6-v2 (384 dimensions),
2. fetch cached pairs (e_i, r_i) for the current session and URL,
3. compute cosine similarity between q and each cached embedding e_i,
4. if the maximum similarity is above a threshold τ (default 0.90), return the cached LLM response and skip the pipeline.

Each URL stores up to 50 cached entries (configurable) with a 5-minute TTL. This short TTL is an intentional freshness trade-off: LLM answers and web content can drift, so the cache shouldn’t last forever.

How much does this actually help?

They observed that in the production workload, about 15–20% of queries within a session were semantic near-duplicates (cosine similarity ≥ 0.90). When the semantic cache hits, it avoids the full pipeline (search agents + LLM synthesis).

They also state that semantic hits save on the order of 3–8 seconds of wall-clock time per avoided provider synthesis call.

The architecture is scoped per session for privacy isolation: unlike some global semantic caches, this doesn’t let one user’s cached response leak into another.

What this build avoids vs previous semantic caching approaches

The paper contrasts their approach with GPTCache and notes three differences:
- Their cache is per-session (GPTCache is global by default).
- They avoid requiring a separate vector database like FAISS/Milvus/Qdrant by storing embeddings directly in Redis.
- GPTCache only addresses semantic dedup; it doesn’t unify session context persistence and URL embedding reuse.

That unified scope is the difference between “cool caching demo” and “deployable assistant.”

Layer 3: Cross-Session URL Embedding Cache to Stop Recomputing the Same Vectors

The third failure is the sneakiest: repeated work happens across sessions, not just within one.

Once you add RAG-style behavior, you inevitably fetch and embed the same popular URLs again and again.

OreoLook’s fix: a global (cross-session) URL Embedding Cache.

One embedding per URL per day (approximately)

Instead of computing embeddings in each session, the system:
- maps URL string → embedding vector,
- stores that embedding in Redis as raw float bytes (float32),
- sets a TTL of 24 hours.

The point isn’t that URLs never change; it’s that popular pages like Wikipedia and major news sites don’t change minute-to-minute. A 24-hour embedding reuse window is a practical compromise.

They also report embedding compute cost timing: roughly ~200 ms per URL locally on CPU. Caching that across sessions is where “hidden costs” disappear.

Memory efficiency: raw bytes beat JSON floats

The paper makes a concrete implementation optimization:
- store embeddings as raw 32-bit float byte arrays,
- avoid JSON array serialization overhead.

For a 384-dimensional embedding, raw bytes take exactly 1,536 bytes in Redis, versus about 3,800 bytes as a JSON array of floats—about 2.5× space savings.

That matters because this cache can accumulate quickly with popular URLs.

What the Production Evaluation Actually Showed Under Load

The evaluation is a historical production snapshot (so don’t treat numbers as timeless guarantees). Still, it’s one of the more useful aspects of the paper: not just “it works,” but “what did it do in real usage.”

Hardware and deployment snapshot

• local stack measured on an 8-vCPU Intel Cascade Lake cloud instance (32 GB RAM), no GPU,
• search assistant ran as three containerized replicas, each 2 vCPU / 2 GB limits,
• Redis container capped at 2 GB, with nginx in front,
• LLM synthesis was remote via provider routing.

Redis keyspace hit rate: a sanity indicator (not a direct user-hit metric)

The paper reports an aggregate Redis keyspace hit rate across DB0/DB1/DB2 of 89.3%, computed as:
- 2,182 keyspace hits
- 262 keyspace misses
- total 2,444 operations, giving 89.3%

Important caveat from the authors: this isn’t a query-level semantic hit rate. It includes internal Redis operations and TTL refreshes. Still, it indicates the system keeps the working set hot and avoids falling back to disk or recomputation.

Estimated contribution by layer

They don’t claim precise request-level attribution (they say request-level measurement of avoided inference remains future work), but they do provide exploratory estimates based on access patterns:

One interesting nuance: Layer 1 may dominate keyspace hits, but Layer 3 has fewer hits because URL embeddings are reused less frequently than session context, even though each hit saves a relatively meaningful CPU chunk.

Compression results: Huffman’s practical sweet spot

They evaluate compression ratios across five production conversation archives. They also compare Huffman vs zlib level 1 vs lz4 on the same archives and find:
- zlib compresses smaller payloads slightly better (especially at larger sizes),
- Huffman remains strong for typical archive sizes (often in the <<5 KB range),
- Huffman’s advantage is mostly operational: pure Python, no native dependencies, small overhead on their payload sizes.

A note on limitations the paper explicitly calls out

To keep this honest: the paper flags several constraints:
- Semantic lookup is brute-force cosine similarity across up to 50 cached embeddings per URL (O(n)), which is fine at current scale but would need an index for much larger deployments.
- Pure-Python Huffman is acceptable for typical archive sizes but a C extension could help for megabyte-scale payloads.
- Archives are compressed but not encrypted at rest—sensitive data deployments should add encryption.

Closing Thoughts: A Caching Architecture You Can Actually Copy

The core insight from this paper is simple but easy to miss: resumable conversations aren’t a feature of the LLM alone—they’re an infrastructure capability. If you want low-latency and controllable costs for LLM web search, caching has to cover more than one thing.

OreoLook’s three-layer approach is effective because each layer targets a specific kind of repetition:
- Session context caching prevents conversation amnesia without unbounded RAM growth.
- Semantic query caching catches rephrasings within short freshness windows and avoids repeated provider inference.
- URL embedding caching eliminates cross-session vector recomputation for popular pages.

And it’s designed with production constraints in mind—Redis partitioning with different TTLs, disk archival with Huffman compression, and a coordinator façade that keeps the integration surface small.

If you’re building an LLM-powered search assistant and you’re feeling that rising bill or sluggish multi-turn behavior, this is a blueprint worth borrowing. Not because it’s flashy—because it’s boringly effective at removing the redundant work your system is currently paying for.

Key Takeaways

• OreoLook is an open-source LLM web search assistant that runs its local browsing, caching, session management, and embedding on commodity CPU hardware, with remote provider LLM synthesis.
• The production failures were predictable: lost session context, semantic rephrasing duplication, and repeated URL embedding computation across sessions.
• The paper’s solution is a three-layer caching architecture:
  • Layer 1: Session context window in Redis (hot) with Huffman-compressed disk overflow (cold), bounded to k=20 recent messages by default.
  • Layer 2: Semantic query cache using cosine similarity (τ=0.90) to reuse prior LLM responses for rephrased near-duplicate queries (TTL 5 minutes; up to 50 cached entries per URL).
  • Layer 3: URL embedding cache shared across sessions, re-embedding each URL at most once per 24 hours (float embeddings stored efficiently as raw bytes).
• Operational design matters: using three Redis logical DBs (DB0 semantic, DB1 URL embeddings, DB2 session context) enables separate TTL policies, monitoring, and safer failure modes.
• In their historical snapshot, they report:
  • 89.3% Redis keyspace hit rate (aggregate indicator of hot working set),
  • estimated semantic near-duplicates of ~15–20% of queries within sessions (similarity ≥ 0.90),
  • semantic hits saving roughly 3–8 seconds by avoiding full pipeline work,
  • URL embedding hits saving roughly ~200 ms per URL avoided embedding computation.
• For builders: if your LLM search costs feel “mysteriously high” as usage grows, start by implementing stateful session caching, then add semantic dedup, and finally add cross-session embedding reuse—those three together address the biggest repetition patterns.