What Is Prefix-Cache-Aware Routing?
Prefix-cache-aware routing is an LLM inference load-balancing technique that sends each request to the model-server replica that already holds the key-value (KV) cache for the request's prompt prefix, so the server can skip recomputing that prefix and return the first token faster. It is also called KV-cache-aware routing, and it replaces cache-blind load balancing with routing that knows what each replica has already computed.
The idea builds on prefix caching in inference servers such as vLLM, where a server reuses the KV cache of a prompt prefix it has already processed instead of recomputing it. That reuse only helps if the request lands on the replica that holds the cache. A normal load balancer does not know which replica that is, so the saving is lost. Prefix-cache-aware routing closes that gap by making the router cache-aware.
Why does prefix-cache-aware routing matter now?
Because the default load balancers waste the most expensive step in inference, and the fix has just become standard, open, and Kubernetes-native. Round-robin and least-request balancers spread traffic without regard to cache contents, so requests that could have reused a cached prefix land on a replica that has to recompute it. That redundant prefill burns GPU time and inflates first-token latency. The problem is worst exactly where production LLM traffic concentrates: long shared system prompts, RAG context, agent scaffolding, and templated prompts, all of which share large prefixes across requests. Through 2025 and 2026, cache-aware routing shipped across the ecosystem, in the open-source llm-d project (IBM, Google, Red Hat), the GKE Inference Gateway, the vLLM production stack, and Envoy AI Gateway, mostly through the Gateway API Inference Extension. It is now a feature a platform team can turn on, not a research prototype.
How does prefix-cache-aware routing work?
The router keeps a view of which prompt prefixes are cached on which replicas, scores each replica by how much of the incoming prompt's prefix it already holds, and routes to the best match. In llm-d this lives in the gateway's Endpoint Picker, which scores candidate pods by the length of the matching cached prefix and sends the request to the highest scorer. After routing, it updates its view so the next request benefits. Session-aware affinity is usually layered on top, so a continuing conversation keeps landing on the replica that holds its history.
Approximate vs precise prefix-cache routing
There are two implementations, trading accuracy against operational overhead.
| Approximate | Precise | |
|---|---|---|
| How it tracks cache state | The router hashes fixed-size prompt blocks and keeps an in-memory index of which prefixes it recently sent to which replica | Replicas emit KV-cache events (add and evict) over ZeroMQ; the router keeps a globally consistent, token-exact index of what each replica holds |
| Tokenization | Estimated from character ratios, no tokenizer needed | Exact token IDs from the inference server's render endpoint |
| Accuracy | Heuristic; can drift from reality when a replica evicts a prefix under memory pressure | Token-exact matching |
| Dependencies | None beyond the router | Inference-server support for KV events, a render endpoint, and ZMQ connectivity |
| Best fit | A fast default with no server integration | Highest cache-hit rate and prefill/decode disaggregation |
Approximate routing is the pragmatic default: it needs no integration and captures most of the benefit. Precise routing wins the last mile of cache-hit rate when the inference servers support KV events and the traffic justifies the extra moving parts.
How much does it help?
The gains are real but depend on how much prefix your traffic actually shares, so treat published figures as illustrative, not guaranteed. In one llm-d test published by Red Hat and IBM, cache-aware routing reached roughly an 87% cache-hit rate and dropped a warm-cache first-token latency from about 2,850 ms to about 340 ms, close to a 90% reduction on those requests. (Red Hat Developers). That was an internal test with heavily shared prefixes, not a controlled public benchmark, so the honest takeaway is directional: the more prefix your requests share, the larger the win, and workloads with long shared system prompts, RAG context, or agent scaffolding benefit most. Workloads where every prompt is unique see little.
Where does prefix-cache-aware routing show up in practice?
It shows up wherever requests share a large, stable prefix: a long system prompt reused across every call, a RAG pipeline that prepends the same retrieved context, an agent that carries the same tool definitions and scaffolding on each step, or a multi-turn chat where the history is the prefix. In those cases the prefill saving is large and repeated, which is why cache-aware routing pairs naturally with an AI gateway and with GPU-efficiency work such as GPU fractioning: all three are about getting more useful work out of the same accelerators.
How CloudRaft helps with prefix-cache-aware routing
CloudRaft builds and tunes LLM inference platforms on Kubernetes, including cache-aware routing. We stand up the inference gateway (llm-d, the GKE Inference Gateway, or an Envoy and Gateway API Inference Extension setup), choose approximate or precise routing to fit your traffic and inference servers, wire prefix caching in vLLM, and measure the result on your own prompts so the cache-hit rate and first-token latency are real, not borrowed from a slide. The outcome is lower latency and more throughput from the GPUs you already pay for. Talk to our team about your inference platform.
Conclusion
Prefix-cache-aware routing fixes a quiet, expensive default: load balancers that scatter requests away from the caches that would have served them. By routing each request to the replica that already holds its prefix, it cuts redundant prefill, lowers first-token latency, and frees GPU capacity, most of all for the shared-prefix traffic that dominates real LLM workloads. It is now standard and open, and the value is in setting it up correctly and measuring it on your own prompts.