Case StudyHow Enkrypt AI ships its AI security platform into air-gapped KubernetesRead the case study
Skip to main content

Glossary

KV Cache: Storing Attention Keys and Values to Speed Up Inference


What Is a KV Cache?

A KV cache stores the attention keys and values of every token a language model has already processed. At each decoding step the model computes keys and values only for the newest token. It reads the rest from the cache. This removes repeated work and makes generation faster. The price is GPU memory, which grows with every token in the sequence.

Without a cache, the model would rerun attention over the entire sequence to produce each new token. Cost per token would grow with context length. With a cache, each step does a constant amount of new projection work plus a read of stored tensors.

Why does the KV cache matter for LLM inference?

Because it sets the limits on speed, concurrency, and context length. The cache lives in GPU memory next to the model weights. Every concurrent request holds its own cache. At long contexts, the cache can exceed the weights in size.

That makes memory, not compute, the bottleneck for many production deployments. Less memory per request means more concurrent requests per GPU — the same economics behind GPU fractioning and partitioning NVIDIA GPUs in Kubernetes. Cache management is therefore a direct cost lever for teams running self-hosted inference.

How does a KV cache work?

Each transformer layer projects every token into a query, a key, and a value. Attention compares the current token's query against all earlier keys, then mixes the matching values. Earlier keys and values never change once computed. The cache keeps them so they are computed once.

Inference runs in two phases:

  • Prefill. The model processes the whole prompt in parallel and fills the cache. This phase is compute-bound.
  • Decode. The model generates one token per step. It reads the full cache and appends one new entry. This phase is memory-bandwidth-bound.

How big is a KV cache?

Cache size per token equals two tensors (key and value), times layers, times KV heads, times head dimension, times bytes per element. Total size multiplies that by sequence length and batch size.

As a rough example, a model with 32 layers, 8 KV heads, head dimension 128, and 16-bit values needs about 128 KB per token. A 32,000-token context then holds roughly 4 GB for a single request. Treat these figures as illustrative, because they vary by architecture.

How do inference engines manage KV cache memory?

Engines reduce waste and reuse cache across requests. The main techniques are:

TechniqueWhat it doesMain benefit
Paged allocationStores cache in fixed-size blocks instead of one contiguous bufferLess fragmentation, higher batch sizes
Prefix cachingReuses cache blocks for prompt prefixes shared across requestsSkips repeated prefill work
Cache quantizationStores keys and values at lower precision, such as FP8Smaller footprint per token
Grouped-query attentionShares key and value heads across query heads in the model designFewer cache tensors per token
OffloadingMoves cold cache entries to CPU memory or diskLarger effective capacity, extra fetch delay
Cache-aware routingSends requests to the replica that already holds the relevant prefixHigher cache-hit rate across replicas

vLLM popularized paged allocation through PagedAttention. Prefix-cache-aware routing and offloading build on the same block structure, and distributed stacks like llm-d, NVIDIA Dynamo, and AIBrix coordinate it across replicas.

What breaks when the KV cache runs out?

When cache memory fills, the server must stop admitting new requests or evict existing ones. Evicted requests either wait in a queue or recompute their cache from scratch. Both raise time to first token and cut throughput. Teams see this as latency spikes under load, especially with long prompts and agent workloads that resend large context.

How CloudRaft helps

CloudRaft builds and tunes LLM inference platforms on Kubernetes, sizing KV cache memory against your real context lengths and concurrency. Talk to our team.

Conclusion

The KV cache is what makes autoregressive generation practical, and it is also what limits it. It converts repeated computation into stored memory. Managing that memory through paging, reuse, compression, and routing decides how many users each GPU can serve. Read more on the wider topic in what is AI inferencing.