Case StudyHow Enkrypt AI ships its AI security platform into air-gapped KubernetesRead the case study
Skip to main content

Glossary

Paged Attention: Managing KV Cache Memory with Virtual Paging


What Is PagedAttention?

PagedAttention is an attention algorithm that stores the KV cache in small fixed-size blocks instead of one contiguous buffer. A block table maps each request to its blocks, which can sit anywhere in GPU memory. Memory is allocated one block at a time as tokens are generated. This removes most fragmentation and lets a GPU hold more concurrent requests.

The technique comes from the 2023 vLLM paper by researchers at UC Berkeley. It borrows its design from operating-system virtual memory. Blocks act as pages, and the block table acts as a page table.

Why does PagedAttention matter for LLM inference?

Because KV cache memory limits how many requests a GPU can serve at once. Before paging, most engines reserved one contiguous buffer per request, sized for the maximum possible output length. Most of that reservation went unused.

The vLLM authors measured that earlier systems wasted 60 to 80 percent of KV cache memory through fragmentation and over-reservation. With paging, waste falls to the last partly filled block of each request. The paper reports 2 to 4 times higher throughput than systems such as FasterTransformer and Orca at similar latency. Treat these numbers as workload-dependent.

How does PagedAttention work?

The engine divides GPU cache memory into physical blocks. Each block holds the keys and values for a fixed number of tokens. vLLM uses 16 tokens per block by default.

Each request keeps a block table. The table lists logical blocks in order and points each one at a physical block. Logical blocks look contiguous to the attention kernel. Physical blocks do not need to be.

At each decoding step the process is simple:

  • Allocate. When the last block fills, the engine grabs one free physical block.
  • Look up. The attention kernel reads the block table and gathers keys and values from each listed block.
  • Free. When a request finishes, its blocks return to the free pool at once.

How does PagedAttention compare with contiguous KV cache allocation?

Paged allocation trades a small lookup cost for far better memory use.

PropertyContiguous allocationPagedAttention
Memory reservedMaximum sequence length per requestOne block at a time, on demand
Internal fragmentationHigh, unused tail of every reservationLow, only the last partial block
External fragmentationGaps between buffers of different sizesNone, all blocks are the same size
Prefix sharingRequires copying the cacheRequests point at the same blocks
Kernel complexitySimple contiguous readsNeeds a custom kernel that follows the table

The custom kernel is the cost. Reading scattered blocks is slightly slower than reading one buffer. In practice the larger batch sizes more than pay for it.

How does PagedAttention enable cache sharing?

Block tables let several requests point at the same physical blocks. This works for parallel sampling, beam search, and shared prompt prefixes. When one request needs to change a shared block, the engine copies it first. This is copy-on-write, the same idea operating systems use for forked processes.

Block-level sharing is also the foundation of prefix caching. The same block structure supports prefix-cache-aware routing, where a router sends requests to the replica that already holds the matching blocks. Distributed stacks such as llm-d, NVIDIA Dynamo, and AIBrix build on that structure across many replicas.

Where is PagedAttention used today?

vLLM introduced it, and it is now the reference design for KV cache management. Other serving engines use block-based cache allocation that follows the same idea. For the broader picture of serving a model in production, see what is AI inferencing.

How CloudRaft helps

CloudRaft builds and tunes LLM inference platforms on Kubernetes, including vLLM block size, memory limits, and batch settings measured on your own traffic. Talk to our team.

Conclusion

PagedAttention applies virtual-memory paging to the KV cache. Fixed-size blocks and a block table cut memory waste and allow sharing across requests. The result is more concurrent requests per GPU with the same model output.