What Is PagedAttention?
PagedAttention is an attention algorithm that stores the KV cache in small fixed-size blocks instead of one contiguous buffer. A block table maps each request to its blocks, which can sit anywhere in GPU memory. Memory is allocated one block at a time as tokens are generated. This removes most fragmentation and lets a GPU hold more concurrent requests.
The technique comes from the 2023 vLLM paper by researchers at UC Berkeley. It borrows its design from operating-system virtual memory. Blocks act as pages, and the block table acts as a page table.
Why does PagedAttention matter for LLM inference?
Because KV cache memory limits how many requests a GPU can serve at once. Before paging, most engines reserved one contiguous buffer per request, sized for the maximum possible output length. Most of that reservation went unused.
The vLLM authors measured that earlier systems wasted 60 to 80 percent of KV cache memory through fragmentation and over-reservation. With paging, waste falls to the last partly filled block of each request. The paper reports 2 to 4 times higher throughput than systems such as FasterTransformer and Orca at similar latency. Treat these numbers as workload-dependent.
How does PagedAttention work?
The engine divides GPU cache memory into physical blocks. Each block holds the keys and values for a fixed number of tokens. vLLM uses 16 tokens per block by default.
Each request keeps a block table. The table lists logical blocks in order and points each one at a physical block. Logical blocks look contiguous to the attention kernel. Physical blocks do not need to be.
At each decoding step the process is simple:
- Allocate. When the last block fills, the engine grabs one free physical block.
- Look up. The attention kernel reads the block table and gathers keys and values from each listed block.
- Free. When a request finishes, its blocks return to the free pool at once.
How does PagedAttention compare with contiguous KV cache allocation?
Paged allocation trades a small lookup cost for far better memory use.
| Property | Contiguous allocation | PagedAttention |
|---|---|---|
| Memory reserved | Maximum sequence length per request | One block at a time, on demand |
| Internal fragmentation | High, unused tail of every reservation | Low, only the last partial block |
| External fragmentation | Gaps between buffers of different sizes | None, all blocks are the same size |
| Prefix sharing | Requires copying the cache | Requests point at the same blocks |
| Kernel complexity | Simple contiguous reads | Needs a custom kernel that follows the table |
The custom kernel is the cost. Reading scattered blocks is slightly slower than reading one buffer. In practice the larger batch sizes more than pay for it.
How does PagedAttention enable cache sharing?
Block tables let several requests point at the same physical blocks. This works for parallel sampling, beam search, and shared prompt prefixes. When one request needs to change a shared block, the engine copies it first. This is copy-on-write, the same idea operating systems use for forked processes.
Block-level sharing is also the foundation of prefix caching. The same block structure supports prefix-cache-aware routing, where a router sends requests to the replica that already holds the matching blocks. Distributed stacks such as llm-d, NVIDIA Dynamo, and AIBrix build on that structure across many replicas.
Where is PagedAttention used today?
vLLM introduced it, and it is now the reference design for KV cache management. Other serving engines use block-based cache allocation that follows the same idea. For the broader picture of serving a model in production, see what is AI inferencing.
How CloudRaft helps
CloudRaft builds and tunes LLM inference platforms on Kubernetes, including vLLM block size, memory limits, and batch settings measured on your own traffic. Talk to our team.
Conclusion
PagedAttention applies virtual-memory paging to the KV cache. Fixed-size blocks and a block table cut memory waste and allow sharing across requests. The result is more concurrent requests per GPU with the same model output.