What Is Chunked Prefill?
Chunked prefill is an LLM scheduling technique that splits a long prompt into fixed-size chunks and processes each chunk in the same forward pass as decode tokens from other requests. Long prompts no longer block token generation for everyone else. The result is smoother streaming and fewer latency spikes, with little loss in overall throughput.
The idea was popularised by the Sarathi-Serve research under the name stall-free batching. Today it is a standard feature in serving engines such as vLLM, where it is on by default in the V1 scheduler.
Why does a long prompt stall other requests?
Inference has two phases with opposite profiles. Prefill processes the whole prompt at once and is compute-bound. Decode generates one token per step and is memory-bandwidth-bound.
In a plain scheduler, a new 30,000-token prompt runs its prefill as one large pass. Every request already decoding must wait for that pass to finish. Users see streaming freeze, which shows up as a spike in inter-token latency.
This problem grows with context length. Agent and RAG workloads routinely send long prompts, so the stalls become frequent. Continuous batching fills the batch efficiently, but it does not by itself stop one big prefill from monopolising a step.
How does chunked prefill work?
The scheduler sets a token budget per step, controlled in vLLM by max_num_batched_tokens. It then fills that budget in priority order.
- Schedule all running decode requests first, one token each.
- Spend the remaining budget on prompt tokens from waiting requests.
- If a prompt exceeds the remaining budget, process only a chunk and resume next step.
Each step now has bounded work, so decode requests advance every step. The partially processed prompt keeps its progress in the KV cache, stored in blocks through paged attention. Chunk N reads the cache written by chunks 1 to N-1, so the final result is identical to unchunked prefill.
Chunked prefill vs prefill-decode disaggregation, what is the difference?
Both fix prefill interference, but at different layers.
| Aspect | Chunked prefill | Prefill-decode disaggregation |
|---|---|---|
| Where phases run | Same GPU, mixed in each step | Separate GPU pools |
| KV cache movement | None | Transferred over the network |
| Complexity | Low, a scheduler setting | High, needs routing and fast links |
| Best fit | Single node or small replica sets | Large fleets with strict latency goals |
Start with chunked prefill. Consider disaggregation only when one pool cannot meet both latency targets.
What are the trade-offs of chunked prefill?
Chunking spreads a prompt over several steps, so a very long prompt can see a slightly later first token. It also re-reads the earlier KV cache for each chunk, adding some memory traffic.
In return, other users get stable inter-token latency, and the GPU stays busier because compute-bound prefill and memory-bound decode share each step. A small chunk protects decode smoothness. A large chunk finishes prompts sooner. Offline batch jobs usually prefer larger budgets, while interactive chat prefers smaller ones.
How do you tune chunked prefill in vLLM?
Set max_num_batched_tokens and measure. Track time to first token, inter-token latency percentiles, and throughput under your real prompt mix. Synthetic short prompts hide the effect entirely, because chunking only matters when long prompts arrive alongside active decodes.
How CloudRaft helps
CloudRaft tunes vLLM schedulers and GPU capacity for production inference on Kubernetes, measured against your own prompt lengths. Talk to our team.
Conclusion
Chunked prefill slices long prompts so decode work never waits for one giant pass. It keeps inter-token latency steady at a small cost in prompt completion time. It is the default in modern vLLM, so the real work is choosing a token budget that fits your traffic.