Case StudyHow Enkrypt AI ships its AI security platform into air-gapped KubernetesRead the case study
Skip to main content

Glossary

Chunked Prefill: Interleaving Prompt Prefills with Decode Steps


What Is Chunked Prefill?

Chunked prefill is an LLM scheduling technique that splits a long prompt into fixed-size chunks and processes each chunk in the same forward pass as decode tokens from other requests. Long prompts no longer block token generation for everyone else. The result is smoother streaming and fewer latency spikes, with little loss in overall throughput.

The idea was popularised by the Sarathi-Serve research under the name stall-free batching. Today it is a standard feature in serving engines such as vLLM, where it is on by default in the V1 scheduler.

Why does a long prompt stall other requests?

Inference has two phases with opposite profiles. Prefill processes the whole prompt at once and is compute-bound. Decode generates one token per step and is memory-bandwidth-bound.

In a plain scheduler, a new 30,000-token prompt runs its prefill as one large pass. Every request already decoding must wait for that pass to finish. Users see streaming freeze, which shows up as a spike in inter-token latency.

This problem grows with context length. Agent and RAG workloads routinely send long prompts, so the stalls become frequent. Continuous batching fills the batch efficiently, but it does not by itself stop one big prefill from monopolising a step.

How does chunked prefill work?

The scheduler sets a token budget per step, controlled in vLLM by max_num_batched_tokens. It then fills that budget in priority order.

  1. Schedule all running decode requests first, one token each.
  2. Spend the remaining budget on prompt tokens from waiting requests.
  3. If a prompt exceeds the remaining budget, process only a chunk and resume next step.

Each step now has bounded work, so decode requests advance every step. The partially processed prompt keeps its progress in the KV cache, stored in blocks through paged attention. Chunk N reads the cache written by chunks 1 to N-1, so the final result is identical to unchunked prefill.

Chunked prefill vs prefill-decode disaggregation, what is the difference?

Both fix prefill interference, but at different layers.

AspectChunked prefillPrefill-decode disaggregation
Where phases runSame GPU, mixed in each stepSeparate GPU pools
KV cache movementNoneTransferred over the network
ComplexityLow, a scheduler settingHigh, needs routing and fast links
Best fitSingle node or small replica setsLarge fleets with strict latency goals

Start with chunked prefill. Consider disaggregation only when one pool cannot meet both latency targets.

What are the trade-offs of chunked prefill?

Chunking spreads a prompt over several steps, so a very long prompt can see a slightly later first token. It also re-reads the earlier KV cache for each chunk, adding some memory traffic.

In return, other users get stable inter-token latency, and the GPU stays busier because compute-bound prefill and memory-bound decode share each step. A small chunk protects decode smoothness. A large chunk finishes prompts sooner. Offline batch jobs usually prefer larger budgets, while interactive chat prefers smaller ones.

How do you tune chunked prefill in vLLM?

Set max_num_batched_tokens and measure. Track time to first token, inter-token latency percentiles, and throughput under your real prompt mix. Synthetic short prompts hide the effect entirely, because chunking only matters when long prompts arrive alongside active decodes.

How CloudRaft helps

CloudRaft tunes vLLM schedulers and GPU capacity for production inference on Kubernetes, measured against your own prompt lengths. Talk to our team.

Conclusion

Chunked prefill slices long prompts so decode work never waits for one giant pass. It keeps inter-token latency steady at a small cost in prompt completion time. It is the default in modern vLLM, so the real work is choosing a token budget that fits your traffic.