What Is Continuous Batching?
Continuous batching is an LLM serving technique that rebuilds the batch at every decoding step. When a request finishes, it leaves at once, and a waiting request takes its place. The GPU keeps running full batches instead of idling while the longest request completes. Throughput rises, and model output stays identical.
The idea was introduced as iteration-level scheduling in the 2022 Orca paper. NVIDIA calls it in-flight batching. vLLM, SGLang, and Hugging Face TGI all implement it.
Why does continuous batching matter for LLM inference?
Because LLM requests have unpredictable lengths. One user asks for a ten-token answer, and another triggers a two-thousand-token essay. A scheduler that ignores this wastes expensive GPU time.
Decoding is also memory-bound. Each step reads the full model weights from GPU memory to produce one token per request. Adding more requests to the step costs little extra, so a fuller batch raises throughput almost for free. Continuous batching keeps batches full at every step. Anyscale reported up to 23 times higher throughput than naive static batching in its vLLM-based tests. Treat that figure as workload-dependent.
How does continuous batching work?
The scheduler runs one loop. At each iteration it decides which requests join the next forward pass:
- Admit. If cache memory and batch limits allow, waiting requests join the running set.
- Step. The engine runs one forward pass and emits one new token per running request.
- Evict. Requests that hit an end-of-sequence token or a length limit leave immediately.
New requests first run a prefill pass over their whole prompt. After that they join the decode batch. Because the batch changes every step, the engine cannot assume uniform sequence lengths. It relies on paged KV cache management to store each request's cache without contiguous buffers.
Static vs continuous batching, what is the difference?
The difference is when the batch is allowed to change.
| Property | Static batching | Continuous batching |
|---|---|---|
| Batch changes | Only between whole batches | At every decoding step |
| Finished requests | Wait for the longest sequence | Leave immediately |
| New requests | Wait for the next batch | Join at the next iteration |
| GPU utilization | Drops as sequences finish | Stays high while traffic is queued |
| Sensitive to output length | Very, wasted compute on padding | Barely, slots are reused |
Static batching still suits offline jobs with uniform lengths. For interactive serving with mixed lengths, continuous batching is the standard choice.
What problems does continuous batching create?
Two interactions need attention. The first is memory. Every running request holds KV cache blocks, so cache capacity caps the batch size before compute does. When memory runs out, the scheduler must queue or preempt requests.
The second is prefill interference. A long prompt can monopolise a forward pass and stall decoding for everyone else. This shows up as spikes in inter-token latency. Chunked prefill splits long prompts into pieces and mixes them with decode work. Continuous batching also pairs with speculative decoding, which raises tokens per step on top of the better batch fill.
How do you tune continuous batching in vLLM?
Three limits matter most. max_num_seqs caps concurrent sequences. max_num_batched_tokens caps the token budget per step. gpu_memory_utilization sets how much GPU memory goes to weights plus cache.
Raise the limits to push throughput. Lower them to protect latency. Measure time to first token and inter-token latency on your own traffic before settling.
How CloudRaft helps
CloudRaft builds and tunes LLM inference platforms on Kubernetes, including batch limits and scheduler settings measured against your latency targets. Talk to our team.
Conclusion
Continuous batching changes the batch at every decoding step, so finished requests free slots immediately. GPUs stay busy, and throughput rises for mixed-length traffic. Memory capacity and prefill spikes are the limits to tune.