What Is Speculative Decoding?
Speculative decoding is an LLM inference technique that cuts latency by drafting several tokens cheaply and verifying them together. A small draft model proposes a short run of tokens. The large target model then checks all of them in one forward pass. It keeps the correct prefix and discards the rest. Output stays statistically identical to normal decoding, but fewer expensive passes are needed.
Why is standard LLM decoding slow?
Because each token needs one full forward pass of the model. Decoding is memory-bandwidth-bound: the GPU reads all model weights to produce a single token. Compute units sit mostly idle during that read.
Verifying several tokens in one pass costs almost the same as generating one. Speculative decoding uses that spare compute. Weight reads are amortized over multiple accepted tokens.
How does speculative decoding work?
The process repeats in a loop:
- Draft. A cheap mechanism proposes k candidate tokens.
- Verify. The target model scores all k positions in one parallel forward pass.
- Accept. Each draft token is accepted or rejected by comparing probabilities. Verification stops at the first rejection.
- Resample. The target model supplies a corrected token at the point of rejection, so every step yields at least one token.
The acceptance rule preserves the target model's output distribution. Speed depends on the acceptance rate: the more draft tokens survive, the fewer target passes per token.
What are the main speculative decoding methods?
| Method | Draft source | Trade-off |
|---|---|---|
| Separate draft model | A small model from the same family | Extra memory; needs a compatible tokenizer |
| N-gram or prompt lookup | Matches in the prompt or earlier output | No extra model; helps mainly on repetitive text |
| Medusa | Extra decoding heads on the target model | Needs head training; verifies a tree of candidates |
| EAGLE and EAGLE-2 | A lightweight head predicting hidden features | High acceptance rates; needs trained draft weights |
| Multi-token prediction (MTP) | Prediction layers built into the model itself | Only available in models trained with MTP |
Serving engines such as vLLM and SGLang, and distributed stacks like llm-d, NVIDIA Dynamo, and AIBrix, support several of these methods.
When does speculative decoding help, and when does it not?
It helps when decoding is memory-bound and the draft is accurate. That means low to moderate batch sizes, interactive chat, and predictable outputs such as code or structured text.
It helps less at high batch sizes. A busy GPU is already compute-bound, so verifying extra draft tokens competes with useful work. A poorly matched draft model also hurts, because rejected tokens waste compute. Measure speedup on your own prompts and load before enabling it in production.
How is speculative decoding different from other inference optimizations?
Speculative decoding reduces the number of sequential target-model steps. Other techniques attack different bottlenecks. KV cache management reduces memory per request. Batching raises throughput across requests. Quantization shrinks weights. These combine: a team can run quantized weights, paged cache, and speculative decoding together when serving LLMs on Kubernetes.
How CloudRaft helps
CloudRaft builds and tunes LLM inference platforms on Kubernetes, and benchmarks speculative decoding against your real traffic before you ship it. Talk to our team.
Conclusion
Speculative decoding trades spare GPU compute for lower latency without changing model output. It works best for latency-sensitive, low-batch serving with a well-matched draft. It is one tool among several, so validate it against your own workload. For background, see what is AI inferencing.