Speculative decoding cuts LLM inference latency by up to 2x without changing a single token of output. A small draft model proposes multiple future tokens, the target model verifies them all in one forward pass, and only verified tokens are emitted — making the speedup mathematically lossless. Production benchmarks on vLLM confirm consistent throughput gains on mixture-of-experts models like Kimi-K2.5 and MiniMax-M2.5, and the technique is now built into every major serving framework including vLLM and SGLang.
How Speculative Decoding Works
Autoregressive LLM decoding generates one token per forward pass. For a 1,000-token response, the serving engine runs roughly 1,000 sequential decode iterations through the full model weights, attention layers, and KV cache machinery. This sequential pattern is the single largest latency bottleneck in LLM serving today, and the GPU is often underutilized because memory bandwidth, not compute, limits each decode step.
Speculative decoding breaks the one-token-at-a-time constraint through a draft-and-verify loop:
- A small, fast draft model proposes the next K candidate tokens in parallel.
- The target model verifies all K tokens in a single forward pass.
- The longest matching prefix is accepted and emitted; the rest are discarded.
- The next draft round starts from the first rejected token.
Every emitted token has been checked by the target model before reaching the user. The speedup is lossless because the output distribution of the target model is preserved exactly — speculative decoding changes only how many tokens the engine produces per forward pass, not which tokens it produces. This guarantee distinguishes it from quantization or distillation, which trade output fidelity for speed.
What Controls the Speedup
Three metrics determine whether speculative decoding helps or hurts your workload. Acceptance rate is the probability that the target model agrees with each draft token. Speculative token count is how many tokens the draft proposes per step. Acceptance length is the average number of tokens accepted per verification round, which follows the theoretical formula based on acceptance rate and speculative token count.
| Metric | Symbol | What it measures | Practical target |
|---|---|---|---|
| Acceptance rate | α | Probability the target accepts each draft token | ≥ 0.6 |
| Speculative token count | γ | Tokens the draft proposes per step | 3–5 |
| Acceptance length | τ | Average tokens accepted per round | Higher is better |
Acceptance rate is the dominant factor. At α ≥ 0.6 and γ ≥ 5, informal vLLM benchmarks achieved 2–3× speedups over baseline decoding. Latency dropped and throughput rose almost linearly with α, while increasing γ only helped when τ was already high — otherwise the extra draft tokens wasted compute on verification that produced rejections. Latency-sensitive applications like chatbots and code completion benefit most because the inter-token latency reduction is felt directly by end users.
Draft Model Quality Is Everything
The acceptance rate depends entirely on how closely the draft model aligns with the target model’s token distribution. Feature-level methods like EAGLE-3 train a lightweight draft module that combines low-, mid-, and high-level semantic features extracted from the target model itself, rather than relying on an unrelated small language model. This tight coupling produces candidates that the verifier is far more likely to accept.
On AMD Instinct MI355X GPUs, EAGLE-3 delivered 1.69× to 2.00× throughput gains on Kimi-K2.5 and up to 1.79× on MiniMax-M2.5 in standardized 1K/1K token sweeps. The same pipeline supports MXFP4 quantization for both target and draft through AMD Quark, reducing memory footprint while keeping the speedup intact. Out-of-the-box draft models work adequately for general chat workloads but struggle with domain-specific tasks or very long contexts. If your workload has unique characteristics, fine-tuning a draft model on representative data produces measurably higher acceptance rates and faster decode cycles.
Memory and Convergence Tradeoffs
You must load both the draft model and the target model into GPU memory simultaneously. On a single GPU, this squeezes space for concurrent requests and can degrade throughput under heavy load. With tensor parallelism across two or more GPUs, the memory pressure eases substantially — speculative decoding maintained its advantage over baseline even at 50 concurrent requests with TP = 2, and time-per-output-token improved by roughly 2×. However, higher speculative token counts (γ = 5) can produce latency spikes under high concurrency that require careful tuning of batch sizes and request scheduling.
For production deployment, follow this checklist before enabling speculative decoding:
- Benchmark under your actual concurrency, sequence lengths, and batching configuration — not synthetic workloads.
- Measure acceptance rate on representative prompts; below 0.5 the overhead rarely pays off.
- Pair with prefill-decode disaggregation to isolate decode-phase gains from prefill-induced latency stalls.
- Validate GPU memory headroom for both models, especially on single-GPU deployments.
- Compare self-hosted cost per token against managed APIs once throughput stabilizes.
The combination of speculative decoding and quantization is where the largest practical gains converge: a well-trained EAGLE-3 draft paired with FP4 or FP8 quantization can push effective throughput to 3–4× over an unoptimized baseline, while remaining bit-exact to the target model’s output distribution.