Speculative decoding lets a large language model emit tokens faster by drafting candidates with a small, cheap model and verifying them in a single forward pass of the target. Google researchers measured a 2X-3X acceleration on T5-XXL with identical outputs, without retraining or architecture changes, and the technique now ships in mainstream serving stacks. The gain is conditional, not universal: vLLM documents speculative decoding as a way to reduce inter-token latency under medium-to-low QPS, memory-bound workloads. On saturated GPUs that already batch heavily, speculation can add overhead instead of removing it, so treat it as a latency tool for low-concurrency paths such as interactive chat, not a blanket throughput upgrade.
How the verification loop works
A draft model or lightweight proposer generates a short block of candidate tokens. The target model scores all candidates in one parallel forward pass, accepting every token that agrees with what it would have sampled and stopping at the first mismatch. Accepted tokens are effectively free: the cost of streaming the big model’s weights through memory is amortized across several tokens at once, which is exactly why the trick pays off on workloads where weight loading dominates compute. The original Google paper frames the scheme as exact decoding that runs the large model in parallel on a smaller model’s proposals, potentially generating several tokens concurrently without changing the output distribution.
Hugging Face’s Transformers implementation guide is explicit about quality: there is no degradation in accuracy because the verification forward pass ensures the same outputs are generated as if the LLM had generated them on its own. The same guide lists the constraints that decide whether the library fits your stack: the assistant and the target must share the same tokenizer, and assisted generation supports greedy search and sampling but not batched inputs. Those limits, plus the operational cost of co-hosting a second model, are why production teams usually land on a serving engine rather than calling generate() directly.
Choosing a speculation method
Method choice is a traffic-pattern decision. vLLM supports model-based methods such as EAGLE, MTP and draft models, alongside lighter n-gram and suffix speculation, and its documentation ranks the model-based family as the best latency reducers while noting that n-gram and suffix options give modest speedups without adding work during peak traffic. Use the qualitative comparison below as a starting point, then validate on your own hardware and sampling settings:
| Method | Low QPS gain | High QPS gain | Notes |
|---|---|---|---|
| EAGLE | High | Medium to high | Strong general-purpose model-based method |
| MTP | High | Medium to high | Best when the target model has native MTP support |
| Draft model | High | Medium | Needs a separate draft model with a shared tokenizer |
| N-gram | Low to medium | Medium | Lightweight and easy to enable |
| Suffix decoding | Low to medium | Medium | No extra draft model; dynamic speculation depth |
| Dynamic speculation | High | Higher than base method | Suits RL workloads or fluctuating QPS |
Configuring vLLM step by step
- Capture a baseline first: TTFT and inter-token latency under your production sampling settings, at the concurrency levels you actually serve.
- Launch with an explicit speculative configuration, for example: vllm serve target-model –speculative-config ‘{“method”: “draft_model”, “model”: “draft-model”, “num_speculative_tokens”: 5}’. The same keys are accepted from Python through LLM(speculative_config={…}), so the setting can live in deployment code instead of CLI flags.
- Reproduce measurements with vLLM’s spec_decode_offline.py example or the benchmark CLI guide instead of ad-hoc single-request timing; single samples exaggerate gains.
- Tune speculation depth against acceptance: more speculative tokens raise the per-step ceiling but waste verification compute whenever acceptance sags. Five is a common starting point, not a law.
- Verify feature compatibility before rollout: pipeline parallelism is not composable with speculative decoding as of vllm<=0.15.0, so multi-node pipeline setups need a different plan.
Measuring acceptance and pitfalls
The number that governs your return is the acceptance rate: the share of drafted tokens the target actually keeps. High acceptance means near-free tokens; low acceptance means you pay draft cost plus full verification and bank almost nothing. vLLM exposes per-request acceptance metrics for exactly this loop. Feed those signals into your existing tracing and evaluation setup and diff outputs on a golden prompt set with speculation on and off before widening rollout.
Two caveats keep expectations honest. Although the method is designed to be lossless, vLLM’s documentation warns that floating-point precision differences and batch-size-driven numerical instability can produce slight output variations, and vLLM does not guarantee stable logprobs across runs — so identical output behavior is a design goal you verify, not assume. And draft models are coupled artifacts: a draft head trained for one target family ages badly when the target is upgraded, so keep speculation assets inside your model deprecation planning cycle instead of treating them as set-and-forget infrastructure.
Sources
- Fast Inference from Transformers via Speculative Decoding (arXiv:2211.17192) — Leviathan, Kalman and Matias, Google Research.
- vLLM documentation: Speculative Decoding — methods, configuration schema and compatibility notes.
- Hugging Face Transformers: Optimizing inference — assisted generation constraints and quality notes.