vLLM’s PagedAttention and iteration-level continuous batching are the two mechanisms that take an autoregressive transformer from the 20–40% GPU utilization typical under static batching toward the 2–4× throughput the SOSP 2023 paper reports. Production value does not come from enabling them — both are on by default — but from sizing the KV cache pool, choosing the right preemption mode, and tuning the prefill/decode token budget so the GPU never stalls on swapped sequences or recomputed prefills. This guide walks through the knobs that move serving throughput and the failure modes they protect against, with the configuration values that matter for a vLLM deployment in 2026. Both mechanisms are mature — the project now lists over 2000 contributors and 200-plus supported model architectures — so the differentiator at the deployment level is configuration discipline, not feature availability.
Why KV cache dominates GPU memory
For a 13B-parameter model on an A100-40GB, the vLLM paper measured approximately 65% of memory consumed by model weights and roughly 30% reserved for KV cache; only a small remainder is activation. The KV portion grows and shrinks per active sequence, which is why fragmentation and over-reservation dominate the cost equation rather than raw compute. For Llama-3-70B in FP16, with 80 layers, 8 grouped-query attention heads, a 128 head dimension, and 2 bytes per element, the per-token cost works out to about 160 KB across all layers. A 2048-token context therefore reserves roughly 327 MB per request whether or not those tokens are ever generated, and a request that finishes in 200 tokens still locks the full reservation. This static reservation is the root cause of the 60–80% waste the original paper measured for existing systems under contiguous pre-allocation, and it is the problem PagedAttention was designed to eliminate.
How PagedAttention reaches 4% waste
The PagedAttention system applies classical operating-system virtual memory ideas to KV cache management. Each active sequence addresses its cache through a logical block table that maps to non-contiguous physical blocks; a block holds the K and V tensors for 16 contiguous tokens by default, configurable via the –block-size flag. Blocks are allocated on demand as tokens are generated, and the gather happens through a custom CUDA kernel at each attention step, selecting FlashAttention-2 or FlashAttention-3 based on GPU capability. Copy-on-write lets parallel sampling and beam search share a parent’s physical blocks by reference and only allocate new blocks as each candidate diverges, returning freed references to the pool with zero copy overhead. Fragmentation shrinks to a maximum of 15 wasted token slots per sequence in the partially filled last block, dropping measured waste to approximately 4% compared with the 60–80% measured under contiguous pre-allocation. One caveat for latency-sensitive single-request workloads: the block-table gather has non-trivial overhead at batch size 1, where a contiguous FlashAttention-2 access pattern can have an edge.
Sizing gpu_memory_utilization correctly
vLLM pre-allocates the KV cache pool as a fraction of total GPU memory controlled by –gpu-memory-utilization, and raising that fraction yields more physical block slots, headroom for max_num_seqs, and lower preemption pressure. Lowering it leaves headroom for co-tenants, activation spikes, or multi-LoRA adapters, but caps concurrency and throughput. To size it deliberately, read the startup log, where vLLM prints the measured KV cache memory value and the total number of blocks it reserved. If total blocks is the binding constraint at your p99 arrival rate, the vLLM optimization docs prescribe four levers in order: raise gpu_memory_utilization, decrease max_num_seqs or max_num_batched_tokens, raise tensor_parallel_size to shard weights, or raise pipeline_parallel_size to spread layers across GPUs. For multi-node or topology-aware placement concerns, pair this with the gang-scheduling and autoscaling patterns covered in our advanced Kubernetes scaling techniques overview.
Preemption modes and the swap penalty
When the KV pool fills beyond its threshold, the scheduler preempts running sequences to free space. vLLM V1 defaults to recompute mode, which drops the sequence and reruns prefill from scratch on re-admission — zero PCIe bandwidth cost, but additional GPU compute on restore. The alternative is swap mode, which serializes a preempted sequence’s physical blocks to CPU DRAM and moves them back over PCIe on re-admission; on a PCIe 4.0 x16 link at roughly 32 GB/s, restoring a long-context sequence on a 70B model can take hundreds of milliseconds.
| Preemption mode | Where state goes | Restore cost | Best for |
|---|---|---|---|
| Recompute (V1 default) | Dropped; prefill rerun | Extra GPU compute, no PCIe transfer | Short prompts, plentiful GPU compute |
| Swap | Serialized to CPU DRAM | PCIe 4.0 x16 (~32 GB/s) copy back; hundreds of ms on long contexts | Long-context workloads where recomputing prefill is expensive |
Sustained preemption is a degradation state rather than a normal operating mode. If the vllm:num_preemptions_total Prometheus counter grows steadily under nominal load, the remedy is to reduce –max-model-len, raise –gpu-memory-utilization, or add capacity before tail latency drifts into SLA breach.
Chunked prefill tuning checklist
Chunked prefill is enabled by default in vLLM V1 whenever possible and lets the scheduler batch decode (memory-bound) and prefill (compute-bound) tokens in the same forward pass. The governing knob is max_num_batched_tokens, whose V0 default is 2048 and which V1 tunes per model and GPU. Lower values improve inter-token latency by interrupting decodes less often; higher values improve time-to-first-token and overall throughput because more prefill tokens fit per step. The official optimization docs recommend setting max_num_batched_tokens above 8192 for throughput, especially for smaller models on large GPUs.
- Start at max_num_batched_tokens = 8192 and measure time-to-first-token plus inter-token latency under your real traffic mix.
- Raise toward max_model_len for batch-oriented workloads that tolerate higher inter-token latency.
- Lower toward 2048 for interactive chat where token-to-token latency is the user-visible metric.
- Re-run a load test after every change; chunked prefill shifts the TTFT/ITL tradeoff, it does not remove it.
- For disaggregated prefill and decode on separate GPU pools, apply the same checklist per pool — see our prefill-decode disaggregation guide.