The number of concurrent users a GPU handles during LLM inference is bounded by key-value cache memory, not by compute throughput. Autoregressive generation accumulates key-value state token by token, and that state must remain resident in GPU memory for the lifetime of each request. Every active request holds KV cache tensors in VRAM, and once that memory budget is exhausted no further requests enter the batch regardless of idle floating-point capacity. Three optimization techniques — PagedAttention, automatic prefix caching, and FP8 KV cache quantization — directly increase the number of simultaneous conversations a single GPU supports. Combined with disaggregated serving architectures now standard in vLLM and NVIDIA Dynamo, these memory-management strategies determine whether your infrastructure serves eight or forty concurrent users on identical hardware.
Why Decode Bottlenecks on Memory
LLM inference splits into two phases with opposite resource profiles. Prefill processes the entire prompt through dense matrix multiplies in a single forward pass, saturating tensor cores and making this phase genuinely compute-bound. Decode generates output tokens one at a time. To produce each token the GPU must read the full model weights and the growing KV cache from HBM while performing comparatively little arithmetic, which makes token generation fundamentally memory-bandwidth-bound. The NVIDIA H100 SXM5 and H200 illustrate this distinction concretely: both deliver identical BF16 compute throughput, yet the H200 with 141 GB at 4.8 TB/s serves far more concurrent users than the H100 with 80 GB at 3.35 TB/s because memory capacity and bandwidth, not peak FLOPs, set the concurrency ceiling during generation.
Calculating Your Real KV Budget
The KV cache cost per token follows a deterministic expression derived from transformer architecture. The formula is two multiplied by the number of transformer layers, the number of KV heads, the head dimension, and the bytes per element. The leading factor of two accounts for storing both key and value tensors. Grouped-query attention shrinks this product by using far fewer KV heads than query heads, which is why models like Llama 3 deploy eight KV heads alongside sixty-four query heads. For a seventy-billion-parameter model with eighty layers, eight KV heads, head dimension 128, and BF16 precision, the per-token cost is 327,680 bytes, approximately 0.31 megabytes. A single request running at 8,192 tokens of context consumes about 2.68 GB of VRAM; the same request at 128K tokens holds 42.9 GB, which exceeds half the total capacity of a flagship H200 GPU. After loading FP8-quantized weights at 70 GB and reserving ten percent for activations, one H200 leaves roughly 57 GB for KV cache, a ceiling of about twenty-one concurrent requests at 8K context before any optimization is applied.
PagedAttention Eliminates Memory Waste
PagedAttention, introduced with vLLM at SOSP 2023, applies operating-system virtual memory and paging techniques to KV cache management. Rather than allocating contiguous VRAM blocks for each request, the system divides GPU memory into fixed-size blocks that can be non-contiguously allocated and referenced through a block table, the same way an operating system maps logical pages to physical frames. This eliminates the internal fragmentation caused by over-reserving cache for maximum sequence length and the external fragmentation that leaves unusable gaps scattered across the memory pool. The original vLLM evaluation documented that existing serving systems allowed a large fraction of KV cache memory to be wasted through fragmentation and redundant duplication, directly limiting achievable batch size. PagedAttention reduced that waste to near-zero and improved throughput by two to four times over prior systems like FasterTransformer and Orca at equivalent latency, with the largest gains appearing for longer sequences and more complex decoding algorithms such as beam search and parallel sampling.
Prefix Caching Avoids Redundant Work
Automatic prefix caching prevents the recomputation of shared context across requests. In production LLM deployments, many requests carry common prefixes — system prompts, tool definitions, retrieved documents, conversation history — that produce identical KV cache blocks. The vLLM implementation identifies each cache block by a hash of its prefix tokens and block contents, then retains blocks whose reference count remains above zero rather than freeing them immediately. A least-recently-used eviction policy clears stale entries when memory pressure rises, preferring to free blocks with longer prefix lengths first. When a new request arrives whose prefix matches cached blocks, the scheduler reuses them directly, skipping redundant prefill. For workloads with shared system prompts and retrieval context, this eliminates most prefill computation.
FP8 Cache and When to Combine
Storing KV cache tensors in eight-bit floating-point format instead of BF16 directly halves the per-token memory cost. The vLLM team validated this approach across Hopper and Blackwell GPU architectures in April 2026, finding that for head dimensions of 64 and 128 the FP8 KV cache reduces the per-token decode cost to fifty-four percent of its BF16 counterpart while preserving near-baseline accuracy. For memory-bound generation this translates to roughly double the concurrent requests at the same VRAM budget. The same study exposed a critical accuracy pitfall: on a 128K needle-in-a-haystack task, an uncorrected FP8 attention kernel dropped retrieval accuracy from ninety-one percent to thirteen percent, a regression traced to imprecise FP32 accumulation in the tensor cores that was subsequently fixed with a two-level accumulation strategy. FP8 KV cache is recommended for head dimensions of 64 and 128 at contexts above seven thousand tokens; shorter contexts should remain on BF16.
The table below summarizes each technique:
| Technique | Concurrency Gain | Complexity | Accuracy Risk |
|---|---|---|---|
| PagedAttention | 2–4× throughput | Default in vLLM | None |
| Prefix caching | Up to 80% prefill cut | Enabled by default | None (identical outputs) |
| FP8 KV cache | ~2× (halved bytes/token) | Single flag | Monitor at 128K+ contexts |
These memory optimizations compose with kernel-level techniques like CUDA graph capture and architectural approaches like disaggregated serving, which separates prefill and decode onto dedicated GPU pools to eliminate head-of-line blocking under concurrent load. While disaggregation itself does not increase raw throughput, it prevents latency spikes that occur when a large prefill job stalls ongoing decode streams. A deployment combining PagedAttention, prefix caching, and FP8 KV cache on an H200 pushes effective concurrency well past the unoptimized ceiling. The recommended sequence: enable PagedAttention and prefix caching first — both are on by default in vLLM — then add FP8 KV cache for longer contexts, and reach for disaggregated serving when a single replica can no longer sustain your traffic.
Sources
- Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., SOSP 2023
- Automatic Prefix Caching RFC — vLLM GitHub
- The State of FP8 KV-Cache and Attention Quantization in vLLM
- Disaggregated Prefilling Documentation — vLLM
- How Many GPUs to Serve an LLM: Capacity Planning — The Wire