KV Cache Memory Sizing for Long-Context LLM Serving

KV cache memory — not model weights — is usually the constraint that decides how many long-context requests a GPU can serve concurrently. The key-value cache for each request is huge and grows and shrinks dynamically, and when it is managed inefficiently, fragmentation and redundant duplication limit the batch size; that is the core finding behind PagedAttention, the mechanism vLLM was built on (arXiv:2309.06180). Before you raise a context limit from 8K to 128K, run one calculation: a per-token formula tells you how much HBM every sequence consumes, and two allocation techniques — paged block allocation and prefix caching — decide how much of that budget you actually waste.

The Per-Token KV Formula

Bytes per token = 2 × layers × KV heads × head dimension × bytes per element. The factor of 2 accounts for the key tensor and the value tensor; the layer count, the KV head count and the head dimension come straight from the model’s config file; bytes per element is 2 for FP16 or BF16 and 1 for INT8. Use the KV head count, not the query head count — under grouped-query attention the cache depends only on how many KV heads the model carries, which is why two models with the same parameter count can have very different serving footprints.

Worked example (my own arithmetic on a hypothetical config, not a vendor measurement): 32 layers, 8 KV heads, head dimension 128, BF16 storage. That gives 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KiB per token. Multiply by context length and by concurrency to get the serving footprint:

The table assumes the example config above; swap in your model’s config values to reproduce it for your stack. The uncomfortable row is the last one: a batch of sixteen 128K-context requests needs 256 GiB of KV state alone, before weights and activation workspace claim their share of the same HBM.

Why Long Contexts Explode Memory

KV cost is linear in context length and linear in concurrency, and it stacks on top of a mostly fixed budget. On one accelerator you divide memory three ways: model weights (fixed at load time), activation workspace (roughly stable per step), and KV cache (the only term that scales with context multiplied by requests). Doubling the context window doubles the KV bill per sequence; raising concurrency multiplies it again; both levers together can push a serving plan off a GPU class that comfortably handled short prompts.

This is the same budget arithmetic that decides whether a managed GPU platform fits a workload at all — the sizing discipline we walked through for Cloud Run GPU inference limits and costs applies, with KV state as the dominant variable. When the budget does not close, you have three levers, in order of least user pain: shorten effective contexts (retrieval instead of stuffing the prompt), pick a model with fewer KV heads, or quantize the KV cache to 1 byte per element instead of 2.

How PagedAttention Removes Waste

A naive allocator reserves contiguous memory for each request’s maximum possible context, so a request that finishes early strands its reservation until the batch drains. vLLM’s answer borrows from operating systems: each block contains the attention keys and values for a fixed number of tokens, and the blocks can be stored in non-contiguous physical memory, with allocation on demand and no per-request reservation (vLLM design notes, kernel design). Blocks are handed out only as tokens are actually processed, so short answers stop paying for context they never reached.

The measured effect is large: near-zero waste in KV cache memory, and the PagedAttention paper reports that vLLM improves the throughput of popular LLMs by 2-4× at the same latency level compared with earlier systems such as FasterTransformer and Orca (arXiv:2309.06180), with the gap widening on longer sequences and more complex decoding.

Prefix Caching Reuses Shared Prompts

Blocks also make reuse cheap. vLLM identifies every KV block by a hash of the tokens inside it plus the prefix before it, so two requests that begin with the same system prompt, RAG preamble or few-shot block map onto the same physical blocks; requests that share a system prompt can reuse the cached KV blocks directly without recomputation (vLLM docs). When free blocks run out, the engine evicts blocks with zero references, preferring the least recently used, which turns the KV cache into an ordinary memory cache with predictable behavior.

One operational caution: cached KV state is plaintext tensor data resident in GPU memory, not an encrypted artifact. If your workload runs under hardware attestation requirements, the same lifecycle rules apply to cached blocks as to the weights themselves — see our setup notes on confidential computing attestation for AI inference.

A Five-Step Sizing Checklist

1. Read layers, num_kv_heads and head_dim from the model config; never guess from a marketing context limit.
2. Compute bytes per token with the formula above; sanity-check it against the serving engine’s reported cache size at startup.
3. Multiply by maximum context length and by target concurrency to get the KV budget per node.
4. Subtract weights and activation workspace from usable HBM; if the remainder is smaller than the KV budget, cut concurrency, then context, then precision.
5. Enable prefix caching, measure the block hit rate on real traffic, and re-run the budget with measured reuse before buying more GPUs.

The checklist turns a capacity panic into arithmetic: once you know bytes per token, every scaling decision — context, concurrency, precision — is a multiplication you can defend in a design review.

Sources

• Efficient Memory Management for Large Language Model Serving with PagedAttention — arXiv:2309.06180
• Automatic Prefix Caching — vLLM documentation
• Paged Attention kernel design — vLLM documentation