PagedAttention is the memory algorithm at the center of vLLM, and it is the biggest single reason self-hosted LLM inference reaches 2–4× the throughput of older serving systems on identical GPUs. It borrows virtual-memory paging from operating systems: rather than reserving one contiguous block of GPU memory per request, vLLM stores each request’s key-value (KV) cache in fixed-size pages allocated on demand, cutting wasted KV cache memory to near zero. For engineering teams paying per GPU-hour, that efficiency maps directly to fewer accelerators and lower per-token cost — provided the surrounding scheduler, batching, and caching settings are tuned to the real workload, not left at defaults.
How PagedAttention Reclaims GPU Memory
Before paging, LLM serving systems stored each request’s KV cache in a contiguous tensor sized to the request’s maximum possible length. The authors of the original PagedAttention paper profiled this approach and found that only 20.4%–38.2% of KV cache memory was used to store actual token states; the rest was consumed by internal fragmentation, external fragmentation, and over-reservation. PagedAttention divides the cache into blocks, each holding keys and values for a fixed number of tokens, and tracks them through a per-request block table — exactly as an operating system maps virtual pages to physical frames. Because all blocks share the same size, external fragmentation disappears, and on-demand allocation removes the internal waste from over-provisioned maximums. The same block granularity also enables copy-on-write sharing of KV cache across sequences, which is what makes parallel sampling and beam search cheap in vLLM.
Continuous Batching Keeps the GPU Busy
Autoregressive generation is memory-bandwidth bound: each decoding step reads the full model weights to emit one token, leaving most of the GPU’s compute idle. The fix is to pack many concurrent requests into the same forward pass — but naive static batching wastes capacity whenever a short request finishes and leaves a slot empty until the whole batch drains. vLLM uses continuous batching, also called iteration-level scheduling, introduced by the Orca system: the scheduler inspects the batch at every decoding step and can admit a new request or evict a finished one without waiting for the others. Combined with PagedAttention’s on-demand memory, this means the GPU is fed fresh work the moment space frees up. In current vLLM versions continuous batching is on by default; the practical lever is --max-num-seqs, which caps how many requests run concurrently. Setting it too low starves the GPU; setting it too high with a fixed KV budget increases preemption, where the scheduler evicts and later recomputes requests, hurting tail latency.
Tuning Batching, Prefill, and Cache
Three settings dominate production throughput, and they interact. --max-num-seqs sets peak concurrency; for a high-traffic OpenAI-compatible API on an 80 GB GPU, values in the 128–512 range are typical, sized so that KV cache pages are not exhausted. --max-num-batched-tokens governs how many tokens — across prefill and decode — the engine processes per step. In vLLM’s V1 engine, chunked prefill is enabled by default whenever possible: the scheduler prioritizes pending decode requests, then fills remaining token budget with prefill chunks, improving both inter-token latency and GPU utilization. The official tuning documentation recommends --max-num-batched-tokens > 8192 for best throughput on smaller models on large GPUs, while smaller values around 2048 reduce inter-token latency when streaming matters more than raw rate. The third lever is Automatic Prefix Caching (APC), set with enable_prefix_caching=True. APC stores the KV cache of previously processed prefixes so a new query sharing that prefix skips recomputation entirely — effective for repeated-document retrieval and multi-turn chat, where it can eliminate most prefill cost.
Beyond engine flags, compile-time and container choices compound. The batch composition of vLLM also shifts floating-point reduction order, which is why identical prompts at temperature 0 can still diverge — relevant when reproducibility matters, as documented for vLLM and SGLang nondeterminism. CUDA graphs plus torch.compile stack on top of PagedAttention for decode-step speedups by removing per-step CPU kernel-launch overhead, while teams that have moved inference to cost-optimized accelerators still face the same KV-cache tuning tradeoffs — as the Trainium2 cost analysis makes clear.
Deploying vLLM on Kubernetes
vLLM is straightforward to run on Kubernetes for scalable serving. The deployment needs a PersistentVolumeClaim and Secret for downloading and storing the model from Hugging Face, a Deployment referencing the vllm/vllm-openai image, and a ClusterIP Service exposing the OpenAI-compatible endpoint on port 8000. The critical operational detail is the readiness and startup probe: the model server must load weights and warm the KV cache before it can serve, and if the probe’s failureThreshold is too low the scheduler kills the container before it becomes ready. vLLM’s deployment guide recommends removing probes, measuring actual cold-start time, then setting a generous failureThreshold and initialDelaySeconds to match. For health checking in clusters, vLLM also exposes the standard gRPC Health Checking Protocol when launched with --grpc, which integrates with Kubernetes native gRPC probes since version 1.24.
Production checklist
Use this checklist when sizing a new vLLM deployment or triaging throughput regressions:
- Profile cold-start time with probes disabled, then set
initialDelaySecondsandfailureThresholdto clear that window. - Size
--max-num-seqsto expected peak concurrency, not to GPU memory limit, to avoid preemption storms. - Set
--max-num-batched-tokensabove 8192 for throughput-bound workloads; lower it toward 2048 to protect streaming latency. - Enable
enable_prefix_caching=Truewhen prompts share prefixes such as retrieval context, chat history, or system prompts. - Monitor Prometheus preemption counters; a rising trend means KV cache is undersized relative to concurrency.
- Allocate at least 2 + N physical CPU cores for N GPUs; the API server, engine core, and each worker run as separate processes.