vLLM’s PagedAttention and iteration-level continuous batching are the two mechanisms that take an autoregressive transformer from the 20–40% GPU utilization typical under static batching toward the 2–4× throughput the SOSP 2023 paper reports. Production value does not come from enabling them — both are on by default — but from …
Editorial team 
Continuous Batching: Why 60% of Your GPU Sits Idle
Naive static batching leaves roughly 60% of an H100 GPU idle during LLM serving, because finished requests hold their slots until the slowest sequence in the batch completes. Continuous batching — iteration-level scheduling introduced in the Orca paper and now the default in vLLM, TensorRT-LLM and TGI — fixes this …