vLLM beats HuggingFace TGI on serving throughput by 3.67x at 100 concurrent requests on a LLaMA-2-7B model, and the gap stretches to 24x under extreme load, according to a November 2025 arXiv study that benchmarked both frameworks on LLaMA-2 models of 7B, 13B, and 70B parameters. TGI still returns the first token faster at the median, which keeps it attractive for chat interfaces. For cloud teams the practical reading is direct: pick vLLM for throughput-bound serving, and weigh roadmap risk heavily before any new TGI deployment, because Hugging Face has shifted the project to maintenance status.
What the study measured
The paper, published as arXiv:2511.17593 by Saicharan Kolluru in November 2025, compares vLLM v0.6.1 against TGI v2.3.0 on a node with four NVIDIA A100 80GB GPUs, an AMD EPYC 7763 processor and 512GB of RAM. LLaMA-2 at 7B and 13B runs on a single GPU, while the 70B model uses tensor parallelism across four GPUs. The evaluation goes beyond a tokens-per-second headline: it measures end-to-end latency percentiles, time-to-first-token (TTFT), time-per-output-token (TPOT), peak GPU memory, GPU utilization and scaling behavior as concurrency climbs. That breadth is what makes the results usable for capacity planning instead of framework debate.
Throughput results favor vLLM
Throughput is where the engines diverge hardest. On LLaMA-2-7B at 100 concurrent requests, vLLM peaked at 15,243 tokens per second against 4,156 for TGI, and the advantage widens to 24x at 200 concurrent requests as TGI’s scheduling saturates while vLLM keeps batching. The gap narrows as models grow — 2.8x for the 13B model and 2.1x for the 70B model with tensor parallelism — because distributed communication overhead taxes both engines in similar proportion. TGI’s 7B throughput saturates beyond roughly 50 concurrent requests, a symptom of memory pressure capping batch size, while vLLM scales close to linearly up to 100-150 requests before plateauing.
| Measurement | vLLM | TGI |
|---|---|---|
| LLaMA-2-7B throughput, 100 concurrent | 15,243 tok/s | 4,156 tok/s |
| LLaMA-2-13B throughput, optimal concurrency | 8,934 tok/s | 3,187 tok/s |
| LLaMA-2-70B throughput, 4-GPU tensor parallel | 3,245 tok/s | 1,544 tok/s |
| 7B p50 time-to-first-token, 25 users | 0.24 s | 0.18 s |
| 7B peak GPU memory | 24.3 GB | 31.7 GB |
Latency trade-offs split
Latency tells a more nuanced story than throughput. Under 25 concurrent users on the 7B model, TGI returned the first token faster at the median (0.18 seconds versus 0.24 for vLLM), which matters for perceived responsiveness in chat products. But vLLM finishes full generations sooner at every percentile: median total latency of 4.82 seconds versus 5.91, and a per-output-token time of 0.019 seconds against 0.023. The divergence grows at the tail, with vLLM holding a 1.5-1.7x edge on p99 total latency. In short, TGI wins the first token and loses the rest of the response.
Memory and GPU utilization
The root cause of the throughput gap is memory management. PagedAttention, the mechanism vLLM borrowed from virtual-memory paging in operating systems, stores attention keys and values in non-contiguous blocks, so waste happens only in the last block of each sequence. In this study vLLM sustained 85-92% GPU utilization under high concurrency while TGI peaked at 68-74%, with TGI’s memory ceiling capping batch sizes and leaving compute idle. PagedAttention cut measured memory consumption by 19-27%: the 7B model peaks at 24.3GB on vLLM versus 31.7GB on TGI, and the 13B at 42.8GB versus 54.2GB. Less memory per sequence means more concurrent sequences per GPU, which is precisely where the throughput advantage originates. If this decision feeds a hardware budget, the H200 and H100 GPU cost comparison covers the other half of the equation.
TGI is now maintenance-only
There is a strategic dimension the benchmark cannot capture. Hugging Face has moved TGI into maintenance mode and now recommends vLLM and SGLang as the engines to use going forward, accepting only minor bug fixes and documentation work on TGI itself. TGI’s architectural legacy is real: its push toward serving engines that rely on transformers model architectures was adopted by the very engines that superseded it, and recent TGI releases already use Flash Attention and paged attention internally. Still, for a new deployment the operational conclusion is hard to escape: the performance leader in this study is also the project with the active roadmap.
How to choose today
A checklist based on the measured profiles:
- Pick vLLM when batch throughput dominates: document processing, classification backfill, synthetic data generation, or any queue that tolerates first-token delay.
- Pick vLLM when GPU memory is the binding constraint; the 19-27% memory saving converts directly into larger batches or smaller nodes.
- Keep TGI only where it is already embedded and TTFT service levels are tight; plan migration toward vLLM or SGLang instead of a green-field TGI build.
- Re-run the benchmark on your own traffic mix, because prompt length, output length and concurrency profile move the crossover points in both directions.
Whichever engine wins, node sizing shapes the bill as much as the software layer does. Teams still mapping the terminology can start with the cloud AI decoding notes for builders before diving into serving-engine documentation.