Prefill-decode disaggregation separates the two phases of LLM inference — prompt processing and token generation — onto dedicated GPU pools, eliminating the head-of-line blocking that causes latency spikes under concurrent load. Production deployments report 1.5x to 2.5x throughput gains, with cache-aware variants like Together AI’s CPD pushing improvements to 40%. …
Editorial team 
vLLM vs SGLang: Which Engine Actually Wins in 2026?
On H100 SXM5 80GB running Llama 3.3 70B Instruct at FP8, SGLang serves 1,920 tokens per second at 50-way concurrency — just 3.8% faster than vLLM’s 1,850. But swap to Llama 3.1 8B, and that gap explodes to 29%: SGLang hits 16,200 tok/s versus vLLM’s 12,500. The inference engine you …
GPU Schedulers Waste 38% Time on Agent Cache Regeneration
Agent Cache Rebuilds Waste 38% GPU When researchers at the University of Hong Kong instrumented a 32-GPU A100 cluster running SWE-bench coding agents on vLLM v0.6.0, they found a number that should bother every platform engineer: 38% of total execution time was spent regenerating KV cache that had been discarded …
Speculative Decoding Cuts MoE Inference Cost by 19%
19% Cheaper Inference, Zero Accuracy Loss Red Hat benchmarked gpt-oss-120B with Eagle3 speculative decoding on vLLM v0.13.0 and measured a 19.4% reduction in cost per 1M output tokens on H200 GPUs running SWE-bench workloads. Not a benchmark trick — production throughput at 200 concurrent requests, with output distributions mathematically identical …