GPU autoscaling on Kubernetes breaks for LLM inference because the Horizontal Pod Autoscaler cannot see accelerators and the node layer takes minutes to add one. The fix is a four-layer stack — DCGM exporter, Prometheus Adapter, HPA, and KEDA over Karpenter — because better GPU allocation sharpens placement without touching …
Editorial team 
Tuning vLLM KV Cache and Preemption in Production Serving
vLLM’s PagedAttention and iteration-level continuous batching are the two mechanisms that take an autoregressive transformer from the 20–40% GPU utilization typical under static batching toward the 2–4× throughput the SOSP 2023 paper reports. Production value does not come from enabling them — both are on by default — but from …
MoE Inference Costs 8.6x GPU Memory of Dense Models
In MoE inference, a 37B-active model can demand roughly 8.6× the GPU memory of a dense model with equivalent per-token compute, because every expert’s weights must stay resident in VRAM even when only a fraction fire on any given token. That single number is why your DeepSeek-V3 serving footprint needs …
Editorial team