GPU autoscaling on Kubernetes breaks for LLM inference because the Horizontal Pod Autoscaler cannot see accelerators and the node layer takes minutes to add one. The fix is a four-layer stack — DCGM exporter, Prometheus Adapter, HPA, and KEDA over Karpenter — because better GPU allocation sharpens placement without touching …
Editorial team 
Attention-FFN Disaggregation: MoE Inference’s Next Step
Attention-FFN disaggregation (AFD) is the next architectural split beyond prefill-decode disaggregation for large mixture-of-experts (MoE) models: it separates the attention modules from the feed-forward expert modules onto distinct GPU pools inside the decode phase, fusing the extra activation transfer with the all-to-all dispatch that MoE expert parallelism already requires. The …
Tuning vLLM KV Cache and Preemption in Production Serving
vLLM’s PagedAttention and iteration-level continuous batching are the two mechanisms that take an autoregressive transformer from the 20–40% GPU utilization typical under static batching toward the 2–4× throughput the SOSP 2023 paper reports. Production value does not come from enabling them — both are on by default — but from …
Model Distillation: 32B Beats o1-Mini at Half the Cost
A 32-billion-parameter student model fine-tuned on DeepSeek-R1’s reasoning traces scored 72.6% Pass@1 on AIME 2024, beating OpenAI’s o1-mini (63.6%) while costing roughly an order of magnitude less per token to serve. Released with the DeepSeek-R1 checkpoints, it is the strongest production evidence yet that knowledge distillation—not larger GPUs—is the dominant …
LLM Inference Nondeterminism: Why Temperature 0 Fails You
LLM inference nondeterminism means identical prompts can return different outputs even at temperature 0, because dynamic batching changes the order of floating-point reductions inside GPU kernels — a property called batch invariance. Thinking Machines Lab measured 80 distinct completions from 1,000 identical requests on Qwen3-235B, and researchers documented up to …
CUDA Graphs + torch.compile: 1.65x LLM Decode Speedup
A single decode step for Llama 3.1 8B on an H100 SXM5 takes 8.4 milliseconds in eager mode. Capture that same forward pass as a CUDA graph and it drops to 5.1 milliseconds — a 1.65× speedup. The reason is not faster math; it is the elimination of CPU-side kernel-launch …
NVIDIA NIM Savings: When Self-Hosting Beats LLM APIs
A single NVIDIA H100 GPU running a self-hosted NIM container costs roughly $1,950 per month on RunPod at $2.69 per hour, yet serves the same OpenAI-compatible /v1/chat/completions endpoint as GPT-4.1 — which bills $6 per million blended tokens. The crossover where NIM beats every per-token API sits around 300–500 million …
Continuous Batching: Why 60% of Your GPU Sits Idle
Naive static batching leaves roughly 60% of an H100 GPU idle during LLM serving, because finished requests hold their slots until the slowest sequence in the batch completes. Continuous batching — iteration-level scheduling introduced in the Orca paper and now the default in vLLM, TensorRT-LLM and TGI — fixes this …
Prefill-Decode Disaggregation: NVIDIA’s 7x Inference Fix
Every LLM inference request is two workloads pretending to be one. Prefill processes your entire prompt in a single compute-bound forward pass — it wants raw FP8 TFLOPS. Decode generates tokens one at a time by streaming KV cache tensors through memory — it wants HBM bandwidth. Running both on …
Prefill Decode Disaggregation Doubles Your LLM Throughput
Prefill-decode disaggregation separates the two phases of LLM inference — prompt processing and token generation — onto dedicated GPU pools, eliminating the head-of-line blocking that causes latency spikes under concurrent load. Production deployments report 1.5x to 2.5x throughput gains, with cache-aware variants like Together AI’s CPD pushing improvements to 40%. …
Editorial team 
vLLM vs SGLang: Which Engine Actually Wins in 2026?
On H100 SXM5 80GB running Llama 3.3 70B Instruct at FP8, SGLang serves 1,920 tokens per second at 50-way concurrency — just 3.8% faster than vLLM’s 1,850. But swap to Llama 3.1 8B, and that gap explodes to 29%: SGLang hits 16,200 tok/s versus vLLM’s 12,500. The inference engine you …
Serverless GPU Cold Starts Take 40s – Here’s How to Fix
The 1000x Latency Gap A cold-start instance on a serverless GPU platform produces its first token after more than 40 seconds. A warm instance generates subsequent tokens in roughly 30 milliseconds. That is a latency ratio of over 1,300:1 between the cold and warm states, and it is the single …