LLM inference nondeterminism means identical prompts can return different outputs even at temperature 0, because dynamic batching changes the order of floating-point reductions inside GPU kernels — a property called batch invariance. Thinking Machines Lab measured 80 distinct completions from 1,000 identical requests on Qwen3-235B, and researchers documented up to …
Editorial team 
CUDA Graphs + torch.compile: 1.65x LLM Decode Speedup
A single decode step for Llama 3.1 8B on an H100 SXM5 takes 8.4 milliseconds in eager mode. Capture that same forward pass as a CUDA graph and it drops to 5.1 milliseconds — a 1.65× speedup. The reason is not faster math; it is the elimination of CPU-side kernel-launch …
Continuous Batching: Why 60% of Your GPU Sits Idle
Naive static batching leaves roughly 60% of an H100 GPU idle during LLM serving, because finished requests hold their slots until the slowest sequence in the batch completes. Continuous batching — iteration-level scheduling introduced in the Orca paper and now the default in vLLM, TensorRT-LLM and TGI — fixes this …
Prefill-Decode Disaggregation: NVIDIA’s 7x Inference Fix
Every LLM inference request is two workloads pretending to be one. Prefill processes your entire prompt in a single compute-bound forward pass — it wants raw FP8 TFLOPS. Decode generates tokens one at a time by streaming KV cache tensors through memory — it wants HBM bandwidth. Running both on …
vLLM vs TensorRT-LLM vs SGLang: H100 Benchmarks 2026
Choosing between vLLM, TensorRT-LLM, and SGLang in 2026 comes down to three questions: how many models you serve, how fast you need to go live, and whether your workload shares prefixes. Benchmarks on H100 80GB with Llama 3.3 70B at FP8 show TensorRT-LLM delivering 13% higher throughput than vLLM at …