GPU capacity planning for LLM inference is one equation followed by three checks: multiply the memory each replica needs (model weights plus KV cache per concurrent request) by the number of replicas your token-throughput target demands, then verify the result against your latency SLO, your provider quota, and a failover …
Editorial team 
KV Cache and Batching: Raising LLM Serving Throughput
LLM serving throughput is decided by two memory-side mechanics before any GPU upgrade matters: how the engine stores the key-value (KV) cache and how it refills batches during decoding. Profiling published with the vLLM paper found that in systems that pre-allocate contiguous cache for each request, only 20.4% to 38.2% …
vLLM vs TGI Benchmarks: Throughput Numbers That Matter
vLLM beats HuggingFace TGI on serving throughput by 3.67x at 100 concurrent requests on a LLaMA-2-7B model, and the gap stretches to 24x under extreme load, according to a November 2025 arXiv study that benchmarked both frameworks on LLaMA-2 models of 7B, 13B, and 70B parameters. TGI still returns the …
Clound IA Decoded: H200 and H100 GPU Costs Compared
Clound ia is a misspelled search for cloud AI, and the traffic behind the typo behaves like any other navigational query: people typing clound ia are usually trying to reach a managed AI platform or to price GPU capacity, not hunting for another definition. For engineering teams in Portugal and …
Kubernetes GPU Autoscaling: Why LLM Inference Stalls
GPU autoscaling on Kubernetes breaks for LLM inference because the Horizontal Pod Autoscaler cannot see accelerators and the node layer takes minutes to add one. The fix is a four-layer stack — DCGM exporter, Prometheus Adapter, HPA, and KEDA over Karpenter — because better GPU allocation sharpens placement without touching …
Attention-FFN Disaggregation: MoE Inference’s Next Step
Attention-FFN disaggregation (AFD) is the next architectural split beyond prefill-decode disaggregation for large mixture-of-experts (MoE) models: it separates the attention modules from the feed-forward expert modules onto distinct GPU pools inside the decode phase, fusing the extra activation transfer with the all-to-all dispatch that MoE expert parallelism already requires. The …
Tuning vLLM KV Cache and Preemption in Production Serving
vLLM’s PagedAttention and iteration-level continuous batching are the two mechanisms that take an autoregressive transformer from the 20–40% GPU utilization typical under static batching toward the 2–4× throughput the SOSP 2023 paper reports. Production value does not come from enabling them — both are on by default — but from …
Model Distillation: 32B Beats o1-Mini at Half the Cost
A 32-billion-parameter student model fine-tuned on DeepSeek-R1’s reasoning traces scored 72.6% Pass@1 on AIME 2024, beating OpenAI’s o1-mini (63.6%) while costing roughly an order of magnitude less per token to serve. Released with the DeepSeek-R1 checkpoints, it is the strongest production evidence yet that knowledge distillation—not larger GPUs—is the dominant …
LLM Inference Nondeterminism: Why Temperature 0 Fails You
LLM inference nondeterminism means identical prompts can return different outputs even at temperature 0, because dynamic batching changes the order of floating-point reductions inside GPU kernels — a property called batch invariance. Thinking Machines Lab measured 80 distinct completions from 1,000 identical requests on Qwen3-235B, and researchers documented up to …
CUDA Graphs + torch.compile: 1.65x LLM Decode Speedup
A single decode step for Llama 3.1 8B on an H100 SXM5 takes 8.4 milliseconds in eager mode. Capture that same forward pass as a CUDA graph and it drops to 5.1 milliseconds — a 1.65× speedup. The reason is not faster math; it is the elimination of CPU-side kernel-launch …
Managed or self-hosted AI inference: compare complete costs
Managed and self-hosted inference should be compared on the same accepted task, quality requirements and traffic pattern. A GPU hourly rate and an API token rate measure different things. Neither establishes a universal crossover or proves that one deployment beats every provider. Define what you are comparing A managed model …
Continuous Batching: Why 60% of Your GPU Sits Idle
Naive static batching leaves roughly 60% of an H100 GPU idle during LLM serving, because finished requests hold their slots until the slowest sequence in the batch completes. Continuous batching — iteration-level scheduling introduced in the Orca paper and now the default in vLLM, TensorRT-LLM and TGI — fixes this …