For most GPU inference deployments, FP8 weight-and-activation quantization is the default pick on Hopper or newer hardware: each weight is stored in one byte instead of two, roughly halving weight memory against BF16 while the quality impact is usually small enough to verify with an afternoon of evaluation. INT8 W8A8 is the fallback for Ampere cards and CPU serving, and INT4 weight-only formats buy the most model-per-GPU headroom when the quality budget allows. vLLM names LLM Compressor as its starting library for quantization, with support for FP8, INT8 and INT4 formats alongside AutoAWQ, BitsAndBytes and GPTQModel, so in practice the decision reduces to hardware support, memory arithmetic and a measured quality check.
What quantization changes
Quantization shrinks a model by storing its weights, and sometimes its activations, in a lower-precision numeric format. The BF16 weights you download from a hub are replaced by 8-bit or 4-bit representations with scaling factors that map compressed values back into a usable range at inference time. Hugging Face describes quantization as lowering the memory requirements of loading and using a model by storing the weights in a lower precision while trying to preserve as much accuracy as possible, and its documentation notes that some methods compress all the way to integer representations such as int8 or int4.
The practical consequence is that a quantized checkpoint moves three deployment variables at once: how many model copies fit on a card, how much room remains for the KV cache that serves concurrent requests, and how much task quality you give up. The first two are pure arithmetic you can do before touching a GPU. The third is workload-dependent and must be measured, never assumed from a model card.
The memory arithmetic per format
Weight memory for a dense model is parameters multiplied by bytes per parameter. The table applies that arithmetic to a 70-billion-parameter model as a sizing example — a calculation, not a measured benchmark:
| Format | Bytes per weight | Approx. weight memory | Where it runs |
|---|---|---|---|
| BF16 baseline | 2 | ~140 GB | Ampere and newer |
| INT8 / FP8 | 1 | ~70 GB | FP8: Ada, Hopper, newer; INT8: broader |
| INT4 weight-only | 0.5 | ~35 GB | Ampere-class and newer GPUs with W4A16 kernels |
The freed VRAM becomes KV cache budget: at INT4, one card holds a larger model or sustains many more concurrent sequences, which is the cheapest capacity win available before any kernel tuning. The arithmetic covers only weights, though — activations, scale factors and framework overhead add a few percent, so size with margin instead of to the byte. If weights plus KV cache still do not fit after weight quantization, quantizing the KV cache itself is the next lever, not a bigger card.
Hardware support decides the recipe
Format choice is constrained before it is a preference. In vLLM, the compatibility chart marks llm-compressor FP8 (W8A8) as supported on Ada, Hopper and AMD GPUs, and unsupported on Volta, Turing, Ampere and the CPU columns — FP8 matrix kernels require hardware that introduced the datatype, which is why an A100 fleet cannot run FP8 W8A8 at all. On the NVIDIA side, the default PyTorch backend of TensorRT-LLM supports FP4 and FP8 quantization on the latest Blackwell and Hopper GPUs. INT8 and INT4 recipes such as W8A8 smooth quantization or W4A16 GPTQ and AWQ exist largely because they run on older and cheaper fleets: on an 80 GB Ampere card, an INT4 W4A16 checkpoint is often the only way to serve a 70-billion-parameter model from a single GPU.
Weight-only versus weight and activation
The second axis is what gets compressed. In weight-only schemes (W4A16, W8A16), only the weights are quantized and activations stay in FP16 or BF16, with kernels that dequantize on the fly. In weight-and-activation schemes (W8A8, FP8 per-tensor or rowwise), both operands of the matrix multiply are compressed. TensorRT-LLM defines quantization as reducing memory footprint and computational cost by converting weights and activations from high-precision floating-point numbers like BF16 to lower-precision data types such as INT8, FP8 or FP4, and its recipe list spans both families: W4A16 GPTQ, W4A8 AWQ, FP8 per tensor, FP8 block scaling and FP8 KV cache.
The trade-off is predictable. Weight-only INT4 usually wins on memory; FP8 W8A8 usually wins on throughput, because tensor cores execute natively in the datatype instead of dequantizing every step. For long-context workloads, quantizing the KV cache alongside the weights matters as much as weight compression, since attention memory scales with sequence length.
A five step selection procedure
- Inventory the serving fleet by GPU architecture: Hopper, Ada or Blackwell opens the FP8 path; Ampere or CPU targets INT8; single memory-starved cards push toward INT4 weight-only.
- Compute the weight budget with the parameters-times-bytes table, reserving explicit headroom for KV cache at your real context length and batch size.
- Shortlist recipes against the vendor compatibility charts, and prefer an already-quantized checkpoint from the model hub over calibrating your own — calibration data choices are an extra failure surface.
- Run a fixed quality gate against a BF16 baseline on your own eval set with numeric pass thresholds; if the gate fails, degrade one step (FP8 to INT8, INT8 to BF16) instead of shipping a regression.
- Benchmark latency and cost per million tokens under production-shaped concurrency, and combine quantization with the speculative decoding setup and the semantic caching guide before buying more GPUs.
One honest limit: no compatibility chart tells you the quality cost on your traffic. Quantization interacts with model, task and prompt distribution, so treat any published quality delta as a prior rather than a guarantee, and keep the BF16 baseline in your regression suite.