Vllm Quantization

---
name: vllm-quantization
allowed-tools: Bash, Read, Write, Edit, Grep, Glob
description: |-
  vLLM datacenter-GPU quantization — picking, configuring, troubleshooting NVFP4, FP8, MXFP4, MXFP8, AWQ, GPTQ, INT8, compressed-tensors, modelopt, quark on H100/H200/B200/B300/GB200/GB300. 30 `--quantization` flag values on current vLLM (31 through v0.27.x, before `bitsandbytes` moved out-of-tree in v0.28.0), KV-cache dtypes (fp8_e4m3, nvfp4, per-token-head, turboquant), MoE backend selection (CUTLASS, TRTLLM, FlashInfer, DeepGEMM, Marlin, Qutlass), producing checkpoints with llm-compressor and NVIDIA ModelOpt (NVFP4_DEFAULT_CFG, FP8_DEFAULT_CFG, W4A16, SmoothQuant+GPTQ), online quantization (`fp8_per_tensor`, `fp8_per_block`, `nvfp4_per_token`), training EAGLE-3/dflash drafters on BF16 targets before PTQ, version gates per vLLM release (v0.14 → v0.27).
when_to_use: |-
  Trigger on `--quantization`, `--kv-cache-dtype`, NVFP4, MXFP4, MXFP8, FP8, W4A16, W8A8, W4A4, AWQ, GPTQ, SmoothQuant, modelopt, compressed-tensors, quark, torchao, bitsandbytes, gguf, TurboQuant, CUTLASS, Marlin, FlashInfer, TRTLLM, DeepGEMM, Qutlass, Machete, `hf_quant_config.json`, `kv_cache_scheme`, `NVFP4_DEFAULT_CFG`, `FP8_DEFAULT_CFG`, llm-compressor, ModelOpt. Symptoms — "garbage after FP8", "NVFP4 NaN", "FP8 KV multi-turn corruption", "MoE kernel not dispatched on SM120", "illegal memory access awq_marlin", "online FP8 drops bias", "modelopt checkpoint won't load". Decisions — NVFP4 vs FP8 on H200 vs B200, quantizing EAGLE-3/dflash drafters, generating a checkpoint vLLM can load. Also implicit — "quantize {model}", "pick quant for {model}", "audit quantization", "deploy-memo quant", "which quant fits {GPU}", "spec-study quantization".
---

# vLLM quantization — operator skill

**Last verified:** 2026-08-11. Claims below are probed against **v0.27.0**
(2026-08-10) — see `references/sources.md` for the per-ref audit table.

**Partial sweep 2026-09-15:** upstream is now **v0.29.0** (2026-09-09), with
v0.28.0 in between. Only the two releases' *breaking changes* were swept and
applied (see the deprecated-formats note below and `references/version-gates.md`).
The per-ref audit table is still at v0.27.0, so treat any claim not marked with a
v0.28/v0.29 note as verified only to v0.27.0.

For production vLLM operators on **H100 / H200 / B200 / B300 / GB200 / GB300** fleets
deciding which quantization format fits a given target model, producing a
checkpoint vLLM will actually load, wiring the right KV-cache dtype, diagnosing
accuracy or throughput regressions after an upgrade, and composing quantization
with speculative decoding / LoRA / MoE.

Pointer-map format: this SKILL.md picks the format and CLI; the files in
`references/` hold the per-format deep dives, exact source pointers, and
troubleshooting cards. Follow the link, don't paraphrase from memory — the
quantization layer moves faster than any other subsystem in vLLM (six formats
landed in v0.19 alone).

## When quantization wins, when it doesn't

Quantization trades weight precision for memory + compute:

- **KV-capacity bound** (long context, high concurrency) — FP8 or NVFP4 **KV
  cache** gives a 2×/4× KV-capacity multiplier; weight format matters much
  less than getting `--kv-cache-dtype` right. Measure `kv_cache_usage_perc`.
- **Memory-bandwidth bound** (small batch, decode-heavy, 70B+ on < 8 GPUs) —
  weight quantization (NVFP4 / FP8 / W4A16) reduces HBM traffic per token,
  giving 1.5–3× decode throughput on a well-matched target+kernel.
- **Compute bound** (prefill, large batch, small model) — quantization may
  not help; Blackwell FP4 Tensor Cores are the first architecture where
  W4A4 actually beats FP8 in compute-bound regimes. On Hopper, W4A16 is
  memory-only — MMA still runs FP16.
- **Multi-node EP / disaggregated serving** — NVFP4 reduces all-to-all by 4×
  vs BF16. DeepSeek-R1 / V3.2 on GB200/GB300 gets most of its throughput
  from NVFP4 over the fabric, not from per-GPU compute (see vLLM WideEP blog).

**Quantized models are not equivalent to the BF16 original.** Always eval on
actual traffic. Stock NVFP4 checkpoints recover ~99 % at 70B+, ~95–98 % at
7B–14B (Red Hat / NVIDIA numbers). Code / math / agentic workloads hit harder.
For the eval itself use the `vllm-benchmarking` skill (vLLM's own harness,
serving-side accuracy + latency) or `aiperf` (vendor-neutral, trace replay and
accuracy benchmarks against any OpenAI-compatible endpoint).

## Format selection — pick once per hardware

| GPU | Weight format recommendation | KV cache | Why |
|---|---|---|---|
| H100 / H200 (SM90) | `fp8` (compressed-tensors) or `modelopt` | `fp8_e4m3` | FP8 native Tensor Cores, CUTLASS/Marlin/DeepGEMM all mature |
| H100 / H200, accuracy-critical | `awq_marlin` / `gptq_marlin` (W4A16) | `fp8_e4m3` | Weight-only INT4 with per-group scales — best accuracy at 4-bit |
| H100 / H200, long-context MoE | `fp8` + DeepGEMM block | `fp8_e4m3` | Block FP8 MoE uses DeepGEMM path, lower activation-scale cost |
| B200 / B300 (SM100 / SM103) | `modelopt_fp4` or compressed-tensors NVFP4 | `fp8_e4m3`, or `nvfp4` (shipped v0.25.0, #42890) | Blackwell has native FP4 Tensor Cores — NVFP4 wins on both memory AND compute |
| B200 / B300, GPT-OSS | `mxfp4` / `gpt_oss_mxfp4` + `VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8=1` | `fp8_e4m3` | Only vendor-supplied format GPT-OSS ships |
| B200 / B300, lower accuracy risk | `modelopt_mxfp8` or online `fp8_per_block` | `fp8_e4m3` | MXFP8 MoE has the newest kernel set (v0.19), better on shapes NVFP4 struggles with |
| GB10 / DGX Spark (SM121) | `fp8` only (NVFP4/MXFP4 kernels brittle on SM121) | `fp8_e4m3` | See #39761 / #37030 / #34817 — desktop-Blackwell quant kernels are not production ready |
| MI300X / MI355X (ROCm, gfx942/gfx950) | `quark` (AMD) — W4A8 MXFP4/FP8 | `fp8_e4m3` (FNUZ-adjusted) | MI300 needs FNUZ scale adjustment; AMD Quark is the validated path |
| CPU | `cpu_awq` (W4A16) or `torchao` | — | Intel path for laptop / dev |

**Cross-hardware rule of thumb:** produce one `NVFP4` checkpoint per model. It
loads on Blackwell natively and on Hopper via emulation ([PR #35733](https://github.com/vllm-project/vllm/pull/35733), v0.19). A separate `fp8` checkpoint is still worth keeping for older Hopper nodes where the
NVFP4 emulation path is slower.

## The `--quantization` flag values (all 31 at v0.27.0)

Ground truth is the `QuantizationMethods = Literal[...]` block in [`vllm/model_executor/layers/quantization/__init__.py`](https://github.com/vllm-project/vllm/blob/v0.27.0/vllm/model_executor/layers/quantization/__init__.py) (`get_quantization_config()` dispatches below it) — grep the symbol, not a line range. Full catalog with file paths, min-capability, kernel map, and notes: `references/formats.md`.

**Production formats (keep in head):**

| Flag | Min SM | Use for |
|---|---|---|
| `fp8` | 89 | Compressed-tensors FP8 W8A8 — the Hopper default |
| `modelopt` | 89 | ModelOpt-exported FP8 (TRT-LLM ecosystem) |
| `modelopt_fp4` | 75 (emulated), 100 (native) | ModelOpt NVFP4 — the Blackwell default |
| `modelopt_mxfp8` | 89 | ModelOpt MXFP8 (MoE + dense) |
| `modelopt_mixed` | 89 | Mixed-precision per-layer checkpoints |
| `compressed-tensors` | varies per scheme | neuralmagic / Red Hat / llm-compressor output |
| `awq_marlin` | 75 | AWQ W4A16 — accuracy-critical INT4 |
| `gptq_marlin` | 75 | GPTQ W4A16 — classic INT4. **v0.30.0 removed group/dynamic activation ordering** ([#54809](https://github.com/vllm-project/vllm/pull/54809)): `g_idx` is ignored and the Marlin/GPTQ/CPU/RDNA3 kernels for it are gone. A checkpoint relying on act-order loses that accuracy recovery silently — prefer `awq_marlin` or `compressed-tensors` there. |
| `mxfp4` / `gpt_oss_mxfp4` | 80 (MoE only on 100) | GPT-OSS ships this |
| `mxfp8` | 80 | Online MXFP8 (v0.19+) |
| `quark` | varies | AMD ROCm path |
| `fp8_per_tensor` / `fp8_per_block` / `int8_per_channel_weight_only` / `online` | 75 | **Online** quantization from BF16 checkpoint — no pre-quant step |

**Deprecated / legacy / narrow:** `fbgemm_fp8` and `fp_quant` are the only two in vLLM's own `DEPRECATED_QUANTIZATION_METHODS` list. Also avoid for new work: `awq` (unfused Triton — use `awq_marlin`), `gptq` (unfused — use `gptq_marlin`), `experts_int8` (use `int8_per_channel_weight_only`), `moe_wna16`, `inc` (Intel Neural Compressor / AutoRound packing — no longer Intel-only compute: v0.30.0 added CUDA support for AutoRound's 2/3/5/6/7-bit widths via a Humming-kernel fallback, #52890), `torchao`. Aliases `auto_awq` / `auto_gptq` resolve to the same configs as `awq` / `gptq`.

**`bitsandbytes` left the tree in v0.28.0** (#43529, listed under Breaking
Changes). It is now an out-of-tree plugin, so on v0.28.0+ the value is not
built in and an existing `--quantization bitsandbytes` deployment needs the
plugin installed rather than a flag change. This is the same shape as `gguf`
moving to `vllm-gguf-plugin` — check the plugin before assuming the flag still
resolves.

**Two quantization-adjacent env vars were removed in v0.29.0**:
`VLLM_TEST_FORCE_FP8_MARLIN` (#52182), superseded by `--linear-backend` /
`--moe-backend`, and `VLLM_ROCM_USE_AITER_FP4_ASM_GEMM` (#53141). v0.28.0 also
removed the deprecated `calculate_kv_scales` runtime KV-scale calculation
(#49389) and `override_attention_dtype` (#48684).

**v0.30.0 removed the `use_fp4_indexer_cache` alias** (#55353): set
`indexer_kv_dtype` directly on `AttentionConfig` for the DSA sparse-attention
indexer K cache (values `auto` / `bf16` / `fp8` / `mxfp4` / `nvfp4`) —
`use_fp4_indexer_cache = True` used to map to `mxfp4`. The field is gone from
`AttentionConfig`, not just ignored — a config that still sets it is no
longer accepted.

**Gone from the in-tree flag list — do not offer them:**

- `gguf` — **migrated out-of-tree** to [`vllm-gguf-plugin`](https://github.com/vllm-project/vllm-gguf-plugin). `pip install vllm-gguf-plugin` first, then `vllm serve <repo>:Q4_K_M --tokenizer <base-model>`. Still experimental; not a datacenter path.
- `cpu_awq` — **folded into `awq_marlin`** ([PR #43841](https://github.com/vllm-project/vllm/pull/43841), merged 2026-05-28). Use `awq_marlin` on CPU.

**Newer values not in older skill copies:** `humming` (Humming mixed-precision, also a `--linear-backend` / `--moe-backend` value), `deepseek_v4_fp8` (DeepSeek-V4 checkpoint config), `fp8_per_channel` and `nvfp4_per_token` (online shorthands; `nvfp4_per_token` added v0.26.0, [#48538](https://github.com/vllm-project/vllm/pull/48538)).

## KV-cache dtypes (all 16 at v0.27.0)

Single dispatch: [`vllm/config/cache.py`](https://github.com/vllm-project/vllm/blob/v0.27.0/vllm/config/cache.py) — grep the `CacheDType = Literal[...]` block, not a line range; it has grown every minor.

- `auto` — match model weight dtype. `float16` / `bfloat16` — pin an explicit unquantized KV dtype.
- `fp8`, `fp8_e4m3`, `fp8_e5m2` — the production path. **E4M3 is default**; E5M2 only for ROCm-specific setups.
- `fp8_inc` (Intel), `fp8_ds_mla` (DeepSeek MLA variant).
- `int4_per_token_head`, `int8_per_token_head`, `fp8_per_token_head` — dynamic per-(token,head) scales computed in-kernel. **No checkpoint scales needed.** Added in [PR #34281](https://github.com/vllm-project/vllm/pull/34281), v0.17.
- `turboquant_k8v4`, `turboquant_4bit_nc`, `turboquant_k3v4_nc`, `turboquant_3bit_nc` — Hadamard-rotated 2-4 bit KV (v0.19, [PR #38479](https://github.com/vllm-project/vllm/pull/38479)).
- `nvfp4` — **shipped, not roadmap.** [#32220](https://github.com/vllm-project/vllm/issues/32220) closed `COMPLETED` 2026-05-04; v0.25.0 landed NVFP4 KV with skip-layers sliding window ([#42890](https://github.com/vllm-project/vllm/pull/42890)) and it is an accepted `CacheDType` value at v0.27.0. Backend support is narrow — see `references/kv-cache.md`.

`--calculate-kv-scales` was **deprecated in v0.19** ([PR #37201](https://github.com/vllm-project/vllm/pull/37201)). Use pre-calibrated scales (LLM Compressor produces them) or let per-token-head scales be computed dynamically.

## Producing a checkpoint

vLLM doesn't quantize — a separate tool does, then vLLM loads the result.
Two production paths exist:

1. **[llm-compressor](https://github.com/vllm-project/llm-compressor)** (vLLM-project) — outputs compressed-tensors format. Preferred for the open ecosystem. Covered in `references/llm-compressor.md`.
2. **[NVIDIA ModelOpt](https://github.com/NVIDIA/TensorRT-Model-Optimizer)** — outputs ModelOpt HF format, also consumable by TRT-LLM and SGLang. Preferred for NVFP4 on Blackwell. Covered in `references/modelopt.md`.

**Quick picker:**

- **FP8 Hopper, no calibration wanted** → llm-compressor `FP8_DYNAMIC` (data-free, ~15 min for a 70B on H100).
- **W4A16 INT4 (AWQ or GPTQ) with best accuracy** → llm-compressor, `AWQModifier` or `GPTQModifier` with 256–512 ultrachat samples.
- **NVFP4 on Blackwell** → ModelOpt `NVFP4_DEFAULT_CFG` or llm-compressor `NVFP4A16` / `NVFP4` scheme (v0.10+; current release **0.13.0**, and note the project ships parallel maintenance lines — see `references/llm-compressor.md`).
- **MXFP4 MoE for GPT-OSS-style models** → use the vendor checkpoint as-is, or ModelOpt MXFP4.
- **KV cache FP8 scales for MLA** → llm-compressor `kv_cache_scheme` block with `strategy: tensor` (per-tensor is stable; per-head is still the experimental path). The MLA multi-turn corruption once flagged here — [#38652](https://github.com/vllm-project/vllm/issues/38652) — was **fixed by PR #37054** and is no longer a reason to avoid FP8 KV on MLA.

Both tools output a HF directory vLLM serves with `--quantization compressed-tensors` (llm-compressor) or `--quantization modelopt` / `--quantization modelopt_fp4` (ModelOpt).

## Speculative decoding drafters

**llm-compressor does not train drafters.** ModelOpt does:
`modelopt/torch/speculative/{eagle,dflash,medusa,plugins}/`, examples in
`examples/speculative_decoding/`. Recipes: `modelopt_recipes/general/speculative_decoding/{eagle3,dflash}.yaml`.

**Critical constraint:** ModelOpt recipes assume **BF16 target** — not
validated with an already-NVFP4 target (base wrapped in `torch.no_grad()`,
so quantized target is theoretically workable but unvalidated). The order is:

```
1. Train drafter on BF16 target   (ModelOpt, ~4-12h on 8×H100)
2. Export drafter HF dir          (scripts/export_hf_checkpoint.py)
3. PTQ target to NVFP4 or FP8     (ModelOpt or llm-compressor)
4. (Optional) PTQ drafter too     (small, minimal accuracy cost)
5. Serve both in vLLM             (--quantization modelopt_fp4 --speculative-config '{...}')
```

Medusa / MTP cannot be trained post-hoc — MTP heads are part of the
pretraining (DeepSeek V3, Qwen3-Next, GLM-4.5 MoE, etc.). Full details +
exact commands: `references/modelopt.md` § speculative-decoding.

For spec-dec runtime tuning (acceptance rate metrics, method selection
per target family, chunked-prefill composability) use the separate
`vllm-speculative-decoding` skill — don't duplicate here.

## Online quantization

Introduced by the v0.14 redesign ([PR #37776](https://github.com/vllm-project/vllm/pull/37776)). Quantizes a BF16 checkpoint **at load time**, no pre-quantization step. Trade-off: peak load memory is BF16 size.

```bash
# Per-tensor FP8 (static scales, simplest)
vllm serve meta-llama/Llama-3.1-70B --quantization fp8_per_tensor

# Per-block FP8 (dynamic per-token activation scales)
vllm serve meta-llama/Llama-3.1-70B --quantization fp8_per_block

# Weight-only INT8
vllm serve meta-llama/Llama-3.1-70B --quantization int8_per_channel_weight_only

# Fine-grained: per-layer-kind override via --quantization-config (JSON or dotted keys)
vllm serve meta-llama/Llama-3.1-70B \
  --quantization fp8_per_tensor \
  --quantization-config '{"moe":{"activation":"mxfp8"},"ignore":["lm_head"]}'
```

The advanced schema is `{linear: {weight, activation}, moe: {weight, activation},
ignore: [...]}`; `linear` / `moe` also accept a bare shorthand string. Names come
from `QUANT_KEY_NAMES` in `vllm/config/quantization.py`. There is no
`--quantization-config-file` flag and no `global_scheme` key.

**v0.30.0 added `targets`** for per-layer online overrides — a dict of
`{pattern: shorthand}` (exact name, `re:`-prefixed regex, or fnmatch, mapping
to `fp8_per_tensor` / `fp8_per_block` / `fp8_per_channel` / `mxfp8` /
`int8_per_channel_weight_only` / `nvfp4_per_token`), mutually exclusive with
`linear`/`moe`. A layer matching no pattern keeps its checkpoint dtype
([#51285](https://github.com/vllm-project/vllm/pull/51285)):

```bash
vllm serve Qwen/Qwen3.5-35B-A3B --quantization online \
  --quantization-config '{"targets":{"model.layers.0.self_attn.o_proj":"fp8_per_tensor","*mlp.experts*":"mxfp4"}}'
```

**v0.30.0 also let online quantization target the unquantized layers of an
already partially pre-quantized checkpoint**, regardless of its own
`quant_method` (modelopt, compressed-tensors, quark) — the checkpoint's own
method still owns its already-quantized layers
([#51392](https://github.com/vllm-project/vllm/pull/51392)):

```bash
# Adds MXFP8 to the dense linear layers of a Quark checkpoint that only
# ships quantized MoE experts.
vllm serve amd/Qwen3.5-35B-A3B-MXFP4 --quantization-config.linear mxfp8
```

**Known gotchas** — [#34129](https://github.com/vllm-project/vllm/issues/34129) (doesn't split MoE across EP, closed `NOT_PLANNED` — won't-fix), [#19020](https://github.com/vllm-project/vllm/issues/19020) / [#32029](https://github.com/vllm-project/vllm/issues/32029) / [#32412](https://github.com/vllm-project/vllm/issues/32412) (multiple active RFCs). Dropped bias weights ([#39663](https://github.com/vllm-project/vllm/issues/39663)) is **fixed in v0.21.0** and no longer a reason to avoid online FP8. For MoE models, still prefer a pre-quantized checkpoint.

## The operator-pain-point shortlist

Internalize these before debugging accuracy / throughput regressions:

1. **`--kv-cache-dtype fp8` on MLA models → garbage on multi-turn** ([#38652](https://github.com/vllm-project/vllm/issues/38652)) — **FIXED, guidance reversed 2026-07-21.** Closed 2026-05-15 with *"Fixed by #37054"*, a PR that merged **2026-03-18** and was already recorded elsewhere in this skill as the v0.19 MLA KV-scale fix. The issue tracker lagged the fix by ~2 months, and this skill carried the blanket avoid-on-DeepSeek/GLM/Kimi warning for longer still. On a current release FP8 KV on MLA is usable — measure it instead of excluding it. See `references/kv-cache.md` items 1 and 5.
2. **Gemma 4 FP8-block → logit saturation / repetitive garbage** ([#39407](https://github.com/vllm-project/vllm/issues/39407), [#39049](https://github.com/vllm-project/vllm/issues/39049)). Use non-block FP8 or FP16. Both issues were closed `NOT_PLANNED` by the inactivity bot on 2026-09-02 with no fix and PR #40391 still open — closed state here is not a fix.
3. **NVFP4 on Qwen3-Next / hybrid-attention models** silently corrupted output when `quantization_config.ignore` missed `linear_attn` layers ([#40252](https://github.com/vllm-project/vllm/issues/40252), fixed + closed 2026-04-20). The underlying pattern still applies to any new hybrid-attention model: always audit the `ignore` list when quantizing non-standard architectures.
4. **Online FP8 dropped bias weights — fixed in v0.21.0** ([#39663](https://github.com/vllm-project/vllm/issues/39663), fix PR #41424). Below v0.21.0, a bias-ed target needs a pre-quantized checkpoint; at or above it, online FP8 is fine for bias. MoE is a separate problem (#34129, won't-fix).
5. **Dynamic FP8 + LoRA-merged model on B200 → non-deterministic degenerate output** ([#39662](https://github.com/vllm-project/vllm/issues/39662)). Pin static FP8.
6. **SM120 (RTX 5090, 6000 Pro) is not a datacenter NVFP4 MoE target** ([#35065](https://github.com/vllm-project/vllm/issues/35065), [#31085](https://github.com/vllm-project/vllm/issues/31085)) — full kernel set is SM100 / SM103 only. Desktop Blackwell is production only for `fp8`.

The hardware-/version-gated traps (B300 TRTLLM hang, ModelOpt-vs-compressed-tensors export drift, MXFP4-linear-falls-back-to-BF16, A100 TurboQuant crash, Qwen3.5 v0.18 KV regression, MXFP8+DeepGEMM pre-v0.19 crash) live in the full triage playbook with symptoms → PR → workaround: `references/troubleshooting.md`.

## Version-gate highlights

Full matrix in `references/version-gates.md`. Load-bearing ones:

- **v0.27.1** — current stable, [released 2026-08-11](https://github.com/vllm-project/vllm/releases/tag/v0.27.1). One change: **quantized DSpark Markov heads** ([#50424](https://github.com/vllm-project/vllm/pull/50424)) — `DSparkMarkovHead.markov_w2` (a `ParallelLMHead`) now accepts and forwards `quant_config`, so **W4A16 `markov_w2` weights including `weight_scale_2` load through the normal quantization dispatch path**. Unquantized behaviour is preserved. A new quantizable surface on a spec-dec drafter; see `vllm-speculative-decoding`. The **container images** shipped first: `vllm/vllm-openai:v0.27.1` (plus `-x86_64`/`-aarch64`/`-cu129`/`-ubuntu2404` variants) were pushed 2026-08-11 10:24-10:42Z, *before* the GitHub release at 10:47Z — so "no release yet" never implies "no image yet". Air-gap staging mirrors that image; it does not wait on a PyPI wheel.
- **v0.27.0** — [2026-08-10](https://github.com/vllm-project/vllm/releases/tag/v0.27.0). `--linear-backend` now honored for ModelOpt W4A16 (#50273); FP4 Qutlass for compressed-tensors (#43229); ModelOpt FP8 emulation on SM80 (#50019); TurboQuant KV quant mode (#50533); compressed-tensors checkpoints for DeepSeek-V4 (#41276) and Kimi-K3 (#50500). **Breaking environment change:** PyTorch 2.13.0 / Triton 3.7.1 / Transformers 5.14.1 (#48155, #49223) — rebuild any custom kernel or plugin image.
- **v0.26** — [v0.26.0](https://github.com/vllm-project/vllm/releases/tag/v0.26.0) (2026-07-27). `nvfp4_per_token` online MoE quantization (#48538); Humming w[2-7]a[4,8] weight-only for compressed-tensors (#46390); CuTe-DSL FlashInfer MXFP4 (#48417); MLA `kv_cache_dtype_skip_layers` (#47309); ROCm HybridW4A16 linear kernel (#40977).
- **v0.25** — [v0.25.1](https://github.com/vllm-project/vllm/releases/tag/v0.25.1) (2026-07-14). **Run v0.25.1+, not v0.25.0**, if serving NVFP4 on multi-GPU: v0.25.0 and earlier corrupt output via the fused allreduce+RMSNorm+quant path on models with Gemma/Qwen-style RMSNorm (PR #48330 — see `vllm-performance-tuning` § garbage `!!!` output). v0.25.0 also removed PagedAttention entirely and made Model Runner V2 the default for all dense models.
- **v0.22–v0.24** — NVFP4 fused MoE for DeepSeek-V4 (#42209), NVFP4 Cutlass linear on the batch-invariant path (#39912), padded NVFP4 quant kernel (#42774), ModelOpt W4A16 NVFP4 fused MoE with mixed-precision dispatch (#42566), FlashInfer cutedsl NVFP4 GEMM (#42235) and cute-dsl MXFP8 linear (#46393), FP8 weight layout canonicalized to `(K, N)` (#44735), `fp8_e5m2` KV cache allowed for non-fp8 checkpoints (#45040), W8A8 int-quant scheme-selection regression fixed (#46860), and an actionable error on group-size/TP mismatch (#46230) in place of an obscure failure.
- **v0.21** — [v0.21.0](https://github.com/vllm-project/vllm/releases/tag/v0.21.0) (2026-05-15); v0.20.0 stable shipped 2026-04-27, followed by v0.20.1 / v0.20.2. Quantization-layer churn continues — re-verify any v0.19-specific claim on upgrade.
- **v0.19** — online MXFP8, `CompressedTensorsW8A8Mxfp8`, ROCm AWQ Marlin, TurboQuant KV, DeepGemm E8M0 fix for Qwen3.5 FP8 on Blackwell, `--calculate-kv-scales` deprecation, Gemma 4 quantized MoE, B300 / GB300 fixes.
- **v0.18** — FP8 KV in Triton MLA decode, FlashInfer Sparse MLA FP8, ModelOpt MXFP8 MoE, AMD Quark W4A8 MXFP4/FP8, MLA crash with AWQ/GPTQ fix.
- **v0.17** — per-head KV scales, SM100 MXFP8 kernels, compressed-tensors as ground-truth, ModelOpt mixed precision, Llama-4 attention quant.
- **v0.16** — NVFP4/FP8 on Turing via emulation, TP>4 for FP4 GEMM, ModelOpt MXFP8 dense.
- **v0.15** — MXFP4 W4A16 for compressed-tensors MoE, FP4 kernel optimization (+65 % on SM100F via 256-bit loads).
- **v0.14** — Online quantization redesign, MXFP4 W4A16 for dense.

## What to read next

- `references/formats.md` — per-format deep dive: kernels, config JSON shapes, min-capability, known caveats.
- `references/llm-compressor.md` — recipe cookbook: FP8_DYNAMIC / W4A16 / AWQ / NVFP4A16 / KV-cache FP8 / model-free PTQ commands with exact calibration budgets and output layouts.
- `references/modelopt.md` — ModelOpt PTQ (`hf_ptq.py`) + speculative-decoding training (EAGLE-3, dflash, MTP constraints) + vLLM loader compatibility.
- `references/kernels.md` — kernel × format × SM dispatch map (Marlin / CUTLASS / DeepGEMM / FlashInfer / TRTLLM / Qutlass / Machete / Triton / Exllamav2).
- `references/kv-cache.md` — KV-cache quantization: dtypes, per-token-head scales, attention-backend compatibility, calibration.
- `references/troubleshooting.md` — symptom → known-issue → fix playbook.
- `references/version-gates.md` — release-by-release quantization changes, v0.14 → v0.27.

## External references

Load source, not paraphrase:

- vLLM docs: [FP8 W8A8](https://docs.vllm.ai/en/latest/features/quantization/fp8/), [Quantized KV Cache](https://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache/), [AMD Quark](https://docs.vllm.ai/en/stable/features/quantization/quark/).
- vLLM recipes: [index](https://docs.vllm.ai/projects/recipes/en/latest/index.html).
- llm-compressor docs: [index](https://docs.vllm.ai/projects/llm-compressor/en/latest/), [NVFP4 W4A4](https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/quantization_w4a4_fp4/), [Qwen3.5 NVFP4 MoE](https://docs.vllm.ai/projects/llm-compressor/en/latest/key-models/qwen3.5/nvfp4-moe-example/).
- compressed-tensors spec: [`quant_scheme.py`](https://github.com/neuralmagic/compressed-tensors/blob/main/src/compressed_tensors/quantization/quant_scheme.py), [overview](https://deepwiki.com/neuralmagic/compressed-tensors/1-overview).
- NVIDIA: [Introducing NVFP4](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/), [NVFP4 KV cache](https://developer.nvidia.com/blog/optimizing-inference-for-long-context-and-large-batch-sizes-with-nvfp4-kv-cache/), [MoE perf leaps on Blackwell](https://developer.nvidia.com/blog/delivering-massive-performance-leaps-for-mixture-of-experts-inference-on-nvidia-blackwell/).
- Red Hat: [Accelerating LLMs with NVFP4](https://developers.redhat.com/articles/2026/02/04/accelerating-large-language-models-nvfp4-quantization), [LLM Compressor 0.9](https://developers.redhat.com/articles/2026/01/16/llm-compressor-090-attention-quantization-mxfp4-support-and-more), [vLLM FP8 foundational](https://developers.redhat.com/articles/2024/07/15/vllm-brings-fp8-inference-open-source-community).
- vLLM blog: [GPT-OSS on Blackwell](https://blog.vllm.ai/2026/02/01/gpt-oss-optimizations.html), [DeepSeek-R1 WideEP on GB200](https://blog.vllm.ai/2026/02/03/dsr1-gb200-part1.html), [DeepSeek-V3.2 on GB300](https://blog.vllm.ai/2026/02/13/gb300-deepseek.html).
- AMD: [FP8 with Quark for vLLM](https://rocm.docs.amd.com/projects/ai-developer-hub/en/latest/notebooks/gpu_dev_optimize/fp8_quantization_quark_vllm.html), [MXFP4 Llama3.3 with Quark](https://rocm.docs.amd.com/projects/ai-developer-hub/en/latest/notebooks/gpu_dev_optimize/mxfp4_quantization_quark_vllm.html).

When in doubt, read the vLLM source — [`vllm/model_executor/layers/quantization/`](https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/layers/quantization/) is the ground truth, and the quantization layer churns fast enough that cached knowledge rots inside a release cycle.