Zeroquant Eval

---
name: zeroquant-eval
description: Evaluates the accuracy and inference latency of post-training quantized Transformer models (BERT and GPT-3-style) on standard NLP benchmarks and language modeling tasks. Use when the user wants to benchmark on GLUE benchmark, 20 zero-shot evaluation tasks, PTB / Wikitext-2 / Wikitext-103, or asks about evaluating this task. Reports average accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2206.01861
  bibtex_key: yao2022zeroquant
  confidence: high
---

# zeroquant-eval

> ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers — Yao et al. (2022) (arXiv:2206.01861, 2022)

## What this evaluates

Evaluates the accuracy and inference latency of post-training quantized Transformer models (BERT and GPT-3-style) on standard NLP benchmarks and language modeling tasks.

## Datasets

- **GLUE benchmark** — total ?; splits: dev (-1); HF `glue`
- **20 zero-shot evaluation tasks** — total ?; splits: test (-1)
- **PTB / Wikitext-2 / Wikitext-103** — total ?; splits: test (-1)

## Metrics

- `average accuracy` **(primary)** — range: percent
  - Mean of task-specific classification or QA accuracies across the benchmark tasks.
- `perplexity (PPL)` — range: other
  - Exponential of the negative average log-likelihood of the ground-truth tokens in the language modeling tasks.
- `latency` — range: other
  - End-to-end average time in milliseconds to generate the first 50 tokens or process the entire model.

## Input / output format

**Input**: Text prompts, sentences, or passages for classification, question answering, or language modeling tasks.

**Output**: Predicted class labels for classification/QA tasks, or generated token sequences for language modeling.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

def compute_ppl(logits, gold):
    log_probs = -F.cross_entropy(logits, gold, reduction='none')
    return torch.exp(log_probs.mean())
```

## Common pitfalls

- The notation WxAy denotes x-bit weight quantization and y-bit activation quantization, with mixed precision (e.g., W4/8) applying different bit-widths to different layers (e.g., MHSA vs FFC).
- Latency speedups are highly dependent on batch size and sequence length; results vary significantly across configurations.
- LKD (Layer-by-Layer Knowledge Distillation) requires a small calibration set and GPU time (seconds to hours), contradicting a strict 'zero-cost' assumption for aggressive quantization.

## Evidence (verbatim from paper)

> Comparing ZeroQuant with PTQ using W8A8, we can reduce the accuracy gap from 1.1% to 0.2% and the perplexity (PPL) gap from 4.7 to 0.2 with no activation range calibration cost.

## Citation

```bibtex
@misc{yao2022zeroquant,
  title={ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers},
  author={Yao et al. (2022)},
  year={2022},
  note={arXiv:2206.01861}
}
```

- arXiv: 2206.01861