Qbert Quantization Eval

---
name: qbert-quantization-eval
description: Evaluates the impact of ultra-low precision weight quantization (down to 2 bits) on BERTBASE performance across standard NLP tasks. It compares Hessian-guided mixed and group-wise quantization strategies against direct quantization baselines to measure accuracy retention versus model compression. Use when the user wants to benchmark on SST-2, MNLI, CoNLL-03, SQuAD, or asks about evaluating this task. Reports Acc.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1909.05840
  bibtex_key: shen2019qbert
  confidence: high
---

# qbert-quantization-eval

> Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT — Shen et al. (2019) (arXiv:1909.05840, 2019)

## What this evaluates

Evaluates the impact of ultra-low precision weight quantization (down to 2 bits) on BERTBASE performance across standard NLP tasks. It compares Hessian-guided mixed and group-wise quantization strategies against direct quantization baselines to measure accuracy retention versus model compression.

## Datasets

- **SST-2** — total ?; splits: dev (-1)
- **MNLI** — total ?; splits: dev (-1)
- **CoNLL-03** — total ?; splits: dev (-1)
- **SQuAD** — total ?; splits: dev (-1)

## Metrics

- `Acc` **(primary)** — range: percent
  - Percentage of correctly predicted class labels out of total instances.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall, computed per class and averaged (macro or micro depending on task convention).
- `Exact Match (EM)` — range: [0, 1]
  - Fraction of predictions where the predicted answer span exactly matches the ground truth span.

## Input / output format

**Input**: Tokenized text sequences (single sentences or sentence pairs) fed into BERTBASE.

**Output**: Predicted class labels for classification tasks (SST-2, MNLI, CoNLL-03) or predicted answer spans for SQuAD.

## Scoring recipe

```python
def compute_metrics(predictions, gold, task):
  if task in ['sst2', 'mnli']:
    return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
  elif task == 'conll03':
    return f1_score(gold, predictions, average='macro')
  elif task == 'squad':
    em = sum(exact_match(p, g) for p, g in zip(predictions, gold)) / len(gold)
    f1 = qa_f1_score(gold, predictions)
    return em, f1
```

## Common pitfalls

- 3-bit quantization is not natively supported in hardware; results often assume casting to 4/8 bits for execution, which affects actual speedup vs memory savings.
- SQuAD fine-tuning exhibits convergence issues (negative Hessian eigenvalues), making it unusually sensitive to quantization compared to other tasks.
- Group-wise quantization accuracy gains saturate at 128 groups; increasing groups further yields diminishing returns while increasing LUT overhead.

## Evidence (verbatim from paper)

> Table 1: Quantization results for BERTBASE on Natural Language Understanding tasks. ... Acc | Size | Size-w/o-e ... Q-BERT | 8 | 8 | 92.88 | 103.9 | 81.2

## Citation

```bibtex
@misc{shen2019qbert,
  title={Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT},
  author={Shen et al. (2019)},
  year={2019},
  note={arXiv:1909.05840}
}
```

- arXiv: 1909.05840