Glue Lm Eval

---
name: glue-lm-eval
description: Evaluates the quantization performance of pre-trained language models (both discriminative BERT-style and generative GPT-style) across standard NLP classification, regression, and language modeling tasks under data-free zero-shot quantization settings. Use when the user wants to benchmark on GLUE, WikiText2, Penn Treebank (PTB), WikiText103, or asks about evaluating this task. Reports GLUE Avg..
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.13315
  bibtex_key: zhu2023zeroshotsharpnessawarequantization
  confidence: high
---

# glue-lm-eval

> Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Models — Zhu et al. (2023) (arXiv:2310.13315, 2023)

## What this evaluates

Evaluates the quantization performance of pre-trained language models (both discriminative BERT-style and generative GPT-style) across standard NLP classification, regression, and language modeling tasks under data-free zero-shot quantization settings.

## Datasets

- **GLUE** — total ?; splits: dev (-1)
- **WikiText2** — total ?; splits: test (-1)
- **Penn Treebank (PTB)** — total ?; splits: test (-1)
- **WikiText103** — total ?; splits: test (-1)

## Metrics

- `Accuracy (Acc.)` — range: [0, 1]
  - Fraction of correctly predicted class labels.
- `Pearson correlation (Pear.)` — range: [-1, 1]
  - Pearson correlation coefficient between predicted and ground-truth continuous scores.
- `Matthew correlation (Mcc.)` — range: [-1, 1]
  - Matthews correlation coefficient for binary classification tasks.
- `Perplexity (PPL)` — range: [0, inf)
  - Exponential of the average negative log-likelihood of the ground-truth tokens.
- `GLUE Avg.` **(primary)** — range: [0, 1]
  - Average of task-specific metrics (Acc., Pear., Mcc.) across all GLUE sub-tasks.

## Input / output format

**Input**: Tokenized text sequences for classification/regression tasks; raw text for language modeling tasks.

**Output**: Class labels or probability distributions for classification; continuous scores for regression; next-token log-probabilities for language modeling.

## Scoring recipe

```python
def compute_metrics(predictions, golds, task_type):
    if task_type == 'classification':
        return np.mean(predictions == golds)
    elif task_type == 'regression':
        return pearsonr(golds, predictions)[0]
    elif task_type == 'binary':
        return matthews_corrcoef(golds, predictions)
    elif task_type == 'lm':
        return np.exp(-np.mean(np.log(predictions + 1e-9)))
```

## Common pitfalls

- Task-specific metrics must be applied correctly (e.g., Mcc. for CoLA, Pear. for STS-B, not Accuracy).
- All reported results are averaged over 5 random seeds to mitigate stochasticity.
- The 'WxAy' notation denotes weight and activation bit-widths respectively, not just weight quantization.
- The method is data-free/zero-shot for quantization; only the teacher model is fine-tuned on task data.

## Evidence (verbatim from paper)

> For evaluation, we report the performance with Accuracy ("Acc.") metric for most tasks, except the Pearson correlation ("Pear:") for STS-B, the Matthew correlation ("Mcc.") for CoLA, the perplexity (PPL) score for language modeling tasks. We report the averaged results over 5 random seeds to avoid stochasticity.

## Citation

```bibtex
@misc{zhu2023zeroshotsharpnessawarequantization,
  title={Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Models},
  author={Zhu et al. (2023)},
  year={2023},
  note={arXiv:2310.13315}
}
```

- arXiv: 2310.13315