Glue Benchmark Eval

---
name: glue-benchmark-eval
description: Evaluates the ability of efficient fine-tuning and structured sparsity methods to maintain performance across a diverse suite of natural language understanding tasks. It probes task-specific classification accuracy and correlation metrics under both full-data and limited-data regimes, as well as pre-training perplexity on a large-scale corpus. Use when the user wants to benchmark on GLUE benchmark, or asks about evaluating this task. Reports GLUE average score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.09169
  bibtex_key: svirsky2026finegates
  confidence: high
---

# glue-benchmark-eval

> Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsity — Svirsky et al. (2026) (arXiv:2602.09169, 2026)

## What this evaluates

Evaluates the ability of efficient fine-tuning and structured sparsity methods to maintain performance across a diverse suite of natural language understanding tasks. It probes task-specific classification accuracy and correlation metrics under both full-data and limited-data regimes, as well as pre-training perplexity on a large-scale corpus.

## Datasets

- **GLUE benchmark** — total ?; splits: test (-1); HF `glue`

## Metrics

- `GLUE average score` **(primary)** — range: [0, 1]
  - Arithmetic mean of task-specific metrics: accuracy for most tasks, Matthew's correlation for CoLA, and Pearson correlation for STSB. MNLI uses overall (matched and mismatched) accuracy.
- `accuracy` — range: [0, 1]
  - Proportion of correctly predicted class labels out of total samples.
- `Matthew's correlation` — range: [-1, 1]
  - Correlation coefficient measuring the quality of binary and multi-class classifications, accounting for true/false positives and negatives.
- `Perplexity` — range: other
  - Exponential of the average negative log-likelihood of the validation tokens on the C4 dataset.

## Input / output format

**Input**: Tokenized text sequences or sentence pairs for classification tasks; raw text tokens for C4 pre-training.

**Output**: Predicted class labels for classification tasks; probability distributions or log-likelihoods for perplexity computation.

## Scoring recipe

```python
def compute_glue_score(predictions, golds):
    scores = []
    for task in ['CoLA', 'STSB', 'MRPC', 'RTE', 'SST2', 'MNLI', 'QNLI', 'QQP']:
        if task == 'CoLA':
            scores.append(matthews_corrcoef(golds[task], predictions[task]))
        elif task == 'STSB':
            scores.append(pearsonr(golds[task], predictions[task]))
        else:
            scores.append(accuracy_score(golds[task], predictions[task]))
    return sum(scores) / len(scores)
```

## Common pitfalls

- MNLI evaluation must combine matched and mismatched test sets into a single overall accuracy score rather than reporting them separately.
- Task-specific metrics differ (accuracy vs. correlation), so the GLUE average is a simple arithmetic mean of these heterogeneous scores, not a weighted average.
- Limited-data experiments use the first 10,000 samples from large datasets rather than standard train/val/test splits.

## Evidence (verbatim from paper)

> We report the overall (matched and mismatched) accuracy for MNLI, Matthew’s correlation for CoLA, Pearson correlation for STSB, and accuracy for other tasks.

## Citation

```bibtex
@misc{svirsky2026finegates,
  title={Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsity},
  author={Svirsky et al. (2026)},
  year={2026},
  note={arXiv:2602.09169}
}
```

- arXiv: 2602.09169