Glue Eval

---
name: glue-eval
description: Evaluates the transfer learning capability of pre-trained language models across a diverse suite of natural language understanding tasks, including sentence classification, semantic textual similarity, and natural language inference. Use when the user wants to benchmark on GLUE, or asks about evaluating this task. Reports GLUE Average.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.06037
  bibtex_key: ji2025confidenceregularized
  confidence: high
---

# glue-eval

> Confidence Regularized Masked Language Modeling using Text Length — Ji et al. (2025) (arXiv:2504.06037, 2025)

## What this evaluates

Evaluates the transfer learning capability of pre-trained language models across a diverse suite of natural language understanding tasks, including sentence classification, semantic textual similarity, and natural language inference.

## Datasets

- **GLUE** — total ?; splits: test (-1); HF `glue`

## Metrics

- `GLUE Average` **(primary)** — range: percent
  - Arithmetic mean of task-specific scores: Matthew's correlation for CoLA, Pearson correlation for STS-b, and accuracy for all other tasks. WNLI is explicitly excluded from the average.

## Input / output format

**Input**: Tokenized text sequences up to 512 tokens, grouped by length during pre-training; standard task-specific prompts during fine-tuning.

**Output**: Class labels for classification tasks; confidence scores for each class used to compute calibration metrics.

## Scoring recipe

```python
def compute_glue_avg(preds, golds, tasks):
    scores = []
    for task in tasks:
        if task == 'CoLA': scores.append(matthews_corrcoef(golds[task], preds[task]))
        elif task == 'STS-b': scores.append(pearsonr(golds[task], preds[task]))
        else: scores.append(accuracy_score(golds[task], preds[task]))
    return sum(scores) / len(scores)
```

## Common pitfalls

- Excluding WNLI from the GLUE average as specified in the paper.
- Reporting task-specific metrics (Matthew's corr, Pearson corr, accuracy) instead of the aggregated average.
- Not averaging results over 7 different random seeds as reported in the tables.

## Evidence (verbatim from paper)

> We evaluated methods on the GLUE benchmark (Wang et al., 2019) and SQuAD 1.1/2.0 datasets (Rajpurkar et al., 2016, 2018). Following Devlin et al. (2019), we excluded WNLI from tasks of GLUE benchmark. We reported Matthew's correlation score for CoLA, Pearson correlations for STS-b, F1 score for SQuAD 1.1/2.0, and accuracy scores for the other tasks.

## Citation

```bibtex
@misc{ji2025confidenceregularized,
  title={Confidence Regularized Masked Language Modeling using Text Length},
  author={Ji et al. (2025)},
  year={2025},
  note={arXiv:2504.06037}
}
```

- arXiv: 2504.06037