Glue Squad2 Eval

---
name: glue-squad2-eval
description: Evaluates the generalization and downstream performance of pretrained language models on a suite of natural language understanding tasks (GLUE) and reading comprehension (SQuAD 2.0). Use when the user wants to benchmark on GLUE, SQuAD 2.0, or asks about evaluating this task. Reports GLUE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2102.08473
  bibtex_key: meng2021cocolm
  confidence: high
---

# glue-squad2-eval

> COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining — Meng et al. (2021) (arXiv:2102.08473, 2021)

## What this evaluates

Evaluates the generalization and downstream performance of pretrained language models on a suite of natural language understanding tasks (GLUE) and reading comprehension (SQuAD 2.0).

## Datasets

- **GLUE** — total ?; splits: train (-1), dev (-1), test (-1)
- **SQuAD 2.0** — total ?; splits: train (-1), validation (-1)

## Metrics

- `GLUE` **(primary)** — range: [0, 1]
  - Macro-average of task-specific metrics across all GLUE tasks.
- `SQuAD 2.0` — range: [0, 1]
  - Maximum of Exact Match and F1 scores between predicted and gold answers.

## Input / output format

**Input**: Text pairs or single text sequences for GLUE tasks; question and context passage for SQuAD 2.0.

**Output**: Class label or regression value for GLUE; span start and end indices (or text) for SQuAD 2.0.

## Scoring recipe

```python
def score(predictions, golds, task):
    if task == 'GLUE':
        return mean([compute_metric(p, g) for p, g in zip(predictions, golds)])
    elif task == 'SQuAD 2.0':
        return max(exact_match(predictions, golds), token_f1(predictions, golds))
# Report median across 5 random seeds
```

## Common pitfalls

- Fine-tuning hyperparameters are tuned per task, introducing variance not attributable to pretraining.
- Results are reported as the median of five random seeds rather than the mean, complicating statistical comparisons.
- Pretraining corpora (Wikipedia, BookCorpus, etc.) may overlap with downstream test sets, potentially inflating scores.

## Evidence (verbatim from paper)

> Downstream Tasks. We use the tasks included in GLUE [54] and SQuAD 2.0 reading compression [41]. Please refer to Appendix A for more details about GLUE tasks. Standard hyperparameter search in fine-tuning is performed, and the search space can be found in Appendix B. The fine-tuning protocols use the open-source implementation of TUPE [26]. The reported results are the median of five random seeds on GLUE and SQuAD.

## Citation

```bibtex
@misc{meng2021cocolm,
  title={COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining},
  author={Meng et al. (2021)},
  year={2021},
  note={arXiv:2102.08473}
}
```

- arXiv: 2102.08473