Glue Sni Eval

---
name: glue-sni-eval
description: Evaluates the downstream generalization and data efficiency of pre-trained language models by fine-tuning them on standard natural language understanding and instruction-following benchmarks. Use when the user wants to benchmark on GLUE, SuperNatural-Instructions (SNI), or asks about evaluating this task. Reports GLUE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.08442
  bibtex_key: kaddour2023minipile
  confidence: high
---

# glue-sni-eval

> The MiniPile Challenge for Data-Efficient Language Models — Kaddour et al. (2023) (arXiv:2304.08442, 2023)

## What this evaluates

Evaluates the downstream generalization and data efficiency of pre-trained language models by fine-tuning them on standard natural language understanding and instruction-following benchmarks.

## Datasets

- **GLUE** — total ?; splits: train (-1), dev (-1), test (-1)
- **SuperNatural-Instructions (SNI)** — total ?; splits: train (-1), test (-1)

## Metrics

- `GLUE` **(primary)** — range: percent
  - Macro-averaged performance across all GLUE sub-tasks, using accuracy, F1, or correlation as appropriate per task, then averaged. Reported as a percentage or normalized score.
- `SNI` — range: percent
  - Exact-match or token-level accuracy across instruction-following tasks, aggregated per task and averaged.

## Input / output format

**Input**: Task-specific prompts, instruction-response pairs, or sentence pairs for fine-tuning and evaluation.

**Output**: Predicted labels, text completions, or numerical answers corresponding to each downstream task.

## Scoring recipe

```python
def compute_metrics(predictions, golds, task_names):
    scores = []
    for preds, golds, task in zip(predictions, golds, task_names):
        if task in ['cola', 'sst-2', 'mrpc', 'qqp', 'mnli', 'qnli', 'rte', 'wnli']:
            scores.append(accuracy_score(golds, preds))
        elif task == 'stsb':
            scores.append(pearsonr(golds, preds)[0])
        else:
            scores.append(f1_score(golds, preds, average='macro'))
    return sum(scores) / len(scores) * 100
```

## Common pitfalls

- GLUE contains diverse sub-tasks with different evaluation metrics (accuracy, F1, correlation); results must be averaged correctly per official GLUE guidelines.
- SNI covers highly varied instruction formats; exact-match vs. token-level scoring can vary significantly across tasks and must be applied consistently.
- Comparisons against baselines require identical fine-tuning hyperparameters and data splits to isolate the effect of pre-training corpus quality.

## Evidence (verbatim from paper)

> We evaluate our pre-trained models on the General Language Understanding Evaluation (GLUE) [55] and SuperNatural-Instructions (SNI) [57] benchmarks. ... Tables 3 and 4 show the results compared against the publicly available checkpoints trained on 2.6x / 745x the amount of data... We observe minor reductions in final downstream performance and conjecture that MiniPile is a well-suited pre-training corpus for common downstream benchmarks.

## Citation

```bibtex
@misc{kaddour2023minipile,
  title={The MiniPile Challenge for Data-Efficient Language Models},
  author={Kaddour et al. (2023)},
  year={2023},
  note={arXiv:2304.08442}
}
```

- arXiv: 2304.08442