Glue Fewshot Eval

---
name: glue-fewshot-eval
description: Evaluates few-shot text classification performance across multiple natural language understanding tasks. It probes a model's ability to generalize from extremely limited labeled examples (16 per class) by generating synthetic training data and fine-tuning a classifier. Use when the user wants to benchmark on GLUE, or asks about evaluating this task. Reports Average performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2211.03044
  bibtex_key: meng2022fewgen
  confidence: high
---

# glue-fewshot-eval

> Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning — Meng et al. (2022) (arXiv:2211.03044, 2022)

## What this evaluates

Evaluates few-shot text classification performance across multiple natural language understanding tasks. It probes a model's ability to generalize from extremely limited labeled examples (16 per class) by generating synthetic training data and fine-tuning a classifier.

## Datasets

- **GLUE** — total ?; splits: train (-1), dev (-1), test (-1); HF `glue`

## Metrics

- `Accuracy` — range: [0, 1]
  - Proportion of correctly predicted labels out of total instances.
- `F1 score` — range: [0, 1]
  - Harmonic mean of precision and recall, computed per class and averaged according to standard GLUE conventions.
- `Matthews correlation` — range: [-1, 1]
  - Correlation coefficient between true and predicted binary classifications, ranging from -1 to 1.
- `Average performance` **(primary)** — range: [0, 1]
  - Mean of task-specific scores (Accuracy, F1, or MCC) across all evaluated GLUE tasks, reported with standard deviation over 5 random few-shot splits.

## Input / output format

**Input**: Raw text sequence (sentence or pair of sentences) for a given GLUE task.

**Output**: Discrete class label corresponding to the task's vocabulary.

## Scoring recipe

```python
def compute_glue_metrics(predictions, gold_labels, task_name):
    if task_name in ['QQP', 'MRPC']:
        return f1_score(gold_labels, predictions, average='macro')
    elif task_name == 'CoLA':
        return matthews_corrcoef(gold_labels, predictions)
    else:
        return accuracy_score(gold_labels, predictions)
```

## Common pitfalls

- The paper uses the official GLUE development set as the test set, not the hidden test server. Evaluators must replicate this split to match reported numbers.
- Few-shot training/dev splits are strictly limited to 16 samples per label, sampled from the original training set. Using the full training set or different few-shot sizes will yield different results.
- STS-B is explicitly excluded because it is a regression task; including it will break the metric calculation and average score.

## Evidence (verbatim from paper)

> We conduct evaluation on all tasks of the GLUE benchmark (Wang et al., 2018) except STS-B which is a regression task. We follow the same data split and evaluation protocol as (Gao et al., 2021): Both D_train and D_dev contain 16 samples per label and are sampled from the original training set with 5 different random seeds. The original development sets are used for testing. For all reported results, we include the average and standard deviation over the 5 different D_train/D_dev splits. F1 score is used as the metric for QQP and MRPC, Matthews correlation for CoLA, and accuracy for the remaining tasks.

## Citation

```bibtex
@misc{meng2022fewgen,
  title={Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning},
  author={Meng et al. (2022)},
  year={2022},
  note={arXiv:2211.03044}
}
```

- arXiv: 2211.03044