Raft Few Shot Classification Eval

---
name: raft-few-shot-classification-eval
description: Evaluates few-shot text classification on real-world tasks with class imbalance and long inputs. It probes a model's ability to leverage limited labeled examples, domain knowledge, and open-domain retrieval to classify text without a validation set. Use when the user wants to benchmark on RAFT, or asks about evaluating this task. Reports macro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2109.14076
  bibtex_key: alex2021raft
  confidence: high
---

# raft-few-shot-classification-eval

> RAFT: A Real-World Few-Shot Text Classification Benchmark — Alex et al. (2021) (arXiv:2109.14076, 2021)

## What this evaluates

Evaluates few-shot text classification on real-world tasks with class imbalance and long inputs. It probes a model's ability to leverage limited labeled examples, domain knowledge, and open-domain retrieval to classify text without a validation set.

## Datasets

- **RAFT** — total ?; splits: train (50), test (-1)

## Metrics

- `macro-F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall, averaged across all classes (macro-averaged) for each dataset, then averaged across all 11 datasets.

## Input / output format

**Input**: Natural language text input, task-specific instructions, and exactly 50 labeled few-shot examples. Models may optionally use open-domain web retrieval and unlabeled test data.

**Output**: A single predicted class label from the provided natural language class names.

## Scoring recipe

```python
def compute_raft_score(predictions, gold, datasets):
    dataset_f1s = []
    for ds in datasets:
        preds = predictions[ds]
        golds = gold[ds]
        # Compute macro-F1 per dataset
        f1 = macro_f1_score(golds, preds)
        dataset_f1s.append(f1)
    return sum(dataset_f1s) / len(dataset_f1s)
```

## Common pitfalls

- Using a validation set for tuning is strictly forbidden; only 50 training examples are provided with no validation split.
- Assuming balanced class distributions; datasets exhibit heavy imbalance, making accuracy misleading and requiring macro-F1.
- Counting the 50-shot limit per class instead of per task; the limit applies to the entire task regardless of number of classes.

## Evidence (verbatim from paper)

> Since some RAFT datasets have substantial class imbalances, we use F1 as our evaluation metric. We compute macro-averaged F1 scores, even for binary datasets. To get an overall score, we average across all datasets.

## Citation

```bibtex
@misc{alex2021raft,
  title={RAFT: A Real-World Few-Shot Text Classification Benchmark},
  author={Alex et al. (2021)},
  year={2021},
  note={arXiv:2109.14076}
}
```

- arXiv: 2109.14076