Fewshot Ft Vs Icl Eval

---
name: fewshot-ft-vs-icl-eval
description: Evaluates the in-domain and out-of-domain generalization capabilities of large language models adapted via few-shot fine-tuning versus in-context learning across standard natural language inference and paraphrase detection benchmarks. Use when the user wants to benchmark on MNLI, RTE, QQP, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.16938
  bibtex_key: mosbach2023fewshot
  confidence: high
---

# fewshot-ft-vs-icl-eval

> Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation — Mosbach et al. (2023) (arXiv:2305.16938, 2023)

## What this evaluates

Evaluates the in-domain and out-of-domain generalization capabilities of large language models adapted via few-shot fine-tuning versus in-context learning across standard natural language inference and paraphrase detection benchmarks.

## Datasets

- **MNLI** — total ?; splits: train (-1), test (-1)
- **RTE** — total ?; splits: train (-1), test (-1)
- **QQP** — total ?; splits: train (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy: the fraction of correctly predicted labels out of the total number of test instances.

## Input / output format

**Input**: Natural language inference and paraphrase detection sentences formatted as few-shot prompts with 16 demonstrations (ICL) or as fine-tuning examples (FT).

**Output**: Classification labels (e.g., entailment/contradiction/neutral for MNLI; entailment/not_entailment for RTE; same/different for QQP).

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Prior studies often compared ICL and FT using mismatched model sizes, leading to biased conclusions about OOD robustness.
- Model selection strategy (e.g., picking checkpoints based on in-domain vs. OOD performance) drastically changes reported OOD generalization results.
- Both methods are highly sensitive to prompt design and training instability, requiring multiple random seeds for reliable evaluation.

## Evidence (verbatim from paper)

> We present the results for in-domain and OOD model performance in Figure 2, comparing both ICL and FT. We perform task adaptation using 16 examples for both strategies. ... On MNLI the largest model (30B) obtains an average performance of 71.4% and a maximum performance of 74.9%. ... — in the x- and y-axis indicates majority class accuracy.

## Citation

```bibtex
@misc{mosbach2023fewshot,
  title={Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation},
  author={Mosbach et al. (2023)},
  year={2023},
  note={arXiv:2305.16938}
}
```

- arXiv: 2305.16938