Metadl Fsl Eval

---
name: metadl-fsl-eval
description: Evaluates few-shot image classification models on real-world domains using a standardized N-way K-shot framework. It probes the ability of models to quickly adapt to new classes with limited labeled examples and generalize across diverse image domains. Use when the user wants to benchmark on MetaDL meta-datasets (1-5), or asks about evaluating this task. Reports average rank.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2206.08138
  bibtex_key: elbaz2022metadl
  confidence: high
---

# metadl-fsl-eval

> Lessons learned from the NeurIPS 2021 MetaDL challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classification — El Baz et al. (2022) (arXiv:2206.08138, 2022)

## What this evaluates

Evaluates few-shot image classification models on real-world domains using a standardized N-way K-shot framework. It probes the ability of models to quickly adapt to new classes with limited labeled examples and generalize across diverse image domains.

## Datasets

- **MetaDL meta-datasets (1-5)** — total ?; splits: test (-1); repo https://github.com/ebadrian/metadl

## Metrics

- `average rank` **(primary)** — range: rank
  - Average of per-dataset ranks across 5 meta-datasets. Lower is better. Per-dataset rank is assigned based on the minimum accuracy observed across 3 independent runs with different random seeds.
- `accuracy` — range: [0, 1]
  - Average classification accuracy over 600 episodes per meta-dataset, computed as correct predictions divided by total query images.

## Input / output format

**Input**: Image classification episodes consisting of a support set (5-way, 5-shot) and a query set drawn from a meta-test set.

**Output**: Predicted class labels for each query image in the episode.

## Scoring recipe

```python
# Per meta-dataset:
run_accuracies = []
for seed in range(3):
    acc = compute_accuracy(model, dataset, n_way=5, n_shot=5, n_episodes=600, seed=seed)
    run_accuracies.append(acc)
dataset_min_acc = min(run_accuracies)

# Across 5 meta-datasets:
dataset_ranks = rank_submissions_by_accuracy(dataset_min_accs)
final_score = mean(dataset_ranks)
```

## Common pitfalls

- Using the mean accuracy across the 3 runs instead of the minimum (worst) accuracy.
- Hardcoding the number of classes for episode generation, which fails when the final phase meta-datasets differ from the feedback phase.
- Tie-breaking relies on submission timestamp rather than performance.

## Evidence (verbatim from paper)

> To increase reproducibility, and avoid participants winning by chance, we ran each submission 3 times, using different random seeds. For each metadata-set, we considered only the lowest accuracy among these 3 runs. Regardless of the metric used we obtained the same top-4 results. The choice between these metrics is tightly linked to the goal of the problem which usually is either to have a 'generalist' or 'specialist' algorithm (Pavao et al., 2021). The competition aims to foster algorithms that are capable of quickly dealing with few-shot image classification problems within a single image domain. Hence, we preferred the average rank metric which emphasized generalist behaviour. Again, for each metadata-set, each algorithm is evaluated on 600 episodes with a 5-way 5-shot configuration. Classes and associated images are drawn from the associated meta-test set. The Accuracy columns are the average accuracy over the 600 episodes of the corresponding meta-dataset.

## Citation

```bibtex
@misc{elbaz2022metadl,
  title={Lessons learned from the NeurIPS 2021 MetaDL challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classification},
  author={El Baz et al. (2022)},
  year={2022},
  note={arXiv:2206.08138}
}
```

- arXiv: 2206.08138