Few Shot Meta Learning Eval

---
name: few-shot-meta-learning-eval
description: Evaluates few-shot classification performance of meta-learning algorithms on standard image datasets. It specifically probes robustness to distribution shift or difficulty by measuring accuracy on dynamically identified 'hard' episodes versus average episodic performance. Use when the user wants to benchmark on CIFAR-FS, mini-ImageNet, tieredImageNet, or asks about evaluating this task. Reports episodic accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2110.11190
  bibtex_key: basu2021hard
  confidence: high
---

# few-shot-meta-learning-eval

> On Hard Episodes in Meta-Learning — Basu et al. (2021) (arXiv:2110.11190, 2021)

## What this evaluates

Evaluates few-shot classification performance of meta-learning algorithms on standard image datasets. It specifically probes robustness to distribution shift or difficulty by measuring accuracy on dynamically identified 'hard' episodes versus average episodic performance.

## Datasets

- **CIFAR-FS** — total ?; splits: train (-1), val (-1), test (-1)
- **mini-ImageNet** — total ?; splits: train (-1), val (-1), test (-1)
- **tieredImageNet** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `episodic accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly classified query images in a few-shot classification episode, averaged over 1,000 test episodes for overall performance, or over the 30 hardest episodes for hard-episode performance.

## Input / output format

**Input**: Few-shot classification episodes comprising a support set (K labeled examples per N classes) and a query set (unlabeled examples to classify).

**Output**: Predicted class labels for each query image in the episode.

## Scoring recipe

```python
correct = 0
total = 0
for episode in test_episodes:
    pred_labels = model.predict(episode.query, episode.support)
    correct += sum(p == g for p, g in zip(pred_labels, episode.ground_truth))
    total += len(episode.query)
return correct / total
```

## Common pitfalls

- Hard episodes are not a static dataset split; they are identified post-hoc as the 30 episodes with the lowest accuracy for each specific meta-learner.
- Model selection uses 2,000 validation episodes every 1,000 training iterations, which differs from the standard 1,000-episode test evaluation.
- 1-shot and 5-shot results are reported separately and must not be averaged across shot counts.

## Evidence (verbatim from paper)

> Finally, we evaluate on 1k test episodes from the test set for each dataset. ... Table 2: Performance of general adversarial training (AT) and adversarial curriculum training (ACT) across different meta-learners on hard episodes. We report the mean accuracy over 30 hardest episodes for each meta-learner.

## Citation

```bibtex
@misc{basu2021hard,
  title={On Hard Episodes in Meta-Learning},
  author={Basu et al. (2021)},
  year={2021},
  note={arXiv:2110.11190}
}
```

- arXiv: 2110.11190