Meta Album Eval

---
name: meta-album-eval
description: Evaluates few-shot image classification performance across 40 diverse datasets spanning 10 domains. It probes a model's ability to adapt to new classes with limited labeled examples (1, 5, 10, 20-shot) in within-domain settings. Use when the user wants to benchmark on Meta-Album, or asks about evaluating this task. Reports average accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2302.08909
  bibtex_key: ullah2023metaalbum
  confidence: high
---

# meta-album-eval

> Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classification — Ullah et al. (2023) (arXiv:2302.08909, 2023)

## What this evaluates

Evaluates few-shot image classification performance across 40 diverse datasets spanning 10 domains. It probes a model's ability to adapt to new classes with limited labeled examples (1, 5, 10, 20-shot) in within-domain settings.

## Datasets

- **Meta-Album** — total 40; splits: train (-1), test (-1); repo https://github.com/ihsaan-ullah/meta-album

## Metrics

- `average accuracy` **(primary)** — range: percent
  - Percentage of correctly classified query images in a few-shot task. Computed per task and averaged across tasks and datasets.
- `average rank` — range: other
  - Ranking of methods based on average accuracy, averaged across datasets.

## Input / output format

**Input**: Support set (labeled images per class) and query set (unlabeled images) for few-shot classification tasks.

**Output**: Predicted class labels for each query image.

## Scoring recipe

```python
For each few-shot task: compute accuracy = (correct predictions / total query images) * 100.
Average accuracy across all tasks within a dataset, then across all datasets.
Compute 95% confidence intervals at the per-task level over 3 runs with 600 tasks per run.
```

## Common pitfalls

- Confidence intervals for average rank are computed at the dataset level, while those for accuracy are computed at the per-task level.
- MatchingNet and ProtoNet use different distance measures (cosine similarity vs squared Euclidean distance), so their 1-shot performances are not directly comparable in the same way.

## Evidence (verbatim from paper)

> Table 5, Table 6, Table 7, and Table 8 display the average accuracy per technique and dataset in the [1, 5, 10, 20]-shot settings, respectively. Note that the 1-shot performance of matching networks and prototypical networks is not the same as they use different distance measures.

## Citation

```bibtex
@misc{ullah2023metaalbum,
  title={Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classification},
  author={Ullah et al. (2023)},
  year={2023},
  note={arXiv:2302.08909}
}
```

- arXiv: 2302.08909