Nlt Few Shot Classification Eval

---
name: nlt-few-shot-classification-eval
description: Evaluates whether few-shot classification methods can adapt to new tasks without using support set labels at test time. It probes the model's ability to cluster or classify query images based solely on support set images and learned representations. Use when the user wants to benchmark on Omniglot, miniImageNet, tieredImageNet, CUB, Meta-Dataset, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1902.08605
  bibtex_key: huang2019fewshot
  confidence: high
---

# nlt-few-shot-classification-eval

> Are Few-Shot Learning Benchmarks too Simple ? Solving them without Task Supervision at Test-Time — Huang et al. (2019) (arXiv:1902.08605, 2019)

## What this evaluates

Evaluates whether few-shot classification methods can adapt to new tasks without using support set labels at test time. It probes the model's ability to cluster or classify query images based solely on support set images and learned representations.

## Datasets

- **Omniglot** — total ?; splits: test (-1)
- **miniImageNet** — total ?; splits: test (-1)
- **tieredImageNet** — total ?; splits: test (-1)
- **CUB** — total ?; splits: test (-1)
- **Meta-Dataset** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly classified query images averaged over 600 test episodes. For transductive settings, it is averaged over episodes with optimal transport post-processing.

## Input / output format

**Input**: Support set images (and labels for LT methods) and query set images, organized into 5-way 5-shot episodes (or 20-way 5-shot for Omniglot). For NLT, support labels are hidden.

**Output**: Predicted class labels for each query image in the episode.

## Scoring recipe

```python
accuracy = 0.0
for episode in test_episodes:
    preds = model(episode.query_images)
    correct = sum(p == g for p, g in zip(preds, episode.query_labels))
    accuracy += correct / len(episode.query_labels)
return (accuracy / len(test_episodes)) * 100
```

## Common pitfalls

- Omniglot's high accuracy without labels stems from invariant class semantics (alphabet characters), not true task adaptation.
- Cross-domain benchmarks (e.g., miniImageNet -> CUB) show a larger gap between LT and NLT, indicating they better test task adaptation.
- Transductive evaluation assumes a uniform label distribution over the query set, which may not hold in practice.

## Evidence (verbatim from paper)

> Test accuracies are computed over 600 test episodes. We run our NLT baseline on four popular few-shot classification benchmarks: Omniglot [20], miniImageNet [42], tieredImageNet [33], and CUB [44].

## Citation

```bibtex
@misc{huang2019fewshot,
  title={Are Few-Shot Learning Benchmarks too Simple ? Solving them without Task Supervision at Test-Time},
  author={Huang et al. (2019)},
  year={2019},
  note={arXiv:1902.08605}
}
```

- arXiv: 1902.08605