Few Shot Classification Eval

---
name: few-shot-classification-eval
description: Evaluates few-shot classification performance under standard and explicit data-shift conditions. It probes a model's ability to generalize from a small number of labeled support samples to query samples across different image domains and distribution shifts. Use when the user wants to benchmark on miniImageNet, Office-Home, Easy-Office-Home, Hard-Office-Home, or asks about evaluating this task. Reports classification accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2101.11354
  bibtex_key: zhu2021combat
  confidence: high
---

# few-shot-classification-eval

> Combat Data Shift in Few-shot Learning with Knowledge Graph — Zhu et al. (2021) (arXiv:2101.11354, 2021)

## What this evaluates

Evaluates few-shot classification performance under standard and explicit data-shift conditions. It probes a model's ability to generalize from a small number of labeled support samples to query samples across different image domains and distribution shifts.

## Datasets

- **miniImageNet** — total 60000; splits: train (64), val (16), test (20)
- **Office-Home** — total 15588; splits: (unstated)
- **Easy-Office-Home** — total ?; splits: train (38), val (12), test (15)
- **Hard-Office-Home** — total ?; splits: train (38), val (12), test (15)

## Metrics

- `classification accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted query samples across all categories in a task. Averaged over 1,000 randomly generated N-way K-shot tasks per dataset, reported with 95% confidence intervals from three random trials.

## Input / output format

**Input**: Image patches of size 84x84 (miniImageNet) or 224x224 (Office-Home variants). Each evaluation task consists of a support set with K labeled samples per class and a query set with 15 unlabeled samples per class.

**Output**: Predicted class labels for each query sample in the task.

## Scoring recipe

```python
correct = 0
total = 0
for _ in range(1000):
    task = sample_task_from_test_set(N_way, K_shot)
    predictions = model.predict(task.query, task.support)
    correct += count_correct(predictions, task.query_labels)
    total += len(task.query_labels)
accuracy = (correct / total) * 100
```

## Common pitfalls

- Baselines on miniImageNet are copied directly from their original papers rather than re-implemented, which complicates fair comparison due to potential implementation differences.
- The constructed Easy/Hard-Office-Home datasets use specific domain assignments (Product vs Clip Art) and training settings (tgt, src+tgt, fulltgt) that differ from standard cross-domain few-shot benchmarks.
- The Knowledge Graph is built externally using WordNet and ImageNet 2012 categories, a non-trivial dependency that is often omitted in reproduction attempts.

## Evidence (verbatim from paper)

> The classification accuracy is computed for all the models by averaging over 1,000 randomly generated new tasks from the test set. All the N-way (with N = 5 and 10) K-shot tasks (with K = 1 and 5) are conducted with K support samples and 15 query samples in each category. All the models are implemented in Pytorch [64] and the average classification accuracy is reported with 95% confidence intervals of three random trials.

## Citation

```bibtex
@misc{zhu2021combat,
  title={Combat Data Shift in Few-shot Learning with Knowledge Graph},
  author={Zhu et al. (2021)},
  year={2021},
  note={arXiv:2101.11354}
}
```

- arXiv: 2101.11354