Few Shot No Labels Eval

---
name: few-shot-no-labels-eval
description: Evaluates few-shot image classification capability using a label-free, similarity-based approach. It probes how well self-supervised visual representations can classify novel classes with only a few key images per class, without any training or test labels. Use when the user wants to benchmark on miniImageNet, CIFAR-100FS, FC100, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2012.13751
  bibtex_key: bharti2020fewshot
  confidence: high
---

# few-shot-no-labels-eval

> Few Shot Learning With No Labels — Bharti et al. (2020) (arXiv:2012.13751, 2020)

## What this evaluates

Evaluates few-shot image classification capability using a label-free, similarity-based approach. It probes how well self-supervised visual representations can classify novel classes with only a few key images per class, without any training or test labels.

## Datasets

- **miniImageNet** — total 60000; splits: train (-1), val (-1), test (-1)
- **CIFAR-100FS** — total 60000; splits: train (-1), val (-1), test (-1)
- **FC100** — total 60000; splits: train (-1), val (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Average classification accuracy over 10,000 randomly sampled N-way K-shot tasks. Computed as the ratio of correctly classified query images to total query images across all tasks.

## Input / output format

**Input**: A set of N-way K-shot tasks. Each task consists of N key images (K per class) and Q=15 query images per class, drawn from C novel classes. Images are resized to 84x84 (miniImageNet) or 32x32 (CIFAR/FC100).

**Output**: Predicted class label for each of the Q query images per class.

## Scoring recipe

```python
total_correct = 0
total_queries = 0
for task in tasks:
    preds = classify(task.key_images, task.query_images) # 1NN or Attention
    total_correct += sum(p == g for p, g in zip(preds, task.golds))
    total_queries += len(preds)
return total_correct / total_queries
```

## Common pitfalls

- Q=15 query images are fixed per class regardless of the shot count K, which differs from standard N-way K-shot protocols that often set Q=K or Q=1.
- Results are averaged over 10,000 randomly sampled tasks with 95% confidence intervals, not evaluated on a single fixed train/val/test split.
- The method is strictly label-free; comparing label counts with baselines requires checking whether they use training, validation, or test labels.

## Evidence (verbatim from paper)

> The classifier is presented with 10,000 tasks and average accuracy is reported. Given a test set consisting of C novel classes, we generate a N-way K-shot task as follows. N classes are uniformly sampled from the set of C classes without replacement. From each class, K key and Q=15 query images are uniformly sampled without replacement.

## Citation

```bibtex
@misc{bharti2020fewshot,
  title={Few Shot Learning With No Labels},
  author={Bharti et al. (2020)},
  year={2020},
  note={arXiv:2012.13751}
}
```

- arXiv: 2012.13751