Matching Networks Eval

---
name: matching-networks-eval
description: Evaluates a model's ability to perform few-shot classification by learning to map a small support set of labeled examples to a classifier for unseen classes without fine-tuning. It probes rapid adaptation and generalization across vision and language modalities using an attention-based non-parametric memory mechanism. Use when the user wants to benchmark on Omniglot, ImageNet, miniImageNet, Penn Treebank, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1606.04080
  bibtex_key: vinyals2016matching
  confidence: high
---

# matching-networks-eval

> Matching Networks for One Shot Learning — Vinyals et al. (2016) (arXiv:1606.04080, 2016)

## What this evaluates

Evaluates a model's ability to perform few-shot classification by learning to map a small support set of labeled examples to a classifier for unseen classes without fine-tuning. It probes rapid adaptation and generalization across vision and language modalities using an attention-based non-parametric memory mechanism.

## Datasets

- **Omniglot** — total ?; splits: train (-1), test (-1)
- **ImageNet** — total ?; splits: train (-1), test (-1)
- **miniImageNet** — total 60000; splits: train (48000), test (12000)
- **Penn Treebank** — total ?; splits: train (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard classification accuracy: the proportion of correctly predicted class labels out of the total number of query examples in the batch. Reported as percentage across N-way and k-shot configurations.

## Input / output format

**Input**: Support set S containing k labeled examples per class from N unseen classes, and a query batch B containing unlabeled examples from the same N classes. For language tasks, a query sentence with a masked token and a support set of sentences with masked tokens and 1-hot labels.

**Output**: Predicted class label (one of N classes) for each example in the query batch.

## Scoring recipe

```python
correct = 0
total = 0
for query_example in query_batch:
    pred_label = model.predict(query_example, support_set)
    if pred_label == gold_label(query_example):
        correct += 1
    total += 1
accuracy = (correct / total) * 100
```

## Common pitfalls

- Confusing N-way (number of classes) with k-shot (examples per class); the protocol strictly defines N as classes and k as support examples per class.
- Ensuring strict disjointness between training classes, support set classes, and query classes to prevent data leakage and unfair advantage.
- Fine-tuning on the support set can cause massive overfitting; the evaluation must explicitly state whether fine-tuning is permitted, as results vary significantly.

## Evidence (verbatim from paper)

> The LSTM language model oracle achieved an upper bound of 72.8% accuracy on the test set. Matching Networks with a simple encoding model achieve 32.4%, 36.1%, 38.2% accuracy on the task with k = 1, 2, 3 examples in the set, respectively.

## Citation

```bibtex
@misc{vinyals2016matching,
  title={Matching Networks for One Shot Learning},
  author={Vinyals et al. (2016)},
  year={2016},
  note={arXiv:1606.04080}
}
```

- arXiv: 1606.04080