Cross Domain Meta Dl Eval

---
name: cross-domain-meta-dl-eval
description: Probes few-shot image classification generalization across diverse domains and highly variable task regimes (2–20 ways, 1–20 shots) without relying on pre-trained backbones. Use when the user wants to benchmark on Meta-Album, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2208.14686
  bibtex_key: [REDACTED]
  confidence: medium
---

# cross-domain-meta-dl-eval

> NeurIPS'22 Cross-Domain MetaDL competition: Design and baseline results — Carrión-Ojeda et al. (2022) (arXiv:2208.14686, 2022)

## What this evaluates

Probes few-shot image classification generalization across diverse domains and highly variable task regimes (2–20 ways, 1–20 shots) without relying on pre-trained backbones.

## Datasets

- **Meta-Album** — total ?; splits: test (-1); repo https://github.com/DustinCarrion/cd-metacl

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly classified query images in each few-shot task, averaged across all tasks.

## Input / output format

**Input**: Image classification tasks with variable numbers of ways (2–20) and shots (1–20) drawn from 10 diverse domains.

**Output**: Class predictions for query images in each task.

## Scoring recipe

```python
correct = 0
total = 0
for task in tasks:
    preds = model.predict(task.query_images)
    correct += sum(p == g for p, g in zip(preds, task.ground_truth))
    total += len(task.query_images)
return correct / total
```

## Common pitfalls

- Using pre-trained backbones is explicitly forbidden; models must be trained de novo.
- Task regimes are not fixed; models must handle variable ways (2–20) and shots (1–20) in a single evaluation.
- Cross-domain generalization is required; performance on seen domains does not guarantee performance on unseen ones.

## Evidence (verbatim from paper)

> Introduces a novel cross-domain meta-learning challenge (NeurIPS'22 Cross-Domain MetaDL) that evaluates models on 'any-way, any-shot' few-shot image classification across 10 diverse domains (e.g., healthcare, ecology), using Meta-Album—a curated meta-dataset of 40 image classification datasets. Unlike prior benchmarks, it emphasizes domain generalization, variable task regimes (2–20 ways, 1–20 shots), and requires 'de novo' training without pre-trained backbones, testing true meta-learning generalization.

## Citation

```bibtex
@misc{carrionojeda2022crossdomainmetadl,
  title={NeurIPS'22 Cross-Domain MetaDL competition: Design and baseline results},
  author={Carrión-Ojeda et al. (2022)},
  year={2022},
  note={arXiv:2208.14686}
}
```

- arXiv: 2208.14686