Meta Omnium Eval

---
name: meta-omnium-eval
description: Evaluates few-shot meta-learners and transfer learning baselines on their ability to generalize across heterogeneous vision tasks including classification, semantic segmentation, keypoint localization, and regression. It specifically probes cross-task knowledge transfer, in-distribution versus out-of-distribution robustness, and the comparative effectiveness of single-task versus multi-task meta-training protocols. Use when the user wants to benchmark on Meta Omnium, or asks about evaluating this task. Reports Average Rank.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.07625
  bibtex_key: bohdal2023metaomnium
  confidence: high
---

# meta-omnium-eval

> Meta Omnium: A Benchmark for General-Purpose Learning-to-Learn — Bohdal et al. (2023) (arXiv:2305.07625, 2023)

## What this evaluates

Evaluates few-shot meta-learners and transfer learning baselines on their ability to generalize across heterogeneous vision tasks including classification, semantic segmentation, keypoint localization, and regression. It specifically probes cross-task knowledge transfer, in-distribution versus out-of-distribution robustness, and the comparative effectiveness of single-task versus multi-task meta-training protocols.

## Datasets

- **Meta Omnium** — total ?; splits: meta-training (-1), meta-validation (-1), meta-testing (-1)

## Metrics

- `accuracy` — range: percent
  - Percentage of correctly predicted class labels on query images within few-shot episodes.
- `mIOU` — range: percent
  - Mean Intersection over Union computed across all semantic classes for predicted segmentation masks.
- `PCK` — range: percent
  - Percentage of Correct Keypoints, measuring localization accuracy where predicted coordinates fall within a normalized distance threshold of the ground truth.
- `Average Rank` **(primary)** — range: other
  - Mean rank of a method across all datasets within a task type (and across task types), computed by ordering methods by performance on each dataset and averaging the resulting ranks.

## Input / output format

**Input**: Few-shot learning episodes consisting of support sets (images with labels, masks, or keypoint coordinates) and query sets for classification, segmentation, keypoint localization, or regression tasks.

**Output**: Predictions for query set items: class labels, pixel-wise segmentation masks, keypoint coordinates, or continuous regression values.

## Scoring recipe

```python
def compute_scores(predictions, gold, task_type):
    if task_type == 'classification':
        return accuracy(predictions, gold)
    elif task_type == 'segmentation':
        return mIOU(predictions, gold)
    elif task_type == 'keypoints':
        return PCK(predictions, gold)
    # For regression, use dataset-specific metric
    return dataset_metric(predictions, gold)

# Aggregate across 600 meta-testing tasks per dataset
# Average scores across datasets within each task type
# Compute Average Rank by sorting methods per dataset and averaging ranks
```

## Common pitfalls

- Evaluating only in-distribution (ID) tasks masks significant performance degradation on out-of-distribution (OOD) datasets, particularly for segmentation and keypoint tasks.
- Assuming multi-task meta-training automatically outperforms single-task training ignores the negative impact of representation heterogeneity across task families.
- Applying standard ImageNet pre-training without conditional evaluation may yield inconsistent or negligible gains in this multi-task meta-learning setting.

## Evidence (verbatim from paper)

> Classification, segmentation, and keypoint results are reported in accuracy (%), mIOU (%), and PCK (%) respectively. The table also reports the average rank of each meta-learner across each dataset, both overall and broken down by ID and OOD datasets.

## Citation

```bibtex
@misc{bohdal2023metaomnium,
  title={Meta Omnium: A Benchmark for General-Purpose Learning-to-Learn},
  author={Bohdal et al. (2023)},
  year={2023},
  note={arXiv:2305.07625}
}
```

- arXiv: 2305.07625