Oven Eval

---
name: oven-eval
description: This benchmark probes a model's ability to recognize visual entities in an open-domain setting, specifically testing zero-shot generalization to entities not seen during training. It evaluates how well a model can align visual inputs with structured knowledge graph descriptions to perform entity retrieval. Use when the user wants to benchmark on OVEN, or asks about evaluating this task. Reports Harmonic Mean (HM) of top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.13675
  bibtex_key: zhou2025seeing
  confidence: high
---

# oven-eval

> Seeing and Knowing in the Wild: Open-domain Visual Entity Recognition with Large-scale Knowledge Graphs via Contrastive Learning — Zhou et al. (2025) (arXiv:2510.13675, 2025)

## What this evaluates

This benchmark probes a model's ability to recognize visual entities in an open-domain setting, specifically testing zero-shot generalization to entities not seen during training. It evaluates how well a model can align visual inputs with structured knowledge graph descriptions to perform entity retrieval.

## Datasets

- **OVEN** — total 6063945; splits: train (6063945), test (15888)

## Metrics

- `Harmonic Mean (HM) of top-1 accuracy` **(primary)** — range: percent
  - The harmonic mean of top-1 accuracy on seen entities and top-1 accuracy on unseen entities. Formula: HM = 2 * (acc_seen * acc_unseen) / (acc_seen + acc_unseen).
- `top-1 accuracy (seen)` — range: percent
  - Percentage of test images correctly matched to their ground-truth entity among entities present in the training set.
- `top-1 accuracy (unseen)` — range: percent
  - Percentage of test images correctly matched to their ground-truth entity among entities absent from the training set.

## Input / output format

**Input**: An input image paired with a text query (entity description or prototype) from a predefined label space.

**Output**: A predicted entity label (top-1 match) from the candidate set, determined by nearest-neighbor search in the shared latent space.

## Scoring recipe

```python
acc_seen = correct_seen / total_seen
acc_unseen = correct_unseen / total_unseen
hm = 2 * (acc_seen * acc_unseen) / (acc_seen + acc_unseen)
return hm
```

## Common pitfalls

- Confusing the seen and unseen entity splits, which have vastly different performance characteristics.
- Using generative captioning metrics instead of retrieval-based top-1 accuracy.
- Overfitting to seen entities significantly degrades zero-shot performance on unseen entities.

## Evidence (verbatim from paper)

> We evaluate the harmonic mean (HM) of the seen and unseen splits (top1 accuracy) after fine tuning on OVEN training set. It serves as our main metric. The evaluation metric is top-1 accuracy for seen and unseen entities, and the harmonic mean (HM) of them.

## Citation

```bibtex
@misc{zhou2025seeing,
  title={Seeing and Knowing in the Wild: Open-domain Visual Entity Recognition with Large-scale Knowledge Graphs via Contrastive Learning},
  author={Zhou et al. (2025)},
  year={2025},
  note={arXiv:2510.13675}
}
```

- arXiv: 2510.13675