Photobench Eval

---
name: photobench-eval
description: Evaluates personalized, intent-driven photo retrieval capabilities that go beyond simple visual matching. It tests a system's ability to fuse multi-source constraints (temporal, spatial, social identity) and correctly abstain when no relevant image exists in a personal album. Use when the user wants to benchmark on PhotoBench, or asks about evaluating this task. Reports Recall@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.01493
  bibtex_key: xu2026photobench
  confidence: high
---

# photobench-eval

> PhotoBench: Beyond Visual Matching Towards Personalized Intent-Driven Photo Retrieval — Tianyi Xu et al. (2026) (arXiv:2603.01493, 2026)

## What this evaluates

Evaluates personalized, intent-driven photo retrieval capabilities that go beyond simple visual matching. It tests a system's ability to fuse multi-source constraints (temporal, spatial, social identity) and correctly abstain when no relevant image exists in a personal album.

## Datasets

- **PhotoBench** — total ?; splits: test (-1); repo https://github.com/LaVieEnRose365/PhotoBench

## Metrics

- `Recall@K` **(primary)** — range: [0, 1]
  - Fraction of ground-truth relevant images retrieved within the top-K ranked results. Evaluated at K ∈ {1, 5, 10, 20}.
- `NDCG@K` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank K, measuring ranked list quality by discounting the relevance of hits at lower positions.
- `Precision/Recall/F1` — range: [0, 1]
  - Set-based metrics for variable-length outputs. Precision is the fraction of returned images that are relevant, Recall is the fraction of relevant images retrieved, and F1 is their harmonic mean.
- `Reject-Precision/Recall/F1` — range: [0, 1]
  - Metrics for Zero-GT queries measuring abstention ability. Reject-Recall is the proportion of empty-GT queries correctly identified as having no matches, while Reject-Precision measures the reliability of empty responses.

## Input / output format

**Input**: Natural language query/intent (potentially implying spatio-temporal or social constraints) over a personal photo album.

**Output**: Fixed-length ranked list of up to K images (for embedding models) or a variable-length set of images/empty set (for agents and mobile systems).

## Scoring recipe

```python
def recall_at_k(preds, gold, k):
    top_k = set(preds[:k])
    return len(top_k & set(gold)) / len(gold) if gold else 0

def set_metrics(preds, gold):
    tp = len(set(preds) & set(gold))
    prec = tp / len(preds) if preds else 0
    rec = tp / len(gold) if gold else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return prec, rec, f1

def reject_metrics(preds, gold):
    is_empty = len(preds) == 0
    return is_empty, is_empty, is_empty
```

## Common pitfalls

- Ignoring Zero-GT queries: Systems must correctly abstain when no relevant photo exists; returning any image counts as a retrieval hallucination.
- Treating retrieval as purely visual matching: Fails to account for non-visual constraints (timestamps, GPS, social roles) that define intent-driven queries.
- Assuming single-label ground truth: PhotoBench uses one-to-many matches with variable ground truth sizes, requiring set-based or ranking metrics rather than exact-match accuracy.

## Evidence (verbatim from paper)

> PhotoBench presents two evaluation challenges: (1) it supports one-to-many matches with variable ground truth sizes, and (2) it includes zero-ground-truth (Zero-GT) queries that require system abstention. Hence, we employ two complementary metric families: Top-K Ranking Metrics. Designed for embedding models that output fixed-length lists. We report Recall@K and NDCG@K with K∈{1,5,10,20}, covering the spectrum from best hit to broad shortlists. Set-Based Metrics. Only suitable for hybrid retrieval systems (i.e., Agents and Phones) that return variable-length sets. We evaluate performance across two query types: Normal Query. We report standard Precision, Recall, and F1 to measure the accuracy of the returned image set against the comprehensive ground truth set. Zero-GT Query. To measure systems’ ability to correctly abstain (reject) when no relevant photo exists, we report Reject-Precision, Reject-Recall, and Reject-F1.

## Citation

```bibtex
@misc{xu2026photobench,
  title={PhotoBench: Beyond Visual Matching Towards Personalized Intent-Driven Photo Retrieval},
  author={Tianyi Xu et al. (2026)},
  year={2026},
  note={arXiv:2603.01493}
}
```

- arXiv: 2603.01493