Ilias Eval

---
name: ilias-eval
description: Evaluates instance-level image retrieval capability, measuring a model's ability to correctly rank specific object instances within a massive, domain-diverse image corpus. It probes robustness to background clutter, scale variations, and the effectiveness of global versus local descriptors for re-ranking. Use when the user wants to benchmark on ILIAS, or asks about evaluating this task. Reports mAP@1k.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.11748
  bibtex_key: kordopatiszilos2025ilias
  confidence: high
---

# ilias-eval

> ILIAS: Instance-Level Image retrieval At Scale — Kordopatis-Zilos et al. (2025) (arXiv:2502.11748, 2025)

## What this evaluates

Evaluates instance-level image retrieval capability, measuring a model's ability to correctly rank specific object instances within a massive, domain-diverse image corpus. It probes robustness to background clutter, scale variations, and the effectiveness of global versus local descriptors for re-ranking.

## Datasets

- **ILIAS** — total 100000000; splits: test (100000000)

## Metrics

- `mAP@1k` **(primary)** — range: [0, 100] percent
  - Mean Average Precision at top-1000. Calculates the average of precision values at the rank positions of all relevant retrieved items, capped at k=1000.

## Input / output format

**Input**: A query (image or text) representing a target object instance, and a candidate database of up to 100M images.

**Output**: A ranked list of the top-1000 candidate images for the query.

## Scoring recipe

```python
def compute_map_at_k(retrieved_indices, ground_truth_indices, k=1000):
    retrieved = retrieved_indices[:k]
    relevant = set(ground_truth_indices)
    hits = 0
    ap_sum = 0.0
    for i, idx in enumerate(retrieved):
        if idx in relevant:
            hits += 1
            ap_sum += hits / (i + 1)
    return (ap_sum / len(relevant)) * 100 if relevant else 0.0
```

## Common pitfalls

- Using in-domain fine-tuned models without acknowledging data leakage from the training sets of comparison datasets (e.g., SOP, GLDv2).
- Confusing the 5M mini-ILIAS subset with the full 100M dataset, as performance drops significantly on the full scale.
- Assuming text-to-image and image-to-image retrieval results are highly correlated; the paper shows they are weakly correlated and benefit from ensembling.

## Evidence (verbatim from paper)

> Table 2. Performance comparison using mAP@1k on ILIAS and mini-ILIAS for global representation models for i2i and t2i.

## Citation

```bibtex
@misc{kordopatiszilos2025ilias,
  title={ILIAS: Instance-Level Image retrieval At Scale},
  author={Kordopatis-Zilos et al. (2025)},
  year={2025},
  note={arXiv:2502.11748}
}
```

- arXiv: 2502.11748