Cirthan Eval

---
name: cirthan-eval
description: Evaluates composed image retrieval capabilities on culturally specific Thangka imagery. It tests the model's ability to align fine-grained sketch+text queries with target images across varying levels of textual semantic granularity, highlighting the domain gap between generic pre-training and specialized cultural retrieval. Use when the user wants to benchmark on CIRThan, or asks about evaluating this task. Reports Recall@K (R@K).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.08411
  bibtex_key: xu2026cirthan
  confidence: high
---

# cirthan-eval

> A Sketch+Text Composed Image Retrieval Dataset for Thangka — Xu et al. (2026) (arXiv:2602.08411, 2026)

## What this evaluates

Evaluates composed image retrieval capabilities on culturally specific Thangka imagery. It tests the model's ability to align fine-grained sketch+text queries with target images across varying levels of textual semantic granularity, highlighting the domain gap between generic pre-training and specialized cultural retrieval.

## Datasets

- **CIRThan** — total 2287; splits: train (1868), test (419); repo https://github.com/jinyuxu-whut/CIRThan

## Metrics

- `Recall@K (R@K)` **(primary)** — range: percent
  - For each composed query, all gallery images are ranked by similarity. Retrieval is considered successful if the ground-truth target image appears within the top-K results. Reported for K ∈ {1, 3, 5} and averaged across all queries.

## Input / output format

**Input**: A composed query consisting of a human-drawn sketch paired with a hierarchical textual description (Level 1, 2, or 3), and a gallery of candidate Thangka images.

**Output**: A ranked list of gallery images by similarity score, or a binary success indicator per query for top-K evaluation.

## Scoring recipe

```python
def compute_recall_at_k(predictions, gold_indices, k):
    hits = 0
    for pred, gold in zip(predictions, gold_indices):
        if gold in pred[:k]:
            hits += 1
    return hits / len(predictions) * 100
```

## Common pitfalls

- Zero-shot methods struggle significantly with domain-specific visual semantics, often failing to align sketch and text cues without in-domain supervision.
- Single-modality baselines (sketch-only or text-only) perform poorly, highlighting the necessity of cross-modal fine-grained alignment for Thangka retrieval.
- Retrieval performance is highly sensitive to textual granularity; Level 3 descriptions consistently yield higher recall than Level 1 or 2.

## Evidence (verbatim from paper)

> Following common practice in composed image retrieval, we evaluate all methods using Recall@K (R@K). For each composed query, all Thangka images in the gallery are ranked by similarity, and retrieval is considered successful if the ground-truth target Thangka appears within the top-K results. We report R@K for $K\in{1,3,5}$, as each query in CIRThan has a single ground-truth target.

## Citation

```bibtex
@misc{xu2026cirthan,
  title={A Sketch+Text Composed Image Retrieval Dataset for Thangka},
  author={Xu et al. (2026)},
  year={2026},
  note={arXiv:2602.08411}
}
```

- arXiv: 2602.08411