Zs Cir Eval

---
name: zs-cir-eval
description: Evaluates a model's ability to retrieve target images based on a reference image and a natural language modification text. It probes fine-grained visual-semantic alignment, compositional reasoning, and ranking precision under varying levels of distractors and semantic transformations. Use when the user wants to benchmark on CIRR, CIRCO, FashionIQ, GeneCIS, or asks about evaluating this task. Reports Recall@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.26330
  bibtex_key: wu2025square
  confidence: high
---

# zs-cir-eval

> SQUARE: Semantic Query-Augmented Fusion and Efficient Batch Reranking for Training-free Zero-Shot Composed Image Retrieval — Wu et al. (2025) (arXiv:2509.26330, 2025)

## What this evaluates

Evaluates a model's ability to retrieve target images based on a reference image and a natural language modification text. It probes fine-grained visual-semantic alignment, compositional reasoning, and ranking precision under varying levels of distractors and semantic transformations.

## Datasets

- **CIRR** — total ?; splits: test (-1)
- **CIRCO** — total ?; splits: test (-1)
- **FashionIQ** — total ?; splits: val (-1)
- **GeneCIS** — total 8032; splits: test (-1)

## Metrics

- `Recall@K` **(primary)** — range: percent
  - For each query, checks if the ground-truth image appears in the top-K retrieved results. The metric is the percentage of queries satisfying this condition across the dataset.
- `mAP@K` — range: percent
  - Mean Average Precision at K. Computes the average precision for each query considering multiple ground-truth images, then averages across all queries up to rank K.
- `Recall_Subset@K` — range: percent
  - Recall@K computed on a restricted candidate pool of semantically similar images and challenging negatives, rather than the full retrieval set.
- `R@10 / R@50` — range: percent
  - Recall at K=10 and K=50, computed separately for each clothing category (Shirt, Dress, Toptee) and averaged across categories.

## Input / output format

**Input**: A reference image, a natural language modification text, and a retrieval candidate pool (images).

**Output**: A ranked list of candidate images (or top-16 subset for the reranking stage).

## Scoring recipe

```python
def compute_recall_at_k(retrieved_indices, gt_idx, k):
    return int(gt_idx in retrieved_indices[:k])

def compute_mAP_at_k(retrieved_indices, gt_indices, k):
    relevant = 0
    sum_prec = 0.0
    for i, idx in enumerate(retrieved_indices[:k]):
        if idx in gt_indices:
            relevant += 1
            sum_prec += relevant / (i + 1)
    return sum_prec / len(gt_indices) if gt_indices else 0.0

# Aggregate over dataset
total_recall = sum(compute_recall_at_k(preds, gt, k) for preds, gt in dataset) / len(dataset)
total_map = sum(compute_mAP_at_k(preds, gts, k) for preds, gts in dataset) / len(dataset)
```

## Common pitfalls

- CIRR and CIRCO results must be submitted to official external evaluation servers; local computation may yield inconsistent results due to hidden test splits or specific ranking protocols.
- FashionIQ results are reported on the validation set, not the test set, which is non-standard for many benchmarks.
- GeneCIS evaluation varies by sub-task: 'Focus on an Attribute' uses a retrieval set of 10 images, while other tasks use 15, affecting absolute Recall@K values.

## Evidence (verbatim from paper)

> We report Recall@K (R@K) with K = 1, 5, and 10 for CIRR, which evaluates whether the ground-truth image appears among the top-K retrieved results. Additionally, we report Recall${}_{\text{Subset}}$@K with K = 1, 2, and 3 in the subset setting of CIRR... For CIRCO, we adopt the mean Average Precision at K (mAP@K) with K = 5, 10, 25, and 50, which accounts for multiple ground-truths per query... For FashionIQ, we compute R@10 and R@50 separately for each clothing category and report the average across categories. In the case of GeneCIS, we evaluate R@K (K = 1, 2, 3) on four distinct tasks...

## Citation

```bibtex
@misc{wu2025square,
  title={SQUARE: Semantic Query-Augmented Fusion and Efficient Batch Reranking for Training-free Zero-Shot Composed Image Retrieval},
  author={Wu et al. (2025)},
  year={2025},
  note={arXiv:2509.26330}
}
```

- arXiv: 2509.26330