Sketch Less Retrieval Eval

---
name: sketch-less-retrieval-eval
description: Evaluates early retrieval performance using partial or low-quality sketches combined with text, measuring how quickly the correct target image appears in the ranked list as the sketch is drawn. It probes robustness to incomplete visual inputs and multimodal fusion. Use when the user wants to benchmark on FS2K-SDE1, FS2K-SDE2, User-SDE, or asks about evaluating this task. Reports m@A, m@B.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.08515
  bibtex_key: dai2024facecaption
  confidence: high
---

# sketch-less-retrieval-eval

> 15M Multimodal Facial Image-Text Dataset — Dawei Dai et al. (2024) (arXiv:2407.08515, 2024)

## What this evaluates

Evaluates early retrieval performance using partial or low-quality sketches combined with text, measuring how quickly the correct target image appears in the ranked list as the sketch is drawn. It probes robustness to incomplete visual inputs and multimodal fusion.

## Datasets

- **FS2K-SDE1** — total ?; splits: train (76609), test (-1)
- **FS2K-SDE2** — total ?; splits: train (23714), test (-1)
- **User-SDE** — total ?; splits: test (110)

## Metrics

- `m@A, m@B` **(primary)** — range: percent
  - m@A is the ranking percentile (higher indicates better early retrieval), and m@B is 1/rank versus percentage of sketch completion. Both capture retrieval efficiency as the sketch is progressively drawn.

## Input / output format

**Input**: A partial sketch of a face and its corresponding text description.

**Output**: A ranked list of target face images.

## Scoring recipe

```python
def compute_slfir(predictions, gold, sketch_pct):
    ranks = [pred_list.index(g) + 1 for pred_list, g in zip(predictions, gold)]
    m_a = np.mean([100 - (r / len(predictions) * 100) for r in ranks])
    m_b = np.mean([1.0 / r for r in ranks]) * 100
    return m_a, m_b
```

## Common pitfalls

- Metrics are task-specific to early retrieval; m@A measures ranking percentile while m@B measures retrieval efficiency relative to sketch completion percentage, requiring careful interpretation.
- User-SDE tests generalization across different painters, which typically shows performance drops compared to controlled datasets due to sketching diversity.

## Evidence (verbatim from paper)

> m@A (the ranking percentile) and m@B (1/rank versus percentage of sketch) were used to capture the retrieval performance for the partial sketches.

## Citation

```bibtex
@misc{dai2024facecaption,
  title={15M Multimodal Facial Image-Text Dataset},
  author={Dawei Dai et al. (2024)},
  year={2024},
  note={arXiv:2407.08515}
}
```

- arXiv: 2407.08515