Mm Bright Eval

---
name: mm-bright-eval
description: This benchmark evaluates reasoning-intensive retrieval capabilities across text-only and multimodal settings. It probes models' ability to align visual and textual information, navigate technical domain queries, and rank relevant documents or images based on complex, multi-modal prompts. Use when the user wants to benchmark on MM-BRIGHT, or asks about evaluating this task. Reports nDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.09562
  bibtex_key: [REDACTED]
  confidence: high
---

# mm-bright-eval

> MM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval — Abdallah et al. (2026) (arXiv:2601.09562, 2026)

## What this evaluates

This benchmark evaluates reasoning-intensive retrieval capabilities across text-only and multimodal settings. It probes models' ability to align visual and textual information, navigate technical domain queries, and rank relevant documents or images based on complex, multi-modal prompts.

## Datasets

- **MM-BRIGHT** — total 2803; splits: test (-1)

## Metrics

- `nDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. Computes the weighted sum of relevance scores for the top 10 retrieved items, normalized by the ideal DCG@10. Tasks 1-3 use binary relevance (0/1), while Task 4 uses graded relevance (0, 1, 2).

## Input / output format

**Input**: A text query, optionally accompanied by one or more query images (for Tasks 2-4), and a candidate corpus of documents and/or images to retrieve from.

**Output**: A ranked list of retrieved corpus items (documents or images), evaluated at the top 10 positions.

## Scoring recipe

```python
def compute_ndcg_at_10(relevance_scores, top_k=10):
    dcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(relevance_scores[:top_k]))
    ideal_scores = sorted(relevance_scores, reverse=True)
    idcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(ideal_scores[:top_k]))
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- Task 4 uses a graded relevance scale (rel=2,1,0) while Tasks 1-3 use binary labels, so scores across tasks are not directly comparable.
- Adding query images often degrades performance for current multimodal models compared to text-only baselines due to insufficient visual reasoning training.
- Models supporting only single-image inputs require vertical concatenation of query images, which can impact retrieval quality.

## Evidence (verbatim from paper)

> Following prior work (Thakur et al., 2021; Nguyen et al., 2016; Su et al., 2024), we use nDCG@10 as the primary metric. Tasks 1-3 use binary relevance labels following BEIR (Thakur et al., 2021). Task 4 uses graded relevance: rel=2 for gold passage with corresponding positive image, rel=1 for gold passage without image, and rel=0 for incorrect passages.

## Citation

```bibtex
@misc{abdallah2026mmbright,
  title={MM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval},
  author={Abdallah et al. (2026)},
  year={2026},
  note={arXiv:2601.09562}
}
```

- arXiv: 2601.09562