Bright Eval

---
name: bright-eval
description: Evaluates a model's ability to perform reasoning-intensive text retrieval by matching complex, domain-diverse queries to relevant documents. It probes deep logical and conceptual alignment between queries and documents, going beyond simple keyword or semantic matching. Use when the user wants to benchmark on BRIGHT, or asks about evaluating this task. Reports nDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.12883
  bibtex_key: su2024bright
  confidence: high
---

# bright-eval

> BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval — Hongjin Su et al. (2024) (arXiv:2407.12883, 2024)

## What this evaluates

Evaluates a model's ability to perform reasoning-intensive text retrieval by matching complex, domain-diverse queries to relevant documents. It probes deep logical and conceptual alignment between queries and documents, going beyond simple keyword or semantic matching.

## Datasets

- **BRIGHT** — total ?; splits: test (-1); repo https://github.com/xlang-ai/BRIGHT

## Metrics

- `nDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. It computes the discounted gain of relevant documents in a ranked list, normalizing by the ideal DCG to yield a score between 0 and 1.

## Input / output format

**Input**: A query (original user query or LLM-generated reasoning trace) and a document corpus to search over.

**Output**: A ranked list of retrieved documents (top-10).

## Scoring recipe

```python
def ndcg_at_10(relevance_scores, k=10):
    dcg = sum(rel / log2(i + 2) for i, rel in enumerate(relevance_scores[:k]))
    ideal = sorted(relevance_scores, reverse=True)[:k]
    idcg = sum(rel / log2(i + 2) for i, rel in enumerate(ideal))
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- LLM-based QA evaluation may not accurately reflect retrieval quality because the generator might fail to integrate retrieved documents, or the evaluator might struggle to compare open-ended answers.
- Trained dense retrievers often underperform on out-of-distribution reasoning queries, while BM25 adapts better to novel query formulations.

## Evidence (verbatim from paper)

> Following prior work (Thakur et al., 2021; Bajaj et al., 2018; Voorhees & Tice, 2000), we use nDCG@10 as the main metric. Results in Table 2 show that BRIGHT is very challenging, with the best model achieving only $24.3\mathrm{nDCG}@10$.

## Citation

```bibtex
@misc{su2024bright,
  title={BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval},
  author={Hongjin Su et al. (2024)},
  year={2024},
  note={arXiv:2407.12883}
}
```

- arXiv: 2407.12883