Needlebench Eval

---
name: needlebench-eval
description: Evaluates large language models' ability to retrieve specific information and perform complex multi-point reasoning within long-context documents. It probes both information-sparse retrieval and information-dense reasoning (Ancestral Trace Challenge) across 32K and 128K token contexts. Use when the user wants to benchmark on NeedleBench, or asks about evaluating this task. Reports Overall.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.11963
  bibtex_key: li2024needlebench
  confidence: high
---

# needlebench-eval

> NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities — Li et al. (2024) (arXiv:2407.11963, 2024)

## What this evaluates

Evaluates large language models' ability to retrieve specific information and perform complex multi-point reasoning within long-context documents. It probes both information-sparse retrieval and information-dense reasoning (Ancestral Trace Challenge) across 32K and 128K token contexts.

## Datasets

- **NeedleBench** — total ?; splits: test (-1); repo https://github.com/open-compass/opencompass

## Metrics

- `Overall` **(primary)** — range: percent
  - Mean accuracy across all sub-tasks (Single-Retrieval, Multi-Retrieval, Multi-Reasoning) for a given context length and language.

## Input / output format

**Input**: Long-context documents (32K or 128K tokens) containing synthetic 'needles' (key facts) interspersed with irrelevant filler text, followed by a natural language query requiring either direct retrieval or multi-step reasoning.

**Output**: Free-text answer generated by the model in response to the query.

## Scoring recipe

```python
def compute_score(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100
# For ATC, scores are averaged across needle counts (2, 4, ..., 512) to produce a Weighted Score.
```

## Common pitfalls

- Models often exhibit an 'under-thinking' phenomenon, prematurely terminating reasoning despite sufficient context.
- Performance varies significantly between English and Chinese, with English typically scoring higher due to pretraining data distribution.
- Scaling parameter count does not guarantee reasoning improvements; architecture and training strategies heavily influence multi-needle performance.

## Evidence (verbatim from paper)

> Overall denotes the mean score across all tasks. Bold denotes the best score among all models, and underline denotes the best score under the same model scale.

## Citation

```bibtex
@misc{li2024needlebench,
  title={NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2407.11963}
}
```

- arXiv: 2407.11963