Deep Research Accuracy Eval

---
name: deep-research-accuracy-eval
description: Evaluates an agent's ability to perform long-horizon, multi-step web research to answer complex factual questions. It probes the model's capacity for iterative search, evidence aggregation, and adaptive reasoning under both reproducible offline constraints and live web environments. Use when the user wants to benchmark on BrowseComp-Plus, BrowseComp, GAIA, xbench-DeepSearch, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.20278
  bibtex_key: li2026openresearcher
  confidence: high
---

# deep-research-accuracy-eval

> OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesis — Li et al. (2026) (arXiv:2603.20278, 2026)

## What this evaluates

Evaluates an agent's ability to perform long-horizon, multi-step web research to answer complex factual questions. It probes the model's capacity for iterative search, evidence aggregation, and adaptive reasoning under both reproducible offline constraints and live web environments.

## Datasets

- **BrowseComp-Plus** — total ?; splits: test (-1)
- **BrowseComp** — total ?; splits: test (-1)
- **GAIA** — total ?; splits: test (-1)
- **xbench-DeepSearch** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Calculated as the percentage of questions for which the agent's final generated answer matches the ground truth. Also reported as Pass@k, measuring the fraction of questions solved by at least one of k sampled trajectories.

## Input / output format

**Input**: A natural language research question. For BrowseComp-Plus, the agent accesses a fixed 15M-document offline corpus via a FAISS index. For other benchmarks, the agent uses the Serper API for live web search.

**Output**: A final textual answer to the research question.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if is_match(p, g))
    return (correct / len(gold)) * 100

def compute_pass_at_k(predictions_per_q, k):
    solved = sum(1 for preds in predictions_per_q if any(is_match(p, gold) for p in preds[:k]))
    return (solved / len(predictions_per_q)) * 100
```

## Common pitfalls

- Confusing the search environment: BrowseComp-Plus uses a fixed offline corpus with a FAISS index, while BrowseComp, GAIA, and xbench-DeepSearch rely on live Serper API calls.
- Assuming longer trajectories always yield better answers: Performance plateaus after ~100 turns, and failed trajectories often use more tools inefficiently rather than correctly.
- Overlooking Pass@k vs Pass@1: Many questions are solvable but only along specific reasoning paths, so evaluating only the first trajectory underestimates capability.

## Evidence (verbatim from paper)

> Our OpenResearcher-30B-A3B achieves 54.8% accuracy on this benchmark, substantially outperforming strong proprietary baselines including GPT-4.1 (36.4%), Claude-4-Opus (36.8%), and DeepSeek-R1 (16.4%).

## Citation

```bibtex
@misc{li2026openresearcher,
  title={OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesis},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2603.20278}
}
```

- arXiv: 2603.20278