Infoseek Eval

---
name: infoseek-eval
description: Evaluates LLMs on complex, multi-step reasoning and agentic search tasks, including single-hop and multi-hop question answering as well as deep research benchmarks requiring web search and synthesis. Use when the user wants to benchmark on NQ, TQA, PopQA, HQA, 2Wiki, MSQ, Bamb, BrowseComp-Plus, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.00375
  bibtex_key: xia2025opendata
  confidence: medium
---

# infoseek-eval

> Open Data Synthesis For Deep Research — Xia et al. (arXiv:2509.00375, 2025)

## What this evaluates

Evaluates LLMs on complex, multi-step reasoning and agentic search tasks, including single-hop and multi-hop question answering as well as deep research benchmarks requiring web search and synthesis.

## Datasets

- **NQ** — total ?; splits: test (-1)
- **TQA** — total ?; splits: test (-1)
- **PopQA** — total ?; splits: test (-1)
- **HQA** — total ?; splits: test (-1)
- **2Wiki** — total ?; splits: test (-1)
- **MSQ** — total ?; splits: test (-1)
- **Bamb** — total ?; splits: test (-1)
- **BrowseComp-Plus** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of instances where the model's final extracted answer matches the ground truth.
- `Search Calls` — range: other
  - Average number of search engine invocations per task instance.

## Input / output format

**Input**: Question or research task prompt requiring multi-step reasoning and web search.

**Output**: Model generates responses enclosed in specific tags: <think> for reasoning, <search> for queries, <information> for retrieved summaries, and <answer> for the final response.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    correct = 0
    search_calls = 0
    for pred, gold_ans in zip(predictions, gold):
        extracted = extract_tag(pred, 'answer')
        if exact_match(extracted, gold_ans):
            correct += 1
        search_calls += count_tag(pred, 'search')
    return {'Accuracy': correct / len(predictions) * 100, 'Search Calls': search_calls / len(predictions)}
```

## Common pitfalls

- Models must strictly follow the XML-like tag structure (<think>, <search>, <answer>); failure to do so breaks answer extraction.
- High Accuracy does not imply efficient search usage; Search Calls are reported separately and can vary significantly across models.
- Benchmarks like BrowseComp-Plus involve noisy web results, where models often lose focus or hallucinate without proper intermediate summarization.

## Evidence (verbatim from paper)

> Table 4: Model performance on the BrowseComp-Plus benchmark for complex reasoning tasks.

| Model | Retriever | Accuracy (%) | Search Calls |
| --- | --- | --- | --- |
| InfoSeeker-3B | BM25 | $16.5$ | $8.24$ |

R = 1 if format and extracted answer are both correct, 0 otherwise

## Citation

```bibtex
@misc{xia2025opendata,
  title={Open Data Synthesis For Deep Research},
  author={Xia et al.},
  year={2025},
  note={arXiv:2509.00375}
}
```

- arXiv: 2509.00375