Ragsearch Eval

---
name: ragsearch-eval
description: Evaluates dense RAG and GraphRAG retrieval backends when integrated into agentic search systems. It probes the agent's ability to dynamically retrieve, reason, and answer general and multi-hop QA queries under both training-free prompting and reinforcement learning paradigms. Use when the user wants to benchmark on NQ, PopQA, TriviaQA, HotpotQA, 2Wiki, Musique, or asks about evaluating this task. Reports Exact Match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.09666
  bibtex_key: fan2026ragsearch
  confidence: high
---

# ragsearch-eval

> Do We Still Need GraphRAG? Benchmarking RAG and GraphRAG for Agentic Search Systems — Fan et al. (2026) (arXiv:2604.09666, 2026)

## What this evaluates

Evaluates dense RAG and GraphRAG retrieval backends when integrated into agentic search systems. It probes the agent's ability to dynamically retrieve, reason, and answer general and multi-hop QA queries under both training-free prompting and reinforcement learning paradigms.

## Datasets

- **NQ** — total ?; splits: test (-1)
- **PopQA** — total ?; splits: test (-1)
- **TriviaQA** — total ?; splits: test (-1)
- **HotpotQA** — total ?; splits: test (-1)
- **2Wiki** — total ?; splits: test (-1)
- **Musique** — total ?; splits: test (-1)

## Metrics

- `Exact Match (EM)` **(primary)** — range: [0, 1]
  - 1 if the predicted answer exactly matches the gold answer string, 0 otherwise. Reported as the overall percentage across the dataset.

## Input / output format

**Input**: A natural language query q and a retrieval backend B (dense RAG or GraphRAG). The agent interacts iteratively, receiving retrieved text chunks or subgraphs wrapped in <information> tags.

**Output**: A final answer enclosed within <answer> and </answer> tags, following an interleaved sequence of <think> and <search> steps.

## Scoring recipe

```python
def compute_em(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred.strip() == gold.strip():
            correct += 1
    return correct / len(golds)
```

## Common pitfalls

- Confusing the agentic control policy with the retrieval backend; the benchmark keeps the agent protocol identical across Dense RAG and GraphRAG to isolate retrieval infrastructure effects.
- Failing to account for the multi-step agentic loop; single-shot baselines are evaluated separately from training-free and RL-based multi-round agents.
- GraphRAG backends require offline graph construction, but the benchmark metric only measures runtime answer correctness, not preprocessing time or cost.

## Evidence (verbatim from paper)

> In RAGSearch, rewards are defined at the trajectory level and focus on task correctness and output validity. Specifically, we combine (i) an outcome-based reward that measures answer correctness (e.g., exact match or task-specific accuracy), and (ii) a format-based reward that encourages the agent to follow the expected interaction and answer format.

## Citation

```bibtex
@misc{fan2026ragsearch,
  title={Do We Still Need GraphRAG? Benchmarking RAG and GraphRAG for Agentic Search Systems},
  author={Fan et al. (2026)},
  year={2026},
  note={arXiv:2604.09666}
}
```

- arXiv: 2604.09666