Ssp Eval

---
name: ssp-eval
description: Evaluates the capability of deep search agents to answer complex factual and multi-hop questions using retrieval-augmented generation and multi-turn reasoning. It probes the agent's ability to dynamically adjust search strategies, verify information via RAG, and synthesize accurate answers under constrained tool-use budgets. Use when the user wants to benchmark on NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, Musique, Bamboogle, or asks about evaluating this task. Reports pass@1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.18821
  bibtex_key: [REDACTED]
  confidence: high
---

# ssp-eval

> Search Self-play: Pushing the Frontier of Agent Capability without Supervision — Lu et al. (2025) (arXiv:2510.18821, 2025)

## What this evaluates

Evaluates the capability of deep search agents to answer complex factual and multi-hop questions using retrieval-augmented generation and multi-turn reasoning. It probes the agent's ability to dynamically adjust search strategies, verify information via RAG, and synthesize accurate answers under constrained tool-use budgets.

## Datasets

- **NQ** — total 500; splits: test (500)
- **TriviaQA** — total 500; splits: test (500)
- **PopQA** — total 500; splits: test (500)
- **HotpotQA** — total 500; splits: test (500)
- **2WikiMultiHopQA** — total 500; splits: test (500)
- **Musique** — total 500; splits: test (500)
- **Bamboogle** — total 125; splits: test (125)

## Metrics

- `pass@1 accuracy` **(primary)** — range: [0, 1]
  - LLM-as-a-judge evaluation where Qwen2.5-32B-Instruct assesses whether the agent's final answer is correct relative to the ground truth. pass@1 denotes the fraction of questions answered correctly on the first attempt.

## Input / output format

**Input**: Natural language question/query.

**Output**: Final answer string generated by the agent after up to 10 rounds of tool-integrated search.

## Scoring recipe

```python
def score(predictions, gold_answers):
    judge = load_llm('Qwen2.5-32B-Instruct')
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        prompt = f'Question: {pred["question"]}\nPredicted Answer: {pred["answer"]}\nGround Truth: {gold}\nIs the predicted answer correct? Answer YES or NO.'
        response = judge.generate(prompt)
        if 'YES' in response.upper():
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Evaluators must sample exactly 500 QA pairs per benchmark (except Bamboogle which uses all 125) to match the paper's protocol.
- The LLM-as-a-judge metric relies on Qwen2.5-32B-Instruct; using a different judge model or prompt may yield different pass@1 scores.
- Search tool calls are strictly limited to 10 rounds per trajectory; exceeding this limit violates the evaluation constraints.

## Evidence (verbatim from paper)

> Following recent work (Gao et al., 2025a), we adopt LLM-as-a-judge as standard metric for evaluation. Qwen2.5-32B-Instruct (Yang et al., 2024a) is deployed as the judge model. All results are reported in terms of pass@1 accuracy. Following the practice in prior works, we randomly sample 500 question-answer (QA) pairs on each benchmark to reduce the evaluation overhead while maintaining statistical reliability. For Bamboogle (Press et al., 2022), all 125 test samples are used for evaluation.

## Citation

```bibtex
@misc{lu2025searchselfplay,
  title={Search Self-play: Pushing the Frontier of Agent Capability without Supervision},
  author={Lu et al. (2025)},
  year={2025},
  note={arXiv:2510.18821}
}
```

- arXiv: 2510.18821