Openseeker Eval

---
name: openseeker-eval
description: Evaluates the capability of web search agents to perform multi-step navigation, complex deep research planning, and precise information retrieval across English and Chinese web environments. It probes the model's ability to synthesize information from noisy, long-horizon browsing trajectories and extract exact answers or reliable summaries. Use when the user wants to benchmark on BrowseComp, BrowseComp-ZH, xbench-DeepSearch, WideSearch, or asks about evaluating this task. Reports accuracy / F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.15594
  bibtex_key: du2026openseeker
  confidence: high
---

# openseeker-eval

> OpenSeeker: Democratizing Frontier Search Agents by Fully Open-Sourcing Training Data — Du et al. (2026) (arXiv:2603.15594, 2026)

## What this evaluates

Evaluates the capability of web search agents to perform multi-step navigation, complex deep research planning, and precise information retrieval across English and Chinese web environments. It probes the model's ability to synthesize information from noisy, long-horizon browsing trajectories and extract exact answers or reliable summaries.

## Datasets

- **BrowseComp** — total 200; splits: test (200)
- **BrowseComp-ZH** — total ?; splits: test (-1)
- **xbench-DeepSearch** — total ?; splits: test (-1)
- **WideSearch** — total ?; splits: test (-1)

## Metrics

- `accuracy / F1 score` **(primary)** — range: percent
  - Exact-match accuracy for BrowseComp, BrowseComp-ZH, and xbench-DeepSearch; token-level F1 for WideSearch. Computed as the proportion of correctly answered instances or token overlap, multiplied by 100 to yield a percentage.

## Input / output format

**Input**: User question $q$ provided in a web browsing environment with access to search and navigation tools.

**Output**: Final answer string extracted from the browsing trajectory, or the sequence of tool calls and reasoning steps.

## Scoring recipe

```python
def compute_metric(predictions, gold):
    correct = 0
    for pred, gold_ans in zip(predictions, gold):
        if normalize_text(pred) == normalize_text(gold_ans):
            correct += 1
    return (correct / len(gold)) * 100
```

## Common pitfalls

- BrowseComp evaluation is restricted to a 200-sample subset due to resource constraints, not the full benchmark.
- Baseline scores are sourced from external technical reports or public leaderboards rather than re-run under identical tool-call limits or context windows.
- Single training run without hyperparameter tuning or data filtering limits reproducibility and generalizability claims.

## Evidence (verbatim from paper)

> Table 1: Comparisons among our OpenSeeker and other search agents. ‘# Samples’ denotes the number of total training data samples; ‘# OS Samples’ denotes the number of open-source data samples; ‘Training’ denotes training techniques (CPT: continual pre-training, SFT: supervised fine-tuning, RL: reinforcement learning); ‘Academic’ denotes whether conducted by pure academic team ($\checkmark$: Yes, $\times$: No); ‘BC-ZH’ denotes BrowseComp-ZH; ‘WideSearch’ denotes item F1 result on the English subset.

## Citation

```bibtex
@misc{du2026openseeker,
  title={OpenSeeker: Democratizing Frontier Search Agents by Fully Open-Sourcing Training Data},
  author={Du et al. (2026)},
  year={2026},
  note={arXiv:2603.15594}
}
```

- arXiv: 2603.15594