Deepwidesearch Eval

---
name: deepwidesearch-eval
description: Evaluates agentic systems' ability to perform wide-scale information collection and deep multi-hop reasoning simultaneously to fill structured result tables. It probes combinatorial search complexity, tool orchestration, reflection, and context management in real-world information-seeking tasks. Use when the user wants to benchmark on DeepWideBenchmark, or asks about evaluating this task. Reports Success Rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.20168
  bibtex_key: lan2025deepwidesearch
  confidence: high
---

# deepwidesearch-eval

> DeepWideSearch: Benchmarking Depth and Width in Agentic Information Seeking — Lan et al. (2025) (arXiv:2510.20168, 2025)

## What this evaluates

Evaluates agentic systems' ability to perform wide-scale information collection and deep multi-hop reasoning simultaneously to fill structured result tables. It probes combinatorial search complexity, tool orchestration, reflection, and context management in real-world information-seeking tasks.

## Datasets

- **DeepWideBenchmark** — total 220; splits: test (-1)

## Metrics

- `Success Rate` **(primary)** — range: percent
  - The percentage of queries where the agent's final output exactly matches the gold-standard table structure and content. Evaluated as Avg@4 (average over 4 attempts) and Pass@4 (best-of-4 success).
- `Row F1 Score` — range: percent
  - Harmonic mean of precision and recall computed over table rows, measuring how well the agent groups related entities together.
- `Item F1 Score` — range: percent
  - Harmonic mean of precision and recall computed over individual table items/cells, measuring fine-grained entity extraction accuracy.
- `Column F1` — range: percent
  - Harmonic mean of precision and recall computed over table columns, measuring the agent's ability to consistently populate specific attribute categories across all rows.
- `Core Entity Accuracy (CE Acc.)` — range: percent
  - Exact match rate of the most critical entities required by the query, regardless of table formatting or peripheral details.

## Input / output format

**Input**: Natural language questions/tasks requiring multi-hop reasoning and broad information collection. Agents receive the task and have access to Google Search API and Webpage Visit tools. HTML content is pre-summarized by the agent's backbone LLM before processing.

**Output**: Markdown-formatted tables containing the requested information, structured with rows, items, and columns corresponding to the query requirements.

## Scoring recipe

```python
def score(predictions, gold):
    # Success Rate: exact match over 4 attempts
    success = 1.0 if predictions == gold else 0.0
    # Parse markdown tables into structured lists
    pred_rows = parse_table(predictions, axis='row')
    gold_rows = parse_table(gold, axis='row')
    row_f1 = f1_score(pred_rows, gold_rows)
    item_f1 = f1_score(parse_table(predictions, axis='item'), parse_table(gold, axis='item'))
    col_f1 = f1_score(parse_table(predictions, axis='col'), parse_table(gold, axis='col'))
    # Core Entity Accuracy: exact set match
    ce_acc = 1.0 if set(predictions.core_entities) == set(gold.core_entities) else 0.0
    return {'success_rate': success, 'row_f1': row_f1, 'item_f1': item_f1, 'col_f1': col_f1, 'ce_acc': ce_acc}
```

## Common pitfalls

- Agents frequently fail due to brittle output formatting, producing invalid markdown tables instead of correctly structured results.
- Overreliance on internal knowledge leads to poor retrieval coverage, while insufficient reasoning before tool calls restricts query precision.
- Context overflow and inference errors during multi-step tool orchestration cause incomplete task solving, especially for models like Gemini 2.5 Pro.

## Evidence (verbatim from paper)

> The complete results are presented in Table [2]. It can be found that most baselines demonstrate near-zero success rates, with only WebSailor (Gemini 2.5 Pro) and WebSailor (Claude Sonnet 4) exceeding 1-2% in Success Rate (Avg@4), confirming the inherent complexity of simultaneously handling deep reasoning and wide-scale information collection.

## Citation

```bibtex
@misc{lan2025deepwidesearch,
  title={DeepWideSearch: Benchmarking Depth and Width in Agentic Information Seeking},
  author={Lan et al. (2025)},
  year={2025},
  note={arXiv:2510.20168}
}
```

- arXiv: 2510.20168