Medbrowsecomp Eval

---
name: medbrowsecomp-eval
description: Evaluates AI agents' ability to perform multi-hop, evidence-grounded medical information retrieval from live, heterogeneous web sources. It probes long-horizon web navigation, tool allocation, source verification, and the capacity to reconcile conflicting or dense biomedical data. Use when the user wants to benchmark on MedBrowseComp-50, MedBrowseComp-605, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.14963
  bibtex_key: chen2025medbrowsecomp
  confidence: high
---

# medbrowsecomp-eval

> MedBrowseComp: Benchmarking Medical Deep Research and Computer Use — Chen et al. (2025) (arXiv:2505.14963, 2025)

## What this evaluates

Evaluates AI agents' ability to perform multi-hop, evidence-grounded medical information retrieval from live, heterogeneous web sources. It probes long-horizon web navigation, tool allocation, source verification, and the capacity to reconcile conflicting or dense biomedical data.

## Datasets

- **MedBrowseComp-50** — total 50; splits: test (50); repo https://github.com/shan23chen/MedBrowseComp
- **MedBrowseComp-605** — total 605; splits: test (605); repo https://github.com/shan23chen/MedBrowseComp

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly answered questions out of the total. A half-point is awarded for specific 2-hop questions where the model retrieves a sub-entity's company name instead of the final answer.

## Input / output format

**Input**: A multi-hop medical information-seeking question requiring navigation of live web sources, with access to agentic tools (e.g., web search, browser, RAG/memory).

**Output**: A direct answer to the question, often requiring specific formatting (e.g., MM-DD-YYYY, YYYY, stock ticker) and verifiable source citations.

## Scoring recipe

```python
def compute_accuracy(predictions, gold, total_questions):
    score = 0
    for pred, gold_ans in zip(predictions, gold):
        if exact_match(pred, gold_ans):
            score += 1.0
        elif is_partial_2hop_match(pred, gold_ans):
            score += 0.5
    return score / total_questions
```

## Common pitfalls

- Agents exhaust tool-call quotas on preliminary tasks, leaving final critical queries unanswered.
- Models cite secondary press releases instead of authoritative sources (e.g., FDA Orange Book), leading to incorrect dates or claims.
- Agents fail to robustly parse dense tables in multi-page PDFs, extracting only partial milestones.
- Greedy attribute extraction causes models to copy the first matching date/field encountered rather than the correct structured metadata.

## Evidence (verbatim from paper)

> Figure [3] summarizes accuracy on MedBrowseComp-50. Across all systems, performance decays monotonically with hop count, corroborating prior evidence that long-horizon web navigation remains an open challenge for frontier LLM agents. Nevertheless, deep research variants–agents that allow iterative browsing steps rather than a single query—had improved performance. For example, O3 deepresearch answers 25.5/50 questions correctly, a 34% relative gain over O3 search (19/50); Gemini-2.5-pro deepsearch shows 75% improvement over its single-shot analogue (24.5 vs. 14). These gains are most pronounced on the hardest 4- and 5-hop splits, where deep research agents more than double the baseline accuracy. ... Half point given to a specific 2-hop question where the model retrieved the sub-entities’ company name instead.

## Citation

```bibtex
@misc{chen2025medbrowsecomp,
  title={MedBrowseComp: Benchmarking Medical Deep Research and Computer Use},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2505.14963}
}
```

- arXiv: 2505.14963