Livemcpbench Eval

---
name: livemcpbench-eval
description: Evaluates LLM agents' capability to dynamically discover, select, and chain Model Context Protocol (MCP) tools to complete complex, multi-step real-world tasks. It probes meta-tool learning and multi-tool collaboration in large-scale, time-varying tool ecosystems. Use when the user wants to benchmark on LiveMCPBench, or asks about evaluating this task. Reports task success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.01780
  bibtex_key: mo2025livemcpbench
  confidence: medium
---

# livemcpbench-eval

> LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools? — Mo Guozhao et al. (2025) (arXiv:2508.01780, 2025)

## What this evaluates

Evaluates LLM agents' capability to dynamically discover, select, and chain Model Context Protocol (MCP) tools to complete complex, multi-step real-world tasks. It probes meta-tool learning and multi-tool collaboration in large-scale, time-varying tool ecosystems.

## Datasets

- **LiveMCPBench** — total 95; splits: test (95)

## Metrics

- `task success rate` **(primary)** — range: percent
  - The percentage of tasks out of the total 95 that the agent successfully completes within the MCP environment. Assessed via LiveMCPEval, an LLM-as-a-Judge framework (using Deepseek-V3) that evaluates multi-step tool invocation trajectories against task requirements. Calculated as (number of tasks judged successful / 95) * 100.

## Input / output format

**Input**: Natural language task prompts requiring interaction with a dynamic pool of 527 tools across 70 MCP servers.

**Output**: Sequential tool invocation trajectories (ReACT-style) and a final binary success/failure status for each task.

## Scoring recipe

```python
def compute_task_success_rate(predictions, gold):
    successful = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (successful / len(gold)) * 100
```

## Common pitfalls

- Using advanced reasoning models (e.g., Deepseek-R1, Claude-Opus-4) as the LLM-as-a-Judge evaluator yields lower human agreement rates (60-70%) due to difficulties processing long execution trajectories.
- The benchmark evaluates dynamic, real-world MCP servers which may change over time, making exact reproducibility dependent on server availability and the specific evaluation timestamp.

## Evidence (verbatim from paper)

> We show the task success rates for different models in Table[2]. We can see that: 1. 1. Meta-Tool-Learning Capabilities in Claude Models. The Claude series demonstrates remarkable meta-tool-learning proficiency, with Claude-Sonnet-4 and Claude-Opus-4 achieving success rates of 78.95% and 70.53% respectively.

## Citation

```bibtex
@misc{mo2025livemcpbench,
  title={LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?},
  author={Mo Guozhao et al. (2025)},
  year={2025},
  note={arXiv:2508.01780}
}
```

- arXiv: 2508.01780