mcpbr is an open-source benchmark runner for evaluating AI agents and MCP servers across 25+ benchmarks including SWE-bench, HumanEval, GSM8K, CyberGym, and more.
Installs into .claude/skills of the current project.
Are you the author of 468 Index 73decf29?
Add the live security badge to your README. It updates with every re-scan.
[](https://www.skillsdirectory.com/skills/tools-only-468-index-73decf29)
---
description: "mcpbr is an open-source benchmark runner for evaluating AI agents and MCP servers across 25+ benchmarks including SWE-bench, HumanEval, GSM8K, CyberGym, and more."
faq:
- q: "What is mcpbr?"
a: "mcpbr (Model Context Protocol Benchmark Runner) is an open-source tool for evaluating AI agents and MCP servers across 25+ benchmarks including SWE-bench, HumanEval, GSM8K, CyberGym, MCPToolBench++, and more. It provides quantitative comparison between tool-assisted and baseline agent performance."
- q: "What models does mcpbr support?"
a: "mcpbr supports Claude models from Anthropic including Claude Opus 4.5, Claude Sonnet 4.5, and Claude Haiku 4.5. Run 'mcpbr models' to see the full list."
- q: "How do I get started with mcpbr?"
a: "Install mcpbr with 'pip install mcpbr', set your ANTHROPIC_API_KEY, run 'mcpbr init' to create a config file, then run 'mcpbr run -c mcpbr.yaml' to start an evaluation."
---
# mcpbr
```bash
pip install mcpbr && mcpbr init && mcpbr run -c mcpbr.yaml -n 1 -v
```
Benchmark your MCP server against real GitHub issues. One command, hard numbers.
---
<p align="center">
<img src="assets/mcpbr-logo.jpg" alt="MCPBR Logo" width="400">
</p>
**Model Context Protocol Benchmark Runner**
[](https://pypi.org/project/mcpbr/)
[](https://www.npmjs.com/package/mcpbr-cli)
[](https://www.python.org/downloads/)
[](https://github.com/greynewell/mcpbr/actions/workflows/ci.yml)
[](https://opensource.org/licenses/MIT)
> Stop guessing if your MCP server actually helps. Get hard numbers comparing tool-assisted vs. baseline agent performance on real GitHub issues.
<p align="center">
<img src="assets/mcpbr-demo.gif" alt="mcpbr in action" width="700">
</p>
## What You Get
Real metrics showing whether your MCP server improves agent performance on SWE-bench tasks. No vibes, just data.
```text
Evaluation Results
Summary
+-----------------+-----------+----------+
| Metric | MCP Agent | Baseline |
+-----------------+-----------+----------+
| Resolved | 8/25 | 5/25 |
| Resolution Rate | 32.0% | 20.0% |
+-----------------+-----------+----------+
Improvement: +60.0%
```
## Why mcpbr?
MCP servers promise to make LLMs better at coding tasks. But how do you *prove* it?
mcpbr runs controlled experiments: same model, same tasks, same environment - the only variable is your MCP server. You get:
- **Apples-to-apples comparison** against a baseline agent
- **Real GitHub issues** from SWE-bench (not toy examples)
- **Reproducible results** via Docker containers with pinned dependencies
## Quick Start
### Option 1: Use Example Configurations (Fastest)
Get started in seconds with our ready-to-use examples:
```bash
# Set your API key
export ANTHROPIC_API_KEY="your-api-key"
# Run your first evaluation
mcpbr run -c examples/quick-start/getting-started.yaml -v
```
This runs 5 SWE-bench tasks with the filesystem server. Expected runtime: 15-30 minutes, cost: $2-5.
!!! tip "Explore 25+ Example Configurations"
Browse the [`examples/`](https://github.com/greynewell/mcpbr/tree/main/examples) directory for:
- **Quick Start**: Getting started, testing servers, comparing models
- **Benchmarks**: SWE-bench Lite/Full, CyberGym basic/advanced
- **MCP Servers**: Filesystem, GitHub, Brave Search, databases, custom servers
- **Scenarios**: Cost-optimized, performance-optimized, CI/CD, regression detection
See the [Examples README](https://github.com/greynewell/mcpbr/tree/main/examples/README.md) for the complete guide.
### Option 2: Generate Custom Configuration
#### 1. Set your API key
```bash
export ANTHROPIC_API_KEY="your-api-key"
```
#### 2. Generate a configuration file
```bash
mcpbr init
```
#### 3. Edit the configuration
Point it to your MCP server:
```yaml
mcp_server:
command: "npx"
args:
- "-y"
- "@modelcontextprotocol/server-filesystem"
- "{workdir}"
env: {}
provider: "anthropic"
agent_harness: "claude-code"
model: "sonnet"
dataset: "SWE-bench/SWE-bench_Lite"
sample_size: 10
timeout_seconds: 300
max_concurrent: 4
```
#### 4. Run the evaluation
```bash
mcpbr run --config mcpbr.yaml
```
## How It Works
mcpbr runs two parallel evaluations for each SWE-bench task:
1. **MCP Agent**: LLM with access to tools from your MCP server
2. **Baseline Agent**: Same LLM without MCP tools
By comparing resolution rates, you can measure the effectiveness of your MCP server for code exploration and bug fixing.
```
Host Machine
+-----------------------------------------------------------+
| mcpbr Harness (Python) |
| - Loads SWE-bench tasks from HuggingFace |
| - Pulls pre-built Docker images |
| - Orchestrates agent runs |
| - Collects results and generates reports |
+----------------------------+------------------------------+
| docker exec
+----------------------------v------------------------------+
| Docker Container (per task) |
| - Repository at correct commit |
| - All dependencies pre-installed |
| - Claude Code CLI runs inside container |
| - Generates patches and runs tests |
+-----------------------------------------------------------+
```
## Next Steps
- [Examples](https://github.com/greynewell/mcpbr/tree/main/examples) - Browse 25+ example configurations
- [FAQ](FAQ.md) - Frequently asked questions
- [Best Practices](best-practices.md) - Guide to effective mcpbr usage
- [Installation](installation.md) - Prerequisites and installation options
- [Configuration](configuration.md) - Full configuration reference
- [CLI Reference](cli.md) - All available commands and options
- [MCP Integration](mcp-integration.md) - How to test your MCP server
- [Benchmarks](benchmarks/index.md) - Available benchmarks and how to use them
- [Evaluation Results](evaluation-results.md) - Understanding output formats
- [Troubleshooting](troubleshooting.md) - Common issues and solutions
---
Built by [Grey Newell](https://greynewell.com)
