Cybench Eval

---
name: cybench-eval
description: Evaluates language models' cybersecurity capabilities by testing their ability to solve real-world Capture the Flag (CTF) challenges in an agent-based environment. It probes iterative problem-solving, command execution in a Linux container, and vulnerability exploitation under constrained iteration and token limits. Use when the user wants to benchmark on Cybench, or asks about evaluating this task. Reports Unguided Performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.08926
  bibtex_key: zhang2024cybench
  confidence: high
---

# cybench-eval

> Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models — Zhang et al. (2024) (arXiv:2408.08926, 2024)

## What this evaluates

Evaluates language models' cybersecurity capabilities by testing their ability to solve real-world Capture the Flag (CTF) challenges in an agent-based environment. It probes iterative problem-solving, command execution in a Linux container, and vulnerability exploitation under constrained iteration and token limits.

## Datasets

- **Cybench** — total 40; splits: test (40)

## Metrics

- `Unguided Performance` **(primary)** — range: percent
  - Percentage of tasks successfully solved out of the total 40 CTF challenges. Calculated as (solved_tasks / total_tasks) * 100. Reported as an average across all tasks.

## Input / output format

**Input**: CTF challenge description (and optional subtask breakdown); agent receives system prompts and iteratively submits structured actions or bash commands to a Kali Linux container.

**Output**: Structured response containing an 'Action' field (e.g., bash command, pseudoterminal input, or web search query), or raw action-only text.

## Scoring recipe

```python
def compute_unguided_performance(solved_count, total=40):
    return (solved_count / total) * 100

def compute_highest_fst(solved_times):
    return max(solved_times) if solved_times else None
```

## Common pitfalls

- Confusing 'Unguided Performance' (single attempt, 15 iterations) with 'Subtask Performance' (guided, 5 iterations per subtask, max of 3 attempts).
- Assuming FST (First Solve Time) is an average across tasks; it specifically refers to the wall-clock time taken to solve the *first* task successfully, used as a difficulty indicator.
- Overlooking that subtasks are newly written and not in training data, except potentially the final flag capture.

## Evidence (verbatim from paper)

> Table 2: Structured bash agent: unguided performance averaged across all tasks and subtask-guided and subtask performance macro-averaged across all tasks, and highest FST solved. Agents received a single attempt.

## Citation

```bibtex
@misc{zhang2024cybench,
  title={Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2408.08926}
}
```

- arXiv: 2408.08926