Terminal Bench 2.0 Eval

---
name: terminal-bench-2.0-eval
description: Evaluates an LLM's ability to execute complex, multi-step terminal commands and tasks in a sandboxed environment. It probes capabilities across software engineering, system administration, data processing, security, and debugging. Use when the user wants to benchmark on Terminal-Bench 2.0, or asks about evaluating this task. Reports TB2.0.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.21193
  bibtex_key: pi2026dataengineering
  confidence: high
---

# terminal-bench-2.0-eval

> On Data Engineering for Scaling LLM Terminal Capabilities — Pi et al. (2026) (arXiv:2602.21193, 2026)

## What this evaluates

Evaluates an LLM's ability to execute complex, multi-step terminal commands and tasks in a sandboxed environment. It probes capabilities across software engineering, system administration, data processing, security, and debugging.

## Datasets

- **Terminal-Bench 2.0** — total ?; splits: test (-1)

## Metrics

- `TB2.0` **(primary)** — range: percent
  - Mean success rate across tasks, reported as average ± standard deviation over multiple runs. Tasks are evaluated in isolated cloud sandboxes using the Daytona framework, which checks for successful completion of terminal commands and task objectives.

## Input / output format

**Input**: Natural language task descriptions requiring multi-step terminal commands and file manipulations.

**Output**: Sequences of terminal commands and actions executed within a containerized environment.

## Scoring recipe

```python
results = []
for task in test_set:
    sandbox = Daytona.create_sandbox()
    trajectory = model.generate(task.prompt)
    success = sandbox.execute_and_verify(trajectory, task.checkpoints)
    results.append(success)
mean_score = sum(results) / len(results)
std_score = std(results)
return mean_score, std_score
```

## Common pitfalls

- Strict filtering of training data (e.g., success-only) discards valuable error-recovery patterns and significantly hurts performance.
- Extending context length beyond 32k tokens or using YaRN2 scaling does not improve TB2.0 scores and may slightly hurt performance due to noisy long-tail trajectories.

## Evidence (verbatim from paper)

> We evaluate Terminal-Task-Gen by benchmarking Nemotron-Terminal on Terminal-Bench 2.0 (TB2.0). As shown in Table [3], our models demonstrate substantial gains: Nemotron-Terminal-8B achieves $13.0\pm 2.2$, a five-fold increase over Qwen3-8B ($2.47\pm 0.5$).

## Citation

```bibtex
@misc{pi2026dataengineering,
  title={On Data Engineering for Scaling LLM Terminal Capabilities},
  author={Pi et al. (2026)},
  year={2026},
  note={arXiv:2602.21193}
}
```

- arXiv: 2602.21193