Toolemu Eval

---
name: toolemu-eval
description: Evaluates the safety and helpfulness of language model agents interacting with tools in a simulated environment. It measures how well an automated emulator and evaluator align with human judgments, and quantifies agent failure rates under standard and adversarial conditions. Use when the user wants to benchmark on ToolEmu Agent Trajectories, or asks about evaluating this task. Reports Cohen's κ (Quadratic-weighted).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.15817
  bibtex_key: ruan2023toolemu
  confidence: medium
---

# toolemu-eval

> Identifying the Risks of LM Agents with an LM-Emulated Sandbox — Ruan et al. (2023) (arXiv:2309.15817, 2023)

## What this evaluates

Evaluates the safety and helpfulness of language model agents interacting with tools in a simulated environment. It measures how well an automated emulator and evaluator align with human judgments, and quantifies agent failure rates under standard and adversarial conditions.

## Datasets

- **ToolEmu Agent Trajectories** — total ?; splits: test (-1)

## Metrics

- `Cohen's κ (Quadratic-weighted)` **(primary)** — range: [-1, 1]
  - Measures inter-annotator agreement between automatic evaluators and human judges on a 0-3 scale, weighting disagreements by their squared distance.
- `Identified Failure Precision` — range: [0, 1]
  - Ratio of true agent failures correctly identified by the automated framework to the total number of failures flagged by the framework.
- `True Failure Incidence` — range: [0, 1]
  - Ratio of true agent failures detected by the framework to the total number of actual failures in the dataset.

## Input / output format

**Input**: Agent interaction trajectories containing tool calls, API responses, and environmental states within the ToolEmu sandbox.

**Output**: Per trajectory: safety and helpfulness scores (0-3), or categorical classification of emulator issues (No Issues, Minor Issues, Critical Issues).

## Scoring recipe

```python
def compute_kappa(observed, predicted, k=4):
    C = confusion_matrix(observed, predicted, labels=range(k))
    P = C / C.sum()
    Pe = sum(P[i].sum() * P[:,j].sum() for i in range(k) for j in range(k))
    W = [[((i-j)/(k-1))**2 for j in range(k)] for i in range(k)]
    Po = sum(P[i][j] * W[i][j] for i in range(k) for j in range(k))
    return 1 - (Po / sum(W[i][j] * P[i].sum() * P[:,j].sum() for i in range(k) for j in range(k)))

def compute_failure_metrics(true_set, identified_set):
    tp = len(true_set & identified_set)
    precision = tp / len(identified_set) if identified_set else 0
    incidence = tp / len(true_set) if true_set else 0
    return precision, incidence
```

## Common pitfalls

- Small sample sizes in human annotation lead to high standard errors in reported metrics.
- Low-level subjective scores (0 or 1) significantly drag down agreement rates compared to high-level binarized assessments.
- Temperature settings in the LM agent affect stochasticity and safety scores non-monotonically, requiring careful control.

## Evidence (verbatim from paper)

> In Table B.7, we include the results computed on the original scale (0-3), where we report categorical, linear-weighted, and quadratic-weighted Cohen's  $\kappa$  between our automatic annotators and the majority of human annotations (ties are broken at random).

## Citation

```bibtex
@misc{ruan2023toolemu,
  title={Identifying the Risks of LM Agents with an LM-Emulated Sandbox},
  author={Ruan et al. (2023)},
  year={2023},
  note={arXiv:2309.15817}
}
```

- arXiv: 2309.15817