Tombench Eval

---
name: tombench-eval
description: Evaluates large language models' Theory of Mind capabilities by testing their ability to infer mental states (beliefs, intentions, emotions) across multiple orders of reasoning using story-based narratives. The benchmark probes whether models can accurately track character perspectives and answer questions about what different agents know or believe in complex social scenarios. Use when the user wants to benchmark on TOMBENCH, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.00026
  bibtex_key: [REDACTED]
  confidence: medium
---

# tombench-eval

> Theory of Mind in Large Language Models: Assessment and Enhancement — Chen et al. (2025) (arXiv:2505.00026, 2025)

## What this evaluates

Evaluates large language models' Theory of Mind capabilities by testing their ability to infer mental states (beliefs, intentions, emotions) across multiple orders of reasoning using story-based narratives. The benchmark probes whether models can accurately track character perspectives and answer questions about what different agents know or believe in complex social scenarios.

## Datasets

- **TOMBENCH** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Percentage of test samples where the model's selected multiple-choice option exactly matches the ground truth answer. Evaluation is based solely on the final answer option to minimize subjective judgment.

## Input / output format

**Input**: A narrative story describing a scenario involving characters' actions and mental states, followed by a multiple-choice question about the characters' beliefs, intentions, or emotions.

**Output**: A single selected option from the provided multiple-choice answers.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_answers):
    if pred.strip().lower() == gold.strip().lower():
        correct += 1
metric_value = correct / len(predictions)
```

## Common pitfalls

- Evaluation relies solely on exact match of the final option, which may not reflect true comprehension of the reasoning process.
- The dataset is bilingual (Chinese and English), so evaluation must be conducted separately or carefully aligned for each language to avoid cross-lingual contamination.

## Evidence (verbatim from paper)

> All test samples in TOMBENCH follow a multiple-choice question answering format, which minimizes subjective judgment. However, the multiple-choice question answering format poses a challenge in determining whether LLMs truly comprehend the questions and answer them correctly, as evaluation is based solely on the final answer option.

## Citation

```bibtex
@misc{chen2025theoryofmind,
  title={Theory of Mind in Large Language Models: Assessment and Enhancement},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2505.00026}
}
```

- arXiv: 2505.00026