Opentom Eval

---
name: opentom-eval
description: This benchmark probes Theory-of-Mind (ToM) reasoning in LLMs by testing their ability to infer psychological mental states (e.g., beliefs, attitudes, intentions) and track physical object locations across naturally generated narratives. It specifically evaluates first- and second-order ToM capabilities under varying narrative lengths and question types. Use when the user wants to benchmark on OpenToM, or asks about evaluating this task. Reports macro-averaged F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.06044
  bibtex_key: xu2024opentom
  confidence: high
---

# opentom-eval

> OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Models — Hainiu Xu et al. (2024) (arXiv:2402.06044, 2024)

## What this evaluates

This benchmark probes Theory-of-Mind (ToM) reasoning in LLMs by testing their ability to infer psychological mental states (e.g., beliefs, attitudes, intentions) and track physical object locations across naturally generated narratives. It specifically evaluates first- and second-order ToM capabilities under varying narrative lengths and question types.

## Datasets

- **OpenToM** — total ?; splits: test (-1)

## Metrics

- `macro-averaged F1 score` **(primary)** — range: [0, 1]
  - Macro-averaged F1 score computed across binary and ternary classification tasks. It calculates the F1 score for each class independently and then takes the unweighted mean, which is necessary because the ground-truth labels are not uniformly distributed.

## Input / output format

**Input**: A narrative featuring personified characters with explicit intentions and motivated actions, followed by a binary or ternary classification question about physical locations or psychological mental states.

**Output**: A single classification label (binary or ternary) corresponding to the correct answer for the posed question.

## Scoring recipe

```python
def compute_macro_f1(predictions, gold_labels, num_classes):
    f1_scores = []
    for c in range(num_classes):
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != c and g == c)
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- Using accuracy instead of macro-F1 is misleading because the dataset's labels are not uniformly distributed.
- High F1 scores do not guarantee genuine ToM reasoning; the paper notes a high 'unfaithful rate' where models may guess correctly without following the narrative's mental states.
- Applying Chain-of-Thought or SimToM prompting universally can degrade performance on certain question types (e.g., Loc_fine or Att) while helping others.

## Evidence (verbatim from paper)

> As all the OpenToM questions are formulated as binary or ternary classification tasks and considering that the labels are not uniformly distributed (Figure A3), we evaluate model performance using the macro-averaged F1 scores (referred to as F1 scores henceforth).

## Citation

```bibtex
@misc{xu2024opentom,
  title={OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Models},
  author={Hainiu Xu et al. (2024)},
  year={2024},
  note={arXiv:2402.06044}
}
```

- arXiv: 2402.06044