Theory Of Mind Qa Eval

---
name: theory-of-mind-qa-eval
description: Evaluates a model's ability to track first-order and second-order false beliefs, distinguishing an agent's mental state from physical reality and memory. It probes whether systems can maintain consistent world-state representations when agents hold incorrect beliefs about object locations or events. Use when the user wants to benchmark on Sally-Anne & Icecream Van Tasks, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1808.09352
  bibtex_key: nematzadeh2018evaluating
  confidence: medium
---

# theory-of-mind-qa-eval

> Evaluating Theory of Mind in Question Answering — Nematzadeh et al. (2018) (arXiv:1808.09352, 2018)

## What this evaluates

Evaluates a model's ability to track first-order and second-order false beliefs, distinguishing an agent's mental state from physical reality and memory. It probes whether systems can maintain consistent world-state representations when agents hold incorrect beliefs about object locations or events.

## Datasets

- **Sally-Anne & Icecream Van Tasks** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly answered questions (belief, reality, memory, or second-order belief) out of the total number of questions. Computed as exact-match or semantic equivalence against the gold answer.

## Input / output format

**Input**: A short narrative describing a scenario with agents, objects, and state/location changes, followed by a specific question (e.g., belief, reality, memory, or second-order belief question).

**Output**: A direct answer to the question, typically a location name (e.g., 'basket', 'box', 'church') or a yes/no response for control questions.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower())
    return correct / len(golds)
```

## Common pitfalls

- Models often answer based on the current physical reality instead of the agent's false belief.
- Performance degrades significantly when irrelevant or noisy sentences are introduced to the context.
- Control questions (reality/memory) are sometimes skipped, leading to chance-level performance on the primary belief question.

## Evidence (verbatim from paper)

> The participants are asked the following questions: - 'Where will Sally look for her marble?' (belief question) - 'Where is the marble really?' (reality question) - 'Where was the marble in the beginning?' (memory question)

## Citation

```bibtex
@misc{nematzadeh2018evaluating,
  title={Evaluating Theory of Mind in Question Answering},
  author={Nematzadeh et al. (2018)},
  year={2018},
  note={arXiv:1808.09352}
}
```

- arXiv: 1808.09352