Mindgames Eval

---
name: mindgames-eval
description: Evaluates large language models' ability to perform higher-order epistemic reasoning and multi-agent belief tracking. It probes whether models can correctly update beliefs based on public announcements and answer True/False questions about agents' knowledge states. Use when the user wants to benchmark on MindGames, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.03353
  bibtex_key: sileo2023mindgames
  confidence: high
---

# mindgames-eval

> MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic — Sileo et al. (2023) (arXiv:2305.03353, 2023)

## What this evaluates

Evaluates large language models' ability to perform higher-order epistemic reasoning and multi-agent belief tracking. It probes whether models can correctly update beliefs based on public announcements and answer True/False questions about agents' knowledge states.

## Datasets

- **MindGames** — total 400; splits: test (400), train (11200), val (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted True/False labels out of the total number of evaluated instances.

## Input / output format

**Input**: A natural language premise describing a multi-agent epistemic scenario, followed by a question and hypothesis formatted as: '<PREMISE> Question: <HYPOTHESIS> True or False ?'

**Output**: The model must generate exactly 'True' or 'False' as the next token/continuation.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Models may exploit lexical shortcuts or hypothesis-only bias rather than performing genuine belief tracking, as evidenced by the deberta-small baseline achieving 83% accuracy.
- Evaluators must strictly adhere to the lm-eval-harness prompt template to ensure perplexity-based scoring is comparable across models.
- The test set is filtered by difficulty using deberta-small confidence and prediction discrepancy; evaluating on the unfiltered raw generation set may yield inflated performance.

## Evidence (verbatim from paper)

> We present 50 test samples per setup to two NLP researchers only instructed to perform entailment detection. Inter-annotator agreement is 0.89, and average accuracy is 94%.

## Citation

```bibtex
@misc{sileo2023mindgames,
  title={MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic},
  author={Sileo et al. (2023)},
  year={2023},
  note={arXiv:2305.03353}
}
```

- arXiv: 2305.03353