Diamonds Eval

---
name: diamonds-eval
description: Evaluates Theory of Mind and participant-centric reasoning in multi-party dialogues by testing a model's ability to track dynamic numerical variables, filter distractors, and reason from specific character perspectives (including false beliefs) rather than using omniscient context. Use when the user wants to benchmark on DIAMONDs, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.12651
  bibtex_key: [REDACTED]
  confidence: high
---

# diamonds-eval

> $\texttt{DIAMONDs}$: A Dataset for $\mathbb{D}$ynamic $\mathbb{I}$nformation $\mathbb{A}$nd $\mathbb{M}$ental modeling $\mathbb{O}$f $\mathbb{N}$umeric $\mathbb{D}$iscussions — Ghosh et al. (2025) (arXiv:2505.12651, 2025)

## What this evaluates

Evaluates Theory of Mind and participant-centric reasoning in multi-party dialogues by testing a model's ability to track dynamic numerical variables, filter distractors, and reason from specific character perspectives (including false beliefs) rather than using omniscient context.

## Datasets

- **DIAMONDs** — total ?; splits: test (-1); repo https://github.com/StonyBrookNLP/diamonds

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. Computed separately for omniscient, participant-centric, true belief, false belief, distractor, and unanswerable subsets.

## Input / output format

**Input**: Multi-party conversational context containing dynamic numerical variables, knowledge asymmetries, and potential distractors, followed by a question that is either omniscient or participant-centric.

**Output**: A direct answer to the question (exact format not specified, evaluated as correct/incorrect).

## Scoring recipe

```python
correct = sum(1 for p, g in zip(predictions, golds) if p == g)
accuracy = (correct / len(golds)) * 100
# Computed separately for subsets: omniscient, participant-centric, true belief, false belief, distractor, unanswerable
```

## Common pitfalls

- Models often fail to filter out irrelevant distractor information when answering participant-centric questions.
- Models struggle with false belief scenarios, frequently answering based on ground truth rather than the participant's outdated knowledge.
- Models exhibit a bias to generate an answer even when questions are underspecified and unanswerable.

## Evidence (verbatim from paper)

> Table 1: Omniscient is the % of correctly answered omniscient questions, Participant Centric is the % of correctly answered participant-centric ones, and Overall is the accuracy across all the question type.

## Citation

```bibtex
@misc{ghosh2025diamonds,
  title={$\texttt{DIAMONDs}$: A Dataset for $\mathbb{D}$ynamic $\mathbb{I}$nformation $\mathbb{A}$nd $\mathbb{M}$ental modeling $\mathbb{O}$f $\mathbb{N}$umeric $\mathbb{D}$iscussions},
  author={Ghosh et al. (2025)},
  year={2025},
  note={arXiv:2505.12651}
}
```

- arXiv: 2505.12651