Twin 2k 500 Eval

---
name: twin-2k-500-eval
description: Evaluates the ability of LLMs to simulate individual-level human behavior across demographic, psychological, cognitive, economic, and behavioral economics domains. It measures test-retest accuracy and replication of known behavioral biases using a large-scale dataset of 2,058 U.S. individuals. Use when the user wants to benchmark on Twin-2K-500, or asks about evaluating this task. Reports test-retest accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.17479
  bibtex_key: toubia2025twin2k500
  confidence: medium
---

# twin-2k-500-eval

> Twin-2K-500: A dataset for building digital twins of over 2,000 people based on their answers to over 500 questions — Toubia et al. (2025) (arXiv:2505.17479, 2025)

## What this evaluates

Evaluates the ability of LLMs to simulate individual-level human behavior across demographic, psychological, cognitive, economic, and behavioral economics domains. It measures test-retest accuracy and replication of known behavioral biases using a large-scale dataset of 2,058 U.S. individuals.

## Datasets

- **Twin-2K-500** — total 2058; splits: full (2058); repo https://github.com/tianyipeng-lab/Digital-Twin-Simulation

## Metrics

- `test-retest accuracy` **(primary)** — range: [0, 1]
  - Calculated as the correlation or agreement rate between a participant's responses in Waves 1–3 and their repeated responses in Wave 4 for the same experimental conditions and questions.

## Input / output format

**Input**: Individual-level answers to 500+ questions spanning demographics, personality, cognitive ability, economic preferences, and behavioral economics experiments, along with assigned experimental conditions.

**Output**: Simulated answers or behavioral predictions for the same questions and conditions, generated by an LLM digital twin conditioned on the individual's profile.

## Scoring recipe

```python
def compute_test_retest_accuracy(wave1_3_responses, wave4_responses):
    aligned = align_by_participant_and_question(wave1_3_responses, wave4_responses)
    accuracy = mean_agreement(aligned)
    return accuracy
```

## Common pitfalls

- Evaluators must match Wave 4 responses to the exact same experimental conditions as Waves 1–3, as conditions were fixed per participant.
- The dataset mixes multiple response formats (Likert scales, binary choices, numerical estimates, open-ended text), requiring format-specific scoring rather than a single generic metric.
- Test-retest accuracy only applies to the 88 questions repeated in Wave 4, not the full 500+ question set.

## Evidence (verbatim from paper)

> Wave 4 repeated the within- and between-subject heuristics and biases experiments from the first three waves as well as the pricing study from wave 3 (88 questions in total). Participants were assigned to the exact same condition in wave 4 as they were in waves 1-3 for each of these experiments, providing us a clean measure of test-retest accuracy.

## Citation

```bibtex
@misc{toubia2025twin2k500,
  title={Twin-2K-500: A dataset for building digital twins of over 2,000 people based on their answers to over 500 questions},
  author={Toubia et al. (2025)},
  year={2025},
  note={arXiv:2505.17479}
}
```

- arXiv: 2505.17479