Stage Icrp Eval

---
name: stage-icrp-eval
description: Evaluates an LLM's ability to role-play as a specific movie character using memory-grounded agent frameworks. It probes consistency with the character's persona, speaking style, and narrative facts across interactive dialogues. Use when the user wants to benchmark on STAGE-ICRP, or asks about evaluating this task. Reports Persona Consistency.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.08510
  bibtex_key: tian2026stage
  confidence: high
---

# stage-icrp-eval

> STAGE: A Benchmark for Knowledge Graph Construction, Question Answering, and In-Script Role-Playing over Movie Screenplays — Qiuyu Tian et al. (2026) (arXiv:2601.08510, 2026)

## What this evaluates

Evaluates an LLM's ability to role-play as a specific movie character using memory-grounded agent frameworks. It probes consistency with the character's persona, speaking style, and narrative facts across interactive dialogues.

## Datasets

- **STAGE-ICRP** — total 1505; splits: test (1505)

## Metrics

- `Persona Consistency` **(primary)** — range: [0, 1]
  - LLM-judged score (via GPT-4o) measuring whether the model's responses align with the character's established persona.
- `Speaking Style Consistency` — range: [0, 1]
  - LLM-judged score measuring whether the model maintains the character's unique linguistic patterns and tone.
- `Narrative Faithfulness` — range: [0, 1]
  - LLM-judged score measuring whether the model's responses remain consistent with verified character facts and plot events.

## Input / output format

**Input**: A persona specification derived from screenplay evidence, optionally augmented with narrative episodic memory summaries and explicit character facts, plus an audience-facing interaction question.

**Output**: First-person natural language response from the perspective of the specified character.

## Scoring recipe

```python
def score_icrp(pred_response, persona_spec, gold_facts):
    persona_score = llm_judge_gpt4o(f'Persona: {persona_spec}\nResponse: {pred_response}\nRate persona consistency.')
    style_score = llm_judge_gpt4o(f'Persona: {persona_spec}\nResponse: {pred_response}\nRate speaking style consistency.')
    faith_score = llm_judge_gpt4o(f'Facts: {gold_facts}\nResponse: {pred_response}\nRate narrative faithfulness.')
    return {'persona_consistency': persona_score, 'speaking_style_consistency': style_score, 'narrative_faithfulness': faith_score}
```

## Common pitfalls

- Evaluation relies entirely on GPT-4o as an LLM judge, so scores may reflect the judge's biases rather than ground-truth character alignment.
- Models are tested with three different memory configurations (Prompt-only, +Episodic Memory, +Episodic Memory + Facts), so performance heavily depends on the memory retrieval setup.
- First-person role-playing responses are inherently subjective, making deterministic scoring impossible without LLM judges.

## Evidence (verbatim from paper)

> Evaluation is conducted along three dimensions: persona consistency, speaking style consistency, and narrative faithfulness. Full prompts and judging criteria are provided in Appendix [D].

## Citation

```bibtex
@misc{tian2026stage,
  title={STAGE: A Benchmark for Knowledge Graph Construction, Question Answering, and In-Script Role-Playing over Movie Screenplays},
  author={Qiuyu Tian et al. (2026)},
  year={2026},
  note={arXiv:2601.08510}
}
```

- arXiv: 2601.08510