Stage Es Eval

---
name: stage-es-eval
description: Assesses an LLM's capacity to abstract scene-level events into concise, free-form descriptions without schema constraints. It evaluates whether the generated events form a coherent, non-redundant structure and remain factually grounded in the screenplay text. Use when the user wants to benchmark on STAGE-ES, or asks about evaluating this task. Reports Event-Structure Consistency.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.08510
  bibtex_key: tian2026stage
  confidence: high
---

# stage-es-eval

> STAGE: A Benchmark for Knowledge Graph Construction, Question Answering, and In-Script Role-Playing over Movie Screenplays — Qiuyu Tian et al. (2026) (arXiv:2601.08510, 2026)

## What this evaluates

Assesses an LLM's capacity to abstract scene-level events into concise, free-form descriptions without schema constraints. It evaluates whether the generated events form a coherent, non-redundant structure and remain factually grounded in the screenplay text.

## Datasets

- **STAGE-ES** — total 13750; splits: test (13750)

## Metrics

- `Event-Structure Consistency` **(primary)** — range: [0, 1]
  - LLM-judged score (via GPT-4o) measuring whether predicted events form a coherent, non-redundant abstraction of the scene.
- `Factual Supportfulness` — range: [0, 1]
  - LLM-judged score measuring whether each predicted event is grounded in explicit screenplay evidence and avoids hallucinations.

## Input / output format

**Input**: Full text of a single movie scene.

**Output**: Concise, free-form event description(s) with no constraints on schema, ordering, or event count.

## Scoring recipe

```python
def score_es(pred_events, gold_scene_text):
    consistency_score = llm_judge_gpt4o(
        prompt=f'Scene: {gold_scene_text}\nPredicted events: {pred_events}\nRate event-structure consistency (0-1).'
    )
    support_score = llm_judge_gpt4o(
        prompt=f'Scene: {gold_scene_text}\nPredicted events: {pred_events}\nRate factual supportfulness (0-1).'
    )
    return {'event_structure_consistency': consistency_score, 'factual_supportfulness': support_score}
```

## Common pitfalls

- Evaluation uses LLM-as-a-judge (GPT-4o) rather than exact string overlap, so scores depend on the judge's prompting and temperature.
- The metric emphasizes event-level abstraction and structural coherence rather than sentence-level fact decomposition, unlike standard FActScore.
- Models are not constrained to a fixed schema or event count, making automated exact-match evaluation impossible.

## Evidence (verbatim from paper)

> Evaluation follows the Event-Structure Consistency framework along two dimensions: event-structure consistency, which measures whether predicted events form a coherent, non-redundant abstraction of the scene; and factual supportfulness, which measures whether each event is grounded in explicit screenplay evidence and avoids unsupported hallucinations.

## Citation

```bibtex
@misc{tian2026stage,
  title={STAGE: A Benchmark for Knowledge Graph Construction, Question Answering, and In-Script Role-Playing over Movie Screenplays},
  author={Qiuyu Tian et al. (2026)},
  year={2026},
  note={arXiv:2601.08510}
}
```

- arXiv: 2601.08510