Stage Kg Eval

---
name: stage-kg-eval
description: Evaluates an LLM's ability to extract and structure narrative knowledge from full-length movie screenplays into canonical graphs. It probes entity and relation recognition while filtering out event-centric noise to focus on stable world-building elements. Use when the user wants to benchmark on STAGE-KG, or asks about evaluating this task. Reports Entity F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.08510
  bibtex_key: tian2026stage
  confidence: high
---

# stage-kg-eval

> STAGE: A Benchmark for Knowledge Graph Construction, Question Answering, and In-Script Role-Playing over Movie Screenplays — Qiuyu Tian et al. (2026) (arXiv:2601.08510, 2026)

## What this evaluates

Evaluates an LLM's ability to extract and structure narrative knowledge from full-length movie screenplays into canonical graphs. It probes entity and relation recognition while filtering out event-centric noise to focus on stable world-building elements.

## Datasets

- **STAGE-KG** — total 150; splits: test (150)

## Metrics

- `Entity F1` **(primary)** — range: [0, 1]
  - Standard F1 score computed over entity matches after normalization and embedding-based similarity matching. Precision and recall are calculated against canonical graphs restricted to reference entities with degree ≥ 2.
- `Relation F1` — range: [0, 1]
  - Standard F1 score for relation extraction, computed under the same degree ≥ 2 and reference-only constraints as entities.

## Input / output format

**Input**: Full screenplay text segmented into 600-token chunks (for zero-shot/GraphRAG baselines) or the complete script, depending on the baseline.

**Output**: Extracted entity-relation graph (nodes and edges) representing the movie's narrative world.

## Scoring recipe

```python
def score_kg(pred_graph, gold_graph):
    gold_filtered = filter_degree_and_type(gold_graph, min_degree=2, exclude_event=True)
    pred_filtered = filter_degree_and_type(pred_graph, min_degree=2, exclude_event=True)
    matched_entities = match_entities(pred_filtered.entities, gold_filtered.entities)
    matched_relations = match_relations(pred_filtered.relations, gold_filtered.relations, matched_entities)
    prec_e = len(matched_entities) / max(len(pred_filtered.entities), 1)
    rec_e = len(matched_entities) / max(len(gold_filtered.entities), 1)
    f1_e = 2 * prec_e * rec_e / (prec_e + rec_e)
    prec_r = len(matched_relations) / max(len(pred_filtered.relations), 1)
    rec_r = len(matched_relations) / max(len(gold_filtered.relations), 1)
    f1_r = 2 * prec_r * rec_r / (prec_r + rec_r)
    return {'entity_f1': f1_e, 'relation_f1': f1_r}
```

## Common pitfalls

- Event nodes and event-centric relations are explicitly excluded from scoring, but models often extract them anyway.
- Only reference entities and relations with a degree ≥ 2 are scored; low-degree or event-specific elements are ignored in the ground truth.
- Entity matching relies on both normalization rules and embedding similarity, not exact string matching.

## Evidence (verbatim from paper)

> Evaluation reports entity and relation precision/recall/F1 against canonical graphs, restricted to narratively salient structure: event nodes and event-centric relations are excluded, and only reference entities and relations with degree ≥ 2 are scored.

## Citation

```bibtex
@misc{tian2026stage,
  title={STAGE: A Benchmark for Knowledge Graph Construction, Question Answering, and In-Script Role-Playing over Movie Screenplays},
  author={Qiuyu Tian et al. (2026)},
  year={2026},
  note={arXiv:2601.08510}
}
```

- arXiv: 2601.08510