Stage Qa Eval

---
name: stage-qa-eval
description: Tests retrieval-augmented question answering over long screenplay contexts. It probes the model's ability to locate relevant information across chunked text and synthesize accurate answers under different retrieval architectures. Use when the user wants to benchmark on STAGE-QA, or asks about evaluating this task. Reports Question Correctness.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.08510
  bibtex_key: tian2026stage
  confidence: high
---

# stage-qa-eval

> STAGE: A Benchmark for Knowledge Graph Construction, Question Answering, and In-Script Role-Playing over Movie Screenplays — Qiuyu Tian et al. (2026) (arXiv:2601.08510, 2026)

## What this evaluates

Tests retrieval-augmented question answering over long screenplay contexts. It probes the model's ability to locate relevant information across chunked text and synthesize accurate answers under different retrieval architectures.

## Datasets

- **STAGE-QA** — total 4162; splits: test (4162)

## Metrics

- `Question Correctness` **(primary)** — range: [0, 1]
  - Binary correctness per question. Answers are generated five times per question; the question is marked correct if any generation is judged correct by GPT-4o.

## Input / output format

**Input**: A curated screenplay question, optionally accompanied by retrieved context chunks (300-token, ~200-token child/~400-token parent, or GraphRAG summaries) depending on the retrieval setting.

**Output**: Free-form natural language answer to the question.

## Scoring recipe

```python
def score_qa(question, retrieved_context):
    answers = [generate_answer(question, retrieved_context, temp=0.2) for _ in range(5)]
    correct = any(llm_judge_gpt4o(f'Q: {question}\nA: {ans}\nIs this correct?') for ans in answers)
    return {'question_correctness': 1.0 if correct else 0.0}
```

## Common pitfalls

- Correctness is judged by GPT-4o, not exact string matching, so paraphrased but correct answers are accepted.
- A question is marked correct if any of the 5 generations is judged correct, which inflates correctness compared to single-generation evaluation.
- Retrieval settings vary (native RAG, hybrid child-parent, GraphRAG), so context length and chunking strategy affect performance.

## Evidence (verbatim from paper)

> For each question, answers are generated five times; a question is marked correct if any generation is judged correct.

## Citation

```bibtex
@misc{tian2026stage,
  title={STAGE: A Benchmark for Knowledge Graph Construction, Question Answering, and In-Script Role-Playing over Movie Screenplays},
  author={Qiuyu Tian et al. (2026)},
  year={2026},
  note={arXiv:2601.08510}
}
```

- arXiv: 2601.08510