Literaryqa Eval

---
name: literaryqa-eval
description: Evaluates long-context language models on their ability to answer complex, abstractive questions about entire literary works. It probes narrative event understanding and semantic correctness, measuring how well generated answers align with human judgment rather than just matching reference strings. Use when the user wants to benchmark on LiteraryQA, or asks about evaluating this task. Reports ROUGE-L.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.13494
  bibtex_key: [REDACTED]
  confidence: high
---

# literaryqa-eval

> LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA — Bonomo et al. (2025) (arXiv:2510.13494, 2025)

## What this evaluates

Evaluates long-context language models on their ability to answer complex, abstractive questions about entire literary works. It probes narrative event understanding and semantic correctness, measuring how well generated answers align with human judgment rather than just matching reference strings.

## Datasets

- **LiteraryQA** — total ?; splits: test (-1); repo https://github.com/sapienzaNLP/literaryQA

## Metrics

- `ROUGE-L` **(primary)** — range: [0, 1]
  - Longest common subsequence between reference and candidate, normalized by reference length.
- `METEOR` — range: [0, 1]
  - F-measure based on unigram matching, incorporating stemming and synonymy.
- `F1` — range: [0, 1]
  - Token-level F1 score (harmonic mean of precision and recall).
- `EM` — range: [0, 1]
  - Exact match: 1 if candidate exactly equals reference, 0 otherwise.
- `BERTScore` — range: [0, 1]
  - Semantic similarity score computed using contextual embeddings from a DeBERTa-XLarge model finetuned for NLI.
- `LLM-as-a-judge` — range: other
  - Score generated by an LLM (e.g., Prometheus 2 7B, GPT-4.1, Claude 3.7 Sonnet) following a system prompt and rubric, typically on a 1–5 scale.

## Input / output format

**Input**: Question, reference answers, and the full book context (or book summary for LLM-as-a-judge evaluation).

**Output**: Numerical score (0–1) for ROUGE-L, METEOR, F1, BERTScore; binary flag for EM; or rubric-based score (1–5) for LLM-as-a-judge.

## Scoring recipe

```python
def compute_metrics(predictions, references):
    scores = {}
    for pred, ref in zip(predictions, references):
        scores['ROUGE-L'] = rouge_l_f1(pred, ref)
        scores['METEOR'] = meteor_score(pred, ref)
        scores['F1'] = token_level_f1(pred, ref)
        scores['EM'] = 1.0 if pred == ref else 0.0
        scores['BERTScore'] = bertscore_similarity(pred, ref, model='DeBERTa-XLarge')
    return scores
```

## Common pitfalls

- n-gram metrics (ROUGE-L, F1, EM) are highly sensitive to length mismatches and reference noise, yielding poor correlation with human judgment.
- LLM-as-a-judge scores vary drastically depending on whether reference answers or book summaries are provided as context.
- System-level correlation (Kendall’s τ) is reported instead of per-instance accuracy, which can obscure individual question failures.

## Evidence (verbatim from paper)

> We include metrics that have been used in literature to evaluate answers on NarrativeQA, namely: ROUGE-L*(Lin, [2004])*, METEOR*(Banerjee and Lavie, [2005])*, token-level F1 (F1) and exact-match (EM) taken from extractive QA*(Yang et al., [2018])*. As our neural-based metric, we use BERTScore*(Zhang et al., [2020])*, which provides a score between 0 and 1 that represents the semantical similarity of two pieces of text.

## Citation

```bibtex
@misc{bonomo2025literaryqa,
  title={LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA},
  author={Bonomo et al. (2025)},
  year={2025},
  note={arXiv:2510.13494}
}
```

- arXiv: 2510.13494