Tempo Sum Eval

---
name: tempo-sum-eval
description: Evaluates text summarization models' temporal generalization by testing on datasets split by publication date, specifically probing how well models handle knowledge-conflicting future articles versus in-distribution past data. Use when the user wants to benchmark on BBC, CNN, or asks about evaluating this task. Reports FactCC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.01951
  bibtex_key: cheang2023tempo-sum
  confidence: high
---

# tempo-sum-eval

> Can LMs Generalize to Future Data? An Empirical Analysis on Text Summarization — Cheang et al. (2023) (arXiv:2305.01951, 2023)

## What this evaluates

Evaluates text summarization models' temporal generalization by testing on datasets split by publication date, specifically probing how well models handle knowledge-conflicting future articles versus in-distribution past data.

## Datasets

- **BBC** — total ?; splits: in-distribution test (150), future test (150); repo https://github.com/NLP2CT/TempoSum
- **CNN** — total ?; splits: in-distribution test (150), future test (150); repo https://github.com/NLP2CT/TempoSum

## Metrics

- `FactCC` **(primary)** — range: [0, 1]
  - Entailment-based faithfulness metric that checks if each claim in the generated summary is logically entailed by the source text using an NLI model.
- `QAFactEval` — range: [0, 1]
  - Uses question-generation and question-answering models to estimate summary faithfulness by verifying if answers to summary-derived questions align with the source text.

## Input / output format

**Input**: Source news article text

**Output**: Generated summary text

## Scoring recipe

```python
def evaluate(predictions, sources):
    factcc_scores = []
    qafe_scores = []
    for pred, src in zip(predictions, sources):
        claims = extract_claims(pred)
        factcc_scores.append(mean([nli_entail(src, c) for c in claims]))
        questions = generate_questions(pred)
        answers = [qa_model(src, q) for q in questions]
        qafe_scores.append(estimate_faithfulness(answers, src))
    return {'FactCC': mean(factcc_scores), 'QAFactEval': mean(qafe_scores)}
```

## Common pitfalls

- Models heavily rely on parametric world knowledge rather than the source text, leading to hallucinations of outdated facts on future data.
- Automatic metrics like FactCC and QAFactEval fail to reliably detect faithfulness improvements on future data, making human evaluation necessary for accurate assessment.

## Evidence (verbatim from paper)

> FactCC: An entailment-based faithfulness evaluation metric proposed by Kryscinski et al. (2020). Previous studies (Pagnoni et al., 2021) show that FactCC has a strong correlation with human judgments on existing benchmarks.

QAFactEval: The state-of-the-art faithfulness evaluation for text summarization (Fabbri et al., 2022). This metric utilizes question-generation and question-answering models to estimate the faithfulness of a summary.

## Citation

```bibtex
@misc{cheang2023tempo-sum,
  title={Can LMs Generalize to Future Data? An Empirical Analysis on Text Summarization},
  author={Cheang et al. (2023)},
  year={2023},
  note={arXiv:2305.01951}
}
```

- arXiv: 2305.01951