Temporal Degradation Eval

---
name: temporal-degradation-eval
description: Evaluates how temporal misalignment between pretraining/fine-tuning data and evaluation data impacts model performance across classification and summarization benchmarks. Use when the user wants to benchmark on PubCLS, NewSum, TwiERC, AIC, PoliAff, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.13169
  bibtex_key: longpre2023pretrainers
  confidence: high
---

# temporal-degradation-eval

> A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity — Longpre et al. (2023) (arXiv:2305.13169, 2023)

## What this evaluates

Evaluates how temporal misalignment between pretraining/fine-tuning data and evaluation data impacts model performance across classification and summarization benchmarks.

## Datasets

- **PubCLS** — total ?; splits: test (-1)
- **NewSum** — total ?; splits: test (-1)
- **TwiERC** — total ?; splits: test (-1)
- **AIC** — total ?; splits: test (-1)
- **PoliAff** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted labels out of total instances.
- `Rouge-L` — range: [0, 1]
  - Recall-oriented F-score computed on the longest common subsequence between generated and reference summaries.

## Input / output format

**Input**: Text instances (news articles, tweets, science articles) or prompts for summarization, partitioned by publication year.

**Output**: Predicted class labels for classification tasks; generated text summaries for NewSum.

## Scoring recipe

```python
def compute_metrics(predictions, golds, task_type):
    if task_type == 'classification':
        correct = sum(1 for p, g in zip(predictions, golds) if p == g)
        return correct / len(golds)
    elif task_type == 'summarization':
        return rouge_l_score(predictions, golds)
```

## Common pitfalls

- Temporal degradation is often conflated with general domain shift; this protocol specifically isolates year-based misalignment.
- Performance drops are persistent and not recoverable via fine-tuning, so evaluating only on aligned years masks the true degradation effect.

## Evidence (verbatim from paper)

> Table 10: Left: Full results on the PubCLS temporal task splits from (Luu et al., 2021). This task evaluates news article source classification, measured with Accuracy. Right: Full results on the NewSum summarization task temporal splits from (Luu et al., 2021), evaluated in Rouge-L.

## Citation

```bibtex
@misc{longpre2023pretrainers,
  title={A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity},
  author={Longpre et al. (2023)},
  year={2023},
  note={arXiv:2305.13169}
}
```

- arXiv: 2305.13169