Temporal Domain Generalization Eval

---
name: temporal-domain-generalization-eval
description: Evaluates a model's ability to generalize to future, unseen temporal domains without full retraining. It measures out-of-distribution accuracy on sequentially arriving target domains after training on historical source domains. Use when the user wants to benchmark on Yearbook, Rotated MNIST (RMNIST), FMoW, Huffpost, Arxiv, CLEAR-10/100, or asks about evaluating this task. Reports OOD_avg accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.26045
  bibtex_key: liu2025tea
  confidence: high
---

# temporal-domain-generalization-eval

> Scaling Up Temporal Domain Generalization via Temporal Experts Averaging — Aoming Liu et al. (arXiv:2509.26045, 2025)

## What this evaluates

Evaluates a model's ability to generalize to future, unseen temporal domains without full retraining. It measures out-of-distribution accuracy on sequentially arriving target domains after training on historical source domains.

## Datasets

- **Yearbook** — total ?; splits: source_train (-1), source_val (-1), target_test (-1)
- **Rotated MNIST (RMNIST)** — total ?; splits: source_train (-1), source_val (-1), target_test (-1)
- **FMoW** — total ?; splits: source_train (-1), source_val (-1), target_test (-1)
- **Huffpost** — total ?; splits: source_train (-1), source_val (-1), target_test (-1)
- **Arxiv** — total ?; splits: source_train (-1), source_val (-1), target_test (-1)
- **CLEAR-10/100** — total ?; splits: source_train (-1), source_val (-1), target_test (-1)

## Metrics

- `OOD_avg accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted class labels on the target domain(s). Computed as the average accuracy across all target domains.

## Input / output format

**Input**: Image or text data samples from sequentially ordered source domains, with the model required to predict class labels for sequentially ordered target domains.

**Output**: Class predictions (or probabilities) for each target domain instance.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Confusing Temporal Domain Generalization (TDG) with standard Domain Generalization (DG) or Continual Domain Generalization (CDGTD), which have different data access and training constraints.
- Failing to respect the chronological domain ordering and specified source-to-target ratios (e.g., 16:5 for Yearbook), which breaks the temporal shift assumption.
- Overlooking computational constraints that force some baselines (e.g., GI, DRAIN) to skip finetuning or use classifier-only variants on larger benchmarks, skewing direct performance comparisons.

## Evidence (verbatim from paper)

> Each dataset is divided into first S source and last F target domains with ratios S:F of: Yearbook (16:5), RMNIST (6:3), FMoW (13:3), Huffpost (4:3), Arxiv (9:7), and CLEAR-10/100 (5:5). Each source domain uses a random 90%-10% train-validation split. We report OOD_avg accuracy (%) across different buffer sizes (1%, 3%, 5%, 10%, 20).

## Citation

```bibtex
@misc{liu2025tea,
  title={Scaling Up Temporal Domain Generalization via Temporal Experts Averaging},
  author={Aoming Liu et al.},
  year={2025},
  note={arXiv:2509.26045}
}
```

- arXiv: 2509.26045