Epidemiological Benchmark Eval

---
name: epidemiological-benchmark-eval
description: Evaluates spatio-temporal graph models on forecasting, stability, and denoising tasks using synthetic epidemiological data generated from PDEs. Probes the model's ability to predict future states, resist noise/dropout, and recover clean signals from corrupted graph time-series. Use when the user wants to benchmark on Epidemiological Synthetic Dataset, or asks about evaluating this task. Reports RMSE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.04140
  bibtex_key: arndt2025synthetic
  confidence: medium
---

# epidemiological-benchmark-eval

> Synthetic Datasets for Machine Learning on Spatio-Temporal Graphs using PDEs — Arndt et al. (2025) (arXiv:2502.04140, 2025)

## What this evaluates

Evaluates spatio-temporal graph models on forecasting, stability, and denoising tasks using synthetic epidemiological data generated from PDEs. Probes the model's ability to predict future states, resist noise/dropout, and recover clean signals from corrupted graph time-series.

## Datasets

- **Epidemiological Synthetic Dataset** — total ?; splits: train (-1), val (-1); repo https://github.com/Jostarndt/Synthetic_Datasets_for_Temporal_Graphs

## Metrics

- `RMSE` **(primary)** — range: other
  - Root Mean Squared Error, calculated as the square root of the average of squared differences between predicted and true values across all samples and time-steps.

## Input / output format

**Input**: Historical spatio-temporal graph node features over time, optionally corrupted with additive Gaussian noise or dropout for stability/denoising tasks.

**Output**: Predicted node features for subsequent time-steps.

## Scoring recipe

```python
def compute_rmse(predictions, targets):
    # predictions, targets: arrays of shape (num_samples, num_time_steps)
    squared_diff = (predictions - targets) ** 2
    mean_squared_error = np.mean(squared_diff)
    return np.sqrt(mean_squared_error)
```

## Common pitfalls

- RMSE is reported per time-step across the validation set rather than as a single aggregate scalar, requiring careful per-step plotting or aggregation.
- Stability and denoising experiments require applying specific noise distributions (Gaussian or Dropout) to the training data, which must exactly match the evaluation setup.
- Models are trained on synthetic data and evaluated on synthetic validation data for stability/denoising, not real-world data, which limits direct real-world performance claims.

## Evidence (verbatim from paper)

> A visual comparison of RMSEs for each time-step in the forecasting experiment for different models can be seen in Fig[10]... Figure 11: Plots of the RMSE of the presented models during the stability experiments for each predicted time-step on the validation dataset. The models were trained on the synthetic dataset, and evaluated on a synthetic dataset with: a) additive Gaussian noise, b) dropout noise.

## Citation

```bibtex
@misc{arndt2025synthetic,
  title={Synthetic Datasets for Machine Learning on Spatio-Temporal Graphs using PDEs},
  author={Arndt et al. (2025)},
  year={2025},
  note={arXiv:2502.04140}
}
```

- arXiv: 2502.04140