Ppg Health Benchmark Eval

---
name: ppg-health-benchmark-eval
description: Evaluates the transferability and emergent capabilities of generalist and specialist time-series foundation models on diverse physiological tasks using PPG and cross-modal signals. Probes classification (e.g., arrhythmia, mental load, activity recognition) and regression (e.g., vital signs, blood chemistry, blood pressure) performance across clinical and ambulatory settings. Use when the user wants to benchmark on Stanford AF, Simband, Real World PPG, MIMIC-III, Sleep-EDF, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.14254
  bibtex_key: kataria2025generalist
  confidence: high
---

# ppg-health-benchmark-eval

> Generalist vs Specialist Time Series Foundation Models: Investigating Potential Emergent Behaviors in Assessing Human Health Using PPG Signals — Kataria et al. (2025) (arXiv:2510.14254, 2025)

## What this evaluates

Evaluates the transferability and emergent capabilities of generalist and specialist time-series foundation models on diverse physiological tasks using PPG and cross-modal signals. Probes classification (e.g., arrhythmia, mental load, activity recognition) and regression (e.g., vital signs, blood chemistry, blood pressure) performance across clinical and ambulatory settings.

## Datasets

- **Stanford AF** — total 41114; splits: train (-1), val (-1), test (-1)
- **Simband** — total 857; splits: train (-1), val (-1), test (-1)
- **Real World PPG** — total 2074; splits: train (1374), test (700)
- **MIMIC-III** — total ?; splits: train (-1), val (-1), test (-1)
- **Sleep-EDF** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted class labels out of total samples. Calculated as correct predictions divided by total test samples.
- `MAE` — range: [0, inf)
  - Mean Absolute Error between predicted and ground-truth continuous values. Standard metric for regression tasks in this benchmark.

## Input / output format

**Input**: Fixed-length PPG signal segments (resampled to 40 Hz, min-max normalized to [0,1]) or cross-modal physiological signals (EEG, CBFV) depending on the downstream task. Segment duration varies by dataset (e.g., 5s, 10s, 25s, 30s).

**Output**: Classification tasks: discrete class labels (e.g., AF/non-AF, stress/non-stress, sleep/wake). Regression tasks: continuous physiological values (e.g., SpO2%, electrolyte concentration, heart rate, blood pressure, ICP).

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels, task_type):
    if task_type == 'classification':
        correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
        return correct / len(gold_labels)
    elif task_type == 'regression':
        errors = [abs(p - g) for p, g in zip(predictions, gold_labels)]
        return sum(errors) / len(errors)
```

## Common pitfalls

- Data leakage from overlapping time windows: PPG segments must strictly precede lab results to preserve temporal causality.
- Participant overlap: Inter-patient splits are required; mixing participants across train/test invalidates generalization claims.
- Signal normalization & resampling: All segments must be min-max normalized and resampled to 40 Hz before evaluation to match model training conditions.

## Evidence (verbatim from paper)

> The tasks are broadly categorized into two types: classification and regression. ... To ensure consistency across samples, we apply min-max normalization to each segment, scaling the data within the range of 0 to 1. ... we employ an inter-patient evaluation scheme in which the training and testing data are derived from different groups of participants to assess the model’s generalizability. ... We report top-1 accuracy on the official test set.

## Citation

```bibtex
@misc{kataria2025generalist,
  title={Generalist vs Specialist Time Series Foundation Models: Investigating Potential Emergent Behaviors in Assessing Human Health Using PPG Signals},
  author={Kataria et al. (2025)},
  year={2025},
  note={arXiv:2510.14254}
}
```

- arXiv: 2510.14254