Vitalbench Eval

---
name: vitalbench-eval
description: Evaluates long-term multivariate time-series forecasting of intraoperative vital signs under three clinically realistic conditions: complete data, variable missingness, and cross-center generalization. It probes a model's ability to handle heterogeneous clinical data, adapt to missing sensor inputs, and generalize across different hospital centers. Use when the user wants to benchmark on VitalDB, MOVER-SIS, or asks about evaluating this task. Reports MAE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.13757
  bibtex_key: [REDACTED]
  confidence: high
---

# vitalbench-eval

> VitalBench: A Rigorous Multi-Center Benchmark for Long-Term Vital Sign Prediction in Intraoperative Care — Cai et al. (2025) (arXiv:2511.13757, 2025)

## What this evaluates

Evaluates long-term multivariate time-series forecasting of intraoperative vital signs under three clinically realistic conditions: complete data, variable missingness, and cross-center generalization. It probes a model's ability to handle heterogeneous clinical data, adapt to missing sensor inputs, and generalize across different hospital centers.

## Datasets

- **VitalDB** — total 962; splits: train (-1), val (-1), test (-1); repo https://github.com/XiudingCai/VitalBench
- **MOVER-SIS** — total 3221; splits: train (-1), val (-1), test (-1); repo https://github.com/XiudingCai/VitalBench

## Metrics

- `MAE` **(primary)** — range: other
  - Mean Absolute Error: average of absolute differences between predicted and actual values. Lower is better.
- `RMSE` — range: other
  - Root Mean Squared Error: square root of the average of squared differences between predicted and actual values. Lower is better.
- `R2` — range: [0, 1]
  - R-squared (Coefficient of Determination): proportion of variance in the target variable explained by the model. Higher is better.
- `CC` — range: [0, 1]
  - Correlation Coefficient (Pearson): measures linear correlation between predicted and actual values. Higher is better.

## Input / output format

**Input**: Multivariate time-series sequences of dynamic physiological variables and static covariates over a 30-minute retrospective window (1 Hz sampling). Input dimensions may vary across samples due to randomly dropped variable channels in Tracks 2 and 3.

**Output**: Predicted time-series values for target vital signs over specified prediction horizons (1, 3, 10, or 30 minutes).

## Scoring recipe

```python
def compute_metrics(pred, gold):
    mae = np.mean(np.abs(pred - gold))
    rmse = np.sqrt(np.mean((pred - gold) ** 2))
    ss_res = np.sum((gold - pred) ** 2)
    ss_tot = np.sum((gold - np.mean(gold)) ** 2)
    r2 = 1 - (ss_res / ss_tot)
    cc = np.corrcoef(pred, gold)[0, 1]
    return {'MAE': mae, 'RMSE': rmse, 'R2': r2, 'CC': cc}
```

## Common pitfalls

- Applying fixed-dimension models (e.g., LightTS, MICN) to Tracks 2 and 3, which require handling variable input dimensions due to missing sensor channels.
- Imputing missing values before training or evaluation, which introduces spurious data and biases predictions compared to the benchmark's recommended masked loss approach.
- Using random data splitting instead of chronological splitting, which violates the temporal validity requirement for intraoperative time-series forecasting.

## Evidence (verbatim from paper)

> For all experiments, we used standard evaluation metrics, including Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), R-squared (R2), and Correlation Coefficient (CC), to ensure a comprehensive and consistent assessment of model performance. Lower values for MAE and RMSE indicate better accuracy, while higher values for R2 and CC suggest improved model fit and correlation.

## Citation

```bibtex
@misc{cai2025vitalbench,
  title={VitalBench: A Rigorous Multi-Center Benchmark for Long-Term Vital Sign Prediction in Intraoperative Care},
  author={Cai et al. (2025)},
  year={2025},
  note={arXiv:2511.13757}
}
```

- arXiv: 2511.13757