Signalmc Med Eval

---
name: signalmc-med-eval
description: This benchmark evaluates biosignal foundation models on synchronized, long-duration single-lead ECG and PPG recordings from emergency department visits. It probes the models' ability to extract clinically meaningful representations for tasks such as age and sex prediction, emergency disposition, laboratory value regression, and ICD-10 diagnosis classification. Use when the user wants to benchmark on SignalMC-MED, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.09940
  bibtex_key: [REDACTED]
  confidence: high
---

# signalmc-med-eval

> SignalMC-MED: A Multimodal Benchmark for Evaluating Biosignal Foundation Models on Single-Lead ECG and PPG — Gustafsson et al. (2026) (arXiv:2603.09940, 2026)

## What this evaluates

This benchmark evaluates biosignal foundation models on synchronized, long-duration single-lead ECG and PPG recordings from emergency department visits. It probes the models' ability to extract clinically meaningful representations for tasks such as age and sex prediction, emergency disposition, laboratory value regression, and ICD-10 diagnosis classification.

## Datasets

- **SignalMC-MED** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/fregu856/SignalMC-MED

## Metrics

- `AUROC` **(primary)** — range: other
  - Area under the receiver operating characteristic curve. Measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
- `Pearson correlation` — range: other
  - Linear correlation coefficient between predicted and true continuous values. Ranges from -1 to 1, where 1 indicates perfect positive linear correlation.

## Input / output format

**Input**: 10-minute single-lead ECG and/or PPG time-series signals, processed as non-overlapping 10-second segments to extract visit-level representations via mean aggregation.

**Output**: Predicted continuous values for regression tasks or class probabilities for classification tasks across 20 clinical benchmark tasks.

## Scoring recipe

```python
def score(predictions, gold, task_type):
    if task_type == 'regression':
        return pearsonr(predictions, gold)
    elif task_type == 'classification':
        return roc_auc_score(gold, predictions)
    return None
```

## Common pitfalls

- Models are evaluated strictly as frozen feature extractors with linear probing, not fine-tuned end-to-end.
- Performance is aggregated across four training data percentages (10%, 25%, 50%, 100%) and averaged over five resampling repetitions, so single-run scores are not directly comparable.
- Multimodal fusion uses late feature-level averaging of independently extracted ECG and PPG representations, not joint multimodal training.

## Evidence (verbatim from paper)

> Performance is reported using Pearson correlation for regression and area under the receiver operating characteristic curve (AUROC) for classification. To assess robustness and data efficiency, downstream models are trained on 10%, 25%, 50%, and 100% of the available train visits with five repeated resamplings.

## Citation

```bibtex
@misc{gustafsson2026signalmcmed,
  title={SignalMC-MED: A Multimodal Benchmark for Evaluating Biosignal Foundation Models on Single-Lead ECG and PPG},
  author={Gustafsson et al. (2026)},
  year={2026},
  note={arXiv:2603.09940}
}
```

- arXiv: 2603.09940