Pulselm Eval

---
name: pulselm-eval
description: This benchmark evaluates multimodal physiological reasoning by testing whether large language models can accurately answer closed-ended questions conditioned on raw photoplethysmography (PPG) waveforms. It probes the model's ability to align continuous biosignal representations with natural language queries across diverse physiological domains and assesses cross-dataset generalization beyond the training distribution. Use when the user wants to benchmark on PulseLM, or asks about evaluating this task. Reports exact-match (EM) accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.03331
  bibtex_key: pham2026pulselm
  confidence: high
---

# pulselm-eval

> PulseLM: A Foundation Dataset and Benchmark for PPG-Text Learning — Pham et al. (2026) (arXiv:2603.03331, 2026)

## What this evaluates

This benchmark evaluates multimodal physiological reasoning by testing whether large language models can accurately answer closed-ended questions conditioned on raw photoplethysmography (PPG) waveforms. It probes the model's ability to align continuous biosignal representations with natural language queries across diverse physiological domains and assesses cross-dataset generalization beyond the training distribution.

## Datasets

- **PulseLM** — total 3150000; splits: train (-1), val (-1), test (-1); repo https://github.com/manhph2211/PulseLM

## Metrics

- `exact-match (EM) accuracy` **(primary)** — range: [0, 1]
  - Strict exact-match classification accuracy, computed by checking if the model's predicted answer exactly matches the ground truth label. Scores are averaged across question paraphrases to reduce sensitivity to surface-level linguistic variation.

## Input / output format

**Input**: A 10-second raw PPG waveform segment and a single natural language question (closed-ended QA prompt).

**Output**: A single answer selected from a predefined candidate set.

## Scoring recipe

```python
def compute_em_accuracy(predictions, gold_labels):
    # predictions and gold_labels are lists of strings
    exact_matches = [1.0 if pred == gold else 0.0 for pred, gold in zip(predictions, gold_labels)]
    return sum(exact_matches) / len(exact_matches)
# Note: Final score is averaged across multiple question paraphrases per physiological label.
```

## Common pitfalls

- Models are evaluated using strict exact-match accuracy, not fuzzy or token-level overlap metrics, so minor formatting or casing differences cause a score of 0.
- Cross-dataset generalization is evaluated by training exclusively on VitalDB and testing on unseen datasets (BCG, PPGBP, Sensors, UCI), rather than using standard held-out splits from the same distribution.
- Performance is averaged across multiple question paraphrases for the same label, which can mask variance in how different phrasings affect model reasoning.

## Evidence (verbatim from paper)

> Performance is measured primarily using a strict exact-match classification accuracy, averaged across question paraphrases to reduce sensitivity to surface-level linguistic variation.

## Citation

```bibtex
@misc{pham2026pulselm,
  title={PulseLM: A Foundation Dataset and Benchmark for PPG-Text Learning},
  author={Pham et al. (2026)},
  year={2026},
  note={arXiv:2603.03331}
}
```

- arXiv: 2603.03331