Ecg Pll Eval

---
name: ecg-pll-eval
description: This benchmark evaluates the robustness of Partial Label Learning (PLL) algorithms for multi-label ECG diagnosis under simulated clinical uncertainty. It probes how well models handle ambiguous candidate label sets generated through random, class-level, and instance-level ambiguity strategies. Use when the user wants to benchmark on PTB-XL, Chapman, or asks about evaluating this task. Reports micro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.11095
  bibtex_key: rahmani2025ecgpartial
  confidence: high
---

# ecg-pll-eval

> Investigating ECG Diagnosis with Ambiguous Labels using Partial Label Learning — Rahmani et al. (2025) (arXiv:2512.11095, 2025)

## What this evaluates

This benchmark evaluates the robustness of Partial Label Learning (PLL) algorithms for multi-label ECG diagnosis under simulated clinical uncertainty. It probes how well models handle ambiguous candidate label sets generated through random, class-level, and instance-level ambiguity strategies.

## Datasets

- **PTB-XL** — total 21837; splits: train (-1), test (-1)
- **Chapman** — total 10646; splits: train (-1), test (-1)

## Metrics

- `micro-F1` **(primary)** — range: [0, 1]
  - Standard micro-averaged F1 score computed across all classes and instances in a multi-label setting, aggregating true positives, false positives, and false negatives globally before calculating precision and recall.
- `AUROC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve, computed per class and micro-averaged for multi-label tasks to measure ranking quality across all decision thresholds.

## Input / output format

**Input**: 12-lead ECG time-series recordings (typically 10 seconds at 500 Hz sampling rate) with corresponding multi-label diagnostic annotations (24 subclasses for PTB-XL).

**Output**: Predicted probability scores for each of the 24 diagnostic classes per ECG recording.

## Scoring recipe

```python
def compute_micro_f1(y_true, y_pred):
    # y_true, y_pred: binary matrices (n_samples, n_classes)
    tp = np.sum((y_true == 1) & (y_pred == 1), axis=0)
    fp = np.sum((y_true == 0) & (y_pred == 1), axis=0)
    fn = np.sum((y_true == 1) & (y_pred == 0), axis=0)
    precision = tp / (tp + fp + 1e-8)
    recall = tp / (tp + fn + 1e-8)
    f1 = 2 * precision * recall / (precision + recall + 1e-8)
    return np.mean(f1)  # Micro-average across classes
```

## Common pitfalls

- The ambiguity generation process relies on three hyperparameters (p, r, epsilon); fixing p=0.5 as in the paper may not reflect real-world noise levels, and results are highly sensitive to these settings.
- Dataset compatibility limits direct comparison: the Chapman dataset lacks the hierarchical taxonomy and cardiologist-derived features required for Class-Level and Instance-Level Cardiologist-Driven ambiguity, restricting it to only three generation strategies.
- Multi-label nature of PTB-XL requires careful thresholding for F1 calculation; the paper reports mean ± std over 3 seeds, so statistical significance testing is needed for method comparisons.

## Evidence (verbatim from paper)

> To evaluate performance under partial supervision, we adopt micro-F1 and AUROC as widely used metrics in ECG classification . The results are reported on a clean held-out test set, while training is performed on versions of the training set augmented with varying levels of label ambiguity introduced by the six candidate label generation strategies described in Section[4.2]. We repeat each experiment across three random seeds, and report the mean and standard deviation of both metrics to assess robustness.

## Citation

```bibtex
@misc{rahmani2025ecgpartial,
  title={Investigating ECG Diagnosis with Ambiguous Labels using Partial Label Learning},
  author={Rahmani et al. (2025)},
  year={2025},
  note={arXiv:2512.11095}
}
```

- arXiv: 2512.11095