Ecg Classification Macro F1 Eval

---
name: ecg-classification-macro-f1-eval
description: Evaluates a model's ability to classify 12-lead electrocardiogram (ECG) signals into multiple clinical diagnoses. It probes the model's robustness to class imbalance and its capacity to learn from long, redundant time-series sequences using self-supervised pre-training and supervised fine-tuning. Use when the user wants to benchmark on Fujiak, PCinC, PTB-XL, or asks about evaluating this task. Reports macro F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.07136
  bibtex_key: zhou2023maskedtransformer
  confidence: high
---

# ecg-classification-macro-f1-eval

> Masked Transformer for Electrocardiogram Classification — Zhou et al. (2023) (arXiv:2309.07136, 2023)

## What this evaluates

Evaluates a model's ability to classify 12-lead electrocardiogram (ECG) signals into multiple clinical diagnoses. It probes the model's robustness to class imbalance and its capacity to learn from long, redundant time-series sequences using self-supervised pre-training and supervised fine-tuning.

## Datasets

- **Fujiak** — total 220251; splits: train (-1), val (-1), test (-1)
- **PCinC** — total 79574; splits: train (-1), val (-1), test (-1)
- **PTB-XL** — total 21836; splits: train (-1), val (-1), test (-1)

## Metrics

- `macro F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of sensitivity (recall) and precision, averaged across all classes (macro average). Computed on the testing set.

## Input / output format

**Input**: 12-lead ECG recordings sampled at 500 Hz over 10 seconds, segmented into chunks of size 25 to form a sequence of length T=200.

**Output**: Multi-label classification predictions (binary indicators for each of the 22-25 clinical diagnosis labels).

## Scoring recipe

```python
def compute_macro_f1(y_true, y_pred):
    f1_scores = []
    for c in range(y_true.shape[1]):
        tp = np.sum((y_true[:, c] == 1) & (y_pred[:, c] == 1))
        fp = np.sum((y_true[:, c] == 0) & (y_pred[:, c] == 1))
        fn = np.sum((y_true[:, c] == 1) & (y_pred[:, c] == 0))
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        f1_scores.append(f1)
    return np.mean(f1_scores)
```

## Common pitfalls

- Using micro or weighted averaging instead of macro averaging, which masks poor performance on rare classes in this highly imbalanced multi-label setting.
- Ignoring the dataset-specific split protocols (e.g., patient-based 8:1:1 split for Fujiak, 10-fold split for PTB-XL), which can cause data leakage if not strictly followed.
- Applying the same hyperparameters for pre-training and fine-tuning; the paper specifies distinct learning rate schedules, momentum values, and regularization (DropPath, layer-wise LR decay) for each phase.

## Evidence (verbatim from paper)

> Therefore, we use the macro F1 score as the major evaluation metric throughout this paper.

## Citation

```bibtex
@misc{zhou2023maskedtransformer,
  title={Masked Transformer for Electrocardiogram Classification},
  author={Zhou et al. (2023)},
  year={2023},
  note={arXiv:2309.07136}
}
```

- arXiv: 2309.07136