Etp Ecg Linear Zero Shot Eval

---
name: etp-ecg-linear-zero-shot-eval
description: This protocol evaluates the transferability and robustness of pre-trained ECG representations by measuring classification performance on downstream cardiac disease datasets. It probes both supervised linear probing capabilities and cross-modal zero-shot generalization to specific cardiac conditions without fine-tuning the encoder. Use when the user wants to benchmark on PTB-XL, CPSC2018, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.07145
  bibtex_key: liu2023etp
  confidence: high
---

# etp-ecg-linear-zero-shot-eval

> ETP: Learning Transferable ECG Representations via ECG-Text Pre-training — Che Liu et al. (2023) (arXiv:2309.07145, 2023)

## What this evaluates

This protocol evaluates the transferability and robustness of pre-trained ECG representations by measuring classification performance on downstream cardiac disease datasets. It probes both supervised linear probing capabilities and cross-modal zero-shot generalization to specific cardiac conditions without fine-tuning the encoder.

## Datasets

- **PTB-XL** — total 21837; splits: train (-1), val (-1), test (-1)
- **CPSC2018** — total 6877; splits: train (-1), val (-1), test (-1)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve. Measures the probability that a classifier ranks a positive example higher than a negative one across all classification thresholds.
- `F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Evaluated per class and averaged.
- `ACC` — range: [0, 1]
  - Classification accuracy, calculated as the number of correct predictions divided by the total number of predictions.

## Input / output format

**Input**: 12-lead ECG signal (500 Hz sampling rate, up to 60s duration) paired with clinical text report. For zero-shot, the input is the ECG signal and a fixed text prompt template 'this ECG indicates [disease_name]'.

**Output**: Predicted diagnostic category label (e.g., NORM, MI, STTC, CD, HYP for PTB-XL; 9 specific arrhythmia/condition labels for CPSC2018). In zero-shot, the category whose prompt embedding has the highest cosine similarity to the ECG embedding.

## Scoring recipe

```python
def linear_eval(encoder, classifier, loader):
    encoder.eval()
    preds, labels = [], []
    for x, y in loader:
        with torch.no_grad():
            feats = encoder(x)
        preds.append(classifier(feats).argmax(dim=1))
        labels.append(y)
    return compute_metrics(torch.cat(preds), torch.cat(labels))

def zero_shot_eval(ecg_enc, text_enc, loader, diseases):
    ecg_enc.eval(); text_enc.eval()
    correct = 0
    for x, y in loader:
        with torch.no_grad():
            e_emb = ecg_enc(x)
        prompts = [f'this ECG indicates {d}' for d in diseases]
        t_embs = text_enc(prompts)
        sim = cosine_similarity(e_emb, t_embs)
        if diseases[sim.argmax(dim=1)] == y: correct += 1
    return correct / len(loader)
```

## Common pitfalls

- The evaluation strictly uses the official train/val/test splits provided by the dataset creators, not random splits, which significantly affects comparability with other works.
- Zero-shot classification relies on a fixed prompt template and cosine similarity, which may not generalize well to diseases with highly variable clinical descriptions.
- Linear evaluation freezes the pre-trained ECG encoder entirely, meaning performance reflects representation quality rather than full fine-tuning capability.

## Evidence (verbatim from paper)

> To do this, we keep the pre-trained ECG encoder fixed and only update a linear classifier that is initialized randomly. This evaluation methodology is applied to two large-scale public ECG datasets with disease-level annotation, PTB-XL and CPSC2018, using Area Under the Curve (AUC) score and F1-score as the primary metrics for performance assessment.

## Citation

```bibtex
@misc{liu2023etp,
  title={ETP: Learning Transferable ECG Representations via ECG-Text Pre-training},
  author={Che Liu et al. (2023)},
  year={2023},
  note={arXiv:2309.07145}
}
```

- arXiv: 2309.07145