Clinical Turing Test Eval

---
name: clinical-turing-test-eval
description: Evaluates the physiological realism and clinical fidelity of synthetic 12-lead ECGs by measuring how often expert clinicians can correctly distinguish them from real clinical recordings, and how accurately they can diagnose specific pathologies in both synthetic and real signals. Use when the user wants to benchmark on MedalCare-XL, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2211.15997
  bibtex_key: gillette2022medalcarexl
  confidence: high
---

# clinical-turing-test-eval

> MedalCare-XL: 16,900 healthy and pathological 12 lead ECGs obtained through electrophysiological simulations — Gillette et al. (2022) (arXiv:2211.15997, 2022)

## What this evaluates

Evaluates the physiological realism and clinical fidelity of synthetic 12-lead ECGs by measuring how often expert clinicians can correctly distinguish them from real clinical recordings, and how accurately they can diagnose specific pathologies in both synthetic and real signals.

## Datasets

- **MedalCare-XL** — total 16900; splits: test (800)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Accuracy = (Number of correctly classified signals) / (Total number of signals evaluated). For pathological cases, it also tracks diagnosis accuracy per pathology class.

## Input / output format

**Input**: 12-lead ECG time-series signals (both synthetic and real clinical recordings) presented without labels.

**Output**: Clinician's binary classification (synthetic vs. real/clinical) and, for pathological cases, a specific pathology diagnosis.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold) * 100
```

## Common pitfalls

- Synthetic signals often lack physiological noise, causing clinicians to misclassify them as real.
- Morphological artifacts like fractionation, improper R-wave propagation, or spiking T-waves lead to false synthetic classifications.
- Pathology misdiagnosis is frequent, e.g., LAO, 1AVB, and MI often mistaken for normal sinus rhythm, while LBBB/RBBB mistaken for MI.

## Evidence (verbatim from paper)

> The six clinicians correctly classified 464 of the 600 cases, which corresponds to an accuracy of 77.33%. On the other side, 136 signals (22.67%) could not be correctly classified, including 62 (10.34%) synthetic and 74 (12.33%) measured ECGs... The two clinicians correctly classified the signals as either measured or clinical in 166 of the 200 cases, which corresponds to an overall accuracy of 83%.

## Citation

```bibtex
@misc{gillette2022medalcarexl,
  title={MedalCare-XL: 16,900 healthy and pathological 12 lead ECGs obtained through electrophysiological simulations},
  author={Gillette et al. (2022)},
  year={2022},
  note={arXiv:2211.15997}
}
```

- arXiv: 2211.15997