Signal Quality Auditing Eval

---
name: signal-quality-auditing-eval
description: Evaluates the effectiveness of various Signal Quality Indices (SQIs) and machine learning models for classifying noisy vs. clean ECG signals, detecting outliers, and denoising time-series data. Use when the user wants to benchmark on Physionet 2011 ECG Challenge (PICC) Set A, MIT-BIH Arrhythmia & NSTDB, or asks about evaluating this task. Reports AUC, Mean Squared Error (MSE).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.00803
  bibtex_key: gao2024signal
  confidence: high
---

# signal-quality-auditing-eval

> Signal Quality Auditing for Time-series Data — Gao et al. (2024) (arXiv:2402.00803, 2024)

## What this evaluates

Evaluates the effectiveness of various Signal Quality Indices (SQIs) and machine learning models for classifying noisy vs. clean ECG signals, detecting outliers, and denoising time-series data.

## Datasets

- **Physionet 2011 ECG Challenge (PICC) Set A** — total 1996; splits: train (998), test (998)
- **MIT-BIH Arrhythmia & NSTDB** — total 18816; splits: train (9408), test (9408)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
- `Accuracy` — range: [0, 1]
  - Fraction of correctly classified samples out of the total number of samples.
- `Mean Squared Error (MSE)` **(primary)** — range: other
  - Average of the squared differences between the predicted denoised ECG values and the actual clean ECG ground truth values.

## Input / output format

**Input**: 10-second 12-lead ECG recordings sampled at 500 Hz, or 512-sample windows of ECG signals.

**Output**: Binary label (0=not noisy, 1=noisy) for classification/outlier tasks; continuous denoised ECG waveform for denoising task.

## Scoring recipe

```python
def evaluate(y_true, y_pred, y_clean=None):
    acc = np.mean(y_true == y_pred)
    fpr, tpr, _ = roc_curve(y_true, y_pred)
    auc = np.trapz(tpr, fpr)
    mse = np.mean((y_clean - y_pred) ** 2) if y_clean is not None else None
    return {'accuracy': acc, 'AUC': auc, 'MSE': mse}
```

## Common pitfalls

- Data leakage across subjects if not using subject-based 5-fold cross-validation.
- Using PhysioNet Set B instead of Set A, which breaks reproducibility with the reported results.
- Comparing denoising MSE across different noise types/SNR levels without normalizing or specifying the exact noise sources (EM, BW, MA, GN).

## Evidence (verbatim from paper)

> We used 5-fold group split cross-validation on the subjects in Set-A of the dataset, so subject data do not overlap in training and testing in any iteration. That yieldED 1996 ECG time-series for training and testing (50% split), each 10 seconds long, recorded at 500 Hz. These raw time-series signals are then featurized via the SQIs mentioned above. This yieldED a straightforward binary classification: 0\=not noisy, 1\=noisy. A Random Forest classifier was further trained on these SQI attributes. Table 3: Table of AUCs and Accuracies (with standard deviations) of 8 methods on subject-based 5-fold cross validation and Random Forest with only single-lead features.

## Citation

```bibtex
@misc{gao2024signal,
  title={Signal Quality Auditing for Time-series Data},
  author={Gao et al. (2024)},
  year={2024},
  note={arXiv:2402.00803}
}
```

- arXiv: 2402.00803