Covid Blues Eval

---
name: covid-blues-eval
description: This benchmark evaluates AI models for detecting COVID-19 infection and assessing lung severity using lung ultrasound videos, clinical variables, and blood count data. It measures how well zero-shot and fine-tuned models generalize to real-world, heterogeneous clinical data compared to human annotators and tabular baselines. Use when the user wants to benchmark on COVID-BLUeS, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.10556
  bibtex_key: wiedemann2025covidblues
  confidence: high
---

# covid-blues-eval

> COVID-BLUeS -- A Prospective Study on the Value of AI in Lung Ultrasound Analysis — Wiedemann et al. (2025) (arXiv:2509.10556, 2025)

## What this evaluates

This benchmark evaluates AI models for detecting COVID-19 infection and assessing lung severity using lung ultrasound videos, clinical variables, and blood count data. It measures how well zero-shot and fine-tuned models generalize to real-world, heterogeneous clinical data compared to human annotators and tabular baselines.

## Datasets

- **COVID-BLUeS** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/NinaWie/COVID-BLUES

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly classified patients (TP+TN)/Total. Reported alongside sensitivity, specificity, and F1-score.
- `sensitivity` — range: [0, 1]
  - True positive rate: TP/(TP+FN).
- `specificity` — range: [0, 1]
  - True negative rate: TN/(TN+FN).
- `F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall: 2*(Precision*Recall)/(Precision+Recall).
- `MAE` — range: other
  - Mean Absolute Error between predicted and actual severity scores.

## Input / output format

**Input**: Lung ultrasound video frames (processed frame-by-frame or via segmentation/localization), optionally concatenated with 55 clinical variables and 11 complete blood count (CBC) features.

**Output**: Binary patient-level classification (COVID-19 positive/negative) or ordinal severity score.

## Scoring recipe

```python
# For binary COVID-19 detection (patient-level)
patient_preds = aggregate_frame_predictions(frame_outputs, aggregation='model_output')
accuracy = sum(pred == gold for pred, gold in zip(patient_preds, gold_labels)) / len(gold_labels)
sensitivity = TP / (TP + FN)
specificity = TN / (TN + FN)
f1 = 2 * precision * recall / (precision + recall)
# For severity scoring
mae = mean(abs(pred_score - true_score))
rmse = sqrt(mean((pred_score - true_score)**2))
```

## Common pitfalls

- Frame-level processing ignores video-level temporal structure, leading to suboptimal performance compared to segmentation-based aggregation.
- Prior studies often use biased datasets with manually selected clear-pathology frames, inflating reported zero-shot accuracy.
- Clinical variables alone (e.g., CBC data) can outperform image-only AI models, making image-only benchmarks clinically misleading.

## Evidence (verbatim from paper)

> The best performance was achieved using the ICLUS segmentation model combined with logistic regression, yielding an accuracy of 79%. However, even this model falls significantly short of the higher accuracies reported in related work. These findings highlight the substantial challenges in accurately classifying the COVID-BLUeS dataset, regardless of the method—whether through human assessment, zero-shot application of AI models, CNN training, or pattern detection techniques.

## Citation

```bibtex
@misc{wiedemann2025covidblues,
  title={COVID-BLUeS -- A Prospective Study on the Value of AI in Lung Ultrasound Analysis},
  author={Wiedemann et al. (2025)},
  year={2025},
  note={arXiv:2509.10556}
}
```

- arXiv: 2509.10556