Fed Echo Eval

---
name: fed-echo-eval
description: Evaluates federated learning algorithms on echocardiogram video segmentation tasks under label incompleteness and high data heterogeneity across multiple medical institutions. It probes how FL methods handle missing annotations and conflicting labels from different clinical sites. Use when the user wants to benchmark on Fed-ECHO, or asks about evaluating this task. Reports DICE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.07050
  bibtex_key: zhang2024fedcvd
  confidence: high
---

# fed-echo-eval

> FedCVD: The First Real-World Federated Learning Benchmark on Cardiovascular Disease Data — Zhang et al. (2024) (arXiv:2411.07050, 2024)

## What this evaluates

Evaluates federated learning algorithms on echocardiogram video segmentation tasks under label incompleteness and high data heterogeneity across multiple medical institutions. It probes how FL methods handle missing annotations and conflicting labels from different clinical sites.

## Datasets

- **Fed-ECHO** — total ?; splits: train (-1), test (-1); repo https://github.com/SMILELab-FL/FedCVD

## Metrics

- `DICE` **(primary)** — range: percent
  - Dice similarity coefficient measuring overlap between predicted and ground truth segmentation masks, expressed as a percentage.
- `Hausdorff distance ($d_H$)` — range: other
  - Maximum distance between the boundaries of the predicted and ground truth segmentation masks.

## Input / output format

**Input**: Echocardiogram video frames from a specific medical institution (client).

**Output**: Segmentation masks for cardiac regions (BG, LV_Endo, LV_Epi, LA).

## Scoring recipe

```python
def compute_dice(pred_mask, gt_mask):
    intersection = np.sum(pred_mask & gt_mask)
    union = np.sum(pred_mask | gt_mask)
    return 2 * intersection / union * 100 if union > 0 else 0.0
def compute_hd(pred_mask, gt_mask):
    pred_pts = np.argwhere(pred_mask)
    gt_pts = np.argwhere(gt_mask)
    if len(pred_pts) == 0 or len(gt_pts) == 0: return float('inf')
    dists = cdist(pred_pts, gt_pts)
    return np.max(np.min(dists, axis=1)) + np.max(np.min(dists, axis=0))
```

## Common pitfalls

- Label incompleteness means some institutions only annotate specific regions; 'Maybe-BG' labels from other institutions must be masked during loss calculation and evaluation.
- Evaluating on global test set without masking unreliable labels inflates error metrics.
- Semi-supervised baselines require extra rounds for labeled client training before FL begins.

## Evidence (verbatim from paper)

> Table 4: The performance of different FL methods on Fed-ECHO, with DICE (%) and $d_{H}$ representing DICE index and Hausdorff distance respectively.

## Citation

```bibtex
@misc{zhang2024fedcvd,
  title={FedCVD: The First Real-World Federated Learning Benchmark on Cardiovascular Disease Data},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2411.07050}
}
```

- arXiv: 2411.07050