Lnmbench Eval

---
name: lnmbench-eval
description: Evaluates the robustness of label noise learning (LNL) methods on medical image classification tasks. It probes model performance under varying noise types (symmetric, instance-dependent, real-world), noise ratios, and class imbalance distributions across multiple imaging modalities. Use when the user wants to benchmark on PathMNIST, DermaMNIST, BloodMNIST, OrganCMNIST, DRTiD, Kaggle DR+, CheXpert, or asks about evaluating this task. Reports average classification accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.09315
  bibtex_key: ma2025lnmbench
  confidence: high
---

# lnmbench-eval

> Benchmarking Real-World Medical Image Classification with Noisy Labels: Challenges, Practice, and Outlook — Ma et al. (2025) (arXiv:2512.09315, 2025)

## What this evaluates

Evaluates the robustness of label noise learning (LNL) methods on medical image classification tasks. It probes model performance under varying noise types (symmetric, instance-dependent, real-world), noise ratios, and class imbalance distributions across multiple imaging modalities.

## Datasets

- **PathMNIST** — total 107180; splits: train (89996), val (10004), test (7180)
- **DermaMNIST** — total 10015; splits: train (7008), val (1002), test (2005)
- **BloodMNIST** — total 17092; splits: train (11959), val (1712), test (3421)
- **OrganCMNIST** — total 23660; splits: train (13000), val (2392), test (8268)
- **DRTiD** — total 3100; splits: train (1600), val (400), test (1100)
- **Kaggle DR+** — total 88702; splits: train (-1), val (-1), test (-1)
- **CheXpert** — total 224316; splits: train (-1), val (-1), test (-1)

## Metrics

- `average classification accuracy` **(primary)** — range: percent
  - Computed as the mean classification accuracy over the last 5 training epochs for each method. Accuracy is calculated as the proportion of correctly predicted labels on the test set.

## Input / output format

**Input**: RGB images resized to 224×224 (or 512×512 for DRTiD and Kaggle DR+) paired with noisy class labels.

**Output**: Predicted class label for each image.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

# Evaluated over the last 5 training epochs
epoch_accuracies = [compute_accuracy(preds_ep, test_labels) for ep in range(last_5_epochs)]
final_metric = sum(epoch_accuracies) / len(epoch_accuracies)
```

## Common pitfalls

- Methods often overfit noisy labels in later training stages, causing a substantial discrepancy between validation and loss accuracies.
- Transition matrix estimation methods perform well on symmetric noise but collapse under instance-dependent noise due to their design assumptions.
- Real-world datasets use different annotation protocols (e.g., expert consensus vs. automated extraction), making direct cross-dataset comparison of noise levels difficult.

## Evidence (verbatim from paper)

> To obtain a stable and fair comparison, we report the average classification accuracy over the last 5 epochs for each method under three noise patterns: symmetric noise, instance-dependent noise, and real-world noise.

## Citation

```bibtex
@misc{ma2025lnmbench,
  title={Benchmarking Real-World Medical Image Classification with Noisy Labels: Challenges, Practice, and Outlook},
  author={Ma et al. (2025)},
  year={2025},
  note={arXiv:2512.09315}
}
```

- arXiv: 2512.09315