Clothing1mpp Eval

---
name: clothing1mpp-eval
description: Evaluates the robustness of image classification models when trained on datasets with inherent label noise and class imbalance. It probes how effectively learning algorithms can filter mislabeled samples and adapt to skewed class distributions without manual curation. Use when the user wants to benchmark on Clothing1mPP, or asks about evaluating this task. Reports Noise Rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.11338
  bibtex_key: liu2024automaticdatasetconstruction
  confidence: high
---

# clothing1mpp-eval

> Automatic Dataset Construction (ADC): Sample Collection, Data Curation, and Beyond — Liu et al. (2024) (arXiv:2408.11338, 2024)

## What this evaluates

Evaluates the robustness of image classification models when trained on datasets with inherent label noise and class imbalance. It probes how effectively learning algorithms can filter mislabeled samples and adapt to skewed class distributions without manual curation.

## Datasets

- **Clothing1mPP** — total 1000000; splits: train (-1), val (-1), test (-1)

## Metrics

- `Noise Rate` **(primary)** — range: percent
  - Percentage of samples in a subset where human annotators disagree with the original label or express uncertainty, aggregated via majority vote or unanimous agreement.
- `Top-1 Accuracy` — range: [0, 1]
  - Standard image classification metric: the proportion of test samples where the model's predicted class matches the ground truth label.

## Input / output format

**Input**: RGB images resized to 256x256 pixels with corresponding class labels.

**Output**: Predicted class label or class probability distribution.

## Scoring recipe

```python
def compute_noise_rate(votes):
    # votes: list of 3 annotator responses per sample
    clean = sum(1 for v in votes if v == ['Yes', 'Yes', 'Yes'])
    return (1 - clean / len(votes)) * 100

def compute_top1_accuracy(preds, labels):
    correct = (preds == labels).sum().item()
    return correct / len(labels)
```

## Common pitfalls

- Confounding label noise with class imbalance: The authors explicitly filter data using Docta and CE to disentangle these two issues before imbalance learning experiments.
- Noise evaluation aggregation: Using majority vote yields 22.15% noise rate, but requiring unanimous agreement retains only 61.25% of samples, significantly affecting downstream learning results.
- Tiny dataset size: The 'tiny' subset used for repeated experiments contains only 50 samples, which may not generalize to full-scale performance.

## Evidence (verbatim from paper)

> For the label noise evaluation task, we utilized a subset of 20,000 samples from the Clothing-ADC dataset, collecting three votes from unique workers for each sample... Using a simple majority vote aggregation, we found that the noise rate in our dataset is 22.15%. However, if a higher level of certainty is required for clean labels, we can apply a more stringent aggregation method, considering more samples as mislabeled.

## Citation

```bibtex
@misc{liu2024automaticdatasetconstruction,
  title={Automatic Dataset Construction (ADC): Sample Collection, Data Curation, and Beyond},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2408.11338}
}
```

- arXiv: 2408.11338