Fashionpedia Eval

---
name: fashionpedia-eval
description: Evaluates joint instance segmentation and fine-grained attribute localization on fashion apparel. It measures how well a model can detect objects, segment them accurately, and correctly assign multiple localized attributes to each instance. Use when the user wants to benchmark on Fashionpedia, or asks about evaluating this task. Reports AP_{IoU + F_1}.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2004.12276
  bibtex_key: [REDACTED]
  confidence: high
---

# fashionpedia-eval

> Fashionpedia: Ontology, Segmentation, and an Attribute Localization Dataset — Jia et al. (2020) (arXiv:2004.12276, 2020)

## What this evaluates

Evaluates joint instance segmentation and fine-grained attribute localization on fashion apparel. It measures how well a model can detect objects, segment them accurately, and correctly assign multiple localized attributes to each instance.

## Datasets

- **Fashionpedia** — total 48825; splits: train (-1), val (-1), test (-1); repo https://fashionpedia.github.io/home/Model_and_API.html

## Metrics

- `AP_{IoU + F_1}` **(primary)** — range: percent
  - Average Precision averaged across 10 IoU thresholds (0.5 to 0.95) and 10 macro F1 thresholds (0.5 to 0.95). A detection is a true positive only if its bounding box IoU exceeds the IoU threshold AND the macro F1 score of its predicted attributes exceeds the F1 threshold.
- `AP_{IoU}` — range: percent
  - Standard COCO Average Precision averaged across 10 IoU thresholds (0.5 to 0.95) and all categories, ignoring attribute constraints.

## Input / output format

**Input**: RGB images resized to 1024px (or 1280px for SpineNet-143) containing fashion apparel. Ground truth provides per-instance bounding boxes, segmentation masks, and multi-label attribute annotations.

**Output**: Per-instance predictions consisting of bounding boxes, binary segmentation masks, and a multi-label vector of predicted attributes.

## Scoring recipe

```python
aps = []
for cat in categories:
    for tau_iou in np.arange(0.5, 1.0, 0.05):
        for tau_f1 in np.arange(0.5, 1.0, 0.05):
            tps = 0
            for pred, gt in matches(preds[cat], gts[cat]):
                if pred.iou(gt) >= tau_iou and pred.macro_f1(gt) >= tau_f1:
                    tps += 1
            aps.append(compute_ap(tps, len(gts[cat])))
return np.mean(aps)
```

## Common pitfalls

- Joint segmentation and attribute localization causes a significant performance drop compared to segmentation-only baselines due to the strict F1 constraint.
- Standard COCO detectors fail on Fashionpedia due to domain shift, showing high background confusion and localization errors.
- Macro F1 thresholding requires precise mask alignment; coarse masks artificially lower attribute F1 scores.

## Evidence (verbatim from paper)

> In the case of instance segmentation and attribute localization, we extend standard COCO metric by adding one more constraint: the macro F1 score for predicted attributes of single detected object with category c (see supplementary material for the average choice of f1-score). We denote the F1 threshold as τF1, and it has the same range as τIoU (τF1 ∈ [0.5 : 0.05 : 0.95]). The main metric AP_{IoU + F1} reports averaged precision score across all 10 IoU thresholds, all 10 macro F1 scores, and all the categories.

## Citation

```bibtex
@misc{jia2020fashionpedia,
  title={Fashionpedia: Ontology, Segmentation, and an Attribute Localization Dataset},
  author={Jia et al. (2020)},
  year={2020},
  note={arXiv:2004.12276}
}
```

- arXiv: 2004.12276