Fashion Compatibility Eval

---
name: fashion-compatibility-eval
description: Evaluates a model's ability to score the visual-semantic compatibility of items within a complete outfit, and to recommend a missing item that best completes a partial outfit. It probes the model's capacity to learn non-transitive, type-aware relationships across different fashion categories. Use when the user wants to benchmark on Maryland Polyvore, Polyvore Outfits, Polyvore Outfits-D, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1803.09196
  bibtex_key: vasileva2018learning
  confidence: high
---

# fashion-compatibility-eval

> Learning Type-Aware Embeddings for Fashion Compatibility — Vasileva et al. (2018) (arXiv:1803.09196, 2018)

## What this evaluates

Evaluates a model's ability to score the visual-semantic compatibility of items within a complete outfit, and to recommend a missing item that best completes a partial outfit. It probes the model's capacity to learn non-transitive, type-aware relationships across different fashion categories.

## Datasets

- **Maryland Polyvore** — total ?; splits: train (17316), test (3076), validation (1407)
- **Polyvore Outfits** — total ?; splits: train (-1), test (-1), validation (-1)
- **Polyvore Outfits-D** — total ?; splits: train (-1), test (-1), validation (-1)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area under the receiver operating characteristic curve. Computed over predicted compatibility scores versus ground truth labels for compatible vs. incompatible outfits.
- `accuracy` — range: [0, 1]
  - Fraction of correctly identified fill-in-the-blank items out of the total number of questions. Evaluated over 4 candidate items per question.

## Input / output format

**Input**: Outfit represented as a set of item embeddings (visual ResNet-18 features and text HGLMM Fisher vectors). For FITB: an outfit missing one item type, plus four candidate embeddings of that type.

**Output**: Compatibility task: scalar compatibility score. FITB task: index of the single most compatible candidate item.

## Scoring recipe

```python
def score_compatibility(y_true, y_pred):
    return roc_auc_score(y_true, y_pred)

def score_fitb(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return correct / len(golds)
```

## Common pitfalls

- Random negative sampling in the original dataset creates invalid outfits (e.g., outfits with only one item type or mismatched categories), making evaluation trivial without category-aware sampling.
- Using unrestricted test negatives without composition filtering artificially inflates performance, as models can dismiss invalid compositions without learning true compatibility.

## Evidence (verbatim from paper)

> In the fashion compatibility task, a candidate outfit is scored as to whether its constitute items are compatible with each other. Performance is evaluated using the average under a receiver operating characteristic curve (AUC). The second task is to select from a set of candidate items (four in this case) in a fill-in-the-blank (FITB) fashion recommendation experiment. The goal is to select the most compatible item with the remainder of the outfit, and performance is evaluated by accuracy on the answered questions.

## Citation

```bibtex
@misc{vasileva2018learning,
  title={Learning Type-Aware Embeddings for Fashion Compatibility},
  author={Vasileva et al. (2018)},
  year={2018},
  note={arXiv:1803.09196}
}
```

- arXiv: 1803.09196