Fashion Ner El Eval

---
name: fashion-ner-el-eval
description: Evaluates a BERT-based Named Entity Recognition pipeline and a binary classifier for candidate entity disambiguation on fashion product descriptions. It probes the model's ability to extract attribute mentions (e.g., material, color) and correctly link them to a knowledge graph ontology under severe data scarcity. Use when the user wants to benchmark on Fashion Product Descriptions (In-house), Fashion EL Disambiguation Dataset, or asks about evaluating this task. Reports f1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2206.01087
  bibtex_key: [REDACTED]
  confidence: high
---

# fashion-ner-el-eval

> Enriching a Fashion Knowledge Graph from Product Textual Descriptions — Barroca et al. (2022) (arXiv:2206.01087, 2022)

## What this evaluates

Evaluates a BERT-based Named Entity Recognition pipeline and a binary classifier for candidate entity disambiguation on fashion product descriptions. It probes the model's ability to extract attribute mentions (e.g., material, color) and correctly link them to a knowledge graph ontology under severe data scarcity.

## Datasets

- **Fashion Product Descriptions (In-house)** — total 900; splits: train (720), val (180)
- **Fashion EL Disambiguation Dataset** — total 2528; splits: train (-1), test (-1)

## Metrics

- `f1-score` **(primary)** — range: [0, 1]
  - Standard binary classification F1-score, calculated as the harmonic mean of precision and recall. Accuracy is explicitly excluded due to severe class imbalance (7.2% positive samples).

## Input / output format

**Input**: NER: Raw product textual description. EL: Product textual description, attribute mention span, and candidate entity string.

**Output**: NER: Token-level entity type tags (e.g., BIO format). EL: Binary label (1 for correct link, 0 for incorrect).

## Scoring recipe

```python
def compute_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
```

## Common pitfalls

- Accuracy is explicitly not reported due to high class imbalance (7.2% positive), so readers should not use or expect accuracy as the evaluation metric.
- The EL dataset is highly synthetic, generated from only 32 manually annotated instances via fuzzy matching, which may limit generalization to out-of-distribution product queries.

## Evidence (verbatim from paper)

> As metrics, we used the standard metrics for binary classification: precision, recall and f1-score. We do not report the accuracy score due to the high unbalanced nature of the dataset.

## Citation

```bibtex
@misc{barroca2022enriching,
  title={Enriching a Fashion Knowledge Graph from Product Textual Descriptions},
  author={Barroca et al. (2022)},
  year={2022},
  note={arXiv:2206.01087}
}
```

- arXiv: 2206.01087