Train O Matic Wsd Eval

---
name: train-o-matic-wsd-eval
description: Evaluates the quality of automatically generated multilingual word sense disambiguation (WSD) training corpora by training a supervised WSD system (IMS) on them and measuring performance on standard WSD benchmark datasets. It probes whether synthetic sense-annotated data can match or exceed manually annotated corpora, particularly for low-resource languages. Use when the user wants to benchmark on Senseval-2, Senseval-3, SemEval-2007, SemEval-2013, SemEval-2015, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1805.04685
  bibtex_key: pasini2018huge
  confidence: high
---

# train-o-matic-wsd-eval

> Huge Automatically Extracted Training Sets for Multilingual Word Sense Disambiguation — Pasini et al. (2018) (arXiv:1805.04685, 2018)

## What this evaluates

Evaluates the quality of automatically generated multilingual word sense disambiguation (WSD) training corpora by training a supervised WSD system (IMS) on them and measuring performance on standard WSD benchmark datasets. It probes whether synthetic sense-annotated data can match or exceed manually annotated corpora, particularly for low-resource languages.

## Datasets

- **Senseval-2, Senseval-3, SemEval-2007, SemEval-2013, SemEval-2015** — total ?; splits: test (-1)

## Metrics

- `F1` **(primary)** — range: percent
  - Harmonic mean of precision and recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). Computed over predicted vs. gold word sense IDs per instance using the unified WSD evaluation framework.

## Input / output format

**Input**: Target word, surrounding context sentence, and candidate word senses from a lexical resource (WordNet or BabelNet).

**Output**: Predicted sense ID for the target word.

## Scoring recipe

```python
def compute_f1(pred_senses, gold_senses):
    tp = sum(1 for p, g in zip(pred_senses, gold_senses) if p == g)
    fp = sum(1 for p, g in zip(pred_senses, gold_senses) if p != g)
    fn = sum(1 for p, g in zip(pred_senses, gold_senses) if p == g and g != 'no-sense')
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return f1 * 100
```

## Common pitfalls

- Parameter tuning (K=500, z=2.0) is performed on a small in-house development set rather than a held-out test set, which may inflate reported F1 scores.
- Comparisons against the MFS (Most Frequent Sense) baseline are included, which can mask improvements on rare or context-dependent senses.
- Different lexical resources (WordNet vs. BabelNet) and sense inventories are used across languages, making cross-lingual F1 scores not directly comparable.

## Evidence (verbatim from paper)

> The evaluation has been performed using the unified evaluation framework for Word Sense Disambiguation made available by Raganato et al. (2017), thus considering the following WSD shared tasks: Senseval-2 (Edmonds and Cotton, 2001), Senseval-3 (Snyder and Palmer, 2004), SemEval-2007 (Navigli et al., 2007), SemEval-2013 (Navigli et al., 2013) and SemEval-2015 (Moro and Navigli, 2015). Table 4: F1 of IMS trained on Train-o-Matic, OMSTI and SemCor, and MFS for the Senseval-2, Senseval-3, SemEval-07, SemEval-13 and SemEval-15 datasets.

## Citation

```bibtex
@misc{pasini2018huge,
  title={Huge Automatically Extracted Training Sets for Multilingual Word Sense Disambiguation},
  author={Pasini et al. (2018)},
  year={2018},
  note={arXiv:1805.04685}
}
```

- arXiv: 1805.04685