Wsi Semcor Eval

---
name: wsi-semcor-eval
description: Evaluates Word Sense Induction (WSI) by clustering contextualized word embeddings to predict sense assignments. It probes a model's ability to capture lexical polysemy and contextual meaning without supervised sense labels, using natural corpus distributions rather than artificially skewed benchmarks. Use when the user wants to benchmark on SemCor, or asks about evaluating this task. Reports F-B^3.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.11686
  bibtex_key: mosolova2026llmwsiremainsunsolved
  confidence: high
---

# wsi-semcor-eval

> In the LLM era, Word Sense Induction remains unsolved — Mosolova et al. (2026) (arXiv:2603.11686, 2026)

## What this evaluates

Evaluates Word Sense Induction (WSI) by clustering contextualized word embeddings to predict sense assignments. It probes a model's ability to capture lexical polysemy and contextual meaning without supervised sense labels, using natural corpus distributions rather than artificially skewed benchmarks.

## Datasets

- **SemCor** — total ?; splits: test (-1)

## Metrics

- `F-B^3` **(primary)** — range: percent
  - A clustering metric that computes the harmonic mean of precision and recall by optimally matching induced clusters to gold sense assignments. It is calculated as F-B^3 = 2 * (precision * recall) / (precision + recall), typically scaled to a percentage.

## Input / output format

**Input**: Contextualized embeddings of target words in their surrounding context, extracted from pre-trained or fine-tuned language models (e.g., BERT, MirrorWiC).

**Output**: A partitioning of the contextualized embeddings into clusters, where each cluster represents an induced word sense.

## Scoring recipe

```python
def compute_fb3(predictions, gold):
    # predictions: list of cluster IDs per instance
    # gold: list of gold sense labels per instance
    # 1. Build confusion matrix between clusters and gold senses
    # 2. Find optimal one-to-one matching (e.g., Hungarian algorithm)
    # 3. Compute precision = sum(matches) / total_clusters
    # 4. Compute recall = sum(matches) / total_senses
    # 5. Return F1 = 2 * (prec * rec) / (prec + rec) * 100
```

## Common pitfalls

- Relying on artificial datasets like SemEval that distort natural polysemy and frequency distributions.
- Assuming complex clustering algorithms will outperform the simple 'one cluster per lemma' (1cpl) baseline, which actually performs best for adjectives and nouns.
- Ignoring part-of-speech variations, as polysemy rates and optimal clustering strategies differ significantly across verbs, nouns, and adjectives.

## Evidence (verbatim from paper)

> Table 3: F-B $^3$ performance across PLMs and clustering algorithms for each POS, for all POS (All POS), and the average over POS weighted by POS proportion in SemCor ( $w$ Avg).

## Citation

```bibtex
@misc{mosolova2026llmwsiremainsunsolved,
  title={In the LLM era, Word Sense Induction remains unsolved},
  author={Mosolova et al. (2026)},
  year={2026},
  note={arXiv:2603.11686}
}
```

- arXiv: 2603.11686