Russe2018 Wsi Eval

---
name: russe2018-wsi-eval
description: Evaluates a model's ability to perform Word Sense Induction (WSI) by clustering contextual usages of target words into sense groups without prior sense labels. It probes the system's capability to handle morphological complexity and free word order in Russian across different sense granularities. Use when the user wants to benchmark on wiki-wiki, bts-rnc, active-dict, or asks about evaluating this task. Reports ARI.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1803.05795
  bibtex_key: panchenko2018russe
  confidence: high
---

# russe2018-wsi-eval

> RUSSE'2018: A Shared Task on Word Sense Induction for the Russian Language — Panchenko et al. (2018) (arXiv:1803.05795, 2018)

## What this evaluates

Evaluates a model's ability to perform Word Sense Induction (WSI) by clustering contextual usages of target words into sense groups without prior sense labels. It probes the system's capability to handle morphological complexity and free word order in Russian across different sense granularities.

## Datasets

- **wiki-wiki** — total ?; splits: test (-1); repo https://github.com/nlpub/russe-wsi-kit
- **bts-rnc** — total ?; splits: test (-1); repo https://github.com/nlpub/russe-wsi-kit
- **active-dict** — total ?; splits: test (-1); repo https://github.com/nlpub/russe-wsi-kit

## Metrics

- `ARI` **(primary)** — range: [-1, 1]
  - Adjusted Rand Index measuring the similarity between the predicted sense clusters and the gold sense labels. Values range from -1 to 1, with 1 indicating perfect agreement between partitions.

## Input / output format

**Input**: Contextual sentences or phrases containing a target word.

**Output**: A partitioning of the contexts into clusters, where each cluster corresponds to a distinct sense of the target word.

## Scoring recipe

```python
def compute_ari(predictions, gold, words):
    ari_scores = []
    for word in unique_words:
        pred_w = [p for p, w in zip(predictions, words) if w == word]
        gold_w = [g for g, w in zip(gold, words) if w == word]
        ari_scores.append(adjusted_rand_score(gold_w, pred_w))
    return sum(ari_scores) / len(ari_scores)
```

## Common pitfalls

- Public and private test sets are non-overlapping; public covers ~1/3 of test words and is used for the leaderboard, while private determines final ranking.
- ARI scores vary drastically across datasets due to sense granularity; wiki-wiki contains coarse homonymous senses yielding high scores, while dictionary datasets contain fine-grained polysemous senses yielding lower scores.
- The task is unsupervised sense clustering; models must not rely on prior sense labels or external sense inventories.

## Evidence (verbatim from paper)

> Tables 2, 3, and 4 present the results of the shared task for the three datasets used for evaluation: wiki-wiki, bts-rnc, and active-dict. Each table lists top 10 best teams with the public and private ARI scores on the test set... Private and public scores were calculated on non-overlapping sets of words, with public words constituting approximately one third of all words in test set of each dataset.

## Citation

```bibtex
@misc{panchenko2018russe,
  title={RUSSE'2018: A Shared Task on Word Sense Induction for the Russian Language},
  author={Panchenko et al. (2018)},
  year={2018},
  note={arXiv:1803.05795}
}
```

- arXiv: 1803.05795