Wsd Rp Accuracy Eval

---
name: wsd-rp-accuracy-eval
description: Evaluates a model's ability to disambiguate word senses for both common nouns and proper nouns exhibiting regular polysemy. It probes contextual understanding and the capacity to leverage structured sense glosses and dot-object type classes to select the correct meaning from a candidate inventory. Use when the user wants to benchmark on WSD dataset (CWN 2.0), RP dataset (Revised Mandarin Chinese Dictionary), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.09758
  bibtex_key: hsieh2024resolving
  confidence: high
---

# wsd-rp-accuracy-eval

> Resolving Regular Polysemy in Named Entities — Shu-Kai Hsieh et al. (2024) (arXiv:2401.09758, 2024)

## What this evaluates

Evaluates a model's ability to disambiguate word senses for both common nouns and proper nouns exhibiting regular polysemy. It probes contextual understanding and the capacity to leverage structured sense glosses and dot-object type classes to select the correct meaning from a candidate inventory.

## Datasets

- **WSD dataset (CWN 2.0)** — total 45784; splits: train (36622), test (9162)
- **RP dataset (Revised Mandarin Chinese Dictionary)** — total 4507; splits: train (3641), test (866)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy: the proportion of test sentences where the model's highest-probability predicted sense matches the ground-truth sense.

## Input / output format

**Input**: Context-gloss pairs formatted as [CLS] TEST-SENT [SEP] TGT, SENSE-DEF, SENSE-EX-SENT [SEP] for WSD, or [CLS] TEST-SENT [SEP] TGT, RPCLASS, RPCLASS-GLOSS [SEP] for RP. Target words are enclosed in angular brackets < >.

**Output**: Probability distribution over candidate senses; the model selects the sense with the highest probability as the prediction.

## Scoring recipe

```python
correct = 0
total = 0
for sentence in test_set:
    candidates = get_candidates(sentence.target_word)
    probs = model.predict(sentence.context, candidates)
    predicted_sense = candidates[probs.argmax()]
    if predicted_sense == sentence.gold_sense:
        correct += 1
    total += 1
accuracy = correct / total
```

## Common pitfalls

- Evaluation is reported per example (sentence), but training batches use flattened context-gloss sequences, which can cause confusion about batch size vs. example count.
- MFS baseline accuracy varies drastically by subset (0% for simple words, 38% for complex) due to lack of sense distribution skew in the simple subset, which can mislead readers expecting standard corpus statistics.
- POS-guided filtering is a pre-processing step to reduce candidates, not the final metric; the model still predicts among filtered candidates.

## Evidence (verbatim from paper)

> We split the WSD dataset into WSD_train, WSD_test; RP dataset into RP_train, RP_test. The model’s prediction accuracies under different conditions are shown in Figure 2. Overall, the model achieves .86 accuracy in the WSD task and .88 in the RP task respectively.

## Citation

```bibtex
@misc{hsieh2024resolving,
  title={Resolving Regular Polysemy in Named Entities},
  author={Shu-Kai Hsieh et al. (2024)},
  year={2024},
  note={arXiv:2401.09758}
}
```

- arXiv: 2401.09758