Semeval2020 Semantic Change Eval

---
name: semeval2020-semantic-change-eval
description: Evaluates a model's ability to detect and rank lexical semantic change over time across multiple languages. It probes both binary classification of whether a word's meaning has changed and graded ranking of the magnitude of that change. Use when the user wants to benchmark on SemEval 2020 Unsupervised Lexical Semantic Change Detection, or asks about evaluating this task. Reports Spearman's rank correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.02891
  bibtex_key: bachphantat2026transparent
  confidence: high
---

# semeval2020-semantic-change-eval

> Transparent Semantic Change Detection with Dependency-Based Profiles — Bach Phan-Tat et al. (2026) (arXiv:2601.02891, 2026)

## What this evaluates

Evaluates a model's ability to detect and rank lexical semantic change over time across multiple languages. It probes both binary classification of whether a word's meaning has changed and graded ranking of the magnitude of that change.

## Datasets

- **SemEval 2020 Unsupervised Lexical Semantic Change Detection** — total ?; splits: test (-1)

## Metrics

- `accuracy` — range: [0, 1]
  - Proportion of correctly classified words. For Subtask 1, a word is labeled as changed if it falls in the top 43% of predicted JSD scores per language.
- `Spearman's rank correlation` **(primary)** — range: [-1, 1]
  - Pearson correlation coefficient between the ranks of predicted change scores (derived from JSD) and the gold change scores.

## Input / output format

**Input**: Target words with associated slot-fillers and context across two time periods, along with gold change scores or labels.

**Output**: Subtask 2: A ranked list of target words by predicted change magnitude. Subtask 1: A binary label (changed/not changed) for each target word, derived by thresholding the top 43% of predicted scores per language.

## Scoring recipe

```python
# Subtask 2 (Spearman's rank correlation)
import scipy.stats as stats
spearman_corr = stats.spearmanr(gold_ranks, predicted_ranks).correlation

# Subtask 1 (Accuracy)
top_k = int(len(words) * 0.43)
predicted_labels = [1 if rank <= top_k else 0 for rank in predicted_ranks]
accuracy = sum(p == g for p, g in zip(predicted_labels, gold_labels)) / len(words)
```

## Common pitfalls

- Using a fixed global threshold for Subtask 1 instead of the per-language top 43% cutoff specified in the shared task.
- Ignoring frequency filtering and POS tag removal, which the authors show substantially impact JSD scores and final rankings.
- Assuming dynamic programming is required for thresholding, as the paper notes a hard-coded 43% cutoff performs comparably.

## Evidence (verbatim from paper)

> Subtask 1 is a binary classification task and is evaluated based on accuracy. Subtask 2 is a ranking task and is evaluated with Spearman's rank correlation. The main focus of our method would be on subtask 2. We then use our subtask 2 scores for subtask 1 classification, following the strategy of Kutuzov et al. (2021).

## Citation

```bibtex
@misc{bachphantat2026transparent,
  title={Transparent Semantic Change Detection with Dependency-Based Profiles},
  author={Bach Phan-Tat et al. (2026)},
  year={2026},
  note={arXiv:2601.02891}
}
```

- arXiv: 2601.02891