Lscdiscovery Eval

---
name: lscdiscovery-eval
description: Evaluates models' ability to detect and rank lexical semantic change in Spanish diachronic corpora. It probes both graded ranking of semantic shift magnitude and binary classification of sense gain/loss or change presence. Use when the user wants to benchmark on LSCDiscovery, or asks about evaluating this task. Reports Spearman rank correlation (SPR), F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.06691
  bibtex_key: zamorareina2022lscdiscovery
  confidence: high
---

# lscdiscovery-eval

> LSCDiscovery: A shared task on semantic change discovery and detection in Spanish — Zamora-Reina et al. (2022) (arXiv:2205.06691, 2022)

## What this evaluates

Evaluates models' ability to detect and rank lexical semantic change in Spanish diachronic corpora. It probes both graded ranking of semantic shift magnitude and binary classification of sense gain/loss or change presence.

## Datasets

- **LSCDiscovery** — total ?; splits: test (-1)

## Metrics

- `Spearman rank correlation (SPR)` **(primary)** — range: [-1, 1]
  - Spearman's rank correlation coefficient ($\rho$) between predicted semantic change scores and gold graded scores. Ranges from -1 to 1, where 1 indicates perfect rank agreement.
- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall for binary classification tasks (Binary Change Detection, Sense Gain, Sense Loss). Calculated as 2 * (P * R) / (P + R).

## Input / output format

**Input**: Diachronic word usage pairs or sequences across time periods for a set of target words.

**Output**: Phase 1: A ranked list or continuous score for each target word indicating semantic shift magnitude. Phase 2: Binary labels (e.g., 0/1 for change/no change, or gain/loss) for each target word.

## Scoring recipe

```python
# Phase 1: Graded Change / COMPARE Discovery
def score_graded(pred_scores, gold_scores):
    return spearmanr(pred_scores, gold_scores).correlation

# Phase 2: Binary Change / Sense Gain / Sense Loss Detection
def score_binary(pred_labels, gold_labels):
    tp = sum(p == g == 1 for p, g in zip(pred_labels, gold_labels))
    fp = sum(p == 1 and g == 0 for p, g in zip(pred_labels, gold_labels))
    fn = sum(p == 0 and g == 1 for p, g in zip(pred_labels, gold_labels))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- Top-performing systems often bypassed modeling the actual annotation procedure (e.g., JSD or COMPARE scores) and instead exploited strong correlations between different scoring metrics, limiting their generalizability.
- Binarizing graded predictions via fixed thresholds (e.g., percentiles) has a hard performance upper bound, as perfectly modeling the underlying continuous score still cannot achieve perfect binary classification.
- Type-based systems consistently underperformed token-based systems in this shared task, contrary to earlier benchmarks, due to biases in contextualized embeddings and aggregation methods.

## Evidence (verbatim from paper)

> Teams are ranked according to SPR score for the Graded Change subtask in decreasing order. The values corresponding to the three best systems are highlighted in bold type. ... Teams are ranked according to F1 score for subtask Change binary in decreasing order.

## Citation

```bibtex
@misc{zamorareina2022lscdiscovery,
  title={LSCDiscovery: A shared task on semantic change discovery and detection in Spanish},
  author={Zamora-Reina et al. (2022)},
  year={2022},
  note={arXiv:2205.06691}
}
```

- arXiv: 2205.06691