Word Embedding Eval

---
name: word-embedding-eval
description: Evaluates the quality of multilingual word embeddings by measuring semantic similarity/relatedness and word categorization accuracy. It probes whether visual grounding improves cross-lingual semantic alignment and clustering of basic-level concepts. Use when the user wants to benchmark on WordSim353, MEN, RW, MTurk, simVerb, SimLex999, Battig, AP, BLESS, ESSLLI-a, ESSLLI-b, ESSLLI-c, Almarsoomi, MC30, Saif40, WordSim, or asks about evaluating this task. Reports Spearman correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2209.03714
  bibtex_key: mohammed2022visualgrounding
  confidence: high
---

# word-embedding-eval

> Visual Grounding of Inter-lingual Word-Embeddings — Mohammed et al. (2022) (arXiv:2209.03714, 2022)

## What this evaluates

Evaluates the quality of multilingual word embeddings by measuring semantic similarity/relatedness and word categorization accuracy. It probes whether visual grounding improves cross-lingual semantic alignment and clustering of basic-level concepts.

## Datasets

- **WordSim353** — total ?; splits: test (-1)
- **MEN** — total ?; splits: test (-1)
- **RW** — total ?; splits: test (-1)
- **MTurk** — total ?; splits: test (-1)
- **simVerb** — total ?; splits: test (-1)
- **SimLex999** — total ?; splits: test (-1)
- **Battig** — total ?; splits: test (-1)
- **AP** — total ?; splits: test (-1)
- **BLESS** — total ?; splits: test (-1)
- **ESSLLI-a** — total ?; splits: test (-1)
- **ESSLLI-b** — total ?; splits: test (-1)
- **ESSLLI-c** — total ?; splits: test (-1)
- **Almarsoomi** — total ?; splits: test (-1)
- **MC30** — total ?; splits: test (-1)
- **Saif40** — total ?; splits: test (-1)
- **WordSim** — total ?; splits: test (-1)

## Metrics

- `Spearman correlation` **(primary)** — range: [-1, 1]
  - Rank-based correlation between predicted similarity/relatedness scores and human-annotated gold scores.
- `Purity score` — range: [0, 1]
  - Sum of the size of the largest cluster for each gold category, divided by the total number of instances.

## Input / output format

**Input**: Word pairs with gold similarity/relatedness scores; sets of nouns with gold category labels. Visual grounding uses image-caption pairs from COCO.

**Output**: Predicted similarity/relatedness scores for word pairs; predicted cluster assignments for categorization tasks.

## Scoring recipe

```python
# Similarity/Relatedness
spearman_rho = spearmanr(gold_scores, pred_scores).correlation

# Categorization (Purity)
purity = 0
for cluster in unique_clusters:
    cluster_labels = gold_labels[cluster]
    purity += max(Counter(cluster_labels).values())
purity /= len(gold_labels)
```

## Common pitfalls

- Confusing word similarity (is-a hierarchy) with word relatedness (association), which some benchmarks conflate.
- Using Pearson correlation instead of Spearman rank correlation for similarity benchmarks, which changes the evaluation metric.
- Not accounting for the fact that BLESS scores are normalized per concept before averaging across relations.

## Evidence (verbatim from paper)

> Following (Bruni et al., 2014; Shahmohammadi et al., 2022), we evaluated our visually grounded word embeddings using similarity/relatedness benchmarks. The task is to estimate the similarity/relatedness score of a pair of words using the Spearman correlation as evaluation metric.

## Citation

```bibtex
@misc{mohammed2022visualgrounding,
  title={Visual Grounding of Inter-lingual Word-Embeddings},
  author={Mohammed et al. (2022)},
  year={2022},
  note={arXiv:2209.03714}
}
```

- arXiv: 2209.03714