Word Learning Eval

---
name: word-learning-eval
description: Evaluates how well multi-modal models learn word meanings and semantic relationships from limited data, comparing visual grounding against language-only baselines across word-relatedness, feature prediction, and POS tagging tasks. Use when the user wants to benchmark on SimLex-999, SimVerb-3500, Word-relatedness dataset (Bruni et al., 2012), or asks about evaluating this task. Reports human-likeness measure.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.13257
  bibtex_key: zhuang2023visualgrounding
  confidence: high
---

# word-learning-eval

> Visual Grounding Helps Learn Word Meanings in Low-Data Regimes — Zhuang et al. (2023) (arXiv:2310.13257, 2023)

## What this evaluates

Evaluates how well multi-modal models learn word meanings and semantic relationships from limited data, comparing visual grounding against language-only baselines across word-relatedness, feature prediction, and POS tagging tasks.

## Datasets

- **SimLex-999** — total ?; splits: test (-1)
- **SimVerb-3500** — total ?; splits: test (-1)
- **Word-relatedness dataset (Bruni et al., 2012)** — total ?; splits: test (-1)

## Metrics

- `human-likeness measure` **(primary)** — range: [0, 1]
  - Correlation between model-pair similarity scores and human similarity judgments. Pairs are ranked by the absolute difference between human and model similarity to compute a normalized human-likeness rank.
- `prediction accuracy` — range: [0, 1]
  - Standard classification accuracy for semantic feature prediction and POS tagging benchmarks.
- `learning efficiency` — range: percent
  - Model performance measured across varying data scales (e.g., 50M tokens, 2.1M image-caption pairs) to assess sample efficiency.

## Input / output format

**Input**: Word pairs or single words paired with image features, full captions, or narrow textual contexts (e.g., 3-word segments). For sentence-level tasks, segmented sentences with center words and associated images.

**Output**: Model similarity scores for word pairs, predicted semantic features, POS tags, or probability distributions over words.

## Scoring recipe

```python
# Compute model similarity for word pairs
model_sim = cosine_similarity(model_emb[w1], model_emb[w2])
human_sim = human_judgments[w1, w2]
# Primary metric: correlation with human judgments
metric = spearmanr(model_sim, human_sim).correlation
# Secondary: accuracy for feature/POS prediction
acc = accuracy_score(gold_labels, predicted_labels)
```

## Common pitfalls

- Visual grounding benefits are highly data-dependent; they disappear or reverse in high-data regimes where distributional text dominates.
- Models often fail to synergistically combine visual and textual signals, instead showing competitive interference.
- Visual models perform poorly on abstract words and verbs, as static images lack dynamic/action information.

## Evidence (verbatim from paper)

> To further analyze differences at the word level, we then develop a human-likeness measure for a pair of words by comparing the judgments of humans and models (see Appendix[A.2] for details). This measure is sorted to get a normalized “rank in human-likeness” measure for each pair of words, where a larger value means more human-like.

## Citation

```bibtex
@misc{zhuang2023visualgrounding,
  title={Visual Grounding Helps Learn Word Meanings in Low-Data Regimes},
  author={Zhuang et al. (2023)},
  year={2023},
  note={arXiv:2310.13257}
}
```

- arXiv: 2310.13257