Lexicon Grounding Eval

---
name: lexicon-grounding-eval
description: Evaluates how well a model learns word meanings and general language modeling performance when trained with lexicon-level contrastive visual grounding. Probes concrete vs. abstract word acquisition, verb relation learning, and next-token prediction accuracy on held-out text. Use when the user wants to benchmark on Word Relatedness, Semantic Feature Prediction, Context Understanding, Lexical Relation Prediction, SimVerb-3500, or asks about evaluating this task. Reports Perplexity.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.14551
  bibtex_key: zhuang2024lexiconcontrastive
  confidence: high
---

# lexicon-grounding-eval

> Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling — Chengxu Zhuang et al. (2024) (arXiv:2403.14551, 2024)

## What this evaluates

Evaluates how well a model learns word meanings and general language modeling performance when trained with lexicon-level contrastive visual grounding. Probes concrete vs. abstract word acquisition, verb relation learning, and next-token prediction accuracy on held-out text.

## Datasets

- **Word Relatedness** — total ?; splits: test (-1)
- **Semantic Feature Prediction** — total ?; splits: test (-1)
- **Context Understanding** — total ?; splits: test (-1)
- **Lexical Relation Prediction** — total ?; splits: test (-1)
- **SimVerb-3500** — total ?; splits: test (-1)

## Metrics

- `Perplexity` **(primary)** — range: other
  - Standard cross-entropy perplexity computed on the held-out set of ungrounded text datasets.
- `Word Relatedness / Semantic Feature Prediction / Context Understanding / Lexical Relation Prediction` — range: [0, 1]
  - Model similarity scores or prediction accuracy compared against human judgment benchmarks. Evaluated via rank correlation or accuracy.
- `Negative Loglikelihood` — range: other
  - Averaged negative log-likelihood of each word across all its appearances in the test dataset to measure per-word prediction performance.

## Input / output format

**Input**: Unconditioned text sequences (ungrounded datasets) for evaluation; image-caption pairs for training. Evaluation uses text-only inputs.

**Output**: Next-token probability distributions or word embeddings for similarity comparison.

## Scoring recipe

```python
def compute_perplexity(log_probs, gold_tokens):
    nll = -sum(log_probs) / len(gold_tokens)
    return math.exp(nll)

def compute_word_relatedness(model_scores, human_scores):
    return pearsonr(model_scores, human_scores)

def compute_nll_per_word(word_predictions, word_counts):
    return sum(word_predictions) / sum(word_counts)
```

## Common pitfalls

- Perplexity is evaluated only on held-out ungrounded datasets, not the grounded image-caption data.
- Word relatedness benchmarks compare model similarity to human judgment ranks, not exact match scores.
- SimVerb-3500 specifically tests verb pairs, where static visual grounding shows limited benefit compared to concrete nouns.

## Evidence (verbatim from paper)

> Furthermore, we find that the LCG algorithm achieves better performance than existing algorithms on general language modeling, measured by perplexity in the held-out set of ungrounded datasets (see Fig.[1]C, the leftmost panel).

## Citation

```bibtex
@misc{zhuang2024lexiconcontrastive,
  title={Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling},
  author={Chengxu Zhuang et al. (2024)},
  year={2024},
  note={arXiv:2403.14551}
}
```

- arXiv: 2403.14551