Semantic Syntactic Word Eval

---
name: semantic-syntactic-word-eval
description: Evaluates whether trained word vector models can capture semantic and syntactic relationships between words through simple algebraic operations in vector space. It probes the model's ability to solve analogy-style questions by measuring how well the vector arithmetic preserves linguistic regularities. Use when the user wants to benchmark on Semantic-Syntactic Word Relationship test set, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1301.3781
  bibtex_key: mikolov2013word2vec
  confidence: high
---

# semantic-syntactic-word-eval

> Efficient Estimation of Word Representations in Vector Space — Mikolov et al. (2013) (arXiv:1301.3781, 2013)

## What this evaluates

Evaluates whether trained word vector models can capture semantic and syntactic relationships between words through simple algebraic operations in vector space. It probes the model's ability to solve analogy-style questions by measuring how well the vector arithmetic preserves linguistic regularities.

## Datasets

- **Semantic-Syntactic Word Relationship test set** — total 19544; splits: test (19544)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of analogy questions where the predicted word exactly matches the gold answer word. Computed as (number of correct predictions / total questions) * 100.

## Input / output format

**Input**: An analogy question defined by two word pairs (A:B :: C:D), where the model receives the word vectors for A, B, and C.

**Output**: A single word token representing the predicted answer D.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if pred == gold:  # Exact match; synonyms count as mistakes
            correct += 1
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Synonyms are explicitly counted as mistakes; only exact word matches are accepted.
- Multi-word entities (e.g., 'New York') are excluded from the test set, so models are only evaluated on single-token words.
- Reaching 100% accuracy is considered impossible due to the lack of morphological information in the models.

## Evidence (verbatim from paper)

> Question is assumed to be correctly answered only if the closest word to the vector computed using the above method is exactly the same as the correct word in the question; synonyms are thus counted as mistakes. This also means that reaching 100% accuracy is likely to be impossible, as the current models do not have any input information about word morphology.

## Citation

```bibtex
@misc{mikolov2013word2vec,
  title={Efficient Estimation of Word Representations in Vector Space},
  author={Mikolov et al. (2013)},
  year={2013},
  note={arXiv:1301.3781}
}
```

- arXiv: 1301.3781