Transition Based Parsing Eval

---
name: transition-based-parsing-eval
description: This evaluation probes a parser's ability to construct accurate projective dependency trees for sentences. It measures how well the model identifies correct syntactic heads and their grammatical relations (labels) under both English and Chinese linguistic conditions. Use when the user wants to benchmark on Penn Treebank (PTB) v5, Chinese Treebank (CTB) v5, or asks about evaluating this task. Reports LAS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1612.05131
  bibtex_key: [REDACTED]
  confidence: high
---

# transition-based-parsing-eval

> Transition-based Parsing with Context Enhancement and Future Reward Reranking — Zhou et al. (2016) (arXiv:1612.05131, 2016)

## What this evaluates

This evaluation probes a parser's ability to construct accurate projective dependency trees for sentences. It measures how well the model identifies correct syntactic heads and their grammatical relations (labels) under both English and Chinese linguistic conditions.

## Datasets

- **Penn Treebank (PTB) v5** — total ?; splits: train (-1), test (-1), dev (-1)
- **Chinese Treebank (CTB) v5** — total ?; splits: train (-1), test (-1), dev (-1)

## Metrics

- `UAS` — range: percent
  - The percentage of words with the correct head in the predicted dependency tree.
- `LAS` **(primary)** — range: percent
  - The percentage of words with both the correct head and the correct dependency label.

## Input / output format

**Input**: Tokenized sentences with part-of-speech tags (predicted via Stanford POS tagger for English, gold-standard for Chinese) and word segmentation (gold-standard for Chinese).

**Output**: A projective dependency tree represented as a list of (head_index, dependency_label) pairs for each word in the sentence.

## Scoring recipe

```python
def compute_uas_las(pred_deps, gold_deps):
    n = len(gold_deps)
    correct_head = sum(1 for p, g in zip(pred_deps, gold_deps) if p[0] == g[0])
    correct_label = sum(1 for (ph, pl), (gh, gl) in zip(pred_deps, gold_deps) if ph == gh and pl == gl)
    return correct_head / n, correct_label / n
```

## Common pitfalls

- The evaluation strictly enforces projective trees; non-projective structures in the original treebanks must be converted or filtered before scoring.
- English evaluation uses predicted POS tags (~97.2% accuracy) rather than gold tags, which introduces external error sources not present in the Chinese evaluation.
- The Chinese dataset uses gold-standard word segmentation and POS tags, creating an asymmetry in input quality between the two languages.

## Evidence (verbatim from paper)

> Besides, all experiments are evaluated with unlabeled attachment score (UAS), the percentage of words with the correct head, and labeled attachment score (LAS), the percentage of words with the correct head and label.

## Citation

```bibtex
@misc{zhou2016transition,
  title={Transition-based Parsing with Context Enhancement and Future Reward Reranking},
  author={Zhou et al. (2016)},
  year={2016},
  note={arXiv:1612.05131}
}
```

- arXiv: 1612.05131