Crohme Hme100k Eval

---
name: crohme-hme100k-eval
description: Evaluates handwritten mathematical expression recognition (HMER) by measuring exact LaTeX sequence matching, tolerant symbol-level error rates, and structural tree prediction accuracy on complex handwritten formulas. Use when the user wants to benchmark on CROHME, HME100K, or asks about evaluating this task. Reports ExpRate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2203.01601
  bibtex_key: yuan2022syntax
  confidence: high
---

# crohme-hme100k-eval

> Syntax-Aware Network for Handwritten Mathematical Expression Recognition — Ye Yuan et al. (2022) (arXiv:2203.01601, 2022)

## What this evaluates

Evaluates handwritten mathematical expression recognition (HMER) by measuring exact LaTeX sequence matching, tolerant symbol-level error rates, and structural tree prediction accuracy on complex handwritten formulas.

## Datasets

- **CROHME** — total 8836; splits: train (8836), test_2014 (986), test_2016 (1147), test_2019 (1199)
- **HME100K** — total 24607; splits: test_easy (7721), test_moderate (10450), test_hard (6436); repo https://github.com/tal-tech/SAN

## Metrics

- `ExpRate` **(primary)** — range: percent
  - Percentage of predicted mathematical expressions that exactly match the ground truth LaTeX sequence.
- `≤1` — range: percent
  - Percentage of predictions tolerating at most one symbol-level error compared to ground truth.
- `≤2` — range: percent
  - Percentage of predictions tolerating at most two symbol-level errors compared to ground truth.
- `ESPR` — range: percent
  - Percentage of expressions whose structural tree is recognized correctly, irrespective of symbol labels.

## Input / output format

**Input**: Handwritten mathematical expression images converted from InkML stroke trajectory data.

**Output**: Predicted LaTeX markup sequence representing the mathematical expression.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    exact = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    err1 = sum(1 for p, g in zip(predictions, golds) if symbol_edit_distance(p, g) <= 1) / len(golds)
    err2 = sum(1 for p, g in zip(predictions, golds) if symbol_edit_distance(p, g) <= 2) / len(golds)
    struct = sum(1 for p, g in zip(predictions, golds) if parse_structure(p) == parse_structure(g)) / len(golds)
    return exact * 100, err1 * 100, err2 * 100, struct * 100
```

## Common pitfalls

- ExpRate requires exact string match, while ≤1/≤2 allow symbol-level deviations; confusing them leads to incorrect benchmark reporting.
- ESPR evaluates only the tree structure (syntax), ignoring symbol labels, which differs from standard sequence matching metrics.
- HME100K test splits are difficulty-stratified by Structural Complexity and Character Length, not randomly sampled.

## Evidence (verbatim from paper)

> Recognition Protocol. Expression recognition rate (ExpRate) is the widely used recognition protocol for mathematical expression recognition, defined as the percentage of predicted mathematical expressions accurately matching the ground truth. ExpRate ≤ 1 and ≤ 2 indicate the expression recognition rates are tolerable at most one or two symbol-level errors. Structure Recognition Protocol. Expression Structure Prediction Rate (ESPR) is used as the structure recognition protocol. ESPR is calculated by the percent of MEs whose structure is recognized correctly irrespective of symbol labels.

## Citation

```bibtex
@misc{yuan2022syntax,
  title={Syntax-Aware Network for Handwritten Mathematical Expression Recognition},
  author={Ye Yuan et al. (2022)},
  year={2022},
  note={arXiv:2203.01601}
}
```

- arXiv: 2203.01601