Mathwriting Eval

---
name: mathwriting-eval
description: Evaluates a model's ability to recognize handwritten mathematical expressions and convert them into normalized LaTeX. It probes both offline (rasterized image) and online (ink coordinate) recognition capabilities, measuring how accurately the model reconstructs complex mathematical structures and symbols. Use when the user wants to benchmark on MathWriting, or asks about evaluating this task. Reports CER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.10690
  bibtex_key: gervais2024mathwriting
  confidence: high
---

# mathwriting-eval

> MathWriting: A Dataset For Handwritten Mathematical Expression Recognition — Gervais et al. (2024) (arXiv:2404.10690, 2024)

## What this evaluates

Evaluates a model's ability to recognize handwritten mathematical expressions and convert them into normalized LaTeX. It probes both offline (rasterized image) and online (ink coordinate) recognition capabilities, measuring how accurately the model reconstructs complex mathematical structures and symbols.

## Datasets

- **MathWriting** — total 630000; splits: train (-1), valid (-1), test (-1)

## Metrics

- `CER` **(primary)** — range: [0, 1]
  - Character Error Rate computed over LaTeX tokens rather than ASCII characters. Calculated as the minimum number of single-character edits (insertions, deletions, substitutions) required to transform the predicted LaTeX string into the ground truth LaTeX string, divided by the length of the ground truth string.
- `Exact Match (EM)` — range: [0, 1]
  - Binary metric indicating whether the predicted LaTeX string exactly matches the ground truth LaTeX string character-for-character.
- `≤ 1 dist` — range: percent
  - Percentage of predictions where the Levenshtein edit distance between the predicted and ground truth LaTeX strings is less than or equal to 1.

## Input / output format

**Input**: InkML format (sequence of coordinates with speed/time information) and/or rasterized bitmap images (black ink on white background).

**Output**: Normalized LaTeX string representing the mathematical expression.

## Scoring recipe

```python
def compute_cer(pred_latex, gold_latex):
    pred_tokens = tokenize_latex(pred_latex)  # Uses Appendix M tokenizer
    gold_tokens = tokenize_latex(gold_latex)
    dist = levenshtein_distance(pred_tokens, gold_tokens)
    return dist / len(gold_tokens)
```

## Common pitfalls

- Using standard ASCII tokenization instead of the specified LaTeX tokenization, which unfairly penalizes Greek letters and mathematical symbols.
- Ignoring the distinction between online (ink coordinates) and offline (rasterized image) input formats, which drastically changes model performance.
- Not using the official reference implementation for tokenization and edit distance, leading to non-comparable CER scores across different runs.

## Evidence (verbatim from paper)

> We propose the following evaluation setup based on MathWriting for the quality of handwriting math expression recognition. - evaluation samples: the test split of MathWriting. - metric: character error rate (CER) [22], where a "character" is a LATEX token as defined by the code in Appendix M. We provide a reference implementation of the evaluation metric at the Github page 1 . We propose the use of CER as a metric to make results comparable to other recognition tasks like text recognition [26, 17], and the use of LATEX tokens instead of ASCII characters so that an error on a single non-latin letter (e.g. \alpha recognized as a) counts as one instead of many.

## Citation

```bibtex
@misc{gervais2024mathwriting,
  title={MathWriting: A Dataset For Handwritten Mathematical Expression Recognition},
  author={Gervais et al. (2024)},
  year={2024},
  note={arXiv:2404.10690}
}
```

- arXiv: 2404.10690