Character Detection Matching

---
name: character-detection-matching
description: Evaluates the visual fidelity and spatial accuracy of formula recognition models by comparing rendered images of predicted and ground-truth LaTeX code at the character level. It addresses the misalignment of text-based metrics with human perception by treating each character as a detectable object in an image. Use when the user has predictions and gold and needs to compute CDM.
metadata:
  skill_kind: metric
  source_arxiv: 2409.03643
  bibtex_key: wang2024image
  confidence: high
---

# character-detection-matching

> Image Over Text: Transforming Formula Recognition Evaluation with Character Detection Matching — Bin Wang et al. (2024) (arXiv:2409.03643, 2024)

## What this evaluates

Evaluates the visual fidelity and spatial accuracy of formula recognition models by comparing rendered images of predicted and ground-truth LaTeX code at the character level. It addresses the misalignment of text-based metrics with human perception by treating each character as a detectable object in an image.

## Datasets

- **UniMER-Test** — total 23757; splits: test (23757)
- **Tiny-Doc-Math** — total 437; splits: test (437)

## Metrics

- `CDM` **(primary)** — range: [0, 1]
  - Image-level character detection matching. Renders predicted and ground-truth LaTeX to images, detects characters as objects, and computes visual similarity with spatial localization. Rendering failures are assigned a score of 0.
- `BLEU` — range: [0, 1]
  - Standard n-gram based text similarity metric, normalized as per prior formula recognition literature.
- `ExpRate` — range: [0, 1]
  - Exact match rate based on character-level string comparison of predicted and ground-truth LaTeX.
- `ExpRate@CDM` — range: [0, 1]
  - Proportion of predictions that achieve a perfect CDM score, indicating completely correct formula recognition.

## Input / output format

**Input**: Formula-level: cropped single formula images. Document-level: PDF pages or screenshots containing multiple formulas.

**Output**: Formula-level: predicted LaTeX string. Document-level: Markdown text containing recognized block formulas.

## Scoring recipe

```python
def compute_cdm(pred_latex, gt_latex):
    pred_img = render_latex(pred_latex)
    gt_img = render_latex(gt_latex)
    if pred_img is None:
        return 0.0
    pred_chars = detect_characters(pred_img)
    gt_chars = detect_characters(gt_img)
    matches = spatial_visual_match(pred_chars, gt_chars)
    return len(matches) / max(len(gt_chars), 1)
```

## Common pitfalls

- Traditional text-based metrics (BLEU, ExpRate) are highly sensitive to LaTeX formatting and style variations, leading to misleading performance rankings across different models.
- Document-level evaluation relies on regex matching to extract block formulas from Markdown outputs, which can fail on non-standard or inline formatting.
- Rendering failures automatically yield a CDM score of 0, which may disproportionately penalize models that produce semantically correct but syntactically unrenderable LaTeX.

## Evidence (verbatim from paper)

> We validate the CDM metric by evaluating several mainstream formula recognition models using both subjective impressions and objective metrics. The models include open-source UniMERNet, Texify, Pix2tex, and the commercial Mathpix API, all tested on the UniMER-Test dataset. The CDM metric relies on the successful rendering of formula images. For models that fail to render images, we assign a CDM score of 0, as rendering failures indicate that the predicted LaTeX code lacks critical elements.

## Citation

```bibtex
@misc{wang2024image,
  title={Image Over Text: Transforming Formula Recognition Evaluation with Character Detection Matching},
  author={Bin Wang et al. (2024)},
  year={2024},
  note={arXiv:2409.03643}
}
```

- arXiv: 2409.03643