Formula Extraction Eval

---
name: formula-extraction-eval
description: Evaluates the ability of PDF document parsers to accurately extract mathematical formulas and preserve their semantic meaning. It probes format variability handling, representational non-uniqueness, and semantic equivalence recognition beyond simple character matching. Use when the user wants to benchmark on PDF Formula Extraction Benchmark, or asks about evaluating this task. Reports LLM-as-a-Judge.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.09874
  bibtex_key: horn2025benchmarking
  confidence: high
---

# formula-extraction-eval

> Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs — Horn et al. (2025) (arXiv:2512.09874, 2025)

## What this evaluates

Evaluates the ability of PDF document parsers to accurately extract mathematical formulas and preserve their semantic meaning. It probes format variability handling, representational non-uniqueness, and semantic equivalence recognition beyond simple character matching.

## Datasets

- **PDF Formula Extraction Benchmark** — total 250; splits: test (250); repo https://github.com/phorn1/pdf-parse-bench

## Metrics

- `LLM-as-a-Judge` **(primary)** — range: 0-10
  - Scores formula pairs on a 0–10 scale based on correctness, completeness, and semantic equivalence. Evaluated using GPT-5-mini.
- `CDM` — range: 0-1
  - Computes character-level precision, recall, and F1-score by rendering formulas, detecting bounding boxes, and performing bipartite matching based on token identity, spatial proximity, and sequential order.

## Input / output format

**Input**: Ground truth mathematical formula and parsed formula output from a PDF parser.

**Output**: A numerical score from 0 to 10.

## Scoring recipe

```python
def score_formula_pair(gt_formula, parsed_formula, model='gpt-5-mini'):
    prompt = f'Score 0-10 on correctness, completeness, semantic equivalence:\nGT: {gt_formula}\nParsed: {parsed_formula}'
    response = model.generate(prompt)
    score = extract_score(response)  # Parse 0-10 float from LLM output
    return score
```

## Common pitfalls

- Assuming text-based metrics (BLEU, Levenshtein) are sufficient; they fail due to format variability and representational non-uniqueness in LaTeX/MathML.
- Relying solely on CDM, which produces false positives for structural errors (superscripts/subscripts) and false negatives for Unicode outputs or semantically equivalent LaTeX variants.

## Evidence (verbatim from paper)

> We evaluated five contemporary LLM models (GPT-5, GPT-5-mini, GPT-5-nano, Gemini-2.5-Flash, and Mistral-Medium-2508), prompting them to score formula pairs on a 0–10 scale based on correctness, completeness, and semantic equivalence. As shown in Figure[2], all models exhibited substantially higher Pearson correlations with human scores (0.69–0.78) than CDM or text-based metrics, with GPT-5 and GPT-5-mini achieving the strongest correlations (0.78). We selected GPT-5-mini as our evaluation model given its competitive performance and cost-effectiveness. Although LLM evaluation is not infallible—occasionally assigning imperfect scores to identically rendered formulas—its correlation with human judgment (0.78) substantially exceeds that of CDM (0.34), making it a more suitable metric for evaluating mathematical formula extraction.

## Citation

```bibtex
@misc{horn2025benchmarking,
  title={Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs},
  author={Horn et al. (2025)},
  year={2025},
  note={arXiv:2512.09874}
}
```

- arXiv: 2512.09874