S2l Eval

---
name: s2l-eval
description: Evaluates models' ability to transcribe spoken mathematical equations and sentences into correct LaTeX syntax. It probes audio-to-text conversion, handling of mathematical symbols, and robustness to syntactic variations in LaTeX formatting. Use when the user wants to benchmark on S2L-equations, S2L-sentences, or asks about evaluating this task. Reports CER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.03542
  bibtex_key: korzh2025speechlatex
  confidence: high
---

# s2l-eval

> Speech-to-LaTeX: New Models and Datasets for Converting Spoken Equations and Sentences — Korzh et al. (2025) (arXiv:2508.03542, 2025)

## What this evaluates

Evaluates models' ability to transcribe spoken mathematical equations and sentences into correct LaTeX syntax. It probes audio-to-text conversion, handling of mathematical symbols, and robustness to syntactic variations in LaTeX formatting.

## Datasets

- **S2L-equations** — total ?; splits: train (-1), test (-1)
- **S2L-sentences** — total ?; splits: train (-1), test (-1)

## Metrics

- `CER` **(primary)** — range: percent
  - Character Error Rate: the ratio of edit operations (insertions, deletions, substitutions) to the length of the ground truth string. Evaluated on normalized, lowercase text to mitigate syntactic differences.
- `TeXBLEU` — range: [0, 1]
  - BLEU score computed on LaTeX token sequences after normalization. Specifically designed to compare LaTeX code while accounting for structural equivalence and formatting variations.

## Input / output format

**Input**: 16kHz audio waveform of spoken mathematical expressions or sentences. For ASR baselines, the input is the raw ASR transcription; for multimodal models, it is the raw audio concatenated with a textual prompt.

**Output**: LaTeX code representing the spoken equation or sentence. For sentences, inline formulas must be correctly embedded within the surrounding text.

## Scoring recipe

```python
def score(predictions, ground_truths):
    cer_scores = []
    texbleu_scores = []
    for pred, gt in zip(predictions, ground_truths):
        # Apply equation normalization to handle spacing/capitalization/font differences
        pred_norm = normalize_latex(pred.lower())
        gt_norm = normalize_latex(gt.lower())
        # Compute CER
        cer = edit_distance(pred_norm, gt_norm) / len(gt_norm)
        cer_scores.append(cer)
        # Compute TeXBLEU (on non-lowercased tokens)
        texbleu = compute_bleu_on_latex_tokens(pred, gt)
        texbleu_scores.append(texbleu)
    return {'CER': sum(cer_scores)/len(cer_scores), 'TeXBLEU': sum(texbleu_scores)/len(texbleu_scores)}
```

## Common pitfalls

- Raw CER heavily penalizes syntactically equivalent LaTeX (e.g., \int_a^b vs \int_{a}^{b}) despite identical mathematical meaning.
- Capitalization and font style variations (e.g., \phi vs \Phi, \mathcal{R} vs r) distort character-level metrics if normalization is skipped.
- For S2L-sentences, inline formulas must be extracted and evaluated separately from surrounding text; evaluating the full string as one unit mixes text and math errors.

## Evidence (verbatim from paper)

> The primary reported metrics are character error rate ($\operatorname{CER}$), and $\operatorname{TeXBLEU}$ *(Jung et al. [2025])* metric, recently specifically proposed for LaTeX comparison. For S2L-equations, predictions and ground truth are compared in LaTeX form, as illustrated in Table[5]. For S2L-sentences, which contain inline formulas within English text, we separately evaluate both equation and text components.

## Citation

```bibtex
@misc{korzh2025speechlatex,
  title={Speech-to-LaTeX: New Models and Datasets for Converting Spoken Equations and Sentences},
  author={Korzh et al. (2025)},
  year={2025},
  note={arXiv:2508.03542}
}
```

- arXiv: 2508.03542