Sentencebench Eval

---
name: sentencebench-eval
description: This benchmark evaluates Persian grapheme-to-phoneme (G2P) systems on sentence-level text, specifically probing their ability to correctly map characters to phonemes and disambiguate homographs using contextual information. It measures both phonetic accuracy and contextual word-sense resolution capabilities. Use when the user wants to benchmark on SentenceBench, or asks about evaluating this task. Reports Homograph Acc. (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.12973
  bibtex_key: fetratqharabagh2025fast
  confidence: high
---

# sentencebench-eval

> Fast, Not Fancy: Rethinking G2P with Rich Data and Rule-Based Models — Mahta Fetrat Qharabagh, Zahra Dehghanian, and Hamid R. Rabiee (2025) (arXiv:2505.12973, 2025)

## What this evaluates

This benchmark evaluates Persian grapheme-to-phoneme (G2P) systems on sentence-level text, specifically probing their ability to correctly map characters to phonemes and disambiguate homographs using contextual information. It measures both phonetic accuracy and contextual word-sense resolution capabilities.

## Datasets

- **SentenceBench** — total ?; splits: test (-1); repo https://github.com/MahtaFetrat/Persian-G2P-Tools-Benchmark

## Metrics

- `PER (%)` — range: percent
  - Phoneme Error Rate, calculated as the normalized edit distance between predicted and reference phoneme sequences.
- `Homograph Acc. (%)` **(primary)** — range: percent
  - Percentage of homograph words correctly assigned the contextually appropriate phoneme sequence.

## Input / output format

**Input**: Persian text (words or sentences).

**Output**: Sequence of phonemes corresponding to the input text.

## Scoring recipe

```python
def compute_metrics(predictions, references):
    per = sum(edit_distance(p, r) / max(len(p), len(r)) for p, r in zip(predictions, references)) / len(predictions)
    hom_acc = sum(1 for p, r in zip(predictions, references) if p == r) / len(predictions)
    return {"PER (%)": per * 100, "Homograph Acc. (%)": hom_acc * 100}
```

## Common pitfalls

- Evaluating on word-level data instead of sentence-level data, which removes the contextual information necessary for homograph disambiguation.
- Reporting only point estimates without standard deviations across multiple random seeds or runs, as the protocol requires mean ± std over 5 independent runs.
- Comparing accuracy metrics without accounting for inference latency, which is critical for real-time accessibility applications like screen readers.

## Evidence (verbatim from paper)

> Table[3] presents the performance of previously available G2P tools on the SentenceBench benchmark. As shown, the only two models that perform well in terms of PER are the neural GE2PE model ([2024]) and the rule-based eSpeak tool ([2024]). However, even these models perform worse than random when it comes to homograph disambiguation.

## Citation

```bibtex
@misc{fetratqharabagh2025fast,
  title={Fast, Not Fancy: Rethinking G2P with Rich Data and Rule-Based Models},
  author={Mahta Fetrat Qharabagh, Zahra Dehghanian, and Hamid R. Rabiee (2025)},
  year={2025},
  note={arXiv:2505.12973}
}
```

- arXiv: 2505.12973