Hebrew G2p Eval

---
name: hebrew-g2p-eval
description: This benchmark evaluates a model's ability to convert unvocalized Hebrew text into fully-specified IPA transcriptions, including accurate stress placement and shva realization. It also measures downstream text-to-speech quality and inference latency to assess real-time applicability. Use when the user wants to benchmark on ILSpeech, SASPEECH, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.12311
  bibtex_key: kolani2025phonikud
  confidence: high
---

# hebrew-g2p-eval

> Phonikud: Hebrew Grapheme-to-Phoneme Conversion for Real-Time Text-to-Speech — Yakov Kolani et al. (2025) (arXiv:2506.12311, 2025)

## What this evaluates

This benchmark evaluates a model's ability to convert unvocalized Hebrew text into fully-specified IPA transcriptions, including accurate stress placement and shva realization. It also measures downstream text-to-speech quality and inference latency to assess real-time applicability.

## Datasets

- **ILSpeech** — total ?; splits: test (-1)
- **SASPEECH** — total ?; splits: test (-1)

## Metrics

- `WER` **(primary)** — range: [0, 1]
  - Word Error Rate: (Substitutions + Deletions + Insertions) / Total Words. For TTS, computed by running ASR on generated audio and comparing to original text.
- `CER` — range: [0, 1]
  - Character Error Rate: (Substitutions + Deletions + Insertions) / Total Characters. Computed similarly to WER.
- `WER^σ` — range: [0, 1]
  - Word Error Rate disregarding stress mismatches. Stress diacritics are stripped from both predictions and gold before alignment.
- `RTF` — range: ratio
  - Real-Time Factor: ratio of system processing/inference time to the duration of the generated audio. Lower is better.

## Input / output format

**Input**: Unvocalized Hebrew text (consonantal script without diacritics).

**Output**: Fully-specified IPA transcription including vowel marks, stress diacritics, and shva realization.

## Scoring recipe

```python
def compute_wer_cer(pred, gold):
    dist = levenshtein_distance(pred, gold)
    return dist / len(gold) if len(gold) > 0 else 0.0

def compute_wer_sigma(pred_ipa, gold_ipa):
    pred_clean = re.sub(r'[ˈˌ]', '', pred_ipa)
    gold_clean = re.sub(r'[ˈˌ]', '', gold_ipa)
    return compute_wer_cer(pred_clean, gold_clean)

def compute_rtf(inference_time_s, audio_duration_s):
    return inference_time_s / audio_duration_s
```

## Common pitfalls

- ASR-based WER/CER for TTS may not penalize phonetic inaccuracies (like stress) if the ASR model confidently transcribes the audio back to the original unvocalized text.
- RTF is calculated differently across baselines: open models use local CPU hardware, while proprietary models use cloud APIs, complicating direct latency comparisons.
- WER^σ explicitly ignores stress mismatches, which can mask critical phonetic errors that significantly impact TTS naturalness.

## Evidence (verbatim from paper)

> We calculate word-and character error rates (WER, CER) and WER when disregarding stress $(\mathrm{WER}^{\sigma})$ . We compare to two baselines: Firstly, we apply the existing SOTA Hebrew diacritizers DictaBERT (Shmidman et al., 2023) and Nakdimon (Gershuni and Pinter, 2022) with our IPA conversion, using reasonable defaults for ambiguous features (e.g. final stress, common in Hebrew).

## Citation

```bibtex
@misc{kolani2025phonikud,
  title={Phonikud: Hebrew Grapheme-to-Phoneme Conversion for Real-Time Text-to-Speech},
  author={Yakov Kolani et al. (2025)},
  year={2025},
  note={arXiv:2506.12311}
}
```

- arXiv: 2506.12311