Psp Accent Eval

---
name: psp-accent-eval
description: Evaluates the phonological accent fidelity and prosodic naturalness of Indic text-to-speech systems across Hindi, Telugu, and Tamil. It decomposes accent into per-phoneme dimensions (retroflex, aspiration, Tamil-zha, vowel-length) and corpus-level distributional metrics, revealing gaps between intelligibility and native-like accent. Use when the user wants to benchmark on PSP Benchmark Sets, or asks about evaluating this task. Reports FAD.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.25476
  bibtex_key: menta2026psp
  confidence: high
---

# psp-accent-eval

> PSP: An Interpretable Per-Dimension Accent Benchmark for Indic Text-to-Speech — Menta et al. (2026) (arXiv:2604.25476, 2026)

## What this evaluates

Evaluates the phonological accent fidelity and prosodic naturalness of Indic text-to-speech systems across Hindi, Telugu, and Tamil. It decomposes accent into per-phoneme dimensions (retroflex, aspiration, Tamil-zha, vowel-length) and corpus-level distributional metrics, revealing gaps between intelligibility and native-like accent.

## Datasets

- **PSP Benchmark Sets** — total 300; splits: pilot (10), golden (300); repo https://github.com/praxelhq/praxy

## Metrics

- `FAD` **(primary)** — range: other
  - Fréchet Audio Distance computed between Wav2Vec2-XLS-R phonetic embeddings of generated audio and a native reference distribution (1000 utterances). Lower values indicate closer alignment to native acoustic distributions.
- `PSD` — range: other
  - Prosodic Signature Divergence measuring distance in a 5D prosodic space (pitch range, log-F0, speech rate, nPVI, log-duration) from native speaker centroids. Lower values indicate more natural prosody.
- `Retroflex/Aspiration/Tamil-zha/Length Fidelity` — range: percent
  - Per-token collapse rates calculated via forced alignment, measuring the percentage of phonological tokens correctly produced versus collapsed. Expressed as a percentage.
- `LLM-WER` — range: percent
  - Word error rate computed by a Qwen-2.5-72B semantic judge over Whisper ASR transcripts to measure semantic intelligibility.

## Input / output format

**Input**: Text prompts in Hindi, Telugu, or Tamil, stratified by phonological density (retroflex-heavy, aspiration-heavy, length-heavy, conjunct-heavy, general). Commercial systems also receive voice gender prompts.

**Output**: Synthesized audio files (WAV) for each text prompt.

## Scoring recipe

```python
def score_system(generated_audios, native_reference_audios, ground_truth_texts):
    # 1. Per-token collapse rates via forced alignment
    aligned = forced_align(generated_audios)
    collapse_rate = sum(1 for t in aligned if t.feature == 'collapsed') / len(aligned)
    
    # 2. FAD using Wav2Vec2-XLS-R embeddings
    gen_emb = wav2vec2_xls_r.encode(generated_audios)
    ref_emb = wav2vec2_xls_r.encode(native_reference_audios)  # 1000 utts
    FAD = frechet_distance(gen_emb.mean(), gen_emb.cov(), ref_emb.mean(), ref_emb.cov())
    
    # 3. PSD from prosodic features
    prosodic_vec = extract_prosody(generated_audios)  # [pitch_range, log_F0, rate, nPVI, log_dur]
    PSD = cosine_distance(prosodic_vec, native_prosodic_centroid)
    
    # 4. LLM-WER
    transcript = whisper.transcribe(generated_audios)
    LLM_WER = qwen2_5_72b.compute_wer(transcript, ground_truth_texts)
    return collapse_rate, FAD, PSD, LLM_WER
```

## Common pitfalls

- WER and FAD/PSD orderings are often inversely correlated; a system with the lowest WER may rank last on accent metrics.
- Pilot sets have sample-size asymmetry (commercial systems use 2 voice genders/20 wavs, open-source use 1 voice/10 wavs), which can skew preliminary collapse rates.
- FAD and PSD are corpus-level metrics requiring large native reference distributions (1000 and ~500 utterances), not per-utterance scores.
- Improving spectral distance (FAD) can sometimes regress prosodic naturalness (PSD) if only acoustic weights are updated without prosodic conditioning.

## Evidence (verbatim from paper)

> Each utterance is scored on all applicable per-phoneme PSP dimensions; corpus-level FAD and PSD are computed once per (system, language) against native reference distributions of 1000 and 500 utterances respectively.

## Citation

```bibtex
@misc{menta2026psp,
  title={PSP: An Interpretable Per-Dimension Accent Benchmark for Indic Text-to-Speech},
  author={Menta et al. (2026)},
  year={2026},
  note={arXiv:2604.25476}
}
```

- arXiv: 2604.25476