L2 Arctic Mispronunciation Eval

---
name: l2-arctic-mispronunciation-eval
description: Probes the ability to detect phoneme-level mispronunciations in second-language (L2) speech by comparing acoustic features of learner speech against a voice-cloned native reference using dynamic time warping on MFCC envelopes. Use when the user wants to benchmark on L2-Arctic, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.10985
  bibtex_key: [REDACTED]
  confidence: high
---

# l2-arctic-mispronunciation-eval

> Pronunciation Deviation Analysis Through Voice Cloning and Acoustic Comparison — Valdivia et al. (2025) (arXiv:2507.10985, 2025)

## What this evaluates

Probes the ability to detect phoneme-level mispronunciations in second-language (L2) speech by comparing acoustic features of learner speech against a voice-cloned native reference using dynamic time warping on MFCC envelopes.

## Datasets

- **L2-Arctic** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly classified instances (both correct and mispronounced) out of the total number of instances.
- `precision` — range: [0, 1]
  - Ratio of true positive mispronunciations detected to the total number of instances predicted as mispronunciations.
- `recall` — range: [0, 1]
  - Ratio of true positive mispronunciations detected to the total number of actual mispronunciations in the ground truth.
- `F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall, calculated as 2 * (precision * recall) / (precision + recall).

## Input / output format

**Input**: Per-instance speech audio of a second-language learner reading a target text, processed into MFCC envelopes and compared against a voice-cloned native reference via dynamic time warping to yield an acoustic deviation distance.

**Output**: Binary classification label (0: correct, 1: mispronunciation) per instance, derived by comparing the acoustic deviation distance against a speaker-specific calibrated threshold.

## Scoring recipe

```python
tp = sum((pred == 1) & (gold == 1))
fp = sum((pred == 1) & (gold == 0))
fn = sum((pred == 0) & (gold == 1))
tn = sum((pred == 0) & (gold == 0))
precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
accuracy = (tp + tn) / (tp + tn + fp + fn)
```

## Common pitfalls

- The evaluation uses speaker-specific acoustic distance thresholds calibrated per individual, so applying a fixed threshold across different speakers or datasets will yield invalid results.
- Class support is uneven across individuals (e.g., EBVS has 110 vs 133 samples), meaning precision and recall are highly sensitive to the chosen decision boundary and may not reflect balanced performance.
- The binary labels (0/1) are derived from continuous DTW distance scores, so the metric does not evaluate raw distance values but rather thresholded classification decisions.

## Evidence (verbatim from paper)

> From the L2-Arctic dataset, our model was applied to four individuals: EBVS, ERMS, MBMPS, and NJS. Classification performance was measured using precision, recall, F1-score, and accuracy.

## Citation

```bibtex
@misc{valdivia2025pronunciation,
  title={Pronunciation Deviation Analysis Through Voice Cloning and Acoustic Comparison},
  author={Valdivia et al. (2025)},
  year={2025},
  note={arXiv:2507.10985}
}
```

- arXiv: 2507.10985