Tunisian Asr Eval

---
name: tunisian-asr-eval
description: Evaluates automatic speech recognition (ASR) models on Tunisian Arabic dialect audio, measuring overall transcription accuracy and code-switching performance for embedded English and French phrases. Use when the user wants to benchmark on LinTO, TunSwitch, or asks about evaluating this task. Reports Word Error Rate (WER).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.02604
  bibtex_key: naouara2025linto
  confidence: high
---

# tunisian-asr-eval

> LinTO Audio and Textual Datasets to Train and Evaluate Automatic Speech Recognition in Tunisian Arabic Dialect — Naouara et al. (2025) (arXiv:2504.02604, 2025)

## What this evaluates

Evaluates automatic speech recognition (ASR) models on Tunisian Arabic dialect audio, measuring overall transcription accuracy and code-switching performance for embedded English and French phrases.

## Datasets

- **LinTO** — total ?; splits: train (-1)
- **TunSwitch** — total ?; splits: test (-1)

## Metrics

- `Word Error Rate (WER)` **(primary)** — range: percent
  - Standard ASR metric: (S + D + I) / N, where S=substitutions, D=deletions, I=insertions, N=number of reference words. Reported as a percentage.
- `F1/Recall/Precision on Latin words` — range: [0, 1]
  - Token-level F1, recall, and precision computed exclusively on Latin-script words (English/French) within code-switched references and predictions.

## Input / output format

**Input**: Audio recordings of Tunisian Arabic speech, frequently code-switched with English or French.

**Output**: Transcribed text using an extended Buckwalter transliteration scheme that encodes both Arabic and Latin characters using ASCII pairs.

## Scoring recipe

```python
def compute_asr_metrics(preds, refs):
    # Standard WER
    wer = edit_distance(preds, refs) / len(refs.split())
    # Latin word metrics (token-level)
    pred_lat = [w for w in preds.split() if is_latin(w)]
    ref_lat = [w for w in refs.split() if is_latin(w)]
    tp = len([w for w in pred_lat if w in ref_lat])
    fp = len([w for w in pred_lat if w not in ref_lat])
    fn = len([w for w in ref_lat if w not in pred_lat])
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return {'wer': wer, 'latin_precision': prec, 'latin_recall': rec, 'latin_f1': f1}
```

## Common pitfalls

- WER can exceed 100% due to high insertion rates, which the paper explicitly notes and should not be clipped or normalized.
- Latin word metrics are computed exclusively on code-switched portions, not the full transcript.
- Statistical confidence intervals are derived via bootstrap resampling, not analytical variance estimates.

## Evidence (verbatim from paper)

> Even if Whisper can recognize some Arabic, it fails to transcribe the Tunisian Arabic Dialect, with Word Error Rates (WER) ranging from 50% to more than 100% (indicating high insertion rates). To assess accuracy for recognition of English and French phrases in a code-switching context, we give F1, recall and precision scores on Latin words. All 95% confidence intervals are computed by performing bootstrap resampling.

## Citation

```bibtex
@misc{naouara2025linto,
  title={LinTO Audio and Textual Datasets to Train and Evaluate Automatic Speech Recognition in Tunisian Arabic Dialect},
  author={Naouara et al. (2025)},
  year={2025},
  note={arXiv:2504.02604}
}
```

- arXiv: 2504.02604