Irish English St Eval

---
name: irish-english-st-eval
description: Evaluates end-to-end speech translation from Irish to English, specifically probing how synthetic audio data and augmentation techniques (noise, VAD) impact model performance in low-resource settings. Use when the user wants to benchmark on IWSLT-2023, FLEURS, Bitesize, SpokenWords, or asks about evaluating this task. Reports chrF++.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.17363
  bibtex_key: moslem2024leveraging
  confidence: high
---

# irish-english-st-eval

> Leveraging Synthetic Audio Data for End-to-End Low-Resource Speech Translation — Moslem (2024) (arXiv:2406.17363, 2024)

## What this evaluates

Evaluates end-to-end speech translation from Irish to English, specifically probing how synthetic audio data and augmentation techniques (noise, VAD) impact model performance in low-resource settings.

## Datasets

- **IWSLT-2023, FLEURS, Bitesize, SpokenWords** — total ?; splits: val (-1), test (-1); repo https://github.com/ymoslem/MT-Preparation

## Metrics

- `chrF++` **(primary)** — range: [0, 1]
  - Character n-gram F-score computed over character-level n-grams (typically 1-6) between predicted and reference translations. Used here to select the best validation checkpoint during training.

## Input / output format

**Input**: Raw Irish audio waveform

**Output**: English text transcript

## Scoring recipe

```python
def compute_chrf(predictions, references, n=6, beta=2.0):
    precisions = []
    recalls = []
    for i in range(1, n+1):
        p = count_ngram_overlap(predictions, references, i)
        r = count_ngram_overlap(references, predictions, i)
        precisions.append(p)
        recalls.append(r)
    p_mean = geometric_mean(precisions)
    r_mean = geometric_mean(recalls)
    return (1 + beta**2) * (p_mean * r_mean) / (beta**2 * p_mean + r_mean)
```

## Common pitfalls

- Using a small early stopping patience on low-resource validation sets can prevent the model from seeing the full data distribution, hurting robustness.
- Synthetic audio generated via TTS lacks natural background noise and silence patterns, potentially inflating validation scores compared to real-world deployment.

## Evidence (verbatim from paper)

> We train the main models with Whisper Medium for at least two epochs, and save the best performing checkpoint based on the chrF++ score on the validation dataset.

## Citation

```bibtex
@misc{moslem2024leveraging,
  title={Leveraging Synthetic Audio Data for End-to-End Low-Resource Speech Translation},
  author={Moslem (2024)},
  year={2024},
  note={arXiv:2406.17363}
}
```

- arXiv: 2406.17363