Mosel Maltese Asr Eval

---
name: mosel-maltese-asr-eval
description: Evaluates automatic speech recognition (ASR) performance on low-resource Maltese speech data. It measures the accuracy of a sequence-to-sequence model in transcribing audio into text after training on filtered open-source speech corpora. Use when the user wants to benchmark on VoxPopuli (Maltese subset), or asks about evaluating this task. Reports Word Error Rate (WER).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.01036
  bibtex_key: gaido2024mosel
  confidence: high
---

# mosel-maltese-asr-eval

> MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languages — Gaido et al. (2024) (arXiv:2410.01036, 2024)

## What this evaluates

Evaluates automatic speech recognition (ASR) performance on low-resource Maltese speech data. It measures the accuracy of a sequence-to-sequence model in transcribing audio into text after training on filtered open-source speech corpora.

## Datasets

- **VoxPopuli (Maltese subset)** — total 9000; splits: train (-1); repo https://github.com/hlt-mt/mosel

## Metrics

- `Word Error Rate (WER)` **(primary)** — range: percent
  - Computed as (S + D + I) / N, where S=substitutions, D=deletions, I=insertions, N=reference word count. Lower values indicate better transcription accuracy.

## Input / output format

**Input**: Audio segments represented as 80 Mel-filterbank features extracted every 10 ms with a 25 ms window.

**Output**: Text transcripts generated as a sequence of tokens from an 8,000-size SentencePiece vocabulary.

## Scoring recipe

```python
def compute_wer(predictions, references):
    # Normalize predictions and references using Whisper Normalizer
    norm_preds = whisper_normalizer(predictions)
    norm_refs = whisper_normalizer(references)
    # Compute Word Error Rate using JiWER
    wer = jiwer.wer(norm_refs, norm_preds)
    return wer
```

## Common pitfalls

- ASR models frequently generate textual hallucinations, such as repetitive spurious words or long noisy strings, especially when background noise or music is present in the audio.
- Metadata language tags may be inconsistent with actual spoken content, leading to misaligned training data if not filtered via automatic Language Identification (LID).
- WER computation requires careful normalization; raw Whisper outputs often contain formatting artifacts that must be stripped before evaluation to avoid inflated error rates.

## Evidence (verbatim from paper)

> Results in Word Error Rate (WER) are computed using the Whisper Normalizer and, then, JiWER for computing the metric.

## Citation

```bibtex
@misc{gaido2024mosel,
  title={MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languages},
  author={Gaido et al. (2024)},
  year={2024},
  note={arXiv:2410.01036}
}
```

- arXiv: 2410.01036