Multilingual Tedx Eval

---
name: multilingual-tedx-eval
description: Evaluates automatic speech recognition, machine translation, and speech translation capabilities on a multilingual corpus of TEDx talks. It probes model robustness to lower-resource conditions, cross-lingual transfer, and the effectiveness of cascaded versus end-to-end modeling paradigms. Use when the user wants to benchmark on Multilingual TEDx Corpus, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2102.01757
  bibtex_key: salesky2021multilingualtedx
  confidence: high
---

# multilingual-tedx-eval

> The Multilingual TEDx Corpus for Speech Recognition and Translation — Salesky et al. (2021) (arXiv:2102.01757, 2021)

## What this evaluates

Evaluates automatic speech recognition, machine translation, and speech translation capabilities on a multilingual corpus of TEDx talks. It probes model robustness to lower-resource conditions, cross-lingual transfer, and the effectiveness of cascaded versus end-to-end modeling paradigms.

## Datasets

- **Multilingual TEDx Corpus** — total ?; splits: test (-1); repo https://github.com/m-wiesner/tedx

## Metrics

- `WER` — range: percent
  - Word Error Rate: the minimum number of insertions, deletions, and substitutions of words required to transform the predicted transcript into the reference transcript, divided by the number of words in the reference.
- `BLEU` **(primary)** — range: percent
  - Cased tokenized BLEU score computed using SACRE-BLEU. Calculated as the geometric mean of modified n-gram precisions with a brevity penalty.

## Input / output format

**Input**: For ASR: 16kHz mono audio files. For MT/ST: source language text sentences (with language ID tags appended at the beginning for multilingual models).

**Output**: For ASR: predicted word transcript. For MT/ST: translated target language text sentence.

## Scoring recipe

```python
def compute_wer(predictions, references):
    total_errors = 0
    total_refs = 0
    for pred, ref in zip(predictions, references):
        total_errors += edit_distance(pred.split(), ref.split())
        total_refs += len(ref.split())
    return (total_errors / total_refs) * 100

def compute_bleu(predictions, references):
    return sacrebleu.corpus_bleu(predictions, [references]) * 100
```

## Common pitfalls

- ASR evaluation removes punctuation and lowercases text, while MT/ST evaluation retains original punctuation and case, creating inconsistent preprocessing standards across tasks.
- Arabic WER results are heavily skewed because the corpus uses Modern Standard Arabic transcripts rather than the spoken dialect actually present in the audio.
- Cascaded ST models are evaluated using 1-best ASR transcripts (beam=10), which ignores ASR errors in the downstream MT model and may overestimate real-world ST performance.

## Evidence (verbatim from paper)

> We computed cased tokenized BLEU scores using SACRE-BLEU [22]. Different community standards for punctuation and case exist between ASR, MT, and ST; while we removed punctuation and lowercased text for ASR evaluation, we did not for translation tasks.

## Citation

```bibtex
@misc{salesky2021multilingualtedx,
  title={The Multilingual TEDx Corpus for Speech Recognition and Translation},
  author={Salesky et al. (2021)},
  year={2021},
  note={arXiv:2102.01757}
}
```

- arXiv: 2102.01757