Fleurs Cer Eval

---
name: fleurs-cer-eval
description: This evaluation probes a model's ability to perform automatic speech recognition in low-resource and zero-supervised settings by leveraging joint speech-text representation learning. It specifically measures how well the model can transcribe unseen languages using only untranscribed audio and graphemic text, without relying on manually labeled speech data. Use when the user wants to benchmark on FLEURS, or asks about evaluating this task. Reports CER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.10027
  bibtex_key: chen2022maestro_u
  confidence: high
---

# fleurs-cer-eval

> Maestro-U: Leveraging joint speech-text representation learning for zero supervised speech ASR — Chen et al. (2022) (arXiv:2210.10027, 2022)

## What this evaluates

This evaluation probes a model's ability to perform automatic speech recognition in low-resource and zero-supervised settings by leveraging joint speech-text representation learning. It specifically measures how well the model can transcribe unseen languages using only untranscribed audio and graphemic text, without relying on manually labeled speech data.

## Datasets

- **FLEURS** — total ?; splits: Group A (-1), Group B (-1)

## Metrics

- `CER` **(primary)** — range: percent
  - Character Error Rate is calculated as the minimum number of character-level edits (insertions, deletions, substitutions) required to transform the predicted transcription into the reference, divided by the reference length and multiplied by 100.

## Input / output format

**Input**: Raw speech audio waveform and, during training/inference, a language identifier embedding. During training, unspoken text (graphemes or byte-level representations) is also provided.

**Output**: A predicted character sequence representing the transcribed speech in the target language's writing system.

## Scoring recipe

```python
def compute_cer(predictions, references):
    total_errors = 0
    total_ref_chars = 0
    for pred, ref in zip(predictions, references):
        dist = levenshtein_distance(pred, ref)
        total_errors += dist
        total_ref_chars += len(ref)
    return (total_errors / total_ref_chars) * 100 if total_ref_chars > 0 else 0.0
```

## Common pitfalls

- Confusing Group A (languages with supervised speech) and Group B (zero-supervised speech) results, as the paper's main contribution targets Group B performance.
- Assuming standard grapheme-based text injection suffices for zero-supervised ASR; the ablation shows language embeddings, duration modeling, and byte-level clustering are critical for unseen scripts.
- Overlooking code-switching effects: South Asian languages contain unavoidable Latin script characters, which artificially inflates grapheme overlap metrics if not filtered.

## Evidence (verbatim from paper)

> We evaluate performance using Character Error Rate (CER) as proposed in [21]. We use a W2V-BERT model trained with untranscribed speech and fine-tuned with transcribed speech as the baseline for all experiments with no other text injection.

## Citation

```bibtex
@misc{chen2022maestro_u,
  title={Maestro-U: Leveraging joint speech-text representation learning for zero supervised speech ASR},
  author={Chen et al. (2022)},
  year={2022},
  note={arXiv:2210.10027}
}
```

- arXiv: 2210.10027