Fleurs Slu Eval

---
name: fleurs-slu-eval
description: Evaluates multilingual spoken language understanding (SLU) across 102 languages for topical classification and 92 languages for spoken multiple-choice QA, testing cross-lingual transfer, speech-to-text translation, and robustness to audio quality variations. Use when the user wants to benchmark on SIB-Fleurs, Belebele-Fleurs, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.06117
  bibtex_key: schmidt2025fleurs_slu
  confidence: high
---

# fleurs-slu-eval

> Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understanding — Schmidt et al. (2025) (arXiv:2501.06117, 2025)

## What this evaluates

Evaluates multilingual spoken language understanding (SLU) across 102 languages for topical classification and 92 languages for spoken multiple-choice QA, testing cross-lingual transfer, speech-to-text translation, and robustness to audio quality variations.

## Datasets

- **SIB-Fleurs** — total ?; splits: train (-1), test (-1)
- **Belebele-Fleurs** — total ?; splits: train (-1), dev (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard classification and multiple-choice QA accuracy: the fraction of instances where the predicted label or choice matches the gold label.
- `cross-entropy` — range: [0, inf)
  - Cross-entropy loss computed over concatenated choice logits to evaluate model confidence and training convergence on the QA task.
- `CER` — range: [0, 1]
  - Character Error Rate computed between reference Flores sentences and ASR transcriptions to quantify utterance quality for subset analysis.

## Input / output format

**Input**: For classification: speech utterances or transcriptions. For QA: paragraph, question, and multiple-choice options in text or speech modality. Speech-LLMs receive an English task description plus in-language input.

**Output**: Predicted class label or correct choice index.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

def compute_cer(reference, hypothesis):
    # Standard edit distance based CER
    return edit_distance(reference, hypothesis) / len(reference)
```

## Common pitfalls

- Audio must be normalized to an RMS level of 0.07 to ensure consistent quality across languages.
- For languages unsupported by the ASR backends, the closest available language must be manually selected for in-language transcription.
- Speech LLMs based on Whisper-v3 are limited to 30 seconds of audio, so longer Belebele-Fleurs instances cannot be evaluated with them.

## Evidence (verbatim from paper)

> We train and validate CS models on the English training and dev sets of Belebele (Bandarkar et al., 2024), respectively. We jointly embed the paragraph, question, and choices with text encoders. We then average the token encodings of each choice $c_{i}\in C$ and project the choice embedding via head $H^{D\times 1}$ to a logit $\mathbf{l_{c_{i}}}$. We minimize the cross-entropy of the concatenated choice logits ${\mathbf{l_{c_{i}}}}_{i\=1}^{|C|}$ to the label choice.

## Citation

```bibtex
@misc{schmidt2025fleurs_slu,
  title={Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understanding},
  author={Schmidt et al. (2025)},
  year={2025},
  note={arXiv:2501.06117}
}
```

- arXiv: 2501.06117