Mtbi Speech Eval

---
name: mtbi-speech-eval
description: Evaluates the automatic speech recognition accuracy and zero-shot generalization capabilities of Speech Large Language Models. It probes robustness across mathematical reasoning, speaker role inference, and prompt adaptation tasks. Use when the user wants to benchmark on LibriSpeech, GSM8K, Generalization Test Set, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.18644
  bibtex_key: xie2025mtbi
  confidence: high
---

# mtbi-speech-eval

> Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleaving — Jingran Xie et al. (2025) (arXiv:2505.18644, 2025)

## What this evaluates

Evaluates the automatic speech recognition accuracy and zero-shot generalization capabilities of Speech Large Language Models. It probes robustness across mathematical reasoning, speaker role inference, and prompt adaptation tasks.

## Datasets

- **LibriSpeech** — total ?; splits: test-clean (-1)
- **GSM8K** — total ?; splits: test (-1)
- **Generalization Test Set** — total ?; splits: test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Standard ASR metric: (Substitutions + Deletions + Insertions) / Total Reference Words. Lower is better.
- `Prompt Generalization Accuracy` — range: percent
  - Percentage of correctly answered prompts after ASR transcription, measuring robustness to speech-text alignment.
- `GSM8K Accuracy` — range: percent
  - Exact-match accuracy on grade-school math word problems, evaluated in zero-shot and 1-shot settings.
- `Speaker Role Accuracy` — range: percent
  - Percentage of correctly identified speaker roles in conversational prompts.

## Input / output format

**Input**: Speech audio input paired with text prompts or task instructions.

**Output**: Text response (transcription for ASR, numerical/text answer for GSM8K, categorical label for Speaker Role).

## Scoring recipe

```python
def compute_metrics(predictions, golds, task):
    if task == 'ASR':
        return compute_wer(predictions, golds)
    elif task in ['GSM8K', 'Speaker Role', 'Prompt Generalization']:
        correct = sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g))
        return correct / len(golds) * 100
```

## Common pitfalls

- ASR SFT models tend to overfit to transcription tasks, prioritizing ASR accuracy over prompt following, which artificially inflates prompt generalization scores.
- Mathematical reasoning tasks are highly sensitive to ASR transcription errors; even minor speech recognition mistakes cascade into incorrect final answers, lowering overall accuracy.

## Evidence (verbatim from paper)

> For evaluation, we use the standard ASR evaluation on the LibriSpeech test-clean dataset with word error rate (WER) as the evaluation metric. Details of the generalization evaluation are in Section 3. During inference, LLM applies a temperature of 0.7 and top-p sampling with a 0.85 threshold, setting a maximum token length of 100 (200 for math tasks) to minimize hallucinations.

## Citation

```bibtex
@misc{xie2025mtbi,
  title={Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleaving},
  author={Jingran Xie et al. (2025)},
  year={2025},
  note={arXiv:2505.18644}
}
```

- arXiv: 2505.18644