Flm Audio Eval

---
name: flm-audio-eval
description: Evaluates native full-duplex audio-language models on speech understanding, speech generation, and real-time conversational capabilities. It measures how well models handle asynchronous text-audio streams, responsiveness to interruptions, and overall dialogue quality compared to specialized ASR/TTS systems and other full-duplex chatbots. Use when the user wants to benchmark on Fleurs-zh, LibriSpeech-clean, LlamaQuestions, Seed-TTS-en, Seed-TTS-zh, Custom Chinese Speech Instruction-Following Set, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.02521
  bibtex_key: yao2025flmaudio
  confidence: high
---

# flm-audio-eval

> FLM-Audio: Natural Monologues Improves Native Full-Duplex Chatbots via Dual Training — Yao et al. (2025) (arXiv:2509.02521, 2025)

## What this evaluates

Evaluates native full-duplex audio-language models on speech understanding, speech generation, and real-time conversational capabilities. It measures how well models handle asynchronous text-audio streams, responsiveness to interruptions, and overall dialogue quality compared to specialized ASR/TTS systems and other full-duplex chatbots.

## Datasets

- **Fleurs-zh** — total ?; splits: test (-1)
- **LibriSpeech-clean** — total ?; splits: test (-1)
- **LlamaQuestions** — total ?; splits: test (-1)
- **Seed-TTS-en** — total ?; splits: test (-1)
- **Seed-TTS-zh** — total ?; splits: test (-1)
- **Custom Chinese Speech Instruction-Following Set** — total ?; splits: test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate: ratio of edit operations (insertions, deletions, substitutions) to the number of words in the reference transcript.
- `Accuracy` — range: [0, 1]
  - Exact-match accuracy for spoken question answering on LlamaQuestions.
- `SIM` — range: [0, 1]
  - Speaker Similarity score computed using a lightly fine-tuned speaker verification model on generated audio.
- `LLM-score` — range: [0, 10]
  - Quality score (0-10) assigned by DeepSeek-V3 comparing candidate textual responses to ground-truth answers.
- `Human Evaluation Ratings` — range: [0, 10]
  - Average ratings from 5 annotators across four dimensions: Helpfulness, Naturalness, Responsiveness, and Robustness.

## Input / output format

**Input**: Audio prompts (converted from text for instruction-following), spoken questions, and reference audio clips for voice cloning/generation tasks.

**Output**: Text responses for ASR/QA/instruction-following tasks; generated audio waveforms for TTS evaluation.

## Scoring recipe

```python
def compute_metrics(predictions, references, generated_audio, reference_audio, candidate_text, ground_truth):
    wer = sum(edit_distance(p, r) / len(r) for p, r in zip(predictions, references)) / len(references)
    acc = sum(1 for p, r in zip(predictions, references) if p == r) / len(references)
    sim = speaker_similarity_model.score(generated_audio, reference_audio)
    llm_score = deepseek_v3.score(candidate_text, ground_truth)
    human_scores = [annotator.rates[dim] for annotator in annotators for dim in dimensions]
    return {'WER': wer, 'Accuracy': acc, 'SIM': sim, 'LLM-score': llm_score, 'Human': mean(human_scores)}
```

## Common pitfalls

- Table 4 explicitly notes that different results for the same model come from different evaluation sources, potentially indicating different inference configurations.
- TTS similarity scores marked with (*) use a lightly fine-tuned similarity model, not the standard open-source speaker verification metric.
- The custom instruction-following benchmark converts text prompts to audio via a TTS pipeline before evaluation, so TTS quality indirectly affects instruction-following scores.

## Evidence (verbatim from paper)

> For ASR, we adopt word error rate (WER) as the primary metric, testing on both Chinese and English benchmarks, including Fleurs-zh *[fleurs]* and LibriSpeech-clean *[librispeech]*. While instruction-following with spoken input is addressed separately in Section[4.3], we also include LlamaQuestions *[llamaquestions]* as a speech-based QA benchmark, reporting accuracy.

## Citation

```bibtex
@misc{yao2025flmaudio,
  title={FLM-Audio: Natural Monologues Improves Native Full-Duplex Chatbots via Dual Training},
  author={Yao et al. (2025)},
  year={2025},
  note={arXiv:2509.02521}
}
```

- arXiv: 2509.02521