Kimi Audio Eval

---
name: kimi-audio-eval
description: Evaluates an audio foundation model's capabilities across automatic speech recognition, general audio understanding, audio-to-text conversational reasoning, and end-to-end speech conversation. Use when the user wants to benchmark on LibriSpeech, FLEURS, AISHELL-1, AISHELL-2, WenetSpeech, Kimi-ASR Internal Testset, MMAU, ClothoAQA, VocalSound, Nonspeech7k, MELD, TUT2017, CochlScene, OpenAudioBench, VoiceBench, or asks about evaluating this task. Reports Word Error Rate (WER).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.18425
  bibtex_key: kimiteam2025kimi-audio
  confidence: high
---

# kimi-audio-eval

> Kimi-Audio Technical Report — KimiTeam et al. (2025) (arXiv:2504.18425, 2025)

## What this evaluates

Evaluates an audio foundation model's capabilities across automatic speech recognition, general audio understanding, audio-to-text conversational reasoning, and end-to-end speech conversation.

## Datasets

- **LibriSpeech** — total ?; splits: test-clean (-1), test-other (-1)
- **FLEURS** — total ?; splits: en (-1)
- **AISHELL-1** — total ?; splits: train (-1)
- **AISHELL-2** — total ?; splits: train (-1)
- **WenetSpeech** — total ?; splits: test-meeting (-1), test-net (-1)
- **Kimi-ASR Internal Testset** — total ?; splits: subset1 (-1), subset2 (-1)
- **MMAU** — total ?; splits: sound (-1), speech (-1)
- **ClothoAQA** — total ?; splits: dev (-1)
- **VocalSound** — total ?; splits: test (-1)
- **Nonspeech7k** — total ?; splits: test (-1)
- **MELD** — total ?; splits: test (-1)
- **TUT2017** — total ?; splits: test (-1)
- **CochlScene** — total ?; splits: dev (-1)
- **OpenAudioBench** — total ?; splits: test (-1)
- **VoiceBench** — total ?; splits: test (-1)

## Metrics

- `Word Error Rate (WER)` **(primary)** — range: percent
  - 100 * (S + D + I) / N, where S=substitutions, D=deletions, I=insertions, N=total words in reference. Calculated using a standardized implementation based on Qwen-2-Audio with consistent text normalization.
- `Accuracy / Benchmark-specific scores` — range: percent
  - Percentage of correct predictions for classification tasks; LLM-judged or exact-match scores for open-ended QA and reasoning tasks.
- `Human Rating` — range: [1, 5]
  - Average score on a 1-5 Likert scale across dimensions: speed control, accent control, emotion control, empathy, and style control.

## Input / output format

**Input**: Raw audio files (speech or environmental) paired with task-specific text prompts or questions.

**Output**: Text transcripts (ASR), categorical labels or free-form text answers (understanding/chat), or synthesized audio waveforms (conversation).

## Scoring recipe

```python
def score_asr(preds, golds):
    # Standardized WER with consistent text normalization
    return sum(wer(p, g) for p, g in zip(preds, golds)) / len(golds)

def score_understanding(preds, golds, task):
    if task in ['MMAU', 'VocalSound', 'Nonspeech7k', 'MELD', 'TUT2017', 'CochlScene']:
        return accuracy(preds, golds)
    else:
        return llm_judge_score(preds, golds, judge='GPT-4o-mini')

def score_conversation(preds, golds):
    # Human evaluation on 1-5 scale
    return average_human_rating(preds)
```

## Common pitfalls

- Inconsistent WER calculation due to varying text normalization practices across different toolkits.
- Relying solely on exact string matching for open-ended audio QA fails to capture semantic correctness of complex LLM responses.
- High sensitivity of model performance to inference parameters like decoding temperature, system prompts, and task prompts.

## Evidence (verbatim from paper)

> We implement a standardized WER calculation (based on Qwen-2-Audio) and integrate GPT-4o-mini as an intelligent judge (following chen2024voicebench) for tasks like audio question answering. This approach overcomes the limitations of inconsistent metrics and simplistic string matching, enabling fair comparison.

## Citation

```bibtex
@misc{kimiteam2025kimi-audio,
  title={Kimi-Audio Technical Report},
  author={KimiTeam et al. (2025)},
  year={2025},
  note={arXiv:2504.18425}
}
```

- arXiv: 2504.18425