Fun Audio Chat Eval

---
name: fun-audio-chat-eval
description: Evaluates a large audio language model's capabilities across spoken question answering, audio understanding, speech recognition, function calling, and instruction following. It probes the model's ability to process speech inputs, generate text/speech outputs, and adhere to complex voice instructions while maintaining speech quality and safety. Use when the user wants to benchmark on VoiceBench, OpenAudioBench, UltraEval-Audio, MMAU, MMAU-Pro, MMSU, Librispeech, Common Voice, Speech-ACEBench, Speech-BFCL, Speech-SmartInteract, VStyle, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.20156
  bibtex_key: tongyifunteam2025funaudiachat
  confidence: high
---

# fun-audio-chat-eval

> Fun-Audio-Chat Technical Report — Tongyi Fun Team et al. (2025) (arXiv:2512.20156, 2025)

## What this evaluates

Evaluates a large audio language model's capabilities across spoken question answering, audio understanding, speech recognition, function calling, and instruction following. It probes the model's ability to process speech inputs, generate text/speech outputs, and adhere to complex voice instructions while maintaining speech quality and safety.

## Datasets

- **VoiceBench** — total ?; splits: test (-1)
- **OpenAudioBench** — total ?; splits: test (-1); HF `baichuan-inc/OpenAudioBench`
- **UltraEval-Audio** — total ?; splits: test (-1); repo https://github.com/OpenBMB/UltraEval-Audio
- **MMAU** — total ?; splits: test (-1)
- **MMAU-Pro** — total ?; splits: test (-1)
- **MMSU** — total ?; splits: test (-1)
- **Librispeech** — total ?; splits: test (-1)
- **Common Voice** — total ?; splits: test (-1)
- **Speech-ACEBench** — total ?; splits: test (-1)
- **Speech-BFCL** — total ?; splits: test (-1)
- **Speech-SmartInteract** — total ?; splits: test (-1); HF `FunAudioLLM/SpeechFCEval`
- **VStyle** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Calculated as the percentage of correctly predicted answers out of the total number of instances. Used for close-ended QA, audio understanding, and function calling tasks.
- `G-Eval` — range: percent
  - An LLM-based evaluation metric that scores the quality of open-ended responses against reference answers or criteria. Scores are normalized to percentages in this paper.
- `ASR-WER` — range: percent
  - Word Error Rate computed by transcribing the model-generated speech using Whisper-v3-large and comparing it against the ground-truth text. Formula: (S+D+I)/N.
- `UTMOS` — range: [1, 5]
  - Self-reported Mean Opinion Score for overall speech quality, rated on a 1-5 scale.
- `Refusal Rate` — range: percent
  - Percentage of safety-benchmark prompts where the model correctly refuses to generate a harmful or unsafe response.
- `LALM evaluation score` — range: [1, 5]
  - Scores assigned by a Large Audio Language Model acting as a judge, evaluating responses on a 1-5 scale across dimensions like acoustic attributes, instruction following, role-play, and empathy.

## Input / output format

**Input**: Speech audio input paired with text instructions or questions.

**Output**: Text responses and/or synthesized speech audio.

## Scoring recipe

```python
def compute_metrics(predictions, gold, task_type):
    if task_type == 'close_ended_qa':
        return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif task_type == 'open_ended_qa':
        return g_eval_score(predictions, gold)  # LLM-based
    elif task_type == 'safety':
        return refusal_rate(predictions)
    elif task_type == 'speech_quality':
        asr_transcripts = whisper_v3_large.predict(predictions)
        wer = word_error_rate(asr_transcripts, gold_text)
        utmos = utmos_score(predictions)
        return wer, utmos
    elif task_type == 'function_calling':
        return execute_and_check(predictions, gold)
    elif task_type == 'instruction_following':
        return lalm_judge_score(predictions, gold)
    return None
```

## Common pitfalls

- G-Eval relies on an external LLM judge, making results sensitive to the judge model/version and potentially non-deterministic across runs.
- ASR-WER is computed using Whisper-v3-large specifically; results are not directly comparable to WER values measured with other ASR models or versions.
- The internal test set for voice empathy is proprietary and not publicly available, limiting full reproducibility of those specific metrics.
- UTMOS is reported as self-evaluated scores rather than independently verified by third-party human listeners.

## Evidence (verbatim from paper)

> Evaluations adhere to the established protocols for each respective benchmark. For S→T and S→S evaluations on Spoken Question Answering benchmarks, we use different metrics depending on the task type: (1) Accuracy is used for close-ended QA tasks including Llama Q., Reasoning QA, TriviaQA, Web Q., SD-QA, MMSU, OpenBookQA, and IFEval; (2) G-Eval is used for open-ended QA tasks including AlpacaEval and CommonEval, which employs LLM-based evaluation to assess response quality; (3) Refusal Rate is reported for AdvBench to measure safety compliance.

## Citation

```bibtex
@misc{tongyifunteam2025funaudiachat,
  title={Fun-Audio-Chat Technical Report},
  author={Tongyi Fun Team et al. (2025)},
  year={2025},
  note={arXiv:2512.20156}
}
```

- arXiv: 2512.20156