Fish Audio S2 Eval

---
name: fish-audio-s2-eval
description: Evaluates speech synthesis models on intelligibility, speaker similarity, and long-form generation across multiple languages. It also assesses subjective qualities like naturalness, instruction-following, and human-level indistinguishability using automated LLM-as-a-Judge and Audio Turing Test frameworks. Use when the user wants to benchmark on Seed-TTS-Eval, CV3-Eval, Minimax Multilingual Testset, Long-TTS-Eval, Audio Turing Test, Emergent TTS Eval, or asks about evaluating this task. Reports WER (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.08823
  bibtex_key: [REDACTED]
  confidence: high
---

# fish-audio-s2-eval

> Fish Audio S2 Technical Report — Liao et al. (2026) (arXiv:2603.08823, 2026)

## What this evaluates

Evaluates speech synthesis models on intelligibility, speaker similarity, and long-form generation across multiple languages. It also assesses subjective qualities like naturalness, instruction-following, and human-level indistinguishability using automated LLM-as-a-Judge and Audio Turing Test frameworks.

## Datasets

- **Seed-TTS-Eval** — total ?; splits: test-zh (-1), test-en (-1), zh-hard (-1)
- **CV3-Eval** — total ?; splits: multilingual-9lang (-1)
- **Minimax Multilingual Testset** — total ?; splits: 24-languages (-1)
- **Long-TTS-Eval** — total ?; splits: English (-1), Chinese (-1)
- **Audio Turing Test** — total 499; splits: ATT-Corpus (499)
- **Emergent TTS Eval** — total ?; splits: 5-dimensions (-1)

## Metrics

- `WER (%)` **(primary)** — range: percent
  - Word Error Rate: ratio of edit operations (insertions, deletions, substitutions) between the ASR-transcribed output and the reference text.
- `CER (%)` — range: percent
  - Character Error Rate: same as WER but computed at the character level, primarily used for Chinese evaluation.
- `SIM (Speaker Similarity)` — range: [0, 1]
  - Cosine similarity between speaker embeddings extracted from generated audio (3s chunks, 1.5s hop) and reference audio using WavLM-large. Reported as mean and standard deviation across chunks.
- `Win-Rate (%)` — range: percent
  - Percentage of pairwise comparisons where the model's output is preferred over a baseline model by the LLM-as-a-Judge or evaluation framework.
- `ATT Posterior Mean` — range: [0, 1]
  - Posterior probability of the 'Human' class from the Audio Turing Test evaluation, estimated via Bayesian inference using the Auto-ATT model.

## Input / output format

**Input**: Reference audio clip and text prompt (optionally with instruction/style tags).

**Output**: Synthesized audio waveform.

## Scoring recipe

```python
def compute_wer_cer(generated_audio, ref_text, lang):
    asr = 'Whisper-large-v3' if lang == 'en' else 'Paraformer-zh'
    transcribed = asr.transcribe(generated_audio, chunk_size=28)
    return edit_distance(transcribed, ref_text) / len(ref_text)

def compute_sim(gen_audio, ref_audio):
    chunks = chunk_audio(gen_audio, duration=3.0, hop=1.5)
    ref_emb = wavlm_large.encode(ref_audio)
    sims = [cosine_similarity(wavlm_large.encode(c), ref_emb) for c in chunks]
    return {'mean': mean(sims), 'std': std(sims)}

def compute_att_posterior(audio):
    return auto_att_model.predict_proba(audio)['Human']
```

## Common pitfalls

- WER/CER values are highly dependent on the specific ASR model (Whisper vs Paraformer) and chunking strategy (28s non-overlapped) used for transcription.
- SIM scores require strict adherence to the 3s chunking with 1.5s hop and WavLM-large embeddings; deviating from this changes the scale.
- ATT results are reported as posterior means from a Bayesian model, not simple classification accuracy, making direct comparison with standard accuracy metrics invalid.

## Evidence (verbatim from paper)

> We assess voice-cloning intelligibility on Seed-TTS-Eval using WER over the test-zh, test-en, and ZH-hard splits. WER is computed by transcribing synthesized audio with Whisper-large-v3 (Radford et al., 2023) for English and Paraformer-zh Gao et al. (2023) for Chinese, following the benchmark protocol; results are reported in Table 1.

## Citation

```bibtex
@misc{liao2026fishaudios2,
  title={Fish Audio S2 Technical Report},
  author={Liao et al. (2026)},
  year={2026},
  note={arXiv:2603.08823}
}
```

- arXiv: 2603.08823