Covocheval

---
name: covocheval
description: Evaluates zero-shot conversational voice cloning systems on their ability to generate natural, expressive speech that matches a target speaker's timbre and spontaneous style without prior training on the target speaker. It measures pronunciation accuracy, speaker similarity, and subjective qualities like naturalness, quality, and spontaneous style. Use when the user wants to benchmark on HQ-Conversations / CoVoC Test Prompts, or asks about evaluating this task. Reports FS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.00064
  bibtex_key: xia2024covocheval
  confidence: high
---

# covocheval

> The ISCSLP 2024 Conversational Voice Clone (CoVoC) Challenge: Tasks, Results and Findings — Xia et al. (2024) (arXiv:2411.00064, 2024)

## What this evaluates

Evaluates zero-shot conversational voice cloning systems on their ability to generate natural, expressive speech that matches a target speaker's timbre and spontaneous style without prior training on the target speaker. It measures pronunciation accuracy, speaker similarity, and subjective qualities like naturalness, quality, and spontaneous style.

## Datasets

- **HQ-Conversations / CoVoC Test Prompts** — total ?; splits: test (-1)

## Metrics

- `CER` — range: percent
  - Character Error Rate computed between the ground truth transcript and the transcript recognized by an external ASR model (paraformer-large) from the synthesized audio.
- `SIM` — range: [-1, 1]
  - Cosine similarity between speaker embeddings extracted from the reference speech and the generated speech using the Resemblyzer tool.
- `SN` — range: [1, 5]
  - Mean Opinion Score (1-5) for speech naturalness, assessing pronunciation correctness, ambiguity, tone changes, and pause naturalness.
- `SQ` — range: [1, 5]
  - Mean Opinion Score (1-5) for speech quality, assessing electronic distortion and voice clarity.
- `SS` — range: [1, 5]
  - Mean Opinion Score (1-5) for speaker similarity, assessing timbre and speaking style match to the target speaker.
- `SSS` — range: [1, 5]
  - Mean Opinion Score (1-5) for speech spontaneous style, assessing colloquial words, laughter, non-rhythmic pauses, stress, and rhythm.
- `FS` **(primary)** — range: [1, 5]
  - Final Score computed as the unweighted average of the four MOS scores: FS = 0.25×SN + 0.25×SQ + 0.25×SS + 0.25×SSS.

## Input / output format

**Input**: Reference audio clip (target speaker) and target text transcript.

**Output**: Synthesized audio waveform matching the target text in the target speaker's voice.

## Scoring recipe

```python
def compute_metrics(reference_audio, predicted_audio, gold_transcript):
    # Objective
    cer = compute_cer(gold_transcript, asr_model(predicted_audio))
    sim = cosine_similarity(resemblyzer_embed(reference_audio), resemblyzer_embed(predicted_audio))
    # Subjective (averaged across 10 raters)
    sn, sq, ss, sss = [rater_score(sample) for rater_score in raters]
    fs = 0.25 * sn + 0.25 * sq + 0.25 * ss + 0.25 * sss
    return {"CER": cer, "SIM": sim, "SN": sn, "SQ": sq, "SS": ss, "SSS": sss, "FS": fs}
```

## Common pitfalls

- CER requires an external ASR model to transcribe the synthesized audio before computing error; it is not a direct string-to-string comparison.
- All audio samples must be resampled to 16 kHz before objective metric computation.
- The Final Score (FS) is a simple unweighted average of the four MOS aspects, despite the formula notation implying a weighted sum.

## Evidence (verbatim from paper)

> The objective evaluation consisted of two aspects: pronunciation accuracy and timbre similarity. We employ Character Error Rate (CER) and cosine similarity for evaluation: Character Error Rate: CER is computed between the ground truth transcript and the recognized transcript. We use an open-source paraformer-large model to recognize the synthesized speech into the corresponding transcription. Speaker Similarity (SIM): We employ the Resemblyzer tool to extract speaker embedding and compute the cosine similarity between the reference speech and generated speech.

## Citation

```bibtex
@misc{xia2024covocheval,
  title={The ISCSLP 2024 Conversational Voice Clone (CoVoC) Challenge: Tasks, Results and Findings},
  author={Xia et al. (2024)},
  year={2024},
  note={arXiv:2411.00064}
}
```

- arXiv: 2411.00064