Multidialog Eval

---
name: multidialog-eval
description: This benchmark evaluates the semantic coherence, acoustic fidelity, and audio-visual synchronization of end-to-end spoken dialogue systems that generate face-to-face conversational audio and video. It probes a model's ability to maintain contextually appropriate dialogue while producing synchronized multimodal outputs without relying on intermediate text representations. Use when the user wants to benchmark on MultiDialog, or asks about evaluating this task. Reports PPL.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.07867
  bibtex_key: park2024lets
  confidence: high
---

# multidialog-eval

> Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation — Park et al. (2024) (arXiv:2406.07867, 2024)

## What this evaluates

This benchmark evaluates the semantic coherence, acoustic fidelity, and audio-visual synchronization of end-to-end spoken dialogue systems that generate face-to-face conversational audio and video. It probes a model's ability to maintain contextually appropriate dialogue while producing synchronized multimodal outputs without relying on intermediate text representations.

## Datasets

- **MultiDialog** — total ?; splits: test (-1)

## Metrics

- `PPL` **(primary)** — range: other
  - Log-perplexity calculated using the DialoGPT model for each utterance and averaged across the test set. Lower values indicate better semantic quality.
- `BLEU` — range: [0, 1]
  - Standard n-gram overlap metric between transcribed synthesized text and ground truth transcripts.
- `FID` — range: other
  - Fréchet Inception Distance measuring the distributional difference between real and generated video features to assess visual quality.
- `SIM` — range: [0, 1]
  - Speaker similarity score computed using the WavLM-Base model to measure acoustic resemblance between target and generated speech.

## Input / output format

**Input**: Audio-visual (AV) speech tokens or raw audio-visual dialogue turns.

**Output**: Synthesized audio-visual dialogue turns (AV speech tokens), which are subsequently transcribed by an ASR model for semantic metric computation.

## Scoring recipe

```python
def compute_ppl(pred_audio_list, gold_text_list):
    # 1. Transcribe synthesized audio using specified ASR (Shi et al. 2021)
    pred_text_list = [asr.transcribe(audio) for audio in pred_audio_list]
    # 2. Compute log-perplexity per utterance using DialoGPT
    ppl_scores = []
    for pred_text in pred_text_list:
        log_probs = dialogpt_model.log_prob(pred_text)
        ppl_scores.append(math.exp(-log_probs / len(pred_text)))
    # 3. Average across test set
    return sum(ppl_scores) / len(ppl_scores)
```

## Common pitfalls

- Failing to transcribe synthesized audio with the specified ASR model before computing text-based metrics like PPL or BLEU.
- Comparing end-to-end AV systems against audio-only baselines without acknowledging the inherent modality mismatch.
- Using different LLM backbones (e.g., not DialoGPT) for PPL calculation, which breaks cross-method comparability.

## Evidence (verbatim from paper)

> We evaluate the semantic quality and the generation quality of both audio and video. For the semantic quality, we first generate transcriptions from the synthesized audio-visual output using an off-the-shelf ASR model Shi et al. ([2021]), and employ standard metrics used for text-based dialogue generation: log-perplexity (PPL), BLEU, METEOR, F1, D-1, and D-2. The log-perplexity is calculated using Dialo-GPT model Zhang et al. ([2019]) and it is calculated for each utterance and averaged across the test set.

## Citation

```bibtex
@misc{park2024lets,
  title={Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation},
  author={Park et al. (2024)},
  year={2024},
  note={arXiv:2406.07867}
}
```

- arXiv: 2406.07867