Full Duplex Bench Eval

---
name: full-duplex-bench-eval
description: Evaluates real-time interactive behaviors in full-duplex spoken dialogue models. It specifically probes turn-taking, pause handling, backchanneling, and interruption management capabilities without relying on human studies. Use when the user wants to benchmark on Full-Duplex-Bench, or asks about evaluating this task. Reports descriptive metrics.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.04721
  bibtex_key: lin2025fullduplexbench
  confidence: medium
---

# full-duplex-bench-eval

> Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities — Lin et al. (2025) (arXiv:2503.04721, 2025)

## What this evaluates

Evaluates real-time interactive behaviors in full-duplex spoken dialogue models. It specifically probes turn-taking, pause handling, backchanneling, and interruption management capabilities without relying on human studies.

## Datasets

- **Full-Duplex-Bench** — total ?; splits: test (-1); repo https://github.com/DanielLin94144/Full-Duplex-Bench

## Metrics

- `descriptive metrics` **(primary)** — range: other
  - Time-synchronous audio alignment combined with descriptive, reproducible measures to evaluate pause handling, backchanneling, turn-taking, and interruption management.

## Input / output format

**Input**: Audio input processed in 30 ms chunks at 16 kHz PCM-16 format, with server-side VAD for segmentation.

**Output**: Audio responses aligned with the original input duration, preserving silence in regions where no response is produced.

## Scoring recipe

```python
def score(predictions, gold):
    # Align prediction audio with gold input audio time-synchronously
    aligned_pred = align_audio_time_synchronous(predictions, gold)
    # Compute descriptive metrics for turn-taking, pause handling, backchanneling, interruption
    metrics = compute_descriptive_metrics(aligned_pred, gold)
    return metrics
```

## Common pitfalls

- Relying on human studies or dataset-specific training instead of automated, generalizable metrics.
- Ignoring real-time streaming constraints (e.g., 30 ms chunking and server-side VAD) when evaluating model responsiveness.

## Evidence (verbatim from paper)

> It uses time-synchronous audio alignment and descriptive, reproducible metrics to enable objective, large-scale comparison of models without relying on human studies or dataset-specific training, addressing critical gaps in current evaluation methods that lack generalizability and automation.

## Citation

```bibtex
@misc{lin2025fullduplexbench,
  title={Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities},
  author={Lin et al. (2025)},
  year={2025},
  note={arXiv:2503.04721}
}
```

- arXiv: 2503.04721