Dualturn Turn Taking Eval

---
name: dualturn-turn-taking-eval
description: This benchmark evaluates a model's ability to predict conversational turn-taking dynamics and agent actions from dual-channel speech audio. It probes the system's capacity to anticipate speech boundaries, detect backchannels, and classify continuous turn-taking states without relying on explicit silence timeouts or external labels. Use when the user wants to benchmark on otoSpeech, Switchboard, or asks about evaluating this task. Reports wF1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.08216
  bibtex_key: rajaa2026dualturn
  confidence: high
---

# dualturn-turn-taking-eval

> DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining — Rajaa (2026) (arXiv:2603.08216, 2026)

## What this evaluates

This benchmark evaluates a model's ability to predict conversational turn-taking dynamics and agent actions from dual-channel speech audio. It probes the system's capacity to anticipate speech boundaries, detect backchannels, and classify continuous turn-taking states without relying on explicit silence timeouts or external labels.

## Datasets

- **otoSpeech** — total ?; splits: test (113)
- **Switchboard** — total ?; splits: test (138)

## Metrics

- `wF1` **(primary)** — range: [0, 1]
  - Weighted F1 score averaged across five agent action classes (Continue, Backchannel, Turn-Shift, etc.). Computed as the sum of per-class F1 scores multiplied by their respective support (true positive count), divided by the total support.
- `AUC avg` — range: [0, 1]
  - Average Area Under the Receiver Operating Characteristic Curve for word-level turn prediction across three classes: Continue (C), Backchannel (B), and Turn-Shift (T).
- `BC F1` — range: [0, 1]
  - F1 score specifically computed for the Backchannel agent action class, measuring the balance between precision and recall for sparse backchannel events.

## Input / output format

**Input**: Dual-channel raw audio streams (or frozen Mimi codec embeddings) representing two speakers in a conversation.

**Output**: Per-channel probabilities or discrete predictions for six turn-taking signals (EOT, HOLD, backchannel, speech onset, etc.), which are linearly aggregated into five agent action classes. For word-level evaluation, binary/multi-class predictions for Continue, Backchannel, and Turn-Shift per word.

## Scoring recipe

```python
def compute_wF1(gold_actions, pred_actions):
    classes = ['Continue', 'Backchannel', 'Turn-Shift', 'Shift', 'Hold']
    f1s, supports = [], []
    for c in classes:
        tp = sum(1 for g, p in zip(gold_actions, pred_actions) if g == c and p == c)
        fp = sum(1 for g, p in zip(gold_actions, pred_actions) if g != c and p == c)
        fn = sum(1 for g, p in zip(gold_actions, pred_actions) if g == c and p != c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
        supports.append(tp + fn)
    return sum(f * s for f, s in zip(f1s, supports)) / sum(supports)

def compute_auc_avg(gold_classes, pred_scores):
    auc_C = roc_auc_score([1 if g == 'C' else 0 for g in gold_classes], pred_scores['C'])
    auc_B = roc_auc_score([1 if g == 'B' else 0 for g in gold_classes], pred_scores['B'])
    auc_T = roc_auc_score([1 if g == 'T' else 0 for g in gold_classes], pred_scores['T'])
    return (auc_C + auc_B + auc_T) / 3
```

## Common pitfalls

- Confusing the 4-second action definition used in the main evaluation with VAP's original 1-second bidirectional event protocol; the paper explicitly tests both to prove gains come from learned representations, not label window adjustments.
- Assuming backchannel (BC) detection is trivial or well-covered by baselines; BC events are highly sparse (<8% of data) and standard models like VAP achieve 0.000 F1 without a dedicated signal.
- Misinterpreting the two-stage training protocol: Stage-1 is purely generative speech pretraining, while Stage-2 drops the generative loss to focus on signal prediction. Adding auxiliary generative loss in Stage-2 actively harms performance by suppressing sparse task learning.

## Evidence (verbatim from paper)

> DualTurn outperformed VAP on all five agent action classes across both evaluation datasets (wF1 0.633 vs. 0.389 on Switchboard, wF1 0.707 vs. 0.461 on otoSpeech), Table[3](#S3.T3 "Table 3 ‣ 3.2 Main Results ‣ 3 Experiments ‣ DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining"). Backchannel detection shows the largest gap, where VAP achieves BC F1 \= 0.000 because it has no dedicated BC signal*[inoue2025backchannel]*. VAP cannot distinguish BC from CT even with an LR probe on its predicted projection probabilities. Trained entirely on self-supervised labels like VAP, DualTurn achieves BC F1 \= 0.349 (chance F1 $\approx$ 0.080) in predicting "the agent should backchannel now".

## Citation

```bibtex
@misc{rajaa2026dualturn,
  title={DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining},
  author={Rajaa (2026)},
  year={2026},
  note={arXiv:2603.08216}
}
```

- arXiv: 2603.08216