Deepdialogue Ser Eval

---
name: deepdialogue-ser-eval
description: Evaluates the emotional expressivity and transferability of a generated multi-turn spoken dialogue dataset by training speech emotion recognition models and measuring their classification performance on held-out and zero-shot test sets. Use when the user wants to benchmark on DeepDialogue (SER subset), RAVDESS, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.19978
  bibtex_key: koudounas2025deepdialogue
  confidence: high
---

# deepdialogue-ser-eval

> DeepDialogue: A Multi-Turn Emotionally-Rich Spoken Dialogue Dataset — Koudounas et al. (2025) (arXiv:2505.19978, 2025)

## What this evaluates

Evaluates the emotional expressivity and transferability of a generated multi-turn spoken dialogue dataset by training speech emotion recognition models and measuring their classification performance on held-out and zero-shot test sets.

## Datasets

- **DeepDialogue (SER subset)** — total ?; splits: train (-1), test (-1)
- **RAVDESS** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard classification accuracy: the proportion of correctly predicted emotion labels out of the total number of instances.
- `macro F1-score` — range: percent
  - Macro-averaged F1-score: the unweighted mean of the F1-scores computed independently for each emotion category.

## Input / output format

**Input**: Audio recordings of dialogue turns paired with ground-truth emotion labels.

**Output**: Predicted emotion category label for each audio instance.

## Scoring recipe

```python
def compute_metrics(preds, golds, num_classes):
    accuracy = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    f1_scores = []
    for c in range(num_classes):
        tp = sum(1 for p, g in zip(preds, golds) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, golds) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, golds) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    macro_f1 = sum(f1_scores) / num_classes
    return accuracy, macro_f1
```

## Common pitfalls

- The training subset is artificially balanced (1,000 turns per emotion category), which may inflate performance compared to real-world imbalanced dialogue data.
- Zero-shot evaluation on RAVDESS suffers from domain shift, so the reported accuracy drop does not necessarily reflect model failure but rather dataset mismatch.
- The exact number of emotion categories is not explicitly stated in the text, making it difficult to verify the macro F1 calculation or compare with other benchmarks.

## Evidence (verbatim from paper)

> All models demonstrate strong performance (around 90%) in both accuracy and macro F1-score when evaluated on a held-out test set from DeepDialogue. To assess the generalizability of the learned representations, we evaluate the best-performing model in a zero-shot setting on the RAVDESS dataset, which shares the same emotion label distribution. As expected, we observe a performance drop due to domain shift; however, results remain strong. Our zero-shot HuBERT-DD achieves an accuracy of 56.6%, closely approaching the performance of a linear-probing baseline (HuBERT-LP, 65.3%) trained directly on RAVDESS.

## Citation

```bibtex
@misc{koudounas2025deepdialogue,
  title={DeepDialogue: A Multi-Turn Emotionally-Rich Spoken Dialogue Dataset},
  author={Koudounas et al. (2025)},
  year={2025},
  note={arXiv:2505.19978}
}
```

- arXiv: 2505.19978