Unseen Speaker Ser Eval

---
name: unseen-speaker-ser-eval
description: Evaluates a model's ability to recognize emotions in speech from speakers it has never encountered during training. It probes cross-speaker generalization and robustness to acoustic variability across multiple languages and recording conditions. Use when the user wants to benchmark on CREMA-D, IEMOCAP, RAVDESS, EmoDB, CaFE, BhavVani, or asks about evaluating this task. Reports WF1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.08931
  bibtex_key: goel2024camu
  confidence: high
---

# unseen-speaker-ser-eval

> Exploring Multilingual Unseen Speaker Emotion Recognition: Leveraging Co-Attention Cues in Multitask Learning — Goel et al. (2024) (arXiv:2406.08931, 2024)

## What this evaluates

Evaluates a model's ability to recognize emotions in speech from speakers it has never encountered during training. It probes cross-speaker generalization and robustness to acoustic variability across multiple languages and recording conditions.

## Datasets

- **CREMA-D** — total ?; splits: train (-1), val (-1), test (-1)
- **IEMOCAP** — total ?; splits: train (-1), val (-1), test (-1)
- **RAVDESS** — total ?; splits: train (-1), val (-1), test (-1)
- **EmoDB** — total ?; splits: train (-1), val (-1), test (-1)
- **CaFE** — total ?; splits: train (-1), val (-1), test (-1)
- **BhavVani** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `WF1` **(primary)** — range: [0, 1]
  - Weighted F1 score: harmonic mean of precision and recall per class, averaged across all classes weighted by their support (number of true instances per class). Accounts for class imbalance.
- `WA` — range: [0, 1]
  - Weighted accuracy: proportion of correctly predicted samples, weighted by class support. Accounts for class imbalance.

## Input / output format

**Input**: Raw audio waveform, optionally processed into MFCCs (40 coefficients, hop length 160) and spectrograms (Hamming window, length 40, hop 10, FFT 800), or 1D embeddings from pre-trained encoders (Wav2Vec2.0, WavLM, HuBERT, Whisper) via average pooling of 2D time-frequency outputs.

**Output**: Categorical emotion label

## Scoring recipe

```python
def compute_metrics(y_true, y_pred, classes):
    correct = sum(1 for t, p in zip(y_true, y_pred) if t == p)
    wa = correct / len(y_true)
    f1_scores = []
    for c in classes:
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
        fp = sum(1 for t, p in zip(y_true, y_pred) if t != c and p == c)
        fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        support = sum(1 for t in y_true if t == c)
        f1_scores.append(f1 * support)
    wf1 = sum(f1_scores) / len(y_true)
    return wa, wf1
```

## Common pitfalls

- Using speaker-dependent splits instead of leave-speaker-out, which leaks acoustic identity and artificially inflates performance.
- Reporting standard accuracy instead of weighted metrics, which masks poor performance on minority emotion classes due to dataset imbalance.
- Averaging metrics per fold instead of pooling predictions across all 10 folds before computing the final score, leading to incorrect weighting.

## Evidence (verbatim from paper)

> For evaluating performance on unseen speakers, we followed 10-fold leave-speaker-out cross validation, wherein each dataset was segmented into 10 folds with each fold containing unique speakers... Note: WA stands for weighted accuracy and WF1 stands for weighted F1 score. These measures account for class imbalance.

## Citation

```bibtex
@misc{goel2024camu,
  title={Exploring Multilingual Unseen Speaker Emotion Recognition: Leveraging Co-Attention Cues in Multitask Learning},
  author={Goel et al. (2024)},
  year={2024},
  note={arXiv:2406.08931}
}
```

- arXiv: 2406.08931