Ser Multilingual Probing Eval

---
name: ser-multilingual-probing-eval
description: Evaluates pre-trained speech models' ability to recognize emotions in audio across multiple languages. It specifically tests how internal layer representations and feature aggregation strategies impact classification performance. Use when the user wants to benchmark on AESDD, CaFE, EmoDB, EMOVO, IEMOCAP, RAVDESS, ShEMO, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.08713
  bibtex_key: singh2023decodingemotions
  confidence: high
---

# ser-multilingual-probing-eval

> Decoding Emotions: A comprehensive Multilingual Study of Speech Models for Speech Emotion Recognition — Anant Singh et al. (2023) (arXiv:2308.08713, 2023)

## What this evaluates

Evaluates pre-trained speech models' ability to recognize emotions in audio across multiple languages. It specifically tests how internal layer representations and feature aggregation strategies impact classification performance.

## Datasets

- **AESDD** — total ?; splits: test (-1)
- **CaFE** — total ?; splits: test (-1)
- **EmoDB** — total ?; splits: test (-1)
- **EMOVO** — total ?; splits: test (-1)
- **IEMOCAP** — total ?; splits: test (-1)
- **RAVDESS** — total ?; splits: test (-1)
- **ShEMO** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - The proportion of correctly predicted emotion labels out of the total number of instances. Calculated as (number of correct predictions) / (total predictions).

## Input / output format

**Input**: Raw speech audio files.

**Output**: Discrete emotion category label.

## Scoring recipe

```python
correct = sum(1 for p, g in zip(predictions, gold) if p == g)
total = len(gold)
return (correct / total) * 100
```

## Common pitfalls

- Relying on final transformer layers or aggregated multi-layer features instead of center layers, which contain the richest contextual features for emotion recognition.
- Using ASR-finetuned models, which discard prosodic information crucial for emotion recognition and consistently underperform non-ASR counterparts.
- Evaluating on a single run without averaging over multiple trials, as the protocol requires each trial to be run five times for statistical validity.

## Evidence (verbatim from paper)

> The exact values of classification accuracies for the dense probing head with the corresponding best layer for each model are shown in Table 3.

## Citation

```bibtex
@misc{singh2023decodingemotions,
  title={Decoding Emotions: A comprehensive Multilingual Study of Speech Models for Speech Emotion Recognition},
  author={Anant Singh et al. (2023)},
  year={2023},
  note={arXiv:2308.08713}
}
```

- arXiv: 2308.08713