Cameo Eval

---
name: cameo-eval
description: Evaluates speech emotion recognition (SER) models across multiple languages and emotional states. It probes a model's ability to map raw audio inputs to discrete emotional categories without relying on speaker or language metadata. Use when the user wants to benchmark on CAMEO, or asks about evaluating this task. Reports macro-averaged F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.11051
  bibtex_key: christop2025cameo
  confidence: high
---

# cameo-eval

> CAMEO: Collection of Multilingual Emotional Speech Corpora — Christop et al. (2025) (arXiv:2505.11051, 2025)

## What this evaluates

Evaluates speech emotion recognition (SER) models across multiple languages and emotional states. It probes a model's ability to map raw audio inputs to discrete emotional categories without relying on speaker or language metadata.

## Datasets

- **CAMEO** — total 41265; splits: test (-1)

## Metrics

- `macro-averaged F1 score` **(primary)** — range: [0, 1]
  - Computes the unweighted mean of the F1 score across all emotion classes, treating each class equally regardless of support.
- `weighted F1 score` — range: [0, 1]
  - Computes the F1 score for each class and averages them, weighting each class by its support (number of true instances).
- `accuracy` — range: [0, 1]
  - The proportion of correctly predicted emotional labels out of the total number of instances.

## Input / output format

**Input**: A textual instruction prompt followed by a single audio file. No additional metadata (speaker identity, gender, or language) is provided during inference.

**Output**: A single-word string corresponding to the predicted emotional state (e.g., 'anger', 'happiness').

## Scoring recipe

```python
def post_process_and_score(pred, gold_labels, threshold=0.57):
    words = pred.lower().split()
    label_scores = {lbl: 0.0 for lbl in gold_labels}
    for word in words:
        for lbl in gold_labels:
            ratio = levenshtein_ratio(word, lbl)
            if ratio >= threshold:
                label_scores[lbl] += ratio
    predicted_label = max(label_scores, key=label_scores.get)
    return 1.0 if predicted_label == gold_labels[0] else 0.0
# Aggregate per-class TP/FP/FN to compute macro/weighted F1 and accuracy
```

## Common pitfalls

- Models frequently output adjectives (e.g., 'angry') instead of the required noun labels (e.g., 'anger'), requiring fuzzy matching rather than exact string comparison.
- Generative models often produce multi-word descriptive responses instead of single-word predictions, necessitating word-splitting and score aggregation.
- The 0.57 Levenshtein threshold is specifically calibrated to noun-adjective pairs for the seven primary emotions and may not generalize to other label formats or languages.

## Evidence (verbatim from paper)

> To capture the performance of the models across different emotions and languages, the evaluation employs widely used metrics, such as macro-averaged F1 score, weighted F1 score and accuracy. ... If a generated response is not an exact match for any of the labels, it is normalized and split into words. Then, the Levenshtein ratio between each target label and each word in the generated response is calculated. Similarity scores below a predefined threshold of 0.57 are filtered out for each label. ... The label with the highest aggregated similarity score is selected as the best match.

## Citation

```bibtex
@misc{christop2025cameo,
  title={CAMEO: Collection of Multilingual Emotional Speech Corpora},
  author={Christop et al. (2025)},
  year={2025},
  note={arXiv:2505.11051}
}
```

- arXiv: 2505.11051