Emogator Eval

---
name: emogator-eval
description: Evaluates machine learning models' ability to classify brief, nonverbal vocal bursts into discrete emotional states. It probes the limits of audio classification on highly ambiguous, short-duration human vocalizations across 30 affective categories. Use when the user wants to benchmark on EmoGator, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2301.00508
  bibtex_key: [REDACTED]
  confidence: high
---

# emogator-eval

> EmoGator: A New Open Source Vocal Burst Dataset with Baseline Machine Learning Classification Methodologies — Buhl (2023) (arXiv:2301.00508, 2023)

## What this evaluates

Evaluates machine learning models' ability to classify brief, nonverbal vocal bursts into discrete emotional states. It probes the limits of audio classification on highly ambiguous, short-duration human vocalizations across 30 affective categories.

## Datasets

- **EmoGator** — total 32130; splits: train (-1), val (-1), test (-1); repo https://github.com/fredbuhl/EmoGator

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - The unweighted mean of per-class F1 scores. Calculated as the average of precision and recall F1 for each of the 30 emotion categories, treating all classes equally regardless of support.

## Input / output format

**Input**: Raw audio waveform files representing short vocal bursts.

**Output**: A single predicted emotion category label from a predefined set of 30 classes.

## Scoring recipe

```python
def f1_score(preds, gold, classes):
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
    return sum(f1s) / len(f1s)
```

## Common pitfalls

- Data augmentation via independent pitch and tempo shifts consistently degraded model performance compared to the original training set.
- Preprocessing steps like denoising or trimming silence from audio endpoints reduced classification accuracy.
- Evaluating on reduced category subsets (e.g., 10 or 16 classes) artificially inflates F1 scores and does not reflect full-dataset capability.

## Evidence (verbatim from paper)

> For one-dimensional convolutional neural networks, the best results against the full dataset were with a 70% / 15% / 15% train/validation/test split, using an 18-layer 1D CNN based on [[31]], but with dropout layers after each convolution. For the full 30-category dataset, the average F1 score was 0.270.

## Citation

```bibtex
@misc{buhl2023emogator,
  title={EmoGator: A New Open Source Vocal Burst Dataset with Baseline Machine Learning Classification Methodologies},
  author={Buhl (2023)},
  year={2023},
  note={arXiv:2301.00508}
}
```

- arXiv: 2301.00508