Iemocap Eval

---
name: iemocap-eval
description: Evaluates how categorical and continuous label ambiguity impacts the performance of unimodal emotion recognition models (text, audio, facial) on the IEMOCAP dataset. It tests whether filtering data by annotator agreement or VAD score dispersion yields cleaner evaluation signals. The protocol highlights the disconnect between rigid single-label benchmarks and the inherent ambiguity of affective data. Use when the user wants to benchmark on IEMOCAP, or asks about evaluating this task. Reports weighted F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.23017
  bibtex_key: larsen2026modelling
  confidence: high
---

# iemocap-eval

> Modelling Emotions is an Elusive Pursuit in Affective Computing — Larsen et al. (2026) (arXiv:2603.23017, 2026)

## What this evaluates

Evaluates how categorical and continuous label ambiguity impacts the performance of unimodal emotion recognition models (text, audio, facial) on the IEMOCAP dataset. It tests whether filtering data by annotator agreement or VAD score dispersion yields cleaner evaluation signals. The protocol highlights the disconnect between rigid single-label benchmarks and the inherent ambiguity of affective data.

## Datasets

- **IEMOCAP** — total ?; splits: test (-1)

## Metrics

- `weighted F1 score` **(primary)** — range: [0, 1]
  - Averages per-class F1 scores weighted by label frequency to account for class imbalance.

## Input / output format

**Input**: Audio, facial, and text modalities of utterances from the IEMOCAP dataset, paired with categorical emotion annotations and/or VAD scores.

**Output**: Predicted categorical emotion label for each utterance.

## Scoring recipe

```python
def weighted_f1(predictions, gold, classes):
    class_counts = Counter(gold)
    total = sum(class_counts.values())
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    weights = [class_counts[c] / total for c in classes]
    return sum(f * w for f, w in zip(f1_scores, weights))
```

## Common pitfalls

- Treating categorical emotion labels as accurate ground truth despite only ~20% full annotator agreement.
- Assuming that filtering by VAD score dispersion will improve model performance, as it actually decreases or shows no improvement.
- Expecting high agreement across text, audio, and facial modalities, when full agreement occurs in only 4.18% of utterances.

## Evidence (verbatim from paper)

> Model performance was evaluated using the weighted F1 score, which averages per-class F1 scores weighted by label frequency to account for class imbalance.

## Citation

```bibtex
@misc{larsen2026modelling,
  title={Modelling Emotions is an Elusive Pursuit in Affective Computing},
  author={Larsen et al. (2026)},
  year={2026},
  note={arXiv:2603.23017}
}
```

- arXiv: 2603.23017