Multimodal Emotion Recognition Eval

---
name: multimodal-emotion-recognition-eval
description: Evaluates a model's ability to recognize emotions in conversational video clips using text, audio, and visual modalities. It probes how well identity-preserving representations and state-space fusion capture emotion-relevant acoustic and facial dynamics across different dataset configurations. Use when the user wants to benchmark on MELD, IEMOCAP, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.14969
  bibtex_key: wang2025qualitycontrolled
  confidence: high
---

# multimodal-emotion-recognition-eval

> Quality-Controlled Multimodal Emotion Recognition in Conversations with Identity-Based Transfer Learning and MAMBA Fusion — Wang & Beigi (2025) (arXiv:2511.14969, 2025)

## What this evaluates

Evaluates a model's ability to recognize emotions in conversational video clips using text, audio, and visual modalities. It probes how well identity-preserving representations and state-space fusion capture emotion-relevant acoustic and facial dynamics across different dataset configurations.

## Datasets

- **MELD** — total ?; splits: test (-1)
- **IEMOCAP** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly predicted emotion labels out of the total number of instances.
- `W-F1` — range: [0, 1]
  - Weighted F1-score across all emotion classes, where each class's F1 is weighted by its support (number of true instances).

## Input / output format

**Input**: Multimodal conversational utterances comprising text transcripts, audio recordings, and video frames, optionally combined with speaker/face identity embeddings.

**Output**: Discrete emotion class label (7 classes for MELD: neutral, joy, sadness, anger, fear, disgust, surprise; 4 classes for IEMOCAP: neutral, sadness, anger, happy+excited).

## Scoring recipe

```python
def compute_metrics(preds, gold):
    accuracy = sum(p == g for p, g in zip(preds, gold)) / len(gold)
    classes = sorted(set(gold))
    weighted_f1 = 0.0
    total = len(gold)
    for c in classes:
        tp = sum(1 for p, g in zip(preds, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        weighted_f1 += f1 * sum(1 for g in gold if g == c)
    weighted_f1 /= total
    return accuracy, weighted_f1
```

## Common pitfalls

- Temporal misalignment between emotional peaks in facial expressions and extracted video frames can degrade visual performance.
- Fear and disgust are frequently confused due to overlapping multimodal patterns in conversational contexts.
- Visual modality alone performs poorly compared to text, requiring careful fusion strategies to avoid performance drops.

## Evidence (verbatim from paper)

> On MELD, the trimodal system (T+V+A) achieves 64.8% accuracy and 64.3% weighted F1-score.

## Citation

```bibtex
@misc{wang2025qualitycontrolled,
  title={Quality-Controlled Multimodal Emotion Recognition in Conversations with Identity-Based Transfer Learning and MAMBA Fusion},
  author={Wang & Beigi (2025)},
  year={2025},
  note={arXiv:2511.14969}
}
```

- arXiv: 2511.14969