Moviegraphs Emotion Eval

---
name: moviegraphs-emotion-eval
description: Predicts multi-label emotions and mental states for movie scenes and individual characters using multimodal inputs (video, dialog, character appearance). It probes long-form video understanding and the ability to integrate visual and linguistic cues for affect recognition. Use when the user wants to benchmark on MovieGraphs, or asks about evaluating this task. Reports mAP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.05634
  bibtex_key: srivastava2023howyoufeelin
  confidence: high
---

# moviegraphs-emotion-eval

> How you feelin'? Learning Emotions and Mental States in Movie Scenes — Srivastava et al. (2023) (arXiv:2304.05634, 2023)

## What this evaluates

Predicts multi-label emotions and mental states for movie scenes and individual characters using multimodal inputs (video, dialog, character appearance). It probes long-form video understanding and the ability to integrate visual and linguistic cues for affect recognition.

## Datasets

- **MovieGraphs** — total 7637; splits: train (-1), val (-1), test (-1)

## Metrics

- `mAP` **(primary)** — range: percent
  - Mean Average Precision across K binary classification problems. AP is computed per label by sorting predictions by confidence and calculating precision-recall area, then mAP averages these APs. Note that AP depends on label frequency.

## Input / output format

**Input**: Multimodal sequence: video frames sampled at 3 fps (up to 300 tokens), dialog utterances encoded via RoBERTa, and character features (face/person detections, tracks, clusters) with time embeddings.

**Output**: K binary predictions (one per emotion/mental state label) for each scene and each character in the scene.

## Scoring recipe

```python
def compute_mAP(preds, gold, K):
    aps = []
    for k in range(K):
        scores = preds[:, k]
        labels = gold[:, k]
        order = np.argsort(-scores)
        sorted_labels = labels[order]
        tp, fp = 0, 0
        precisions = []
        for y in sorted_labels:
            if y == 1: tp += 1
            else: fp += 1
            precisions.append(tp / (tp + fp))
        ap = np.mean(precisions) if tp > 0 else 0.0
        aps.append(ap)
    return np.mean(aps) * 100
```

## Common pitfalls

- The test set is significantly harder than the validation set, causing consistent performance drops across all models.
- Average Precision (AP) is sensitive to label frequency, so performance varies considerably between frequent and rare emotions.
- Character-level prediction is inherently more challenging than scene-level prediction due to occlusion and limited visibility.

## Evidence (verbatim from paper)

> We use the original splits from MovieGraphs. As we have  $K$  binary classification problems, we adopt mean Average Precision (mAP) to measure model performance (similar to Atomic Visual Actions [25]). Note that AP also depends on the label frequency.

## Citation

```bibtex
@misc{srivastava2023howyoufeelin,
  title={How you feelin'? Learning Emotions and Mental States in Movie Scenes},
  author={Srivastava et al. (2023)},
  year={2023},
  note={arXiv:2304.05634}
}
```

- arXiv: 2304.05634