Emotrans Eval

---
name: emotrans-eval
description: Evaluates multimodal LLMs on understanding, reasoning, and predicting fine-grained emotion transitions in short video clips. It probes capabilities across four progressive tasks: detecting whether an emotion change occurs, identifying before/after emotion states, generating evidence-grounded reasoning for transitions, and predicting the next emotion state. Use when the user wants to benchmark on EmoTrans, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.23348
  bibtex_key: he2026emotrans
  confidence: high
---

# emotrans-eval

> EmoTrans: A Benchmark for Understanding, Reasoning, and Predicting Emotion Transitions in Multimodal LLMs — He Hu et al. (2026) (arXiv:2604.23348, 2026)

## What this evaluates

Evaluates multimodal LLMs on understanding, reasoning, and predicting fine-grained emotion transitions in short video clips. It probes capabilities across four progressive tasks: detecting whether an emotion change occurs, identifying before/after emotion states, generating evidence-grounded reasoning for transitions, and predicting the next emotion state.

## Datasets

- **EmoTrans** — total 3274; splits: test (3274); repo https://github.com/Emo-gml/EmoTrans

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted labels for classification tasks (ECD, ESI, NEP). Calculated as (correct predictions / total instances) * 100.
- `BERTScore` — range: [0, 1]
  - Contextual embedding-based similarity between generated reasoning explanations and ground-truth rationales, computed using BERT embeddings.
- `ROUGE-L` — range: [0, 1]
  - Longest common subsequence overlap between generated and reference text, measuring structural similarity.
- `BLEU4` — range: [0, 1]
  - 4-gram precision with brevity penalty comparing generated explanations to reference rationales.
- `LLM-Score` — range: other
  - Automated scoring of generated reasoning quality using a separate LLM judge against ground-truth explanations.

## Input / output format

**Input**: Video clip (audio+visual) with aligned modalities, person identification attributes (e.g., clothing, position), and a task-specific natural language prompt.

**Output**: Task-dependent: (1) Binary label (0 or 1) for emotion change; (2) Tuple of before/after emotion labels (e_bef, e_aft) or single label e; (3) Natural language rationale explaining the transition or stability; (4) Single predicted emotion label.

## Scoring recipe

```python
def score_emotrans(predictions, golds, task):
    if task in ['ECD', 'ESI', 'NEP']:
        correct = sum(1 for p, g in zip(predictions, golds) if p == g)
        return correct / len(golds) * 100
    elif task == 'ETR':
        bert = compute_bertscore(predictions, golds)
        rouge = compute_rouge_l(predictions, golds)
        bleu = compute_bleu4(predictions, golds)
        llm = compute_llm_score(predictions, golds)
        return {'BERTScore': bert, 'ROUGE-L': rouge, 'BLEU4': bleu, 'LLM-Score': llm}
```

## Common pitfalls

- Multi-person videos require per-person predictions; models must correctly attribute emotions to specific individuals using provided attributes, otherwise predictions are misaligned.
- Temporal tolerance is 0.5 seconds for change points; exact timestamp matching will incorrectly penalize otherwise valid predictions.
- Emotion labels are strictly categorical (7 classes: Happiness, Sadness, Anger, Fear, Disgust, Surprise, Neutral); dimensional or intensity-based outputs are invalid.

## Evidence (verbatim from paper)

> For classification tasks (ECD, ESI, and NEP), results are reported in Accuracy (%). Avg. denotes the mean Accuracy over ECD, ESI, and NEP. Red and blue denote the best and second-best results among all models.

## Citation

```bibtex
@misc{he2026emotrans,
  title={EmoTrans: A Benchmark for Understanding, Reasoning, and Predicting Emotion Transitions in Multimodal LLMs},
  author={He Hu et al. (2026)},
  year={2026},
  note={arXiv:2604.23348}
}
```

- arXiv: 2604.23348