Emoscene Eval

---
name: emoscene-eval
description: Probes a model's ability to perform context-aware, multi-dimensional emotion understanding by predicting Plutchik’s 8 basic emotions from rich textual scenarios. It specifically tests zero-shot multi-label emotion prediction and evaluates whether models can capture emotional entanglement (co-occurrence) rather than treating emotion dimensions as independent. Use when the user wants to benchmark on EmoScene, or asks about evaluating this task. Reports Macro F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.00819
  bibtex_key: [REDACTED]
  confidence: medium
---

# emoscene-eval

> Emotion Entanglement and Bayesian Inference for Multi-Dimensional Emotion Understanding — Kotaprolu et al. (2026) (arXiv:2604.00819, 2026)

## What this evaluates

Probes a model's ability to perform context-aware, multi-dimensional emotion understanding by predicting Plutchik’s 8 basic emotions from rich textual scenarios. It specifically tests zero-shot multi-label emotion prediction and evaluates whether models can capture emotional entanglement (co-occurrence) rather than treating emotion dimensions as independent.

## Datasets

- **EmoScene** — total 4731; splits: test (4731)

## Metrics

- `Macro F1` **(primary)** — range: [0, 1]
  - Unweighted mean of the F1 scores across the 8 Plutchik emotion classes. F1 per class is calculated as 2 * (Precision * Recall) / (Precision + Recall).
- `Precision` — range: [0, 1]
  - Ratio of correctly predicted positive instances to the total predicted positives for each emotion dimension.
- `Recall` — range: [0, 1]
  - Ratio of correctly predicted positive instances to the total actual positives for each emotion dimension.

## Input / output format

**Input**: Context-rich textual scenario descriptions.

**Output**: Logit scores corresponding to 'Yes' and 'No' for each of the 8 Plutchik basic emotions, optionally accompanied by a natural language explanation and a confidence score.

## Scoring recipe

```python
def compute_macro_f1(predictions, gold):
    f1_scores = []
    for i in range(8):
        tp = sum(1 for p, g in zip(predictions, gold) if p[i] == 1 and g[i] == 1)
        fp = sum(1 for p, g in zip(predictions, gold) if p[i] == 1 and g[i] == 0)
        fn = sum(1 for p, g in zip(predictions, gold) if p[i] == 0 and g[i] == 1)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- Models often treat the 8 emotion dimensions as independent, ignoring known emotional co-occurrence patterns (entanglement).
- Zero-shot prompting without task-specific fine-tuning leads to significant drops in Macro F1, especially for smaller models.
- Provided confidence scores do not reliably correlate with actual prediction accuracy.

## Evidence (verbatim from paper)

> Existing LLMs perform poorly in zero-shot multi-label emotion prediction (best Macro F1: 0.501), revealing limitations in context-aware emotional reasoning.

## Citation

```bibtex
@misc{kotaprolu2026emoscene,
  title={Emotion Entanglement and Bayesian Inference for Multi-Dimensional Emotion Understanding},
  author={Kotaprolu et al. (2026)},
  year={2026},
  note={arXiv:2604.00819}
}
```

- arXiv: 2604.00819