Express Emotion Recognition Eval

---
name: express-emotion-recognition-eval
description: Evaluates language models' ability to recognize and decompose fine-grained human emotions from self-disclosed narratives. It probes whether models can align with human emotional expressions across 10 Plutchik-based dimensions (8 basic emotions + 2 sentiments) rather than just predicting surface-level emotion words. Use when the user wants to benchmark on EXPRESS, or asks about evaluating this task. Reports F1_V.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.09593
  bibtex_key: shu2025fluent
  confidence: high
---

# express-emotion-recognition-eval

> Fluent but Unfeeling: The Emotional Blind Spots of Language Models — Shu et al. (2025) (arXiv:2509.09593, 2025)

## What this evaluates

Evaluates language models' ability to recognize and decompose fine-grained human emotions from self-disclosed narratives. It probes whether models can align with human emotional expressions across 10 Plutchik-based dimensions (8 basic emotions + 2 sentiments) rather than just predicting surface-level emotion words.

## Datasets

- **EXPRESS** — total 33679; splits: test (-1); repo https://github.com/Computing-for-Social-Good-CSG/express-emotion-recognition.git

## Metrics

- `Acc_L` — range: [0, 1]
  - Lexical accuracy: the proportion of instances where the model's predicted emotion word exactly matches the human self-disclosed emotion word.
- `Acc_V` — range: [0, 1]
  - Vector accuracy: the proportion of instances where the model's predicted 10-dimensional emotion vector exactly matches the ground-truth human vector.
- `F1_V` **(primary)** — range: [0, 1]
  - Average Vector F-1 Score: macro-averaged F1 score computed across the 10 emotion dimensions of the predicted vs. ground-truth emotion vectors.

## Input / output format

**Input**: Text of a human self-disclosed emotional experience (Reddit post), optionally accompanied by few-shot examples or CoT instructions.

**Output**: A predicted emotion word and/or a 10-dimensional binary/continuous vector representing the decomposed emotion according to Plutchik’s Wheel.

## Scoring recipe

```python
def score(predictions, golds):
    acc_l = sum(1 for p, g in zip(predictions, golds) if p.word == g.word) / len(predictions)
    acc_v = sum(1 for p, g in zip(predictions, golds) if p.vector == g.vector) / len(predictions)
    f1_v = 0.0
    for dim in range(10):
        tp = sum(1 for p, g in zip(predictions, golds) if p.vector[dim] == 1 and g.vector[dim] == 1)
        fp = sum(1 for p, g in zip(predictions, golds) if p.vector[dim] == 1 and g.vector[dim] == 0)
        fn = sum(1 for p, g in zip(predictions, golds) if p.vector[dim] == 0 and g.vector[dim] == 1)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1_v += 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    f1_v /= 10
    return acc_l, acc_v, f1_v
```

## Common pitfalls

- CoT prompting consistently degrades performance on this task, contrary to its success in many reasoning benchmarks.
- Segmenting long posts into 512-token chunks does not disadvantage models with larger context windows, as full-post evaluation yields nearly identical results.
- Models often predict semantically plausible but contextually shallow emotions, overusing words like 'anxious' or 'frustrated' while missing human nuances like 'disheartened' or 'panicked'.

## Evidence (verbatim from paper)

> Here, we present our findings on the emotion recognition capabilities of LLMs evaluated on the EXPRESS dataset. ... $Acc_{L}$ ranged from 0.051 to 0.318, while $Acc_{V}$, slightly higher, ranged from 0.097 to 0.388. $F1_{V}$ ranged from 0.434 to 0.711, compared to a baseline of randomly generated vectors at 0.322.

## Citation

```bibtex
@misc{shu2025fluent,
  title={Fluent but Unfeeling: The Emotional Blind Spots of Language Models},
  author={Shu et al. (2025)},
  year={2025},
  note={arXiv:2509.09593}
}
```

- arXiv: 2509.09593