Eve Emotion Recognition Eval

---
name: eve-emotion-recognition-eval
description: Evaluates vision-language models' ability to recognize evoked emotions from images in a zero-shot setting. It probes their robustness to prompt perturbations and measures sentiment bias in predicting positive vs. negative emotions. Use when the user wants to benchmark on EvE, or asks about evaluating this task. Reports weighted F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.05660
  bibtex_key: bhattacharyya2025eve
  confidence: high
---

# eve-emotion-recognition-eval

> Evaluating Vision-Language Models for Emotion Recognition — Bhattacharyya et al. (2025) (arXiv:2502.05660, 2025)

## What this evaluates

Evaluates vision-language models' ability to recognize evoked emotions from images in a zero-shot setting. It probes their robustness to prompt perturbations and measures sentiment bias in predicting positive vs. negative emotions.

## Datasets

- **EvE** — total ?; splits: test (-1); repo https://github.com/sreebhattacharyya/Eve_Benchmark

## Metrics

- `weighted F1 score` **(primary)** — range: [0, 1]
  - Computed by string-matching the model's parsed single-word prediction against the ground truth class label, then calculating the weighted F1 score across all emotion classes.

## Input / output format

**Input**: An image I and a prompt P describing emotion labels for k discrete emotion classes.

**Output**: A single-word emotion prediction.

## Scoring recipe

```python
predictions = [parse_single_word(model_output) for model_output in model_outputs]
tp, fp, fn = {cls: 0 for cls in classes}, {cls: 0 for cls in classes}, {cls: 0 for cls in classes}
for p, g in zip(predictions, gold_labels):
    if p == g: tp[g] += 1
    elif p != g: fp[p] += 1
    if p != g: fn[g] += 1
precisions = [tp[c]/(tp[c]+fp[c]) if (tp[c]+fp[c])>0 else 0 for c in classes]
recalls = [tp[c]/(tp[c]+fn[c]) if (tp[c]+fn[c])>0 else 0 for c in classes]
supports = [sum(1 for g in gold_labels if g==c) for c in classes]
f1s = [2*p*r/(p+r) if (p+r)>0 else 0 for p, r in zip(precisions, recalls)]
weighted_f1 = sum(f * s for f, s in zip(f1s, supports)) / sum(supports)
```

## Common pitfalls

- String-matching is used for evaluation, so any paraphrased or multi-word model output is automatically counted as incorrect.
- Model performance is highly sensitive to prompt structure, particularly the order of emotion labels provided in the prompt.
- Open-vocabulary or persona-based prompting significantly degrades F1 scores compared to simple classification prompts.

## Evidence (verbatim from paper)

> The responses are parsed and string-matched with the ground truth class labels, and weighted F1 scores are calculated.

## Citation

```bibtex
@misc{bhattacharyya2025eve,
  title={Evaluating Vision-Language Models for Emotion Recognition},
  author={Bhattacharyya et al. (2025)},
  year={2025},
  note={arXiv:2502.05660}
}
```

- arXiv: 2502.05660