Emotion Recognition Eval

---
name: emotion-recognition-eval
description: Evaluates vision models' ability to classify emotions in images across multiple affective categories. It also measures the alignment between emotions expressed in text prompts and those visually present in generated images. Use when the user wants to benchmark on EmoSet, or asks about evaluating this task. Reports macro-averaged F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.05985
  bibtex_key: mehta2024emotionalimages
  confidence: high
---

# emotion-recognition-eval

> Emotional Images: Assessing Emotions in Images and Potential Biases in Generative Models — Mehta et al. (2024) (arXiv:2411.05985, 2024)

## What this evaluates

Evaluates vision models' ability to classify emotions in images across multiple affective categories. It also measures the alignment between emotions expressed in text prompts and those visually present in generated images.

## Datasets

- **EmoSet** — total ?; splits: test (-1)

## Metrics

- `macro-averaged F1-score` **(primary)** — range: [0, 1]
  - Unweighted mean of per-class F1-scores. F1 is the harmonic mean of precision and recall for each emotion class.
- `Precision` — range: [0, 1]
  - Ratio of true positive predictions to all positive predictions for each class.
- `Recall` — range: [0, 1]
  - Ratio of true positive predictions to all actual positive instances for each class.

## Input / output format

**Input**: RGB images of generated or real scenes; text prompts for cross-modal comparison.

**Output**: Discrete emotion labels from a predefined set (e.g., Fear, Amusement, Awe, Anger, Sadness, Excitement, Contentment, Disgust).

## Scoring recipe

```python
def compute_macro_f1(preds, gold):
    classes = set(preds) | set(gold)
    f1s = []
    for c in classes:
        tp = sum(p == c and g == c for p, g in zip(preds, gold))
        fp = sum(p == c and g != c for p, g in zip(preds, gold))
        fn = sum(p != c and g == c for p, g in zip(preds, gold))
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
    return sum(f1s) / len(f1s)
```

## Common pitfalls

- Zero-shot multimodal models heavily over-predict specific emotions (e.g., Amusement) while failing on others, distorting macro-averaged metrics.
- Auto-captioning pipelines (image-to-text-to-emotion) underperform direct vision models due to modality transfer loss.
- Cross-modality correlation analysis shows weak alignment for positive emotions, making prompt-image alignment assessments noisy.

## Evidence (verbatim from paper)

> Among these models, Google ViT emerged as the strongest performer, with an overall macro-averaged F1-score higher than all other methods.

## Citation

```bibtex
@misc{mehta2024emotionalimages,
  title={Emotional Images: Assessing Emotions in Images and Potential Biases in Generative Models},
  author={Mehta et al. (2024)},
  year={2024},
  note={arXiv:2411.05985}
}
```

- arXiv: 2411.05985