Facial Emotion Analysis Eval

---
name: facial-emotion-analysis-eval
description: Evaluates vision-language models on facial emotion analysis tasks, including fine-grained action unit detection, categorical emotion recognition, and grounded natural language reasoning over facial expressions. The protocol tests both recognition accuracy and the model's ability to generate interpretable, AU-grounded explanations. Use when the user wants to benchmark on DISFA, BP4D, RAF-AU, FER2013, AffectNet, RAF-DB, FABA-Instruct, FEA-20K, or asks about evaluating this task. Reports F1 score, Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.10254
  bibtex_key: wu2025facialr1
  confidence: high
---

# facial-emotion-analysis-eval

> Facial-R1: Aligning Reasoning and Recognition for Facial Emotion Analysis — Jiulong Wu et al. (2025) (arXiv:2511.10254, 2025)

## What this evaluates

Evaluates vision-language models on facial emotion analysis tasks, including fine-grained action unit detection, categorical emotion recognition, and grounded natural language reasoning over facial expressions. The protocol tests both recognition accuracy and the model's ability to generate interpretable, AU-grounded explanations.

## Datasets

- **DISFA** — total 87192; splits: train (52392), test (27654)
- **BP4D** — total 146847; splits: train (100813), test (46034)
- **RAF-AU** — total 4601; splits: train (3479), test (853)
- **FER2013** — total 35887; splits: train (28709), test (3589)
- **AffectNet** — total 303330; splits: train (287618), test (3493)
- **RAF-DB** — total 29672; splits: train (12271), test (3068)
- **FABA-Instruct** — total 14379; splits: train (6060), test (314)
- **FEA-20K** — total 19425; splits: train (17737), test (1688); repo https://github.com/RobitsG/Facial-R1

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall across all action unit classes, typically averaged macro to handle class imbalance.
- `Accuracy` **(primary)** — range: [0, 1]
  - Ratio of correctly predicted emotion categories to total test samples.
- `SEGE` — range: [0, 1]
  - Aggregates AU recognition F1 score and ROUGE-L score of generated textual descriptions to evaluate joint recognition and reasoning quality.
- `ROUGE-L` — range: [0, 1]
  - Longest common subsequence overlap between generated reasoning text and ground-truth reference text.
- `GPT-4o-mini Evaluator` — range: [0, 10]
  - Automated LLM-as-a-judge scoring semantic similarity between generated and ground-truth reasonings on a 0–10 scale.

## Input / output format

**Input**: Single facial image (grayscale or color). For reasoning tasks, the image is accompanied by a text prompt requesting emotion/AU analysis or explanation.

**Output**: Predicted action unit labels (binary/multi-label), emotion category labels (multi-class), or natural language text describing the emotion and reasoning.

## Scoring recipe

```python
def compute_metrics(predictions, golds, task, dataset):
    if task == 'AU_recognition':
        return f1_score(golds, predictions, average='macro')
    elif task == 'emotion_recognition':
        return accuracy_score(golds, predictions)
    elif task == 'emotion_reasoning':
        rouge_l = rouge_l_score(golds['text'], predictions['text'])
        if dataset == 'FABA-Instruct':
            au_f1 = f1_score(golds['au'], predictions['au'], average='macro')
            return 0.5 * au_f1 + 0.5 * rouge_l
        elif dataset == 'FEA-20K':
            return rouge_l
        elif dataset == 'FEA-20K_reliability':
            return gpt4o_mini_evaluate(golds['text'], predictions['text'], scale=10)
```

## Common pitfalls

- AU recognition is a multi-label binary classification task; using standard multi-class accuracy will severely underreport performance.
- Official splits for DISFA and BP4D are frame-level; inconsistent frame extraction or sampling changes dataset size and skews results.
- SEGE metric combines AU F1 and ROUGE-L; failing to normalize or weight them correctly breaks comparability with prior work.
- GPT-4o-mini evaluator scores are non-deterministic and sensitive to prompt phrasing, requiring fixed temperature and system prompts for reproducibility.

## Evidence (verbatim from paper)

> For AU recognition, we employ the F1 score*(mavadati2013disfa; zhang2014bp4d)*, ensuring a balanced evaluation of precision and recall. Regarding emotion recognition, we adopt accuracy (Acc.) as the primary metric, following established protocols*(li2017RAF-DB)*. For AU-based emotion reasoning, we employ task-specific metrics. Specifically, on the FABA-Instruct dataset, we utilize the SEGE metric proposed in FABA*(li2024faba)*, which aggregates the AU recognition F1 and the ROUGE-L score of textual descriptions. For the FEA-20K dataset, we report ROUGE-L to measure the textual similarity between generated and ground-truth reasonings. Additionally, to comprehensively evaluate the reliability of emotion reasoning, we propose employing GPT-4o-mini*(hurst2024gpt4o)* as an automated evaluator for measuring semantic similarity between generated and ground-truth reasonings, with scores ranging from 0 to 10.

## Citation

```bibtex
@misc{wu2025facialr1,
  title={Facial-R1: Aligning Reasoning and Recognition for Facial Emotion Analysis},
  author={Jiulong Wu et al. (2025)},
  year={2025},
  note={arXiv:2511.10254}
}
```

- arXiv: 2511.10254