Facebehaviornet Eval

---
name: facebehaviornet-eval
description: Evaluates a multi-task facial analysis model's ability to jointly predict continuous affect (valence/arousal), discrete facial expressions, and facial action units from in-the-wild and lab-controlled face images. It probes the model's capacity for task-coupled learning and generalization across heterogeneous annotation schemes and domains. Use when the user wants to benchmark on Aff-Wild, AffectNet, AFEW, RAF-DB, EmotioNet, DISFA, BP4D, BP4D+, or asks about evaluating this task. Reports CCC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1910.11111
  bibtex_key: kollias2019facebehavior
  confidence: high
---

# facebehaviornet-eval

> Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network — Kollias et al. (2019) (arXiv:1910.11111, 2019)

## What this evaluates

Evaluates a multi-task facial analysis model's ability to jointly predict continuous affect (valence/arousal), discrete facial expressions, and facial action units from in-the-wild and lab-controlled face images. It probes the model's capacity for task-coupled learning and generalization across heterogeneous annotation schemes and domains.

## Datasets

- **Aff-Wild** — total 1250000; splits: train (-1), val (-1)
- **AffectNet** — total 1000000; splits: train (-1), val (-1)
- **AFEW** — total 1809; splits: val (-1)
- **RAF-DB** — total 15200; splits: val (-1)
- **EmotioNet** — total 1000000; splits: val (-1)
- **DISFA** — total 260000; splits: val (-1)
- **BP4D** — total 223000; splits: val (-1)
- **BP4D+** — total ?; splits: val (-1)

## Metrics

- `CCC` **(primary)** — range: [-1, 1]
  - Concordance Correlation Coefficient. Measures precision and accuracy of the predicted mean relative to the true mean: CCC = 2*rho*sigma_x*sigma_y / (sigma_x^2 + sigma_y^2 + (mu_x - mu_y)^2).
- `F1 score` — range: [0, 1]
  - Harmonic mean of precision and recall. Used for AU detection and expression classification on most datasets.
- `Total accuracy` — range: [0, 1]
  - Percentage of correctly classified samples. Used for AFEW.
- `Mean diagonal of confusion matrix` — range: [0, 1]
  - Average of correct classification rates per class. Used for RAF-DB.
- `UAR` — range: [0, 1]
  - Unweighted Average Recall. Average of recall scores across all classes. Used for EmotioNet expression classification.

## Input / output format

**Input**: 96x96x3 RGB facial images with intensity normalized to [-1, 1], extracted using SSH detector and aligned via 5 facial landmarks.

**Output**: Continuous values for valence and arousal; categorical labels for basic/compound expressions; binary/intensity scores for Action Units (AUs).

## Scoring recipe

```python
import numpy as np
from sklearn.metrics import f1_score, accuracy_score

def compute_ccc(y_true, y_pred):
    mean_true, mean_pred = np.mean(y_true), np.mean(y_pred)
    var_true, var_pred = np.var(y_true), np.var(y_pred)
    cov = np.cov(y_true, y_pred)[0, 1]
    return (2 * cov) / (var_true + var_pred + (mean_true - mean_pred)**2)

def score_dataset(y_true, y_pred, metric_type):
    if metric_type == 'CCC':
        return compute_ccc(y_true, y_pred)
    elif metric_type == 'F1':
        return f1_score(y_true, y_pred, average='macro')
    elif metric_type == 'Accuracy':
        return accuracy_score(y_true, y_pred)
    elif metric_type == 'UAR':
        recalls = [np.mean(y_pred[y_true==c] == c) for c in np.unique(y_true)]
        return np.mean(recalls)
    return None
```

## Common pitfalls

- Datasets like AffectNet, AFEW, BP4D, and BP4D+ lack official test splits; the authors repurpose the validation set for testing and split the training set 85/15.
- Each dataset uses a different primary metric (CCC for VA, F1 for AUs, accuracy for AFEW, mean diagonal for RAF-DB, composite scores for EmotioNet).
- EmotioNet AU evaluation averages mean F1 and mean accuracy across all AUs, which differs from standard macro-F1.

## Evidence (verbatim from paper)

> We use: i) the CCC for Aff-Wild (CCC was the evaluation criterion of Aff-Wild Challenge) and Affectnet, ii) the total accuracy for AFEW (this metric was the evaluation criterion of the EmotiW Challenges), the mean diagonal value of the confusion matrix for RAF-DB (this criterion was selected for evaluating the performance on this database by [li2017reliable]), the F1 score for AffectNet, iii) the F1 score for DISFA, BP4D and BP4D+ (this metric was the evaluation criterion of the FERA 2015 and 2017 Challenges); for AU detection in EmotioNet the Challenge’s metric was the average between: a) the mean (across all AUs) F1 score and b) the mean (across all AUs) accuracy; for the expression classification, it was the average between: a) the mean (across all emotions) F1 score and b) the unweighted average recall (UAR) over all emotion categories.

## Citation

```bibtex
@misc{kollias2019facebehavior,
  title={Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network},
  author={Kollias et al. (2019)},
  year={2019},
  note={arXiv:1910.11111}
}
```

- arXiv: 1910.11111