Emersk Emotion Recognition Eval

---
name: emersk-emotion-recognition-eval
description: Evaluates a multimodal emotion recognition system that fuses facial, posture, and gait cues with situational knowledge to classify emotional states. It probes the model's ability to generalize across posed and wild settings, different data modalities, and subject-independent splits. Use when the user wants to benchmark on FER-2013, CAER-S, FABO, EWalk, GroupWalk, GEMEP, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.08657
  bibtex_key: palash2023emersk
  confidence: high
---

# emersk-emotion-recognition-eval

> EMERSK -- Explainable Multimodal Emotion Recognition with Situational Knowledge — Palash et al. (2023) (arXiv:2306.08657, 2023)

## What this evaluates

Evaluates a multimodal emotion recognition system that fuses facial, posture, and gait cues with situational knowledge to classify emotional states. It probes the model's ability to generalize across posed and wild settings, different data modalities, and subject-independent splits.

## Datasets

- **FER-2013** — total 32298; splits: test (-1)
- **CAER-S** — total 70000; splits: test (-1)
- **FABO** — total 206; splits: test (-1)
- **EWalk** — total 1384; splits: test (-1)
- **GroupWalk** — total 45; splits: test (-1)
- **GEMEP** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Accuracy = #N_c / #N_t, where #N_c is the number of correctly predicted items and #N_t is the total number of items in the test dataset.
- `mAP` — range: [0, 1]
  - mean Average Precision, calculated as the average of Average Precision (AP) values across all emotion classes, where AP is the area under the precision-recall curve.

## Input / output format

**Input**: Images or video frames containing face, posture, and/or gait modalities, often accompanied by background/contextual information.

**Output**: Discrete emotion class label (e.g., Neutral, Sadness, Happiness, Anger, Surprise, Fear, Disgust, Boredom, Puzzlement, Anxiety, Contempt) and optionally a textual explanation based on situational knowledge.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

def compute_mAP(predictions, gold_labels, classes):
    ap_scores = []
    for cls in classes:
        # Extract binary relevance and confidence scores for cls
        # Compute precision-recall curve and area under it
        ap_scores.append(calculate_ap_for_class(cls, predictions, gold_labels))
    return sum(ap_scores) / len(classes)
```

## Common pitfalls

- Stand-alone unimodal modules use simplified architectures to preserve multimodal speed, so they may underperform complex literature baselines despite the multimodal system excelling.
- Evaluation splits on GEMEP drastically change results: Leave-One-Subject-Out (LOSO) yields ~79.4% accuracy, while allowing subject overlap yields ~99.02%, making split definition critical for fair comparison.

## Evidence (verbatim from paper)

> The test accuracy of our method is computed as follows: Accuracy = #N_c / #N_t Where, #N_c indicates the number of items correctly predicted and #N_t indicates the total number of items in the test dataset. Additionally, we use the mean average precision (mAP) score as the performance metric in this dataset. mAP is a widely used evaluation metric which is a measures of the quality of the algorithm. Average precision (AP) is the area under the precision-recall curve, which measures how well the algorithm retrieves relevant instances. The mAP is calculated as the average of the AP values for all the classes in the dataset.

## Citation

```bibtex
@misc{palash2023emersk,
  title={EMERSK -- Explainable Multimodal Emotion Recognition with Situational Knowledge},
  author={Palash et al. (2023)},
  year={2023},
  note={arXiv:2306.08657}
}
```

- arXiv: 2306.08657