Egonormia Eval

---
name: egonormia-eval
description: Evaluates vision-language models' ability to understand and reason about physical-social norms in egocentric video scenarios. It probes whether models can correctly select normative actions, justify them, and identify plausible alternatives in conflict-prone situations. Use when the user wants to benchmark on EgoNormia, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.20490
  bibtex_key: rezaei2025egonormia
  confidence: high
---

# egonormia-eval

> EgoNormia: Benchmarking Physical Social Norm Understanding — Rezaei et al. (2025) (arXiv:2502.20490, 2025)

## What this evaluates

Evaluates vision-language models' ability to understand and reason about physical-social norms in egocentric video scenarios. It probes whether models can correctly select normative actions, justify them, and identify plausible alternatives in conflict-prone situations.

## Datasets

- **EgoNormia** — total 2053; splits: full (1853), verified (200)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Proportion of correct predictions out of total instances. Calculated separately for action selection, justification, and combined (Both).
- `IoU` — range: [0, 1]
  - Intersection over Union between the set of predicted plausible actions and the set of ground-truth plausible actions. Used for the third subtask where multiple contextually-sensible choices exist.

## Input / output format

**Input**: Three evaluation settings: (a) Blind: questions only; (b) Pipeline: rich text scene description (generated by Gemini 1.5 Flash) concatenated with questions; (c) Video: egocentric video sampled at 1 fps and concatenated left-to-right into a single image, plus questions. Chain-of-thought (CoT) prompting is applied to all non-reasoning models.

**Output**: Multiple-choice selection for the correct action and its justification, plus identification of plausible alternative actions. Models generate text responses following CoT prompting.

## Scoring recipe

```python
def compute_accuracy(preds, golds):
    correct = sum(1 for p, g in zip(preds, golds) if p == g)
    return correct / len(golds)

def compute_iou(preds, golds):
    iou_scores = []
    for p, g in zip(preds, golds):
        p_set, g_set = set(p), set(g)
        union = len(p_set | g_set)
        iou_scores.append(len(p_set & g_set) / union if union > 0 else 0.0)
    return sum(iou_scores) / len(iou_scores)
```

## Common pitfalls

- Input modality heavily influences results; performance drops ~20-26% when visual input is removed (Blind setting).
- The third subtask uses IoU instead of Accuracy because multiple alternative actions are considered correct.
- CoT prompting is required for non-reasoning models to achieve comparable performance, which may inflate latency.

## Evidence (verbatim from paper)

> Accuracy is used in the first two subtasks with a single ground-truth answer; intersection over union (IoU) is used on the third subtask, where multiple contextually-sensible action choices exist.

## Citation

```bibtex
@misc{rezaei2025egonormia,
  title={EgoNormia: Benchmarking Physical Social Norm Understanding},
  author={Rezaei et al. (2025)},
  year={2025},
  note={arXiv:2502.20490}
}
```

- arXiv: 2502.20490