Simmmdg Eval

---
name: simmmdg-eval
description: Evaluates a model's ability to generalize across unseen domains in multi-modal action recognition. It probes feature disentanglement, cross-modal translation for missing modalities, and robustness to domain shifts in video, audio, and optical flow inputs. Use when the user wants to benchmark on EPIC-Kitchens, HAC, or asks about evaluating this task. Reports Top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.19795
  bibtex_key: dong2023simmmdg
  confidence: high
---

# simmmdg-eval

> SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalization — Dong et al. (2023) (arXiv:2310.19795, 2023)

## What this evaluates

Evaluates a model's ability to generalize across unseen domains in multi-modal action recognition. It probes feature disentanglement, cross-modal translation for missing modalities, and robustness to domain shifts in video, audio, and optical flow inputs.

## Datasets

- **EPIC-Kitchens** — total ?; splits: D1 (-1), D2 (-1), D3 (-1)
- **HAC** — total 3381; splits: H (-1), A (-1), C (-1)

## Metrics

- `Top-1 accuracy` **(primary)** — range: percent
  - Standard classification accuracy: the proportion of correctly predicted action labels out of the total number of test instances.

## Input / output format

**Input**: Multi-modal inputs (video, audio, and/or optical flow) representing human actions across different source domains.

**Output**: Predicted action class label from the predefined set (8 classes for EPIC-Kitchens, 7 for HAC).

## Scoring recipe

```python
def top1_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Domain generalization splits are defined by source/target domain combinations (e.g., D2, D3 → D1) rather than fixed train/val/test sets.
- Missing modality handling drastically changes results; zero-filling embeddings often underperforms cross-modal translation or even unimodal baselines.
- Backbone architecture (I3D vs. SlowFast/ResNet-18) significantly shifts absolute accuracy scores, so comparisons must match the backbone.

## Evidence (verbatim from paper)

> We report the Top-1 accuracy for all experiments. The EPIC-Kitchens dataset includes eight actions ('put', 'take', 'open', 'close', 'wash', 'cut', 'mix', and 'pour') recorded in three different kitchens, forming three separate domains D1, D2, and D3. Our HAC dataset consists of seven actions ('sleeping', 'watching tv', 'eating', 'drinking', 'swimming', 'running', and 'opening door') performed by humans, animals, and cartoon figures, forming three different domains H, A, and C. We collect 3381 video clips from the internet with around 1000 samples for each domain.

## Citation

```bibtex
@misc{dong2023simmmdg,
  title={SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalization},
  author={Dong et al. (2023)},
  year={2023},
  note={arXiv:2310.19795}
}
```

- arXiv: 2310.19795