Har Multimodal Classification Eval

---
name: har-multimodal-classification-eval
description: Probes the ability of multimodal and unimodal models to recognize human activities from wearable sensor, pose, and video data. It evaluates classification performance, data efficiency across low-data regimes (1-100% training fractions), and zero-shot transfer capability to unseen real-world datasets. Use when the user wants to benchmark on MM-Fit, MHEALTH, MyoGym, MotionSense, or asks about evaluating this task. Reports Macro F1-Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.03857
  bibtex_key: fritsch2024mujo
  confidence: high
---

# har-multimodal-classification-eval

> MuJo: Multimodal Joint Feature Space Learning for Human Activity Recognition — Fritsch et al. (2024) (arXiv:2406.03857, 2024)

## What this evaluates

Probes the ability of multimodal and unimodal models to recognize human activities from wearable sensor, pose, and video data. It evaluates classification performance, data efficiency across low-data regimes (1-100% training fractions), and zero-shot transfer capability to unseen real-world datasets.

## Datasets

- **MM-Fit** — total ?; splits: train (-1), test (-1)
- **MHEALTH** — total ?; splits: train (-1), test (-1)
- **MyoGym** — total ?; splits: train (-1), test (-1)
- **MotionSense** — total ?; splits: train (-1), test (-1)

## Metrics

- `Macro F1-Score` **(primary)** — range: [0, 1]
  - Unweighted mean of recall per class. Calculated as the average of per-class F1 scores across all activity classes.
- `Top-k Accuracy` — range: [0, 1]
  - Proportion of instances where the true class appears in the top-k predicted probabilities or ranks.

## Input / output format

**Input**: Windowed sensor data (accelerometer), 3D pose sequences, or pre-extracted video/text feature vectors. Multimodal inputs are formed by concatenating these feature representations.

**Output**: Predicted activity class label from a fixed vocabulary, optionally including a 'NULL' class for unknown/irrelevant activities.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels, k=1):
    classes = sorted(set(predictions) | set(gold_labels))
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    macro_f1 = sum(f1_scores) / len(f1_scores)
    correct = sum(1 for p, g in zip(predictions, gold_labels) if g in p[:k])
    topk_acc = correct / len(gold_labels)
    return macro_f1, topk_acc
```

## Common pitfalls

- Including a 'NULL' class for unknown activities significantly lowers scores across all datasets and modalities.
- Domain shift between synthetic pre-training data (FiMAD) and real-world evaluation sensors causes performance degradation.
- Pose modality mismatch due to different extraction pipelines (IMUTube vs. MM-Fit) reduces multimodal gains.

## Evidence (verbatim from paper)

> Table 1 shows that for our method the pre-trained models with trainable weights consistently outperform the baseline with respect to the Macro F1-Score on all datasets, regardless of whether 2% or 100% of the training data are used.

## Citation

```bibtex
@misc{fritsch2024mujo,
  title={MuJo: Multimodal Joint Feature Space Learning for Human Activity Recognition},
  author={Fritsch et al. (2024)},
  year={2024},
  note={arXiv:2406.03857}
}
```

- arXiv: 2406.03857