Mmhu Eval

---
name: mmhu-eval
description: Evaluates multimodal models on human behavior understanding in autonomous driving contexts, covering motion prediction, text-to-motion generation, and behavior question-answering. It probes a model's ability to interpret video frames, predict future human motion, generate plausible driving-scene motions from text, and answer safety-critical behavior questions. Use when the user wants to benchmark on MMHU, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.12463
  bibtex_key: li2025mmhu
  confidence: high
---

# mmhu-eval

> MMHU: A Massive-Scale Multimodal Benchmark for Human Behavior Understanding — Li et al. (2025) (arXiv:2507.12463, 2025)

## What this evaluates

Evaluates multimodal models on human behavior understanding in autonomous driving contexts, covering motion prediction, text-to-motion generation, and behavior question-answering. It probes a model's ability to interpret video frames, predict future human motion, generate plausible driving-scene motions from text, and answer safety-critical behavior questions.

## Datasets

- **MMHU** — total 57340; splits: train (56500), test (840)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly answered close-ended questions about human behaviors in driving scenes. Computed as correct predictions divided by total test instances.
- `F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall for multi-class behavior classification. Balances false positives and false negatives in VQA and intention prediction tasks.
- `MPJPE` — range: other
  - Mean Per Joint Position Error. Average Euclidean distance between predicted and ground-truth 3D joint coordinates across all joints and frames.
- `FID` — range: other
  - Fréchet Inception Distance. Measures the Wasserstein-2 distance between feature distributions of real and generated motion sequences. Lower values indicate better generation quality.

## Input / output format

**Input**: Multimodal input consisting of 4 to 6 sampled video frames and a text prompt or close-ended question describing a specific human behavior or motion trajectory.

**Output**: Text answer for VQA tasks; 3D/2D joint coordinates or motion sequences for prediction/generation tasks.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return correct / len(gold)

def compute_f1(predictions, gold, labels):
    tp = sum(1 for p, g in zip(predictions, gold) if p == g)
    prec = tp / max(1, sum(1 for p in predictions if p in labels))
    rec = tp / max(1, sum(1 for g in gold if g in labels))
    return 2 * prec * rec / max(1e-9, prec + rec)

def compute_mpjpe(pred_joints, gt_joints):
    return np.mean(np.sqrt(np.sum((pred_joints - gt_joints)**2, axis=-1)))
```

## Common pitfalls

- Models are evaluated on different test sets depending on the task: MMHU-T for VQA/motion, 3DPW for motion prediction generalization, and JAAD for intention prediction.
- VQA baselines use only 4-6 frames per query, which may underrepresent temporal context compared to full video inputs.
- Motion generation metrics (FID) are computed on generic datasets but applied to driving scenes, highlighting a domain gap rather than absolute performance.

## Evidence (verbatim from paper)

> We evaluated the pre-trained baselines on MMHU-T, to unify the time argument used in computing the MPJPE metric, we use the frame id to replace it. All baselines are evaluated on the same frames.

## Citation

```bibtex
@misc{li2025mmhu,
  title={MMHU: A Massive-Scale Multimodal Benchmark for Human Behavior Understanding},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2507.12463}
}
```

- arXiv: 2507.12463