Humanvbench Eval

---
name: humanvbench-eval
description: This benchmark evaluates the human-centric video understanding capabilities of multimodal large language models (MLLMs). It specifically probes inner emotion perception, outer behavioral manifestations, and cross-modal speech-visual alignment through 16 fine-grained multiple-choice tasks. Use when the user wants to benchmark on HumanVBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.17574
  bibtex_key: zhou2024humanvbench
  confidence: high
---

# humanvbench-eval

> HumanVBench: Exploring Human-Centric Video Understanding Capabilities of MLLMs with Synthetic Benchmark Data — Ting Zhou et al. (2024) (arXiv:2412.17574, 2024)

## What this evaluates

This benchmark evaluates the human-centric video understanding capabilities of multimodal large language models (MLLMs). It specifically probes inner emotion perception, outer behavioral manifestations, and cross-modal speech-visual alignment through 16 fine-grained multiple-choice tasks.

## Datasets

- **HumanVBench** — total ?; splits: test (-1); repo https://github.com/modelscope/data-juicer

## Metrics

- `accuracy` **(primary)** — range: percent
  - Calculated as the proportion of correctly answered multiple-choice questions out of the total number of questions. Reported alongside random guess performance and graduate-level human baselines for reference.

## Input / output format

**Input**: Video clips (with or without audio) accompanied by a multiple-choice question (N choose 1) probing human-centric attributes like emotion, behavior, or speech-visual alignment.

**Output**: A single selected option from the provided multiple-choice options.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Fixed 8-frame sampling introduces temporal noise, causing models to misclassify neutral or specific emotions (e.g., interpreting mouth-opening frames as surprise).
- Visual-only MLLMs lack audio input, causing them to degrade to speech action recognition rather than performing true lip-reading or speech-visual alignment.
- Time-specific tasks require explicit timestamp integration in prompts, but many models lack native support, leading to poor temporal reasoning without manual intervention.

## Evidence (verbatim from paper)

> All QAs were framed as multiple-choice questions (N choose 1, with N varying across different test samples), reporting both accuracy and the performance of random guesses and graduate-level humans for reference.

## Citation

```bibtex
@misc{zhou2024humanvbench,
  title={HumanVBench: Exploring Human-Centric Video Understanding Capabilities of MLLMs with Synthetic Benchmark Data},
  author={Ting Zhou et al. (2024)},
  year={2024},
  note={arXiv:2412.17574}
}
```

- arXiv: 2412.17574