Mm Inference Eval

---
name: mm-inference-eval
description: This evaluation probes the effectiveness and efficiency of dynamic sparse attention methods for long-context vision-language models (VLMs). It tests the model's ability to perform long-video understanding, retrieve specific visual or mixed-modality information from extremely long contexts (Needle in a Haystack), and maintain accuracy while reducing computational cost and latency. Use when the user wants to benchmark on Video Understanding Benchmarks, V-NIAH, MM-NIAH, or asks about evaluating this task. Reports task accuracy / official benchmark score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.16083
  bibtex_key: li2025mm inference
  confidence: high
---

# mm-inference-eval

> MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attention — Li et al. (2025) (arXiv:2504.16083, 2025)

## What this evaluates

This evaluation probes the effectiveness and efficiency of dynamic sparse attention methods for long-context vision-language models (VLMs). It tests the model's ability to perform long-video understanding, retrieve specific visual or mixed-modality information from extremely long contexts (Needle in a Haystack), and maintain accuracy while reducing computational cost and latency.

## Datasets

- **Video Understanding Benchmarks** — total ?; splits: test (-1), val (-1)
- **V-NIAH** — total ?; splits: test (-1)
- **MM-NIAH** — total ?; splits: test (-1)

## Metrics

- `task accuracy / official benchmark score` **(primary)** — range: percent
  - Uses the official metrics and scripts provided by each benchmark (ActNet-QA, EgoSchema, Next-QA, PerceptionTest, VideoDC, VideoMME). Typically exact-match or multiple-choice accuracy for QA tasks, and standard captioning/retrieval scores where applicable.
- `end-to-end latency` — range: other
  - Wall-clock time measured for the full pre-filling/inference pass on a single NVIDIA A100 using bfloat16 and greedy decoding. Reported in seconds or milliseconds.
- `kernel-level latency` — range: other
  - Wall-clock time measured specifically for the attention kernel computation, used to isolate the speedup from sparse attention patterns.

## Input / output format

**Input**: Long-video inputs ranging from 110 to 4.5k frames (up to ~1.1M tokens), paired with text queries for QA, captioning, or retrieval tasks. MM-NIAH inputs additionally contain 25% text segments inserted at the document level across different frames.

**Output**: Model-generated answers or selected choices for QA/retrieval tasks. Latency is measured as wall-clock time per pass.

## Scoring recipe

```python
```python
# For accuracy metrics
import official_benchmark_scripts
predictions = model.generate(input_video, input_query)
gold_answers = load_gold_answers(dataset)
metric_value = official_benchmark_scripts.compute_score(predictions, gold_answers)

# For latency
import time
torch.cuda.synchronize()
start = time.time()
model.generate(input_video, input_query) # or model.prefill()
torch.cuda.synchronize()
latency = time.time() - start
```
```

## Common pitfalls

- Static sparse patterns (e.g., A-shape, Tri-shape) show notable performance drops on multi-choice VQA tasks like EgoSchema, even when FLOPs are matched.
- Sparse indices trained on textual contexts fail to generalize to visual modalities; modality-aware permutation is required to maintain performance across mixed-modality boundaries.
- Latency measurements must use greedy decoding and bfloat16 on an A100 to ensure stable, comparable results as specified in the setup.

## Evidence (verbatim from paper)

> We evaluate our method on three general long-video tasks: long-video understanding, Video Needle in a Haystack, and Video-Text Needle in a Haystack. Our evaluation uses the official metrics and scripts provided by these tasks. Latency experiments are performed on a single NVIDIA A100 using bfloat16, with greedy decoding to ensure stable results.

## Citation

```bibtex
@misc{li2025mm inference,
  title={MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attention},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2504.16083}
}
```

- arXiv: 2504.16083