Multimodal Benchmarks Eval

---
name: multimodal-benchmarks-eval
description: Evaluates multimodal perception and reasoning capabilities across image, video, and audio understanding tasks. It probes the model's ability to process heterogeneous modalities and answer complex questions or transcribe speech accurately. Use when the user wants to benchmark on AI2D, MMMU, MMStar, OCRBench, MMVet, Mathvista, LongVideoBench, DiDeMo, AVQA, MVBench, Video-MME, Aishell1, LibriSpeech, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.19023
  bibtex_key: gao2025ordmo
  confidence: high
---

# multimodal-benchmarks-eval

> OrdMoE: Preference Alignment via Hierarchical Expert Group Ranking in Multimodal Mixture-of-Experts LLMs — Gao et al. (2025) (arXiv:2511.19023, 2025)

## What this evaluates

Evaluates multimodal perception and reasoning capabilities across image, video, and audio understanding tasks. It probes the model's ability to process heterogeneous modalities and answer complex questions or transcribe speech accurately.

## Datasets

- **AI2D** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: test (-1)
- **MMStar** — total ?; splits: test (-1)
- **OCRBench** — total ?; splits: test (-1)
- **MMVet** — total ?; splits: test (-1)
- **Mathvista** — total ?; splits: test (-1)
- **LongVideoBench** — total ?; splits: test (-1)
- **DiDeMo** — total ?; splits: test (-1)
- **AVQA** — total ?; splits: test (-1)
- **MVBench** — total ?; splits: test (-1)
- **Video-MME** — total ?; splits: test (-1)
- **Aishell1** — total ?; splits: test (-1)
- **LibriSpeech** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 100] percent
  - Percentage of correctly answered questions out of the total number of questions in the benchmark.
- `word error rate (WER)` — range: percent
  - Standard speech recognition metric calculating the ratio of substitutions, deletions, and insertions to the reference transcript.

## Input / output format

**Input**: Multimodal inputs (images, video frames, audio clips) paired with text prompts or questions. Videos are uniformly sampled to 128 frames.

**Output**: Text responses containing the model's predicted answer or transcription.

## Scoring recipe

```python
def compute_score(predictions, golds, modality):
    if modality == 'audio':
        return calculate_wer(predictions, golds)  # Lower is better
    else:
        correct = sum(1 for p, g in zip(predictions, golds) if p == g)
        return (correct / len(golds)) * 100  # Higher is better
```

## Common pitfalls

- Video evaluation uses a fixed 128-frame sampling strategy, which may discard critical temporal context for long videos.
- Audio WER is lower-is-better while visual/text accuracy is higher-is-better; averaging across modalities requires careful direction normalization.
- Benchmarks like MMVet and MathVista often rely on LLM-as-judge or custom grading rubrics, introducing potential scorer bias not detailed in the protocol.

## Evidence (verbatim from paper)

> Our model is evaluated across a diverse set of audio, image, and video understanding tasks to comprehensively assess its multimodal perception and reasoning capabilities. For image-text understanding, we select a suite of challenging multimodal and vision-centric benchmarks: AI2D, MMMU, MMStar, OCRBench, MMVet, and Mathvista. For video understanding, we include LongVideoBench, DiDeMo, AVQA, MVBench, and Video-MME, using a uniform sampling strategy that extract 128 frames per video. For audio performance, we evaluate the model on standard speech recognition benchmarks, including public Chinese datasets such as Aishell1 and English datasets such as LibriSpeech. As shown in Table 2, OrdMoE consistently improves over the baseline across all modalities: it reduces word error rate (WER) on audio benchmarks (lower is better), and increases accuracy on both image and video understanding tasks (higher is better).

## Citation

```bibtex
@misc{gao2025ordmo,
  title={OrdMoE: Preference Alignment via Hierarchical Expert Group Ranking in Multimodal Mixture-of-Experts LLMs},
  author={Gao et al. (2025)},
  year={2025},
  note={arXiv:2511.19023}
}
```

- arXiv: 2511.19023