Mir Eval

---
name: mir-eval
description: Evaluates multimodal large language models on progressive, interleaved multi-image reasoning tasks. It probes the model's ability to perform structured, step-by-step reasoning across multiple images, including text-to-region alignment, cross-image relationship modeling, and analytical inference. Use when the user wants to benchmark on MIR, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.17040
  bibtex_key: du2025mir
  confidence: high
---

# mir-eval

> From Easy to Hard: The MIR Benchmark for Progressive Interleaved Multi-Image Reasoning — Du et al. (2025) (arXiv:2509.17040, 2025)

## What this evaluates

Evaluates multimodal large language models on progressive, interleaved multi-image reasoning tasks. It probes the model's ability to perform structured, step-by-step reasoning across multiple images, including text-to-region alignment, cross-image relationship modeling, and analytical inference.

## Datasets

- **MIR** — total 22257; splits: spatial (-1), Sequential (-1), Analytical (-1); repo https://github.com/Shelly-coder239/MIRBench

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly matched answers. Due to the long, structured reasoning outputs, rule-based matching is unreliable; the authors uniformly employ GPT to match model outputs with ground truth answers.

## Input / output format

**Input**: Interleaved multi-image sequences paired with a question requiring step-by-step reasoning.

**Output**: Structured reasoning process consisting of five explicit steps: Summary, Caption, Text2Region, Region2Region, and Conclusion, followed by the final answer.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    matches = 0
    for pred, gold_ans in zip(predictions, gold):
        # GPT is used to determine if the prediction matches the ground truth
        if gpt_match_answer(pred, gold_ans):
            matches += 1
    return (matches / len(predictions)) * 100
```

## Common pitfalls

- Long model outputs often fail to align with ground truth using standard rule-based answer matching, necessitating GPT-based evaluation.
- Sequential tasks use images extracted from videos, which may not accurately represent the full video context, leading to artificially low performance on this sub-task.

## Evidence (verbatim from paper)

> During the evaluation phase, since lmms-eval relies on rule-based answer matching, long outputs often struggle to align with the correct answers. Therefore, we uniformly employ GPT to match model outputs with the ground truth. This evaluation scheme ensures the reliability and comprehensiveness of the experimental results.

## Citation

```bibtex
@misc{du2025mir,
  title={From Easy to Hard: The MIR Benchmark for Progressive Interleaved Multi-Image Reasoning},
  author={Du et al. (2025)},
  year={2025},
  note={arXiv:2509.17040}
}
```

- arXiv: 2509.17040