Milebench Eval

---
name: milebench-eval
description: Evaluates Multimodal Large Language Models (MLLMs) on long-context, multi-image comprehension. It probes capabilities like needle-in-a-haystack retrieval, image retrieval, temporal reasoning across multiple images, and semantic understanding in long multimodal contexts. Use when the user wants to benchmark on MileBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.18532
  bibtex_key: song2024milebench
  confidence: high
---

# milebench-eval

> MileBench: Benchmarking MLLMs in Long Context — Song et al. (2024) (arXiv:2404.18532, 2024)

## What this evaluates

Evaluates Multimodal Large Language Models (MLLMs) on long-context, multi-image comprehension. It probes capabilities like needle-in-a-haystack retrieval, image retrieval, temporal reasoning across multiple images, and semantic understanding in long multimodal contexts.

## Datasets

- **MileBench** — total 6440; splits: test (6440)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice and needle-in-a-haystack questions.
- `ROUGE-L` — range: percent
  - n-gram-based metric used for open-ended generation tasks.

## Input / output format

**Input**: Interleaved images and text prompts/questions. When input length exceeds the model's maximum context length, the instruction is kept and the interleaved image-text question is truncated from the left.

**Output**: Text answers generated via greedy decoding, with a designated generation length between 1 and 512 tokens.

## Scoring recipe

```python
def compute_metric(prediction, gold, task_type):
    if task_type in ['multiple_choice', 'needle_in_haystack']:
        return 1.0 if prediction.strip() == gold.strip() else 0.0
    elif task_type == 'open_ended':
        return rouge_l_score(prediction, gold)
```

## Common pitfalls

- Many open-source models partially answer needle-in-a-haystack tasks (e.g., outputting only part of a numeric string) without fully matching, resulting in zero accuracy despite partial retrieval.
- Input truncation from the left when exceeding context limits may discard critical question information, as the instruction and question are deemed critical.
- Models may generate image captions instead of following multi-image QA instructions due to lack of multi-image training data.

## Evidence (verbatim from paper)

> Metrics for each dataset, as shown in Table[6], are consistent with the original work for tasks built on previous datasets. For open-ended generation tasks, the popular n-gram-based metric ROUGE-L is adopted, and accuracy is the metric for multiple-choice and needle-in-a-haystack tasks.

## Citation

```bibtex
@misc{song2024milebench,
  title={MileBench: Benchmarking MLLMs in Long Context},
  author={Song et al. (2024)},
  year={2024},
  note={arXiv:2404.18532}
}
```

- arXiv: 2404.18532