Mig Bench Eval

---
name: mig-bench-eval
description: Evaluates a model's ability to perform free-form, multi-image visual grounding by localizing specified objects across multiple input images based on natural language instructions. It probes cross-image reasoning, spatial understanding, and the capacity to follow complex, unstructured queries without relying on chain-of-thought abstractions. Use when the user wants to benchmark on MIG-Bench, or asks about evaluating this task. Reports Acc_0.5.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.05767
  bibtex_key: li2025migician
  confidence: high
---

# mig-bench-eval

> Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models — You Li et al. (2025) (arXiv:2501.05767, 2025)

## What this evaluates

Evaluates a model's ability to perform free-form, multi-image visual grounding by localizing specified objects across multiple input images based on natural language instructions. It probes cross-image reasoning, spatial understanding, and the capacity to follow complex, unstructured queries without relying on chain-of-thought abstractions.

## Datasets

- **MIG-Bench** — total ?; splits: test (-1)

## Metrics

- `Acc_0.5` **(primary)** — range: percent
  - Accuracy of object localization where a prediction is considered correct if the Intersection over Union (IoU) between the predicted bounding box and the ground truth bounding box exceeds 0.5.

## Input / output format

**Input**: A set of multiple input images accompanied by a free-form natural language instruction describing the target object(s) to locate.

**Output**: Bounding box coordinates (typically normalized or pixel-based) corresponding to the predicted location of the target object in the input image(s).

## Scoring recipe

```python
correct = 0
total = len(predictions)
for pred_box, gt_box in zip(predictions, ground_truth):
    iou = calculate_iou(pred_box, gt_box)
    if iou > 0.5:
        correct += 1
accuracy = (correct / total) * 100
```

## Common pitfalls

- IoU threshold of 0.5 is standard but may not align with stricter localization requirements in some downstream tasks or datasets.
- Handling queries that reference objects across different images requires correct cross-image alignment, which models often struggle with without explicit grounding training.
- Coordinate format (normalized vs. absolute pixels) must match the evaluation script to avoid IoU calculation errors or off-by-one boundary issues.

## Evidence (verbatim from paper)

> For the evaluation in our proposed MIG-Bench, we use the conventional metric $	ext{Acc}_{0.5}$ in referring expression comprehension[[20]]. This metric measures the accuracy of object localization, defining a prediction as correct if the Intersection over Union (IoU) with the ground truth is greater than 0.5.

## Citation

```bibtex
@misc{li2025migician,
  title={Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models},
  author={You Li et al. (2025)},
  year={2025},
  note={arXiv:2501.05767}
}
```

- arXiv: 2501.05767