Mc Bench Eval

---
name: mc-bench-eval
description: This benchmark evaluates multi-context visual grounding, requiring models to localize target objects across multiple images using open-ended, context-rich text prompts. It probes cross-image reasoning, fine-grained instance localization, and the ability to correctly group and reject irrelevant instances. Use when the user wants to benchmark on MC-Bench, or asks about evaluating this task. Reports AP50.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.12332
  bibtex_key: xu2024mcbench
  confidence: high
---

# mc-bench-eval

> MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs — Xu et al. (2024) (arXiv:2410.12332, 2024)

## What this evaluates

This benchmark evaluates multi-context visual grounding, requiring models to localize target objects across multiple images using open-ended, context-rich text prompts. It probes cross-image reasoning, fine-grained instance localization, and the ability to correctly group and reject irrelevant instances.

## Datasets

- **MC-Bench** — total 2000; splits: test (2000)

## Metrics

- `Accuracy (Acc)` — range: percent
  - Binary metric checking whether the model correctly identifies which images contain the target objects described in the prompt, ignoring fine-grained location and instance count.
- `AP50` **(primary)** — range: percent
  - Average Precision at an Intersection over Union (IoU) threshold of 0.5. For prompts describing multiple instance groups, a Hungarian algorithm matches predicted groups to ground-truth groups to maximize mean IoU before computing precision.

## Input / output format

**Input**: A sequence or horizontally concatenated set of multiple images paired with an open-ended text prompt describing one or more target instances.

**Output**: A set of bounding boxes for the predicted target instances, optionally assigned to groups corresponding to the prompt's descriptions.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truth):
    # predictions & ground_truth: list of dicts with 'image_idx', 'boxes', 'group_id'
    acc = 1.0 if set(p['image_idx'] for p in predictions) == set(g['image_idx'] for g in ground_truth) else 0.0
    pred_groups = group_by_id(predictions)
    gt_groups = group_by_id(ground_truth)
    matched_ious = hungarian_match(pred_groups, gt_groups)
    ap50 = compute_ap(matched_ious, iou_threshold=0.5)
    return acc, ap50
```

## Common pitfalls

- Models frequently fail to assign correct groups, often predicting only a single group for prompts requiring multiple.
- Models tend to generate redundant low-confidence bounding boxes on irrelevant/negative images, which artificially inflates localization but severely degrades Accuracy (Acc).
- Performance drops significantly for small/medium objects and negative samples compared to large objects.

## Evidence (verbatim from paper)

> Accuracy (Acc) is used to confirm whether the models can correctly identify which images contain the objects indicated by each text prompt, where the instance quantity and fine-grained location information is not considered. We choose average precision (AP50) as the instance-level metric to verify whether the models can locate the target instances with multi-context inputs. For samples where the text prompt describes multiple groups of instances, we first apply Hungarian algorithm to match each predicted group to the most appropriate ground-truth group, ensuring that the mean intersection over union (IoU) across all predictions is maximized.

## Citation

```bibtex
@misc{xu2024mcbench,
  title={MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs},
  author={Xu et al. (2024)},
  year={2024},
  note={arXiv:2410.12332}
}
```

- arXiv: 2410.12332