Gitm Mr Eval

---
name: gitm-mr-eval
description: Evaluates visual-linguistic relation understanding by requiring models to classify image-text matching, ground matched objects with bounding boxes, and identify mismatched relations from candidates. It specifically probes data efficiency and length generalization capabilities in out-of-distribution settings. Use when the user wants to benchmark on GITM-MR, or asks about evaluating this task. Reports Match%.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.01236
  bibtex_key: wu2023grounded
  confidence: high
---

# gitm-mr-eval

> Grounded Image Text Matching with Mismatched Relation Reasoning — Wu et al. (2023) (arXiv:2308.01236, 2023)

## What this evaluates

Evaluates visual-linguistic relation understanding by requiring models to classify image-text matching, ground matched objects with bounding boxes, and identify mismatched relations from candidates. It specifically probes data efficiency and length generalization capabilities in out-of-distribution settings.

## Datasets

- **GITM-MR** — total ?; splits: train (-1), test (-1), ood_test (-1)

## Metrics

- `Match%` **(primary)** — range: percent
  - Classification accuracy for determining whether the text describes the image.
- `Grounding%` — range: percent
  - Recall@1 for object grounding. A case is correct only if the model predicts a match and the predicted bounding box has an IoU ≥ 0.5 with the ground-truth box.
- `MRR%` — range: percent
  - Top-1 accuracy for mismatch reasoning. A case is correct only if the model correctly classifies it as a mismatch and selects the exact ground-truth relation from the candidate set.

## Input / output format

**Input**: An image and a natural language sentence describing potential objects or relations in the image.

**Output**: A binary classification label (match/mismatch), bounding box coordinates for matched objects, and a selected candidate relation ID for mismatched cases.

## Scoring recipe

```python
def compute_metrics(preds, gold):
    match_acc = sum(p == g for p, g in zip(preds['match'], gold['match'])) / len(gold['match'])
    ground_correct = sum(1 for p_box, g_box, p_m, g_m in zip(preds['boxes'], gold['boxes'], preds['match'], gold['match']) if p_m == g_m and iou(p_box, g_box) >= 0.5)
    grounding_acc = ground_correct / len(gold['boxes'])
    mrr_acc = sum(p == g for p, g in zip(preds['mismatch_rel'], gold['mismatch_rel'])) / len(gold['mismatch_rel'])
    return match_acc, grounding_acc, mrr_acc
```

## Common pitfalls

- Grounding accuracy requires both correct match classification AND IoU ≥ 0.5; failing either yields zero points.
- MRR is strict top-1 accuracy, not a ranking metric; models must correctly classify mismatch AND pick the exact candidate.
- Out-of-distribution length generalization tests often reveal overfitting to training sentence lengths rather than true compositional reasoning.

## Evidence (verbatim from paper)

> The evaluation metrics include classification accuracy for three subtasks. The grounding result for a matched case is considered as correct when it is identified as matching and the predicted box has at least 0.5 IoU with its ground-truth location. The grounding accuracy (i.e. Recall@1) is the ratio of correctly grounded cases. For mismatch reasoning, a mismatched case needs to be correctly classified and the mismatched relation should be accurately selected from the candidate set. The MRR accuracy is the top-1 accuracy among the candidates.

## Citation

```bibtex
@misc{wu2023grounded,
  title={Grounded Image Text Matching with Mismatched Relation Reasoning},
  author={Wu et al. (2023)},
  year={2023},
  note={arXiv:2308.01236}
}
```

- arXiv: 2308.01236