Padt Unified Vision Eval

---
name: padt-unified-vision-eval
description: Evaluates a multimodal large language model's ability to perform visual grounding, segmentation, open-vocabulary detection, and referring image captioning by predicting structured visual outputs directly from interleaved visual reference tokens and text. Use when the user wants to benchmark on RefCOCO/+/g, COCO 2017, RIC, or asks about evaluating this task. Reports IoU@0.5 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.01954
  bibtex_key: su2025patchasdecodabletoken
  confidence: high
---

# padt-unified-vision-eval

> Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs — Su et al. (2025) (arXiv:2510.01954, 2025)

## What this evaluates

Evaluates a multimodal large language model's ability to perform visual grounding, segmentation, open-vocabulary detection, and referring image captioning by predicting structured visual outputs directly from interleaved visual reference tokens and text.

## Datasets

- **RefCOCO/+/g** — total ?; splits: val (-1), test-A (-1), test-B (-1), test (-1)
- **COCO 2017** — total ?; splits: val (-1)
- **RIC** — total ?; splits: val (-1)

## Metrics

- `IoU@0.5 accuracy` **(primary)** — range: percent
  - Accuracy is computed as the fraction of predictions where the Intersection over Union (IoU) between the predicted bounding box and the ground-truth box exceeds 0.5.
- `cIoU` — range: [0, 1]
  - centroid IoU, measuring the overlap between predicted and ground-truth segmentation masks while penalizing centroid displacement.
- `AP@[50:95]` — range: [0, 1]
  - Average Precision averaged over IoU thresholds from 0.50 to 0.95 in steps of 0.05, standard for COCO object detection evaluation.
- `CIDEr-D, Meteor, ROUGE-L, BLEU-4, GP, GR` — range: [0, 1] | percent
  - Standard NLP metrics for caption quality (CIDEr-D, Meteor, ROUGE-L, BLEU-4) combined with GreedyPrecision (GP) and GreedyRecall (GR) for grounding accuracy in referring image captioning.

## Input / output format

**Input**: An image and a natural language query/prompt describing the target object or scene.

**Output**: A sequence of Visual Reference Tokens (VRTs) interleaved with text, which are decoded into bounding boxes, segmentation masks, or descriptive captions.

## Scoring recipe

```python
def score_rec(pred_box, gt_box):
    iou = compute_iou(pred_box, gt_box)
    return 1.0 if iou > 0.5 else 0.0

def score_res(pred_mask, gt_mask):
    return compute_centroid_iou(pred_mask, gt_mask)

def score_coco(pred_boxes, gt_boxes):
    return coco_eval.compute_ap(pred_boxes, gt_boxes, iou_thresh=[0.50, 0.55, ..., 0.95])

def score_ric(pred_caption, gt_captions, gt_caption):
    return {
        'CIDEr-D': cider_score(pred_caption, gt_captions),
        'Meteor': meteor_score(pred_caption, gt_captions),
        'ROUGE-L': rouge_l_score(pred_caption, gt_caption),
        'BLEU-4': bleu4_score(pred_caption, gt_caption),
        'GP': greedy_precision(pred_caption, gt_caption),
        'GR': greedy_recall(pred_caption, gt_caption)
    }
```

## Common pitfalls

- Using coordinate-based text representations instead of the paper's Visual Reference Tokens (VRTs) will break the unified paradigm and yield significantly lower performance.
- Evaluating on RefCOCOg requires using the official 'test' split rather than 'val' to match reported numbers, as test-A/B splits are specific to RefCOCO/RefCOCO+.
- For COCO open-vocabulary detection, models must predict bounding boxes without relying on closed-set class labels; using fixed class priors inflates AP scores.

## Evidence (verbatim from paper)

> The Referring Expression Comprehension (REC) task evaluates an MLLM’s ability to localize objects given natural language descriptions, where a prediction is considered correct if its IoU with the ground-truth box exceeds 50%. We adopt cIoU as the evaluation metric, and results are reported in Tab.[2]. PaDT and PaDT Pro (3B) deliver strong improvements, reaching 1.45 CIDEr, 0.304 Meteor, 0.501 ROUGE-L, 0.467 BLEU-4, and top detection scores of 82.3% GreedyPrecision (GP) and 45.1% GreddyRecall (GR).

## Citation

```bibtex
@misc{su2025patchasdecodabletoken,
  title={Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs},
  author={Su et al. (2025)},
  year={2025},
  note={arXiv:2510.01954}
}
```

- arXiv: 2510.01954