Instructpart Eval

---
name: instructpart-eval
description: Evaluates Vision-Language Models' ability to perform fine-grained visual grounding and instruction reasoning for part segmentation. It probes whether models can infer task-relevant object parts from natural language instructions or oracle prompts, and assesses their capacity for affordance learning in human-robot interaction contexts. Use when the user wants to benchmark on InstructPart, or asks about evaluating this task. Reports gIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.18291
  bibtex_key: wan2025instructpart
  confidence: high
---

# instructpart-eval

> InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning — Wan et al. (2025) (arXiv:2505.18291, 2025)

## What this evaluates

Evaluates Vision-Language Models' ability to perform fine-grained visual grounding and instruction reasoning for part segmentation. It probes whether models can infer task-relevant object parts from natural language instructions or oracle prompts, and assesses their capacity for affordance learning in human-robot interaction contexts.

## Datasets

- **InstructPart** — total 2400; splits: train (-1), test (-1)

## Metrics

- `gIoU` **(primary)** — range: percent
  - Average of all per-image Intersection-over-Union (IoU) scores across the dataset.
- `cIoU` — range: percent
  - Cumulative intersection over cumulative union across all images in the dataset.
- `P@50` — range: percent
  - Precision metric where a prediction is considered a true positive if the IoU ratio exceeds 0.5.
- `P@50:95` — range: percent
  - Precision averaged across IoU thresholds from 0.50 to 0.95 in increments of 0.05.

## Input / output format

**Input**: RGB image paired with a natural language instruction (either an oracle referring prompt containing explicit object/part names, or a task-oriented reasoning prompt requiring implicit part inference).

**Output**: Binary segmentation mask corresponding to the specified part.

## Scoring recipe

```python
def compute_metrics(pred_masks, gt_masks):
    ious = [np.sum(p & g) / np.sum(p | g) for p, g in zip(pred_masks, gt_masks)]
    gIoU = np.mean(ious) * 100
    cIoU = (np.sum([np.sum(p & g) for p, g in zip(pred_masks, gt_masks)]) /
            np.sum([np.sum(p | g) for p, g in zip(pred_masks, gt_masks)])) * 100
    p50 = np.mean([1.0 if iou > 0.5 else 0.0 for iou in ious]) * 100
    p50_95 = np.mean([np.mean([1.0 if iou > t else 0.0 for t in np.arange(0.50, 0.96, 0.05)]) for iou in ious]) * 100
    return gIoU, cIoU, p50, p50_95
```

## Common pitfalls

- Confusing gIoU (per-image average) with cIoU (dataset-level cumulative intersection/union), which can yield significantly different values on imbalanced datasets.
- P@50:95 requires averaging precision across multiple IoU thresholds (0.50 to 0.95 step 0.05), not just evaluating at a single threshold.
- GPT-4V cannot output masks directly; the paper's pipeline uses Grounding-DINO + SAM, so poor results may stem from the grounding/masking step rather than the VLM's reasoning.

## Evidence (verbatim from paper)

> To evaluate our approach, we use standard metrics in LISA, namely gIoU and cIoU. gIoU reflects the average of all per-image Intersection-over-Unions (IoUs), while cIoU is defined by the cumulative intersection over the cumulative union. To evaluate the precision of the models, we adopt Precision@50 (P@50) metric as the previous referring segmentation works and develop a Precision@50:95 (P@50:95) metric according to COCO. The P@50 metric considers a mask to be a true positive when the IoU ratio exceeds 0.5, and P@50:95 calculates across a range of IoU thresholds from 0.50 to 0.95 with increments of 0.05, then averages across all the thresholds.

## Citation

```bibtex
@misc{wan2025instructpart,
  title={InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning},
  author={Wan et al. (2025)},
  year={2025},
  note={arXiv:2505.18291}
}
```

- arXiv: 2505.18291