Paco Lvis Instruct Eval

---
name: paco-lvis-instruct-eval
description: Evaluates a model's ability to follow complex natural-language instructions to segment specific object instances in images. It probes fine-grained instance grounding while maintaining concept-level recall across simple and complex prompts. Use when the user wants to benchmark on PACO-LVIS-Instruct, or asks about evaluating this task. Reports gIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.04585
  bibtex_key: li2025sam3i
  confidence: high
---

# paco-lvis-instruct-eval

> SAM3-I: Segment Anything with Instructions — Li et al. (2025) (arXiv:2512.04585, 2025)

## What this evaluates

Evaluates a model's ability to follow complex natural-language instructions to segment specific object instances in images. It probes fine-grained instance grounding while maintaining concept-level recall across simple and complex prompts.

## Datasets

- **PACO-LVIS-Instruct** — total 45308; splits: train (-1), val (-1), test (-1)

## Metrics

- `gIoU` **(primary)** — range: [0, 1]
  - Average Intersection-over-Union (IoU) across all images in the dataset.
- `P@50` — range: percent
  - Percentage of test samples where the predicted segmentation mask achieves an IoU of at least 0.5 with the ground-truth mask.

## Input / output format

**Input**: An input image paired with a natural-language instruction specifying the target instance (concept, simple, or complex instruction).

**Output**: A predicted binary segmentation mask for the target instance specified by the instruction.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truths):
    ious = [iou(pred, gt) for pred, gt in zip(predictions, ground_truths)]
    gIoU = sum(ious) / len(ious)
    P50 = sum(1 for i in ious if i >= 0.5) / len(ious) * 100
    return gIoU, P50
```

## Common pitfalls

- Comparing against baselines that use iterative multi-round agent checking (e.g., 3 rounds) rather than single-pass inference, which inflates baseline performance unfairly.
- The instruction-mask pairs are generated via a specific pipeline; evaluating on out-of-distribution instructions without this generation context may yield inconsistent results.
- The paper notes this is a preliminary feasibility study, so results should not be interpreted as state-of-the-art for the broader PIS task.

## Evidence (verbatim from paper)

> To train and evaluate PIS models, we extend the PACO-LVIS*[paco]* benchmark with rich natural-language instructions, resulting in PACO-LVIS-Instruct. The dataset includes 45,308 images, 210,876 object-level masks, and 843,504 positive instructions, with an average instruction length of 15.3 words. The train/val/test splits follow those of the original PACO-LVIS benchmark, and all instruction–mask pairs are produced using the pipeline introduced in Sec.[5]. For evaluation, we adopt the gIoU and P@50 metrics. Here gIoU *[lai2024lisa]* is defined by the average of all per-image Intersection-over-Union (IoU). Precision@0.5 (P@50 *[hu2016segmentation]*) measures the percentage of test samples whose predicted segmentation mask achieves an Intersection-over-Union (IoU) of at least 0.5 with the corresponding ground-truth mask. The percent symbol is omitted for readability.

## Citation

```bibtex
@misc{li2025sam3i,
  title={SAM3-I: Segment Anything with Instructions},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2512.04585}
}
```

- arXiv: 2512.04585