Point It Out Eval

---
name: point-it-out-eval
description: Evaluates vision-language models' embodied reasoning and visual grounding capabilities across three hierarchical stages: referred-object localization, task-driven pointing, and multi-step visual trace prediction in real-world scenarios. Use when the user wants to benchmark on Point-It-Out (PIO), or asks about evaluating this task. Reports score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.25794
  bibtex_key: xue2025pointitout
  confidence: medium
---

# point-it-out-eval

> Point-It-Out: Benchmarking Embodied Reasoning for Vision Language Models in Multi-Stage Visual Grounding — Xue et al. (2025) (arXiv:2509.25794, 2025)

## What this evaluates

Evaluates vision-language models' embodied reasoning and visual grounding capabilities across three hierarchical stages: referred-object localization, task-driven pointing, and multi-step visual trace prediction in real-world scenarios.

## Datasets

- **Point-It-Out (PIO)** — total ?; splits: test (-1); repo https://github.com/xavihart/PIO

## Metrics

- `score` **(primary)** — range: [0, 1]
  - Normalized accuracy score (0 to 1) for bounding box or point prediction in stages S1 and S2. Exact threshold or IoU convention is not specified in the provided text.
- `rating` — range: other
  - Human and GPT-based assessment score for multi-step trajectory generation in stage S3, reported on a 1 to 5 scale.

## Input / output format

**Input**: Visual input (images/video) paired with text instructions specifying a target object, task constraint, or action to perform.

**Output**: Bounding box coordinates or point coordinates for S1/S2; a sequence of coordinates forming a visual trace/trajectory for S3.

## Scoring recipe

```python
def score_localization(pred, gt):
    # Check if prediction matches ground truth within unspecified threshold
    return 1.0 if matches_threshold(pred, gt) else 0.0

def score_trace(pred, gt):
    # Aggregate human and GPT ratings on a 1-5 scale
    return average(human_ratings + gpt_ratings)

# Pipeline
for stage in [S1, S2]:
    acc = mean([score_localization(p, g) for p, g in zip(preds, golds)])
for stage in [S3]:
    rating = score_trace(preds, golds)
```

## Common pitfalls

- Models often succeed on simple language reasoning tasks but fail significantly on fine-grained part localization and affordance/contact prediction.
- Grounding-specialized models (e.g., MoLMO, Qwen) excel at isolated localization but underperform in multi-step temporal planning (S3) compared to general-purpose VLMs like GPT-4o or Gemini-2.5 Pro.
- Evaluation combines automated localization metrics with subjective human and GPT-based ratings for trace quality, requiring careful alignment of scoring rubrics.

## Evidence (verbatim from paper)

> Even strong models like Qwen and MoLMO score below 0.5. In S2, while most models handle recommendation well, they struggle with affordance and contact prediction. Although MoLMO is the best in affordance, it still scores below 0.4. ... Gemini-2.5-Pro achieves almost 4 out of 5 in Figure[8] (right), it can be attributed to the inclusion of embodied data and grounding data in the strong model.

## Citation

```bibtex
@misc{xue2025pointitout,
  title={Point-It-Out: Benchmarking Embodied Reasoning for Vision Language Models in Multi-Stage Visual Grounding},
  author={Xue et al. (2025)},
  year={2025},
  note={arXiv:2509.25794}
}
```

- arXiv: 2509.25794