Robointer Vqa Eval

---
name: robointer-vqa-eval
description: Evaluates embodied reasoning capabilities of vision-language models on robotic manipulation tasks. It probes spatial understanding and generation (e.g., object grounding, grasp pose prediction) and temporal understanding and generation (e.g., motion trace reconstruction, multi-step planning) across diverse indoor and tabletop scenarios. Use when the user wants to benchmark on RoboInter-VQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.09973
  bibtex_key: li2026robointer
  confidence: high
---

# robointer-vqa-eval

> RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulation — Hao Li et al. (2026) (arXiv:2602.09973, 2026)

## What this evaluates

Evaluates embodied reasoning capabilities of vision-language models on robotic manipulation tasks. It probes spatial understanding and generation (e.g., object grounding, grasp pose prediction) and temporal understanding and generation (e.g., motion trace reconstruction, multi-step planning) across diverse indoor and tabletop scenarios.

## Datasets

- **RoboInter-VQA** — total ?; splits: train (-1), val (-1), test (7246); repo https://github.com/InternRobotics/RoboInter

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly answered selection, judgment, and generation tasks. For spatial/temporal generation tasks, correctness is determined by IoU thresholds for bounding boxes or exact match for categorical outputs.

## Input / output format

**Input**: Video frames or image sequences paired with text prompts conditioning on varying prior information (e.g., past subtasks, overall instructions, or initial waypoints).

**Output**: Text responses for selection/judgment tasks; 2D bounding boxes, grasp poses, placement proposals, or gripper coordinates for spatial generation; motion traces or multi-step action plans for temporal generation.

## Scoring recipe

```python
def score(predictions, gold):
    correct = 0
    for pred, gold_item in zip(predictions, gold):
        if gold_item['type'] in ['selection', 'judgment']:
            if pred.strip().lower() == gold_item['answer'].strip().lower():
                correct += 1
        elif gold_item['type'] in ['spatial_gen', 'temporal_gen']:
            if compute_iou(pred['coords'], gold_item['coords']) >= 0.5:
                correct += 1
    return correct / len(gold)
```

## Common pitfalls

- Information leakage between training and validation sets if the 7,246-video evaluation pool is not strictly held out.
- Generation tasks vary in difficulty based on contextual completeness (e.g., with/without initial waypoints), requiring careful stratified reporting.
- Raw data lacks reliable camera parameters, so end-effector traces and 3D-to-2D projections rely on estimated calibration matrices that may introduce noise.

## Evidence (verbatim from paper)

> To prevent information leakage between training and validation, we carefully designate 7,246 videos as the evaluation pool with the remaining data used for training, and sample validation sets for each question category from this pool. Tasks are organized along two axes: (i) intermediate representation type (spatial vs. temporal) and (ii) target capability (understanding vs. generation).

## Citation

```bibtex
@misc{li2026robointer,
  title={RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulation},
  author={Hao Li et al. (2026)},
  year={2026},
  note={arXiv:2602.09973}
}
```

- arXiv: 2602.09973