Ir3d Bench Eval

---
name: ir3d-bench-eval
description: Evaluates vision-language models' ability to understand 3D scenes by generating executable scene descriptions from a single 2D image. It shifts evaluation from passive captioning to active reconstruction, probing geometric layout, spatial reasoning, object appearance, and semantic attributes. Use when the user wants to benchmark on IR3D-Bench, or asks about evaluating this task. Reports Pixel Distance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.23329
  bibtex_key: liu2025ir3dbench
  confidence: high
---

# ir3d-bench-eval

> IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering — Parker Liu et al. (arXiv:2506.23329, 2025)

## What this evaluates

Evaluates vision-language models' ability to understand 3D scenes by generating executable scene descriptions from a single 2D image. It shifts evaluation from passive captioning to active reconstruction, probing geometric layout, spatial reasoning, object appearance, and semantic attributes.

## Datasets

- **IR3D-Bench** — total ?; splits: test (-1)

## Metrics

- `Pixel Distance` **(primary)** — range: [0, 1]
  - Mean L2 distance between predicted and ground-truth object centers. Lower values indicate better localization.
- `Bounding Box IoU` — range: [0, 1]
  - Intersection over Union between predicted and ground-truth bounding boxes. Higher values indicate better spatial extent estimation.
- `Relational Accuracy` — range: [0, 1]
  - Accuracy of predicted inter-object spatial relations (e.g., proximity, containment, relative position) compared to ground truth.
- `CLIP Score` — range: [0, 1]
  - Cosine similarity between CLIP embeddings of predicted and ground-truth object appearance attributes (color, size, material, shape).
- `LLM Score` — range: [0, 1]
  - Score assigned by an LLM judge evaluating the predicted object appearance and overall scene layout against ground truth.

## Input / output format

**Input**: A single 2D image accompanied by a prompt instructing the VLM to extract geometric information (shape, size, position, material) for each object. Camera intrinsic and extrinsic parameters are fixed and provided to ensure consistent reconstruction.

**Output**: A structured JSON file containing predicted object attributes, bounding boxes, centers, and scene layout.

## Scoring recipe

```python
def compute_metrics(pred_json, gt_json):
    pred_objs = parse_json(pred_json)
    gt_objs = parse_json(gt_json)
    pixel_dist = mean_l2(pred_objs.center, gt_objs.center)
    bbox_iou = compute_iou(pred_objs.bbox, gt_objs.bbox)
    rel_acc = accuracy(pred_objs.relations, gt_objs.relations)
    clip_score = clip_similarity(pred_objs.appearance, gt_objs.appearance)
    llm_score = llm_judge(pred_objs.layout, gt_objs.layout)
    return {
        'pixel_distance': pixel_dist,
        'bbox_iou': bbox_iou,
        'relational_accuracy': rel_acc,
        'clip_score': clip_score,
        'llm_score': llm_score
    }
```

## Common pitfalls

- Several models fail to produce valid JSON outputs, indicating either insufficient 3D understanding or incompatibility with the task format.
- Models often achieve high appearance recognition scores but struggle significantly with spatial reasoning and inter-object relations.
- The benchmark assumes fixed camera parameters; models cannot infer them, which constrains the evaluation to scene attribute prediction rather than full camera recovery.

## Evidence (verbatim from paper)

> Relational Accuracy, which captures reasoning over inter-object spatial relations, is below 0.3 for most models (GPT-4o: 0.28, Gemini: 0.26), showing persistent errors in understanding relative positions, proximity, and containment.

## Citation

```bibtex
@misc{liu2025ir3dbench,
  title={IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering},
  author={Parker Liu et al.},
  year={2025},
  note={arXiv:2506.23329}
}
```

- arXiv: 2506.23329