Real 3dqa Eval

---
name: real-3dqa-eval
description: Evaluates whether 3D-LLMs genuinely comprehend 3D spatial relationships rather than relying on linguistic shortcuts or text-only priors. It filters out 3D-independent questions and measures consistency across viewpoint rotations to penalize superficial pattern matching. Use when the user wants to benchmark on Real-3DQA, or asks about evaluating this task. Reports Viewpoint Rotation Score (VRS).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.23523
  bibtex_key: ma2026do3dlarge
  confidence: high
---

# real-3dqa-eval

> Do 3D Large Language Models Really Understand 3D Spatial Relationships? — Ma et al. (2026) (arXiv:2603.23523, 2026)

## What this evaluates

Evaluates whether 3D-LLMs genuinely comprehend 3D spatial relationships rather than relying on linguistic shortcuts or text-only priors. It filters out 3D-independent questions and measures consistency across viewpoint rotations to penalize superficial pattern matching.

## Datasets

- **Real-3DQA** — total ?; splits: test (-1)

## Metrics

- `Viewpoint Rotation Score (VRS)` **(primary)** — range: percent
  - Measures robustness to spatial variations on a 0–100% scale. For each batch of 4 questions (original + 3 rotations), calculate P_k as the percentage of instances where at least k questions are answered correctly (k ∈ {1,2,3,4}). VRS is the mean of P_1 through P_4. Higher values indicate true 3D understanding.
- `refined exact match (EM_R)` — range: percent
  - Standard exact-match evaluation adapted for situated question answering, as defined in prior SQA3D literature.

## Input / output format

**Input**: Situation description containing 3D spatial context, a question, and optionally a scene graph. For VRS evaluation, inputs are grouped into batches of four related questions (original viewpoint plus 90°, 180°, and 270° rotations) sharing the same underlying layout.

**Output**: Model's predicted answer string for each question.

## Scoring recipe

```python
correct_counts = []
for pred_batch, gold_batch in zip(predictions, golds):
    correct_counts.append(sum(1 for p, g in zip(pred_batch, gold_batch) if p == g))
N_total = len(correct_counts)
P_k = []
for k in range(1, 5):
    N_k = sum(1 for c in correct_counts if c >= k)
    P_k.append((N_k / N_total) * 100)
return sum(P_k) / 4
```

## Common pitfalls

- Models can achieve high scores on standard SQA3D by exploiting linguistic shortcuts or text-only priors without actual 3D reasoning.
- VRS requires consistency across all four viewpoint rotations; failing to answer all variants correctly significantly penalizes the score.
- The benchmark explicitly filters out questions answerable by GPT-4o-mini from text alone, so models relying purely on language priors will fail.

## Evidence (verbatim from paper)

> During the evaluation, each batch consists of four related questions: the original and its three rotated variants (90°, 180° and 270°). After processing the entire data set, we calculate the percentage of instances in which the model correctly answers at least $k$ questions, where $k\in{1,2,3,4}$. Formally, let $N_{k}$ be the number of instances where at least $k$ questions in a batch are answered correctly, and $N_{\text{total}}$ be the total number of instances. The percentage for each $k$ is given by: $P_{k}\=\frac{N_{k}}{N_{\text{total}}}\times 100$, where $P_{k}$ represents the accuracy percentage for answering at least $k$ questions correctly. The final VRS metric is computed as the mean of these percentages: $\text{VRS}\=\frac{1}{4}\sum_{k\=1}^{4}P_{k}$.

## Citation

```bibtex
@misc{ma2026do3dlarge,
  title={Do 3D Large Language Models Really Understand 3D Spatial Relationships?},
  author={Ma et al. (2026)},
  year={2026},
  note={arXiv:2603.23523}
}
```

- arXiv: 2603.23523