C2rope 3d Vqa Eval

---
name: c2rope-3d-vqa-eval
description: Evaluates a 3D large multimodal model's ability to perform spatial reasoning and visual question answering on multi-view 3D scene data. It probes the model's capacity to retain early visual context, understand spatial relationships, and generate accurate text responses to complex 3D scene queries. Use when the user wants to benchmark on ScanQA, SQA3D, or asks about evaluating this task. Reports EM@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.10551
  bibtex_key: ye2026c2rope
  confidence: high
---

# c2rope-3d-vqa-eval

> C^2ROPE: Causal Continuous Rotary Positional Encoding for 3D Large Multimodal-Models Reasoning — Ye et al. (2026) (arXiv:2602.10551, 2026)

## What this evaluates

Evaluates a 3D large multimodal model's ability to perform spatial reasoning and visual question answering on multi-view 3D scene data. It probes the model's capacity to retain early visual context, understand spatial relationships, and generate accurate text responses to complex 3D scene queries.

## Datasets

- **ScanQA** — total 33400; splits: test (-1), val (-1)
- **SQA3D** — total 19000; splits: test (-1), val (-1)

## Metrics

- `EM@1` **(primary)** — range: [0, 1]
  - Exact Match at 1: 1 if the generated response exactly matches the ground truth answer (case-insensitive, stripped), else 0. Averaged over the dataset.
- `EM@R` — range: [0, 1]
  - Exact Match Recall: Measures the recall of exact matches, likely across multiple generated candidates or a relaxed matching criterion as used in the SQA3D benchmark.
- `B-4` — range: [0, 1]
  - BLEU-4: N-gram precision up to 4-grams, typically with brevity penalty, comparing generated text to reference answers.
- `MET` — range: [0, 1]
  - METEOR: Metric for Evaluation of Translation with Explicit ORdering, considering synonymy, stemming, and alignment between generated and reference text.
- `RGE` — range: [0, 1]
  - ROUGE-L: Recall-Oriented Understudy for Gisting Evaluation based on the longest common subsequence between generated and reference text.
- `CIDEr` — range: [0, 1]
  - Consensus-based Image Description Evaluation: TF-IDF weighted n-gram similarity that measures consensus between generated captions and multiple references.

## Input / output format

**Input**: Multi-view 3D scene images (16 views per scene, each encoded into 576 2D patches, totaling 9,216 visual tokens) paired with a natural language question/instruction.

**Output**: A single natural language text response answering the question.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    em1 = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower()) / len(golds)
    # EM@R, BLEU-4, METEOR, ROUGE-L, CIDEr computed via standard NLG evaluation libraries (e.g., sacrebleu, rouge-score, cider)
    return {'EM@1': em1, 'EM@R': 'computed via standard NLG libraries', 'B-4': 'computed via standard NLG libraries', 'MET': 'computed via standard NLG libraries', 'RGE': 'computed via standard NLG libraries', 'CIDEr': 'computed via standard NLG libraries'}
```

## Common pitfalls

- Models often hallucinate spatial relationships when visual tokens are long (9,216 patches), leading to incorrect answers despite high BLEU/CIDEr scores.
- EM@R is not standard exact match; it likely measures recall of exact matches across multiple generated candidates or a relaxed matching criterion, which can be ambiguous without code.
- Evaluation relies on human-annotated (ScanQA) vs GPT-4-generated (SQA3D) questions, which may differ in difficulty and phrasing, affecting cross-dataset comparison.

## Evidence (verbatim from paper)

> As reported in Tab.[I], compared to the baseline LLaVA-3D, our method achieves consistent improvements across all five evaluation metrics on ScanQA, with gains of +4.3 on EM@1, +8.5 on B-4, +13.4 on MET, +2.5 on RGE, and +18.1 on CIDEr.

## Citation

```bibtex
@misc{ye2026c2rope,
  title={C^2ROPE: Causal Continuous Rotary Positional Encoding for 3D Large Multimodal-Models Reasoning},
  author={Ye et al. (2026)},
  year={2026},
  note={arXiv:2602.10551}
}
```

- arXiv: 2602.10551