Gaussianvlm Eval

---
name: gaussianvlm-eval
description: Evaluates a 3D vision-language model's ability to perform object-centric and scene-centric reasoning tasks, including captioning, question answering, embodied planning, and dialogue. It probes spatial grounding, semantic abstraction, and robust generalization to out-of-domain real-world scene representations. Use when the user wants to benchmark on ScanRefer, ScanQA, Nr3D, SQA3D, 3D-LLM ScanNet subset, ScanNet++ (OOD object counting), or asks about evaluating this task. Reports Exact-match accuracy (EM1).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.00886
  bibtex_key: halacheva2025gaussianvlm
  confidence: high
---

# gaussianvlm-eval

> GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyond — Halacheva et al. (2025) (arXiv:2507.00886, 2025)

## What this evaluates

Evaluates a 3D vision-language model's ability to perform object-centric and scene-centric reasoning tasks, including captioning, question answering, embodied planning, and dialogue. It probes spatial grounding, semantic abstraction, and robust generalization to out-of-domain real-world scene representations.

## Datasets

- **ScanRefer** — total ?; splits: test (-1)
- **ScanQA** — total ?; splits: test (-1)
- **Nr3D** — total ?; splits: test (-1)
- **SQA3D** — total ?; splits: test (-1)
- **3D-LLM ScanNet subset** — total ?; splits: test (-1)
- **ScanNet++ (OOD object counting)** — total 1000; splits: val (1000)

## Metrics

- `CIDEr` — range: percent
  - Consensus-based Image Description Evaluation metric that weights n-grams by their IDF scores to penalize common words and reward distinctive phrases.
- `BLEU-4` — range: percent
  - Precision-based metric computing the geometric mean of modified n-gram precisions up to order 4, with a brevity penalty.
- `METEOR` — range: percent
  - Metric that aligns predictions and references at word/phrase level, incorporating synonym matching, stemming, and exact matches to compute a harmonic mean of precision and recall.
- `ROUGE` — range: percent
  - Recall-Oriented Understudy for Gisting Evaluation; measures overlap of n-grams, longest common subsequence, or word sequences between prediction and reference.
- `Sentence-BERT similarity (Sim)` — range: [0, 1]
  - Computes cosine similarity between dense vector embeddings of the predicted and reference texts using a pre-trained Sentence-BERT model.
- `Exact-match accuracy (EM1)` **(primary)** — range: percent
  - Percentage of instances where the model's generated answer exactly matches the ground truth answer string.
- `Accuracy` — range: percent
  - Proportion of correctly answered questions, specifically used for the out-of-domain object counting task.

## Input / output format

**Input**: 3D scene representation encoded as 40k randomly sampled Gaussian splats (or point clouds for baselines) paired with a natural language prompt or question.

**Output**: Free-form text generation (captions, answers, plans, or dialogue turns).

## Scoring recipe

```python
def compute_metrics(predictions, references):
    scores = {}
    scores['EM'] = sum(1 for p, r in zip(predictions, references) if p.strip() == r.strip()) / len(predictions)
    scores['Accuracy'] = scores['EM']
    scores['CIDEr'] = compute_cider(predictions, references)
    scores['BLEU-4'] = compute_bleu(predictions, references, n=4)
    scores['METEOR'] = compute_meteor(predictions, references)
    scores['ROUGE'] = compute_rouge(predictions, references)
    scores['Sim'] = compute_sbert_similarity(predictions, references)
    return scores
```

## Common pitfalls

- BLEU-4 and CIDEr are excluded for object-centric tasks because they reward superficial n-gram overlap and can assign misleadingly high scores to captions that describe context correctly but identify the wrong object.
- Exact-match accuracy is used for SQA3D and object counting, but semantic similarity metrics (Sentence-BERT, METEOR, ROUGE) are preferred for scene-centric tasks to capture paraphrasing and partial matches without penalizing valid alternative phrasings.

## Evidence (verbatim from paper)

> For scene-centric tasks, where captions and answers typically encompass diverse and richly descriptive content, we report standard metrics including CIDEr, BLEU-4, METEOR, ROUGE, exact-match accuracy, and Sentence-BERT similarity. For object-centric tasks, we exclude BLEU-4 and CIDEr. BLEU, a precision-based metric, and CIDEr are overly sensitive to superficial n-gram overlap, rendering them unsuitable for evaluating long-form object captions.

## Citation

```bibtex
@misc{halacheva2025gaussianvlm,
  title={GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyond},
  author={Halacheva et al. (2025)},
  year={2025},
  note={arXiv:2507.00886}
}
```

- arXiv: 2507.00886