Perla 3d Eval

---
name: perla-3d-eval
description: Evaluates a 3D vision-language model's ability to answer questions about indoor scenes and generate dense captions for 3D instances. It probes fine-grained spatial reasoning, object attribute recognition, and scene understanding by comparing generated text against ground-truth annotations using standard natural language generation metrics. Use when the user wants to benchmark on ScanNet, or asks about evaluating this task. Reports CiDEr.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.19774
  bibtex_key: mei2024perla
  confidence: high
---

# perla-3d-eval

> PerLA: Perceptive 3D Language Assistant — Mei et al. (2024) (arXiv:2411.19774, 2024)

## What this evaluates

Evaluates a 3D vision-language model's ability to answer questions about indoor scenes and generate dense captions for 3D instances. It probes fine-grained spatial reasoning, object attribute recognition, and scene understanding by comparing generated text against ground-truth annotations using standard natural language generation metrics.

## Datasets

- **ScanNet** — total 1513; splits: train (1201), val (312)

## Metrics

- `CiDEr` **(primary)** — range: percent
  - TF-IDF weighted n-gram overlap between predicted and ground-truth text. Scores are scaled to [0, 100] following standard NLG evaluation conventions.
- `BLEU-4` — range: percent
  - 4-gram precision with brevity penalty, scaled to [0, 100].
- `METEOR` — range: percent
  - Unigram precision/recall with synonymy and stemming matching, scaled to [0, 100].
- `Rouge-L` — range: percent
  - Longest common subsequence F1 score between predictions and references, scaled to [0, 100].

## Input / output format

**Input**: 3D point cloud scene (40,000 randomly sampled points) paired with a natural language question (for QA) or used alone (for dense captioning).

**Output**: Natural language text response (for QA) or localized 3D instance descriptions with bounding box references (for dense captioning).

## Scoring recipe

```python
def evaluate(predictions, golds, task='qa'):
    scores = {}
    scores['CiDEr'] = cider(predictions, golds)
    scores['BLEU-4'] = bleu(predictions, golds, n=4)
    scores['METEOR'] = meteor(predictions, golds)
    scores['Rouge-L'] = rouge(predictions, golds, 'L')
    if task == 'dense_captioning':
        for thresh in [0.25, 0.5]:
            for m in ['CiDEr', 'BLEU-4', 'METEOR', 'Rouge-L']:
                scores[f'{m}@{thresh}'] = apply_iou_threshold(predictions, golds, m, thresh)
    return scores
```

## Common pitfalls

- Classification-based (CLS) methods select from a fixed answer set while generation-based (GEN) methods produce free text; direct metric comparison requires careful handling of answer vocabularies and tokenization.
- Dense captioning metrics use m@$k$IoU thresholds that vary by dataset (0.25 for ScanRefer, 0.5 for Nr3D); failing to specify the threshold leads to incomparable results.
- Point clouds are downsampled to 40,000 random points per scene, which may underrepresent fine-grained geometry compared to full-resolution scans and bias performance evaluations.

## Evidence (verbatim from paper)

> We follow LL3DA’s evaluation protocol*[[9]]* to evaluate the quality of output responses. We use the abbreviations C, B4, M and R for CiDEr*[[55]]*, BLEU-4*[[43]]*, METEOR*[[3]]*, and Rouge-L*[[34]]*, respectively.

## Citation

```bibtex
@misc{mei2024perla,
  title={PerLA: Perceptive 3D Language Assistant},
  author={Mei et al. (2024)},
  year={2024},
  note={arXiv:2411.19774}
}
```

- arXiv: 2411.19774