Multi3drefer Eval

---
name: multi3drefer-eval
description: Evaluates 3D visual grounding models on their ability to localize zero, single, or multiple objects in 3D scenes based on natural language descriptions. It tests whether models can correctly identify target objects while handling ambiguous references, distractors, and zero-target cases. Use when the user wants to benchmark on Multi3DRefer, or asks about evaluating this task. Reports Acc@0.5.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.05251
  bibtex_key: zhang2023multi3drefer
  confidence: high
---

# multi3drefer-eval

> Multi3DRefer: Grounding Text Description to Multiple 3D Objects — Zhang et al. (2023) (arXiv:2309.05251, 2023)

## What this evaluates

Evaluates 3D visual grounding models on their ability to localize zero, single, or multiple objects in 3D scenes based on natural language descriptions. It tests whether models can correctly identify target objects while handling ambiguous references, distractors, and zero-target cases.

## Datasets

- **Multi3DRefer** — total ?; splits: train (3826), val (2862), test (13178)

## Metrics

- `Acc@0.5` **(primary)** — range: percent
  - Percentage of correctly grounded instances where the Intersection over Union (IoU) between the predicted and ground-truth bounding box exceeds 0.5.
- `F1@0.5` — range: percent
  - Harmonic mean of precision and recall computed at an IoU threshold of 0.5, evaluating both correct localization and correct rejection of zero-target cases.

## Input / output format

**Input**: 3D point cloud scene (coordinates, normals, multi-view features), natural language description, and optionally ground-truth or predicted bounding boxes for object proposals.

**Output**: Predicted 3D bounding box (or set of boxes) corresponding to the text description.

## Scoring recipe

```python
def compute_acc_at_05(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if iou(p, g) >= 0.5)
    return (correct / len(gold)) * 100

def compute_f1_at_05(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if iou(p, g) >= 0.5)
    fp = len(predictions) - tp
    fn = len(gold) - tp
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- Must explicitly state whether ground-truth or predicted bounding boxes are used, as performance drops significantly with predicted boxes.
- Zero-target cases require models to correctly reject all proposals; failing to do so heavily penalizes recall and F1.
- Distractors (objects of the same class as the target) are explicitly included in some splits and drastically lower performance if not handled.

## Evidence (verbatim from paper)

> We report F1 scores with GT boxes on Multi3DRefer val set.

## Citation

```bibtex
@misc{zhang2023multi3drefer,
  title={Multi3DRefer: Grounding Text Description to Multiple 3D Objects},
  author={Zhang et al. (2023)},
  year={2023},
  note={arXiv:2309.05251}
}
```

- arXiv: 2309.05251