Objvariantensemble Eval

---
name: objvariantensemble-eval
description: Evaluates 3D grounding models' spatial reasoning and fine-grained object distinction capabilities by testing their ability to locate a target object among visually similar distractors in ensembled point cloud scenes. Use when the user wants to benchmark on OVE (ObjVariantEnsemble), or asks about evaluating this task. Reports ACC@0.25.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.14837
  bibtex_key: cao2024objvariantensemble
  confidence: high
---

# objvariantensemble-eval

> ObjVariantEnsemble: Advancing Point Cloud LLM Evaluation in Challenging Scenes with Subtly Distinguished Objects — Cao et al. (2024) (arXiv:2412.14837, 2024)

## What this evaluates

Evaluates 3D grounding models' spatial reasoning and fine-grained object distinction capabilities by testing their ability to locate a target object among visually similar distractors in ensembled point cloud scenes.

## Datasets

- **OVE (ObjVariantEnsemble)** — total 75000; splits: test (-1)

## Metrics

- `ACC@0.25` **(primary)** — range: [0, 1]
  - Accuracy calculated as the fraction of correctly localized objects where the Intersection over Union (IoU) between the predicted and ground-truth 3D bounding boxes is at least 0.25.

## Input / output format

**Input**: A 3D point cloud scene containing a target object and multiple distractors, accompanied by a text query specifying the target's distinguishing attributes or spatial relationship.

**Output**: A 3D bounding box (coordinates) or spatial location corresponding to the target object in the scene.

## Scoring recipe

```python
correct = 0
for pred_box, gt_box in zip(predictions, ground_truth):
    iou = calculate_3d_iou(pred_box, gt_box)
    if iou >= 0.25:
        correct += 1
return correct / len(predictions)
```

## Common pitfalls

- Models may exploit coarse color or shape cues present in training data, failing when only spatial location distinguishes the target.
- Scale mismatches and overlapping bounding boxes in ensembled scenes can artificially inflate or deflate IoU scores if preprocessing is not standardized.
- The benchmark's fine-grained distinction types (location-only vs location+shape/color) require separate evaluation to avoid averaging out specific reasoning failures.

## Evidence (verbatim from paper)

> Performance(ACC@0.25) of 3D grounding models on OVE across different numbers of distractors, considering only location.

## Citation

```bibtex
@misc{cao2024objvariantensemble,
  title={ObjVariantEnsemble: Advancing Point Cloud LLM Evaluation in Challenging Scenes with Subtly Distinguished Objects},
  author={Cao et al. (2024)},
  year={2024},
  note={arXiv:2412.14837}
}
```

- arXiv: 2412.14837