Search3d Lerf Eval

---
name: search3d-lerf-eval
description: Evaluates a model's ability to perform open-vocabulary segmentation and localization in 3D radiance fields, specifically testing its capacity to understand and process hierarchical queries (e.g., object parts relative to whole objects) versus simple object-level queries. Use when the user wants to benchmark on Search3D (adapted), LERF dataset, or asks about evaluating this task. Reports mIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.21441
  bibtex_key: weijler2025openhype
  confidence: high
---

# search3d-lerf-eval

> OpenHype: Hyperbolic Embeddings for Hierarchical Open-Vocabulary Radiance Fields — Weijler et al. (2025) (arXiv:2510.21441, 2025)

## What this evaluates

Evaluates a model's ability to perform open-vocabulary segmentation and localization in 3D radiance fields, specifically testing its capacity to understand and process hierarchical queries (e.g., object parts relative to whole objects) versus simple object-level queries.

## Datasets

- **Search3D (adapted)** — total ?; splits: test (-1)
- **LERF dataset** — total ?; splits: test (-1)

## Metrics

- `mIoU` **(primary)** — range: percent
  - Mean Intersection over Union computed between the predicted segmentation mask and the ground truth annotated masks across all queries.
- `accuracy` — range: percent
  - Localization accuracy defined as the percentage of queries where the pixel with the highest similarity score falls inside the ground truth bounding box of the target object.

## Input / output format

**Input**: Hierarchical text prompts (e.g., '[OBJ]', '[PART] of a [OBJ]', or compositional phrases like 'leg of the blue chair') provided to a radiance field model for open-vocabulary query.

**Output**: Predicted segmentation mask and a similarity score map (or bounding box derived from the highest similarity pixel) for each query.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truth):
    miou_scores = []
    acc_scores = []
    for query, (pred_mask, sim_map) in predictions.items():
        gt_mask, gt_bbox = ground_truth[query]
        iou = intersection_over_union(pred_mask, gt_mask)
        miou_scores.append(iou)
        top_pixel = argmax(sim_map)
        acc_scores.append(1 if inside_bbox(top_pixel, gt_bbox) else 0)
    return {
        'mIoU': sum(miou_scores) / len(miou_scores) * 100,
        'accuracy': sum(acc_scores) / len(acc_scores) * 100
    }
```

## Common pitfalls

- Models often suffer from a 'bag-of-words' effect, incorrectly segmenting the entire object when queried for a specific part.
- Localization accuracy heavily depends on the similarity score map resolution and the exact definition of the bounding box overlap.
- Hierarchical queries require compositional understanding; simple concatenation of object and part labels often fails without explicit hierarchical modeling.

## Evidence (verbatim from paper)

> We report results on the task of segmentation using Mean Intersection Over Union (mIoU (Mean Intersection Over Union)) computed between the predicted mask and the annotated masks, as well as performance on the task of localization, quantified as accuracy. For localization, we follow LERF*[[15]]* and consider a true prediction if the pixel with the highest similarity score is inside the bounding box of the object.

## Citation

```bibtex
@misc{weijler2025openhype,
  title={OpenHype: Hyperbolic Embeddings for Hierarchical Open-Vocabulary Radiance Fields},
  author={Weijler et al. (2025)},
  year={2025},
  note={arXiv:2510.21441}
}
```

- arXiv: 2510.21441