Openfungraph Eval

---
name: openfungraph-eval
description: Evaluates a model's ability to predict open-vocabulary functional 3D scene graphs from posed RGB-D images. It specifically probes the detection of objects and interactive elements, as well as the inference of their functional relationships (e.g., switch controls light) in real-world indoor spaces. Use when the user wants to benchmark on SceneFun3D, FunGraph3D, or asks about evaluating this task. Reports Recall@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.19199
  bibtex_key: zhang2025openfungraph
  confidence: high
---

# openfungraph-eval

> Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces — Zhang et al. (2025) (arXiv:2503.19199, 2025)

## What this evaluates

Evaluates a model's ability to predict open-vocabulary functional 3D scene graphs from posed RGB-D images. It specifically probes the detection of objects and interactive elements, as well as the inference of their functional relationships (e.g., switch controls light) in real-world indoor spaces.

## Datasets

- **SceneFun3D** — total ?; splits: val (8), test (12)
- **FunGraph3D** — total ?; splits: test (-1)

## Metrics

- `Recall@K` **(primary)** — range: [0, 1]
  - Node recall is the fraction of ground-truth nodes successfully retrieved within top-K, where success requires non-zero 3D IoU and CLIP embedding cosine similarity ranking. Triplet recall requires all three components (object, element, relationship) to be retrieved in top-K. Node association and edge prediction are derived from triplet recall counts.

## Input / output format

**Input**: Posed RGB-D images of indoor environments.

**Output**: Functional 3D scene graph containing nodes (objects and interactive elements) with 3D coordinates and open-vocabulary labels, and edges representing functional relationships between them.

## Scoring recipe

```python
def compute_recall(pred_items, gt_items, K, sim_fn):
    retrieved = 0
    for gt in gt_items:
        if any(sim_fn(pred, gt) > 0 and rank(pred, gt) <= K for pred in pred_items):
            retrieved += 1
    return retrieved / len(gt_items)

# Node recall uses 3D IoU > 0 and CLIP cosine similarity
node_recall = compute_recall(pred_nodes, gt_nodes, K=3 or 10, sim_fn=clip_cosine)

# Triplet recall requires all 3 components retrieved in top-K
triplet_recall = compute_recall(pred_triplets, gt_triplets, K=5 or 10, sim_fn=bert_cosine)
node_assoc = triplet_recall / len(gt_triplets)
edge_pred = triplet_recall / node_assoc
```

## Common pitfalls

- Baselines using ground-truth instance segmentation for graph inference are unfair; must use predicted nodes for fair comparison.
- Triplet evaluation is strict: all three components (object, interactive element, relationship) must be individually retrieved within top-K.
- Relationship retrieval uses BERT embeddings for open-vocabulary predicates, not CLIP.

## Evidence (verbatim from paper)

> To evaluate open-vocabulary functional 3D scene graphs effectively, a new quantitative metric is essential. Existing approaches, such as ConceptGraph*[[27]], rely on subjective human assessments, while Open3DSG*[[41]] approaches evaluation as a label retrieval task, assuming all ground-truth nodes are known, an assumption that diverges from our real-world setting. To address this, we extend the Open3DSG Recall@K metric*[[41]] with a node detection component, using spatial overlap between predicted and ground-truth nodes, inspired by evaluation techniques on 2D scene graph generation*[[106], [89], [50], [87], [88]].

## Citation

```bibtex
@misc{zhang2025openfungraph,
  title={Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2503.19199}
}
```

- arXiv: 2503.19199