Irefvla Eval

---
name: irefvla-eval
description: Evaluates a model's ability to ground referential language in 3D scenes when references are imperfect or ambiguous. It probes whether the model can correctly identify existing objects, detect non-existent references, and generate plausible alternative objects based on spatial and semantic reasoning. Use when the user wants to benchmark on IRef-VLA, or asks about evaluating this task. Reports score_sim.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.17406
  bibtex_key: zhang2025irefvla
  confidence: high
---

# irefvla-eval

> IRef-VLA: A Benchmark for Interactive Referential Grounding with Imperfect Language in 3D Scenes — Zhang et al. (2025) (arXiv:2503.17406, 2025)

## What this evaluates

Evaluates a model's ability to ground referential language in 3D scenes when references are imperfect or ambiguous. It probes whether the model can correctly identify existing objects, detect non-existent references, and generate plausible alternative objects based on spatial and semantic reasoning.

## Datasets

- **IRef-VLA** — total 4700000; splits: train (-1), val (-1), test (-1); repo https://github.com/HaochenZ11/IRef-VLA

## Metrics

- `score_sim` **(primary)** — range: [0, 1]
  - Calculates the normalized weighted match between aspects of the original referential statement S and the suggested alternative S'. Aspects (classes, attributes, spatial relations) are weighted by importance (λ_i). Formula: score_sim = Σ(λ_i * 1{a_i ∈ A(S')}) / Σ(λ_i).
- `TP/FP/TN/FN` — range: other
  - Binary classification counts used to assess how well the model identifies object existence based on the referential statement.

## Input / output format

**Input**: A referential statement (S) describing a target object within a 3D scene.

**Output**: Either the identified object if it exists, or an explicit indication that the object was not found, followed by a suggested alternative object.

## Scoring recipe

```python
def compute_grounding_metrics(predictions, golds):
    tp = sum(1 for p, g in zip(predictions, golds) if p == g == 'exists')
    fp = sum(1 for p, g in zip(predictions, golds) if p == 'exists' and g != 'exists')
    tn = sum(1 for p, g in zip(predictions, golds) if p != 'exists' and g != 'exists')
    fn = sum(1 for p, g in zip(predictions, golds) if p != 'exists' and g == 'exists')
    return tp, fp, tn, fn

def compute_score_sim(statement_aspects, alt_aspects, weights):
    numerator = sum(w * (1 if aspect in alt_aspects else 0) for aspect, w in zip(statement_aspects, weights))
    denominator = sum(weights)
    return numerator / denominator if denominator > 0 else 0.0
```

## Common pitfalls

- Assuming the referred object always exists in the scene, unlike standard referential grounding benchmarks that only require retrieval.
- Relying solely on the heuristic score_sim metric, which may not fully capture human intent; the authors note human-labeled scores are preferable but scale-limited.
- Confusing this task with embodied navigation benchmarks (e.g., ObjectNav) that evaluate agent planning and movement rather than pure language-to-object grounding.

## Evidence (verbatim from paper)

> For the grounding and search subtask, we use binary classification metrics—true positive (TP), false positive (FP), true negative (TN), and false negative (FN)—to assess how well the model can identify object existence based on a referential statement.

To quantitatively assess the quality of retrieved object alternatives, we use a heuristic scoring system. We calculate a similarity score $score_{sim}$ based on how well each suggestion matches aspects of the referential statement, such as object classes, attributes, and spatial relations.

## Citation

```bibtex
@misc{zhang2025irefvla,
  title={IRef-VLA: A Benchmark for Interactive Referential Grounding with Imperfect Language in 3D Scenes},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2503.17406}
}
```

- arXiv: 2503.17406