Ifc Bench V2 Eval

---
name: ifc-bench-v2-eval
description: Tests an LLM's ability to extract, compute, and reason over heterogeneous Building Information Modeling (BIM) data (IFC files) using adaptive code execution or static baselines. It probes robustness to data heterogeneity, documentation retrieval, and tool augmentation. Use when the user wants to benchmark on ifc-bench v2, or asks about evaluating this task. Reports aggregate accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2605.01698
  bibtex_key: hellin2026bim
  confidence: high
---

# ifc-bench-v2-eval

> BIM Information Extraction Through LLM-based Adaptive Exploration — Hellin et al. (2026) (arXiv:2605.01698, 2026)

## What this evaluates

Tests an LLM's ability to extract, compute, and reason over heterogeneous Building Information Modeling (BIM) data (IFC files) using adaptive code execution or static baselines. It probes robustness to data heterogeneity, documentation retrieval, and tool augmentation.

## Datasets

- **ifc-bench v2** — total 1027; splits: train/dev (513), test (514); repo https://github.com/sylvainHellin/cobbie

## Metrics

- `aggregate accuracy` **(primary)** — range: [0, 1]
  - A non-abstained answer is rated correct only when all four substantive criteria (faithfulness, completeness, transparency, relevance) are simultaneously satisfied. Formula: accuracy = |{i: c1,i ∧ c2,i ∧ c3,i ∧ c4,i}| / N.

## Input / output format

**Input**: Natural-language question, reference IFC model path, and optionally pre-fetched documentation or available tools.

**Output**: A final natural-language answer or an explicit abstention message.

## Scoring recipe

```python
def compute_aggregate_accuracy(predictions, gold, criteria):
    correct = 0
    for i in range(len(predictions)):
        if not predictions[i].abstained:
            if all(criteria[i][c] for c in [1, 2, 3, 4]):
                correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Abstention is distinct from system errors (crashes, timeouts), which are excluded from evaluation.
- Faithfulness thresholds are category-dependent: Category 1 requires grounding in BIM data only, while Category 4 permits stated assumptions alongside data evidence.
- The metric is deliberately conservative; a single criterion failure across the four substantive criteria marks the entire answer incorrect.

## Evidence (verbatim from paper)

> The primary metric reported throughout this study is aggregate accuracy: a non-abstained answer is rated correct only when all four substantive criteria (1–4) are simultaneously satisfied: accuracy = |{i: c1,i ∧ c2,i ∧ c3,i ∧ c4,i}| / N. This metric is deliberately conservative: a single criterion failure marks the answer incorrect.

## Citation

```bibtex
@misc{hellin2026bim,
  title={BIM Information Extraction Through LLM-based Adaptive Exploration},
  author={Hellin et al. (2026)},
  year={2026},
  note={arXiv:2605.01698}
}
```

- arXiv: 2605.01698