Built Bench Eval

---
name: built-bench-eval
description: Evaluates the ability of pre-trained text embedding models to capture domain-specific semantic alignment for built asset information. The benchmark probes clustering, information retrieval, and document reranking capabilities using technical terminology from architectural, structural, mechanical, and electrical systems. Use when the user wants to benchmark on BuiltBench, or asks about evaluating this task. Reports task-specific metrics.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.12056
  bibtex_key: [REDACTED]
  confidence: medium
---

# built-bench-eval

> Benchmarking pre-trained text embedding models in aligning built asset information — Shahinmoghadam et al. (2024) (arXiv:2411.12056, 2024)

## What this evaluates

Evaluates the ability of pre-trained text embedding models to capture domain-specific semantic alignment for built asset information. The benchmark probes clustering, information retrieval, and document reranking capabilities using technical terminology from architectural, structural, mechanical, and electrical systems.

## Datasets

- **BuiltBench** — total ?; splits: test (-1); repo https://github.com/mehrzadshm/built-bench-paper

## Metrics

- `task-specific metrics` **(primary)** — range: percent
  - The paper reports scores based on task-specific metrics detailed in the original task descriptions. Clustering uses label-based similarity, retrieval uses matching accuracy/recall, and reranking uses ranking quality metrics. Scores are aggregated as averages across subtasks.

## Input / output format

**Input**: Text strings representing built asset product names or descriptions derived from IFC and buildingSMART Data Dictionary. Queries are either short product names or longer descriptions; documents are product descriptions.

**Output**: Embedding vectors for each text string. For clustering: cluster assignments. For retrieval/reranking: ranked list of document indices or similarity scores.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    # Compute task-specific metrics per subtask as defined in original descriptions
    clustering_score = compute_clustering_metric(predictions, gold) * 100
    retrieval_score = compute_retrieval_metric(predictions, gold) * 100
    reranking_score = compute_reranking_metric(predictions, gold) * 100
    # Average across all six subtasks (s2s/p2p for each task)
    return (clustering_score + retrieval_score + reranking_score) / 3
```

## Common pitfalls

- Assuming performance on general-purpose benchmarks (e.g., MTEB) transfers directly to domain-specific built asset data.
- Ignoring the significant performance drop on short-to-short (S2S) clustering tasks due to lack of contextual clues in short product names.
- Failing to apply model-specific prompts for instruction-tuned models, leading to inconsistent evaluation across base and instruction variants.

## Evidence (verbatim from paper)

> We applied a 1:3 positive-to-negative sampling ratio to create a balanced yet challenging evaluation set, ensuring that models must distinguish effectively between relevant and irrelevant documents. Table 2: Average scores of benchmarked models per task, based on the task-specific metrics mentioned in the task descriptions.

## Citation

```bibtex
@misc{shahinmoghadam2024builtbench,
  title={Benchmarking pre-trained text embedding models in aligning built asset information},
  author={Shahinmoghadam et al. (2024)},
  year={2024},
  note={arXiv:2411.12056}
}
```

- arXiv: 2411.12056