Construction Site 10k Eval

---
name: construction-site-10k-eval
description: Evaluates vision-language models on construction site safety inspection tasks, including image captioning, safety rule violation detection, reasoning, and visual grounding of specific objects. Use when the user wants to benchmark on ConstructionSite 10k, or asks about evaluating this task. Reports IoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.11011
  bibtex_key: chen2025construction
  confidence: high
---

# construction-site-10k-eval

> Are Large Pre-trained Vision Language Models Effective Construction Safety Inspectors? — Chen et al. (2025) (arXiv:2508.11011, 2025)

## What this evaluates

Evaluates vision-language models on construction site safety inspection tasks, including image captioning, safety rule violation detection, reasoning, and visual grounding of specific objects.

## Datasets

- **ConstructionSite 10k** — total 10013; splits: test (-1)

## Metrics

- `IoU` **(primary)** — range: [0, 1]
  - Intersection over Union: Area of overlap between predicted and ground-truth bounding boxes divided by their union area.
- `SPICE` — range: [0, 1]
  - Scene Graph Informed Caption Evaluation: Parses captions into scene graphs (objects, attributes, relationships) and computes F1 score of matching tuples.
- `Precision` — range: [0, 1]
  - True Positives / (True Positives + False Positives) for multi-label rule selection.
- `Recall` — range: [0, 1]
  - True Positives / (True Positives + False Negatives) for multi-label rule selection.
- `LLM-Judge Score` — range: [0, 6]
  - Sum of three criteria (Relevance, Equivalence, Specificity) scored 0-2 by a Llama 3 8B judge, max total 6.

## Input / output format

**Input**: Image + system prompt ('You are a construction site inspector...') + user prompt (task-specific, e.g., safety rule question with 4 rules to choose from, requesting rule ID, explanation, and bounding box)

**Output**: For captioning: a single-paragraph text description. For VQA: a selected violated rule ID, a textual explanation, and a bounding box coordinate.

## Scoring recipe

```python
def score(preds, golds):
    # IoU for grounding
    iou = overlap(preds.box, golds.box) / union(preds.box, golds.box)
    # Precision/Recall for rule selection
    tp = sum(p == g == 1 for p, g in zip(preds.rules, golds.rules))
    fp = sum(p == 1 and g == 0 for p, g in zip(preds.rules, golds.rules))
    fn = sum(p == 0 and g == 1 for p, g in zip(preds.rules, golds.rules))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    # LLM Judge for reasoning (only if rule correctly selected)
    judge = llm_judge.evaluate(preds.reasoning, golds.reasoning, criteria=['relevance', 'equivalence', 'specificity'])
    return {'IoU': iou, 'Precision': prec, 'Recall': rec, 'LLM_Judge': judge}
```

## Common pitfalls

- IoU is only computed for correctly identified violations in the VQA task, meaning false positives in rule selection are not penalized in the grounding score.
- Few-shot in-context learning differs between GPT models (5 images+captions) and LLaVA (5 captions-only), making cross-model comparison sensitive to modality alignment.
- LLM judge scores depend heavily on the 3-shot examples and beam search settings (k=5, seed=20) used during evaluation.

## Evidence (verbatim from paper)

> The evaluation metrics used for the image captioning task are: SPICE, CIDEr-D, METEOR, BERTScore, and CLIPScore... For multi-label classification (i.e. choosing the violated rules), the evaluation metrics are precision and recall for each rule. For the visual grounding, we use IoU.

## Citation

```bibtex
@misc{chen2025construction,
  title={Are Large Pre-trained Vision Language Models Effective Construction Safety Inspectors?},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2508.11011}
}
```

- arXiv: 2508.11011