Mobile Construction Eval

---
name: mobile-construction-eval
description: Tests a robot's ability to simultaneously navigate and construct a target structure in 1D/2D/3D grid worlds under partial observability and environmental uncertainty. It evaluates the bi-directional coupling between localization and manipulation planning. Use when the user wants to benchmark on Mobile Construction Benchmark, or asks about evaluating this task. Reports IoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2103.16732
  bibtex_key: [REDACTED]
  confidence: high
---

# mobile-construction-eval

> Simultaneous Navigation and Construction Benchmarking Environments — Han et al. (2021) (arXiv:2103.16732, 2021)

## What this evaluates

Tests a robot's ability to simultaneously navigate and construct a target structure in 1D/2D/3D grid worlds under partial observability and environmental uncertainty. It evaluates the bi-directional coupling between localization and manipulation planning.

## Datasets

- **Mobile Construction Benchmark** — total ?; splits: test (-1)

## Metrics

- `IoU` **(primary)** — range: [0, 1]
  - Intersection over Union between the terminal grid state G^T and the ground-truth design D. Computed as the sum over all grid cells l of min(G^T(l), D(l)) / max(G^T(l), D(l)).

## Input / output format

**Input**: Partially observable grid world state with a limited sensing region; target design D (static) or dynamic curve/triangle sequence (dynamic).

**Output**: Action sequence consisting of 'move' or 'drop brick' commands to construct the target design.

## Scoring recipe

```python
def compute_iou(terminal_grid, target_design):
    intersection = sum(min(terminal_grid[l], target_design[l]) for l in grid_cells)
    union = sum(max(terminal_grid[l], target_design[l]) for l in grid_cells)
    return intersection / union if union > 0 else 0.0
```

## Common pitfalls

- IoU is evaluated only at the terminal state, ignoring intermediate construction progress or partial matches.
- Dynamic tasks use fixed groups of designs for testing, which may not reflect generalization to unseen dynamic patterns.
- 3D environments introduce obstacles and agent/obstacle volume, drastically increasing difficulty and often causing agents to learn to only move rather than build.

## Evidence (verbatim from paper)

> Evaluation metric: we use the IoU score as our evaluation criteria which is measured between the terminal grid state $G^{T}$ and the ground-truth design $D$. Here, we use IoU as our evaluation criteria because it is more straightforward for us to evaluate the quality of the built structure to the ground-truth design $D$. The IoU is defined as $IoU=\frac{G^{T}\cap D}{G^{T}\cup D}=\sum_{l\in\mathcal{L}}\frac{\min(G^{T}(l),D(l))}{\max(G^{T}(l),D(l))}.$

## Citation

```bibtex
@misc{han2021simultaneous,
  title={Simultaneous Navigation and Construction Benchmarking Environments},
  author={Han et al. (2021)},
  year={2021},
  note={arXiv:2103.16732}
}
```

- arXiv: 2103.16732