Booster Eval

---
name: booster-eval
description: Evaluates stereo and monocular depth/disparity estimation models on images containing specular and transparent surfaces, which violate standard non-Lambertian assumptions and cause significant performance degradation in existing networks. Use when the user wants to benchmark on Booster, or asks about evaluating this task. Reports bad-2.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2301.08245
  bibtex_key: zama_ramirez2023booster
  confidence: high
---

# booster-eval

> Booster: a Benchmark for Depth from Images of Specular and Transparent Surfaces — Zama Ramirez et al. (2023) (arXiv:2301.08245, 2023)

## What this evaluates

Evaluates stereo and monocular depth/disparity estimation models on images containing specular and transparent surfaces, which violate standard non-Lambertian assumptions and cause significant performance degradation in existing networks.

## Datasets

- **Booster** — total ?; splits: train (-1), test (-1)

## Metrics

- `bad-2` **(primary)** — range: percent
  - Percentage of pixels where the absolute disparity error exceeds 2 pixels.
- `bad-4` — range: percent
  - Percentage of pixels where the absolute disparity error exceeds 4 pixels.
- `bad-6` — range: percent
  - Percentage of pixels where the absolute disparity error exceeds 6 pixels.
- `bad-8` — range: percent
  - Percentage of pixels where the absolute disparity error exceeds 8 pixels.
- `MAE` — range: other
  - Mean Absolute Error between predicted and ground-truth disparity values.
- `RMSE` — range: other
  - Root Mean Square Error between predicted and ground-truth disparity values.

## Input / output format

**Input**: Stereo image pairs (left and right views) for balanced/unbalanced benchmarks; single image for monocular benchmark.

**Output**: Dense disparity maps (or depth maps) at the input resolution.

## Scoring recipe

```python
def compute_metrics(pred_disp, gt_disp, valid_mask):
    err = np.abs(pred_disp - gt_disp)
    bad2 = 100 * np.mean(err[valid_mask] > 2)
    bad4 = 100 * np.mean(err[valid_mask] > 4)
    bad6 = 100 * np.mean(err[valid_mask] > 6)
    bad8 = 100 * np.mean(err[valid_mask] > 8)
    mae = np.mean(err[valid_mask])
    rmse = np.sqrt(np.mean(err[valid_mask]**2))
    return {'bad-2': bad2, 'bad-4': bad4, 'bad-6': bad6, 'bad-8': bad8, 'MAE': mae, 'RMSE': rmse}
```

## Common pitfalls

- Predictions are often upsampled via nearest-neighbor interpolation after processing at half/quarter resolution, which can inflate errors compared to downsampling ground truth.
- The benchmark evaluates on 'All' and 'Cons' (consistent) pixels, but occlusions are noted as not the main difficulty; material segmentation masks (Classes 0-3) are used to isolate specular/transparent regions.
- Disparity maps must be multiplied by the upsampling factor when comparing quarter-resolution predictions to full-resolution ground truth.

## Evidence (verbatim from paper)

> e.g., 33.07 vs 8.13 bad-2 for CREStereo on all-pixels in the Middlebury benchmark. This evidence confirms that high resolution is not the only challenge in our benchmark, but networks also struggle due to the presence of transparent and specular surfaces in our dataset.

## Citation

```bibtex
@misc{zama_ramirez2023booster,
  title={Booster: a Benchmark for Depth from Images of Specular and Transparent Surfaces},
  author={Zama Ramirez et al. (2023)},
  year={2023},
  note={arXiv:2301.08245}
}
```

- arXiv: 2301.08245