Indoor Lidar Eval

---
name: indoor-lidar-eval
description: Evaluates 3D object detection and BEV perception capabilities on indoor robotic platforms using LiDAR point clouds. It probes a model's ability to classify indoor objects and localize them with 3D bounding boxes, specifically highlighting the sim-to-real transfer gap in controlled indoor environments. Use when the user wants to benchmark on INDOOR-LiDAR, or asks about evaluating this task. Reports Mean IoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.12377
  bibtex_key: [REDACTED]
  confidence: high
---

# indoor-lidar-eval

> INDOOR-LiDAR: Bridging Simulation and Reality for Robot-Centric 360 degree Indoor LiDAR Perception -- A Robot-Centric Hybrid Dataset — Haichuan Li et al. (arXiv:2512.12377, 2025)

## What this evaluates

Evaluates 3D object detection and BEV perception capabilities on indoor robotic platforms using LiDAR point clouds. It probes a model's ability to classify indoor objects and localize them with 3D bounding boxes, specifically highlighting the sim-to-real transfer gap in controlled indoor environments.

## Datasets

- **INDOOR-LiDAR** — total ?; splits: simulated test (-1), real-world test (-1)

## Metrics

- `Precision (P)` — range: [0, 1]
  - True positives divided by total predicted positives. Measures the fraction of predicted detections that are correct classifications.
- `Mean IoU` **(primary)** — range: [0, 1]
  - Average Intersection over Union between matched predicted and ground-truth bounding boxes across all true positives. Higher values indicate better geometric alignment.
- `Acc@IoU0.25` — range: [0, 1]
  - Fraction of predictions with IoU ≥ 0.25 against ground truth. Measures localization tolerance at a low threshold.
- `Acc@IoU0.50` — range: [0, 1]
  - Fraction of predictions with IoU ≥ 0.50 against ground truth. Standard detection threshold.
- `Acc@IoU0.75` — range: [0, 1]
  - Fraction of predictions with IoU ≥ 0.75 against ground truth. Measures strict localization accuracy.
- `L1 distance error` — range: meters
  - Manhattan distance between predicted and ground-truth box centers. Lower values indicate better localization.
- `L2 distance error` — range: meters
  - Euclidean distance between predicted and ground-truth box centers. Lower values indicate better localization.

## Input / output format

**Input**: Dense 3D LiDAR point clouds with intensity maps captured from diverse indoor environments (offices, labs, dining halls, cafes, stairwells).

**Output**: Predicted 3D bounding boxes with class labels and spatial coordinates.

## Scoring recipe

```python
def evaluate(preds, gts):
    tp, fp = 0, 0
    ious, l1s, l2s = [], [], []
    for pred in preds:
        best_iou, best_gt = 0, None
        for gt in gts:
            if pred['class'] == gt['class']:
                iou = calculate_iou(pred['bbox'], gt['bbox'])
                if iou > best_iou:
                    best_iou, best_gt = iou, gt
        if best_iou > 0.5:
            tp += 1
            ious.append(best_iou)
            l1s.append(l1_dist(pred['bbox'], best_gt['bbox']))
            l2s.append(l2_dist(pred['bbox'], best_gt['bbox']))
        else:
            fp += 1
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    mean_iou = sum(ious) / len(ious) if ious else 0
    return precision, mean_iou, l1s, l2s
```

## Common pitfalls

- Models exhibit drastic performance degradation when evaluated on real-world data compared to simulated data, making sim2real transfer a critical evaluation dimension.
- Different architectures dominate different metrics; e.g., GroupFree3D excels at classification precision while PointRCNN leads in geometric localization (Mean IoU/L1/L2).
- Rare or ambiguous object categories (e.g., 'All other') consistently yield 0.00 precision across all evaluated models, requiring careful per-class reporting.

## Evidence (verbatim from paper)

> To provide a multifarious evaluation of model performance, we report metrics across three aspects of the detection task: • Classification Performance: We use Precision (P) to evaluate a model's ability to correctly classify detection. - Bounding Box Quality: We evaluate the geometric accuracy of the predicted bounding boxes using several metrics: Mean IoU across all true positives, Accuracy at different IoU thresholds, and the L1 and L2 distance errors.

## Citation

```bibtex
@misc{li2025indoorlidar,
  title={INDOOR-LiDAR: Bridging Simulation and Reality for Robot-Centric 360 degree Indoor LiDAR Perception -- A Robot-Centric Hybrid Dataset},
  author={Haichuan Li et al.},
  year={2025},
  note={arXiv:2512.12377}
}
```

- arXiv: 2512.12377