Carlaocc Eval

---
name: carlaocc-eval
description: Evaluates 3D occupancy prediction models on their ability to reconstruct semantic and instance-level voxel grids from camera inputs. It probes geometric completeness, occlusion reasoning, and instance discrimination in complex autonomous driving scenes. Use when the user wants to benchmark on CarlaOcc, or asks about evaluating this task. Reports mIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.27238
  bibtex_key: feng2026carlaocc
  confidence: high
---

# carlaocc-eval

> An Instance-Centric Panoptic Occupancy Prediction Benchmark for Autonomous Driving — Feng et al. (2026) (arXiv:2603.27238, 2026)

## What this evaluates

Evaluates 3D occupancy prediction models on their ability to reconstruct semantic and instance-level voxel grids from camera inputs. It probes geometric completeness, occlusion reasoning, and instance discrimination in complex autonomous driving scenes.

## Datasets

- **CarlaOcc** — total ?; splits: test (-1)

## Metrics

- `mIoU` **(primary)** — range: percent
  - Mean Intersection-over-Union across all semantic classes. Computed as the average of IoU scores for each class, where IoU is the intersection over union of predicted and ground truth voxel masks.
- `PQ` — range: percent
  - Panoptic Quality for instance-level occupancy. It is the product of Segmentation Quality (SQ, mean IoU of matched instances) and Recognition Quality (RQ, precision of instance matching).

## Input / output format

**Input**: Multi-camera RGB images, depth, semantic maps, LiDAR, and semantic LiDAR point clouds. All inputs are processed at a unified voxel size of 0.2 m.

**Output**: Voxel-wise semantic occupancy grid and instance-level panoptic labels (semantic class + instance ID) at 0.2 m resolution.

## Scoring recipe

```python
def compute_miou(pred, gt, num_classes):
    ious = []
    for c in range(num_classes):
        pred_c = (pred == c)
        gt_c = (gt == c)
        inter = np.sum(pred_c & gt_c)
        union = np.sum(pred_c | gt_c)
        ious.append(inter / union if union > 0 else 1.0)
    return np.mean(ious) * 100

def compute_pq(pred_instances, gt_instances, iou_thresh=0.5):
    tp, fp, fn = 0, 0, 0
    sq_sum = 0
    for p_inst in pred_instances:
        best_iou = 0
        best_gt = None
        for g_inst in gt_instances:
            iou = compute_iou(p_inst, g_inst)
            if iou > best_iou:
                best_iou, best_gt = iou, g_inst
        if best_iou >= iou_thresh:
            tp += 1
            sq_sum += best_iou
            gt_instances.remove(best_gt)
        else:
            fp += 1
    fn = len(gt_instances)
    sq = sq_sum / tp if tp > 0 else 0
    rq = tp / (tp + fp + fn) if (tp + fp + fn) > 0 else 0
    return sq * rq * 100
```

## Common pitfalls

- Models are evaluated at a fixed 0.2 m voxel resolution, which differs from the dataset's native 0.05 m resolution.
- Panoptic metrics require instance-level matching, which is significantly harder than semantic segmentation and often yields low scores on complex scenes.
- Occluded regions must be inferred rather than just predicted for visible voxels, as the ground truth includes complete 3D layout.

## Evidence (verbatim from paper)

> During evaluation, we report Intersection-over-Union (IoU) and mean Intersection-over-Union (mIoU) in semantic occupancy prediction evaluation, while panoptic occupancy prediction is assessed using Panoptic Quality (PQ), Segmentation Quality (SQ), Recognition Quality (RQ), and mIoU to jointly measure the semantic segmentation quality and instance-level completeness.

## Citation

```bibtex
@misc{feng2026carlaocc,
  title={An Instance-Centric Panoptic Occupancy Prediction Benchmark for Autonomous Driving},
  author={Feng et al. (2026)},
  year={2026},
  note={arXiv:2603.27238}
}
```

- arXiv: 2603.27238