Spacesense Bench Eval

---
name: spacesense-bench-eval
description: Evaluates multi-modal spacecraft perception and pose estimation across 2D/3D segmentation, object detection, monocular depth estimation, and orientation estimation. Probes zero-shot generalization to unseen spacecraft configurations and robustness to long-tail class distributions and metallic surface reflections. Use when the user wants to benchmark on SpaceSense-Bench, or asks about evaluating this task. Reports mIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.09320
  bibtex_key: [REDACTED]
  confidence: high
---

# spacesense-bench-eval

> SpaceSense-Bench: A Large-Scale Multi-Modal Benchmark for Spacecraft Perception and Pose Estimation — Aodi Wu et al. (arXiv:2603.09320, 2026)

## What this evaluates

Evaluates multi-modal spacecraft perception and pose estimation across 2D/3D segmentation, object detection, monocular depth estimation, and orientation estimation. Probes zero-shot generalization to unseen spacecraft configurations and robustness to long-tail class distributions and metallic surface reflections.

## Datasets

- **SpaceSense-Bench** — total 136; splits: train (117), test (14); repo https://github.com/wuaodi/SpaceSense-Bench

## Metrics

- `mIoU` **(primary)** — range: [0, 1]
  - Mean Intersection-over-Union averaged across all 7 semantic classes. Computed as the mean of per-class IoU scores.
- `aAcc` — range: [0, 1]
  - Overall pixel accuracy, calculated as the total number of correctly classified pixels divided by the total number of pixels.
- `mAP@0.5` — range: [0, 1]
  - Mean Average Precision at IoU threshold 0.5 across all classes.
- `mAP@0.5:0.95` — range: [0, 1]
  - Mean Average Precision averaged over IoU thresholds from 0.5 to 0.95 in steps of 0.05.
- `fwIoU` — range: [0, 1]
  - Frequency-weighted IoU, where per-class IoU is weighted by the frequency of that class in the ground truth.
- `AbsRel` — range: [0, inf)
  - Absolute relative error between predicted and ground truth depth, averaged over valid pixels.
- `Spearman rank correlation` — range: [-1, 1]
  - Spearman's rho measuring the monotonic relationship between predicted and ground truth depth values.
- `MAAE` — range: [0, 180]
  - Mean Axis Angular Error, measuring the average angular difference between predicted and ground truth orientation axes.

## Input / output format

**Input**: RGB images, depth maps, and/or LiDAR point clouds of spacecraft models rendered from various viewpoints. 2D tasks use 640×640 resolution images.

**Output**: Per-pixel or per-point semantic class labels, bounding boxes for detected objects, continuous depth values, and 6-DoF orientation angles/axes.

## Scoring recipe

```python
def compute_miou(pred_masks, gt_masks, num_classes=7):
    ious = []
    for c in range(num_classes):
        pred_c = (pred_masks == c)
        gt_c = (gt_masks == c)
        intersection = np.logical_and(pred_c, gt_c).sum()
        union = np.logical_or(pred_c, gt_c).sum()
        ious.append(intersection / union if union > 0 else 1.0)
    return np.mean(ious)
# For detection: compute AP@0.5 and AP@0.5:0.95 using standard COCO-style matching.
# For depth: AbsRel = mean(|d_pred - d_gt| / d_gt); Spearman = rank_correlation(d_pred, d_gt).
# For orientation: MAAE = mean(angular_distance(axis_pred, axis_gt)).
```

## Common pitfalls

- Small components (e.g., thrusters, omni-antennas) occupy <0.2% of foreground pixels and suffer from severe long-tail performance drops.
- Zero-shot evaluation requires models to generalize to entirely unseen spacecraft geometries not seen during training.
- Depth estimation requires least-squares affine alignment on the satellite region before computing metrics like AbsRel.

## Evidence (verbatim from paper)

> For 2D semantic segmentation, we train FCN[[11](#bib.bib10 "Fully convolutional networks for semantic segmentation")]* (ResNet-50), DeepLabV3+*[[5](#bib.bib11 "Encoder-decoder with atrous separative convolution for semantic image segmentation")]* (ResNet-50), SegFormer*[[23](#bib.bib12 "SegFormer: simple and efficient design for semantic segmentation with transformers")]* (MiT-B3), and Mask2Former*[[6](#bib.bib13 "Masked-attention mask transformer for universal image segmentation")]* (Swin-Base), all initialized from ImageNet pre-trained weights, and report mean Intersection-over-Union (mIoU) and overall pixel accuracy (aAcc). For object detection, we train YOLO26*[[4](#bib.bib15 "YOLO26: an analysis of nms-free end to end framework for real-time object detection")]* at five scales (n/s/m/l/x) and report mAP@0.5 and mAP@0.5:0.95. For 3D point cloud segmentation, we train PMFNet*[[27](#bib.bib14 "Perception-aware multi-sensor fusion for 3d lidar semantic segmentation")]* (ResNet-34) which fuses RGB and LiDAR, and report mIoU and frequency-weighted IoU (fwIoU). For monocular depth estimation, we evaluate Depth Anything V2*[[24](#bib.bib16 "Depth anything v2")]* (ViT-S/B/L) in a zero-

## Citation

```bibtex
@misc{wu2026spacesensebench,
  title={SpaceSense-Bench: A Large-Scale Multi-Modal Benchmark for Spacecraft Perception and Pose Estimation},
  author={Aodi Wu et al.},
  year={2026},
  note={arXiv:2603.09320}
}
```

- arXiv: 2603.09320