Octscenes Eval

---
name: octscenes-eval
description: Evaluates object-centric learning models on real-world tabletop scenes to measure their ability to segment foreground objects and background, as well as reconstruct scenes from single-image, video, or multi-view inputs. Use when the user wants to benchmark on OCTScenes-A, OCTScenes-B, or asks about evaluating this task. Reports ARI-O.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.09682
  bibtex_key: huang2023octscenes
  confidence: high
---

# octscenes-eval

> OCTScenes: A Versatile Real-World Dataset of Tabletop Scenes for Object-Centric Learning — Huang et al. (2023) (arXiv:2306.09682, 2023)

## What this evaluates

Evaluates object-centric learning models on real-world tabletop scenes to measure their ability to segment foreground objects and background, as well as reconstruct scenes from single-image, video, or multi-view inputs.

## Datasets

- **OCTScenes-A** — total ?; splits: train (90000), val (3000), test (3000)
- **OCTScenes-B** — total ?; splits: train (144000), val (3000), test (3000)

## Metrics

- `ARI-O` **(primary)** — range: [0, 1]
  - Adjusted Rand Index computed only on object pixels. Measures the similarity between predicted and ground-truth object clusters, adjusted for chance. Higher values indicate better segmentation.
- `AMI-O` — range: [0, 1]
  - Adjusted Mutual Information computed only on object pixels. Measures mutual information between predicted and ground-truth clusters, adjusted for chance. Higher values indicate better segmentation.
- `mIoU` — range: [0, 1]
  - mean Intersection over Union. Computes the average IoU across all object classes/predicted clusters. Higher values indicate better overlap with ground truth.
- `MSE` — range: [0, inf)
  - Mean Squared Error between reconstructed and original images at the pixel level. Lower values indicate better reconstruction, though it favors blurry results.
- `LPIPS` — range: [0, 1]
  - Learned Perceptual Image Patch Similarity. Measures perceptual difference in feature space between reconstructed and original images. Lower values indicate better reconstruction aligned with human perception.

## Input / output format

**Input**: RGB-D images resized to 128x128. Scenes are split into sub-scenes with 10-frame intervals. Inputs vary by method type: single-frame images, video sequences, or multi-view frames.

**Output**: Predicted object segmentation masks (cluster assignments) and reconstructed RGB images.

## Scoring recipe

```python
def evaluate(pred_masks, gt_masks, pred_img, gt_img):
    ari_o = adjusted_rand_index(pred_masks, gt_masks, ignore_bg=True)
    ami_o = adjusted_mutual_info(pred_masks, gt_masks, ignore_bg=True)
    miou = mean_intersection_over_union(pred_masks, gt_masks)
    mse = np.mean((pred_img - gt_img) ** 2)
    lpips = perceptual_similarity(pred_img, gt_img)
    return {'ARI-O': ari_o, 'AMI-O': ami_o, 'mIoU': miou, 'MSE': mse, 'LPIPS': lpips}
```

## Common pitfalls

- Models often fail to segment the background as a single cluster, instead splitting it into multiple parts to capture complex background variations.
- Reconstruction metrics can be inconsistent: pixel-level MSE may be low while feature-level LPIPS is high (or vice versa), depending on whether the model uses a mixture-based or transformer-based decoder.
- Methods relying on random slot initialization (e.g., SLATE, STEVE) exhibit high performance variance across different random seeds and hyperparameters.

## Evidence (verbatim from paper)

> We assess segmentation quality with Adjusted Rand Index (ARI) [21], Adjusted Mutual Information (AMI) [36], and mean Intersection over Union (mIoU). ARI and AMI, which measure the congruence between two data clusters, are robust indicators of superior segmentation performance as their values increase. mIoU, a standard metric for evaluating object segmentation, provides a quantifiable measure of the overlap between the predicted and ground truth segmentation. We further refine our analysis by introducing the terms AMI-A and ARI-A, signifying calculations that consider both the objects and the background, and AMI-O and ARI-O, which focus solely on the objects. We rely on Minimize Squared Error (MSE) and Learned Perceptual Image Patch Similarity (LPIPS) [37] to evaluate the quality of reconstruction, both of which indicate better reconstruction performance at lower values.

## Citation

```bibtex
@misc{huang2023octscenes,
  title={OCTScenes: A Versatile Real-World Dataset of Tabletop Scenes for Object-Centric Learning},
  author={Huang et al. (2023)},
  year={2023},
  note={arXiv:2306.09682}
}
```

- arXiv: 2306.09682