Occam Eval

---
name: occam-eval
description: Evaluates whether object-centric representations derived from zero-shot segmentation masks enable robust zero-shot classification under spurious background correlations, and compares them against slot-based OCL methods on unsupervised object discovery. Use when the user wants to benchmark on Movi-C, Movi-E, UrbanCars, ImageNet-D, ImageNet-9, Waterbirds, CounterAnimals, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.07092
  bibtex_key: [REDACTED]
  confidence: high
---

# occam-eval

> Are We Done with Object-Centric Learning? — Rubinstein et al. (2025) (arXiv:2504.07092, 2025)

## What this evaluates

Evaluates whether object-centric representations derived from zero-shot segmentation masks enable robust zero-shot classification under spurious background correlations, and compares them against slot-based OCL methods on unsupervised object discovery.

## Datasets

- **Movi-C** — total 1000; splits: test (1000)
- **Movi-E** — total 1000; splits: test (1000)
- **UrbanCars** — total ?; splits: test (-1)
- **ImageNet-D** — total ?; splits: test (-1)
- **ImageNet-9** — total ?; splits: test (-1)
- **Waterbirds** — total ?; splits: test (-1)
- **CounterAnimals** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy: the proportion of correctly predicted labels out of the total number of samples.
- `worst group accuracy (WGA)` — range: [0, 1]
  - The minimum accuracy computed across predefined spurious correlation groups (e.g., background types). It measures robustness by penalizing models that rely on spurious cues.
- `FG-ARI` — range: [0, 1]
  - Foreground adjusted Rand index: a clustering similarity metric that evaluates how well predicted instance masks match ground truth foreground objects, ignoring background pixels.
- `mBO` — range: [0, 1]
  - Mean best overlap: computes the maximum intersection-over-union (IoU) between predicted and ground truth masks for each object, averaged across objects and accounting for background pixels.

## Input / output format

**Input**: Input images containing foreground objects and spurious background cues (or synthetic scenes for object discovery). For the OCCAM pipeline, the model receives the original image and predicted object masks.

**Output**: Class predictions (zero-shot classification) or mask-based feature representations. For object discovery, binary/instance masks per object.

## Scoring recipe

```python
def compute_accuracy(preds, gold):
    return sum(1 for p, g in zip(preds, gold) if p == g) / len(gold)

def compute_wga(preds, gold, groups):
    group_accs = {}
    for p, g, grp in zip(preds, gold, groups):
        group_accs.setdefault(grp, []).append(p == g)
    return min(sum(accs)/len(accs) for accs in group_accs.values())
```

## Common pitfalls

- Assuming slot-based OCL is required for object-centric representations, whereas zero-shot segmentation masks achieve superior performance with far fewer training samples.
- Confusing standard accuracy with Worst Group Accuracy (WGA); WGA specifically measures robustness by taking the minimum accuracy across spurious background groups, making it sensitive to background correlations.
- Overlooking that foreground mask selection remains a critical bottleneck; even with high-quality segmentation, poor foreground detection degrades downstream classification performance.

## Evidence (verbatim from paper)

> We measure model performance using the standard metric used in the respective benchmark: accuracy and worst group accuracy (WGA). We provide per-benchmark comparisons for reference, including results from other relevant methods, citing them alongside their names in the tables.

## Citation

```bibtex
@misc{rubinstein2025objectcentric,
  title={Are We Done with Object-Centric Learning?},
  author={Rubinstein et al. (2025)},
  year={2025},
  note={arXiv:2504.07092}
}
```

- arXiv: 2504.07092