Physically Disentangled Eval

---
name: physically-disentangled-eval
description: Evaluates the utility of physically disentangled scene representations (geometry, albedo, lighting, camera) for downstream vision tasks. Probes robustness to out-of-distribution lighting and viewpoints, and measures how well learned features transfer to clustering, linear classification, and segmentation benchmarks. Use when the user wants to benchmark on CelebA, Buffy, BBT, RAF-DB, CelebA Mask, ShapeNet Cars, or asks about evaluating this task. Reports clustering_accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2204.05281
  bibtex_key: klinghoffer2022physicallydisentangled
  confidence: high
---

# physically-disentangled-eval

> Physically Disentangled Representations — Klinghoffer et al. (2022) (arXiv:2204.05281, 2022)

## What this evaluates

Evaluates the utility of physically disentangled scene representations (geometry, albedo, lighting, camera) for downstream vision tasks. Probes robustness to out-of-distribution lighting and viewpoints, and measures how well learned features transfer to clustering, linear classification, and segmentation benchmarks.

## Datasets

- **CelebA** — total 6000; splits: test (6000)
- **Buffy** — total 568; splits: test (568)
- **BBT** — total 644; splits: test (644)
- **RAF-DB** — total 3068; splits: test (3068)
- **CelebA Mask** — total 27012; splits: train (24127), test (2885)
- **ShapeNet Cars** — total 1000; splits: test (1000)

## Metrics

- `clustering_accuracy` **(primary)** — range: [0, 1]
  - Weighted cluster purity: Acc = (1/N) * sum_{c=1}^{|C|} n_c * p_c, where N is total samples, n_c is samples in cluster c, and p_c is the fraction of the most frequent class in cluster c relative to n_c.
- `F1-score` — range: [0, 1]
  - Weighted average F1-score computed over clusters or classes.
- `mIoU` — range: [0, 1]
  - Mean intersection over union across all segmentation classes.
- `pixel_accuracy` — range: [0, 1]
  - Fraction of correctly classified pixels across the entire dataset.

## Input / output format

**Input**: 64x64 RGB images (or video frames). For downstream tasks, the model extracts 1x256 geometry and albedo feature embeddings per image.

**Output**: Cluster assignments per sample, predicted class labels, or pixel-wise segmentation masks.

## Scoring recipe

```python
def clustering_accuracy(labels, clusters):
    N = len(labels)
    acc = 0.0
    for c in unique_clusters:
        n_c = count(labels in c)
        p_c = max(count(label in c for label in unique_labels)) / n_c
        acc += n_c * p_c
    return acc / N

def linear_accuracy(predictions, gold):
    return sum(p == g for p, g in zip(predictions, gold)) / len(gold)

def miou(pred_masks, gt_masks, num_classes):
    ious = []
    for cls in range(num_classes):
        intersection = (pred_masks == cls) & (gt_masks == cls)
        union = (pred_masks == cls) | (gt_masks == cls)
        ious.append(intersection.sum() / union.sum())
    return sum(ious) / num_classes
```

## Common pitfalls

- For video datasets (Buffy/BBT), features must be averaged per video track before clustering, not evaluated frame-by-frame.
- Linear classification experiments require varying the labeled training set size (100, 500, or 1000 samples) and testing both frozen and fine-tuned pre-trained encoders.
- Only geometry and albedo features are used for downstream tasks unless explicitly stated otherwise; other physical parameters are ignored.

## Evidence (verbatim from paper)

> To measure clustering performance, we report clustering accuracy (also known as weighted cluster purity) and F1-score. F1-score is computed using a weighted average. Clustering accuracy is computed by assigning the most common ground truth label for a cluster to all points in the cluster, as defined below. ... For linear classification, we report accuracy using a threshold of 50%. For segmentation, we report mean intersection over union (mIoU) and pixel accuracy.

## Citation

```bibtex
@misc{klinghoffer2022physicallydisentangled,
  title={Physically Disentangled Representations},
  author={Klinghoffer et al. (2022)},
  year={2022},
  note={arXiv:2204.05281}
}
```

- arXiv: 2204.05281