Pose Aware Ssl Eval

---
name: pose-aware-ssl-eval
description: Evaluates the ability of self-supervised visual representations to capture geometric pose information and semantic content. It probes absolute and relative pose estimation accuracy, as well as semantic classification performance, across in-domain, out-of-domain, and real-world settings. Use when the user wants to benchmark on Carvana, Synthetic dataset [8], or asks about evaluating this task. Reports relative pose estimation accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.14973
  bibtex_key: wang2024poseaware
  confidence: high
---

# pose-aware-ssl-eval

> Pose-Aware Self-Supervised Learning with Viewpoint Trajectory Regularization — Wang et al. (2024) (arXiv:2403.14973, 2024)

## What this evaluates

Evaluates the ability of self-supervised visual representations to capture geometric pose information and semantic content. It probes absolute and relative pose estimation accuracy, as well as semantic classification performance, across in-domain, out-of-domain, and real-world settings.

## Datasets

- **Carvana** — total 5088; splits: test (5088)
- **Synthetic dataset [8]** — total ?; splits: train (-1)

## Metrics

- `semantic classification accuracy` — range: percent
  - Accuracy of a linear probe trained on frozen 512-dim feature representations.
- `absolute pose estimation accuracy` — range: percent
  - Accuracy of a weighted k-nearest neighbor classifier on frozen feature representations.
- `relative pose estimation accuracy` **(primary)** — range: percent
  - Accuracy of a two-layer MLP probe on concatenated 1024-dim features from two views, predicting discretized relative pose deltas.

## Input / output format

**Input**: RGB images of objects from varying viewpoints. Single images for absolute pose and semantic classification. Pairs of images for relative pose estimation.

**Output**: Discretized semantic class labels. Discretized absolute pose coordinates/angles. Discretized relative pose deltas.

## Scoring recipe

```python
def evaluate(features, labels, task):
    if task == 'semantic':
        logits = linear_probe(features)
        return accuracy(logits, labels)
    elif task == 'absolute_pose':
        preds = weighted_knn_predict(features, gallery_features, gallery_labels, k=5)
        return accuracy(preds, labels)
    elif task == 'relative_pose':
        z1, z2 = encoder(img1), encoder(img2)
        concat_feat = concatenate(z1, z2)
        preds = mlp_probe(concat_feat)
        return accuracy(preds, relative_labels)
```

## Common pitfalls

- Using the last feature layer instead of mid-layer representations (e.g., 'conv3') significantly underestimates pose estimation performance, missing up to 20% gains.
- Evaluating only on in-domain synthetic data ignores the critical out-of-domain generalization capability that the method aims to improve.

## Evidence (verbatim from paper)

> We evaluate with a linear classification on top of the frozen representation from the feature layer with dimension 512. We employ a weighted k-nearest neighbor classifier as used in [56] on the representations from the feature layer. These representations are concatenated (resulting in a 1024-dim feature), and a simple probe of a two-layer perceptron with a hidden dimension of 1024 is used to predict the relative pose. Relative pose estimation is more computationally efficient but is generally harder as it relies on only two views for inference.

## Citation

```bibtex
@misc{wang2024poseaware,
  title={Pose-Aware Self-Supervised Learning with Viewpoint Trajectory Regularization},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2403.14973}
}
```

- arXiv: 2403.14973