Cmc Downstream Eval

---
name: cmc-downstream-eval
description: Evaluates the transferability and quality of self-supervised multiview representations by measuring downstream performance on image classification, video action recognition, and semantic segmentation tasks. Use when the user wants to benchmark on ImageNet, UCF-101, HMDB-51, NYU-Depth-V2, STL-10, or asks about evaluating this task. Reports Top-1 classification accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1906.05849
  bibtex_key: tian2019contrastive
  confidence: high
---

# cmc-downstream-eval

> Contrastive Multiview Coding — Tian et al. (2019) (arXiv:1906.05849, 2019)

## What this evaluates

Evaluates the transferability and quality of self-supervised multiview representations by measuring downstream performance on image classification, video action recognition, and semantic segmentation tasks.

## Datasets

- **ImageNet** — total ?; splits: test (-1)
- **UCF-101** — total ?; splits: test (-1)
- **HMDB-51** — total ?; splits: test (-1)
- **NYU-Depth-V2** — total 1449; splits: test (-1)
- **STL-10** — total ?; splits: test (-1)

## Metrics

- `Top-1 classification accuracy (%)` **(primary)** — range: percent
  - Percentage of correctly predicted class labels out of the total number of test samples.
- `Top-5 classification accuracy (%)` — range: percent
  - Percentage of test samples where the true label appears in the top 5 predicted classes.
- `Pixel Accuracy (%)` — range: percent
  - Percentage of correctly classified pixels out of the total number of pixels in the segmentation task.
- `mIoU (%)` — range: percent
  - Mean Intersection over Union across all semantic classes, computed as the average of IoU per class.

## Input / output format

**Input**: RGB images converted to specific channel splits (e.g., L and ab in Lab space, Y and DbDr in YCbCr), video frames, optical flow maps, depth maps, surface normals, or semantic labels. For contrastive pre-training, 128x128 patches are randomly cropped from original images.

**Output**: Class predictions from a linear classifier or segmentation decoder trained on frozen encoder features.

## Scoring recipe

```python
def compute_accuracy(preds, targets):
    correct = (preds == targets).sum()
    return correct / len(targets) * 100

def compute_miou(preds, targets, num_classes):
    ious = []
    for c in range(num_classes):
        intersection = ((preds == c) & (targets == c)).sum()
        union = ((preds == c) | (targets == c)).sum()
        ious.append(intersection / union if union > 0 else 0)
    return sum(ious) / num_classes * 100
```

## Common pitfalls

- Freezing the pre-trained encoder weights before training the downstream linear classifier or decoder, as required by the protocol.
- Using single-crop evaluation for ImageNet accuracy rather than multi-crop or test-time augmentation.
- Evaluating different view combinations (e.g., L vs ab vs depth) requires retraining the contrastive model from scratch, not just swapping input channels.

## Evidence (verbatim from paper)

> The quality of such a representation is evaluated by freezing the weights of encoder and training linear classifier on top of each layer. Table 1: Top-1 / Top-5 Single crop classification accuracy (%) on ImageNet with a supervised logistic regression classifier.

## Citation

```bibtex
@misc{tian2019contrastive,
  title={Contrastive Multiview Coding},
  author={Tian et al. (2019)},
  year={2019},
  note={arXiv:1906.05849}
}
```

- arXiv: 1906.05849