Cinic 10 Eval

---
name: cinic-10-eval
description: Evaluates image classification performance under significant domain shift between synthetic (CIFAR-10) and real-world/downsampled (ImageNet) sources. Probes model robustness to distributional bias and class-level statistical divergence across training and test domains. Use when the user wants to benchmark on CINIC-10, or asks about evaluating this task. Reports Test Error.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1810.03505
  bibtex_key: darlow2018cinic10
  confidence: high
---

# cinic-10-eval

> CINIC-10 is not ImageNet or CIFAR-10 — Darlow et al. (2018) (arXiv:1810.03505, 2018)

## What this evaluates

Evaluates image classification performance under significant domain shift between synthetic (CIFAR-10) and real-world/downsampled (ImageNet) sources. Probes model robustness to distributional bias and class-level statistical divergence across training and test domains.

## Datasets

- **CINIC-10** — total 270000; splits: train (-1), val (-1), test (-1); repo https://github.com/BayesWatch/cinic-10

## Metrics

- `Test Error` **(primary)** — range: percent
  - Computed as 1 minus top-1 accuracy. The paper reports the mean and standard deviation across multiple experimental runs.

## Input / output format

**Input**: 32×32 RGB images with corresponding class labels from 10 categories.

**Output**: Predicted class label (or class probabilities) for each image.

## Scoring recipe

```python
def compute_test_error(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    accuracy = correct / len(gold_labels)
    return (1 - accuracy) * 100  # Returns percentage
```

## Common pitfalls

- Models may exploit synthetic artifacts present in the downsampled ImageNet images rather than learning robust features.
- The domain shift between CIFAR-10 and ImageNet causes distributional bias that standard data augmentation may not fully mitigate.
- Reporting only mean test error without confidence intervals or multiple seeds can mask high variance in domain adaptation performance.

## Evidence (verbatim from paper)

> Table 1: CINIC-10 benchmarks.  

<table><tr><td>Model</td><td>No. Parameters</td><td>Test Error</td></tr>...

## Citation

```bibtex
@misc{darlow2018cinic10,
  title={CINIC-10 is not ImageNet or CIFAR-10},
  author={Darlow et al. (2018)},
  year={2018},
  note={arXiv:1810.03505}
}
```

- arXiv: 1810.03505