Imagenet 21k Pretraining For The Masses Arxiv 2104 10972v4

# ImageNet-21K Pretraining for the Masses

## One-line decision
Use this skill when you want to use ImageNet-21K for vision encoder pretraining with proper preprocessing and training recipes. Avoid it when ImageNet-1K or CLIP pretraining is sufficient.

## Skill metadata
- **Skill type**: imagenet21k-preprocessing
- **Paper kind**: operational-method
- **Actionability**: high
- **Evidence quality**: full_paper

## Goal
Provide standardized preprocessing and training recipes for using ImageNet-21K (14M images, 21K classes) for vision model pretraining, making large-vocabulary supervised pretraining accessible.

## Problem signature
- Modality: ImageNet-21K images for supervised pretraining.
- Data state: ImageNet-21K cleaned and preprocessed for efficient training.
- Scale regime: 14M images, 21K classes.
- Model requirement: Any ViT or CNN for supervised pretraining.

## Use when
- You want supervised pretraining on 21K classes.
- ImageNet-1K is too small for your pretraining.
- You need a large-vocabulary vision encoder.

## Do not use when
- ImageNet-1K pretraining is sufficient.
- CLIP pretraining is preferred.
- 21K classes are unnecessary.

## Required inputs
- **imagenet_21k**: Full ImageNet-21K dataset.
- **preprocessing_pipeline**: Cleaning and preprocessing pipeline.
- **training_recipe**: Optimized training hyperparameters.

## Optional inputs
- **class_hierarchy**: WordNet class hierarchy for analysis.

## Outputs
- **preprocessed_21k**: Cleaned ImageNet-21K for training.
- **pretrained_model**: Vision model pretrained on 21K classes.

## Assumptions and prerequisites
- 21K classes provide richer supervision than 1K.
- Proper preprocessing significantly improves training.
- Supervised pretraining on 21K complements CLIP.

## Procedure
1. **Preprocess ImageNet-21K**
   Action: Clean, deduplicate, and balance the 21K dataset.
   Why: Raw 21K has quality issues.
   Note: See paper for details.
2. **Design training recipe**
   Action: Optimize hyperparameters for 21K pretraining.
   Why: 21K training needs different recipes than 1K.
   Note: See paper for details.
3. **Pretrain vision model**
   Action: Train ViT or CNN on preprocessed 21K.
   Why: Produces a strong vision encoder.
   Note: See paper for details.
4. **Transfer to downstream**
   Action: Fine-tune on target tasks.
   Why: Validates pretraining quality.
   Note: See paper for details.

## Parameters to set
- **preprocessing_steps** — Role: Cleaning steps for 21K. How to set: Remove broken, too-small images. Default/range: Standard cleaning. Effect: Cleaning improves training quality.
- **training_epochs** — Role: Pretraining duration. How to set: 80-300 epochs depending on model. Default/range: 80. Effect: More epochs improve features.

## Validation checks
- 21K-pretrained should outperform 1K-pretrained on transfer.
- Preprocessing should measurably improve training.
- The recipe should be reproducible.

## Failure modes
- 21K has noisier labels than 1K.
- Some classes have very few images.
- Preprocessing may remove valid data.

## Adaptation notes for VLM training
- ImageNet-21K provides supervised pretraining for VLM vision encoders.
- Use alongside CLIP pretraining for complementary features.
- 21K vocabulary covers more visual concepts.

## Implementation notes
- Use the provided preprocessing code.
- Apply the recommended training recipe.
- Compare to 1K-only pretraining.

## Evidence from the paper
- ImageNet-21K pretraining improves transfer over 1K.
- Proper preprocessing is critical for 21K training.
- 21K vocabulary provides richer supervision.
- Standardized recipes make 21K pretraining accessible.

## Source paper
- **Title**: ImageNet-21K Pretraining for the Masses
- **Year**: 2021
- **Venue**: NeurIPS Workshops
- **Paper ID**: arxiv-2104.10972v4
- **URL**: http://arxiv.org/abs/2104.10972v4
- **arXiv ID**: 2104.10972v4