Scaling Up Visual And Vision Language Representation Learning With Noisy Text Supervision Arxiv 2102 05918v2

# Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

## One-line decision
Use this skill when you have a massive noisy image-text dataset and want to train a dual-encoder model that is robust to caption noise. Avoid it when you need clean, curated data or your dataset is too small for noisy supervision to work.

## Skill metadata
- **Skill type**: noisy-web-data-training
- **Paper kind**: operational-method
- **Actionability**: high
- **Evidence quality**: full_paper

## Goal
Demonstrate that vision-language representation learning can scale effectively with over 1 billion noisy image-text pairs from the web without expensive manual filtering.

## Problem signature
- Modality: image-text pairs with noisy alt-text supervision.
- Data state: over 1.8 billion noisy image-alt-text pairs from the web, minimally filtered.
- Scale regime: 1.8 billion image-text pairs for pretraining at scale.
- Model requirement: ALIGN dual-encoder with EfficientNet image encoder and BERT text encoder.

## Use when
- You have web-scale noisy image-text data and want to avoid expensive manual cleaning.
- You need a dual-encoder model robust to noisy captions.
- You want to show that scaling data size can compensate for noise.

## Do not use when
- You need high-precision captions for tasks like detailed image description.
- Your dataset is small enough that noise dominates signal.
- You need region-level or pixel-level supervision.

## Required inputs
- **noisy_image_text_pairs**: Large-scale image-alt-text pairs from web crawl with minimal filtering.
- **contrastive_loss**: Normalized softmax contrastive loss for dual-encoder training.
- **image_encoder**: EfficientNet or similar CNN/ViT architecture.

## Optional inputs
- **text_encoder**: BERT-based text encoder for the text branch.
- **basic_filters**: Simple frequency-based filtering to remove pornographic or very short alt-text.

## Outputs
- **align_model**: Dual-encoder model with aligned image-text embeddings trained on 1.8B noisy pairs.
- **image_text_retrieval_model**: Model suitable for cross-modal retrieval tasks.

## Assumptions and prerequisites
- Scale can compensate for noise in web-crawled alt-text.
- Simple filtering (removing very short or obviously bad text) is sufficient.
- Contrastive learning is robust to moderate label noise.

## Procedure
1. **Collect raw image-alt-text pairs**
   Action: Crawl the web and extract image-alt-text co-occurrences at billion scale.
   Why: Massive scale provides diverse visual concept coverage.
   Note: The ALIGN dataset contains 1.8B pairs.
2. **Apply minimal filtering**
   Action: Remove pairs with very short alt-text, pornographic content, or images below minimum resolution.
   Why: Removes the most obvious noise without expensive curation.
   Note: Filtering rules are simple frequency and length thresholds.
3. **Train dual-encoder with contrastive loss**
   Action: Train EfficientNet + BERT dual-encoder with normalized softmax contrastive loss.
   Why: Contrastive learning aligns image and text in a shared space.
   Note: Use very large batch sizes for effective negatives.
4. **Evaluate on retrieval and classification**
   Action: Test on Flickr30K, MSCOCO retrieval, and ImageNet zero-shot classification.
   Why: Standard benchmarks measure cross-modal alignment quality.
   Note: See paper for details.
5. **Compare to CLIP and supervised baselines**
   Action: Show that noisy 1.8B pairs outperform smaller curated datasets.
   Why: Demonstrates the data scaling hypothesis.
   Note: See paper for details.

## Parameters to set
- **dataset_size** — Role: Total number of noisy pairs. How to set: Collect as many as feasible from web crawl. Default/range: 1.8B. Effect: Larger noisy datasets improve performance despite noise.
- **min_alt_text_length** — Role: Minimum caption length to keep a pair. How to set: Remove very short captions (e.g., <3 words). Default/range: ~3 words. Effect: Removes trivially noisy pairs.
- **batch_size** — Role: Contrastive batch size. How to set: Scale to thousands for effective negatives. Default/range: 16384. Effect: Larger batches improve contrastive learning.

## Validation checks
- Flickr30K and MSCOCO retrieval metrics should exceed CLIP baselines.
- ImageNet zero-shot accuracy should improve with more data even with noise.
- Training should converge despite noisy supervision.

## Failure modes
- Noisy alt-text may teach wrong associations for rare concepts.
- Scale alone may not help for concepts not represented in web data.
- EfficientNet architecture may limit resolution-dependent tasks.

## Adaptation notes for VLM training
- The noisy-data-at-scale philosophy applies to VLM pretraining data collection.
- Minimal filtering can be augmented with CLIP score filtering post-hoc.
- The ALIGN approach validates that noise tolerance is a key VLM data strategy.

## Implementation notes
- Use distributed training with data parallelism for billion-scale datasets.
- Monitor the noise rate in random samples during training.
- Consider progressive filtering: start with noisy data, refine later.

## Evidence from the paper
- ALIGN leverages a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps.
- A simple dual-encoder architecture trained with contrastive loss achieves state-of-the-art results on Flickr30K (95.3% R@1) and MSCOCO retrieval.
- The scale of the dataset compensates for its noise and enables strong visual and cross-modal representations.
- ALIGN achieves 76.4% top-1 accuracy on ImageNet in a zero-shot setting.

## Source paper
- **Title**: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- **Year**: 2021
- **Venue**: ICML
- **Paper ID**: arxiv-2102.05918v2
- **URL**: http://arxiv.org/abs/2102.05918v2
- **arXiv ID**: 2102.05918v2