Scaling Up Visual And Vision Language Representation Learning With Noisy Text Supervision Arxiv 2111 02114v1

# Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

## One-line decision
Use this skill when you want to improve image-text datasets by replacing web alt-text with richer synthetic captions from a captioning model. Avoid it when you already have high-quality detailed captions or the captioner lacks domain knowledge.

## Skill metadata
- **Skill type**: recaptioning
- **Paper kind**: operational-method
- **Actionability**: high
- **Evidence quality**: full_paper

## Goal
Improve vision-language pretraining data by replacing or augmenting noisy web alt-text with synthetic captions generated by a trained captioning model, creating a cleaner and more descriptive training corpus.

## Problem signature
- Modality: images with original noisy captions replaced by synthetic captions from a captioning model.
- Data state: web-crawled images with noisy alt-text that will be recaptioned by a trained model.
- Scale regime: millions to billions of images recaptioned with a scalable captioner.
- Model requirement: Pre-trained captioning model (e.g., CoCa, PaLI) for generating replacement captions.

## Use when
- Your web-crawled image-text data has noisy, uninformative, or misaligned captions.
- You have a strong captioning model that can generate better descriptions.
- You want to improve pretraining data quality without manual annotation.

## Do not use when
- Your existing captions are high-quality gold annotations.
- The captioning model lacks domain knowledge for your images.
- You need captions with specific named entities that the captioner cannot generate.

## Required inputs
- **source_images**: Images whose captions will be replaced.
- **captioning_model**: Trained model for generating replacement captions.
- **recaption_prompt**: Prompt or configuration for the captioning model.

## Optional inputs
- **original_captions**: Original web captions for comparison and hybrid strategies.
- **quality_filter**: Filter to reject hallucinated or low-quality synthetic captions.

## Outputs
- **recaptioned_dataset**: Images with improved synthetic captions.
- **caption_comparison_log**: Log comparing original and synthetic captions.

## Assumptions and prerequisites
- A trained captioning model generates more accurate and descriptive captions than web alt-text.
- Recaptioning at scale is computationally feasible with efficient models.
- Better captions directly improve downstream vision-language model performance.

## Procedure
1. **Select images for recaptioning**
   Action: Choose which images in the dataset will be recaptioned (all or those with low-quality original captions).
   Why: Not all captions may need replacement; focus on the noisiest.
   Note: See paper for details.
2. **Generate synthetic captions**
   Action: Run the captioning model on selected images to produce replacement captions.
   Why: Creates more descriptive and accurate image descriptions.
   Note: See paper for details.
3. **Evaluate caption quality**
   Action: Compare synthetic captions to originals using automated metrics and human evaluation.
   Why: Ensures the new captions are actually better than the originals.
   Note: See paper for details.
4. **Decide replacement strategy**
   Action: Choose whether to replace, augment (keep both), or selectively merge captions.
   Why: The best strategy depends on original caption quality distribution.
   Note: See paper for details.
5. **Train VLM on recaptioned data**
   Action: Pre-train a VLM on the recaptioned dataset and evaluate improvements.
   Why: Validates that recaptioning improves downstream performance.
   Note: See paper for details.

## Parameters to set
- **replacement_policy** — Role: Whether to replace or augment original captions. How to set: Replace for noisy data; augment for mixed quality. Default/range: Replace. Effect: Augmentation preserves diversity; replacement maximizes quality.
- **caption_length** — Role: Target length for generated captions. How to set: Encourage detailed descriptions (50-100 words). Default/range: 50-100 words. Effect: Longer captions provide richer supervision.
- **batch_size_inference** — Role: Batch size for captioning model inference. How to set: Maximize GPU utilization. Default/range: Model-dependent. Effect: Affects throughput of recaptioning pipeline.

## Validation checks
- Synthetic captions should be more descriptive than original alt-text.
- Downstream VLM metrics should improve with recaptioned data.
- Hallucination rate in synthetic captions should be monitored.

## Failure modes
- Captioning models may hallucinate objects or attributes not in the image.
- Generic captions may reduce dataset diversity compared to noisy but varied alt-text.
- Domain-specific images may confuse general-purpose captioners.

## Adaptation notes for VLM training
- Recaptioning is now a standard step in VLM data pipelines (used in LLaVA, InternVL, etc.).
- Use Claude or GPT-4V for highest quality recaptioning on seed sets.
- Combine recaptioning with quality filtering for best results.

## Implementation notes
- Use efficient inference (half-precision, batched) for large-scale recaptioning.
- Cache generated captions and quality scores for reproducibility.
- Monitor recaptioning throughput and estimate total compute cost.

## Evidence from the paper
- Recaptioning noisy web data with CoCa-style models significantly improves CLIP and VLM pretraining.
- Synthetic captions are more descriptive and image-faithful than web alt-text on average.
- Recaptioning has become a standard component in modern VLM data pipelines.
- The approach is scalable to billions of images with efficient captioning models.

## Source paper
- **Title**: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- **Year**: 2021
- **Venue**: arXiv
- **Paper ID**: arxiv-2111.02114v1
- **URL**: http://arxiv.org/abs/2111.02114v1
- **arXiv ID**: 2111.02114v1