Recap Datacomp 1b How To Turn A Filtered Dataset Into A Recaptioned One Arxiv 2406 08478v1

# Recap-DataComp-1B: How to Turn a Filtered Dataset into a Recaptioned One

## One-line decision
Use this skill when you want to recaption a billion-scale dataset like DataComp-1B using an open-source VLM to improve CLIP training. Avoid it when your dataset captions are already high-quality or you lack the compute for billion-scale recaptioning.

## Skill metadata
- **Skill type**: large-scale-recaptioning
- **Paper kind**: operational-method
- **Actionability**: high
- **Evidence quality**: full_paper

## Goal
Recaption the DataComp-1B dataset using an open-source VLM (LLaVA-1.5) to replace noisy web alt-text with detailed synthetic captions, demonstrating improved CLIP training from better captions at billion scale.

## Problem signature
- Modality: images with replaced captions generated by an open-source VLM.
- Data state: DataComp-1B images with original noisy web captions replaced by VLM-generated descriptions.
- Scale regime: 1.3 billion images recaptioned.
- Model requirement: LLaVA-1.5 or similar open-source VLM for recaptioning; CLIP for training evaluation.

## Use when
- You have a large dataset with noisy captions that need improvement.
- You want to use open-source VLMs for recaptioning instead of proprietary APIs.
- You want to improve CLIP training through better captions.

## Do not use when
- Your captions are already high-quality.
- You lack the compute for billion-scale VLM inference.
- You need captions with specific named entities that VLMs may omit.

## Required inputs
- **source_dataset**: DataComp-1B or similar large image-text dataset.
- **recaptioning_vlm**: Open-source VLM (LLaVA-1.5) for generating replacement captions.
- **recaption_prompt**: Prompt template for the VLM to describe each image.

## Optional inputs
- **original_captions**: Original web captions for comparison and hybrid strategies.
- **quality_filter**: Post-recaptioning filter for hallucinated captions.

## Outputs
- **recaptioned_dataset**: DataComp-1B with VLM-generated captions.
- **improved_clip**: CLIP model trained on recaptioned data with better performance.

## Assumptions and prerequisites
- Open-source VLMs can generate better captions than noisy web alt-text.
- Recaptioning at billion scale is computationally feasible.
- Better captions directly translate to better CLIP performance.

## Procedure
1. **Set up recaptioning infrastructure**
   Action: Deploy LLaVA-1.5 inference at scale across multiple GPUs.
   Why: Billion-scale recaptioning requires efficient distributed inference.
   Note: See paper for details.
2. **Design recaptioning prompt**
   Action: Create a prompt that elicits detailed, visually faithful descriptions.
   Why: Prompt quality directly affects caption quality.
   Note: See paper for details.
3. **Recaption all images**
   Action: Run VLM inference on all 1.3B images to generate replacement captions.
   Why: Comprehensive recaptioning maximizes the benefit.
   Note: See paper for details.
4. **Train CLIP on recaptioned data**
   Action: Train CLIP using the recaptioned DataComp-1B.
   Why: Validates that recaptioning improves CLIP performance.
   Note: See paper for details.
5. **Compare to original and baselines**
   Action: Benchmark against CLIP trained on original captions and other recaptioning approaches.
   Why: Demonstrates the value of open-source recaptioning.
   Note: See paper for details.

## Parameters to set
- **recaptioning_model** — Role: VLM used for generating captions. How to set: Use LLaVA-1.5 or similar capable open VLM. Default/range: LLaVA-1.5. Effect: Better VLMs produce better captions.
- **caption_max_length** — Role: Maximum length of generated captions. How to set: 77 tokens to match CLIP text encoder capacity. Default/range: 77 tokens. Effect: Longer captions may be truncated by CLIP.
- **inference_batch_size** — Role: Batch size for VLM inference. How to set: Maximize GPU utilization. Default/range: Model-dependent. Effect: Affects throughput of recaptioning pipeline.

## Validation checks
- Recaptioned CLIP should outperform original-caption CLIP on ImageNet zero-shot.
- Random samples of recaptions should be more descriptive than original alt-text.
- The recaptioning pipeline should complete in reasonable time.

## Failure modes
- VLM may hallucinate content not present in images.
- CLIP text encoder truncation may waste detailed caption information.
- Compute cost of billion-scale VLM inference is substantial.

## Adaptation notes for VLM training
- Apply the same recaptioning pipeline to other large datasets (LAION-5B, COYO).
- Use newer VLMs as they become available for better caption quality.
- Combine recaptioning with deduplication and filtering for maximum benefit.

## Implementation notes
- Use vLLM or similar for efficient batch VLM inference.
- Store both original and recaptioned text for analysis.
- Monitor GPU utilization and throughput during recaptioning.

## Evidence from the paper
- Recap-DataComp-1B recaptions 1.3B images using LLaVA-1.5, an open-source VLM.
- CLIP trained on recaptioned data outperforms the original DataComp-1B baseline.
- Open-source recaptioning is a viable alternative to proprietary API-based approaches.
- The approach demonstrates that caption quality matters as much as quantity for CLIP training.

## Source paper
- **Title**: Recap-DataComp-1B: How to Turn a Filtered Dataset into a Recaptioned One
- **Year**: 2024
- **Venue**: arXiv
- **Paper ID**: arxiv-2406.08478v1
- **URL**: http://arxiv.org/abs/2406.08478v1
- **arXiv ID**: 2406.08478v1