Sharegpt4v Improving Large Multi Modal Models With Better Captions Arxiv 2311 12793v2

# ShareGPT4V: Improving Large Multi-Modal Models with Better Captions

## One-line decision
Use this skill when you want to generate high-quality detailed captions using GPT-4V and then train a captioner to scale caption generation. Avoid it when you have sufficient high-quality captions or cannot afford GPT-4V API calls for the seed set.

## Skill metadata
- **Skill type**: high-quality-caption-generation
- **Paper kind**: operational-method
- **Actionability**: high
- **Evidence quality**: full_paper

## Goal
Generate highly detailed and accurate image captions using GPT-4V on a seed set, then train an open-source captioner (Share-Captioner) to scale this caption quality to millions of images for VLM pretraining.

## Problem signature
- Modality: images with detailed synthetic captions generated by GPT-4V and an open-source captioner.
- Data state: existing images from various sources needing higher quality captions than their original alt-text or short captions.
- Scale regime: 100K GPT-4V seed captions scaled to 1.2M captions via Share-Captioner.
- Model requirement: GPT-4V for seed caption generation; a fine-tuned captioning model (Share-Captioner) for scaling.

## Use when
- You need detailed, high-quality captions for VLM pretraining data.
- You can afford GPT-4V API calls for a seed set of ~100K images.
- You want to train a scalable open-source captioner for large-scale recaptioning.

## Do not use when
- Your existing captions are already detailed and accurate.
- You cannot afford any GPT-4V API calls for seed generation.
- You need captions for a domain where GPT-4V lacks expertise.

## Required inputs
- **source_images**: Images to be captioned (from COCO, SAM, web crawl, etc.).
- **gpt4v_api**: Access to GPT-4V for generating seed captions on ~100K images.
- **base_captioning_model**: A VLM that can be fine-tuned as Share-Captioner (e.g., LLaVA-7B).

## Optional inputs
- **caption_quality_filter**: Filter to remove hallucinated or low-quality generated captions.
- **original_captions**: Original captions to compare against for quality assessment.

## Outputs
- **sharegpt4v_captions**: 1.2M high-quality detailed captions for VLM pretraining.
- **share_captioner**: Open-source captioning model capable of generating GPT-4V-quality captions at scale.
- **improved_vlm**: VLM trained on the higher-quality captions showing improved performance.

## Assumptions and prerequisites
- GPT-4V generates more detailed and accurate captions than existing alt-text or short captions.
- A fine-tuned captioner can reproduce GPT-4V caption quality at a fraction of the cost.
- Better captions during pretraining improve downstream VLM performance.

## Procedure
1. **Generate seed captions with GPT-4V**
   Action: Use GPT-4V to generate detailed captions for ~100K diverse images.
   Why: Creates a high-quality seed set for training the scalable captioner.
   Note: Use structured prompts requesting comprehensive descriptions of content, spatial relationships, attributes.
2. **Train Share-Captioner**
   Action: Fine-tune a VLM (e.g., LLaVA-7B) on the 100K GPT-4V captions to create Share-Captioner.
   Why: Enables scaling GPT-4V-quality captions to millions of images without API cost.
   Note: See paper for details.
3. **Scale caption generation**
   Action: Use Share-Captioner to generate detailed captions for 1.2M images.
   Why: Provides high-quality captions at scale for VLM pretraining.
   Note: See paper for details.
4. **Replace original captions in pretraining data**
   Action: Substitute the generated captions for original alt-text or short captions in pretraining data.
   Why: Higher quality captions improve vision-language alignment during pretraining.
   Note: See paper for details.
5. **Train VLM on improved captions**
   Action: Pre-train a VLM using the ShareGPT4V captions and evaluate on benchmarks.
   Why: Validates that caption quality improvements translate to model improvements.
   Note: See paper for details.

## Parameters to set
- **seed_set_size** — Role: Number of images captioned by GPT-4V. How to set: ~100K provides sufficient diversity for training Share-Captioner. Default/range: 100K. Effect: Larger seed sets may improve captioner quality but increase API cost.
- **caption_detail_level** — Role: Level of detail in generated captions. How to set: Use prompts requesting spatial, attribute, and relationship descriptions. Default/range: Highly detailed. Effect: More detailed captions provide richer supervision.
- **scale_target** — Role: Total number of captions to generate with Share-Captioner. How to set: Scale to cover your pretraining dataset. Default/range: 1.2M. Effect: More captions improve pretraining coverage.

## Validation checks
- Share-Captioner captions should be comparable in detail and accuracy to GPT-4V captions on held-out images.
- VLM trained on ShareGPT4V captions should outperform one trained on original captions.
- Random samples should be audited for hallucination and caption faithfulness.

## Failure modes
- GPT-4V may hallucinate details for ambiguous or low-quality images.
- Share-Captioner may degrade in quality on out-of-distribution images.
- Detailed captions may introduce verbosity that doesn't help downstream tasks.

## Adaptation notes for VLM training
- Replace GPT-4V with Claude or other VLMs for seed caption generation.
- The captioner-scaling approach generalizes to video and 3D data.
- Use ShareGPT4V captions to improve any VLM's pretraining data quality.

## Implementation notes
- Use structured prompts for GPT-4V to ensure consistent caption format.
- Batch API calls efficiently to minimize cost.
- Store both original and generated captions for comparison and rollback.

## Evidence from the paper
- ShareGPT4V produces 1.2M highly detailed captions by training Share-Captioner on 100K GPT-4V-generated captions.
- Replacing original captions with ShareGPT4V captions in pretraining improves LLaVA performance across multiple benchmarks.
- Share-Captioner generates captions comparable in quality to GPT-4V at a fraction of the cost.
- ShareGPT4V-7B achieves competitive performance with GPT-4V on several visual understanding benchmarks.

## Source paper
- **Title**: ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
- **Year**: 2023
- **Venue**: ECCV
- **Paper ID**: arxiv-2311.12793v2
- **URL**: http://arxiv.org/abs/2311.12793v2
- **arXiv ID**: 2311.12793v2