Dclm Datacomp For Language Models Arxiv 2406 11794v3

# DCLM: DataComp for Language Models

## One-line decision
Use this skill when you want a benchmark-driven approach to evaluate text data filtering strategies for LLM training, analogous to DataComp for CLIP. Avoid it when you are not filtering text data for LLM/VLM training.

## Skill metadata
- **Skill type**: language-model-data-benchmark
- **Paper kind**: operational-method
- **Actionability**: high
- **Evidence quality**: full_paper

## Goal
Extend the DataComp framework to language model training, providing a standardized benchmark for evaluating text data filtering and curation strategies on Common Crawl data.

## Problem signature
- Modality: text from Common Crawl with evaluable filtering strategies.
- Data state: large Common Crawl text pool requiring filtering for LLM training.
- Scale regime: 240 trillion token candidate pool filtered to various sizes.
- Model requirement: Standard transformer LLM for benchmarking filtering strategies.

## Use when
- You want to benchmark text data filtering strategies.
- You need principled guidance for LLM text data curation.
- You want to compare filtering approaches systematically.

## Do not use when
- You are not working with text data filtering.
- You have a well-established data pipeline.
- You need multimodal-specific filtering.

## Required inputs
- **common_crawl_pool**: 240T token Common Crawl candidate pool.
- **filtering_strategy**: Strategy for selecting a training subset.
- **evaluation_suite**: Downstream benchmarks for LLM evaluation.

## Optional inputs
- **model_based_filter**: Trained quality classifier for text filtering.

## Outputs
- **filtered_dataset**: Training data selected by the filtering strategy.
- **benchmark_results**: Downstream performance of LLM trained on filtered data.

## Assumptions and prerequisites
- Standardized benchmarking reveals best filtering practices.
- Filtering strategies have measurable impact on LLM quality.
- Results from smaller scales predict larger scale outcomes.

## Procedure
1. **Define candidate pool**
   Action: Establish the 240T token Common Crawl pool as the universe.
   Why: Common pool enables fair comparison.
   Note: See paper for details.
2. **Apply filtering strategy**
   Action: Run the candidate filtering strategy on the pool.
   Why: Filtering is the variable under study.
   Note: See paper for details.
3. **Train LLM on filtered data**
   Action: Train a standardized LLM on the filtered subset.
   Why: Fixed model ensures differences are data-driven.
   Note: See paper for details.
4. **Evaluate on benchmarks**
   Action: Run downstream evaluations.
   Why: Standardized evaluation enables comparison.
   Note: See paper for details.

## Parameters to set
- **pool_scale** — Role: Size of candidate pool. How to set: Use DCLM's provided pools. Default/range: 240T tokens. Effect: Larger pools allow more aggressive filtering.
- **filter_type** — Role: Type of filtering approach. How to set: Compare heuristic, classifier-based, and model-based. Default/range: Various. Effect: Different approaches have different strengths.

## Validation checks
- Filtered data should outperform random sampling baseline.
- Quality classifiers should beat heuristic filters.
- Results should be reproducible.

## Failure modes
- Optimal filters may not transfer to different pool compositions.
- Small-scale experiments may not predict large-scale outcomes perfectly.
- The benchmark may not capture all dimensions of data quality.

## Adaptation notes for VLM training
- DCLM filtering insights apply to the text component of VLM training.
- Use DCLM-validated quality classifiers for your own text data.
- The benchmark methodology extends to multimodal data filtering.

## Implementation notes
- Use the DCLM codebase and provided pools.
- Report results at standardized scales for comparison.
- Track per-benchmark improvements from filtering.

## Evidence from the paper
- DCLM extends DataComp to language model data filtering benchmarking.
- Quality classifier-based filtering achieves the best results.
- The 240T token pool enables systematic filtering experiments.
- DCLM-filtered data produces competitive LLM models.

## Source paper
- **Title**: DCLM: DataComp for Language Models
- **Year**: 2024
- **Venue**: NeurIPS
- **Paper ID**: arxiv-2406.11794v3
- **URL**: http://arxiv.org/abs/2406.11794v3
- **arXiv ID**: 2406.11794v3