Phi 3 Technical Report A Highly Capable Language Model Locally On Your Phone Arxiv 2404 14219v2

# Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

## One-line decision
Use this skill when you want to train high-capability small models using heavily filtered web data augmented with synthetic textbook-quality data. Avoid it when you are training a large model where data quality filtering is less impactful.

## Skill metadata
- **Skill type**: synthetic-textbook-data
- **Paper kind**: operational-method
- **Actionability**: high
- **Evidence quality**: full_paper

## Goal
Train a highly capable small language model (3.8B) by combining heavily filtered web data with synthetic textbook-quality data generated by LLMs, demonstrating that data quality enables small models to punch well above their weight.

## Problem signature
- Modality: text: heavily filtered web data + LLM-generated synthetic textbook data.
- Data state: web data filtered for educational value + synthetically generated educational content.
- Scale regime: 3.3 trillion tokens of curated and synthetic data.
- Model requirement: Phi-3 transformer architecture (3.8B parameters).

## Use when
- You want maximum capability from a small model.
- You can invest in data quality filtering and synthetic generation.
- You need deployment-efficient models.

## Do not use when
- You are training a large model where quantity matters more.
- You cannot afford synthetic data generation.
- You need a general-purpose large model.

## Required inputs
- **web_data**: Web text heavily filtered for educational quality.
- **synthetic_data**: LLM-generated textbook-quality educational content.
- **quality_filters**: Aggressive filters for selecting educational content.

## Optional inputs
- **domain_specific_data**: Additional domain-specific educational content.

## Outputs
- **phi3_training_data**: 3.3T tokens of quality-curated data.
- **phi3_model**: 3.8B model with strong reasoning capability.

## Assumptions and prerequisites
- Data quality is the primary determinant of small model capability.
- Synthetic textbook-quality data provides the highest quality training signal.
- Aggressive filtering can match or exceed the value of more data.

## Procedure
1. **Filter web data aggressively**
   Action: Apply quality classifiers to select only educational, reasoning-rich web content.
   Why: Quality over quantity maximizes small model capability.
   Note: See paper for details.
2. **Generate synthetic textbook data**
   Action: Use LLMs to generate educational content mimicking textbook quality.
   Why: Synthetic data fills gaps in web data quality.
   Note: See paper for details.
3. **Combine filtered and synthetic data**
   Action: Mix aggressively filtered web data with synthetic content.
   Why: Both sources contribute complementary quality training signal.
   Note: See paper for details.
4. **Train Phi-3**
   Action: Train the 3.8B model on the curated data.
   Why: Quality data enables strong capability from a small model.
   Note: See paper for details.

## Parameters to set
- **web_filter_threshold** — Role: Aggressiveness of web data filtering. How to set: Keep only the highest quality web content. Default/range: Very aggressive. Effect: More aggressive filtering improves quality at the cost of quantity.
- **synthetic_ratio** — Role: Ratio of synthetic to filtered web data. How to set: Significant synthetic component. Default/range: Task-dependent. Effect: More synthetic data provides higher quality training signal.

## Validation checks
- Phi-3-mini should outperform much larger models on reasoning benchmarks.
- Data quality should demonstrably drive performance.
- The model should handle diverse tasks despite small size.

## Failure modes
- Aggressive filtering may create domain gaps.
- Synthetic data may lack the diversity of web data.
- Small models have inherent capacity limitations.

## Adaptation notes for VLM training
- The quality-focused approach applies to VLM training data curation.
- Use synthetic data to augment filtered web data for VLM pretraining.
- Data quality insights transfer to multimodal settings.

## Implementation notes
- Invest in quality classifier development.
- Generate synthetic data at scale using strong LLMs.
- Monitor capability across diverse benchmarks.

## Evidence from the paper
- Phi-3-mini (3.8B) matches Mixtral-8x7B on many benchmarks through data quality.
- Heavily filtered web data + synthetic textbook data enables remarkable small model capability.
- Data quality is the primary driver of model capability for small models.
- The approach demonstrates that careful data curation can substitute for model scale.

## Source paper
- **Title**: Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
- **Year**: 2024
- **Venue**: arXiv
- **Paper ID**: arxiv-2404.14219v2
- **URL**: http://arxiv.org/abs/2404.14219v2
- **arXiv ID**: 2404.14219v2