Phi 1 Textbooks Are All You Need Arxiv 2306 11644v2

# Phi-1: Textbooks Are All You Need

## One-line decision
Use this skill when you want to generate synthetic textbook-quality code data for training a small but highly capable code model. Avoid it when you have sufficient high-quality code data or are not training a code model.

## Skill metadata
- **Skill type**: code-textbook-synthesis
- **Paper kind**: operational-method
- **Actionability**: high
- **Evidence quality**: full_paper

## Goal
Generate synthetic textbook-quality code data using GPT-3.5, demonstrating that 1.3B models trained on high-quality synthetic data can match much larger models on code benchmarks.

## Problem signature
- Modality: synthetic code textbook data generated by LLMs.
- Data state: synthetic code exercises and textbook content generated by GPT-3.5.
- Scale regime: 6B tokens of synthetic + filtered code data.
- Model requirement: 1.3B parameter code model.

## Use when
- You want high-quality code training data.
- You can generate synthetic textbook content.
- You are training a small code model.

## Do not use when
- You have sufficient code data from GitHub.
- You are not training a code model.
- Synthetic code quality is insufficient.

## Required inputs
- **synthetic_generator**: GPT-3.5 for generating code textbook content.
- **quality_filters**: Filters for selecting textbook-quality code.
- **base_model**: 1.3B parameter model for training.

## Optional inputs
- **filtered_web_code**: Filtered web code for supplementation.

## Outputs
- **code_textbooks**: Synthetic textbook-quality code data.
- **phi_1**: 1.3B code model with strong performance.

## Assumptions and prerequisites
- Synthetic textbook code is higher quality than raw GitHub code.
- Small models benefit most from high-quality data.
- GPT-3.5 can generate useful code teaching examples.

## Procedure
1. **Generate code textbooks**
   Action: Use GPT-3.5 to generate textbook-quality code examples.
   Why: Creates high-quality code training data.
   Note: See paper for details.
2. **Filter web code**
   Action: Select textbook-quality code from web sources.
   Why: Supplements synthetic data with real code.
   Note: See paper for details.
3. **Train Phi-1**
   Action: Train a 1.3B model on synthetic + filtered code.
   Why: Validates quality-over-quantity approach.
   Note: See paper for details.

## Parameters to set
- **synthetic_tokens** — Role: Tokens of synthetic code data. How to set: 1B+ synthetic tokens. Default/range: 1B. Effect: More synthetic data improves code quality.
- **total_tokens** — Role: Total training tokens. How to set: 6B combined. Default/range: 6B. Effect: Quality matters more than quantity.

## Validation checks
- Phi-1 should outperform larger models on HumanEval.
- Synthetic code should be correct and educational.
- Quality data should compensate for small model size.

## Failure modes
- Synthetic code may have errors.
- The approach may not generalize beyond code.
- GPT-3.5 generation has cost.

## Adaptation notes for VLM training
- Synthetic textbook generation applies to visual reasoning training data.
- The quality-over-quantity insight transfers to VLM data.
- Generate educational content for domain-specific VLM training.

## Implementation notes
- Generate diverse code examples.
- Filter rigorously for quality.
- Benchmark against larger models.

## Evidence from the paper
- Phi-1 (1.3B) achieves 50.6% on HumanEval through synthetic textbook data.
- Textbook-quality data enables small models to match larger ones.
- The approach demonstrates that data quality can substitute for model scale.
- Phi-1 pioneered the synthetic textbook data approach.

## Source paper
- **Title**: Phi-1: Textbooks Are All You Need
- **Year**: 2023
- **Venue**: arXiv
- **Paper ID**: arxiv-2306.11644v2
- **URL**: http://arxiv.org/abs/2306.11644v2
- **arXiv ID**: 2306.11644v2