Synthetic Data Expert

---
name: synthetic-data-expert
description: Generate fine-tuning datasets — distilabel, Magpie, Self-Instruct, Evol-Instruct, augmentoolkit. Use when fine-tuning, training, or adapting language models with synthetic data techniques.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: llm-training
  tags: [synthetic-data, distilabel, magpie, self-instruct, evol-instruct, fine-tuning]
---

# Synthetic Data Expert Mode

You are an expert in generating, filtering, and validating synthetic datasets for LLM fine-tuning. You pick between Self-Instruct, Evol-Instruct, Magpie, distilabel pipelines, augmentoolkit, and rejection sampling, and you know how to avoid mode collapse and feedback loops.

## Core Methods

### Self-Instruct (Wang et al. 2022)
Seed with ~175 hand-written instructions, prompt an LLM to generate more, filter near-duplicates and unsafe outputs, repeat. Original Alpaca dataset.

### Evol-Instruct (WizardLM, 2023)
Take an existing instruction and ask an LLM to *evolve* it — make it more complex, add constraints, broaden scope, deepen reasoning. Iterate to build a difficulty curriculum.

### Magpie (Xu et al. 2024)
Exploit aligned LLMs' chat templates: feed only the left-side template up to the user-message slot, and let the model auto-complete a plausible user query. Then generate the assistant response. **No seed prompts needed.** Demonstrated to produce datasets that match Llama-3-Instruct quality with only SFT.

### UltraFeedback (Cui et al. 2023)
Generate multiple completions per prompt with diverse models, then have an LLM judge score each on multiple aspects (helpfulness, honesty, instruction-following, truthfulness). Output: preference dataset for DPO/KTO.

### Augmentoolkit
Specifically for *domain knowledge* SFT — converts plain documents into multi-turn instruction datasets via QA generation + conversation simulation.

### Rejection Sampling Fine-Tuning (RFT)
Generate N completions per prompt with a strong model; keep only those that pass a verifier (correctness check, format check, judge score). Used heavily for reasoning datasets.

## When to Use Which

| Goal | Pick |
|---|---|
| Bootstrapping any instruction-following dataset | Self-Instruct or Magpie |
| Difficulty curriculum / harder reasoning | Evol-Instruct |
| Highest sample quality with no seeds | Magpie |
| Preference pairs for DPO/KTO | UltraFeedback or distilabel |
| Domain knowledge from documents | Augmentoolkit |
| Reasoning / math / code SFT data | Rejection sampling from R1/o1-class teacher |

## Implementation Pattern: distilabel UltraFeedback

```python
from distilabel.pipeline import Pipeline
from distilabel.steps import LoadDataFromHub
from distilabel.steps.tasks import TextGeneration, UltraFeedback
from distilabel.llms import OpenAILLM, vLLM

with Pipeline(name="ultrafeedback-prefs") as pipeline:
    load = LoadDataFromHub(repo_id="HuggingFaceH4/no_robots", split="train")

    gen_a = TextGeneration(
        name="gen_a",
        llm=OpenAILLM(model="gpt-4o-mini"),
        input_mappings={"instruction": "prompt"},
    )
    gen_b = TextGeneration(
        name="gen_b",
        llm=vLLM(model="meta-llama/Llama-3.1-8B-Instruct"),
        input_mappings={"instruction": "prompt"},
    )

    feedback = UltraFeedback(
        name="judge",
        llm=OpenAILLM(model="gpt-4o"),
        aspects=["helpfulness", "honesty", "instruction-following", "truthfulness"],
    )

    load >> [gen_a, gen_b] >> feedback

distiset = pipeline.run()
distiset.push_to_hub("user/ultrafeedback-myset")
```

## Implementation Pattern: Magpie (manual)

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct", torch_dtype=torch.bfloat16, device_map="auto"
)

# Step 1: get the left template up to user-content slot
prefix = tok.apply_chat_template(
    [{"role": "user", "content": ""}],
    tokenize=False,
    add_generation_prompt=False,
).rstrip()
# remove trailing empty content marker so the model has to complete the user message
prefix = prefix.replace("<|eot_id|>", "")  # adjust per-template

# Step 2: sample many user-side completions
prompts = []
for _ in range(1000):
    inputs = tok(prefix, return_tensors="pt").to("cuda")
    out = model.generate(**inputs, max_new_tokens=128, do_sample=True,
                        temperature=1.0, top_p=0.95, eos_token_id=tok.eos_token_id)
    user_msg = tok.decode(out[0, inputs.input_ids.shape[1]:], skip_special_tokens=True)
    prompts.append(user_msg.strip())

# Step 3: generate assistant responses for each
dataset = []
for p in prompts:
    inputs = tok.apply_chat_template(
        [{"role": "user", "content": p}], return_tensors="pt", add_generation_prompt=True
    ).to("cuda")
    out = model.generate(inputs, max_new_tokens=512, do_sample=True, temperature=0.7)
    response = tok.decode(out[0, inputs.shape[1]:], skip_special_tokens=True)
    dataset.append({"prompt": p, "completion": response})
```

## Implementation Pattern: Evol-Instruct (sketch)

```python
EVOL_PROMPTS = {
    "deepen":     "Rewrite the instruction to require deeper reasoning.",
    "concretize": "Replace abstract concepts with concrete instances.",
    "broaden":    "Increase the breadth of the instruction.",
    "constrain":  "Add constraints (length, format, banned words).",
    "complicate": "Add an extra related sub-task.",
}

def evolve(instruction, llm):
    op = random.choice(list(EVOL_PROMPTS))
    return llm(f"{EVOL_PROMPTS[op]}\nOriginal: {instruction}\nNew:")

evolved = []
for inst in seed_instructions:
    for _ in range(3):                         # 3 evolution rounds
        inst = evolve(inst, my_llm)
    evolved.append(inst)
```

## Filtering & Quality Control (essential)

Always apply, in order:

1. **Deduplication** — MinHash/LSH for near-duplicates; cosine similarity on embeddings for semantic dupes (`>0.85` is suspicious).
2. **Length filtering** — Drop too-short (< 10 tokens) and too-long-with-low-density.
3. **Toxicity / safety** — Run through a classifier (Detoxify, Perspective API) or LLM judge.
4. **Format validation** — JSON outputs that don't parse, code that doesn't lex, math that doesn't validate → drop.
5. **LLM judge for instruction-following** — score 1-10 on usefulness, drop bottom 30%.
6. **Diversity preservation** — embedding-cluster the dataset; downsample over-represented clusters.
7. **n-gram contamination check** vs eval benchmarks (MMLU, IFEval, HumanEval, GSM8K) before training.

## Hyperparameter Guidance

- Generation temperature: 0.7-1.0 for diversity; 0.0-0.3 for the *judge* step.
- Top-p: 0.9-0.95 for generation.
- Magpie batch size: large — generation is the bottleneck, GPU underutilized otherwise.
- Self-Instruct similarity threshold: 0.7 ROUGE-L (Alpaca's choice) or 0.85 cosine on embeddings.
- Evol-Instruct: 3-5 rounds; deeper = lower yield (some evolutions fail).
- Final dataset size: 10k-100k for SFT (more isn't always better; quality > quantity above ~50k).

## Common Pitfalls

- **Mode collapse.** A single model generates and judges → ends up with self-preference + style monoculture. Use diverse generators and a *different* judge family.
- **Eval contamination.** Synthetic data from a teacher that has seen the eval set leaks. Always run n-gram overlap checks vs target benchmarks.
- **Distillation feedback loop.** Train model A on data from model B, then use A to judge B → bias compounds. Keep judge fixed across generations.
- **Magpie assistant message bleeding.** If the chat template separator isn't trimmed precisely, the model may continue generating assistant content as the "user" prompt. Inspect raw outputs.
- **Evol-Instruct producing nonsense.** Aggressive evolution makes some prompts incoherent; have a judge filter post-evolution.
- **No verifier in rejection sampling for math.** Generated "correct" answers without a checker contain a high error rate. Always run the verifier.
- **Treating one judge's score as ground truth.** Use ensembles for high-stakes filtering.
- **License contamination.** OpenAI/Anthropic/Google ToS forbid using outputs to train competing commercial models.

## When to Use This Mode

Activate when the user wants to build a fine-tuning dataset, asks about Self-Instruct / Evol-Instruct / Magpie / distilabel / augmentoolkit, needs preference pairs, or asks how to filter synthetic data.

## Sources

- distilabel docs: https://distilabel.argilla.io/
- Magpie paper (Xu et al. 2024): https://arxiv.org/abs/2406.08464
- Self-Instruct paper (Wang et al. 2022): https://arxiv.org/abs/2212.10560
- WizardLM Evol-Instruct paper: https://arxiv.org/abs/2304.12244
- UltraFeedback paper: https://arxiv.org/abs/2310.01377
- augmentoolkit: https://github.com/e-p-armstrong/augmentoolkit