Installs into .claude/skills of the current project.
Are you the author of Data Preparation?
Add the live security badge to your README. It updates with every re-scan.
[](https://www.skillsdirectory.com/skills/vectorspacelab-data-preparation-084ddd1c)
---
name: data-preparation
description: "Guides pix2tex formula extraction, tokenizer and dataset pickle
creation, LaTeX rendering, scraping boundaries, and data-format
troubleshooting."
disable-model-invocation: true
metadata:
disco-role: operating
license: MIT
---
# Data Preparation
Use this sub-skill when the user needs to prepare image/LaTeX pairs for
LaTeX-OCR, extract formulas from TeX-like documents, create a tokenizer, build
`Im2LatexDataset` pickle files, render formulas to PNGs, or plan safe data
acquisition. Use this before training and evaluation.
## Quick Route
1. Read [references/data-formats.md](references/data-formats.md) for equation
text files, image naming, tokenizer JSON, and dataset pickle expectations.
2. Read [references/rendering-and-preprocessing.md](references/rendering-and-preprocessing.md)
before rendering LaTeX to PNG or normalizing formulas.
3. Read [references/data-acquisition.md](references/data-acquisition.md) before
web, Wikipedia, StackExchange, arXiv, or Google Drive acquisition.
4. Read [references/troubleshooting.md](references/troubleshooting.md) for
missing system tools, bad formulas, naming mismatch, and import quirks.
5. Use [scripts/extract_math_snippets.py](scripts/extract_math_snippets.py) for
safe local formula extraction, and
[scripts/check_dataset_inputs.py](scripts/check_dataset_inputs.py) before
running the package dataset builder.
## Typical Dataset Build Flow
```bash
# Validate inputs first; this does not create a pickle.
python scripts/check_dataset_inputs.py --equations math.txt --images images/
# Then build the package dataset pickle when the validation looks right.
python -m pix2tex.dataset.dataset \
--equations math.txt \
--images images/ \
--tokenizer tokenizer.json \
--out dataset.pkl
```
To train a new tokenizer from formulas only:
```bash
python -m pix2tex.dataset.dataset --equations math.txt --vocab-size 8000 --out tokenizer.json
```
## Boundaries
- Training/evaluation commands belong in
[../training-and-evaluation/SKILL.md](../training-and-evaluation/SKILL.md).
- OCR inference and image capture belong in
[../ocr-inference/SKILL.md](../ocr-inference/SKILL.md).
- Network scraping, large downloads, TeX rendering, and dataset pickle creation
can be long or system-dependent; confirm the user's budget and tools before
running them.
