Data Preparation

---
name: data-preparation
description: "Use BERT-pytorch to build, load, and debug vocabularies and
  dataset inputs from tab-separated sentence pairs."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# data-preparation

Use this sub-skill when the task is about corpus layout, vocabulary files, `bert-vocab`, `WordVocab`, `BERTDataset`, or pickle vocab troubleshooting.

## What this covers

- Build a vocabulary from a corpus with `bert-vocab` or the Python API.
- Read and write `WordVocab` pickle files.
- Inspect `BERTDataset` inputs and outputs.
- Diagnose malformed tab-separated rows, missing corpus files, and streaming-mode pitfalls.

## What to do first

- If the corpus is not two sentences per line separated by one tab, fix the corpus before building a vocab.
- If you are using streaming mode, pass an explicit `corpus_lines` value; for small jobs, prefer `on_memory=True`.
- Use `scripts/build_vocab_smoke.py` when you want a safe tiny fixture that proves both vocab creation and dataset loading.
- If the task is actually about model construction, training, or checkpoints, hand off to `../training/SKILL.md`.

## Read these references

- `references/corpus-format.md`: corpus layout, special tokens, dataset fields, and the tiny smoke recipe.
- `references/troubleshooting.md`: malformed rows, pickle problems, streaming-mode issues, and tokenization mistakes.
- `../training/SKILL.md`: follow this route after you have a valid corpus and vocab.

## Related scripts

- `scripts/build_vocab_smoke.py`: build a tiny vocab, reload it, and inspect one dataset item.
- `../../scripts/make_tiny_corpus.py`: create a deterministic two-line corpus fixture for experiments.

## Common outputs

- `WordVocab` stores special tokens at fixed ids: pad 0, unk 1, eos 2, sos 3, mask 4.
- `BERTDataset.__getitem__` returns `bert_input`, `bert_label`, `segment_label`, and `is_next` tensors.
- `WordVocab.load_vocab()` expects a pickle created by this package, not a text vocabulary.

## Boundary reminders

- Do not solve model initialization or checkpointing here.
- Do not hide corpus-format bugs behind training troubleshooting.
- Keep the route focused on data layout, vocab serialization, and dataset inspection.