Data Preparation

---
name: data-preparation
description: "Validate and prepare raw text-classification inputs, label
  dictionaries, n-gram expansions, and cache expectations for the legacy
  brightmart/text_classification workflows."
disable-model-invocation: true
metadata:
  disco-role: operating
license: MIT
---

# Data Preparation

Use this sub-skill when you need to validate or normalize repo-style text-classification data before a model-specific workflow runs.

This sub-skill is for the legacy TensorFlow 1.x / TFLearn code paths that back the generated repo skill. Its bundled helpers are plain Python 3.7-compatible scripts and do not require TensorFlow, but they must still respect the repo's older data conventions.

## Covers
- raw single-label lines: `text __label__label`
- raw multi-label lines with repeated or space-separated `__label__` markers
- prediction TSV rows: `question_id<TAB>text`
- relation / two-sentence rows: `text1<TAB>text2 __label__label`
- vocabulary and label dictionaries
- HDF5 cache keys and pickle tuple shapes
- adjacent n-gram expansion
- multi-hot labels and fixed top-label alignment
- seq2seq label shift tokens `_GO`, `_END`, `_PAD`

## Read first
- `references/data-formats.md`
- `references/workflows.md`
- `references/troubleshooting.md`

## Use the bundled scripts
- `scripts/validate_text_classification_data.py`
- `scripts/generate_ngrams.py`

## Do not use for
- model architecture selection
- training loop behavior
- ensemble/logit fusion
- downstream task choice outside data preparation