Data Preparation

---
name: data-preparation
description: "Prepare and validate face.evoLVe identity-folder datasets,
  low-shot balancing, augmentation notes, and validation bcolz layouts."
disable-model-invocation: true
metadata:
  disco-role: operating
license: MIT
---

# data-preparation

Use this sub-skill when the request is about preparing or checking face.evoLVe
training data in identity folders, pruning low-shot classes safely, or
understanding validation pair files and public dataset/model-zoo artifacts.

## Use this sub-skill for

- ImageFolder-style identity folders for PyTorch training.
- PaddlePaddle `NormalDataset` / balancing datasets that read identity folders
  directly.
- Hidden-file cleanup and class-count validation.
- Low-shot removal with a dry-run first.
- RandAugment-style augmentation notes.
- Validation `bcolz` carray folders plus matching `_list.npy` files.
- Public data/model-zoo naming, downloads, and licensing cautions.

## Route elsewhere when needed

- Need MTCNN detection, landmarks, alignment, or resize preprocessing ->
  `face-alignment`.
- Need PyTorch config editing, model construction, or training loops ->
  `pytorch-training`.
- Need checkpoint feature extraction or verification metrics ->
  `feature-extraction-verification`.
- Need PaddlePaddle training, quantization, or deployment -> `paddle-workflows`.

## Read or run

- Read `references/data-preparation.md` for folder rules, low-shot handling,
  augmentation notes, and validation-pair layout.
- Read `references/data-and-model-zoo.md` before choosing a public dataset or
  model artifact.
- Read `references/troubleshooting.md` when class folders, hidden files,
  `bcolz`, or downloads fail.
- Run `scripts/check_image_folder.py` to validate a tiny fixture or a real
  identity-folder root before training or pruning.
- Run `scripts/remove_lowshot_safe.py` to report or prune classes below a
  threshold without mutating the source tree by default.

## Typical decisions

- Validate first, prune second, and re-validate after any change.
- Prefer weighted sampling in training when the user wants imbalance handling
  without deleting identities.
- Treat validation pair files as a separate artifact family from ImageFolder
  training data.
- Treat public downloads as external, optional artifacts, not bundled files.

## Guardrails

- Keep the dataset root flat at the identity-folder level; do not mix in
  alignment or checkpoint logic here.
- Remove or report hidden files such as `.DS_Store` before pruning or training.
- Count only real image files when deciding whether a class is low-shot.
- Do not assume the repo checkout includes large datasets, pair archives, or
  model weights.