Training And Data Pipeline

---
name: training-and-data-pipeline
description: "Plan Protenix training data roots, custom CIF preprocessing, safe
  training and fine-tuning commands, DDP, W&B, checkpoints, and data-layout
  troubleshooting without launching expensive jobs."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# Protenix Training and Data Pipeline

Use this sub-skill when a task involves Protenix training or fine-tuning data roots, released or custom bioassembly data, CCD cache decisions, index CSVs, training command planning, DDP, W&B, checkpoints, or safe preflight checks.

## Start Here

1. Classify the task as data-root inspection, custom CIF preprocessing, CCD-cache planning, single-process training, multi-GPU/DDP training, or fine-tuning from checkpoints.
2. For data-root or index questions, run `scripts/check_training_data_layout.py`; it is read-only and uses only the Python standard library.
3. For custom CIF preprocessing, use `scripts/build_prepare_training_data_command.py` to print the command for later approval; read `references/preprocessing.md` before recommending the actual job.
4. For training or fine-tuning, use `scripts/build_training_command.py` to print a no-run command; read `references/training-workflows.md` and `references/configuration.md` before adapting overrides.
5. Treat full data downloads, CCD refreshes, MSA/template generation, preprocessing, and training as expensive operations that require explicit resource and side-effect approval.

## References

- `references/data-layout.md` explains `PROTENIX_ROOT_DIR`, released data components, MSA/template and RNA MSA directories, downloader modes, and safe root checks.
- `references/preprocessing.md` explains CIF-to-bioassembly preprocessing, cluster files, distillation mode, CCD cache handling, output `.pkl.gz` payloads, and index CSV schema.
- `references/training-workflows.md` explains train and fine-tune command anatomy, DDP with `torchrun`, W&B defaults, checkpoint behavior, kernels, output directories, and resource cautions.
- `references/configuration.md` explains Protenix config parsing, dot-notation overrides, required fields, dataset overrides, optimizer/scheduler settings, and validation order.
- `references/troubleshooting.md` maps common data, preprocessing, CUDA/DDP, W&B, checkpoint, config, and dependency failures to safe next actions.

## Bundled Tools

- `scripts/check_training_data_layout.py`: read-only checker for a `PROTENIX_ROOT_DIR`-style root and optional index CSV schemas.
- `scripts/build_prepare_training_data_command.py`: no-run builder for CIF preprocessing commands; it can include CCD-cache advisory text but does not refresh CCD data.
- `scripts/build_training_command.py`: no-run builder for training and fine-tuning commands with W&B disabled by default and optional `torchrun` launcher output.

## Route Elsewhere

- Single-job prediction, `protenix pred`, batch inference, and inference output interpretation belong to `../cli-and-inference/SKILL.md`.
- MSA/template search mechanics, input-prep CLI flows, and RNA MSA search details belong to `../msa-template-and-prep/SKILL.md`.
- Low-level kernels, model internals, advanced model config internals, and performance tuning beyond launch planning belong to `../advanced-model-configuration/SKILL.md`.

## Safety Rules

- Do not run data download, CCD refresh, CIF preprocessing, MSA/template search, or training commands as smoke tests.
- Default generated training commands to `--use_wandb false`; enable W&B only when the user confirms credentials, project naming, network policy, and non-interactive behavior.
- Prefer `python -m runner.train` for single-process training and `torchrun -m runner.train` for DDP command planning.
- Do not recommend `--triangle_attention deepspeed` unless the user has a compatible CUDA, DeepSpeed, Pydantic, and CUTLASS setup.
- Keep generated commands and notes portable: use user-supplied paths and placeholders, not local checkout paths or environment prefixes.