Data Pipelines

---
name: data-pipelines
description: "Create, inspect, convert, split, and validate SchNetPack atomistic
  datasets backed by ASE databases and built-in dataset modules."
disable-model-invocation: true
metadata:
  disco-role: operating
license: NOASSERTION
---

# SchNetPack Data Pipelines

Use this sub-skill when the task is about SchNetPack datasets: creating or loading ASE DB files, checking unit metadata, choosing properties, configuring `AtomsDataModule`, preparing split files, using transforms before batching, converting legacy metadata, or diagnosing data-loading errors.

## Start Here

1. Identify whether the user has a custom ASE DB (`.db`) or a built-in dataset module such as `QM9`, `MD17`, `rMD17`, `MD22`, `ANI1`, `ISO17`, `QM7X`, `MaterialsProject`, `OrganicMaterialsDatabase`, or `TMQM`.
2. Inspect dataset metadata before training: SchNetPack expects `_distance_unit`, `_property_unit_dict`, and `atomrefs` in the ASE DB metadata.
3. Use `ASEAtomsData` or `load_dataset` for direct reads and `AtomsDataModule` for train/validation/test splitting, batching, transforms, and unit conversion.
4. Keep training/Hydra command construction in `../training-configs/SKILL.md`; keep representation/output modules in `../models-atomistic/SKILL.md`; keep calculators, MD, and LAMMPS in `../interfaces-md/SKILL.md`.

## Primary References

- `references/data-formats.md` covers custom ASE DB creation, metadata, built-in dataset modules, split files, transforms, and validation snippets.
- `references/api-reference.md` lists the important classes, signatures, methods, split strategies, loader behavior, stats helpers, and CLI conversion helper usage.
- `references/troubleshooting.md` maps common unit, property, split, workdir, and legacy-conversion errors to concrete diagnostics and fixes.
- `scripts/convert_ase_units.py` updates legacy ASE DB metadata and atomrefs without relying on SchNetPack's source checkout.

## Safe Workflow

- For a tiny custom dataset, create the `.db` with `ASEAtomsData.create(..., distance_unit="Ang", property_unit_dict={...})`, add systems, reload with `ASEAtomsData`, and then wrap it in `AtomsDataModule`.
- For legacy ASE DBs, run `python sub-skills/data-pipelines/scripts/convert_ase_units.py --help` first, then update units on a copy of the database.
- For split problems, inspect the existing `split.npz` arrays before training; stale split files are reused and validated against requested counts.
- Avoid long downloads or training in diagnosis loops. Use tiny DBs and `num_workers=0` or `1` for fast local validation.