Distributed And Backends

---
name: distributed-and-backends
description: "Choose and troubleshoot DALLE-pytorch CUDA, DeepSpeed, Horovod,
  Apex, Docker, sparse attention, and distributed checkpoint paths."
metadata:
  disco-role: operating
disable-model-invocation: true
license: MIT
---

# Distributed and backend workflows

Use this sub-skill for CUDA readiness, DeepSpeed, Horovod, Apex AMP, Docker GPU images, sparse attention, distributed parser flags, and checkpoint behavior.

## Read first

- `references/distributed-backends.md` for backend flags, module availability, sparse attention, precision, and checkpoint notes.
- `references/docker-and-system-setup.md` for Docker/source-build caveats.
- `references/troubleshooting.md` for backend import, CUDA, Apex, Horovod, and DeepSpeed failures.
- `scripts/check_backend_availability.py` to inspect safe backend availability without launching training.

## Typical routes

| Request | Action |
| --- | --- |
| "Can I use DeepSpeed?" | Run/describe backend availability check, confirm `deepspeed` import, decide whether sparse attention or only distributed training is needed. |
| "What does `--deepspeed` do?" | Explain parser wrapping and that `args.deepspeed` maps to the `DeepSpeed` backend when available. |
| "Use Horovod" | Confirm `horovod.torch` import and use `horovodrun`-style launch; batch size semantics differ from DeepSpeed. |
| "Install Apex or sparse attention" | Treat as CUDA/source-build task requiring explicit approval; do not run bundled source install scripts blindly. |
| "Choose `attn_types`" | Use package attention options; only `sparse` needs DeepSpeed sparse attention, while `full`, `axial_row`, `axial_col`, and `conv_like` are package-level options. |

## Boundary notes

- Single-process VAE training belongs to `../vae-training/SKILL.md`.
- Single-process DALL-E training/data/tokenizer issues belong to `../dalle-training/SKILL.md`.
- Generation belongs to `../generation-and-ranking/SKILL.md`.

## Safety rules

- Do not run source builds, `sudo apt-get`, Docker builds, or distributed launchers without explicit approval.
- A CUDA allocation smoke is not proof that full training works; it only proves torch can see a device.
- Keep DeepSpeed/Apex/Horovod optional unless the user's selected workflow requires them.