Open Clip

---
name: open-clip
description: "Route OpenCLIP model loading, inference, training, audio CLAP,
  NaFlex generative workflows, evaluation, checkpoint conversion, and package
  troubleshooting tasks."
disable-model-invocation: true
metadata:
  disco-role: operating
license: NOASSERTION
---

# OpenCLIP

Use this repo skill for OpenCLIP (`open_clip_torch`) tasks involving CLIP-style vision-language models, CoCa, CLAP audio-text models, NaFlex variable-resolution pipelines, GenLIP/GenLAP generative models, training commands, zero-shot evaluation, retrieval metrics, checkpoint conversion, and Hugging Face export planning.

## Start Here

- Install for inference with `pip install open_clip_torch`; install selected extras only when needed, such as `pip install 'open_clip_torch[training]'` for training/data utilities and `pip install 'open_clip_torch[audio]'` for CLAP audio workflows.
- Run `python scripts/check_open_clip_env.py` to confirm importability, package version, model registry access, and optional audio/training modules before deep troubleshooting.
- Read `references/repo-provenance.md` before refreshing or comparing this skill to a source checkout.
- Read `references/troubleshooting.md` for cross-cutting install, optional dependency, cache/download, torch/CUDA, and stale-skill issues.
- Read `references/source-script-map.md` when a user asks about bundled helpers or why original long-running scripts are not directly reproduced.

## Route By Task

| User intent | Read first | Why |
| --- | --- | --- |
| Load a model, choose `pretrained`, tokenize text, preprocess images, create embeddings, or inspect model config | `sub-skills/model-inference/SKILL.md` | Owns `create_model*`, `get_tokenizer`, transforms, pretrained tags, `hf-hub:`/`local-dir:`, CoCa inference, and no-download smoke checks. |
| Build or debug training commands, datasets, task wrappers, precision, FSDP, `torch.compile`, optimizer/scheduler, checkpoints, or parser defaults | `sub-skills/training/SKILL.md` | Owns `python -m open_clip_train.main`, dict batch contracts, CSV/WebDataset/synthetic data, task-era wrappers, and training-safe helper scripts. |
| Work with CLAP or NaFlexCLAP audio models, audio transforms, `webdataset-audio`, HF audio zero-shot, or HF CLAP checkpoint conversion | `sub-skills/audio-clap/SKILL.md` | Owns audio configs, audio dict shapes, audio zero-shot flags, optional audio extras, and CLAP conversion troubleshooting. |
| Use `--use-naflex`, variable image/audio token budgets, patch dictionaries, NaFlex conversion, GenLIP, GenLAP, generative scoring, or probe planning | `sub-skills/naflex-generative/SKILL.md` | Owns NaFlex data config, `patches`/`patch_coord`/`patch_valid`, GenLIP/GenLAP model differences, and expensive-script planning helpers. |
| Build zero-shot classifiers, ImageNet zero-shot routes, retrieval metrics, checkpoint key reports, state-dict conversion/export, or HF Hub push plans | `sub-skills/evaluation-conversion/SKILL.md` | Owns contrastive evaluation, retrieval chunking, checkpoint prefix/EMA diagnostics, conversion/export routing, and safe key-inspection helpers. |

## Common Decisions

- Use `model-inference` for ordinary CLIP/CoCa embedding tasks; use `evaluation-conversion` only when the task asks for classifier construction, retrieval metrics, checkpoint keys, conversion, or export.
- Use `audio-clap` for audio data and CLAP zero-shot; do not force ImageNet/image zero-shot workflows onto audio-only models.
- Use `naflex-generative` when images or audio are represented as patch dictionaries or when a GenLIP/GenLAP model lacks contrastive `encode_text` semantics.
- Use `training` for command construction and data validation; route model-family-specific NaFlex or audio details to the owning sub-skill.
- Treat pretrained downloads, Hugging Face datasets, checkpoint loading, probe training, and large training/evaluation runs as explicit user-authorized operations, not default smoke checks.

## Bundled Helpers

- `scripts/check_open_clip_env.py` verifies imports, package metadata, registry counts, optional audio/training imports, and selected torch backend facts without downloading weights.
- `sub-skills/model-inference/scripts/inference_smoke.py` creates a no-download model/tokenizer/preprocess smoke by default.
- `sub-skills/training/scripts/training_arg_report.py` reports parsed training defaults without building models, data loaders, or distributed process groups.
- `sub-skills/training/scripts/validate_csv_dataset.py` validates CSV/TSV path and caption columns without starting training.
- `sub-skills/audio-clap/scripts/audio_config_report.py` and `sub-skills/audio-clap/scripts/clap_zero_shot_args.py` inspect audio config and plan HF audio zero-shot flags without downloading datasets.
- `sub-skills/naflex-generative/scripts/naflex_config_report.py` and `sub-skills/naflex-generative/scripts/genlip_scoring_args.py` validate token-budget/generative scoring settings before expensive runs.
- `sub-skills/evaluation-conversion/scripts/zero_shot_classifier_smoke.py` and `sub-skills/evaluation-conversion/scripts/checkpoint_key_report.py` provide no-download classifier and checkpoint diagnostics.

## Guardrails

- Do not ask future agents to run OpenCLIP repository examples, tests, notebooks, or shell launchers as runtime instructions; use bundled helpers and references instead.
- Do not assume every optional dependency is installed. Training, audio, Hugging Face tokenizers, datasets, and backend features have separate dependencies and failure modes.
- Do not run download-prone pretrained/HF workflows unless the user explicitly allows network/cache access.
- Do not treat skipped native tests or heavyweight examples as passing; use the relevant helper and report safety limits.
- If the source repository commit, package version, public APIs, or major docs changed since `references/repo-provenance.md`, refresh this skill before relying on fine details.