Openrlhf

---
name: openrlhf
description: "Use OpenRLHF for Ray/vLLM/DeepSpeed RLHF workflows, including
  dataset preparation, SFT/RM/DPO training, PPO-family RL and agent training,
  runtime operations, reward serving, LoRA merging, and troubleshooting."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# OpenRLHF

Use this repo skill when a user asks about OpenRLHF, a Ray + vLLM + DeepSpeed RLHF framework for supervised fine-tuning, reward modeling, preference optimization, PPO-family reinforcement learning, custom rewards, multi-turn agents, and distributed training operations.

OpenRLHF workflows are usually GPU/distributed and can trigger model downloads, Ray clusters, long training runs, or service processes. Prefer planning, validating, and generating commands first; run expensive operations only after confirming the user wants execution and the environment is ready.

## Fast Routing

- **Dataset schemas and validation**: use `sub-skills/data-preparation/` for SFT records, reward-model chosen/rejected pairs, DPO preferences, PPO prompt datasets, chat templates, mixed dataset probabilities, VLM image placeholders, and the bundled dataset validator.
- **SFT, reward-model, and DPO/IPO/cDPO training**: use `sub-skills/supervised-preference-training/` for `train_sft`, `train_rm`, `train_dpo`, LoRA/packing/DeepSpeed/checkpoint/logging flags, and safe command skeleton generation.
- **PPO, REINFORCE++, GRPO/RLOO, Ray/vLLM, custom rewards, and agents**: use `sub-skills/rl-agent-training/` for `train_ppo_ray`, hybrid engine placement, async rollout, single-turn reward functions, multi-turn agents, VLM agents, and OpenAI-compatible agent executors.
- **Installation, diagnostics, serving, LoRA merge, and utilities**: use `sub-skills/operations-and-utilities/` for dependency/runtime checks, CUDA/flash-attn/Ray/NCCL issues, reward-model serving, LoRA adapter merging, DeepSpeed checkpoint conversion, and Docker/system caveats.

## Start Safely

1. Identify the user’s target workflow: data preparation, supervised/preference training, RL/agent training, or runtime utility operations.
2. Check whether the request requires execution or only a plan, command, script, config review, or troubleshooting answer.
3. If execution is requested, classify safety first: local schema validation is usually safe; full training, Ray/vLLM startup, reward serving, Docker/NVIDIA setup, model downloads, and checkpoint conversion are expensive or stateful.
4. Route to the nearest sub-skill and read its linked references before giving detailed commands.
5. For cross-cutting failures, read `references/troubleshooting.md` and then the workflow-specific troubleshooting file in the owning sub-skill.

## Minimal Package Facts

- Distribution name: `openrlhf`.
- Import module: `openrlhf`.
- Version represented by this skill: `0.10.4`.
- Python requirement from packaging metadata: Python 3.10 or newer.
- Core dependency stack: PyTorch, Transformers, DeepSpeed, Ray, vLLM-related optional extras, flash-attn, datasets, PEFT/LoRA tooling, logging integrations, and CUDA-capable GPU infrastructure for realistic training.

A top-level `import openrlhf` only proves package metadata/import visibility. It does not prove CUDA, flash-attn, DeepSpeed, Ray, vLLM, model download, or distributed training readiness. Use the operations sub-skill for runtime checks.

## Common Workflows

- **Prepare SFT data**: route to `data-preparation`, validate keys and chat-template assumptions, then use `supervised-preference-training` to generate a `train_sft` command.
- **Train a reward model**: route to `data-preparation` for chosen/rejected pairs, use `supervised-preference-training` for `train_rm`, and use `operations-and-utilities` if the reward model will be served over HTTP.
- **Run DPO/IPO/cDPO**: route to `supervised-preference-training`; verify prompt/chosen/rejected fields through `data-preparation` first.
- **Run PPO or REINFORCE++ with Ray/vLLM**: route to `rl-agent-training`; use `operations-and-utilities` for Ray cluster, NCCL, CUDA, and vLLM runtime readiness.
- **Add custom reward or multi-turn agent logic**: route to `rl-agent-training` and generate a bundled template before adapting user logic.
- **Diagnose install or runtime failures**: route to `operations-and-utilities`; check optional dependency and GPU/backend status before changing training flags.

## References

- `references/repo-provenance.md`: source commit, package version, evidence paths, dirty-state notes, and refresh baseline.
- `references/troubleshooting.md`: cross-cutting install/import/backend/routing issues shared by all sub-skills.

## Boundaries

This skill is a self-contained guide distilled from repository source, docs, examples, scripts, and tests. Do not depend on the original checkout at runtime. If a user needs a reusable helper, use the bundled scripts under this skill or adapt their contents into the user’s project.

Do not run original OpenRLHF training examples or system scripts as smoke tests by default. Treat them as evidence for command shape unless the user explicitly asks to run them and the environment, GPUs, data, models, credentials, and time budget are suitable.