Remote Reward Models

---
name: remote-reward-models
description: "Operate Align-Anything remote reward servers, payload checks, and
  PPO remote-RM wiring."
metadata:
  disco-role: operating
disable-model-invocation: true
license: Apache 2.0
---

# Remote Reward Models

Use this sub-skill when you need to:
- start a local remote reward server
- validate the HTTP payload contract
- wire PPO training to a remote reward endpoint
- debug address, port, dataset, reward-type, or reward-function failures

## Fast route
1. Read `references/remote-rm-contract.md`.
2. Start the server with `scripts/start_remote_rm_template.sh` or the `python -m align_anything.models.remote_rm.run_reward_server` pattern.
3. Validate `/get_reward` with `scripts/probe_remote_rm_payload.py`.
4. If PPO still fails, check `references/troubleshooting.md`.
5. For PPO remote RM, use the adapted qwen2_5_vl launch pattern captured here: start the reward server first, export `REMOTE_RM_URL`, then run `deepspeed --module align_anything.trainers.text_to_text.ppo_remote_rm`.

## What this sub-skill covers
- Flask reward server lifecycle
- reward function registration and dataset-backed math verification
- `RemoteRewardModel` client calls and retries
- PPO remote RM config keys and launch order
- common failures around payload shape, reward type, dataset mapping, and tokenizer mismatch

## Primary facts
- Server entry point: `python -m align_anything.models.remote_rm.run_reward_server`
- HTTP endpoint: POST `/get_reward`
- Client class: `align_anything.models.remote_rm.remote_rm_client.RemoteRewardModel`
- PPO trainer: `align_anything.trainers.text_to_text.ppo_remote_rm`
- Math verifier reward: `math_verifier_reward_function`
- Bundled launch helpers: `scripts/start_remote_rm_template.sh`, `scripts/probe_remote_rm_payload.py`

## What is out of scope
- editing source implementation outside the skill tree
- production hardening, auth, or multi-tenant deployment
- non-HTTP reward training paths that do not use the remote RM server