On Policy Actor Critic

---
name: on-policy-actor-critic
description: "Router for REINFORCE, baseline, actor-critic, A2C, and PPO workflows."
metadata:
  disco-role: operating
disable-model-invocation: true
license: MIT
---

# On-policy Actor-Critic

Use this sub-skill for the repo's on-policy classic-control workflows in Char02, Char03, Char04, and Char07.

## Route here when the task says
- "run REINFORCE on CartPole"
- "compare baseline vs no-baseline"
- "inspect actor-critic updates"
- "use the A2C multiprocessing helper"
- "inspect PPO clipping"
- "play back a saved policy"

## Keep this route focused on
- REINFORCE and policy-gradient baselines in Char02.
- Actor-critic variants in Char03.
- A2C with `SubprocVecEnv` in Char04.
- PPO on CartPole, MountainCar, and Pendulum in Char07.
- Playback of saved policy pickles from the on-policy family.

## Do not use this route for
- Tabular control or GridWorld toy examples — route to the tabular-control route.
- DQN or other value-based discrete control — route to the value-based-discrete-control route.
- Off-policy continuous-control families such as DDPG, SAC, or TD3 — route to the off-policy continuous-control route.
- Standalone plotting or results aggregation.

## Read first
- `references/workflows.md` for the algorithm comparison, checkpoint map, and the exact source-script roles.
- `references/troubleshooting.md` for `env.seed`, old Gym step signatures, multiprocessing, checkpoint loading, TensorBoard paths, and headless plotting failures.

## Bundled helpers
- `scripts/multiprocessing_env.py` for the reusable subprocess vector-env helper used by A2C.
- `scripts/playback_saved_policy.py` for bounded playback of saved policy pickles from the REINFORCE / actor-critic family.

## Fast choice guide
- No critic, episode returns only, `saved_log_probs`: REINFORCE.
- Policy plus value head, advantage from `G - V`: actor-critic or baseline.
- Multiple env processes and short rollouts: A2C.
- `clip_param`, ratio clipping, minibatches, repeated epochs: PPO.
- A saved `.pkl` policy object that should be evaluated rather than trained: playback helper.

## Compatibility notes
- The inspection environment verified Gym 0.23.1 with classic-control envs and 4-value `step(...)` tuples.
- The source scripts still use legacy `env.seed(...)` and legacy env IDs such as `CartPole-v0`, `MountainCar-v0`, and `Pendulum-v0`.
- Treat `Pendulum-v0` as a compatibility note; use `Pendulum-v1` in the inspected environment when you need a modern substitute.
- This route is CPU-first, even though CUDA is available in the inspection environment.