P3o Surrogate Loss

---
name: p3o_surrogate_loss
description: Build decomposed P3O surrogate loss terms with clipped replay gradients and behavior-to-target KL regularization.
---

# P3O Surrogate Loss

Use this skill when a recovery or implementation must preserve the P3O objective from Equation 14. It is not a full RL framework; it computes deterministic scalar components that can be embedded in a learner.

## Inputs
- On-policy score-gradient proxies and advantages.
- Off-policy score-gradient proxies, advantages, and importance ratios.
- Target and behavior action distributions for KL estimation.
- `clip_threshold` and `kl_coefficient`, usually from the ESS scheduler.

## Outputs
- `on_policy`, `off_policy`, `kl_penalty`, and `objective` scalar values.
- Clipped ratios used for replay data.

## Workflow
1. Multiply on-policy advantages by score-gradient proxies.
2. Clip each replay importance ratio at `c` and multiply by replay advantage and score proxy.
3. Compute `KL(beta || pi_theta)` over behavior and target distributions.
4. Combine terms as `on + off - lambda * KL`.

## Source Boundary
Use this skill with the paper, module documents, generated artifacts, and ordinary package documentation. Do not read or depend on the original P3O repository.

## Validation
Run `python scripts/<script>.py --self-test` or `python -m pytest tests` from the skill directory. The bundled tests use only the Python standard library.