Ppo Clipped Surrogate Objective

---
name: ppo_clipped_surrogate_objective
description: Compute and validate PPO clipped probability-ratio surrogate objectives and trust-region-style diagnostics.
---

# PPO Clipped Surrogate Objective

Use this skill when a PPO implementation, recovery harness, or audit needs the paper's clipped surrogate objective from Equation 7. Do not use it as a generic ratio clamp: the selected term must be the minimum of the unclipped and clipped advantage-weighted objectives.

## Inputs

- `new_log_probs`: log probabilities under the updated/current policy.
- `old_log_probs`: log probabilities under the behavior policy.
- `advantages`: advantage estimates aligned with the sampled actions.
- `clip_epsilon`: clipping half-width, often `0.2`.

## Outputs

- Per-sample ratios, unclipped terms, clipped terms, and selected terms.
- Mean objective and minimization loss.
- Clip fraction and approximate KL diagnostics.

## Workflow

1. Compute `ratio = exp(new_log_prob - old_log_prob)`.
2. Compute `ratio * advantage` for the conservative-policy-iteration surrogate term.
3. Clamp the ratio to `[1 - epsilon, 1 + epsilon]` and multiply by the same advantage.
4. Select the smaller objective contribution per sample.
5. Average selected terms and negate for minimization.
6. Record diagnostics for recovery analysis.

## Validation

Run `python tests/test_surrogate.py` from this skill directory. The tests cover positive-advantage upper clipping, negative-advantage lower clipping, and in-range no-op behavior.

## Limitations

This skill computes the policy objective only. Value loss, entropy bonus, rollout generation, and optimizer execution belong to separate modules.