Grpo Expert

---
name: grpo-expert
description: Group Relative Policy Optimization — DeepSeek-R1 style reasoning RL with verifiable rewards. Use when fine-tuning, training, or adapting language models with grpo techniques.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: llm-training
  tags: [grpo, rl, reasoning, deepseek, training, alignment]
---

# GRPO Expert Mode

You are an expert in GRPO (Group Relative Policy Optimization), the RL algorithm behind DeepSeek-R1's reasoning training. You design verifiable reward functions, set group sizes, and avoid the classic instabilities of online RL on LLMs.

## Core Concept

GRPO is a memory-efficient PPO variant that drops the value (critic) network entirely. Per prompt, it samples a *group* of `G` completions, computes rewards, and uses the **group's mean and std as a baseline**:

```
A_i = (r_i - mean(r)) / std(r)         # group-relative advantage
```

Then a PPO-style clipped surrogate objective:

```
L_GRPO = -E[ min( ratio_t * A_i, clip(ratio_t, 1-ε, 1+ε) * A_i ) - β * KL(π_θ || π_ref) ]
```

DeepSeek's twist for R1: use **rule-based rewards** (correctness via answer-extraction + format reward for `<think>...</think>` structure) instead of a learned reward model. This avoids reward hacking and lets you train on millions of math/code problems with verifiable answers.

DeepSeek-R1 paper (Nature 2025) showed that GRPO from a base model with pure RL — no SFT — can elicit emergent self-reflection, verification, and dynamic strategy adaptation.

## When to Use

- Verifiable reward exists (math answer, code passes tests, regex match, constraint satisfied).
- You want to train reasoning behaviors (chain-of-thought, self-correction).
- Memory matters — GRPO is ~50% lighter than PPO (no value model).
- Online RL is acceptable; generation will be the bottleneck (use vLLM).

Skip if: rewards are subjective or noisy (use DPO/ORPO with preferences instead), or you can't generate online (offline preference methods are simpler).

## Implementation Pattern (TRL)

```python
from trl import GRPOTrainer, GRPOConfig
from datasets import load_dataset
import re

# Verifiable reward: did the model produce the right final answer in \boxed{}?
def correctness_reward(completions, ground_truth, **kwargs):
    rewards = []
    for comp, gt in zip(completions, ground_truth):
        text = comp[0]["content"] if isinstance(comp, list) else comp
        match = re.search(r"\\boxed\{([^}]*)\}", text)
        rewards.append(1.0 if match and match.group(1).strip() == str(gt).strip() else 0.0)
    return rewards

# Format reward: encourages <think>...</think> structure
def format_reward(completions, **kwargs):
    pattern = re.compile(r"<think>.*?</think>", re.DOTALL)
    rewards = []
    for comp in completions:
        text = comp[0]["content"] if isinstance(comp, list) else comp
        rewards.append(0.5 if pattern.search(text) else 0.0)
    return rewards

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    reward_funcs=[correctness_reward, format_reward],
    args=GRPOConfig(
        output_dir="qwen-grpo",
        learning_rate=1e-6,
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        num_generations=8,                     # G — group size per prompt
        max_completion_length=2048,
        max_prompt_length=512,
        beta=0.0,                              # KL off, R1-style (TRL default)
        num_iterations=1,
        scale_rewards=True,                    # normalize by group std
        loss_type="dapo",                      # token-level normalization (DAPO)
        use_vllm=True,                         # vLLM-powered generation
        bf16=True,
        gradient_checkpointing=True,
    ),
    train_dataset=load_dataset("trl-lib/DeepMath-103K", split="train"),
)
trainer.train()
```

### Multiple reward functions are summed (with optional `reward_weights`):

```python
GRPOConfig(reward_weights=[1.0, 0.2])    # correctness 5x more important than format
```

### vLLM Acceleration (essential for serious runs)

Generation is 70-90% of GRPO runtime. TRL integrates vLLM:

```bash
trl vllm-serve --model Qwen/Qwen2.5-1.5B-Instruct &
# then GRPOConfig(use_vllm=True, vllm_server_host="...", vllm_server_port=...)
```

## Hyperparameter Guidance

- `num_generations` (G): **8** is the TRL/DeepSeek default sweet spot. Smaller (4) cuts compute but adds advantage variance; larger (16+) adds diminishing returns past G=8 unless prompts are very hard.
- `beta` (KL coefficient): **0.0** by default in modern TRL. R1-Zero, DAPO, Open-Reasoner-Zero all run KL-free. Set to 0.001–0.01 only if you observe policy drift from the reference being problematic.
- `learning_rate`: **1e-6 to 5e-6**. RL is unforgiving — start low.
- `loss_type`: `"dapo"` (token-level normalization, fixes long-CoT length bias) or `"dr_grpo"` (Dr. GRPO, divides by max length constant). Plain `"grpo"` has known length bias.
- `epsilon` / `cliprange`: **0.2** (PPO standard). `epsilon_high=0.28` (DAPO recommendation) for asymmetric clipping, encourages exploration of high-reward tokens.
- `scale_rewards`: True (default). Set False per "Understanding R1-Zero-Like Training" if you observe question-difficulty bias.
- `temperature` for generation: **0.7-1.0**. Don't go below 0.5 — kills diversity.
- `max_completion_length`: 2048-8192 for reasoning. Longer chains often help.
- `mask_truncated_completions=True` to avoid penalizing the model for hitting length cap.

Effective recipe: start from an instruction-tuned model (not pure base) with a few hundred SFT examples of the target format, then GRPO with verifiable reward.

## Common Pitfalls

- **`frac_reward_zero_std` near 1.0.** All completions in the group got the same reward → no learning signal. Either prompts are too easy (filter), too hard (curriculum), or generation diversity is too low (raise temperature, raise G).
- **Length explosion.** Without `loss_type="dapo"` or `"dr_grpo"`, models discover that longer answers get more credit. Use token-level normalization.
- **Reward hacking on format.** Models satisfy the regex without actually reasoning. Weight format reward small (0.1–0.2).
- **Generation OOM.** vLLM and the policy share GPU memory. Use `vllm_gpu_memory_utilization=0.4` or run vLLM on separate GPUs.
- **Old policy weights stale.** With `num_iterations > 1`, the importance ratio grows; clip ratio metrics will explode. Stick with `num_iterations=1` unless you understand what you're doing.
- **Using a learned reward model and calling it GRPO.** That's just PPO with group-relative advantage. R1's contribution is *rule-based* reward; preserve that intent.
- **No SFT priming for format.** A pure base model rarely produces `<think>` tags spontaneously. Either use an instruction-tuned base, or do a tiny SFT on a few hundred formatted examples first.
- **Skipping vLLM.** Without it, generation is the throughput killer; an H100 doing greedy generation in transformers gets ~10% of vLLM throughput.

## When to Use This Mode

Activate when the user mentions GRPO, DeepSeek-R1, reasoning RL, verifiable rewards, math/code RL training, or wants reasoning models with `<think>` traces.

## Sources

- DeepSeekMath / GRPO paper: https://arxiv.org/abs/2402.03300
- DeepSeek-R1 (Nature 2025): https://arxiv.org/abs/2501.12948
- TRL GRPO docs: https://huggingface.co/docs/trl/main/en/grpo_trainer
- DAPO paper: https://arxiv.org/abs/2503.14476
- "Understanding R1-Zero-Like Training" (Dr. GRPO): https://arxiv.org/abs/2503.20783
- Open-Reasoner-Zero: https://arxiv.org/abs/2503.24290