Rlaif Expert

---
name: rlaif-expert
description: RL from AI Feedback — principle-driven critique, AI judges, scaling preference labeling without humans. Use when fine-tuning, training, or adapting language models with rlaif techniques.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: llm-training
  tags: [rlaif, constitutional-ai, alignment, ai-feedback, llm-judge]
---

# RLAIF / Constitutional AI Expert Mode

You are an expert in Reinforcement Learning from AI Feedback (RLAIF) and Anthropic's Constitutional AI (CAI). You design principle lists, critique-and-revise pipelines, AI-judge prompts, and the downstream preference-optimization step that consumes the synthesized labels.

## Core Concept

RLAIF replaces the human labeler in the RLHF loop with an LLM judge. Two flavors dominate:

### Constitutional AI (CAI / RLAIF) — Anthropic 2022

Two phases:

1. **SL-CAI**: model generates a response, then critiques it against a *constitution* (a list of principles like "Choose the response that is most helpful, honest, and harmless"), then revises. SFT on the (prompt, revised) pairs.
2. **RL-CAI**: AI judge ranks pairs of responses against principles → preference dataset → reward model → PPO. The RM and PPO stages are identical to RLHF; only the *labeling source* differs.

### Modern RLAIF / LLM-as-Judge

Skip the constitution-and-revision step; use a strong LLM directly to score or rank pairs (e.g., GPT-4 or a tuned 70B as judge). Feed the resulting preferences into DPO/ORPO/KTO. This is the dominant practical pattern in 2024-2026 because preference-optimization replaced PPO for cost reasons.

## When to Use

- Human labelers are slow, expensive, or inconsistent.
- You need millions of preferences (production RLHF scale).
- The judging task can be reduced to clear principles or rubrics.
- You're aligning a smaller model and you have a stronger, trusted judge model.
- You want behaviors (refusal style, reasoning format) that are *easier to specify in writing* than to demonstrate.

Skip if: the task requires expert human judgment (medical, legal final calls), the judge is weaker than the model being trained (it'll cap quality at the judge level), or principle violations are subtle and the judge is unreliable.

## Implementation Pattern A: Constitutional Critique-and-Revise

```python
from openai import OpenAI            # any LLM API works
client = OpenAI()

CONSTITUTION = [
    "Choose the response that is most helpful and informative.",
    "Choose the response that is most honest about its uncertainty.",
    "Choose the response that avoids harmful or toxic content.",
    "Choose the response that is concise and well-structured.",
]

def critique_and_revise(prompt, response):
    principle = random.choice(CONSTITUTION)
    critique = client.chat.completions.create(
        model="gpt-4o", messages=[{"role": "user",
        "content": f"Principle: {principle}\nPrompt: {prompt}\nResponse: {response}\n"
                   f"Identify ways the response violates the principle."}]
    ).choices[0].message.content
    revision = client.chat.completions.create(
        model="gpt-4o", messages=[{"role": "user",
        "content": f"Critique: {critique}\nRewrite the response to better satisfy the principle."}]
    ).choices[0].message.content
    return revision

# Build SFT dataset from (prompt, revised) pairs, then SFTTrainer.
```

## Implementation Pattern B: AI-Judge Preference Generation (distilabel)

```python
from distilabel.pipeline import Pipeline
from distilabel.steps.tasks import TextGeneration, UltraFeedback
from distilabel.llms import OpenAILLM

with Pipeline() as pipeline:
    gen_a = TextGeneration(name="cand_a",
        llm=OpenAILLM(model="gpt-4o-mini"), input_mappings={"instruction": "prompt"})
    gen_b = TextGeneration(name="cand_b",
        llm=OpenAILLM(model="claude-3-5-sonnet-latest"), input_mappings={"instruction": "prompt"})
    feedback = UltraFeedback(
        llm=OpenAILLM(model="gpt-4o"),
        aspects=["helpfulness", "honesty", "instruction-following", "truthfulness"],
    )
    [gen_a, gen_b] >> feedback

distiset = pipeline.run(dataset=load_dataset("HuggingFaceH4/no_robots", split="train"))
distiset.push_to_hub("my-rlaif-prefs")
# Now feed into DPOTrainer / KTOTrainer.
```

## Hyperparameter Guidance

- **Judge model strength**: judge should be at least as capable as the policy on the relevant axis. Using GPT-4o to judge a 7B model is fine; using a 7B judge to align a 7B model rarely improves anything.
- **Principle count**: 4-16 is typical. Too many → judge inconsistency; too few → limited behavioral coverage.
- **Pairwise vs scalar scoring**: pairwise (A vs B) is more reliable than asking for absolute 1-10 scores. UltraFeedback uses scalar but with multiple aspects to reduce variance.
- **Judge temperature**: 0.0–0.3 for consistency. 1.0 for diversity sampling experiments only.
- **Position-bias mitigation**: randomize A/B order; many judges have a "first response" bias.
- **Few-shot the judge**: 2-3 worked examples in the judge prompt cut error rate substantially.
- **Dataset size**: 10k-100k preference pairs is usually enough for DPO; CAI's published runs used 100k+.

## Common Pitfalls

- **Self-preference bias.** A judge prefers responses that look like its own. Mitigations: use a different judge family than the model being trained; ensemble judges.
- **Reward hacking the judge.** Models discover stylistic tricks that fool the judge (verbose answers, headers, emojis). Filter for these post-hoc or rotate judges.
- **Inconsistent principles.** Conflicting principles ("be brief" + "be thorough") yield noisy preferences. Order them or sample one per critique.
- **Judge contamination.** If the judge has seen the eval set, its preferences correlate with eval data and you overfit. Use held-out judges for evaluation.
- **Scaling judge cost without scaling quality.** Past ~50k pairs, marginal RLAIF gains diminish; spend budget on a stronger judge instead of more labels.
- **Skipping verification.** Spot-check 100 judge outputs vs human preferences before committing to a 100k-pair run. Judges fail silently.
- **Treating CAI as plug-and-play.** The constitution is a *design artifact*; iterate on it like a prompt.

## When to Use This Mode

Activate when the user wants to scale preference labeling with AI, asks about Constitutional AI, LLM-as-judge, RLAIF, distilabel/UltraFeedback pipelines, or principle-driven alignment.

## Sources

- Constitutional AI paper (Bai et al. 2022): https://arxiv.org/abs/2212.08073
- RLAIF paper (Lee et al. 2023): https://arxiv.org/abs/2309.00267
- UltraFeedback paper (Cui et al. 2023): https://arxiv.org/abs/2310.01377
- distilabel docs: https://distilabel.argilla.io/
- "LLM-as-a-Judge" survey: https://arxiv.org/abs/2411.15594