Self Rag Expert

---
name: self-rag-expert
description: Self-RAG — model decides when to retrieve and critiques evidence with reflection tokens. Use when building or optimizing retrieval-augmented generation pipelines with self rag.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: rag-advanced
  tags: [rag, self-rag, reflection-tokens, adaptive-retrieval, asai]
---

# Self-RAG Expert Mode

You are an expert in Self-RAG (Asai et al., ICLR 2024, arXiv:2310.11511). Standard RAG retrieves *every time*, then forces the LLM to use the context whether useful or not. Self-RAG trains the model to make four decisions on the fly: **whether to retrieve at all**, **which retrieved passages are relevant**, **whether the generation is supported by them**, and **whether the overall answer is useful**. Each decision is emitted as a special "reflection token" the model learned to predict.

## Core Concept

Four reflection tokens (each multi-class):

| Token | When emitted | Values |
|---|---|---|
| `Retrieve` | At each generation step / segment | `[Yes]`, `[No]`, `[Continue]` |
| `IsRel` | Per retrieved passage | `[Relevant]`, `[Irrelevant]` |
| `IsSup` | Per generated segment vs each passage | `[Fully supported]`, `[Partially supported]`, `[No support]` |
| `IsUse` | At end of answer | `[Utility:1]` … `[Utility:5]` |

Inference algorithm (simplified):

```
for segment in answer:
    emit Retrieve token
    if Retrieve == [Yes]:
        passages = retriever(query)
        for p in passages:
            emit IsRel for p
            if IsRel == [Relevant]:
                emit segment_p (continuation conditioned on p)
                emit IsSup for segment_p vs p
        pick best segment by combined IsRel + IsSup (+ utility) score
    else:
        emit segment without retrieval
emit IsUse for full answer
```

The trained 7B/13B Self-RAG models beat ChatGPT and Llama2-chat baselines on PopQA, TriviaQA, ALCE-ASQA, etc. by avoiding spurious retrievals and self-checking groundedness.

## When It Helps

- **Mixed query corpora** where some queries don't need retrieval (chitchat, calculation, common knowledge) — saves cost and avoids context pollution.
- **Hallucination-sensitive tasks**: medical, legal, factual QA. Reflection tokens give per-segment groundedness signals.
- **Long-form generation**: per-segment retrieval keeps citations tight to claims, not just a one-shot context dump.
- **Custom-trained models**: if you can fine-tune, you get the full Self-RAG benefit.

## When It Hurts

- **You can't fine-tune**: the original Self-RAG approach trains a Llama-2 7B/13B from scratch on a curated dataset with a critic LLM. The "tokens" only mean something to the trained model. Off-the-shelf prompting Self-RAG is approximated, not equivalent.
- **Latency-sensitive**: per-segment retrieval decisions add LLM round trips.
- **Small / focused corpora** where you'd always want to retrieve anyway.
- **You already have a strong agentic RAG stack with verification** — Self-RAG and agentic RAG with verifier loops cover similar ground.

## Implementation Patterns

### Use the released Self-RAG model

```python
# selfrag/selfrag_llama2_7b on HuggingFace
from vllm import LLM, SamplingParams
model = LLM("selfrag/selfrag_llama2_7b", dtype="half")
prompt = (
    "### Instruction:\nWhen did the first iPhone launch?\n\n### Response:\n"
)
out = model.generate(
    [prompt], SamplingParams(temperature=0.0, top_p=1.0, max_tokens=256, skip_special_tokens=False)
)
# Output contains [Retrieve], [Relevant], [Fully supported], [Utility:5] tokens.
# A wrapper parses these and routes retrieval calls.
```

The original repo provides a runner that handles the retrieve-on-token loop and beam scoring across passages.

### Approximate Self-RAG with prompted GPT/Claude

If you can't run the trained model, you can approximate the four checks with explicit prompted steps. This is *not* Self-RAG proper, but captures the spirit:

```python
def adaptive_rag(q, llm, retriever):
    # 1. Decide if retrieval is needed
    need = llm.invoke(
        f"Does answering this require external knowledge? Reply YES/NO.\nQuestion: {q}"
    ).strip().upper()
    if need.startswith("NO"):
        return llm.invoke(q)

    passages = retriever.invoke(q)

    # 2. Filter by relevance
    rel = [p for p in passages
           if "YES" in llm.invoke(f"Is this passage relevant to '{q}'?\n\n{p.text}\n\nYES/NO:").upper()]
    if not rel:
        return "I don't have enough information to answer."

    # 3. Generate with retrieved context
    answer = llm.invoke(f"Answer using only this context:\n\n{format(rel)}\n\nQ: {q}")

    # 4. Self-check support
    grounded = llm.invoke(f"Is every claim in the answer supported by the context? YES/NO.\n\nContext:\n{format(rel)}\n\nAnswer:\n{answer}").upper()
    if "NO" in grounded:
        answer += "\n\n(Note: some claims may not be fully supported by retrieved context.)"
    return answer
```

### LangGraph implementation

LangGraph provides a Self-RAG cookbook that wires up retrieve → grade documents → generate → grade hallucinations → grade answer with explicit state nodes and conditional edges. Useful as a starting graph.

## Eval / Tuning

- **Compare against always-retrieve baseline** on a query mix that includes "doesn't need retrieval" examples. Measure cost reduction, faithfulness lift, latency change.
- **Faithfulness / IsSup precision**: Ragas faithfulness metric on supported claims. Self-RAG should improve this.
- **No-answer / refusal rate**: Self-RAG should refuse more often when context is bad. Calibrate against ground truth.
- **Citation precision**: per-segment passage attribution should improve.
- **Token budget**: every reflection emission and every per-segment beam costs tokens. Measure end-to-end vs vanilla.

## Common Pitfalls

- **Treating prompted Self-RAG as the trained version**: prompted approximations are unreliable; reflection tokens in the trained model carry learned semantics.
- **Threshold tuning of reflection scores**: combined ranking (`IsRel * IsSup * IsUse`) needs weights. Stick close to the paper unless you have eval data.
- **Skipping the grade-passage step in approximated versions**: that's where most of the precision lift lives.
- **Always-retrieve fallback when grader fails**: if the grader can't decide, prefer retrieve — false-negative retrieval is worse than the cost of an extra search.
- **Confusing Self-RAG with CRAG**: CRAG (Yan et al.) corrects with web search fallback when retrieval is bad; Self-RAG decides whether to retrieve and grades evidence inline. Often combined.
- **Stale model**: the released Self-RAG models are Llama-2 based. For modern quality, you may need to retrain on Llama-3/4 or use a CRAG-style critic on top of a strong base model.

## When to Use This Mode

Use Self-RAG (or a Self-RAG-style adaptive pipeline) when:

- Workload mixes retrieval-needed and retrieval-unnecessary queries.
- Hallucination cost is high and you need per-claim grounding signals.
- You can run a fine-tuned model or accept approximated, prompt-driven adaptive RAG.

Skip when:

- Every query needs the corpus (and the savings from skipping retrieval are zero).
- Latency budget can't absorb extra grader calls.
- A simpler hybrid + rerank + verifier already meets your faithfulness target.

## Sources

- Asai et al., "Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection" (ICLR 2024) — https://arxiv.org/abs/2310.11511
- Self-RAG project page — https://selfrag.github.io/
- Self-RAG repo — https://github.com/AkariAsai/self-rag
- Self-RAG model cards — https://huggingface.co/selfrag/selfrag_llama2_7b
- LangGraph Self-RAG cookbook — https://langchain-ai.github.io/langgraph/tutorials/rag/langgraph_self_rag/
- LlamaIndex Self-RAG pack — https://github.com/run-llama/llama_index/tree/main/llama-index-packs/llama-index-packs-self-rag