Retroagent Dual Intrinsic Feedback

---
name: retroagent-dual-intrinsic-feedback
title: "RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback"
version: 0.0.2
engine: skillxiv-v0.0.2-claude-opus-4.6
license: MIT
url: "https://arxiv.org/abs/2603.08561"
keywords: [RL, Agent Learning, Self-Reflection, Intrinsic Feedback, Memory]
description: "Train LLM agents to evolve by generating dual intrinsic feedback signals—numerical rewards for capability progress and language lessons for future reuse. Combines hindsight reflection with memory-augmented RL using SimUtil-UCB retrieval."
---

# Technique: Dual Intrinsic Feedback via Hindsight Self-Reflection

Traditional RL agents rely on sparse binary rewards (task success/failure), limiting learning signals for long-horizon reasoning. RetroAgent augments this with **dual intrinsic feedback**: capability-evolution rewards that measure incremental progress, plus natural-language lessons extracted from trajectories and reused strategically via memory.

This approach shifts optimization from isolated problem-solving toward continuous agent evolution, enabling agents to learn from partial successes and build persistent knowledge.

## Core Concept

RetroAgent generates two complementary feedback signals:

1. **Intrinsic Numerical Feedback**: A scalar reward measuring progress toward subtask completion relative to historical baselines, enabling exploration of partial achievements not yet constituting full success.

2. **Intrinsic Language Feedback**: Natural-language lessons distilled from trajectories and stored in memory, retrieved during subsequent training via a principled strategy balancing relevance, utility, and coverage.

The hindsight mechanism conditions on successful outcomes, then computes how step probability changes given success, effectively performing causal filtering of responsible actions.

## Architecture Overview

- **Reflection module**: Generates capabilities-evolution rewards from trajectory analysis
- **Lesson extraction**: Distills actionable insights from successful and near-successful trajectories
- **Memory buffer**: Persistent storage of lessons with metadata (semantic embedding, utility score, coverage tag)
- **SimUtil-UCB retriever**: Balances semantic similarity, historical utility, and exploration coverage
- **In-context or RL reflection**: Optional variants for higher quality or joint optimization

## Implementation Steps

### Step 1: Compute Capability-Evolution Rewards

Generate numerical feedback based on incremental progress toward task completion.

```python
import torch
from collections import defaultdict

class CapabilityEvolutionReward:
    def __init__(self):
        self.subtask_progress = defaultdict(float)

    def compute_reward(self, trajectory, task_subtasks):
        """
        trajectory: list of (action, state) pairs
        task_subtasks: list of subtask descriptions
        """
        completed_subtasks = set()

        for subtask in task_subtasks:
            # Check if trajectory addresses this subtask
            # (simplified; real implementation uses semantic matching)
            if self.trajectory_addresses_subtask(trajectory, subtask):
                completed_subtasks.add(subtask)

        # Baseline: historical max completion for this task
        historical_max = self.subtask_progress[task_id]
        current_completion = len(completed_subtasks) / len(task_subtasks)

        # Capability-evolution reward: progress relative to history
        intrinsic_reward = max(0, current_completion - historical_max)
        self.subtask_progress[task_id] = max(historical_max, current_completion)

        return intrinsic_reward

    def trajectory_addresses_subtask(self, trajectory, subtask):
        # In practice, use semantic matching with MLLM
        return True  # Placeholder
```

### Step 2: Extract and Store Lessons

Distill natural-language lessons from trajectories and populate memory.

```python
class LessonMemory:
    def __init__(self, capacity=10000, embedding_model=None):
        self.capacity = capacity
        self.lessons = []
        self.embedding_model = embedding_model
        self.utility_scores = []
        self.access_counts = defaultdict(int)

    def add_lesson(self, trajectory, outcome, lesson_text):
        """Store a lesson with metadata."""
        lesson_entry = {
            'text': lesson_text,
            'trajectory': trajectory,
            'outcome': outcome,
            'embedding': self.embedding_model.encode(lesson_text),
            'utility': 1.0,  # Initial utility
            'timestamp': time.time(),
            'coverage_tag': self.compute_coverage_tag(lesson_text)
        }

        if len(self.lessons) >= self.capacity:
            # Evict oldest lesson
            self.lessons.pop(0)
        else:
            self.lessons.append(lesson_entry)

    def compute_coverage_tag(self, lesson_text):
        """Assign coverage tag (e.g., 'arithmetic', 'reasoning')."""
        # Simplified; use semantic clustering in practice
        return 'general'
```

### Step 3: SimUtil-UCB Lesson Retrieval

Balance semantic relevance, historical utility, and exploration coverage.

```python
import numpy as np

def retrieve_lessons_simutil_ucb(memory, query_text, k=3, exploration_bonus=0.1):
    """
    Retrieve lessons using SimUtil-UCB strategy.

    exploration_bonus: UCB weight for under-explored lessons
    """
    query_embedding = memory.embedding_model.encode(query_text)

    scores = []
    for i, lesson in enumerate(memory.lessons):
        # Similarity: cosine distance to query
        similarity = np.dot(
            query_embedding / (np.linalg.norm(query_embedding) + 1e-8),
            lesson['embedding'] / (np.linalg.norm(lesson['embedding']) + 1e-8)
        )

        # Utility: exponential moving average of historical benefit
        utility = lesson['utility']

        # UCB exploration bonus: fewer accesses = higher bonus
        access_count = memory.access_counts[i]
        ucb_bonus = exploration_bonus / (access_count + 1)

        # Combined score
        combined_score = similarity + utility + ucb_bonus
        scores.append(combined_score)

    # Top-k retrieval
    top_indices = np.argsort(scores)[-k:][::-1]
    retrieved_lessons = [memory.lessons[i] for i in top_indices]

    # Update access counts for UCB
    for i in top_indices:
        memory.access_counts[i] += 1

    return retrieved_lessons
```

### Step 4: In-Context Reflection for Higher Quality

Use trajectory pairs to generate lessons via in-context comparison.

```python
def in_context_reflection(model, success_trajectory, near_miss_trajectory):
    """
    Compare trajectories to extract actionable lessons.
    """
    prompt = f"""
    Successful trajectory:
    {format_trajectory(success_trajectory)}

    Near-miss trajectory:
    {format_trajectory(near_miss_trajectory)}

    What key decision or reasoning pattern made the difference?
    Extract a concise lesson (1-2 sentences) that future attempts should follow.
    """

    lesson = model.generate(prompt, max_tokens=100)
    return lesson.strip()
```

## Practical Guidance

**When to Use:**
- Long-horizon reasoning tasks where binary rewards are too sparse
- Multi-step problems requiring persistent knowledge across episodes
- Scenarios where agents benefit from learning from partial successes
- Interactive task synthesis environments

**When NOT to Use:**
- Short-horizon problems with immediate, clear rewards
- Extremely budget-constrained inference (memory overhead)
- Domains where lessons don't transfer across tasks

**Hyperparameter Tuning:**
- **Lesson memory capacity**: 1000-10000 depending on task diversity
- **Exploration bonus weight**: 0.1-1.0 determines coverage emphasis
- **Utility update rate**: Use exponential moving average with α=0.9
- **Lesson extraction frequency**: Every 2-5 successes balances coverage and redundancy

**Common Pitfalls:**
- Lesson memory bloat causing retrieval slowdown (implement TTL or importance-based eviction)
- Overly generic lessons that don't transfer (enforce specificity in extraction)
- Under-exploration of coverage space if UCB bonus too weak
- Insufficient trajectory diversity in early training (warm-up phase helpful)

## Reference

[RetroAgent paper on arXiv](https://arxiv.org/abs/2603.08561)