Reasoning Palette

---
name: reasoning-palette
title: "Reasoning Palette: Modulating Reasoning via Latent Contextualization"
version: 0.0.2
engine: skillxiv-v0.0.2-claude-opus-4.6
license: MIT
url: https://arxiv.org/abs/2512.17206
keywords: [reinforcement-learning, reasoning, exploration, latent-space, vae]
description: "Overcome token-level randomness limitations in RL by shifting exploration to latent reasoning strategies. Train a VAE encoding diverse reasoning patterns, sample latents during RL, decode to prefix embeddings steering internal reasoning—enabling structured exploration across math, coding, and QA with interpretable, controllable behavior."
---

## Overview

Reasoning Palette addresses a fundamental inefficiency in LLM reinforcement learning: standard sampling schemes create limited trajectory diversity because token-level randomness doesn't generate sufficiently different reasoning approaches. This framework shifts exploration to a learned latent space of reasoning strategies.

## Core Technique

The key insight is that reasoning diversity should come from high-level strategy changes, not token-level noise.

**VAE-Based Latent Strategy Space:**
Learn a continuous space encoding diverse reasoning patterns from multi-domain data.

```python
# VAE for reasoning strategy encoding
class ReasoningStrategyVAE:
    def __init__(self, latent_dim=16):
        self.encoder = Encoder()  # Question-answer → latent
        self.decoder = Decoder()  # Latent → prefix embedding
        self.latent_dim = latent_dim

    def encode_strategies(self, qa_pairs):
        """
        Encode diverse reasoning patterns from mathematical,
        coding, and QA domains into shared latent space.
        """
        strategies = []
        for question, answer in qa_pairs:
            # Embed both question and answer
            q_embed = embedding_model(question)
            a_embed = embedding_model(answer)

            # Mean-pool to get strategy representation
            strategy_rep = (q_embed + a_embed) / 2

            # Encode into latent space
            latent = self.encoder(strategy_rep)
            strategies.append(latent)

        return torch.stack(strategies)

    def sample_and_decode(self):
        """
        Sample from Gaussian latent space and decode to
        prefix embeddings that steer reasoning.
        """
        # Sample from standard normal
        z = torch.randn(1, self.latent_dim)

        # Decode to prefix embedding
        prefix_embedding = self.decoder(z)

        return prefix_embedding
```

**Prefix Conditioning for Reasoning Steering:**
Decoded latent vectors become learnable token prefixes prepended to prompts.

```python
def apply_strategy_prefix(model, question, latent_sample):
    """
    Prepend latent-decoded prefix to question.
    Prefix steers the model's internal reasoning before generation.
    """
    # Decode latent to prefix embedding
    prefix_embedding = vae_decoder(latent_sample)

    # Prepend to question tokens
    prompt_with_prefix = [prefix_embedding] + tokenize(question)

    # Forward pass (prefix modulates reasoning trajectory)
    output = model.generate(prompt_with_prefix)

    return output
```

**Scheduled Integration in RL Training:**
During RL, latent-guided prefixes enable structured exploration that transitions from exploration to exploitation.

```python
def rl_training_with_latent_guidance(model, dataset, epochs=10):
    """
    RL training with scheduled latent-prefix guidance.
    Early epochs: wide exploration via diverse latents
    Late epochs: exploit best strategies
    """
    for epoch in range(epochs):
        # Compute exploration schedule
        exploration_factor = 1.0 - (epoch / epochs)  # Decrease over time

        for batch in dataset:
            # Sample latent strategy
            if random.random() < exploration_factor:
                # Exploration phase: sample diverse latents
                latent = vae.sample_latent()
            else:
                # Exploitation phase: use best latent
                latent = best_latent_so_far

            # Generate with strategy prefix
            prefix = vae_decoder(latent)
            trajectory = model.generate_with_prefix(batch, prefix)

            # RL update
            reward = compute_reward(trajectory)
            model.update_via_grpo(trajectory, reward)

            # Optional: update best latent
            if reward > best_reward:
                best_latent_so_far = latent
                best_reward = reward
```

## When to Use This Technique

Use Reasoning Palette when:
- RL training for reasoning tasks shows poor trajectory diversity
- Token-level randomness creates insufficient strategy variation
- Multi-domain reasoning (math, coding, QA) training
- Interpretability of reasoning strategies is valuable

## When NOT to Use This Technique

Avoid this approach if:
- Single-domain or narrow reasoning required (simpler methods suffice)
- Computational overhead of VAE training unacceptable
- Token-level randomness adequately covers strategy space
- Real-time online learning requires immediate strategy adaptation

## Implementation Notes

The framework requires:
- VAE training on diverse question-answer pairs
- Integration of prefix embeddings into model architecture
- RL training loop with scheduled exploration
- Supervised fine-tuning warm-up to sensitize model to prefixes

## Key Performance

- Consistent improvements on mathematical reasoning benchmarks
- More diverse and explorative trajectories than token-level sampling
- Interpretable strategy representations
- Controllable reasoning behavior

## References

- VAE-based latent space for reasoning strategies
- Prefix conditioning for internal reasoning modulation
- Scheduled RL integration for exploration-exploitation
- Multi-domain reasoning pattern learning