Test Time Rl Vlm

---
name: test-time-rl-vlm
title: "Test-Time Reinforcement Learning for Vision Language Models: TTRV"
version: 0.0.2
engine: skillxiv-v0.0.2-claude-opus-4.6
license: MIT
url: https://arxiv.org/abs/2510.06783
keywords: [test-time-adaptation, reinforcement-learning, vision-language, self-improvement, unlabeled-data]
description: "Adapt vision-language models at inference without labeled data by generating multiple predictions and rewarding high-frequency outputs. Trigger: improve VLM accuracy on deployment with self-generated supervision signals."
---

# Test-Time Reinforcement Learning for Vision Language Models (TTRV)

## Core Concept

TTRV enables VLMs to **self-improve during inference** without any labeled training data. By generating multiple predictions on the same test sample and using frequency-based rewards, the model learns to refine its outputs in real-time. This achieves up to 52.4% accuracy gains on object recognition and matches GPT-4V performance without retraining.

The key insight: When a model produces the same answer multiple times under different sampling conditions, that consensus is a reliable learning signal.

## Architecture Overview

- **Multiple Sampling**: Generate k predictions per test sample with stochastic decoding
- **Frequency-Based Rewards**: Reward outputs that appear in multiple samples (consensus)
- **Entropy Regularization**: Penalize overly uncertain distributions to maintain quality
- **GRPO Adaptation**: Apply Group Relative Policy Optimization at test time
- **No Labeled Data Required**: Uses model's own outputs as supervision

## Implementation Steps

### 1. Design Frequency-Based Reward Signal

Create a reward function that identifies consensus predictions. Multiple similar outputs suggest higher confidence in that answer.

```python
def frequency_based_reward(predictions, temperature=1.0):
    """
    Compute reward based on prediction frequency across samples.

    Args:
        predictions: List of strings from k independent generations
        temperature: Sharpness of reward (higher = more selective)

    Returns:
        List of rewards, one per prediction
    """
    # Tokenize predictions for similarity matching
    tokens = [set(p.lower().split()) for p in predictions]

    rewards = []
    for i, pred in enumerate(predictions):
        # Count how many other predictions are similar
        # (Jaccard similarity > threshold)
        similarity_threshold = 0.7
        similar_count = 0

        for j, other in enumerate(predictions):
            if i != j:
                intersection = len(tokens[i] & tokens[j])
                union = len(tokens[i] | tokens[j])
                jaccard = intersection / (union + 1e-8)

                if jaccard > similarity_threshold:
                    similar_count += 1

        # Reward proportional to consensus
        base_reward = similar_count / (len(predictions) - 1)

        # Temperature scaling
        reward = base_reward ** (1 / temperature)
        rewards.append(reward)

    return torch.tensor(rewards, dtype=torch.float32)
```

### 2. Implement Entropy Regularization

Balance exploration with quality control. Overly diverse outputs waste compute; overly confident ones miss refinement opportunities.

```python
def entropy_regularized_reward(predictions, logits, alpha=0.1):
    """
    Combine frequency reward with entropy penalty.

    Args:
        predictions: List of decoded outputs
        logits: Model's logits for each prediction
        alpha: Weight of entropy regularization (0.05-0.15 typical)

    Returns:
        Regularized rewards
    """
    # Base frequency reward
    freq_rewards = frequency_based_reward(predictions)

    # Entropy of output distribution
    probs = torch.softmax(logits, dim=-1)
    entropy = -torch.sum(probs * torch.log(probs + 1e-8), dim=-1)

    # Normalize entropy to [0, 1]
    entropy_norm = entropy / torch.log(torch.tensor(logits.shape[-1], dtype=torch.float32))

    # Penalize high entropy (encourages consensus)
    # but don't penalize zero entropy (overfitting)
    entropy_penalty = torch.clamp(entropy_norm - 0.3, min=0)

    regularized_rewards = freq_rewards - alpha * entropy_penalty

    return regularized_rewards
```

### 3. Collect Test-Time Rollouts

At inference on a test sample, generate multiple predictions and collect trajectories for RL.

```python
class TestTimeRolloutCollector:
    def __init__(self, model, num_samples=4):
        self.model = model
        self.num_samples = num_samples

    def collect_rollout(self, image, task_type="object_recognition"):
        """
        Generate k predictions and compute rewards.

        Args:
            image: Input image tensor
            task_type: "object_recognition", "vqa", etc.

        Returns:
            Trajectory with predictions, logits, and rewards
        """
        predictions = []
        logits_list = []
        log_probs_list = []

        # Generate multiple samples
        for _ in range(self.num_samples):
            # Use stochastic decoding (temperature > 0)
            output = self.model.generate(
                image,
                task_type=task_type,
                max_new_tokens=50,
                temperature=0.8,  # Encourage diversity
                return_logits=True
            )

            predictions.append(output["text"])
            logits_list.append(output["logits"])
            log_probs_list.append(output["log_prob"])

        # Stack tensors
        logits = torch.stack(logits_list)  # (k, vocab_size)

        # Compute frequency and entropy rewards
        rewards = entropy_regularized_reward(
            predictions,
            logits,
            alpha=0.1
        )

        return {
            "predictions": predictions,
            "logits": logits,
            "log_probs": log_probs_list,
            "rewards": rewards,
            "num_samples": self.num_samples
        }
```

### 4. Apply GRPO at Test Time

Use Group Relative Policy Optimization to refine the model's policy based on test-time rewards. This is lightweight enough for deployment.

```python
def grpo_update_at_test_time(model, rollout, lr=0.01):
    """
    Apply one step of GRPO using test-time rollout.

    Args:
        model: VLM to adapt
        rollout: Output from collect_rollout()
        lr: Learning rate for this update

    Returns:
        Updated model, loss value
    """
    rewards = rollout["rewards"]
    log_probs = torch.stack(rollout["log_probs"])

    # Group relative advantage
    baseline = torch.mean(rewards)
    advantages = rewards - baseline

    # Normalize advantages for stability
    advantages = (advantages - torch.mean(advantages)) / (torch.std(advantages) + 1e-8)

    # Policy gradient loss
    policy_loss = -(advantages * log_probs).mean()

    # Entropy bonus (prevent mode collapse)
    entropy = -torch.mean(torch.sum(
        torch.exp(log_probs) * log_probs,
        dim=-1
    ))

    total_loss = policy_loss - 0.01 * entropy

    # Single gradient step (lightweight for inference)
    total_loss.backward()

    with torch.no_grad():
        # Manual parameter update (in-place)
        for param in model.parameters():
            if param.grad is not None:
                param.data -= lr * param.grad
                param.grad.zero_()

    return model, total_loss.item()
```

### 5. Full Inference Loop with Adaptation

Orchestrate the test-time adaptation pipeline.

```python
class AdaptiveVLMInference:
    def __init__(self, model, num_adaptation_steps=3):
        self.model = model
        self.num_adaptation_steps = num_adaptation_steps
        self.collector = TestTimeRolloutCollector(model)

    def infer_with_adaptation(self, image, task_type="object_recognition"):
        """
        Perform inference with test-time RL adaptation.

        Args:
            image: Input image
            task_type: Task specification

        Returns:
            Final prediction after adaptation
        """
        # Step 1: Initial prediction
        initial = self.model.generate(image, task_type=task_type)
        print(f"Initial prediction: {initial['text']}")

        # Step 2: Iterative adaptation
        for step in range(self.num_adaptation_steps):
            # Collect rollout
            rollout = self.collector.collect_rollout(image, task_type)

            # Apply GRPO update
            self.model, loss = grpo_update_at_test_time(
                self.model,
                rollout,
                lr=0.01 * (0.9 ** step)  # Decay learning rate
            )

            # Check consensus
            most_common = max(
                set(rollout["predictions"]),
                key=rollout["predictions"].count
            )
            consensus_score = rollout["predictions"].count(most_common) / len(rollout["predictions"])

            print(f"Step {step+1}: consensus={consensus_score:.2f}, loss={loss:.4f}")

            # Early stopping if high confidence
            if consensus_score > 0.9:
                break

        # Step 3: Final prediction with adapted model
        final = self.model.generate(image, task_type=task_type, temperature=0.0)
        return final["text"]
```

## Practical Guidance

**Hyperparameters:**
- **Num samples per test image**: 4-8 (balance quality vs. latency)
- **Temperature for generation**: 0.8 (encourage diversity for reward signal)
- **Entropy regularization weight**: 0.05-0.15
- **Test-time learning rate**: 0.01 per step, decayed by 0.9x
- **Adaptation steps**: 3-5 (diminishing returns after)

**When to Use:**
- VLM deployment where labeled data unavailable
- Real-time improvement needed on specific domains
- Low-latency requirements (TTRV adds minimal overhead)
- Task types: object recognition, VQA, visual reasoning

**When NOT to Use:**
- Tasks where model outputs highly variable (low consensus)
- Streaming applications where image stays in view briefly
- Tasks requiring labeled ground truth for supervision
- Settings where model must be frozen (e.g., compliance)

## Reference

[TTRV: Test-Time Reinforcement Learning for Vision Language Models](https://arxiv.org/abs/2510.06783) — arXiv:2510.06783