Tars Token Adaptive Hallucination Reduction

---
name: tars-token-adaptive-hallucination-reduction
title: TARS Min-Max Token-Adaptive Preference Strategy
version: 0.0.2
engine: skillxiv-v0.0.2-claude-opus-4.6
license: MIT
url: https://arxiv.org/abs/2507.21584
keywords: [multimodal-llm, hallucination-reduction, preference-optimization, token-adaptive, robustness]
description: "Token-adaptive preference optimization framework using min-max formulation to reduce multimodal LLM hallucination. Achieves 50% hallucination reduction using min-max distributional robustness while preserving visual grounding."
---

## TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction

TARS addresses the critical problem of hallucination in multimodal large language models through a novel token-level adaptive preference optimization approach. Rather than treating preference signals as fixed targets, TARS uses min-max optimization to train models that are robust to preference label uncertainty while maintaining visual grounding.

### Core Concept

The key insight is that hallucinations arise when language models overfit to spurious patterns in preference data, disconnecting outputs from actual visual content. TARS reformulates preference optimization as a min-max problem:

- **Outer maximization**: Shift the model's token distribution to satisfy preference constraints
- **Inner minimization**: Perturb preferences within a semantic budget to simulate uncertainty
- **Equilibrium**: Train model to be robust against preference variations that don't compromise visual grounding

This approach reduces hallucinations from 26.4% to 13.2% using only 4.8k preference samples, achieving state-of-the-art performance.

### Architecture Overview

The framework consists of:

- **Preference Encoding**: Represent human preferences as token-level target distributions
- **Semantic Constraint Layer**: Define allowed perturbations without changing meaning
- **Min-Max Optimizer**: Outer loop for distribution shift, inner loop for robustness
- **Visual Grounding Preservation**: Ensure steering doesn't disconnect from image content
- **Token-Level Adaptation**: Apply different strengths of preference steering per token

### Implementation Steps

**Step 1: Encode preferences as token-level distributions**

Convert pairwise preference labels into token-level target distributions:

```python
import torch
import torch.nn.functional as F
from typing import List, Tuple, Dict

class PreferenceEncoder:
    """Encodes human preferences as token-level distributions"""

    def __init__(self, tokenizer, vocabulary_size: int):
        self.tokenizer = tokenizer
        vocab_size = vocabulary_size

    def encode_preference_pair(self, preferred_text: str,
                               dispreferred_text: str) -> Tuple[torch.Tensor, torch.Tensor]:
        """
        Convert preferred vs dispreferred outputs to target distributions.

        Args:
            preferred_text: The preferred response
            dispreferred_text: The dispreferred response

        Returns:
            (preferred_dist, dispreferred_dist) of shape (seq_len, vocab_size)
        """
        preferred_tokens = self.tokenizer.encode(preferred_text)
        dispreferred_tokens = self.tokenizer.encode(dispreferred_text)

        # Create one-hot distributions for each sequence
        max_len = max(len(preferred_tokens), len(dispreferred_tokens))
        preferred_dist = torch.zeros(max_len, self.vocab_size)
        dispreferred_dist = torch.zeros(max_len, self.vocab_size)

        for i, token_id in enumerate(preferred_tokens):
            preferred_dist[i, token_id] = 1.0

        for i, token_id in enumerate(dispreferred_tokens):
            dispreferred_dist[i, token_id] = 1.0

        return preferred_dist, dispreferred_dist

    def create_preference_labels(self, pairs: List[Tuple[str, str]]) -> torch.Tensor:
        """
        Create batch of preference target distributions.

        Args:
            pairs: List of (preferred, dispreferred) text pairs

        Returns:
            Tensor of shape (batch_size, seq_len, vocab_size)
        """
        all_prefs = []
        for preferred, dispreferred in pairs:
            pref_dist, _ = self.encode_preference_pair(preferred, dispreferred)
            all_prefs.append(pref_dist)

        # Pad to same length
        max_len = max(p.shape[0] for p in all_prefs)
        batch_prefs = torch.zeros(len(all_prefs), max_len, self.vocab_size)

        for i, pref in enumerate(all_prefs):
            batch_prefs[i, :pref.shape[0], :] = pref

        return batch_prefs
```

This represents preferences as target probability distributions over tokens.

**Step 2: Implement min-max optimization with perturbation**

The core training loop applies min-max optimization to achieve robustness:

```python
class MinMaxPreferenceOptimizer:
    """Min-max optimization for robust preference learning"""

    def __init__(self, model, tokenizer, epsilon: float = 0.1):
        self.model = model
        self.tokenizer = tokenizer
        self.epsilon = epsilon  # Perturbation budget
        self.vocab_size = model.config.vocab_size

    def compute_preference_loss(self, logits: torch.Tensor,
                               target_dist: torch.Tensor,
                               positions: torch.Tensor) -> torch.Tensor:
        """
        Compute loss between predicted logits and preference targets.

        Args:
            logits: Model output logits, shape (batch, seq_len, vocab_size)
            target_dist: Target probability distribution over tokens
            positions: Which positions correspond to preference targets

        Returns:
            Scalar loss value
        """
        # Extract logits at preference positions
        pred_probs = F.softmax(logits, dim=-1)

        # KL divergence from model distribution to target
        loss = F.kl_div(
            F.log_softmax(logits, dim=-1),
            target_dist.clamp(min=1e-8),
            reduction='batchmean'
        )

        return loss

    def perturbation_step(self, target_dist: torch.Tensor,
                         step_size: float = 0.01) -> torch.Tensor:
        """
        Inner optimization: find worst-case perturbation within semantic budget.

        Args:
            target_dist: Current preference target
            step_size: Optimization step size

        Returns:
            Perturbed distribution that remains semantically valid
        """
        perturbed = target_dist.clone().detach().requires_grad_(True)
        optimizer = torch.optim.Adam([perturbed], lr=step_size)

        for _ in range(5):  # Inner loop iterations
            # Compute model loss on perturbed targets
            logits = self.model(...)  # Forward pass
            loss = self.compute_preference_loss(logits, perturbed, mask=None)

            # Maximize loss (adversarial): find worst perturbation
            loss_adv = -loss

            optimizer.zero_grad()
            loss_adv.backward()
            optimizer.step()

            # Project back to semantic constraint set
            perturbed.data = self._project_to_semantic_budget(perturbed.data)

        return perturbed.detach()

    def _project_to_semantic_budget(self, dist: torch.Tensor) -> torch.Tensor:
        """
        Project distribution back to semantic constraint set.
        Keep similar tokens with higher weight.
        """
        # Reproject to valid probability distribution
        dist = F.softmax(dist, dim=-1)

        # Constraint: perturbed distribution cannot exceed epsilon away from original
        # (in terms of TV divergence or other semantic distance metric)
        # Simplified: clamp large changes

        return dist

    def min_max_training_step(self, images: torch.Tensor,
                             prompts: torch.Tensor,
                             preference_targets: torch.Tensor) -> Tuple[torch.Tensor, float]:
        """
        Single min-max training step: maximize robustness to preference perturbations.

        Args:
            images: Visual input
            prompts: Text prompts
            preference_targets: Target distributions for next tokens

        Returns:
            (updated_model_state, loss_value)
        """
        # Outer loop: optimize model parameters
        optimizer = torch.optim.AdamW(self.model.parameters(), lr=1e-5)

        for outer_step in range(3):  # Outer iterations

            # Inner loop: find worst perturbation
            perturbed_targets = self.perturbation_step(preference_targets)

            # Forward pass with model
            outputs = self.model(images, prompts)
            logits = outputs.logits

            # Compute loss on perturbed targets
            loss = self.compute_preference_loss(logits, perturbed_targets, positions=None)

            # Outer optimization: minimize loss on worst-case perturbation
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

        return self.model.state_dict(), loss.item()
```

This implements the core min-max algorithm that creates robustness to preference uncertainty.

**Step 3: Implement token-level adaptive steering**

Apply different preference strengths to different tokens:

```python
class TokenAdaptivePreference:
    """Adaptively applies preference steering per token"""

    def __init__(self, model):
        self.model = model

    def compute_token_importance(self, logits: torch.Tensor,
                                target_dist: torch.Tensor) -> torch.Tensor:
        """
        Compute importance weight for each token based on preference strength.

        Args:
            logits: Model predictions
            target_dist: Preference targets

        Returns:
            Importance weights of shape (batch, seq_len)
        """
        pred_probs = F.softmax(logits, dim=-1)

        # Importance = entropy difference (high-entropy targets need stronger steering)
        target_entropy = -(target_dist * torch.log(target_dist + 1e-8)).sum(dim=-1)
        pred_entropy = -(pred_probs * torch.log(pred_probs + 1e-8)).sum(dim=-1)

        importance = (target_entropy - pred_entropy).clamp(min=0)

        # Normalize to [0, 1]
        importance = importance / (importance.max() + 1e-8)

        return importance

    def apply_adaptive_preference(self, logits: torch.Tensor,
                                 target_dist: torch.Tensor,
                                 base_strength: float = 1.0) -> torch.Tensor:
        """
        Shift logits toward preference targets with token-adaptive strength.

        Args:
            logits: Model logits
            target_dist: Preference targets
            base_strength: Base preference steering strength

        Returns:
            Modified logits
        """
        # Compute per-token importance weights
        importance = self.compute_token_importance(logits, target_dist)

        # Compute preference direction
        pred_probs = F.softmax(logits, dim=-1)
        pref_direction = target_dist - pred_probs  # Shape: (batch, seq_len, vocab)

        # Apply adaptive steering
        batch_size, seq_len, vocab_size = logits.shape

        for b in range(batch_size):
            for t in range(seq_len):
                # Scale steering by token importance
                strength = base_strength * importance[b, t].item()

                # Move logits toward preference targets
                logits[b, t, :] = logits[b, t, :] + strength * pref_direction[b, t, :]

        return logits
```

This enables the system to focus preference steering on critical tokens.

**Step 4: Preserve visual grounding during steering**

Ensure that preference steering doesn't disconnect from image content:

```python
class VisualGroundingPreserver:
    """Ensures preference optimization preserves visual grounding"""

    def __init__(self, model, vision_encoder):
        self.model = model
        self.vision_encoder = vision_encoder

    def compute_visual_consistency(self, image: torch.Tensor,
                                  output_tokens: torch.Tensor) -> float:
        """
        Measure how well output is grounded in visual content.

        Compute correlation between image features and output token embeddings.
        """
        # Extract visual features
        image_features = self.vision_encoder(image)  # (image_dim,)

        # Extract output token embeddings
        token_embeddings = self.model.get_token_embeddings(output_tokens)  # (seq_len, emb_dim)

        # Compute attention-weighted consistency
        batch_size, seq_len, vocab_size = output_tokens.shape
        similarities = []

        for t in range(seq_len):
            # Similarity between this token's embedding and image features
            token_emb = token_embeddings[t]
            similarity = F.cosine_similarity(
                image_features.unsqueeze(0),
                token_emb.unsqueeze(0)
            )
            similarities.append(similarity)

        # Average similarity across tokens
        visual_consistency = torch.mean(torch.stack(similarities))

        return visual_consistency.item()

    def grounding_aware_loss(self, logits: torch.Tensor,
                            target_dist: torch.Tensor,
                            image: torch.Tensor,
                            lambda_grounding: float = 0.1) -> torch.Tensor:
        """
        Combine preference loss with visual grounding preservation.

        Args:
            logits: Model predictions
            target_dist: Preference targets
            image: Visual input
            lambda_grounding: Weight for grounding constraint

        Returns:
            Total loss that preserves grounding
        """
        # Preference alignment loss
        preference_loss = F.kl_div(
            F.log_softmax(logits, dim=-1),
            target_dist,
            reduction='batchmean'
        )

        # Visual grounding penalty
        predicted_tokens = torch.argmax(logits, dim=-1)
        grounding_score = self.compute_visual_consistency(image, predicted_tokens)

        # Total loss: preference optimization + grounding preservation
        total_loss = preference_loss - lambda_grounding * grounding_score

        return total_loss
```

This ensures hallucination reduction doesn't sacrifice visual fidelity.

**Step 5: Training loop with all components**

Integrate all components into an end-to-end training procedure:

```python
class TARSTrainer:
    """Complete TARS training with preference, perturbation, and grounding"""

    def __init__(self, model, vision_encoder, tokenizer,
                 epsilon: float = 0.1, lambda_grounding: float = 0.1):
        self.model = model
        self.vision_encoder = vision_encoder
        self.pref_encoder = PreferenceEncoder(tokenizer, model.config.vocab_size)
        self.optimizer = MinMaxPreferenceOptimizer(model, tokenizer, epsilon)
        self.token_adaptive = TokenAdaptivePreference(model)
        self.grounding = VisualGroundingPreserver(model, vision_encoder)
        self.lambda_grounding = lambda_grounding

    def train_step(self, batch: Dict) -> float:
        """
        Single TARS training step.

        Args:
            batch: Contains 'images', 'prompts', 'preferred_text', 'dispreferred_text'

        Returns:
            Loss value
        """
        images = batch['images']
        prompts = batch['prompts']
        preferred = batch['preferred_text']
        dispreferred = batch['dispreferred_text']

        # Encode preferences as distributions
        pref_targets = self.pref_encoder.create_preference_labels(
            list(zip(preferred, dispreferred))
        )

        # Forward pass
        outputs = self.model(images, prompts)
        logits = outputs.logits

        # Apply token-adaptive preference steering
        adapted_logits = self.token_adaptive.apply_adaptive_preference(
            logits, pref_targets, base_strength=1.0
        )

        # Compute loss with visual grounding preservation
        loss = self.grounding.grounding_aware_loss(
            adapted_logits, pref_targets, images,
            lambda_grounding=self.lambda_grounding
        )

        # Min-max optimization step
        self.model.train()
        loss.backward()

        torch.nn.utils.clip_grad_norm_(self.model.parameters(), 1.0)

        optimizer = torch.optim.AdamW(self.model.parameters(), lr=1e-5)
        optimizer.step()
        optimizer.zero_grad()

        return loss.item()

    def train_epoch(self, dataloader, num_epochs: int = 3):
        """Train for multiple epochs"""
        for epoch in range(num_epochs):
            total_loss = 0.0
            for batch_idx, batch in enumerate(dataloader):
                loss = self.train_step(batch)
                total_loss += loss

                if batch_idx % 100 == 0:
                    print(f"Epoch {epoch}, Batch {batch_idx}, Loss: {loss:.4f}")

            avg_loss = total_loss / len(dataloader)
            print(f"Epoch {epoch} average loss: {avg_loss:.4f}")
```

This trains models with robust preference optimization while preserving visual grounding.

### Practical Guidance

**When to use TARS:**
- Reducing hallucinations in multimodal language models
- Fine-tuning with limited preference data (4-10k examples)
- When visual grounding must be preserved
- Training data with some label noise or disagreement
- Production systems requiring reliable image-to-text alignment

**When NOT to use TARS:**
- Pure language models without vision (use standard DPO instead)
- Systems with extremely high-quality, consistent preference labels
- Real-time training on streaming data (min-max optimization adds overhead)
- When hallucination is acceptable tradeoff for other capabilities

**Key hyperparameters:**

- `epsilon`: Perturbation budget (0.05-0.15 typical; larger = more robust but slower)
- `lambda_grounding`: Grounding preservation weight (0.05-0.2; higher = more visual fidelity)
- `base_strength`: Preference steering magnitude (0.5-2.0)
- `inner_steps`: Perturbation loop iterations (3-7 typical)
- `outer_steps`: Model parameter update iterations (2-4 typical)

**Expected improvements:**

- Hallucination reduction: 40-50% typical improvement
- Visual accuracy: Maintained at >95% with proper lambda_grounding
- Training efficiency: ~2-3x fewer preference examples needed vs DPO
- Training time: Add ~30-40% overhead due to min-max loop

**Dataset requirements:**
- Preference pairs with hallucinations (dispreferred) vs accurate (preferred)
- Minimum 4.8k pairs; performance scales up to ~20k
- Good coverage of different hallucination types

### Reference

TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction. arXiv:2507.21584