Vtam Video Tactile Action Models

---
name: vtam-video-tactile-action-models
title: "VTAM: Video-Tactile-Action Models for Contact-Rich Robotic Manipulation"
version: 0.0.3
engine: skillxiv-v0.0.3-claude-opus-4.6
license: MIT
url: "https://arxiv.org/abs/2603.23481"
keywords: [Tactile Sensing, Video-Action Models, Multimodal Learning, Robotic Manipulation, Contact Dynamics]
description: "Replace vision-only Video-Action Models with a video-tactile fusion architecture using tactile regularization loss to prevent visual dominance, improving contact-rich manipulation success from baseline to 90% on complex tasks (80% improvement on high-precision pick-and-place). Effective when robots interact with objects requiring fine-grained force awareness, partial visual observability, or contact state transitions that vision alone cannot capture."
category: "Component Innovation"
---

## What This Skill Does

Augment pretrained video transformers with tactile sensor streams and a cross-modal regularization loss that enforces balanced attention between visual and tactile modalities, enabling robots to handle contact-rich manipulation tasks where vision alone provides incomplete state information.

## The Component Swap

**Old component:** Video-Action Models (VAMs) that encode world state from video tokens alone, missing critical tactile cues about contact forces, pressure, and transition events.

```python
# Traditional VAM: vision-only
class VideoActionModel(nn.Module):
    def __init__(self, video_encoder, action_head):
        super().__init__()
        self.video_encoder = video_encoder

    def forward(self, video_frames, language_instruction):
        # video_frames: [B, T, 3, H, W]
        video_tokens = self.video_encoder(video_frames)  # [B, T*N, D]
        action = self.action_head(video_tokens)
        return action  # Missing tactile information
```

**New component:** Multimodal fusion with explicit tactile stream and regularization to prevent visual latent dominance.

```python
class VTAM(nn.Module):
    def __init__(self, video_encoder, tactile_encoder, action_head, hidden_dim=256):
        super().__init__()
        self.video_encoder = video_encoder      # Pretrained video transformer
        self.tactile_encoder = tactile_encoder  # Lightweight tactile processing
        self.action_head = action_head
        self.hidden_dim = hidden_dim

        # Cross-modal fusion layer
        self.fusion = nn.MultiheadAttention(
            embed_dim=hidden_dim,
            num_heads=8,
            batch_first=True
        )

        # Tactile regularization loss weight
        self.tactile_loss_weight = 0.1

    def forward(self, video_frames, tactile_data, language_instruction):
        """
        video_frames: [B, T, 3, H, W]
        tactile_data: [B, T, 6] - 3 force axes + 3 torque axes per timestep
        language_instruction: [B, L] - tokenized instruction
        """
        # Encode both modalities
        video_tokens = self.video_encoder(video_frames)      # [B, T*N_v, D]
        tactile_tokens = self.tactile_encoder(tactile_data)  # [B, T, D]

        # Expand tactile to match video token count (repeat across spatial tokens)
        # This ensures tactile info is available for every spatial region
        tactile_tokens_expanded = tactile_tokens.unsqueeze(2).expand(
            -1, -1, video_tokens.shape[1] // tactile_data.shape[1], -1
        ).reshape(tactile_tokens.shape[0], -1, self.hidden_dim)

        # Multimodal fusion with cross-attention
        video_query = video_tokens
        tactile_key_value = tactile_tokens_expanded

        fused, attn_weights = self.fusion(
            video_query,
            tactile_key_value,
            tactile_key_value
        )

        # Predict action
        action = self.action_head(fused)

        return action, attn_weights

    def compute_tactile_regularization_loss(self, attn_weights):
        """
        Ensure tactile modality receives sufficient attention weight.
        Prevents visual dominance that would ignore contact information.

        Loss encourages: max(tactile_attn) > threshold
        """
        # attn_weights: [B, T*N_v, T*N_t]
        # Sum attention from each video token to all tactile tokens
        tactile_attention_per_token = attn_weights.sum(dim=-1).mean()

        # Loss: penalize if tactile attention is too low
        # Target: at least 10-20% of total attention budget to tactile
        target_tactile_attn = 0.15
        loss_tactile_reg = (target_tactile_attn - tactile_attention_per_token) ** 2

        return loss_tactile_reg

    def training_loss(self, action_pred, action_true, attn_weights):
        """Total loss combines action prediction and tactile regularization."""
        loss_action = F.mse_loss(action_pred, action_true)
        loss_tactile = self.compute_tactile_regularization_loss(attn_weights)

        total_loss = loss_action + self.tactile_loss_weight * loss_tactile
        return total_loss
```

The key swap is from single-modality (video-only) to dual-stream fusion with a regularization term that prevents the model from ignoring tactile information, even when visual cues are available. This forces the model to learn to interpret contact states that vision cannot represent.

## Performance Impact

**Contact-rich manipulation tasks:**
- Average success rate: **90%** (substantial improvement from baseline VAM)
- Contact state detection: significantly improved

**Potato chip pick-and-place (high-precision, high-force task):**
- Improvement over π0.5 baseline: **80% relative improvement**
- Baseline struggles with fragile object requiring precise force control
- VTAM's tactile feedback enables appropriate gripper modulation

**Task complexity correlation:**
- Simple reach tasks: minimal tactile benefit
- Object placement: moderate benefit
- Contact-sensitive manipulation (chip, egg, deformable): substantial benefit (20-80% improvement)

## When to Use

- Contact-rich manipulation tasks (object insertion, pushing, deformable objects)
- Scenarios where visual observability is partial (gripper occludes object)
- When force control is critical (assembly, delicate object handling)
- Tasks with contact state transitions (initial contact → grasp → lift)
- Robotic systems equipped with tactile sensors (F/T wrist sensor, tactile skin)

## When NOT to Use

- Purely visual tasks where contact is incidental (reaching without contact)
- If tactile sensors are unavailable or unreliable
- On systems with latency constraints (tactile encoding adds compute)
- When tactile data is noisy and unreliable for your sensors
- Very simple tasks where vision alone is sufficient (overhead not justified)

## Implementation Checklist

**1. Tactile encoder setup:**
- Ensure tactile data pipeline: 6-axis F/T sensor (or equivalent) → normalized to [-1, 1]
- Synchronize tactile and video timestamps (often tactile runs at higher frequency; downsample or aggregate)
- Create lightweight tactile encoder: 2-3 layer MLP or 1D conv

```python
class TactileEncoder(nn.Module):
    def __init__(self, input_size=6, hidden_size=256, output_size=256):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(input_size, hidden_size),
            nn.ReLU(),
            nn.Linear(hidden_size, output_size)
        )

    def forward(self, tactile_data):
        # tactile_data: [B, T, 6] → [B, T, output_size]
        return self.net(tactile_data)
```

**2. Architecture integration:**
```python
# Minimal swap: add tactile branch to existing VAM
vtam = VTAM(
    video_encoder=your_pretrained_video_encoder,
    tactile_encoder=TactileEncoder(input_size=6),
    action_head=your_action_head,
    hidden_dim=256
)

# Forward pass now accepts tactile data
action = vtam(video_frames, tactile_data, instruction)
```

**3. Fine-tuning strategy:**
- Freeze pretrained video encoder weights (or use LoRA for light fine-tuning)
- Train tactile encoder and fusion layer on your task
- Typical: 10K-50K steps on task-specific data

**4. Verification:**
- Measure task success rate on contact-sensitive tasks
- Compare: VAM baseline (vision only) vs. VTAM
- Ablate: remove tactile regularization loss to confirm it's necessary
- Check attention weights: verify tactile tokens receive >10% of attention

**5. Hyperparameter tuning if needed:**
- Tactile loss weight: start at 0.1; increase to 0.5 if visual still dominates
- Target tactile attention: 0.15 (15% of budget); adjust based on task criticality
- Tactile encoder hidden size: match video encoder dimension (e.g., 256)
- Learning rate: 1e-4 for frozen video encoder, 1e-3 for trainable encoder

**6. Known issues:**
- Tactile sensor noise: add moving average filter or Kalman filter for smoothing
- Synchronization drift: drift between video and tactile timestamps causes misalignment
- Limited contact data: if contact-rich trajectories are rare, use data augmentation (simulate contacts)
- Generalization: model trained on one robot/sensor may not transfer; validate on your hardware
- Cold start: if tactile info is absent early in episode, model may learn to ignore it; ensure diverse training trajectories

## Related Work

This builds on multimodal fusion architectures (CLIP, BLIP) and extends VAM-style video prediction with sensor fusion patterns. The tactile regularization loss parallels auxiliary task learning (MTL) and relates to balanced contrastive objectives in multimodal transformers. Inspired by human sensorimotor control, where tactile feedback is essential for object manipulation.