Vla Rft World Model Reinforcement Robotics

---
name: vla-rft-world-model-reinforcement-robotics
title: "VLA-RFT: Reinforcement Fine-Tuning via World Model Simulation"
version: 0.0.2
engine: skillxiv-v0.0.2-claude-opus-4.6
license: MIT
url: https://arxiv.org/abs/2510.00406
keywords: [VLA, world-models, reinforcement-learning, robotics, efficiency]
description: "Fine-tune Vision-Language-Action models using learned world models as simulators, eliminating costly real-world or physics-simulation RL. Train robust robot policies in 400 steps via GRPO with model-generated verified rewards."
---

# VLA-RFT: Reinforcement Fine-Tuning via World Model Simulation

VLA-RFT addresses the sample inefficiency of robot learning by using a lightweight learned world model (138M parameters) as a simulator to generate verified rewards for policy optimization. This eliminates sim-to-real gaps and real-world cost while enabling rapid fine-tuning.

## Core Architecture

- **Lightweight world model**: 138M-parameter model predicting next visual frames
- **Data-driven simulator**: Uses model predictions to compute rewards without physics engine
- **GRPO-based RL**: Advantage-based optimization on simulated trajectories
- **Verified rewards**: Geometric and kinematic metrics (GIoU, trajectory distance)
- **Rapid convergence**: 400 training steps sufficient for task learning

## Implementation Steps

Train lightweight world model for robotic prediction:

```python
# Build world model from demonstration data
from vla_rft import WorldModel, VLAPolicy, RLTrainer

world_model = WorldModel(
    vision_encoder="clip",
    prediction_heads=["next_frame", "action_feasibility"],
    model_size="lightweight",  # 138M parameters
    prediction_horizon=1  # single-step prediction
)

# Train on demonstration trajectories
world_model.train_on_demonstrations(
    data=robot_demonstrations,  # ~100-200 trajectories
    learning_rate=1e-4,
    num_epochs=10,
    reconstruction_loss="mse",
    action_feasibility_loss="bce"
)
```

Execute VLA reinforcement fine-tuning with world model rewards:

```python
# Setup VLA policy and RL trainer
vla_policy = VLAPolicy(
    base_model="pretrained_vla",  # e.g., Qwen2.5-VL-7B
    action_space="continuous",
    device="cuda"
)

rl_trainer = RLTrainer(
    policy=vla_policy,
    world_model=world_model,
    algorithm="GRPO",
    num_rollouts=4,
    num_training_steps=400
)

# Define verified reward function using world model
def compute_reward(trajectory, target):
    """Compute geometric reward from predicted trajectory"""
    # Use world model to predict visual progression
    predicted_frames = world_model.predict_frames(
        initial_frame=trajectory.initial_image,
        actions=trajectory.actions
    )

    # GIoU reward: grasping quality
    final_predicted_frame = predicted_frames[-1]
    predicted_object_bbox = extract_bbox(final_predicted_frame)
    giou_reward = compute_giou(
        predicted=predicted_object_bbox,
        target=target.bbox
    )

    # Trajectory distance: path efficiency
    predicted_path = extract_trajectory(predicted_frames)
    traj_distance = compute_path_distance(
        actual=predicted_path,
        optimal=target.optimal_path
    )

    # Combined reward
    total_reward = 0.6 * giou_reward + 0.4 * (1 - traj_distance)
    return total_reward

# GRPO training loop
for step in range(num_training_steps):
    # Generate rollouts using current policy
    trajectories = rl_trainer.rollout_batch(
        num_rollouts=4,
        task=current_task
    )

    # Compute rewards using world model predictions
    rewards = []
    for trajectory in trajectories:
        reward = compute_reward(trajectory, current_task.target)
        rewards.append(reward)

    # Update policy with GRPO
    loss = rl_trainer.compute_grpo_loss(
        trajectories=trajectories,
        rewards=rewards
    )

    loss.backward()
    optimizer.step()

    if step % 50 == 0:
        print(f"Step {step}: Avg reward = {np.mean(rewards):.3f}")

print("Fine-tuning complete")
```

## Practical Guidance

**When to use VLA-RFT:**
- Training robot policies when real-world access is expensive
- Tasks where physics simulation doesn't capture important visual details
- Rapid prototyping requiring quick iteration cycles
- Domain adaptation where transfer from simulation to real robots is problematic

**When NOT to use:**
- Scenarios where world model prediction quality is poor
- Very complex long-horizon tasks (single-step prediction may accumulate errors)
- Real-time control where computation overhead unacceptable
- Highly contact-sensitive manipulations (friction, forces)

**Hyperparameters:**
- **World model size (138M)**: Tradeoff between prediction quality and inference speed
- **GRPO num_rollouts (4)**: Standard for vision-based tasks
- **Training steps (400)**: Sufficient for most tabletop tasks; increase to 1000 for complex
- **GIoU weight (0.6)**: Emphasize grasping quality; increase to 0.7 if precision critical
- **Traj distance weight (0.4)**: Balance between accuracy and efficiency
- **Learning rate (1e-4)**: Reduce to 5e-5 for more stable fine-tuning

## Training Data Requirements

- **Demonstration trajectories**: 100-200 expert demonstrations
- **Task specifications**: Object positions, target configurations
- **Compute**: ~1-2 GPU hours for world model pretraining + ~30 min RL fine-tuning

## Benchmark Results

Evaluated on LIBERO manipulation benchmark:
- **In-domain tasks**: 85-92% success rate
- **Out-of-domain visual variants**: 78-85% success
- **Sim-to-real transfer**: 72-80% (modest gap from simulation)
- **Real robot validation**: Tested on two platform types

## Advantages Over Alternatives

vs. Pure Imitation Learning:
- Robust to distribution shift via RL
- Better generalization to unseen object configurations

vs. Physics Simulation (MuJoCo, PyBullet):
- No sim-to-real gap from incorrect physics parameters
- Visual learning transfers directly to real world

vs. Real-World RL:
- 10-100x faster data collection (no real robot hours)
- Safe exploration (no risk of robot damage)
- Parallel training across multiple task variations

## World Model Design Choices

**Single-step prediction**: Prevents accumulation errors; each action generates reward independently
**Vision-only**: Avoids proprioceptive/force sensor requirements; works with monocular RGB
**Lightweight (138M)**: Balance between quality and inference latency

## References

Builds on world models for reinforcement learning and data-driven simulation for robotics.