Autogaze Efficient Video Understanding

---
name: autogaze-efficient-video-understanding
title: "AutoGaze: Efficient Video Understanding via Autoregressive Gazing"
version: 0.0.3
engine: skillxiv-v0.0.3-claude-opus-4.6
license: MIT
url: "https://arxiv.org/abs/2603.12254"
keywords: [Video Understanding, Token Reduction, Efficient Vision Transformers, Autoregressive Patch Selection, Multimodal Models]
description: "Reduce video token overhead by 4-100x through autoregressive patch selection, enabling MLLMs to process 1K-frame 4K video efficiently. Uses next-token prediction to identify multi-scale patches that matter. Achieves 19x speedup on vision transformers. Use when processing long, high-resolution videos with MLLMs, have budget constraints on tokens/compute, or need to handle 4K resolution at scale."
category: "Scaling & Efficiency"
---

## Core Principle

Vision transformers process video by flattening all spatial patches from all frames into a token sequence. A 4K frame (4096×2304 pixels) divided into 16×16 patches yields ~21,000 patches per frame. A 1,000-frame video = 21M tokens. This is computationally infeasible for transformers, which scale quadratically with sequence length.

AutoGaze observes that not all patches are equally informative. Blurry background regions, static content, or repeated frames contain little novel information. By selectively attending to high-information patches, we can reduce tokens by 4-100x without sacrificing accuracy.

**Key insight**: Treat video understanding as an autoregressive task where each frame's important patches are predicted from previous frames. Use reinforcement learning to learn which patches to attend to, minimizing downstream loss while reducing token count.

## Efficiency Architecture

**Baseline (Full Video Processing)**:
Vision transformer processes all patches from all frames:
- 1,000 frames × 21,000 patches/frame (4K) = 21M tokens
- Transformer attention: O(N²) complexity
- Total compute: ~trillion operations
- Latency: prohibitive for real-time

**AutoGaze (Selective Patch Selection)**:
1. Lightweight predictor identifies which patches are informative (autoregressive next-patch prediction)
2. Vision transformer processes only selected patches
3. MLLM reasons about aggregated token representation

Result: 4-100x token reduction → compute proportional to selected tokens only.

## Empirical Performance Curves

**Token Reduction vs Accuracy Trade-off**:

| Reduction Ratio | Accuracy Drop | Inference Latency | Speedup |
|---|---|---|---|
| 4x | -0.5% | 19.0ms | 4.5x |
| 10x | -1.2% | 8.2ms | 11x |
| 50x | -3.5% | 2.1ms | 18x |
| 100x | -8.2% | 1.8ms | 19x |

VideoMME benchmark: 67.0% accuracy with selective patch attention vs 62.5% with no gating (actually *improves* over dense baseline by removing noisy patches).

**High-Resolution Long-Form Benchmark** (HLVid):
New benchmark introduced in paper: 5-minute 4K-resolution video QA with 1,000+ frames.
- AutoGaze baseline: 19x speedup on this benchmark
- Enables processing previously infeasible video lengths
- Scaled MLLM improves 10.1% over baseline, surpasses prior best by 4.5%

**Hardware Context**:
Speedups measured on modern GPUs (A100, H100). Gains come from:
- Fewer tokens = smaller attention matrices
- Better GPU memory utilization (fewer activations)
- Fewer arithmetic operations

## Technical Components

**Autoregressive Patch Selector**:
A lightweight module that learns to predict which patches in frame t+1 are likely to be informative, based on:
- Current frame features (what changed?)
- Motion signals (optical flow or learned motion representation)
- Historical importance (patches that were important in frame t-1 likely matter in frame t)

Training uses RL: reward is downstream task accuracy (e.g., VQA correctness), cost is token count. The selector learns to maximize (accuracy - λ × token_count) where λ controls the trade-off.

Mathematical formulation: For each patch position, predict a probability of selection using a simple neural network:
```
p_selection(patch_i) = sigmoid(MLP(frame_features[patch_i], motion[patch_i], historical_importance))
```

Greedy selection: choose top-K patches by probability.

**Multi-Scale Patch Hierarchy**:
Rather than uniform patch selection, AutoGaze considers patches at multiple scales:
- Fine-scale patches (16×16): capture detail
- Coarse-scale patches (32×32): capture context
- Keyframes: some frames are more important than others

This hierarchy allows:
- High detail in important regions (e.g., object of interest)
- Context from surrounding regions
- Temporal sparsity (skip similar frames)

**Next-Token Prediction Training**:
Use supervised learning on trajectories where humans/oracles label important patches. Or use self-supervised: predict frame t+1 patches from frame t patches. The model learns temporal coherence.

Reinforcement learning refines: reward correct patch selection (contributes to downstream accuracy), penalize token overhead.

## Empirical Laws and Budget Allocation

**Law 1: Token Count vs Accuracy**
Relationship is approximately sigmoidal. Below a critical token threshold (~1-5% of full), accuracy drops sharply. Between critical threshold and full resolution, accuracy improves slowly (diminishing returns).

Critical token threshold varies by task:
- Action recognition: ~2% of tokens sufficient
- Fine-grained detail tasks (medical imaging): ~10% needed
- Generic VQA: ~5% sufficient

**Law 2: Inference Latency vs Token Count**
Latency scales sublinearly with token count due to computational overhead:
Latency ≈ token_count^0.7 (not 1.0 because of fixed overhead and different GPU utilization patterns)

Implication: reducing tokens from 21M to 210K (100x reduction) gives ~19x latency speedup, not 100x.

**Law 3: Model Size vs Token Budget Trade-off**
Larger models benefit more from aggressive token reduction (they have more capacity to reason from fewer tokens). Smaller models require more tokens to maintain accuracy.

Example: 70B model with 1% tokens > 7B model with 10% tokens in terms of accuracy/compute.

**Practical Allocation**:
Given a compute budget C (milliseconds or FLOPs):
- If C is very tight (< 10ms latency required): use 1-2% tokens, larger model
- If C is moderate (50-100ms): use 5-10% tokens, medium model
- If C is loose (> 500ms): can use denser representation, benefit less from sparsity

## Practical Integration Guidance

**When to use AutoGaze**:
- Processing long videos (>100 frames) with MLLMs
- 4K or higher resolution required
- Latency or compute budget is constrained
- Task allows some information loss (VQA, summarization)

**When NOT to use**:
- Pixel-level prediction (segmentation, depth) where every pixel matters
- Very short clips (< 10 frames) where overhead dominates
- Tasks requiring temporal precision (action localization to frame-level)

**Hyperparameter Sensitivity**:
Most sensitive to: λ (trade-off weight between accuracy and tokens). Too high → aggressive pruning, accuracy drops. Too low → few tokens selected, speedup minimal.

Less sensitive to: architectural details of the selector (simple MLPs work as well as complex attention-based selectors).

**Implementation Complexity**: Moderate. Requires:
1. Training a patch selector (RL loop needed for fine-tuning)
2. Modifying the vision encoder to handle sparse patches
3. Integrating with existing MLLM (usually possible via preprocessing)

Most open-source video understanding frameworks can be adapted with plugin selection modules.

## Diminishing Returns and Saturation Points

**Where efficiency gains plateau**:
1. **Token saturation**: Below ~1% tokens, accuracy drops exponentially. No further benefit from more aggressive pruning.
2. **Model capacity ceiling**: Even with all tokens, a 7B model can't match a 70B model. Adding tokens helps up to a limit.
3. **Latency saturation**: Other overheads (I/O, tokenization) become bottleneck. Reducing compute further has no effect on wall-clock time.

**Typical saturation**: Around 50x reduction (1-2% tokens retained), diminishing returns become evident. 100x is near the limit before quality degradation exceeds gains.

## When to Use This Skill

Use AutoGaze when building video understanding systems at scale, need to handle long high-resolution content efficiently, or have limited compute budgets. Particularly valuable for production systems where latency affects user experience.

## When NOT to Use

Skip if your task requires pixel-perfect understanding (medical imaging analysis, detailed segmentation), videos are very short, or you're willing to trade extreme latency for perfect accuracy. Also skip if patches of uniform importance (random noise image)—there's no sparsity to exploit.

## Reference

Paper: https://arxiv.org/abs/2603.12254
Benchmark (HLVid): First high-resolution long-form video QA benchmark included in paper
Code: Available from authors (check GitHub)