Raft Optical Flow Eval

---
name: raft-optical-flow-eval
description: Evaluates dense optical flow estimation accuracy and generalization across synthetic and real-world driving scenes. It measures pixel-wise displacement error and outlier rates on clean and final passes of benchmark datasets. Use when the user wants to benchmark on Sintel, KITTI, or asks about evaluating this task. Reports EPE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2003.12039
  bibtex_key: teed2020raft
  confidence: high
---

# raft-optical-flow-eval

> RAFT: Recurrent All-Pairs Field Transforms for Optical Flow — Teed et al. (2020) (arXiv:2003.12039, 2020)

## What this evaluates

Evaluates dense optical flow estimation accuracy and generalization across synthetic and real-world driving scenes. It measures pixel-wise displacement error and outlier rates on clean and final passes of benchmark datasets.

## Datasets

- **Sintel** — total ?; splits: train (-1), test (-1)
- **KITTI** — total ?; splits: train (-1), test (-1)

## Metrics

- `EPE` **(primary)** — range: pixels
  - Average Euclidean distance between predicted and ground truth flow vectors over all valid pixels.
- `F1-epe` — range: percent
  - Percentage of pixels where the end-point error exceeds a threshold (typically 3 pixels).
- `F1-all` — range: percent
  - Percentage of all pixels (including invalid/out-of-bounds) where the end-point error exceeds a threshold.

## Input / output format

**Input**: Pairs of consecutive image frames.

**Output**: Dense flow field of shape (H, W, 2) representing horizontal and vertical pixel displacements.

## Scoring recipe

```python
def compute_epe(pred_flow, gt_flow, valid_mask=None):
    diff = pred_flow - gt_flow
    epe_per_pixel = np.sqrt(diff[..., 0]**2 + diff[..., 1]**2)
    if valid_mask is not None:
        epe_per_pixel = epe_per_pixel[valid_mask]
    return np.mean(epe_per_pixel)
```

## Common pitfalls

- Failing to use the correct number of inference updates (32 for Sintel, 24 for KITTI), which significantly impacts EPE.
- Not distinguishing between Sintel's 'clean' and 'final' passes, which have different noise levels and require separate evaluation.
- Confusing pretraining (FlyingChairs/Things) with finetuning splits, as performance varies drastically based on whether the model is evaluated on cross-dataset generalization or dataset-specific finetuning.

## Evidence (verbatim from paper)

> We evaluate RAFT on Sintel and KITTI. Following previous works, we pretrain our network on FlyingChairs and FlyingThings, followed by dataset specific finetuning. Unless otherwise noted, we evaluate after 32 flow updates on Sintel and 24 on KITTI. Our method achieves an average EPE (end-point-error) of 1.43 on the Sintel(train) clean pass, which is a 29% lower error than FlowNet2.

## Citation

```bibtex
@misc{teed2020raft,
  title={RAFT: Recurrent All-Pairs Field Transforms for Optical Flow},
  author={Teed et al. (2020)},
  year={2020},
  note={arXiv:2003.12039}
}
```

- arXiv: 2003.12039