Optical Flow Kitti Sintel Eval

---
name: optical-flow-kitti-sintel-eval
description: This evaluation protocol measures the accuracy of predicted optical flow fields against ground truth displacement vectors across varying motion magnitudes and occlusion conditions. It probes a model's ability to handle both small, fine-grained movements and large, robust displacements using standard video sequence benchmarks. Use when the user wants to benchmark on KITTI2012, KITTI2015, MPI-Sintel, or asks about evaluating this task. Reports Out-Noc, EPE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1611.05607
  bibtex_key: schuster2016optical
  confidence: high
---

# optical-flow-kitti-sintel-eval

> Optical Flow Requires Multiple Strategies (but only one network) — Schuster et al. (2016) (arXiv:1611.05607, 2016)

## What this evaluates

This evaluation protocol measures the accuracy of predicted optical flow fields against ground truth displacement vectors across varying motion magnitudes and occlusion conditions. It probes a model's ability to handle both small, fine-grained movements and large, robust displacements using standard video sequence benchmarks.

## Datasets

- **KITTI2012** — total ?; splits: train (-1), test (-1)
- **KITTI2015** — total ?; splits: train (-1), test (-1)
- **MPI-Sintel** — total ?; splits: test (-1)

## Metrics

- `Out-Noc` **(primary)** — range: percent
  - Percentage of pixels with Euclidean error greater than 3 pixels, calculated only over non-occluded pixels.
- `Fl-all` — range: percent
  - Percentage of outlier pixels (Euclidean error greater than 3 pixels) across all image regions.
- `EPE` **(primary)** — range: pixels
  - End-point error averaged over all pixels (or restricted to specific displacement ranges like s0-10 or s40+).
- `Fl` — range: percent
  - Percentage of pixels with an error value larger than 120.

## Input / output format

**Input**: Paired images (or image patches) representing consecutive frames in a video sequence.

**Output**: Dense optical flow field (2D displacement vectors for each pixel or patch).

## Scoring recipe

```python
def compute_metrics(pred_flow, gt_flow, mask=None):
    err = np.sqrt(np.sum((pred_flow - gt_flow)**2, axis=-1))
    outlier_kitti = (err > 3.0).astype(float)
    if mask is not None: outlier_kitti = outlier_kitti[mask]
    out_noc = np.mean(outlier_kitti) * 100
    fl_sintel = np.mean((err > 120).astype(float)) * 100
    epe = np.mean(err)
    return {'Out-Noc': out_noc, 'Fl': fl_sintel, 'EPE': epe}
```

## Common pitfalls

- KITTI's Out-Noc metric explicitly excludes occluded pixels; reporting overall error without masking will overstate performance.
- MPI-Sintel's Fl metric uses a much higher outlier threshold (120 pixels) than KITTI (3 pixels), requiring separate thresholding logic.
- EPE is often reported globally, but the protocol emphasizes analyzing it within specific displacement ranges (e.g., s0-10, s40+) to evaluate multi-strategy behavior.

## Evidence (verbatim from paper)

> Out-Noc is the percentage of pixels with euclidean error $>$ 3 pixels out of the non-occluded pixels. Fl-all is the percentage of outliers (pixels with euclidean error $>$ 3 pixels). The EPE (end-point-error) is averaged over all the pixels and the two right columns contain only the EPE of pixels within the displacement range mentioned in the title. The Fl column presents an evaluation of the the outlier percentage, which, although not provided by this benchmark, was calculated from the error figures presented for each scene that have higher pixel values for larger errors. Fl is the percentage of pixels with a value larger than 120.

## Citation

```bibtex
@misc{schuster2016optical,
  title={Optical Flow Requires Multiple Strategies (but only one network)},
  author={Schuster et al. (2016)},
  year={2016},
  note={arXiv:1611.05607}
}
```

- arXiv: 1611.05607