Kitti Fc Eval

---
name: kitti-fc-eval
description: Evaluates the robustness of optical flow estimation models when subjected to various digital, illumination, weather, noise, and blur corruptions. It measures both absolute performance degradation and relative robustness compared to clean data across in-domain and out-of-domain training settings. Use when the user wants to benchmark on KITTI-FC, or asks about evaluating this task. Reports EPE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.14865
  bibtex_key: [REDACTED]
  confidence: high
---

# kitti-fc-eval

> Benchmarking the Robustness of Optical Flow Estimation to Corruptions — Yi et al. (2024) (arXiv:2411.14865, 2024)

## What this evaluates

Evaluates the robustness of optical flow estimation models when subjected to various digital, illumination, weather, noise, and blur corruptions. It measures both absolute performance degradation and relative robustness compared to clean data across in-domain and out-of-domain training settings.

## Datasets

- **KITTI-FC** — total ?; splits: train (-1), test (-1); repo https://github.com/ZhonghuaYi/optical_flow_robustness_benchmark

## Metrics

- `EPE` **(primary)** — range: pixels
  - End-Point Error: the mean Euclidean distance between predicted and ground-truth flow vectors across all valid pixels.
- `CRE` — range: pixels
  - Corruption Robustness Error: quantifies absolute robustness by measuring the increase in EPE under corruption compared to clean data.
- `CREr` — range: ratio or percent
  - Relative CRE: quantifies relative robustness drop, typically calculated as the ratio or percentage change in EPE degradation between evaluation settings (e.g., OOD vs ID).

## Input / output format

**Input**: Paired image frames (or single frame with applied corruption) for optical flow estimation.

**Output**: 2D optical flow field (horizontal and vertical displacement vectors per pixel).

## Scoring recipe

```python
def compute_epe(pred_flow, gt_flow):
    return np.mean(np.sqrt(np.sum((pred_flow - gt_flow)**2, axis=-1)))

def compute_cre(epe_clean, epe_corrupted):
    return epe_corrupted - epe_clean

def compute_crer(epe_clean_id, epe_corrupted_id, epe_clean_ood, epe_corrupted_ood):
    deg_id = compute_cre(epe_clean_id, epe_corrupted_id)
    deg_ood = compute_cre(epe_clean_ood, epe_corrupted_ood)
    return (deg_id - deg_ood) / deg_ood if deg_ood > 0 else 0.0
```

## Common pitfalls

- Confusing Out-Of-Domain (OOD) evaluation (trained on FlyingThings/Sintel/HD1K) with In-Domain (ID) evaluation (fine-tuned on KITTI-FC train split).
- Overlooking that unsupervised methods are only evaluated in the ID setting, while supervised models are tested in both.
- Failing to apply the exact 24 corruptions (7 temporal, 17 classical) including the upgraded PSF blur simulation method.

## Evidence (verbatim from paper)

> We first report the EPE of all 29 model variants to give the optical flow estimation performance in Tab. 2, then discuss the absolute robustness CRE and relative robustness CREr in Fig. 4 for deep research on optical flow robustness.

## Citation

```bibtex
@misc{yi2024benchmarking,
  title={Benchmarking the Robustness of Optical Flow Estimation to Corruptions},
  author={Yi et al. (2024)},
  year={2024},
  note={arXiv:2411.14865}
}
```

- arXiv: 2411.14865