Blinkflow Eval

---
name: blinkflow-eval
description: Evaluates the accuracy and robustness of event-based optical flow estimation models on complex scenes with dynamic objects, occlusions, and high-frequency motion. It measures how well models generalize to unseen scenarios and handle fine-grained flow details compared to traditional rigid/static scene benchmarks. Use when the user wants to benchmark on BlinkFlow, DSEC, MVSEC, or asks about evaluating this task. Reports AEE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2303.07716
  bibtex_key: li2023blinkflow
  confidence: high
---

# blinkflow-eval

> BlinkFlow: A Dataset to Push the Limits of Event-based Optical Flow Estimation — Li et al. (2023) (arXiv:2303.07716, 2023)

## What this evaluates

Evaluates the accuracy and robustness of event-based optical flow estimation models on complex scenes with dynamic objects, occlusions, and high-frequency motion. It measures how well models generalize to unseen scenarios and handle fine-grained flow details compared to traditional rigid/static scene benchmarks.

## Datasets

- **BlinkFlow** — total 33000; splits: train (33000), test (-1)
- **DSEC** — total ?; splits: train (-1), test (-1)
- **MVSEC** — total ?; splits: train (-1), test (-1)

## Metrics

- `AEE` **(primary)** — range: pixels
  - Average Euclidean distance between predicted and ground truth optical flow vectors per pixel.
- `Outlier` — range: percent
  - Percentage of pixels where endpoint error exceeds both 3 pixels and 5% of the ground truth flow magnitude.
- `AE` — range: degrees
  - Angle in degrees between normalized predicted and normalized ground truth flow vectors.
- `N-PE` — range: percent
  - Percentage of pixels with flow error magnitude exceeding N pixels (computed for N=1, 2, 3).

## Input / output format

**Input**: Event stream data (and optionally corresponding frames) for consecutive time steps.

**Output**: 2D optical flow field (u, v vectors) for each pixel in the frame.

## Scoring recipe

```python
def compute_metrics(pred_flow, gt_flow):
    ee = np.sqrt((pred_flow[:,:,0] - gt_flow[:,:,0])**2 + (pred_flow[:,:,1] - gt_flow[:,:,1])**2)
    aee = np.mean(ee)
    mag_gt = np.sqrt(gt_flow[:,:,0]**2 + gt_flow[:,:,1]**2)
    outlier = np.mean((ee > 3) & (ee > 0.05 * mag_gt)) * 100
    norm_pred = pred_flow / (np.linalg.norm(pred_flow, axis=2, keepdims=True) + 1e-6)
    norm_gt = gt_flow / (np.linalg.norm(gt_flow, axis=2, keepdims=True) + 1e-6)
    ae = np.degrees(np.arccos(np.clip(np.sum(norm_pred * norm_gt, axis=2), -1.0, 1.0)))
    n_pe = {f"{n}-PE": np.mean(ee > n) * 100 for n in [1, 2, 3]}
    return {"AEE": aee, "Outlier": outlier, "AE": ae, **n_pe}
```

## Common pitfalls

- DSEC and MVSEC ground truth is computed via post-processing (motion field multiplied by time interval dt), which introduces bias and inaccuracies compared to BlinkFlow's direct renderer ground truth.
- Models easily overfit to DSEC/MVSEC training data (outdoor driving scenes), leading to severe performance degradation on indoor or complex dynamic scenes.
- The Outlier metric uses a dual threshold (>3 pixels AND >5% of magnitude), not just a fixed pixel threshold.

## Evidence (verbatim from paper)

> We report the optical flow accuracy in terms of the Average Endpoint Error (AEE), outlier, Angular Error (AE) and N-pixel erros (N-PE). AEE measures the Euclidean distance between the predicted flow and the ground truth. Outlier is the percentage of points with endpoint error greater than 3 pixels and 5% of the magnitude. AE measures the angle between the normalized optical flow predictions and normalized ground truth. N-PE indicates the percentage of flow errors higher than N pixels in magnitude and we compute 3-PE, 2-PE, 1-PE for flow outliers analysis from coarse to fine.

## Citation

```bibtex
@misc{li2023blinkflow,
  title={BlinkFlow: A Dataset to Push the Limits of Event-based Optical Flow Estimation},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2303.07716}
}
```

- arXiv: 2303.07716