Multi Human Optical Flow Eval

---
name: multi-human-optical-flow-eval
description: Evaluates optical flow estimation models on synthetic single- and multi-human video sequences. It probes the model's ability to handle complex human poses, occlusions, and fine-grained motion on small body parts like fingers and hands. Use when the user wants to benchmark on SHOF, MHOF, or asks about evaluating this task. Reports EPE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1910.11667
  bibtex_key: ranjan2019learningmulti
  confidence: high
---

# multi-human-optical-flow-eval

> Learning Multi-Human Optical Flow — Ranjan et al. (arXiv:1910.11667, 2019)

## What this evaluates

Evaluates optical flow estimation models on synthetic single- and multi-human video sequences. It probes the model's ability to handle complex human poses, occlusions, and fine-grained motion on small body parts like fingers and hands.

## Datasets

- **SHOF** — total 146020; splits: train (135153), test (10867)
- **MHOF** — total 111312; splits: train (86259), test (13236), val (11817)

## Metrics

- `EPE` **(primary)** — range: pixels
  - Average End Point Error: the mean Euclidean distance between predicted and ground truth optical flow vectors across all pixels (or a specified mask).
- `MCI` — range: other
  - Motion Compensated Intensity: measures reconstruction error after warping the first frame using the estimated flow. Lower values indicate better flow estimation.

## Input / output format

**Input**: Pair of consecutive frames from synthetic human motion sequences (256x256 for SHOF, 640x640 for MHOF).

**Output**: 2-channel optical flow field (horizontal and vertical displacement per pixel).

## Scoring recipe

```python
def compute_epe(pred_flow, gt_flow, body_mask=None):
    if body_mask is not None:
        pred_flow = pred_flow[body_mask]
        gt_flow = gt_flow[body_mask]
    diff = np.sqrt((pred_flow[:,:,0] - gt_flow[:,:,0])**2 + (pred_flow[:,:,1] - gt_flow[:,:,1])**2)
    return np.mean(diff)
```

## Common pitfalls

- Evaluating on the full image including static background inflates performance; the protocol recommends masking out background to compute body-only EPE.
- Architectures like FlowNet struggle with the small motions prevalent in SHOF, so dataset choice and motion scale must be considered when comparing methods.
- Resolution differences (256x256 vs 640x640) significantly impact training speed and the ability to resolve fine details like fingers.

## Evidence (verbatim from paper)

> We quantitatively evaluate optical flow methods on the MHOF dataset using motion compensated intensity metric. ... Therefore, we mask out all errors of background pixels and compute the average EPE only on body pixels (see Table[5]).

## Citation

```bibtex
@misc{ranjan2019learningmulti,
  title={Learning Multi-Human Optical Flow},
  author={Ranjan et al.},
  year={2019},
  note={arXiv:1910.11667}
}
```

- arXiv: 1910.11667