Visual Quality Assessment Eval

---
name: visual-quality-assessment-eval
description: Evaluates the perceptual visual quality of interpolated frames generated by optical flow methods against human judgments. It measures how well traditional objective metrics like RMSE correlate with crowdsourced subjective quality ratings across multiple video sequences. Use when the user wants to benchmark on Middlebury, or asks about evaluating this task. Reports SROCC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1901.05362
  bibtex_key: men2019visual
  confidence: high
---

# visual-quality-assessment-eval

> Technical Report on Visual Quality Assessment for Frame Interpolation — Men et al. (2019) (arXiv:1901.05362, 2019)

## What this evaluates

Evaluates the perceptual visual quality of interpolated frames generated by optical flow methods against human judgments. It measures how well traditional objective metrics like RMSE correlate with crowdsourced subjective quality ratings across multiple video sequences.

## Datasets

- **Middlebury** — total ?; splits: test (-1)

## Metrics

- `SROCC` **(primary)** — range: [-1, 1]
  - Spearman Rank Order Correlation Coefficient measuring the monotonic relationship between RMSE-based rankings and subjective quality rankings. Computed with 95% confidence intervals via bootstrapping (1000 iterations) and Fisher transform.
- `subjective quality score` — range: [0, 1]
  - Scale values reconstructed from pairwise comparisons using Thurstone's model, linearly rescaled to [0, 1] by anchoring the worst-quality image to 0 and the ground-truth image to 1.

## Input / output format

**Input**: Pairwise comparisons of interpolated frames from 141 optical flow methods across 8 Middlebury sequences, presented to crowd workers alongside two anchor images (worst quality and ground truth).

**Output**: Binary preference choices per pair, aggregated into rescaled subjective quality scores in [0, 1].

## Scoring recipe

```python
def compute_protocol(prefs, rmse_values):
    # Reconstruct scores via Thurstone's model with anchors
    raw_scores = thurstone_reconstruct(prefs, anchors=[0, 1])
    # Rescale to [0, 1]
    subjective_scores = rescale_to_zero_one(raw_scores)
    # Compute Spearman rank correlation
    srocc = spearman_rank_correlation(rmse_values, subjective_scores)
    # Bootstrap confidence intervals
    ci_95 = bootstrap_ci(srocc, n_iterations=1000)
    return srocc, ci_95, subjective_scores
```

## Common pitfalls

- RMSE correlates poorly with human perception for frame interpolation due to artifacts not captured by pixel-wise error.
- Subjective scores require strict anchor calibration (worst/ground-truth) to ensure comparability across separate crowdsourcing jobs.
- Crowd workers must pass accuracy filters (≥70% on test questions) to be included in the final trusted worker pool.

## Evidence (verbatim from paper)

> Table II shows the bootstrapped (after 1000 iterations) SROCC correlation values accompanied with confidence intervals (95%) between the ranking in Middlebury benchmark (i.e., ranking according to RMSE) and the re-ranking according to our subjective study.

## Citation

```bibtex
@misc{men2019visual,
  title={Technical Report on Visual Quality Assessment for Frame Interpolation},
  author={Men et al. (2019)},
  year={2019},
  note={arXiv:1901.05362}
}
```

- arXiv: 1901.05362