Saliency Prediction Driving Eval

---
name: saliency-prediction-driving-eval
description: Evaluates the ability of saliency prediction models to accurately identify visually salient regions and semantic objects in autonomous driving scenarios. It probes whether models can capture critical driving elements like pedestrians and approaching vehicles while mitigating center-bias and peripheral vision neglect. Use when the user wants to benchmark on BDD-A, DR(eye)VE, JAAD, or asks about evaluating this task. Reports D_KL.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1911.10455
  bibtex_key: pal2019looking
  confidence: high
---

# saliency-prediction-driving-eval

> Looking at the right stuff: Guided semantic-gaze for autonomous driving — Pal et al. (2019) (arXiv:1911.10455, 2019)

## What this evaluates

Evaluates the ability of saliency prediction models to accurately identify visually salient regions and semantic objects in autonomous driving scenarios. It probes whether models can capture critical driving elements like pedestrians and approaching vehicles while mitigating center-bias and peripheral vision neglect.

## Datasets

- **BDD-A** — total ?; splits: test (-1)
- **DR(eye)VE** — total ?; splits: test (-1)
- **JAAD** — total ?; splits: test (-1)

## Metrics

- `D_KL` **(primary)** — range: [0, ∞)
  - Asymmetric Kullback-Leibler divergence between ground truth and predicted saliency distributions. Lower values indicate better alignment.
- `CC` — range: [-1, 1]
  - Pearson’s Cross Correlation coefficient between flattened ground truth and predicted saliency maps. Higher values indicate better similarity.
- `F1_score` — range: [0, 1]
  - Fβ-score with β²=1 (equal weight to precision and recall) computed on binary segmentation masks derived from predicted and ground truth maps.
- `MAE` — range: [0, 1]
  - Mean Absolute Error measuring pixel-wise difference between predicted saliency map and ground truth.

## Input / output format

**Input**: RGB image frames from driving scenarios (e.g., day/night, city/highway, intersections, crowded streets).

**Output**: Continuous saliency maps (probability distributions over pixels) and/or binary semantic segmentation masks for evaluated objects.

## Scoring recipe

```python
def compute_metrics(pred, gt):
    P, Q = pred.flatten(), gt.flatten()
    P, Q = P/P.sum(), Q/Q.sum()
    dk = np.sum(Q * np.log(Q/P))
    cc = np.corrcoef(P, Q)[0,1]
    P_bin, Q_bin = (P>0.5).astype(int), (Q>0.5).astype(int)
    tp, fp, fn = np.sum(P_bin&Q_bin), np.sum(P_bin&~Q_bin), np.sum(~P_bin&Q_bin)
    prec, rec = tp/(tp+fp+1e-8), tp/(tp+fn+1e-8)
    f1 = (2*prec*rec)/(prec+rec+1e-8)
    mae = np.mean(np.abs(pred-gt))
    return dk, cc, f1, mae
```

## Common pitfalls

- Using discrete fixation metrics (AUC, NSS, IG) instead of continuous distribution metrics, which fail to capture object boundaries crucial for driving.
- Applying the standard β²=0.3 weighting for Fβ-score instead of β²=1, which underweights recall and ignores the high cost of false negatives in driving.
- Evaluating semantic-centric metrics against raw gaze ground-truth instead of using Mask R-CNN segmented maps as the semantic baseline.

## Evidence (verbatim from paper)

> For the first category, we choose two distribution-based metrics - Kullback-Leibler Divergence ($\mathrm{D_{KL}}$), and Pearson’s Cross Correlation ($\mathrm{CC}$). $\mathrm{D_{KL}}$ is an asymmetric dissimilarity metric, that penalizes FN more than FP. $\mathrm{CC}$, on the other hand is a symmetric similarity metric which equally affects both FN and FP... In the second category, we again consider two metrics - namely $\mathrm{F}$-$\mathrm{score}$, which measures region similarity of detection, and Mean Absolute Error ($\mathrm{MAE}$), which gives pixel-wise accuracy.

## Citation

```bibtex
@misc{pal2019looking,
  title={Looking at the right stuff: Guided semantic-gaze for autonomous driving},
  author={Pal et al. (2019)},
  year={2019},
  note={arXiv:1911.10455}
}
```

- arXiv: 1911.10455