Wake Vision Eval

---
name: wake-vision-eval
description: Evaluates the robustness and accuracy of TinyML person detection models across diverse demographic, environmental, and visual conditions. It benchmarks binary classification performance on large-scale, real-world image datasets tailored for resource-constrained devices. Use when the user wants to benchmark on Wake Vision, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.00892
  bibtex_key: banbury2024wakevision
  confidence: high
---

# wake-vision-eval

> Wake Vision: A Tailored Dataset and Benchmark Suite for TinyML Computer Vision Applications — Banbury et al. (2024) (arXiv:2405.00892, 2024)

## What this evaluates

Evaluates the robustness and accuracy of TinyML person detection models across diverse demographic, environmental, and visual conditions. It benchmarks binary classification performance on large-scale, real-world image datasets tailored for resource-constrained devices.

## Datasets

- **Wake Vision** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Top-level test accuracy calculated as the percentage of correctly classified binary labels on the held-out test set.
- `F1 score` — range: [0, 1]
  - Harmonic mean of precision and recall, computed per fine-grained subgroup (e.g., age, lighting, distance) and averaged across three models per dataset.

## Input / output format

**Input**: 224x224x3 RGB images representing real-world scenes for binary person detection.

**Output**: Binary classification label (person vs. non-person/wake word).

## Scoring recipe

```python
def compute_accuracy(preds, gold):
    return sum(p == g for p, g in zip(preds, gold)) / len(gold) * 100

def compute_f1(preds, gold):
    tp = sum(p == 1 and g == 1 for p, g in zip(preds, gold))
    fp = sum(p == 1 and g == 0 for p, g in zip(preds, gold))
    fn = sum(p == 0 and g == 1 for p, g in zip(preds, gold))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- Models trained on VWW may outperform Wake Vision models on the VWW test set due to domain shift, not superior generalization.
- Fine-grained F1 scores are averaged across three models per dataset, not reported per individual model.

## Evidence (verbatim from paper)

> The Wake Vision model exhibits superior robustness across the challenging settings exercised by our benchmarking suite. For instance, on the “Depictions” benchmark, which evaluates performance on images containing persons, non-person objects, or no depictions, the Wake Vision model achieves an F1 score of 0.71 for person depictions, outperforming the VWW model’s 0.66.

## Citation

```bibtex
@misc{banbury2024wakevision,
  title={Wake Vision: A Tailored Dataset and Benchmark Suite for TinyML Computer Vision Applications},
  author={Banbury et al. (2024)},
  year={2024},
  note={arXiv:2405.00892}
}
```

- arXiv: 2405.00892