Keyword Spotting Efficiency Eval

---
name: keyword-spotting-efficiency-eval
description: Evaluates the energy efficiency and inference speed of various hardware platforms (CPU, GPU, neuromorphic chips) running a keyword spotting neural network on audio data. It measures how power consumption and latency scale with network size and batch configuration. Use when the user wants to benchmark on Keyword Spotting Dataset, or asks about evaluating this task. Reports energy cost per inference (J).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1812.01739
  bibtex_key: blouw2018benchmarking
  confidence: high
---

# keyword-spotting-efficiency-eval

> Benchmarking Keyword Spotting Efficiency on Neuromorphic Hardware — Blouw et al. (2018) (arXiv:1812.01739, 2018)

## What this evaluates

Evaluates the energy efficiency and inference speed of various hardware platforms (CPU, GPU, neuromorphic chips) running a keyword spotting neural network on audio data. It measures how power consumption and latency scale with network size and batch configuration.

## Datasets

- **Keyword Spotting Dataset** — total ?; splits: test (-1); repo https://github.com/abr/power_benchmarks

## Metrics

- `energy cost per inference (J)` **(primary)** — range: Joules (J)
  - Dynamic power (W) = mean runtime power (W) - mean idle power (W). Energy cost per inference (J) = dynamic power (W) / inference speed (inferences/sec).

## Input / output format

**Input**: Audio frames with a 10ms stride fed into a two-layer spiking neural network.

**Output**: Keyword classification output, plus logged runtime power, idle power, and inference timestamps for metric computation.

## Scoring recipe

```python
dynamic_power_watts = mean_runtime_power_watts - mean_idle_power_watts
energy_cost_joules = dynamic_power_watts / inferences_per_second
# Compute 95% CI via bootstrapping on log-transformed values
ci_lower, ci_upper = bootstrap_ci(log(energy_cost_joules), n=1000, alpha=0.05)
```

## Common pitfalls

- Loihi and Movidius only support a batch size of 1, making direct batched comparisons invalid.
- Scaling experiments fix batch size at 1 and ignore trained weights, measuring only topology/size effects on speed and energy.
- Confidence intervals must be computed on log-transformed power values, not raw values.

## Evidence (verbatim from paper)

> To explain, the mean idle power consumption is subtracted from the mean runtime power consumption to give the mean dynamic power consumption in Watts in the fourth column. Since one Watt is equivalent to one joule per second, we can divide the dynamic power consumption value by the number of inferences per second, listed in the fifth column, to get the mean energy cost per inference, listed in the final column.

## Citation

```bibtex
@misc{blouw2018benchmarking,
  title={Benchmarking Keyword Spotting Efficiency on Neuromorphic Hardware},
  author={Blouw et al. (2018)},
  year={2018},
  note={arXiv:1812.01739}
}
```

- arXiv: 1812.01739