Hw Nas Bench Eval

---
name: hw-nas-bench-eval
description: Evaluates hardware-aware neural architecture search (HW-NAS) algorithms by measuring how effectively they discover network topologies that optimize the trade-off between classification accuracy and on-device inference latency for specific target hardware. Use when the user wants to benchmark on HW-NAS-Bench, or asks about evaluating this task. Reports top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2103.10584
  bibtex_key: li2021hwnasbench
  confidence: high
---

# hw-nas-bench-eval

> HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark — Li et al. (2021) (arXiv:2103.10584, 2021)

## What this evaluates

Evaluates hardware-aware neural architecture search (HW-NAS) algorithms by measuring how effectively they discover network topologies that optimize the trade-off between classification accuracy and on-device inference latency for specific target hardware.

## Datasets

- **HW-NAS-Bench** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/RICE-EIC/HW-NAS-Bench

## Metrics

- `top-1 accuracy` **(primary)** — range: percent
  - Percentage of correctly classified samples in the test set.
- `latency (ms)` — range: other
  - On-device inference time measured in milliseconds for a single forward pass.

## Input / output format

**Input**: NAS search space definition (e.g., FBNet), target dataset (e.g., CIFAR-100), and target hardware device specification.

**Output**: A searched neural network architecture (topology and weights) optimized for the specified target device.

## Scoring recipe

```python
def evaluate(architecture, dataset, target_device, hw_bench):
    acc = compute_top1_accuracy(architecture, dataset)
    lat = hw_bench.query_latency(architecture, target_device)
    return {"top-1 accuracy": acc, "latency (ms)": lat}
```

## Common pitfalls

- Assuming latency or energy measurements from one hardware device transfer directly to another without re-measurement.
- Optimizing for a generic latency constraint rather than querying the exact device-specific hardware cost provided by the benchmark.
- Ignoring the negligible overhead of querying the benchmark during the search process, which can skew runtime comparisons.

## Evidence (verbatim from paper)

> Benchmark Setting. We adopt a SOTA HW-NAS algorithm, ProxylessNAS*(Cai et al., [2018])* for this experiment. As an example to use our HW-NAS-Bench, we use ProxylessNAS to search over the FBNet*(Wu et al., [2019])* search space on CIFAR-100*(Krizhevsky et al., [2009])*, when targeting different devices in our HW-NAS-Bench by simply querying the corresponding device’s measured/estimated hardware-cost, which has negligible overhead as compared to the HW-NAS algorithm itself, without the need for hardware expertise or knowledge during the whole HW-NAS. Table 5 illustrates that the searched architectures achieve the lowest latency among all architectures when the target devices of HW-NAS are the same as the one used to measure the architecture’s on-device inference latency.

## Citation

```bibtex
@misc{li2021hwnasbench,
  title={HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark},
  author={Li et al. (2021)},
  year={2021},
  note={arXiv:2103.10584}
}
```

- arXiv: 2103.10584