Bnn Nvm Benchmark Eval

---
name: bnn-nvm-benchmark-eval
description: This benchmark evaluates the inference accuracy and hardware performance of binary neural networks (BNNs) deployed on non-volatile memory crossbar architectures. It probes how hardware constraints like ADC resolution and first-layer input precision affect model accuracy, latency, energy efficiency, and chip area. Use when the user wants to benchmark on ImageNet, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.06227
  bibtex_key: huang2023bnnnvm
  confidence: high
---

# bnn-nvm-benchmark-eval

> Comprehensive Benchmarking of Binary Neural Networks on NVM Crossbar Architectures — Huang et al. (2023) (arXiv:2308.06227, 2023)

## What this evaluates

This benchmark evaluates the inference accuracy and hardware performance of binary neural networks (BNNs) deployed on non-volatile memory crossbar architectures. It probes how hardware constraints like ADC resolution and first-layer input precision affect model accuracy, latency, energy efficiency, and chip area.

## Datasets

- **ImageNet** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Top-1 classification accuracy computed on the ImageNet validation set after training with binary weights and activations.
- `latency` — range: other
  - Inference latency per layer and end-to-end, measured in nanoseconds via the NeuroSim simulator in pipelined mode.
- `energy consumption` — range: other
  - Total energy used for inference, measured in picojoules, accounting for ADC resolution and memory cell operations.
- `throughput` — range: other
  - Number of inference operations completed per second, derived from latency and pipelining parameters.
- `chip area` — range: other
  - Physical area occupied by the NVM crossbar and peripheral circuitry, measured in square micrometers.

## Input / output format

**Input**: Layer-level model structures (input activation/weight dimensions, pooling/activation parameters) and intermediate activation/weight data from trained BNN models, processed through a NeuroSim PyTorch wrapper.

**Output**: Simulated hardware metrics (chip area, latency, energy consumption, throughput, efficiency) and inference accuracy scores for each model and ADC resolution configuration.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels, sim_metrics):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    accuracy = correct / len(gold_labels)
    latency = sim_metrics['latency']
    energy = sim_metrics['energy_consumption']
    throughput = sim_metrics['throughput']
    area = sim_metrics['chip_area']
    return {'accuracy': accuracy, 'latency': latency,
            'energy_consumption': energy, 'throughput': throughput, 'chip_area': area}
```

## Common pitfalls

- NeuroSim defaults to high-precision floating-point input for the first layer; failing to add a bit-serialization module will yield unrealistic accuracy results for BNNs.
- Hardware metrics scale exponentially with ADC resolution and are highly sensitive to first-layer input precision, making direct comparisons across different ADC settings without normalization misleading.

## Evidence (verbatim from paper)

> To demonstrate the impact of ADC resolution on both accuracy and hardware performance, we perform tests on each model using various ADC resolutions and collect their hardware performance data, including chip area, latency, energy consumption, throughput, and efficiency.

## Citation

```bibtex
@misc{huang2023bnnnvm,
  title={Comprehensive Benchmarking of Binary Neural Networks on NVM Crossbar Architectures},
  author={Huang et al. (2023)},
  year={2023},
  note={arXiv:2308.06227}
}
```

- arXiv: 2308.06227