Finn Bnn Inference Eval

---
name: finn-bnn-inference-eval
description: Evaluates the inference performance of binarized neural networks (BNNs) accelerated on FPGAs using the FINN framework. It measures classification throughput, latency, and accuracy across standard image datasets to assess hardware efficiency and resource utilization. Use when the user wants to benchmark on MNIST, CIFAR-10, SVHN, or asks about evaluating this task. Reports classification throughput (FPS).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1612.07119
  bibtex_key: umuroglu2016finn
  confidence: high
---

# finn-bnn-inference-eval

> FINN: A Framework for Fast, Scalable Binarized Neural Network Inference — Umuroglu et al. (2016) (arXiv:1612.07119, 2016)

## What this evaluates

Evaluates the inference performance of binarized neural networks (BNNs) accelerated on FPGAs using the FINN framework. It measures classification throughput, latency, and accuracy across standard image datasets to assess hardware efficiency and resource utilization.

## Datasets

- **MNIST** — total ?; splits: test (10000)
- **CIFAR-10** — total ?; splits: test (10000)
- **SVHN** — total ?; splits: test (10000)

## Metrics

- `classification throughput (FPS)` **(primary)** — range: [0, ∞)
  - FPS = N / T, where N is the number of processed images (10,000) and T is the measured execution time in seconds.
- `latency (μs)` — range: [0, ∞)
  - Time taken to classify a single image, measured in microseconds.
- `accuracy (%)` — range: [0, 100]
  - Percentage of correctly classified images out of the total test set.
- `power consumption (W)` — range: [0, ∞)
  - Measured as P_chip (FPGA internal power via PMBus) and P_wall (total board power via wall meter).

## Input / output format

**Input**: Binary or low-precision images (28x28 for MNIST, 32x32 for CIFAR-10/SVHN) initialized in shared DRAM.

**Output**: 10-bit one-hot vector (for fully connected topologies) or 10-element 16-bit vector (for convolutional topology) indicating class logits/probabilities.

## Scoring recipe

```python
def evaluate(accelerator, test_images, ground_truth):
    N = len(test_images)
    start_time = time.time()
    predictions = accelerator.run(test_images)
    end_time = time.time()
    fps = N / (end_time - start_time)
    accuracy = sum(1 for p, g in zip(predictions, ground_truth) if p == g) / N
    return fps, accuracy
```

## Common pitfalls

- Confusing P_chip (FPGA internal power) with P_wall (total board power), as P_wall includes a ~7W idle baseline that dominates low-power measurements.
- Assuming higher folding factors linearly improve throughput; they are constrained by integer folding and BRAM/LUT limits, often causing resource bottlenecks or unused capacity.
- Comparing FPS across different datasets without accounting for varying image resolutions and network complexities.

## Evidence (verbatim from paper)

> The host code runs on the Cortex-A9 cores of the Zynq. It initializes 10000 images with test data in the Zynq's shared DRAM, launches and times the accelerator execution to measure classification throughput, then measures accuracy by comparing against the correct classifications.

## Citation

```bibtex
@misc{umuroglu2016finn,
  title={FINN: A Framework for Fast, Scalable Binarized Neural Network Inference},
  author={Umuroglu et al. (2016)},
  year={2016},
  note={arXiv:1612.07119}
}
```

- arXiv: 1612.07119