Mlperf Abfp Eval

---
name: mlperf-abfp-eval
description: Evaluates the inference accuracy of deep neural networks when simulated with Adaptive Block Floating-Point (ABFP) number representation and analog-to-digital converter (ADC) noise. It probes how tile width, amplification gain, and bitwidth affect model quality, and compares the effectiveness of Quantization-Aware Training (QAT) versus Differential Noise Finetuning (DNF) in recovering baseline float32 performance. Use when the user wants to benchmark on MLPerf datacenter inference benchmark, or asks about evaluating this task. Reports top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.06287
  bibtex_key: basumallik2022adaptive
  confidence: high
---

# mlperf-abfp-eval

> Adaptive Block Floating-Point for Analog Deep Learning Hardware — Basumallik et al. (2022) (arXiv:2205.06287, 2022)

## What this evaluates

Evaluates the inference accuracy of deep neural networks when simulated with Adaptive Block Floating-Point (ABFP) number representation and analog-to-digital converter (ADC) noise. It probes how tile width, amplification gain, and bitwidth affect model quality, and compares the effectiveness of Quantization-Aware Training (QAT) versus Differential Noise Finetuning (DNF) in recovering baseline float32 performance.

## Datasets

- **MLPerf datacenter inference benchmark** — total ?; splits: test (-1)

## Metrics

- `top-1 accuracy` **(primary)** — range: percent
  - Fraction of correctly classified images out of the total number of images in the dataset.
- `mAP` — range: percent
  - Mean Average Precision across all object classes, computed as the area under the precision-recall curve.

## Input / output format

**Input**: Pre-trained PyTorch checkpoints with bfloat16 input tensors. Convolutions are converted to tiled matrix-multiplications via im2col; other layers (batch-norm, pooling, non-linearities) compute in float32 before converting back to bfloat16.

**Output**: Model predictions (class probabilities or bounding boxes) in bfloat16 format, evaluated against ground-truth labels.

## Scoring recipe

```python
# For classification (ResNet50)
correct = sum(pred == true_label for pred, true_label in predictions)
accuracy = correct / len(predictions) * 100

# For detection (SSD-ResNet34)
mAP = compute_mean_average_precision(predictions, ground_truth) # Standard MLPerf protocol
```

## Common pitfalls

- Higher amplification gain does not universally improve quality; it can increase noise standard deviation and cause accuracy drops depending on tile width and layer sensitivity.
- Reducing weight/input bitwidths from 8 to 6 bits has negligible impact on quality compared to changes in tile width or gain.
- DNF speed and accuracy depend heavily on selectively injecting noise only into high-variance layers, unlike vanilla QAT which applies noise globally.

## Evidence (verbatim from paper)

> Table III: Comparing QAT and DNF on ResNet50 and SSD-ResNet34. Bold values highlight methods that improved the DNN to above 99% of the original float32 metric. The metrics are top-1 accuracy (ResNet50) and mAP (SSD-ResNet34).

## Citation

```bibtex
@misc{basumallik2022adaptive,
  title={Adaptive Block Floating-Point for Analog Deep Learning Hardware},
  author={Basumallik et al. (2022)},
  year={2022},
  note={arXiv:2205.06287}
}
```

- arXiv: 2205.06287