Integer Quantization Eval

---
name: integer-quantization-eval
description: Evaluates the classification and detection accuracy, as well as inference latency, of neural networks quantized to 8-bit integer arithmetic on mobile ARM CPUs compared to floating-point baselines. Use when the user wants to benchmark on ImageNet, COCO, Face detection dataset, Face attributes dataset, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1712.05877
  bibtex_key: jacob2017quantization
  confidence: high
---

# integer-quantization-eval

> Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference — Benoit Jacob et al. (arXiv:1712.05877, 2017)

## What this evaluates

Evaluates the classification and detection accuracy, as well as inference latency, of neural networks quantized to 8-bit integer arithmetic on mobile ARM CPUs compared to floating-point baselines.

## Datasets

- **ImageNet** — total ?; splits: val (-1)
- **COCO** — total ?; splits: val (-1)
- **Face detection dataset** — total ?; splits: test (-1)
- **Face attributes dataset** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted class labels out of total samples.
- `mAP` — range: percent
  - Mean Average Precision computed over IoU thresholds from 0.5 to 0.95 for object detection.
- `latency (ms)` — range: ms
  - Inference time measured in milliseconds on single-threaded or multi-threaded ARM CPU cores.

## Input / output format

**Input**: Preprocessed image inputs fed into quantized or floating-point MobileNet/ResNet/Inception models.

**Output**: Predicted class labels, bounding boxes, or attribute values, along with measured inference latency in milliseconds.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truth, latencies):
    correct = sum(1 for p, g in zip(predictions, ground_truth) if p == g)
    accuracy = correct / len(ground_truth) * 100
    ap_scores = []
    for cls in classes:
        prec, rec = compute_precision_recall_curve(predictions[cls], ground_truth[cls])
        ap_scores.append(interpolate_ap(prec, rec))
    mAP = sum(ap_scores) / len(classes)
    avg_latency = sum(latencies) / len(latencies)
    return accuracy, mAP, avg_latency
```

## Common pitfalls

- Latency is highly dependent on specific ARM micro-architectures (Snapdragon 835/821) and thread counts, making cross-device comparisons difficult.
- Accuracy is often reported as relative degradation from the floating-point baseline rather than absolute values.
- Quantization bit-depths for weights and activations are ablated independently, which can obscure total model size vs. accuracy tradeoffs.

## Evidence (verbatim from paper)

> Integer-only quantized MobileNets achieve higher accuracies than floating-point MobileNets given the same run time budget. The accuracy gap is quite substantial (~10%) for Snapdragon 835 LITTLE cores at the 33ms latency needed for real-time (30 fps) operation.

## Citation

```bibtex
@misc{jacob2017quantization,
  title={Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference},
  author={Benoit Jacob et al.},
  year={2017},
  note={arXiv:1712.05877}
}
```

- arXiv: 1712.05877