Imagenet Mcu Eval

---
name: imagenet-mcu-eval
description: Evaluates image classification accuracy on ultra-constrained microcontrollers (MCUs) with strict SRAM and Flash limits, measuring the trade-off between model quantization, memory footprint, and performance. Use when the user wants to benchmark on ImageNet, or asks about evaluating this task. Reports Top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.19076
  bibtex_key: lin2024tinyml
  confidence: high
---

# imagenet-mcu-eval

> Tiny Machine Learning: Progress and Futures — Ji Lin et al. (2024) (arXiv:2403.19076, 2024)

## What this evaluates

Evaluates image classification accuracy on ultra-constrained microcontrollers (MCUs) with strict SRAM and Flash limits, measuring the trade-off between model quantization, memory footprint, and performance.

## Datasets

- **ImageNet** — total ?; splits: val (-1)

## Metrics

- `Top-1 accuracy` **(primary)** — range: percent
  - Percentage of validation images where the model's highest-confidence prediction matches the ground truth label.
- `Top-5 accuracy` — range: percent
  - Percentage of validation images where the ground truth label appears in the model's top 5 predictions.

## Input / output format

**Input**: Standard ImageNet validation images processed through a quantized (int8/int4) neural network optimized for MCU hardware constraints.

**Output**: Predicted class labels (top-1 and top-5) for each input image.

## Scoring recipe

```python
For each image in the ImageNet validation set:
  pred = model.predict(image)
  if pred == ground_truth: top1_correct += 1
  if ground_truth in pred[:5]: top5_correct += 1
top1_acc = (top1_correct / total_images) * 100
top5_acc = (top5_correct / total_images) * 100
```

## Common pitfalls

- Hardware memory limits (SRAM/Flash) are hard constraints; exceeding them causes out-of-memory (OOM) failures rather than just slowdowns.
- Quantization precision (int8 vs int4) significantly impacts both accuracy and memory footprint, requiring careful system-algorithm co-design.

## Evidence (verbatim from paper)

> We compared MCUNet with existing state-of-the-art solutions on ImageNet classification under two hardware settings: 256kB SRAM/1MB Flash and 512kB SRAM/2MB Flash. The goal is to achieve the highest ImageNet Top-1 accuracy on resource-constrained MCUs (Table IV).

## Citation

```bibtex
@misc{lin2024tinyml,
  title={Tiny Machine Learning: Progress and Futures},
  author={Ji Lin et al. (2024)},
  year={2024},
  note={arXiv:2403.19076}
}
```

- arXiv: 2403.19076