Nestdnn Eval

---
name: nestdnn-eval
description: Evaluates the inference accuracy, computational cost, memory footprint, and switching overhead of a multi-capacity deep learning architecture compared to independent baseline models across six mobile vision classification tasks. It also benchmarks a resource-aware scheduler's ability to maintain accuracy and frame rate under dynamic runtime memory constraints. Use when the user wants to benchmark on CIFAR-10, ImageNet-50, ImageNet-100, GTSRB, Adience-Gender, Places-32, or asks about evaluating this task. Reports Top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1810.10090
  bibtex_key: fang2018nestdnn
  confidence: high
---

# nestdnn-eval

> NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Vision — Fang et al. (2018) (arXiv:1810.10090, 2018)

## What this evaluates

Evaluates the inference accuracy, computational cost, memory footprint, and switching overhead of a multi-capacity deep learning architecture compared to independent baseline models across six mobile vision classification tasks. It also benchmarks a resource-aware scheduler's ability to maintain accuracy and frame rate under dynamic runtime memory constraints.

## Datasets

- **CIFAR-10** — total 60000; splits: train (50000), test (10000)
- **ImageNet-50** — total 65000; splits: train (63000), test (2000)
- **ImageNet-100** — total 126000; splits: train (121000), test (5000)
- **GTSRB** — total ?; splits: test (-1)
- **Adience-Gender** — total ?; splits: test (-1)
- **Places-32** — total ?; splits: test (-1)

## Metrics

- `Top-1 accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly classified images out of the total test set. Calculated as correct predictions divided by total predictions.
- `Computational cost (GFLOPs)` — range: GFLOPs
  - Total number of billion floating-point operations required for a single forward pass through the network.
- `Memory footprint (MB)` — range: MB
  - Total model size in megabytes, calculated from the number of parameters and weights stored in memory.
- `Model switching overhead` — range: MB & Joules
  - Average page-in and page-out memory usage (MB) and energy consumption (Joules) when transitioning between model variants during runtime.

## Input / output format

**Input**: RGB images resized to dataset-specific dimensions (e.g., 32×32 for CIFAR-10, 224×224 for ImageNet/GTSRB/Places), fed into VGG-16 or ResNet-50 backbones.

**Output**: Predicted class labels for classification; system-level reports of inference accuracy, frame processing rate, memory usage, GFLOPs, and energy consumption.

## Scoring recipe

```python
correct = sum(1 for p, g in zip(predictions, labels) if p == g)
accuracy = correct / len(labels)
memory_mb = model_size_bytes / (1024**2)
gflops = sum(conv_flops + fc_flops for layer in model)
energy_joules = measure_power_over_time(duration_seconds)
switching_overhead_mb = page_in_size_mb + page_out_size_mb
```

## Common pitfalls

- Baseline models are pre-trained on ImageNet before fine-tuning, which significantly inflates their accuracy compared to training from scratch.
- The scheduler benchmark uses a synthetic workload (random app creation/killing every second) rather than real-world application traces.
- Energy measurements rely on a hardware Monsoon power monitor on a single device (Galaxy S8), not software estimation or cross-device averaging.

## Evidence (verbatim from paper)

> Figure 6 illustrates the comparison between descendent models and baseline models across six mobile vision applications. For each application, we show the top-1 accuracies of both descendant models and baseline models as a function of model size. To compare the performance between our resource-aware approach and the resource-agnostic status quo approach, we have designed a benchmark that emulates runtime application queries in diverse scenarios... We repeat the simulation 100 times and report the average runtime performance.

## Citation

```bibtex
@misc{fang2018nestdnn,
  title={NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Vision},
  author={Fang et al. (2018)},
  year={2018},
  note={arXiv:1810.10090}
}
```

- arXiv: 1810.10090