Smtm Mobile Cnn Eval

---
name: smtm-mobile-cnn-eval
description: Evaluates the latency reduction, accuracy loss, memory overhead, energy saving, and early exit performance of a semantic memory caching mechanism (SMTM) for accelerating CNN inference on mobile devices. Use when the user wants to benchmark on UCF101, CIFAR-100 (long-tail), or asks about evaluating this task. Reports latency reduction.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2112.02644
  bibtex_key: li2021boosting
  confidence: high
---

# smtm-mobile-cnn-eval

> Boosting Mobile CNN Inference through Semantic Memory — Li et al. (2021) (arXiv:2112.02644, 2021)

## What this evaluates

Evaluates the latency reduction, accuracy loss, memory overhead, energy saving, and early exit performance of a semantic memory caching mechanism (SMTM) for accelerating CNN inference on mobile devices.

## Datasets

- **UCF101** — total 70928; splits: test (70928)
- **CIFAR-100 (long-tail)** — total 1442; splits: test (1442)

## Metrics

- `latency reduction` **(primary)** — range: percent
  - Percentage reduction in average processing time per frame compared to the no-cache baseline. Calculated as (1 - T_SMTM / T_baseline) * 100%.
- `accuracy loss` — range: percent
  - Drop in top-1 classification accuracy compared to the baseline model without caching.
- `memory overhead` — range: other
  - Additional RAM usage in megabytes required to store semantic centers and frequency tables.
- `energy saving` — range: percent
  - Percentage reduction in energy consumption measured via on-device PMIC (voltage × current × time).
- `early exit ratio` — range: percent
  - Percentage of input frames that trigger an early exit at a specific network layer based on semantic similarity threshold τ.

## Input / output format

**Input**: Raw video frames (224×224 for UCF101, 32×32 for CIFAR-100) processed by CNN architectures (AlexNet, GoogleNet, ResNet50, MobileNet V2, VGG16).

**Output**: Top-1 class prediction and early-exit flag indicating whether inference terminates at the current layer based on semantic similarity threshold τ.

## Scoring recipe

```python
# Compute baseline metrics (no-cache CPU/GPU)
T_base = mean(latency(no_cache_baseline))
E_base = mean(energy(no_cache_baseline))
Acc_base = top1_accuracy(no_cache_baseline)

# Compute SMTM metrics
T_smtm = mean(latency(smtm_predictions))
E_smtm = mean(energy(smtm_predictions))
Acc_smtm = top1_accuracy(smtm_predictions)
Exit_counts = count_exits_per_layer(smtm_predictions)

latency_reduction = (1 - T_smtm / T_base) * 100
accuracy_loss = Acc_base - Acc_smtm
energy_saving = (1 - E_smtm / E_base) * 100
early_exit_ratio = Exit_counts / total_samples * 100
```

## Common pitfalls

- Fair comparison with DeepCache/DeepMon requires matching ncnn configurations and explicitly disabling SIMD acceleration, as stated in the experimental setup.
- Energy measurement cannot rely on single inferences due to PMIC sampling limits; the device must be forced into an infinite inference loop to average voltage and current.
- The CIFAR-100 evaluation uses a custom long-tail split of 1,442 images rather than the standard 10,000-image test set.

## Evidence (verbatim from paper)

> We use five metrics to comprehensively evaluate the performance of SMTM: latency reduction (Section 8.2), accuracy loss (Section 8.3), memory overhead (Section 8.4), energy saving (Section 8.5), and early exit ratio (Section 8.6).

## Citation

```bibtex
@misc{li2021boosting,
  title={Boosting Mobile CNN Inference through Semantic Memory},
  author={Li et al. (2021)},
  year={2021},
  note={arXiv:2112.02644}
}
```

- arXiv: 2112.02644