Imagenet Multiplexing Eval

---
name: imagenet-multiplexing-eval
description: Evaluates the trade-off between inference latency, energy consumption, and classification accuracy when dynamically routing image inputs between a lightweight mobile model and a powerful cloud model using a learned neural multiplexer. Use when the user wants to benchmark on ImageNet ILSVRC 2012, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2001.05870
  bibtex_key: eshratifar2020multiplexing
  confidence: high
---

# imagenet-multiplexing-eval

> Runtime Deep Model Multiplexing for Reduced Latency and Energy Consumption Inference — Eshratifar et al. (2020) (arXiv:2001.05870, 2020)

## What this evaluates

Evaluates the trade-off between inference latency, energy consumption, and classification accuracy when dynamically routing image inputs between a lightweight mobile model and a powerful cloud model using a learned neural multiplexer.

## Datasets

- **ImageNet ILSVRC 2012** — total ?; splits: train (-1), val (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Top-1 classification accuracy: the fraction of input images correctly classified by the selected model (mobile or cloud).
- `latency` — range: other
  - End-to-end inference time measured in milliseconds (ms), including computation and communication overheads.
- `energy_consumption` — range: other
  - Energy consumed per inference measured in millijoules (mJ) using an INA226 power sensor on the mobile device.

## Input / output format

**Input**: RGB image passed to a 4-layer CNN multiplexer, which outputs a continuous routing score between 0 and 1.

**Output**: Binarized routing decision (0 = local/mobile, 1 = cloud) based on a 0.5 threshold, followed by the class label predicted by the selected model.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

# Latency and energy are measured via system timers and INA226 sensor respectively,
# not computed from (predictions, gold). They are averaged over the test set.
```

## Common pitfalls

- Directly comparing hybrid latency/energy to mobile-only baselines without accounting for the multiplexer overhead and communication costs, which the authors explicitly note is an unfair comparison since the hybrid approach prioritizes accuracy gains.
- Assuming the multiplexer threshold is universally fixed at 0.5; the paper demonstrates that sweeping thresholds yields different optimal values (e.g., 0.288 for cloud-based API inference).
- Overlooking that the reported accuracy improvements stem primarily from offloading hard examples to the cloud, rather than the multiplexer itself improving classification capability.

## Evidence (verbatim from paper)

> Table I: The latency, percentage of local inference, and accuracy of mobile-only, cloud-only and hybrid (multiplexing) methods.

## Citation

```bibtex
@misc{eshratifar2020multiplexing,
  title={Runtime Deep Model Multiplexing for Reduced Latency and Energy Consumption Inference},
  author={Eshratifar et al. (2020)},
  year={2020},
  note={arXiv:2001.05870}
}
```

- arXiv: 2001.05870