Workload Allocation Eval

---
name: workload-allocation-eval
description: Evaluates the latency performance of AI workload allocation strategies across hierarchical cloud/edge/device computing environments for latency-sensitive medical ICU applications. It measures how effectively dynamic routing minimizes end-to-end response time when processing and transmission delays are factored in. Use when the user wants to benchmark on Edge AIBench ICU Applications (MIMIC-III derived), or asks about evaluating this task. Reports response time.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2002.03493
  bibtex_key: hao2020aioriented
  confidence: high
---

# workload-allocation-eval

> AI-oriented Medical Workload Allocation for Hierarchical Cloud/Edge/Device Computing — Hao et al. (2020) (arXiv:2002.03493, 2020)

## What this evaluates

Evaluates the latency performance of AI workload allocation strategies across hierarchical cloud/edge/device computing environments for latency-sensitive medical ICU applications. It measures how effectively dynamic routing minimizes end-to-end response time when processing and transmission delays are factored in.

## Datasets

- **Edge AIBench ICU Applications (MIMIC-III derived)** — total 18; splits: inference (18)

## Metrics

- `response time` **(primary)** — range: milliseconds
  - Response time is formulated as the sum of processing delay (model FLOPs divided by device FLOPS) and transmission delay (inference data size divided by network bandwidth). The evaluation aims to minimize total response time across dynamically routed workloads.

## Input / output format

**Input**: Pre-trained LSTM models for three ICU applications (short-of-breath alerts, life-death prediction, patient phenotype classification) paired with inference data of varying sizes (64 KB to 2048 KB).

**Output**: The system outputs the assigned execution layer (cloud, edge, or device) for each workload and the measured response time.

## Scoring recipe

```python
def calc_response_time(workload, device):
    # Processing delay: model FLOPs / device FLOPS
    proc_delay = workload['flops'] / device['flops']
    # Transmission delay: data size / network bandwidth
    trans_delay = workload['data_size_kb'] / device['bandwidth_kb_s']
    return proc_delay + trans_delay
```

## Common pitfalls

- The models are trained offline on the cloud; the evaluation only measures online inference latency, not training time or prediction accuracy.
- Workloads are synthetic size variations (64-2048 KB) applied to fixed pre-trained models, not a standard train/validation/test split.
- Network parameters (latency/bandwidth) are fixed per link (cloud-device vs edge-device) and do not vary dynamically during the experiment.

## Evidence (verbatim from paper)

> It formulates response time as a function of processing and transmission delays, derives an optimal single-job allocation model based on computational and network trade-offs, and extends it to a multi-job scheduling algorithm that minimizes total response time through dynamic workload routing across layers.

## Citation

```bibtex
@misc{hao2020aioriented,
  title={AI-oriented Medical Workload Allocation for Hierarchical Cloud/Edge/Device Computing},
  author={Hao et al. (2020)},
  year={2020},
  note={arXiv:2002.03493}
}
```

- arXiv: 2002.03493