Elastic Scaling Eval

---
name: elastic-scaling-eval
description: Evaluates a deep learning job scheduler's ability to dynamically adjust GPU allocations and batch sizes to maximize cluster throughput and minimize job completion times. It probes how well the system handles compute-bound, communication-bound, and non-elastic workloads under varying job arrival patterns. Use when the user wants to benchmark on CIFAR100, Food101, or asks about evaluating this task. Reports SJS Efficiency.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2006.13878
  bibtex_key: saxena2020elastic
  confidence: high
---

# elastic-scaling-eval

> Effective Elastic Scaling of Deep Learning Workloads — Saxena et al. (2020) (arXiv:2006.13878, 2020)

## What this evaluates

Evaluates a deep learning job scheduler's ability to dynamically adjust GPU allocations and batch sizes to maximize cluster throughput and minimize job completion times. It probes how well the system handles compute-bound, communication-bound, and non-elastic workloads under varying job arrival patterns.

## Datasets

- **CIFAR100** — total ?; splits: train (-1)
- **Food101** — total ?; splits: train (-1)

## Metrics

- `SJS Efficiency` **(primary)** — range: other
  - Opt_Sch_Time / Act_Sch_Time, where Opt_Sch_Time is the sum of job lengths on a single GPU, and Act_Sch_Time is the sum of (actual GPUs used × time duration). Measures average scaling efficiency of scheduled jobs.
- `Job Drop Ratio` — range: other
  - Number of jobs dropped / Total number of jobs. Measures the proportion of jobs rejected due to resource constraints.

## Input / output format

**Input**: Job arrival requests characterized by category (compute/communication bound, elasticity range), batch size constraints (min/max), and arrival pattern (Poisson-distributed low, high, or bursty rates).

**Output**: Scheduling decisions specifying the number of GPUs allocated per job and the batch size per GPU at each optimization interval.

## Scoring recipe

```python
# Calculate Opt_Sch_Time and Act_Sch_Time from scheduler logs
opt_time = sum(job_length_on_1_gpu for job in scheduled_jobs)
act_time = sum(num_gpus_allocated[job] * job_duration[job] for job in scheduled_jobs)
sjs_efficiency = opt_time / act_time

# Calculate Job Drop Ratio
drop_ratio = num_dropped_jobs / total_arrived_jobs
```

## Common pitfalls

- Confusing GPU-only elasticity (baseline) with batch-size elasticity (proposed method).
- Applying standard system benchmarks (e.g., SPEC, Kounev metrics) instead of the paper's Opt/Act GPU time metrics.
- Assuming model accuracy or convergence speed is the primary evaluation criterion; the paper explicitly focuses on cluster throughput and job completion time.

## Evidence (verbatim from paper)

> We measure and report the performance in terms of two metrics (i) Scheduled Job Scaling (SJS) Efficiency, defined as Opt_Sch_Time / Act_Sch_Time, and (ii) Job Drop Ratio, defined as Number of jobs dropped / Total number of jobs.

## Citation

```bibtex
@misc{saxena2020elastic,
  title={Effective Elastic Scaling of Deep Learning Workloads},
  author={Saxena et al. (2020)},
  year={2020},
  note={arXiv:2006.13878}
}
```

- arXiv: 2006.13878