Themisio Io Sharing Eval

---
name: themisio-io-sharing-eval
description: Evaluates a policy-driven I/O sharing framework for burst buffers by measuring how effectively it allocates bandwidth, maintains fairness, and reduces interference across concurrent workloads. It probes the system's ability to enforce primitive and composite sharing policies under varying load conditions and compares performance against baseline schedulers. Use when the user wants to benchmark on ThemisIO Benchmark & Application Suite, or asks about evaluating this task. Reports sustained I/O throughput.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.11615
  bibtex_key: karrels2023themisio
  confidence: high
---

# themisio-io-sharing-eval

> Fine-grained Policy-driven I/O Sharing for Burst Buffers — Karrels et al. (2023) (arXiv:2306.11615, 2023)

## What this evaluates

Evaluates a policy-driven I/O sharing framework for burst buffers by measuring how effectively it allocates bandwidth, maintains fairness, and reduces interference across concurrent workloads. It probes the system's ability to enforce primitive and composite sharing policies under varying load conditions and compares performance against baseline schedulers.

## Datasets

- **ThemisIO Benchmark & Application Suite** — total ?; splits: test (-1)

## Metrics

- `sustained I/O throughput` **(primary)** — range: other
  - Measured in GB/s or MB/s, calculated as the median throughput over 1-second sampling intervals during concurrent execution. Represents the effective bandwidth allocated to each job under a specific sharing policy.
- `I/O-induced slowdown` — range: percent
  - Percentage reduction in application performance when running with a background I/O job under a sharing policy, compared to exclusive access. Calculated as (T_exclusive - T_shared) / T_exclusive * 100%.

## Input / output format

**Input**: Concurrent I/O requests from multiple jobs/applications, tagged with metadata (user, group, node count/job size), submitted to a remote-shared burst buffer managed by ThemisIO.

**Output**: Time-series throughput measurements (MB/s or GB/s) sampled at 1-second intervals, latency per I/O operation, and aggregate slowdown percentages relative to baseline exclusive execution.

## Scoring recipe

```python
def compute_slowdown(t_exclusive, t_shared):
    return ((t_exclusive - t_shared) / t_exclusive) * 100

def compute_throughput_metrics(samples):
    median_tp = median(samples)
    std_tp = std(samples)
    return median_tp, std_tp

# For each policy P and job pair:
# 1. Run jobs concurrently, record 1s-interval throughput samples
# 2. Compute median and std dev for each job
# 3. Compare against exclusive baseline to get slowdown %
```

## Common pitfalls

- Throughput is sampled at 1-second intervals, which masks microsecond-level I/O operation latencies and introduces measurement artifacts.
- Asynchronous I/O workloads may show slowdowns exceeding theoretical fair-share bounds due to non-linear shifts from computation-bound to I/O-latency-bound execution.
- Initial startup overhead (~10 seconds) in ThemisIO can cause temporary throughput imbalances, especially in composite policy hierarchies.

## Evidence (verbatim from paper)

> The sustained I/O throughput of ThemisIO is comparable to the state of the art production system. The sustained peak throughput of ThemisIO is 19.8GB / s, which is 13.5% and 13.7% higher than that of GIFT and TBF, respectively. ThemisIO also shows a lower standard deviation of the throughput of Job 2 with a value of 504MB / s, compared with 626MB / s for GIFT and 845MB / s for TBF.

## Citation

```bibtex
@misc{karrels2023themisio,
  title={Fine-grained Policy-driven I/O Sharing for Burst Buffers},
  author={Karrels et al. (2023)},
  year={2023},
  note={arXiv:2306.11615}
}
```

- arXiv: 2306.11615