Eurosys Reproducibility

---
name: eurosys-reproducibility
description: Use when hardening the reproducibility story of a EuroSys paper — recording hardware and software provenance for every number, taming performance variance with repeated runs and dispersion reporting, versioning workloads and traces, and writing an availability statement that survives both double-blind review and the sysartifacts AEC.
---

# EuroSys Reproducibility

Use this while experiments are still running — reproducibility retrofitted in
deadline week is transcription, not engineering. The venue context: EuroSys
papers live on measured performance claims, the community runs a badge-granting
artifact evaluation (`sysartifacts.github.io`), and SIGOPS has publicly digested
five years of EuroSys AE lessons (sigops.org blog, 2025; rendered 2026-07-08).
A paper whose numbers cannot be regenerated by its own authors three months
later fails both review-time scrutiny and post-acceptance AE.

## The provenance ledger

Keep one machine-readable record per reported number. Minimum fields:

| Field | Why EuroSys reviewers care |
|---|---|
| Commit hash of system under test | "Which version got 2.1x?" is a real AE question |
| Baseline name + version + config | Untuned-baseline suspicion is the venue's default |
| Hardware: CPU/RAM/NIC/storage, topology | Systems results rarely transfer across boxes |
| OS/kernel, key library versions | Kernel changes move I/O and scheduler numbers |
| Workload/trace + generator seed | Trace provenance is checked, not assumed |
| Repetitions, warm-up policy | Distinguishes measurement from anecdote |
| Timestamp + raw-output path | Lets you rebuild any figure from raw logs |

## Variance discipline

Single-run numbers are the most common silent reproducibility failure in
systems evaluation:

- Repeat every headline measurement enough times to see its spread; report
  median plus an explicit dispersion measure (stdev, IQR, or min–max), and say
  in the caption which one it is.
- Isolate noise sources you control: pin frequencies, disable turbo where it
  distorts comparisons, note co-located load, randomize run order across
  systems so drift does not favor yours.
- Tail metrics (p99 and beyond) need far more samples than means; state the
  sample count whenever a tail latency is claimed.
- When a difference is within run-to-run spread, say so — EuroSys reviewers
  reward calibrated claims over uniform victory narratives.

## Automation floor

The practical bar: any figure regenerates from raw data with one command.

```bash
# Layout that keeps figures honest
experiments/
  fig7_throughput/
    run.sh        # executes the sweep, writes results/*.csv with metadata header
    plot.py       # reads results/, emits fig7.pdf — no hand-edited numbers
    results/      # raw outputs, never overwritten, one dir per run timestamp
make fig7         # the only path by which fig7.pdf ever changes
```

If a plot was ever touched manually, its provenance is broken and the AEC will
find the discrepancy before you do.

## Availability statement, two audiences

- **Review time (double-blind):** describe what exists — "an anonymized
  repository containing the system, workload generator, and run scripts
  accompanies the submission" — without leaking the lab's identity through
  URLs, paths, or commit authors.
- **Camera-ready / AE time:** replace with the DOI-backed archive and the badge
  set being sought. Restricted traces (production data, partner NDAs) need an
  honest fallback: a synthetic generator calibrated to the trace's published
  statistics, with the calibration method described.

## Restricted evidence, stated honestly

Some EuroSys evidence legitimately cannot ship — production traces under
NDA, partner clusters, proprietary workloads. The honest pattern:

- Name the restriction and its scope precisely ("the ingestion trace from
  operator X cannot be released; its summary statistics are in Table 3").
- Ship a calibrated synthetic substitute and the calibration procedure, so
  external readers can approximate the regime.
- Keep at least one headline result on fully public inputs; a paper whose
  every number depends on unreleasable data asks reviewers for faith the
  venue does not trade in.
- Never let the availability paragraph imply more openness than the AEC
  will find; the badge process makes overstatement visible in print.

## Pre-deadline reproducibility drill

One week before the paper gate, run the drill on a machine that never ran
the experiments:

1. Clean checkout, environment build from the lockfile alone — record every
   undocumented step it turns out to need.
2. Regenerate two figures end to end: one cheap, one expensive.
3. Diff regenerated numbers against the draft's numbers; investigate any
   drift beyond the reported dispersion.
4. Fix the documentation, not just the outcome — the drill's product is the
   README the AEC will eventually read.

## Quick self-test

1. Can a new student regenerate Figure 7 from a clean checkout in one command?
2. Does every table cell trace to a raw log file with hardware metadata?
3. Are repetition counts and dispersion visible for every performance claim?
4. Would the availability paragraph survive both anonymity and the AEC?

## Output format

```text
[Repro grade] regenerable / scripted-with-gaps / manual
[Ledger coverage] <numbers with full provenance / total reported numbers>
[Variance findings] <single-run claims, missing dispersion, undersampled tails>
[Workload provenance] <traces and generators with version + seed status>
[Availability draft] <review-time text and camera-ready plan>
```