Eurosys Experiments

---
name: eurosys-experiments
description: Use when designing or auditing the evaluation of a EuroSys paper — choosing workloads that earn the word realistic, tuning baselines beyond their defaults, decomposing end-to-end wins into per-mechanism gains, measuring overheads and worst cases, and sizing experiments to the claims the paper actually makes.
---

# EuroSys Experiments

Use this while the evaluation is still designable. EuroSys reviewers treat the
evaluation as the paper's testimony about itself: the design section says what
should happen, and §Evaluation is cross-examination. Plan it as a set of
questions with falsifiable answers, not as a benchmark tour.

## The four-layer evidence stack

| Layer | Question it answers | Typical EuroSys instrument |
|---|---|---|
| End-to-end | Does the system win where users live? | Full application or serving workload, realistic scale |
| Decomposition | Which mechanism buys which fraction? | Factor analysis: enable components one at a time |
| Cost | What does the win spend? | Memory/CPU/network overhead, code and ops complexity |
| Boundary | Where does it stop winning? | Adversarial mixes, saturation, failure injection |

A submission with only the first layer reads as a demo; the middle layers are
what convert "it is faster" into "we understand why it is faster", which is
the systems-research standard of proof.

## Workload realism, argued not asserted

- Prefer published traces and standard suites with named versions; when using
  a synthetic generator, calibrate it against a real distribution and say how.
- Match the workload to the claim's regime: a memory-efficiency claim needs
  memory pressure; a tail-latency claim needs load near saturation, not at 30%.
- Scale honestly: if the pitch is rack-scale, a two-node result needs an
  explicit extrapolation argument or a scoped-down claim.
- Report the workload's own parameters (skew, read ratio, arrival process) so
  the experiment is reconstructible without your cluster.

## Baseline fairness — the venue's sharpest knife

The reflexive EuroSys reviewer question is "did they tune the baseline?"
Answer it before it is asked:

- Run baselines at their documented best configuration for your hardware, and
  cite where that configuration comes from.
- Include the strongest *deployed* alternative, not only research prototypes —
  losing to a well-tuned production system on some axis is survivable;
  omitting it is not.
- Same hardware, same workload generator, same measurement harness for every
  system; any asymmetry gets a sentence of justification.
- When your system loses a metric, plot it anyway and explain the tradeoff.

## Experiment matrix as an artifact

Freeze the plan in a machine-checkable form before running:

```yaml
# eval-matrix.yaml — one row per claim the paper will make
- claim: "cuts p99 GET latency ≥40% under skewed load"
  figure: fig8
  workload: {trace: twitter-cache-2020, skew: zipf-0.99, load: 0.85sat}
  systems: [ours@v1.4, baselineA@3.2-tuned, baselineB@1.9-tuned]
  reps: 10
  metrics: [p50, p99, p999, throughput]
  status: pending
```

The matrix doubles as the reproducibility ledger (`eurosys-reproducibility`)
and later as the artifact's claims map (`eurosys-artifact-evaluation`).

## Result invalidators to design out early

Each of these has sunk otherwise strong EuroSys evaluations; each is cheap
to prevent and expensive to discover in a review:

- Measuring through a bottleneck that is not the system under test — a
  saturated client, a slow log disk, a debug build of a dependency.
- Comparing your warm system against cold baselines (or vice versa)
  because warm-up policy was never standardized across systems.
- A "scalability" curve whose x-axis grows load and resources together,
  so nothing about scaling behavior is actually isolated.
- Reporting means over runs that include documented failures — decide the
  failure-handling policy for measurements before running them.
- Config drift between the end-to-end and decomposition experiments, so
  the component gains do not sum to anything resembling the headline.

## Sizing runs for the claim

- Comparative bar charts: ≥5–10 repetitions with dispersion shown; single-run
  bars invite a one-line rejection rationale.
- Tail latencies: enough requests that the quoted percentile has support —
  a p999 from 10k requests is ten samples of noise.
- Ablations: vary one factor per experiment; a config delta of two changes
  attributes nothing.
- Timeboxing: reserve cluster time for the boundary layer explicitly; it is
  always the layer teams drop under deadline pressure and the layer reviewers
  miss loudest.

## Reporting floor for the paper

- Hardware and topology for every experiment, once, in a table the
  reader can find (§Evaluation setup), with per-figure deviations noted.
- Software versions for every system in every comparison, including yours.
- Load points quoted with the metric ("85% of measured peak throughput"),
  never as bare client counts whose meaning depends on the cluster.
- Repetition count and dispersion type in every caption that shows a
  comparison.
- A sentence on measurement methodology for anything subtle: how latency
  is timestamped, where the measurement harness sits, what it costs.

## Output format

```text
[Claim -> evidence map] <each paper claim: figure, workload, systems, reps>
[Stack coverage] end-to-end / decomposition / cost / boundary: present or missing
[Baseline fairness] <tuning provenance and deployed-alternative status>
[Realism audit] <trace provenance, load regime, scale honesty>
[Highest-value missing run] <the one experiment to schedule next>
```