Hr Benchmarking

---
name: hr-benchmarking
description: Benchmark equivalent work with repeated timing, CPU and memory measurements under stated inputs and controls, and check whether saved results support a performance claim. Use when asked to benchmark, to measure which option is faster or uses less memory, or to time agent or model runs. Not for public model leaderboard research.
---

# Benchmarking

Follow the repository and host's commands, resource limits, authorization and
landing rules. Reuse an existing workload and runner before building a harness.
Read [the tool reference](references/tools.md) when selecting a benchmark,
profiler or correctness check, or interpreting its measurements.

## Define the comparison

State the question, baseline, candidate, representative input and timed region.
Exercise ordinary supported inputs and cases that disable an optimization.
State any restrictions behind a performance claim.

For a requested candidate set, record its inventory and distinguish
configurations or capacity modes. Keep untested, invalid and unsupported
candidates visible with reasons. A subset comparison establishes results only
for its tested scope, not a winner across the full inventory.

Test the newest stable release of each candidate. When the version installed
or deployed differs, say so in the results, and ask which to test if the
request does not settle it. When told to use an older version, use it and
offer the newest as an alternative.

Before running, list the selected engines, workloads, metrics, controls and
repetitions. Count the planned runs explicitly, including controls and stages;
state which decision each retained dimension helps make. Choose combinations
for that decision rather than automatically running every combination. Use a
broader matrix when its required coverage is stated. State the benchmark's
expected wall-clock time, waits included, before the first run.
Name the measurement unit: request, operation, batch or whole command. State
whether setup, startup, compilation, input loading, retries and cleanup count.

Record exact runtime, compiler, dependency and runner versions, build flags,
input identity and source revisions. Freeze copied quality and control artifacts;
record their hashes and the executed artifact's identity. Agreement across
runtimes alone does not verify the intended artifact. Keep these fixed across
repeated arms; reject mixed versions when resuming a checkpoint.

Compare runtime knobs within the same build. Use equivalent outputs, release
settings and comparable tuning. An untuned arm compares configurations rather
than implementations.

Keep warm and cold conditions explicit. Reset state or use fresh processes
where the question requires independence. Record warmup and cache conditions.
A reused service needs equivalent state across arms, not an arbitrary restart.

## Confirm the work

Validate outputs, consumed results and error counts outside the timed region
where practical. Confirm requests arrived and asynchronous work was awaited.
Include failures, timeouts and unresolved attempts in both arms' denominators.
A faster wrong result or skipped operation is not a performance improvement.

Identify the limiting resource from CPU versus wall time, resource counters or
a separate profile. Check that throughput is possible given the machine and
input. When the generator limits the result, report that limit.

Before changing a component to remove a suspected cost, attribute time to that
component in the real workload with a profile or a counter. For fast feedback,
benchmark each changed piece through its actual production entry point with
representative fixed inputs and equivalent consumed outputs. Do not benchmark a
copied implementation or stub out work material to the comparison.

Compare each piece before and after against its immediate predecessor. Before
running, declare integration milestones when changed pieces enter the consuming
workload. Keep that workload before any piece changes as the original baseline.
At those milestones, compare the integrated workload with both its immediate
predecessor and the original baseline to track cumulative effects. Plan applicable
caller and integration checks for copying, serialization, startup and orchestration
costs that isolation excludes. Run those checks at the milestones and report
results or specific coverage limits. Re-time the change in the workload that uses
it at those milestones; a full-application run is not required after every piece.
An isolated gain does not establish an end-to-end gain. A setting that ran 7.6
times faster in isolation changed a 33 s indexing run by 0.1 s.

Select metrics for the question and limiting resource: wall time or latency,
memory, disk I/O, CPU, throughput, errors and output size as relevant. State each
metric's unit, process scope and interval; distinguish peak process memory from
retained memory, logical bytes from physical storage I/O, and warm from cold cache
state. Cache hits and delayed writeback can hide storage costs; record unavailable
counters and attribution limits rather than inferring physical I/O from logical
bytes. Do not require irrelevant additional counters; retain the CPU and memory
observations required below.

## Measure without changing the experiment

Keep measured files, toolchains and services unchanged, and keep other work on
the machine from competing with the run.

Prefer removing competing work to raising priority. Record effective CPU and
I/O scheduling policies and memory protections; keep them equal across arms.
Use normal scheduling and all available CPUs for ordinary measurements. Omit
automatic `nice` and `taskset` wrappers; verify inherited priority and affinity
rather than assuming their omission resets either. Use restricted affinity or
changed priority only for representative deployment settings, an explicit resource
agreement or a stated sensitivity question. Record the reason and effective values.
Preserve frozen protocols and report constrained results within their tested scope.
Read [scheduling controls](references/tools.md#scheduling-controls) before
choosing them.

Apply worker limits to requested processing concurrency. Report sampled OS-thread
counts separately; runtime background threads are not processing workers. A
thread count alone does not establish CPU contention; keep it diagnostic unless
the experiment separately justifies using it to reject a run.

Before collecting results, set the criteria for rejecting a run: competing
load, swapping and memory-limit or OOM events, each with its metric and
threshold. Check the machine against them before each arm and watch them
throughout the measured interval. A run that fails them, or whose inputs change, is rejected even when the benchmark caused the
failure. A rejection invalidates that measurement; it does not by itself show
an implementation defect. Keep rejected runs outside the accepted sample and
report them; a later passing run neither erases a rejection nor loosens the
criteria.

Attribute CPU, load and resident memory to the recorded benchmark process tree
or resource scope before calling them competing work. Expected benchmark
utilization is not contention, and total host CPU and load average include it.
Keep an activity counter diagnostic until separate evidence shows its cutoff
matters. Collect boundary counters adjacent to the measured work, with
unrelated probes outside that interval, and record sample times. A counter
interval extending past completion cannot by itself establish overlap; keep
post-run observations diagnostic and let the machine settle before the next arm.

Before claiming negligible observer impact or a small decisive gain, test
sensitivity to observation separately with balanced controls and a preset
precision criterion. Retain inconclusive controls; do not subtract observer CPU
from workload wall time. For remote agent or model evaluations, record the
timing regime and concurrent local work.

Screen correctness and quality first, then use one run per selected arm to
explore timing and resource behavior. Preserve screened-out results and their
reasons. Label screening exploratory; one run does not support a performance
claim. A correctness-only check can use one candidate run; report its coverage.

Before exploratory runs, define the decision-relevant signal for selecting
comparisons for confirmation. Choose comparisons that meet that signal, plus
decision-critical comparisons, for repeated confirmation; keep necessary controls.
Set the confirmation plan before its runs and update the explicit run count
when the scope changes.

The default stopping rule for a performance claim is 3 runs per arm. Add 2 more
per arm only when the arms' ranges overlap or the effect is near the noise,
meaning the gap is within the variation criterion set below. 5 per arm is the
cap. If 5 runs per arm leave the comparison unresolved, report it as
inconclusive rather than adding runs. Use another count only when the user asks
for it or a run is too long to repeat 3 times, and declare it before the first
run, with the reason. Alternate the arms in every run so drift reaches each arm.

For a performance claim, assess run-to-run variation from those alternating runs.
Treat variation as already known only when earlier repeated runs of the
same variant, build and input used the same workload, host, run conditions and
variation statistic, and met the repetition count or stopping rule declared for
this claim. Otherwise include an A/A arm,
which runs the same variant twice in the same alternation. Its spread is that
comparison's own noise floor, and it can supply the preset variation statistic.
Before the first run used in that claim, set a reproducible variation criterion
(statistic and threshold), and the stopping rule if it differs from the default.

For short or tail metrics, such as the longest stall in a run, give the
threshold an absolute floor or use a percentile or count metric; a percentage
of a single small maximum can fail on the workload's own variation.
Retain every result. Keep pilots that informed the plan separate from the
claim's sample. Capture raw results, exit status, elapsed time, CPU and memory
observations. Report monitoring overhead separately from the
measured workload; batch compatible probes when they dominate elapsed time.
Report the selected repetition count or stopping rule, variation statistic and
threshold, sample count, median and range. Always give the run count and the
range with any central value, never a bare average. Apply the preset
criterion and report a gap within run-to-run variation as no clear change. Profile
separately from the runs used to claim a speed difference.

When using saved evidence for a performance claim, bind archived inputs, source,
cohort and results to their recorded identities and artifact hashes. Validate
historical reports against that frozen archive; later changes in the live
checkout alone do not invalidate historical evidence. Verify run identities and
stages against the declared plan.

Recompute qualification from recorded raw counters and preset limits; an accepted
summary or changed label is insufficient. Reject missing, duplicate or
misclassified required observations. When an evidence reader exists, put these
checks and focused corruption regressions there. After a rebase or merge
touches the reader or a check it imports, run the integrated reader over the
saved archive before landing, and compare its accepted and excluded records
with the frozen reader's. A shared check changed on the base branch can reject
records that the frozen reader accepted.

Record memory by what it measures. Whole-process peak RSS includes startup,
inputs and dependencies. Heap snapshots describe retained objects; allocation
profiles describe allocation activity. Forced-GC probes are separate from
ordinary latency measurements. Match retention semantics and payloads when
comparing caches; equal requested capacities need not mean equal memory use.
For cache-retention probes, keep the cache owner reachable through the final
observation; otherwise collection of the owner can be mistaken for release by
the cache operation.

For WeakRef-based retention probes, use bounded allocation batches and calibrated
collection boundaries after creating or dereferencing targets. Avoid observer
arrays or locals that keep measured values strongly reachable; preserve the
intended owner lifetime. Report live payload, cache occupancy and process RSS
separately. Keep this collection protocol outside ordinary latency runs; yielding
or forced collection alone does not guarantee that a target is reclaimed.

Batch latency percentiles describe batches, not individual requests.

## Report and decide

Keep the full workload, versions, conditions, repetitions, output identity,
errors and measurement units in the measurement record. Include excluded costs,
coverage limits, collector overhead, scheduling, resource caps, pre-run
machine checks and rejected attempts there.

Pull requests and other change summaries show the workload, baseline and
candidate, measured results, relevant checks and limits that affect the claim.
Omit operational measurement details unless the reader needs them to interpret
the result or asks for them. A claimed difference with unknown limits, unequal
work or missing repetitions is unverified or inconclusive.

Show numeric before/after results in a table with a benefit column containing
percent change or a times improvement. Use the metric's direction and label
worsening, unchanged or inconclusive results. With a zero or missing baseline,
show an absolute change or state that the benefit cannot be calculated.

There is no minimum gain size. A favorable difference counts as a gain when it
lies outside the variation criterion declared before the first run (the comparison's
own run-to-run spread, from its A/A arm or already-known variation, as a
percentage of the baseline) and the outputs are identical. Report a gap inside
that criterion as no clear change, and name the statistic and threshold beside
it. An unfavorable difference outside it is a regression; label it worsening.
Report memory and CPU costs beside the gain. If the variation criterion cannot support a percentage, keep its result
separate and label unsupported differences inconclusive. Changing presentation
does not change frozen acceptance or rejection gates.

Show a comparison of three or more variants or conditions as a chart beside the
before/after table, unless every row is no clear change. Choose the chart for the
question:

| Question                         | Chart                                      |
| -------------------------------- | ------------------------------------------ |
| Repeated runs of each variant    | Dot plot of every run, mean marked         |
| Conditions crossed with variants | Grouped bars, baseline as a reference line |
| Ranking of options               | Sorted horizontal bars                     |
| Resource cost (CPU, memory)      | Bars per variant, one panel per resource   |
| Change with load or time         | Line chart                                 |

Put units and run counts on the axes, start bars at zero, mark a gap inside the
variation criterion as no clear change, and give the chart an explicit background colour so
it reads the same on a light or a dark page.
In a pull request, the headline chart and when it may be skipped come from
`hr-writing-pr`; this table applies to any other chart.

For agent, model, rule or skill comparisons, distinguish development cases
from held-out cases and report runs per case. Check each case for a ceiling: a
case every arm already passes cannot show a pass-rate improvement. Report it
apart from the cases that could, and report any metric the case does not
saturate (time, cost, graded quality) on its own. Include a one-line instruction
control when claiming an improvement from a rule or skill. Keep helper,
setup and retry costs visible. Before collecting results, define a per-case
contribution threshold. If a case exceeds it, report results both with and
without that case.

For bug-review evaluations, freeze the supported input contract and label
procedure before calls. A passing reference suite does not certify a clean
control. Adjudicate additional findings against that contract. A confirmed extra
defect makes the affected specificity estimate unavailable; retain earlier
receipts and introduce a new control version. Leave disputed domains unresolved.

Report provider completion, output-format compliance and independently checked
artifact acceptance separately. Distinguish observed call time from verification
and rework time. Keep returned usage units separate from attributable account
charges; unknown attribution remains unknown.

Record every experiment, rejected ones included, with the observed result and
what was decided. End with adopt, reject or inconclusive, and the evidence for that
decision. Any implementation or publication still follows the user's scope.