Zbench

---
name: zbench
description: Measures interactive zsh latency with zsh-bench. Use for slow-shell diagnosis, performance comparisons, or regression bisection.
---

# zsh-bench Integration

Proper benchmarking of interactive zsh using [romkatv/zsh-bench](https://github.com/romkatv/zsh-bench). Measures real user-visible latency, NOT `time zsh -lic exit` (which is meaningless).

## Commands

```bash
hey zbench              # Run + display with threshold indicators (auto-compares if baseline exists)
hey zbench-save         # Run + save as baseline + append history
hey zbench-compare      # Run + explicit diff against baseline
hey zbench-check        # Exit non-zero if over threshold (for git bisect)
hey zbench-baseline     # Show saved baseline (no run)
hey zbench-history      # Show TSV history
```

All commands accept extra zsh-bench args: `hey zbench --iters 4` for quick runs. When editing the repo-local harness before rebuilding, validate with `./bin/hey zbench --iters 4`; the installed `hey` on `PATH` may still point at the previous generation. In Nushell, zbench subcommands must use `def --wrapped ... [...args]` so flags such as `--iters` pass through to `zsh-bench`.

## Metrics & Thresholds

From romkatv's blind perception study — values at or below threshold are indistinguishable from zero:

| Metric                 | Threshold | What it means                             |
| ---------------------- | --------- | ----------------------------------------- |
| `first_prompt_lag_ms`  | 50ms      | Time to see prompt after opening terminal |
| `first_command_lag_ms` | 150ms     | Time until first command can execute      |
| `command_lag_ms`       | 10ms      | Delay between Enter and next prompt       |
| `input_lag_ms`         | 20ms      | Keystroke-to-screen latency               |

Indicators: 🟢 ≤50% (headroom) · 🟡 ≤100% (imperceptible) · 🟠 ≤200% (noticeable) · 🔴 >200% (sluggish)

`exit_time_ms` is shown but **not** used for thresholds — it doesn't measure interactive performance.

## Git Bisect Workflow

Find which commit made the shell slow:

```bash
git bisect start
git bisect bad HEAD
git bisect good <known-good-commit>
git bisect run hey zbench-check
```

`zbench-check` exits non-zero when any metric exceeds its threshold.

## File Layout

```
benchmarks/zsh-bench/
├── <Host>.json              # Current baseline per host
└── history/
    └── <Host>.tsv           # Append-only history (timestamp, git_rev, metrics)
packages/zsh-bench/           # Nix package (romkatv/zsh-bench with internal/ helpers)
bin/hey.d/zbench.nu           # Nushell hey subcommands
bin/zbench-report             # Python — parse, compare, format results
```

Baselines are per-host (`MacTraitor-Pro.json`, `Seqeratop.json`) because hardware varies.

## Typical Workflow

```bash
# 1. Establish baseline on a clean build
hey zbench-save

# 2. Make zsh config changes
vim config/zsh/.zshrc
hey rebuild

# 3. Check for regressions
hey zbench                    # Shows comparison vs baseline

# 4. If satisfied, update baseline
hey zbench-save
```

## Autoresearch Optimization Prompt

For a multi-iteration optimization loop, use `pi-autoresearch` to drive small
zsh config experiments and `zsh-bench` to measure whether each experiment helped.
Do not use autoresearch instead of zsh-bench; use it as the edit → benchmark →
keep/revert loop around zsh-bench.

Run the local prompt template with `pi-autoresearch`:

```bash
pi -e npm:pi-autoresearch "$(cat .pi/prompts/zbench-autoresearch.md)"
```

If you prefer to install the extension persistently first:

```bash
pi install npm:pi-autoresearch
cat .pi/prompts/zbench-autoresearch.md
```

Canonical prompt file: `.pi/prompts/zbench-autoresearch.md`.

## Debugging Slow Startup

### Phase Timing Script

Don't guess — measure. Paste this into `zsh -c '...'` to time each phase of startup:

```zsh
zsh -c '
zmodload zsh/datetime
export XDG_CONFIG_HOME="${XDG_CONFIG_HOME:-$HOME/.config}"
export XDG_CACHE_HOME="${XDG_CACHE_HOME:-$HOME/.cache}"
export ZDOTDIR="${ZDOTDIR:-$XDG_CONFIG_HOME/zsh}"
export ZSH_CACHE="${ZSH_CACHE:-$XDG_CACHE_HOME/zsh}"
function _source { [[ -f "$1" ]] && source "$1"; }
function _cache {
  local cache_dir="$XDG_CACHE_HOME/zsh"; local cache_file="$cache_dir/$1.zsh"
  if [[ ! -f "$cache_file" ]] || [[ "$commands[$1]" -nt "$cache_file" ]]; then
    mkdir -p "$cache_dir"; "$@" > "$cache_file"; fi
  source "$cache_file"
}

t0=$EPOCHREALTIME
source $ZDOTDIR/.zshenv 2>/dev/null; t1=$EPOCHREALTIME
source $ZDOTDIR/config.zsh; t2=$EPOCHREALTIME
# ... add phases matching your .zshrc ...
source $ZDOTDIR/completion.zsh 2>/dev/null; t3=$EPOCHREALTIME
_source $ZDOTDIR/extra.zshrc; t4=$EPOCHREALTIME

printf "zshenv:     %4.0fms\n" $(( (t1-t0)*1000 ))
printf "config:     %4.0fms\n" $(( (t2-t1)*1000 ))
printf "completion: %4.0fms\n" $(( (t3-t2)*1000 ))
printf "extra:      %4.0fms\n" $(( (t4-t3)*1000 ))
printf "TOTAL:      %4.0fms\n" $(( (t4-t0)*1000 ))
'
```

Adapt phases to match the actual `.zshrc`. The gap between this total and `zsh-bench` is overhead from `/etc/zshrc` (nix-darwin generated) and deferred plugin loading.

To drill into `extra.zshrc`, time each `source` line individually — one slow alias file can dominate.

### Known Culprits (ranked by typical impact)

| Culprit                            | Typical cost  | Fix                                                                                                                                                                                |
| ---------------------------------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Redundant compinit**             | 2000-3000ms   | Ensure compinit runs exactly once. Check EOF of `.zshrc`, `/etc/zshrc`, and completion.zsh — easy to end up with 2+ calls. Use `compinit -C -d "$cache"` with 24h staleness check. |
| **Nix store globs**                | 200-400ms     | `for f in /nix/store/*foo*/*.zsh` is slow — thousands of dirs. Cache the resolved path to a file.                                                                                  |
| **Shell startup file scanning**    | 100-500ms     | Functions that `grep`/`sed` across many files at startup (e.g., fixing session files). Move to on-demand or a cron job.                                                            |
| **Uncached `eval "$(tool init)"`** | 40-100ms each | `brew shellenv`, `direnv hook zsh`, `fnm env`, `zoxide init zsh`, `entire completion zsh`. Use `_cache` pattern to write output to file, re-eval only when binary changes.         |
| **Double `brew shellenv`**         | 40-80ms       | nix-homebrew adds `eval "$(brew shellenv)"` to `/etc/zshrc`. If you handle it in `.zshenv`, set `enableZshIntegration = false` in nix-homebrew config.                             |
| **Plugin manager overhead**        | 10-40ms       | Antidote's `antidote load` does staleness checks. If static file exists, source it directly and skip antidote init entirely.                                                       |
| **Deferred plugins**               | 0ms startup   | antidote `kind:defer` is free at startup but zsh-bench won't detect `has_syntax_highlighting`/`has_autosuggestions`. This is fine.                                                 |

### The `_cache` Pattern

Central to fast startup. Already defined in `.zshrc`:

```zsh
function _cache {
  local cache_dir="$XDG_CACHE_HOME/zsh"
  local cache_file="$cache_dir/$1.zsh"
  if [[ ! -f "$cache_file" ]] || [[ "$commands[$1]" -nt "$cache_file" ]]; then
    mkdir -p "$cache_dir"
    "$@" > "$cache_file"
  fi
  source "$cache_file"
}

# Usage:
_cache zoxide init zsh        # instead of eval "$(zoxide init zsh)"
_cache direnv hook zsh        # instead of eval "$(direnv hook zsh)"
_cache entire completion zsh  # instead of source <(entire completion zsh)
```

Invalidates when the binary changes (`$commands[$1]` mtime check). Delete `~/.cache/zsh/*.zsh` to force regeneration.

### Replay Mode

Use `zsh-bench --iters 1 --scratch-dir /tmp/zbench-debug` then `dbg/replay --scratch-dir /tmp/zbench-debug` to watch what zsh-bench actually sees. For worktree experiments that should read the repo copy of `config/zsh`, run with `ZDOTDIR=$PWD/config/zsh ZSH_CACHE=$HOME/.cache/zsh`; otherwise zsh-bench may exercise the currently installed Home Manager generation.

## Key Design Decisions

- Uses zsh-bench's **non-raw output** (median of 16 iterations) for stable numbers.
- **`--raw`** gives per-iteration arrays — useful for variance analysis but not default.
- Baselines stored as JSON for easy programmatic comparison.
- History stored as TSV for easy `column -t`, `awk`, or import into spreadsheets.
- Regression detection: flags changes > 20% or > 5ms (whichever is larger).