Perfup

---
name: perfup
description: "Autonomous performance optimization: research, PoC, benchmark, implement, review, PR"
disable-model-invocation: true
allowed-tools:
  - Bash
  - Read
  - Write
  - Edit
  - Glob
  - Grep
  - Task
  - WebSearch
  - WebFetch
  - Skill
  - TaskCreate
  - TaskUpdate
  - TaskList
  - TaskGet
  - EnterPlanMode
  - ExitPlanMode
  - AskUserQuestion
---

# /perfup — Autonomous Performance Optimization

Inspired by [karpathy/autoresearch](https://github.com/karpathy/autoresearch): you are an autonomous performance researcher for vllm-mlx. You propose optimizations, benchmark them, keep what works, discard what doesn't, and ship a production PR.

## Key Files

- **Results log**: `reports/perfup-results.tsv` — append-only experiment log (commit, metric, status, description)
- **Optimization queue**: `memory/knowledge/perf_optimization_queue.md` — ranked list of candidates
- **Memory index**: `memory/MEMORY.md` — what's been done, what's known
- **Benchmark script**: `scripts/benchmark_engines.py`
- **Model for benchmarking**: Check memory for current model path. If unavailable, ask user.

## The 6 Phases

### Phase 1: Research

Read existing state, then discover new opportunities.

1. Read `memory/knowledge/perf_optimization_queue.md` and `memory/MEMORY.md`
2. If `$ARGUMENTS` is provided (e.g. `/perfup decode`), focus on that area. Otherwise broad search.
3. Scan codebase for optimization opportunities:
   - Use Task(subagent_type=Explore) on critical paths
   - Search for TODO/FIXME/PERF/HACK comments
   - Check ml-explore/mlx-lm recent releases (`gh release list --repo ml-explore/mlx-lm --limit 5`)
4. WebSearch for latest MLX inference optimizations if needed
5. Produce candidate list, each with: problem, solution, estimated impact, effort, coverage, risk

### Phase 2: Prioritize

Score and rank. Persist to memory.

1. Score each candidate (1-5 per axis):
   - **Impact**: Performance gain magnitude (5 = >2x)
   - **Ease**: Implementation effort (5 = <1 day)
   - **Coverage**: Models that benefit (5 = all)
   - **Safety**: Regression risk (5 = zero)
2. Sort by composite = Impact x Ease x Coverage x Safety
3. Update `memory/knowledge/perf_optimization_queue.md`:
   - Completed items → "Completed" section (date + results)
   - Failed/rejected → "Rejected" section (reason)
   - Active queue → "Queue" section with [P0]-[P3] tags
4. Present top 3 to user. Wait for confirmation before proceeding.

### Phase 3: PoC Experiment Loop

**This is the core loop. Inspired by autoresearch: try, measure, keep or discard. Repeat.**

```
SETUP:
  git checkout -b perfup/<optimization-name>
  Record baseline metrics (run benchmark on current code)
  Initialize reports/perfup-results.tsv if not exists

LOOP:
  1. Implement minimal PoC change in code
  2. git commit -m "perfup: <brief description>"
  3. Run benchmark: python3.12 scripts/benchmark_engines.py (or custom)
     Redirect output: > reports/perfup-run.log 2>&1
  4. Extract metrics from log (TTFT, decode tok/s, etc.)
  5. Record to reports/perfup-results.tsv:
     commit<TAB>decode_tps<TAB>ttft_ms<TAB>status<TAB>description
  6. DECISION:
     - If metric improved: KEEP. Log "keep" status. This is the new baseline.
     - If metric same or worse: DISCARD. Log "discard". git reset --hard to previous keep.
     - If crashed: Log "crash". Try to fix (1-2 attempts). If unfixable, discard and move on.
  7. If improvement confirmed and significant (>5%): break loop → Phase 4
  8. If no candidate works after trying top 3: inform user and stop.
```

**Rules for the loop:**
- Each PoC should be MINIMAL — smallest change that tests the hypothesis
- Benchmark must run on a REAL model (not mocks)
- If benchmark takes too long or model not loaded, ask user
- Do NOT ask "should I continue?" between iterations — just keep going
- DO stop and ask if you need user action (download model, start server, etc.)

### Phase 4: Full Implementation

PoC validated. Now build it properly.

1. Clean up or rewrite the PoC code for production quality
2. Enter plan mode — design clean architecture, tests, docs
3. Implement:
   - Clean code, proper error handling, logging
   - Unit tests matching existing patterns in `tests/`
   - No hacks, no dead code
4. Run full test suite: `python3.12 -m pytest tests/ -v`
5. Run benchmark again — confirm improvement matches PoC

### Phase 5: Review Loop

Independent review via Codex.

1. Invoke: `/review-loop <description of optimization>`
2. Address all findings (P0 = blocker, P1 = should fix, P2 = nice to have)
3. After review passes, run final benchmarks on all relevant models
4. Update README/docs with new benchmark numbers if applicable

### Phase 6: PR & Ship

1. Ensure all changes are on `perfup/<name>` or `feat/<name>` branch
2. Push to `raullenchai` remote (NEVER origin, NEVER main directly)
3. Create PR:
   ```
   gh pr create --repo raullenchai/vllm-mlx --base main
   ```
   PR body must include:
   - **Summary**: What was optimized and why
   - **Benchmark results**: Before/after table from perfup-results.tsv
   - **Test plan**: How to verify
4. Update memory:
   - Move optimization to "Completed" in `perf_optimization_queue.md` with PR#, date, confirmed speedup
   - Remove from todo if applicable
5. Present PR URL to user

---

## Results TSV Format

```
commit	decode_tps	ttft_ms	status	description
a1b2c3d	68.4	245	baseline	current main branch
b2c3d4e	72.1	240	keep	reduce redundant mx.eval in decode loop
c3d4e5f	67.9	248	discard	speculative prefill chunking
d4e5f6g	0.0	0	crash	fused MoE kernel (import error)
```

## Focus Areas

If `$ARGUMENTS` provided:
- `ttft` — Time to first token (prefill optimization)
- `decode` — Decode throughput (tok/s)
- `tools` — Tool calling accuracy/reliability
- `accuracy` — Model output quality
- `memory` — Memory usage / longer contexts
- `prefill` — Prefill speed
- `cache` — Cache hit rate / prompt reuse
- No argument → broad research across all areas

## Important Rules

1. **Benchmark proves everything.** No optimization ships without measured improvement.
2. **Memory is truth.** `perf_optimization_queue.md` is the canonical record of what's tried/works/failed.
3. **Git discipline.** Feature branch → PR on raullenchai/vllm-mlx. Never push to main.
4. **Keep it simple.** A small improvement with clean code beats a large improvement with ugly code. Removing code for equal performance is a win.
5. **Ask only when blocked.** Don't ask "should I continue?" — just keep iterating. Ask only for user actions (model download, server restart, etc.).