Installs into .claude/skills of the current project.
Are you the author of Pipeline Health?
Add the live security badge to your README. It updates with every re-scan.
[](https://www.skillsdirectory.com/skills/nightgauge-pipeline-health)
---
name: pipeline-health
description: Comprehensive pipeline health analysis across 7 dimensions — token economics,
cost health, stage effectiveness, model routing, reliability, self-improvement
loop health, and pipeline velocity. Cross-references all telemetry data
sources to generate actionable findings and auto-create improvement issues.
Use periodically or after a batch of pipeline runs to assess overall pipeline health.
license: Apache-2.0
metadata:
author: nightgauge
version: "1.2.0"
source: https://github.com/nightgauge/nightgauge
chainable: true
allowed-tools: Read Write Edit Glob Grep Bash Task
context: fork
agent: pipeline-optimizer
model: haiku
---
# Nightgauge Pipeline Health
## Description
Comprehensive pipeline health analysis across 7 dimensions. Uses deterministic
Python3/jq for data extraction and metric computation; AI interprets results
into human-readable findings with severity-classified recommendations.
**For a quick point-in-time snapshot, use `/nightgauge:pipeline-audit`
instead.**
**Use Cases:**
- Weekly/bi-weekly pipeline health reviews
- Cost health and token economics analysis
- Identifying reliability patterns and failure modes
- Tracking whether past recommendations improved metrics
- Comparing current period against baseline
- Auto-creating improvement issues for actionable findings
**When to Use:**
- On a regular cadence (weekly recommended) for continuous improvement
- After significant pipeline changes to verify impact
- When pipeline costs or failure rates seem abnormal
- Before planning pipeline optimization sprints
- When you need **comprehensive analysis** — pipeline-health cross-references 7
data sources and tracks recommendations over time. For a quick efficiency
check, use `/nightgauge:pipeline-audit`.
**Relationship to Pipeline Audit:**
Pipeline health is complementary to pipeline audit — not a replacement:
| Aspect | Pipeline Audit | Pipeline Health |
| ------------ | --------------------------- | ------------------------------------------- |
| Scope | 5 analysis categories | 7 analysis dimensions |
| Data sources | Execution history primarily | Cross-references 7 data sources |
| Output | Findings + recommendations | Findings + auto-issue creation with full AC |
| Tracking | Point-in-time snapshot | Recommendation tracking over time |
| Comparison | None | Baseline period comparison |
| Use case | Quick efficiency check | Comprehensive health assessment |
## Invocation
| Tool | Command |
| ----------- | ----------------------------------------- |
| Claude Code | `/nightgauge:pipeline-health [options]` |
| Copilot | Invoke via Agent Skills extension |
| Cursor | Run via Agent Skills or direct SKILL.md |
| Headless | `claude -p "/nightgauge:pipeline-health"` |
## Arguments
### Core Options
| Argument | Description | Default |
| ------------------------- | -------------------------------------------------- | --------- |
| `--period N` | Analyze last N days | `7` |
| `--since DATE` | Start date (YYYY-MM-DD) | - |
| `--until DATE` | End date (YYYY-MM-DD) | - |
| `--dimensions DIMS` | Comma-separated dimensions to analyze | `all` |
| `--create-issues` | Auto-create GitHub issues for findings | `false` |
| `--severity LEVEL` | Min severity for `--create-issues` | `high` |
| `--dry-run` | Show what issues would be created without creating | `false` |
| `--compare-to-baseline` | Compare current vs previous period | `false` |
| `--track-recommendations` | Check past recommendation effectiveness | `false` |
| `--format FORMAT` | Output format: `summary`, `json`, `both` | `summary` |
### Examples
```bash
# Weekly health check (default: last 7 days, summary format)
/nightgauge:pipeline-health
# Analyze last 30 days
/nightgauge:pipeline-health --period 30
# Analyze specific date range
/nightgauge:pipeline-health --since 2026-02-01 --until 2026-02-14
# Only analyze cost and reliability dimensions
/nightgauge:pipeline-health --dimensions cost-health,reliability
# Preview what issues would be created
/nightgauge:pipeline-health --create-issues --dry-run
# Create issues for medium+ severity findings
/nightgauge:pipeline-health --create-issues --severity medium
# Compare current week against previous week
/nightgauge:pipeline-health --compare-to-baseline
# Track whether past recommendations improved metrics
/nightgauge:pipeline-health --track-recommendations
# JSON output for programmatic consumption
/nightgauge:pipeline-health --format json
```
---
## Prerequisites
- `python3` installed (for reliable JSONL parsing — `cat | jq` drops records)
- `nightgauge` CLI installed and authenticated (`nightgauge forge auth login`)
- `jq` installed for JSON processing (fallback data sources only)
- Git repository with `.nightgauge/` directory
- At least one data source populated (see Data Sources below)
**CRITICAL**: This skill runs headless. Do NOT use AskUserQuestion. Make
autonomous decisions or fail with clear error.
---
## Data Sources
The skill reads data from 8 sources in priority order, with graceful fallback
when sources are unavailable:
| Source | Location | What It Provides |
| ---------------------- | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Execution History | `pipeline/history/*.jsonl` | Per-run tokens, costs, durations, stage breakdowns |
| History Index | `pipeline/history/index.json` | Lightweight run summaries for fast filtering |
| Post-Pipeline Analysis | `checkout/analysis/*.json` | Model performance metrics, failure analysis |
| A/B Experiments | `checkout/analysis/experiments/*.jsonl` | Experiment results and variant comparisons |
| Past Health Reports | `pipeline/health-report-*.json` | Historical findings for recommendation tracking |
| Health Series | `checkout/health/trends.jsonl` | The one health time-series: one schema-2 entry per pipeline run, 90-day retention. `overall_score` is `null` (status `no-data`) when no dimension had data; `excluded_dimensions` names every dimension that did not contribute and why. |
`pipeline/...` locations are in the clone's pipeline state directory
(`nightgauge layout path pipeline`); `checkout/...` locations are in this
checkout's directory (`nightgauge layout path checkout`).
Each data source is optional. The skill analyzes whatever is available and notes
which sources were missing in the output report.
### Querying Dimension Trends
Use `HealthTrendsWriter.read()` from the SDK for programmatic access:
```typescript
// Last 20 runs for sparkline display
const entries = await HealthTrendsWriter.read(workspaceRoot, { limit: 20 });
// Last 30 days for trend analysis
const since = new Date();
since.setDate(since.getDate() - 30);
const entries = await HealthTrendsWriter.read(workspaceRoot, {
startDate: since,
endDate: new Date(),
});
```
---
## Analysis Dimensions
### 1. Token Economics
Identifies wasteful token usage patterns and optimization opportunities:
- Per-stage token averages and distribution
- Cache hit rate trends (prompt caching effectiveness)
- **Per-stage cache hit rate** (Issue #3804): the canonical
`cache_read / (cache_read + cache_creation + input)` ratio computed per stage,
surfaced as `perStageCacheHitRate.<stage>` metrics. Stages with no cacheable
input report no metric (treated as "no data"), never `0%`. Each stage with
enough samples whose rate falls below its resolved threshold
(`pipeline.cache.stage_alert_thresholds.<stage>`, else
`pipeline.cache.alert_threshold`, default 40%) emits a low-reuse finding —
`high` severity below 10%, `medium` otherwise. This is the same threshold and
formula the `nightgauge-pipeline-audit` skill uses, so the two surfaces
agree by construction.
- Input/output token ratios per stage
- Outlier detection (runs with abnormally high token usage)
- Redundant file reads across stages
### 2. Cost Health
Financial analysis and cost optimization:
- Total cost trends (daily, weekly)
- Per-run cost distribution with outlier detection
- Cost-per-stage breakdown
- Cost by issue size/type/priority correlation
- Projected monthly cost at current rate
### 3. Stage Effectiveness
Timing and reliability analysis per pipeline stage:
- Per-stage duration baselines (median, avg, p90, max)
- First-attempt success rates per stage
- Retry frequency and impact on total duration
- Bottleneck detection (stages >1.5x slower than average)
- Stage skip effectiveness analysis
### 4. Model Routing
AI model selection analysis. **This dimension is not computed here** — it is
the SDK's `analyzeModelRouting` (`HealthAnalysisEngine`, dimension
`model-routing`), fed per executed stage by `flattenRunRecords` (#461): each
record carries that stage's `model` and `selectionSource` from
`stages[*].model_selection`. Report the engine's `score`, `metrics` and
`findings` for this dimension verbatim; do not re-derive them from the raw
JSONL. The engine's metrics are:
- `autoSelectionSuccessRate` — success rate for stages whose dispatched model
ran unsubstituted (`selectionSource === "scheduler"`)
- `underRoutingCount` / `overRoutingCount` — lightweight model on L/XL work
that failed; heavyweight model on XS/S work that succeeded first try
- `distinctModelCount`, `model.{name}.successRate`,
`model.{name}.effectiveCostPerSuccess` — usage distribution, quality and
cost impact per model
### 5. Reliability & Failure Patterns
Failure analysis and pattern detection:
- Failure rate by stage, issue type, and complexity
- Common failure modes and root causes
- Mean time to recovery (MTTR) per failure type
- Cascading failure detection (one stage failure causing downstream failures)
- Flaky stage detection (intermittent failures)
### 6. Learning Effectiveness
Effectiveness of the pipeline's self-tuning mechanisms:
- Self-tuning action frequency and types
- Impact of self-tuning on subsequent runs
- Recommendation implementation rate
- Closed-loop effectiveness (did changes improve target metrics?)
- Stale recommendations (old findings never addressed)
#### 6.5 Skill-Catalog Usage (telemetry)
The pipeline records outcomes for the 6 core stages, but the other ~37 skills
have no outcome record. The PreToolUse(Skill) hook closes that gap by logging
invocations to `skills/usage.jsonl` in this checkout's directory
(`nightgauge layout path checkout skills/usage.jsonl`). Read the rollup with:
```bash
nightgauge skills usage --json
```
Surface as findings:
- **Never-triggered skills** (`never_seen: true`) — a skill that never fires is
usually a `description:` that isn't matching its intended triggers (fix the
description) or a genuinely dead skill (retire it).
- **Under-triggering vs. expectation** — skills you expect on a cadence
(backlog-groom, docs-watch) that show stale `last_seen`.
- **Popular skills** — the high-`trigger_count` skills worth investing in
(better gotchas, verification, scripts).
Feed never-triggered / under-triggering skills into the improvement-issue
generation below, just like any other dimension finding.
### 7. Pipeline Velocity
Throughput and efficiency trends:
- Issues completed per day/week
- Average end-to-end pipeline duration
- Time-in-stage distribution
- Queue wait times (time between stages)
- Velocity trend (improving, stable, degrading)
---
## Workflow
### Phase 0: Environment Preflight
### Phase 0: Environment Preflight
**PURPOSE**: Verify the nightgauge binary is available and the environment
is healthy before executing any pipeline stage. Skills halt immediately with
actionable instructions when this check fails.
Three-layer check:
1. **Binary discovery** — shell-level, runs before invoking the binary
2. **Doctor check** — binary-level, full environment health via `doctor --json`
3. **Project environment** — the checks the project declares under
`preflight.environment` (#101); opt-in, silent when none are declared
The binary discovery cascade below is **provider-neutral** (issue 4029): it works
identically under every adapter (Claude, Codex, Gemini, …) and references no
VSCode-extension-specific path. The host that spawns the skill (the VSCode
extension's `skillRunner`, or the Go `nightgauge` binary in auto/CLI mode)
resolves the binary authoritatively and exports `NIGHTGAUGE_BIN`, which
this cascade honors first. The remaining steps (PATH, the development-only repo
`bin/` and worktree canonical-repo `bin/`, `~/go/bin`) are fallbacks for direct
or terminal invocation.
This intentionally **diverges** from
`claude-plugins/nightgauge/hooks/lib/guard.sh` (issue 3262 → issue 4029): guard.sh is
Claude-Code-only and retains a trailing `~/.vscode/extensions/...` glob to serve
the standalone-terminal-Claude case (where no host exports `NIGHTGAUGE_BIN`).
Skills must stay portable Markdown and cannot reference that VSCode-only path —
so the shared resolution order (`NIGHTGAUGE_BIN` → PATH → gated repo → gated
canonical → `~/go/bin`) stays in sync, and only guard.sh carries the extra Claude-only
fallback. The `nightgauge preflight skill-portability` gate enforces that
no skill reintroduces a `.vscode/extensions` path, and that every cascade keeps
the `NIGHTGAUGE_DEV_REPO_BIN` gate (#2758): a foreign repository's
`bin/nightgauge` must never run.
After the cascade resolves `$BINARY`, the preflight prepends
`dirname($BINARY)` to `PATH` so subsequent bare `nightgauge ...`
invocations later in the same skill body resolve through PATH rather than
failing with "command not found" (issue 3262).
```bash
# Layer 1: provider-neutral binary discovery (NIGHTGAUGE_BIN → PATH → gated repo
# bin → gated canonical-repo bin → ~/go/bin). No VSCode-extension path — see issue 4029.
# Repo bins need NIGHTGAUGE_DEV_REPO_BIN=1 and a verified core checkout (#2758).
BINARY="${NIGHTGAUGE_BIN:-}"
[ -n "$BINARY" ] && [ ! -x "$BINARY" ] && BINARY=""
[ -z "$BINARY" ] && BINARY=$(command -v nightgauge 2>/dev/null || echo "")
if [ -z "$BINARY" ] && [ "${NIGHTGAUGE_DEV_REPO_BIN:-}" = "1" ]; then
for NG_ROOT in "$(git rev-parse --show-toplevel 2>/dev/null)" \
"$(cd "$(git rev-parse --git-common-dir 2>/dev/null)/.." 2>/dev/null && pwd)"; do
[ -x "$NG_ROOT/bin/nightgauge" ] || continue
grep -qx 'module github.com/nightgauge/nightgauge' "$NG_ROOT/go.mod" 2>/dev/null || continue
NG_ORIGIN="$(git -C "$NG_ROOT" config --get remote.origin.url 2>/dev/null)"
case "${NG_ORIGIN%.git}" in
git@github.com:nightgauge/nightgauge | https://github.com/nightgauge/nightgauge | \
ssh://git@github.com/nightgauge/nightgauge) BINARY="$NG_ROOT/bin/nightgauge"; break ;;
esac
done
fi
[ -z "$BINARY" ] && [ -x "$HOME/go/bin/nightgauge" ] && BINARY="$HOME/go/bin/nightgauge"
if [ -z "$BINARY" ]; then
echo "ERROR: nightgauge binary not found via NIGHTGAUGE_BIN, PATH or ~/go/bin" >&2
echo "" >&2
echo "Install via: go install github.com/nightgauge/nightgauge/cmd/nightgauge@latest" >&2
echo "Or download from: https://github.com/nightgauge/nightgauge/releases" >&2
echo "Run \`nightgauge doctor\` after installing to verify your environment." >&2
exit 1
fi
# Make the resolved binary callable via bare `nightgauge ...` later in this skill body (issue 3262).
export PATH="$(dirname "$BINARY"):$PATH"
# Per-repo GitHub token: export GH_TOKEN so every `gh` call in this skill
# authenticates as the *current repo's* configured user instead of the machine-
# global `gh auth` active account. `forge auth token` resolves per-repo (config
# github_user/token → GITHUB_TOKEN → `gh auth token --user <github_user>`), so
# concurrent sessions/workspaces owned by different GitHub users each use their
# own token. GH_TOKEN outranks GITHUB_TOKEN and the gh keyring. Only resolve when
# unset (an upstream env value — e.g. the VSCode extension's terminal env —
# wins); never fail preflight over it. Mirrors hooks/lib/guard.sh (sync, issue 3262).
if [ -z "${GH_TOKEN:-}" ]; then
IB_REPO_TOKEN="$("$BINARY" forge auth token 2>/dev/null || true)"
if [ -n "$IB_REPO_TOKEN" ]; then
export GH_TOKEN="$IB_REPO_TOKEN"
export GITHUB_TOKEN="${GITHUB_TOKEN:-$IB_REPO_TOKEN}"
fi
unset IB_REPO_TOKEN
fi
# Layer 2: full environment health check via doctor --json
DOCTOR_RESULT=$("$BINARY" doctor --json 2>/dev/null)
DOCTOR_EXIT=$?
if [ "$DOCTOR_EXIT" -eq 2 ]; then
echo "ERROR: Environment check failed — nightgauge doctor reports broken environment." >&2
FAILED_CHECKS=$(printf '%s\n' "$DOCTOR_RESULT" | jq -r '.failed_checks[]? // empty' 2>/dev/null)
if [ -n "$FAILED_CHECKS" ]; then
echo "Failing check(s): $(echo "$FAILED_CHECKS" | tr '\n' ',' | sed 's/,$//')" >&2
fi
printf '%s\n' "$DOCTOR_RESULT" | jq -r '.errors[]' >&2 2>/dev/null || true
INSTALL_MSG=$(printf '%s\n' "$DOCTOR_RESULT" | jq -r '.install_instructions // empty' 2>/dev/null)
[ -n "$INSTALL_MSG" ] && echo "$INSTALL_MSG" >&2
exit 1
fi
if [ "$DOCTOR_EXIT" -eq 1 ]; then
echo "WARNING: Environment has non-critical issues:" >&2
printf '%s\n' "$DOCTOR_RESULT" | jq -r '.warnings[]' >&2 2>/dev/null || true
# Continue — warnings do not block skill execution
fi
# Layer 3: project-declared environment checks (preflight.environment, #101).
# Opt-in: with none declared this prints nothing and continues. The verb
# writes one `[environment-halt]` line per failed check, with its remedy.
if ! "$BINARY" preflight environment >/dev/null; then
echo "[environment-halt] Project environment check failed — fix the environment, not the code." >&2
exit 1
fi
```
**Exit codes from `nightgauge doctor`**:
| Code | Meaning | Skill behavior |
| ---- | ------------------------------- | --------------------------------------- |
| 0 | Healthy | Continue |
| 1 | Degraded (warnings only) | Continue with warning printed to stderr |
| 2 | Broken (required checks failed) | Halt immediately with error details |
**Layer 3** halts the stage when any check declared under
`preflight.environment` in the workspace's `.nightgauge/config.yaml` fails
(exit 1) or the declaration is invalid (exit 2). The failure is in the
operator's environment, not the code: do not edit code to work around it, do
not run the remedy yourself, and end the stage. The pipeline records it as
`environment_halt` and holds the issue for an operator.
Do not theorize about the cause beyond what is printed here — the failing
check name (`failed_checks`) and its `.errors[]` message are the complete
diagnosis (#277).
---
### Phase 1: Parse Arguments and Detect Data Sources
#### Step 1.1: Parse Arguments
Extract options from invocation:
```bash
PERIOD_DAYS=7
SINCE_DATE=""
UNTIL_DATE=""
DIMENSIONS="all"
CREATE_ISSUES=false
MIN_SEVERITY="high"
DRY_RUN=false
COMPARE_BASELINE=false
TRACK_RECOMMENDATIONS=false
OUTPUT_FORMAT="summary"
# Parse arguments from invocation
# --period N: set PERIOD_DAYS
# --since YYYY-MM-DD: set SINCE_DATE
# --until YYYY-MM-DD: set UNTIL_DATE
# --dimensions DIMS: set DIMENSIONS (comma-separated)
# --create-issues: set CREATE_ISSUES=true
# --severity LEVEL: set MIN_SEVERITY
# --dry-run: set DRY_RUN=true
# --compare-to-baseline: set COMPARE_BASELINE=true
# --track-recommendations: set TRACK_RECOMMENDATIONS=true
# --format FORMAT: set OUTPUT_FORMAT
```
#### Step 1.2: Validate Arguments
```bash
# Validate date formats
if [ -n "$SINCE_DATE" ]; then
if ! printf '%s\n' "$SINCE_DATE" | grep -qE '^[0-9]{4}-[0-9]{2}-[0-9]{2}$'; then
echo "ERROR: --since must be YYYY-MM-DD format (got: $SINCE_DATE)"
exit 1
fi
fi
if [ -n "$UNTIL_DATE" ]; then
if ! printf '%s\n' "$UNTIL_DATE" | grep -qE '^[0-9]{4}-[0-9]{2}-[0-9]{2}$'; then
echo "ERROR: --until must be YYYY-MM-DD format (got: $UNTIL_DATE)"
exit 1
fi
fi
# Validate dimensions
VALID_DIMS="token-economics,cost-health,stage-effectiveness,model-routing,reliability,self-improvement,pipeline-velocity"
if [ "$DIMENSIONS" != "all" ]; then
for dim in $(echo "$DIMENSIONS" | tr ',' '\n'); do
if ! printf '%s\n' "$VALID_DIMS" | grep -q "$dim"; then
echo "ERROR: Unknown dimension '$dim'. Valid: $VALID_DIMS"
exit 1
fi
done
fi
# Validate --dry-run requires --create-issues
if [ "$DRY_RUN" = true ] && [ "$CREATE_ISSUES" = false ]; then
echo "WARNING: --dry-run has no effect without --create-issues"
fi
```
#### Step 1.3: Locate Data Sources
Check for each data source with priority ordering and graceful fallbacks:
```bash
SOURCES_FOUND=()
SOURCES_MISSING=()
# Check each data source
PIPELINE_DIR="$(nightgauge layout path pipeline)"
HISTORY_DIR="${PIPELINE_DIR}/history"
if ls "${HISTORY_DIR}"/*.jsonl 2>/dev/null | head -1 > /dev/null; then
SOURCES_FOUND+=("execution-history")
else
SOURCES_MISSING+=("execution-history")
fi
if [ -f "${HISTORY_DIR}/index.json" ]; then
SOURCES_FOUND+=("history-index")
else
SOURCES_MISSING+=("history-index")
fi
if [ -f "$(nightgauge layout path checkout health/trends.jsonl)" ]; then
SOURCES_FOUND+=("health-series")
else
SOURCES_MISSING+=("health-series")
fi
ANALYSIS_DIR="$(nightgauge layout path checkout analysis)"
if ls "${ANALYSIS_DIR}"/*.json 2>/dev/null | head -1 > /dev/null; then
SOURCES_FOUND+=("post-pipeline-analysis")
else
SOURCES_MISSING+=("post-pipeline-analysis")
fi
if ls "${ANALYSIS_DIR}"/experiments/*.jsonl 2>/dev/null | head -1 > /dev/null; then
SOURCES_FOUND+=("ab-experiments")
else
SOURCES_MISSING+=("ab-experiments")
fi
if ls "${PIPELINE_DIR}"/health-report-*.json 2>/dev/null | head -1 > /dev/null; then
SOURCES_FOUND+=("past-health-reports")
else
SOURCES_MISSING+=("past-health-reports")
fi
if [ ${#SOURCES_FOUND[@]} -eq 0 ]; then
echo "WARNING: No pipeline data sources found."
echo " Expected: ${HISTORY_DIR}/*.jsonl (primary)"
echo " Run some pipeline stages first to generate data."
exit 0
fi
echo "Data sources found: ${SOURCES_FOUND[*]}"
echo "Data sources missing: ${SOURCES_MISSING[*]}"
```
---
### Phase 2: Data Aggregation (Deterministic)
**IMPORTANT**: Use Python3 for data extraction. Do NOT read raw JSONL into AI
context — extract only aggregated metrics into `/tmp/health_*.json` intermediate
files.
#### Step 2.1: Extract Execution History
When execution history JSONL files are available:
```python
# /tmp/health_extract_history.py — deterministic metric extraction
import json, glob, subprocess, sys
from collections import defaultdict
from pathlib import Path
from datetime import datetime, timedelta
HISTORY_DIR = subprocess.run(
["nightgauge", "layout", "path", "pipeline", "history"],
capture_output=True, text=True, check=True,
).stdout.strip()
PERIOD_DAYS = int(sys.argv[1]) if len(sys.argv) > 1 else 7
SINCE_DATE = sys.argv[2] if len(sys.argv) > 2 else ""
UNTIL_DATE = sys.argv[3] if len(sys.argv) > 3 else ""
# Compute date range
if not SINCE_DATE:
SINCE_DATE = (datetime.now() - timedelta(days=PERIOD_DAYS)).strftime("%Y-%m-%d")
if not UNTIL_DATE:
UNTIL_DATE = datetime.now().strftime("%Y-%m-%d")
# Parse all JSONL records within date range
records = []
for f in sorted(glob.glob(f"{HISTORY_DIR}/*.jsonl")):
file_date = Path(f).stem
if file_date < SINCE_DATE or file_date > UNTIL_DATE:
continue
with open(f) as fh:
for line in fh:
line = line.strip()
if not line:
continue
try:
r = json.loads(line)
if r.get("record_type") == "run":
records.append(r)
except json.JSONDecodeError:
pass
# Build stage-level metrics
STAGE_NAMES = ["issue-pickup", "feature-planning", "feature-dev",
"feature-validate", "pr-create", "pr-merge"]
stage_durations = defaultdict(list)
stage_statuses = defaultdict(lambda: defaultdict(int))
stage_tokens = defaultdict(lambda: {"input": [], "output": [],
"cache_read": [], "cache_creation": []})
for r in records:
for stage in STAGE_NAMES:
s = r.get("stages", {}).get(stage, {})
if not s:
continue
d = s.get("duration_ms", 0)
if d > 0:
stage_durations[stage].append(d)
stage_statuses[stage][s.get("status", "unknown")] += 1
ps = r.get("tokens", {}).get("per_stage", {}).get(stage, {})
for key in ("input", "output", "cache_read", "cache_creation"):
val = ps.get(key, 0)
if val > 0:
stage_tokens[stage][key].append(val)
# Build per-run metrics
run_metrics = []
for r in records:
t = r.get("tokens", {})
run_metrics.append({
"issue_number": r.get("issue_number"),
"title": r.get("title", ""),
"outcome": r.get("outcome"),
"total_duration_ms": r.get("total_duration_ms", 0),
"started_at": r.get("started_at", ""),
"total_input": t.get("total_input", 0),
"total_output": t.get("total_output", 0),
"total_cache_read": t.get("total_cache_read", 0),
"total_cache_creation": t.get("total_cache_creation", 0),
"estimated_cost_usd": t.get("estimated_cost_usd", 0),
"labels": r.get("labels", []),
"size": r.get("size"),
"type": r.get("type"),
"priority": r.get("priority"),
"stage_durations": {
s: r.get("stages", {}).get(s, {}).get("duration_ms", 0)
for s in STAGE_NAMES
},
"stage_statuses": {
s: r.get("stages", {}).get(s, {}).get("status", "skipped")
for s in STAGE_NAMES
},
})
output = {
"runs_analyzed": len(records),
"date_from": SINCE_DATE,
"date_to": UNTIL_DATE,
"run_metrics": run_metrics,
"stage_durations": {k: v for k, v in stage_durations.items()},
"stage_statuses": {k: dict(v) for k, v in stage_statuses.items()},
"stage_tokens": {k: {tk: tv for tk, tv in v.items()}
for k, v in stage_tokens.items()},
}
with open("/tmp/health_execution.json", "w") as f:
json.dump(output, f, indent=2)
print(f"Extracted {len(records)} runs to /tmp/health_execution.json")
```
```bash
python3 /tmp/health_extract_history.py ${PERIOD_DAYS} "${SINCE_DATE}" "${UNTIL_DATE}"
```
#### Step 2.2: Extract the Health Series
```bash
# Extract health trend scores using Go binary (replaces Python script)
nightgauge health trends --limit 50 --json > /tmp/health_scores.json 2>/tmp/health_scores_warn.txt
SCORE_COUNT=$(jq 'length' /tmp/health_scores.json 2>/dev/null || echo 0)
echo "Extracted ${SCORE_COUNT} health trend entries to /tmp/health_scores.json"
[ -s /tmp/health_scores_warn.txt ] && cat /tmp/health_scores_warn.txt
```
#### Step 2.3: Extract Post-Pipeline Analysis
```bash
# Aggregate analysis files into single summary
ANALYSIS_DIR="$(nightgauge layout path checkout analysis)"
python3 -c "
import json, glob
analyses = []
for f in sorted(glob.glob('${ANALYSIS_DIR}/*.json')):
try:
with open(f) as fh:
analyses.append({'file': f, 'data': json.load(fh)})
except (json.JSONDecodeError, IOError):
pass
with open('/tmp/health_analysis.json', 'w') as f:
json.dump({'records': len(analyses), 'analyses': analyses[-20:]}, f, indent=2)
print(f'Extracted {len(analyses)} analysis files')
"
```
#### Step 2.5: Extract A/B Experiment Data
```bash
ANALYSIS_DIR="$(nightgauge layout path checkout analysis)"
python3 -c "
import json, glob
experiments = []
for f in sorted(glob.glob('${ANALYSIS_DIR}/experiments/*.jsonl')):
try:
with open(f) as fh:
for line in fh:
line = line.strip()
if line:
experiments.append(json.loads(line))
except (json.JSONDecodeError, IOError):
pass
with open('/tmp/health_experiments.json', 'w') as f:
json.dump({'records': len(experiments), 'experiments': experiments[-30:]}, f, indent=2)
print(f'Extracted {len(experiments)} experiment records')
"
```
#### Step 2.6: Extract Gate Metrics (Issue #1412)
```bash
# Extract gate metrics with hit rates using Go binary (replaces jq aggregation)
nightgauge health gate-metrics --json > /tmp/health_gates.json 2>/tmp/health_gates_warn.txt
GATE_COUNT=$(jq 'length' /tmp/health_gates.json 2>/dev/null || echo 0)
if [ "$GATE_COUNT" -eq 0 ]; then
echo "No gate metrics found — gate hit-rate analysis will be skipped"
else
echo "Extracted ${GATE_COUNT} gate aggregates to /tmp/health_gates.json"
fi
[ -s /tmp/health_gates_warn.txt ] && cat /tmp/health_gates_warn.txt
```
---
### Phase 3: Analysis per Dimension
Read `/tmp/health_*.json` intermediate files (produced in Phase 2) and compute
metrics per dimension. Only analyze dimensions specified by `--dimensions`.
For each dimension, compute all metrics listed in the Analysis Dimensions
section above. Write per-dimension results to `/tmp/health_dim_*.json`.
**IMPORTANT**: Use `estimated_cost_usd` directly from records — do NOT
recalculate from token counts.
#### Overall score
A dimension marked `insufficient-data` / `no-data` has **no score** — it is
rendered `N/A` and it does **not** vote. The overall score is the weighted mean
over the dimensions that have data, re-normalised to their weights:
```text
overall = round( Σ score[d] × weight[d] / Σ weight[d] ) over d with data
clamped to [0, 100]; N/A when no dimension has data
```
Never substitute a placeholder (50, "fair") for a starved dimension — that
asserts mediocrity from absence of evidence. This is the same rule the SDK
`HealthAnalysisEngine` runs (`computeOverallHealthScore`); both are pinned to
the shared corpus in `overall-score-corpus.json` next to this file, so change
the corpus first when the rule changes.
Correlate findings across dimensions to identify root causes:
- Cost spike + model routing change → model selection caused cost increase
- Reliability drop + self-tuning action → self-tuning may have degraded quality
- Velocity decrease + stage duration increase → bottleneck stage identified
- Token increase + new data source → context size growth detected
Write cross-references to `/tmp/health_cross_ref.json`.
---
### Phase 5: Finding Generation (Probabilistic)
AI interprets computed metrics into severity-classified findings:
```json
{
"dimension": "token-economics | cost-health | stage-effectiveness | model-routing | reliability | self-improvement | pipeline-velocity",
"severity": "critical | high | medium | low | info",
"title": "Short description of finding",
"description": "Detailed explanation with supporting metrics",
"estimated_impact": "$X.XX/run or X% improvement",
"recommendation": "Specific actionable step to address this",
"acceptance_criteria": ["AC1", "AC2"],
"related_dimensions": ["other-dimension"]
}
```
**Severity Classification Rules:**
| Severity | Criteria |
| ---------- | --------------------------------------------------------- |
| `critical` | >50% waste or >30% failure rate |
| `high` | >25% waste or >15% failure rate or clear cost opportunity |
| `medium` | >10% waste or noticeable inefficiency |
| `low` | Minor optimization opportunity |
| `info` | Informational — positive trend or baseline observation |
**Gate Hit-Rate Finding Rules (Issue #1412):**
Read `/tmp/health_gates.json` (produced in Step 2.6). For each gate in the
`stage-effectiveness` dimension:
- Gate with `hit_rate > 0.5` (>50% of runs catch defects) → `info`: "High-value
gate — [gate_name] catches defects in [hit_rate]% of runs"
- Gate with `hit_rate < 0.05` AND `invocations > 10` (<5% catch rate with 10+
samples) → `low`: "Rarely-catching gate — [gate_name] catches defects in only
[hit_rate]% of runs; consider repositioning or reviewing necessity"
- No gate data (empty `/tmp/health_gates.json`) → skip gate finding entirely
---
### Phase 6: Recommendation Tracking
Only executes when `--track-recommendations` is passed.
Load past health reports from `health-report-*.json` in the clone's pipeline
state directory (`nightgauge layout path pipeline`).
For each past recommendation:
1. Check if a corresponding issue was created and completed
2. Compare the target metric before and after the recommendation period
3. Classify as: implemented-effective, implemented-ineffective, not-implemented,
partially-implemented
Output recommendation tracking results alongside current findings.
---
### Phase 7: Baseline Comparison
Only executes when `--compare-to-baseline` is passed.
Compare current period against the immediately preceding period of equal length:
- Current period: `--since` to `--until` (or last N days)
- Baseline period: Equal duration immediately before current period
For each dimension, compute delta (absolute and percentage) for key metrics.
Flag significant regressions (>10% worse) and improvements (>10% better).
---
### Phase 8: Issue Creation
Only executes when `--create-issues` is passed and `--dry-run` is false.
#### Step 8.1: Filter by Severity
Only create issues for findings at or above `--severity` threshold.
#### Step 8.2: Check for Existing Issues
Before creating, search for duplicates across both open and recently closed
issues:
```bash
REPO="${NIGHTGAUGE_REPO:-$(nightgauge git repo-slug)}"
# Open matches (`issue list --search` is scoped to open issues)
nightgauge issue list --repo "$REPO" --search "${FINDING_KEYWORDS}" --json | jq '[(. // [])[] | {number, title}]'
# Closed matches (no list verb reaches closed issues: GraphQL search)
nightgauge forge graphql -f query='query($q: String!) { search(query: $q, type: ISSUE, first: 20) { nodes { ... on Issue { number title } } } }' \
-f q="repo:$REPO is:issue is:closed ${FINDING_KEYWORDS}" | jq '.data.search.nodes'
# Open epics
nightgauge forge issue list --repo "$REPO" --labels "type:epic" --json | jq '[.[:30][] | {number, title}]'
```
Skip creation if a matching issue exists.
#### Step 8.3: Create Issues
When creating 3+ findings as issues, create as an epic with sub-issues:
```bash
# Create epic
nightgauge forge issue create \
--title "epic: Pipeline health findings — ${SUMMARY}" \
--body "${EPIC_BODY}" \
--labels "type:epic,priority:high,size:L"
# Create sub-issues. --labels takes label NAMES and resolves them itself
# (#1214); an unknown name fails before anything is created.
nightgauge issue create-sub <EPIC_NUMBER> \
--title "<FINDING_TITLE>" --body "<FINDING_BODY>" \
--labels "type:bug,priority:medium"
# Sync epic to project board
nightgauge project add <EPIC_NUMBER>
```
For `--dry-run`, output what would be created without actually creating.
For fewer than 3 findings, create standalone issues.
#### Step 8.4: Report Created Issues
Output list of created issues with numbers, titles, and URLs.
---
### Phase 9: Output Report
#### Step 9.1: Write JSON Report
Write the structured report with
`nightgauge layout write pipeline health-report-YYYY-MM-DD.json` (content on
stdin, or `--from FILE` after writing a temp file outside the git directory):
```json
{
"schema_version": "1.0",
"analysis_period": {
"from": "YYYY-MM-DD",
"to": "YYYY-MM-DD",
"period_days": 7,
"data_sources_found": ["execution-history", "health-series"],
"data_sources_missing": ["ab-experiments"]
},
"summary": {
"total_cost_usd": 45.0,
"avg_cost_per_run": 3.0,
"total_runs": 15,
"success_rate": 0.87,
"avg_duration_minutes": 45,
"total_tokens": 5000000,
"cache_hit_rate": 0.82
},
"overall_score": 77,
"dimensions": {
"token-economics": { "status": "analyzed", "metrics": {}, "findings": [] },
"cost-health": { "status": "analyzed", "metrics": {}, "findings": [] },
"stage-effectiveness": {
"status": "analyzed",
"metrics": {},
"findings": []
},
"model-routing": { "status": "no-data", "metrics": {}, "findings": [] },
"reliability": { "status": "analyzed", "metrics": {}, "findings": [] },
"self-improvement": { "status": "no-data", "metrics": {}, "findings": [] },
"pipeline-velocity": { "status": "analyzed", "metrics": {}, "findings": [] }
},
"summary": {
"total_cost_usd": 45.0,
"avg_cost_per_run": 3.0,
"total_runs": 15,
"success_rate": 0.87,
"avg_duration_minutes": 45,
"total_tokens": 5000000,
"cache_hit_rate": 0.82,
"gate_effectiveness": {
"total_invocations": 75,
"by_gate": [
{
"gate_name": "build",
"invocations": 15,
"catches": 2,
"hit_rate": 0.13
},
{
"gate_name": "unit-tests",
"invocations": 15,
"catches": 3,
"hit_rate": 0.2
},
{
"gate_name": "integration-tests",
"invocations": 15,
"catches": 1,
"hit_rate": 0.07
},
{
"gate_name": "type-check",
"invocations": 15,
"catches": 4,
"hit_rate": 0.27
},
{
"gate_name": "lint",
"invocations": 15,
"catches": 1,
"hit_rate": 0.07
}
]
}
},
"findings": [],
"cross_references": [],
"recommendations": [],
"baseline_comparison": null,
"recommendation_tracking": null,
"issues_created": [],
"created_at": "ISO-8601"
}
```
#### Step 9.2: Write Markdown Summary
Write the human-readable report with
`nightgauge layout write pipeline health-report-YYYY-MM-DD.md`:
```
PIPELINE HEALTH REPORT
═══════════════════════════════════════════════════════════
Analysis Period: 2026-02-07 to 2026-02-14 (7 days, 15 runs)
Data Sources: execution-history, health-series (5/7 available)
SUMMARY
───────────────────────────────────────────────────────────
Total Cost: $45.00 (avg $3.00/run)
Total Tokens: 5,000,000 (avg 333,333/run)
Success Rate: 87% (13/15 runs)
Avg Duration: 45 minutes
Cache Hit Rate: 82%
DIMENSION HEALTH
───────────────────────────────────────────────────────────
Overall: ████████░░░░ 77% (5 of 7 dimensions scored)
Token Economics: ██████████░░ 83% [HEALTHY]
Cost Health: ████████░░░░ 67% [WARNING]
Stage Effectiveness: █████████░░░ 75% [HEALTHY]
Model Routing: ░░░░░░░░░░░░ N/A [NO DATA]
Reliability: ███████████░ 92% [HEALTHY]
Self-Improvement: ░░░░░░░░░░░░ N/A [NO DATA]
Pipeline Velocity: ████████░░░░ 67% [WARNING]
FINDINGS (6 total: 1 high, 3 medium, 2 info)
───────────────────────────────────────────────────────────
[HIGH] Cost Health: 3 runs exceeded $5.00 threshold
→ Estimated impact: $4.50/week savings with model routing
→ Recommendation: Route S/XS issues to Sonnet
...
GATE HIT-RATES (Issue #1412)
───────────────────────────────────────────────────────────
Gate Invocations Catches Hit Rate
───────────────── ─────────── ─────── ────────
type-check 15 4 27%
unit-tests 15 3 20%
build 15 2 13%
integration-tests 15 1 7%
lint 15 1 7%
RECOMMENDATIONS (sorted by impact)
───────────────────────────────────────────────────────────
1. Route simple issues to Sonnet (~$4.50/week, medium effort)
2. Optimize feature-validate tests (~20% stage speedup, low effort)
3. Enable prompt caching for docs/ (~10% token reduction, low effort)
───────────────────────────────────────────────────────────
Run with --create-issues to auto-create GitHub issues for findings.
Next: /nightgauge:pipeline-health --create-issues --severity medium
```
If `--format json`, write only JSON. If `--format summary`, write only markdown.
If `--format both`, write both files and output the markdown summary.
---
### Self-Assessment Epilogue
### Self-Assessment Epilogue
**PURPOSE**: Evaluate whether this skill's instructions matched reality during
this execution. This phase is **non-blocking** — skip entirely if any main phase
failed. A perfectly working skill produces **no output** from this phase.
> See [docs/SKILL_SELF_ASSESSMENT.md](https://github.com/nightgauge/nightgauge/blob/main/docs/SKILL_SELF_ASSESSMENT.md) for
> the full strategy, synthesis algorithm, and integration architecture.
#### When to Skip
- Any prior phase exited with an error before completing
- The skill was cancelled or timed out
- Running in `--dry-run` mode
#### Step 1: Evaluate Execution Friction
Review the execution that just completed. Answer these questions honestly:
1. **Command failures**: Did any command, script, or binary call in the skill
instructions fail? (e.g., script not found, wrong flags, missing binary)
2. **Workarounds**: Did you have to deviate from the skill instructions to
accomplish the goal? (e.g., used `gh` directly because a referenced script
didn't exist, skipped a step that referenced a nonexistent file)
3. **Stale references**: Did any file path, function name, API endpoint, or tool
referenced in the instructions not exist in the current codebase?
4. **Unclear instructions**: Were any instructions ambiguous enough that you had
to guess at the intended behavior?
5. **Missing instructions**: Was there a significant step you had to figure out
on your own that should have been documented in this skill?
**If ALL answers are "no" — write nothing and complete the skill normally.** The
goal is silence when everything works.
#### Step 2: Write Assessment Record
Only if friction was detected in Step 1. Write a single JSON file,
`assessments/{STAGE_NAME}-${ISSUE_NUMBER}.json` in the clone's pipeline state
directory, through the binary (JSON on stdin; never a direct file write):
```bash
ISSUE_NUMBER="${NIGHTGAUGE_ISSUE_NUMBER:-$(git branch --show-current | sed -n 's#^[^/]*/\([0-9]*\)-.*#\1#p')}"
: "${ISSUE_NUMBER:?set NIGHTGAUGE_ISSUE_NUMBER or check out the issue branch}"
nightgauge layout write pipeline "assessments/{STAGE_NAME}-${ISSUE_NUMBER}.json" <<'JSON'
{ ... the record below ... }
JSON
```
The assessment record MUST follow this schema:
```json
{
"schema_version": "1",
"skill": "{STAGE_NAME}",
"skill_file": "skills/nightgauge-{STAGE_NAME}/SKILL.md",
"issue_number": 42,
"timestamp": "2026-03-10T14:30:00Z",
"friction": [
{
"type": "command_failure",
"severity": "high",
"description": "hooks/lib/add-to-project.sh not found — script was deleted",
"skill_line_hint": "claude-plugins/nightgauge/hooks/lib/add-to-project.sh <issue-number>",
"actual_resolution": "Used gh api graphql to add issue to project board directly",
"suggested_fix": "Replace with: nightgauge project add <issue-number>"
}
]
}
```
**Friction types**: `command_failure`, `workaround`, `stale_reference`,
`unclear_instruction`, `missing_instruction`
**Severity levels**:
- `high` — instruction is **broken**. Required manual workaround to complete.
- `medium` — instruction is **misleading**. Agent adapted without user help.
- `low` — instruction is **suboptimal**. No functional impact.
**Rules**:
- **One record per execution** — multiple friction items go in the `friction`
array, not separate files.
- **Be specific** — quote the exact instruction text that was wrong. Not "some
commands didn't work" but "Step 5.2 calls `hooks/lib/add-to-project.sh` but
this script was deleted in commit 65915701."
- **Suggest the fix** — every finding MUST include `suggested_fix` with the
concrete SKILL.md change needed. Not "update the docs" but "replace
`hooks/lib/add-to-project.sh <N>` with `nightgauge project add <N>`."
- **Don't invent friction** — only report issues you actually encountered during
this execution. Do not speculate about potential problems.
#### Step 3: Validate and Complete
```bash
# Validate JSON if written
ISSUE_NUMBER="${NIGHTGAUGE_ISSUE_NUMBER:-$(git branch --show-current | sed -n 's#^[^/]*/\([0-9]*\)-.*#\1#p')}"
: "${ISSUE_NUMBER:?set NIGHTGAUGE_ISSUE_NUMBER or check out the issue branch}"
ASSESSMENT_FILE="$(nightgauge layout path pipeline "assessments/{STAGE_NAME}-${ISSUE_NUMBER}.json")"
if [ -f "$ASSESSMENT_FILE" ]; then
python3 -m json.tool "$ASSESSMENT_FILE" > /dev/null 2>&1 || \
echo "WARNING: Assessment record is not valid JSON" >&2
fi
```
---
## Output Format
### JSON Schema
See Phase 9 Step 9.1 for the complete JSON report structure.
### Report Files
| File | Format | When Written |
| ---------------------------------------- | -------- | ------------------------------------- |
| `pipeline/health-report-YYYY-MM-DD.json` | JSON | `--format json` or `--format both` |
| `pipeline/health-report-YYYY-MM-DD.md` | Markdown | `--format summary` or `--format both` |
Both are written with `nightgauge layout write pipeline <name>`.
---
## Error Handling
| Condition | Action |
| ------------------------------------------------- | -------------------------------------------------------------- |
| No data sources found | Output friendly message, exit 0 |
| Malformed JSONL records | Skip bad records, report count at end |
| python3 not installed | Error with install instructions |
| forge not authenticated | Error with `nightgauge forge auth login` instructions |
| `--create-issues` without findings | Output "No findings at severity threshold" |
| Insufficient data for dimension | Mark dimension as "insufficient-data", skip analysis |
| `--track-recommendations` with no past reports | Output "No past reports found for tracking" |
| `--compare-to-baseline` with insufficient history | Output "Insufficient history for baseline comparison" |
| `issue create-sub` label error | Use `nightgauge forge graphql` addLabelsToLabelable separately |
| Duplicate finding exists | Skip creation, report as already tracked |
| `--dry-run` without `--create-issues` | Warn, continue with analysis only |
---
## Pipeline Position
```
UTILITIES (not part of main pipeline)
/nightgauge:pipeline-health
↑
Use on regular cadence (weekly recommended)
(pipeline/ = the clone's pipeline state dir: nightgauge layout path pipeline)
Reads: pipeline/history/*.jsonl
Reads: checkout/health/trends.jsonl (the health series)
Reads: checkout/analysis/*.json
Reads: checkout/analysis/experiments/*.jsonl
Reads: pipeline/health-report-*.json
Writes: pipeline/health-report-YYYY-MM-DD.json (via layout write)
Writes: pipeline/health-report-YYYY-MM-DD.md (via layout write)
```
This is a standalone utility skill. It does not affect pipeline state and can be
run at any time without interfering with active pipeline runs.
---
## Configuration
This skill reads configuration from `.nightgauge/config.yaml`:
| Config Key | Default | Description |
| ------------------------- | ------- | ----------------------------------- |
| `project.number` | - | GitHub Project number for issues |
| `health.default_period` | `7` | Default `--period` value (days) |
| `health.default_severity` | `high` | Default severity for issue creation |
| `health.auto_create` | `false` | Auto-create issues without flag |
---
**Author:** nightgauge **License:** Apache-2.0
