Regime Scan

---
name: regime-scan
description: Scan the whole US market once a day for trend direction and sentiment turns. Folds index trend, breadth, VIX term structure, credit spreads, and defensive rotation into one 🟢/🟡/🔴 state plus divergence flags, logging each day so the slope (the real turn signal) shows across runs. Use to gauge market direction and whether sentiment is rolling over (index near highs but internals weakening) before adding risk vs raising cash. Triggers on "scan the market", "market trend", "sentiment turn", "market health check", "is the market topping", "risk-on or risk-off", "breadth", "market regime", "tape health", "should I de-risk". The level ABOVE momentum-scan, which finds which NAMES work; this judges whether the MARKET is healthy. NOT for single-ticker analysis (use yfinance), picking stocks to buy (use momentum-scan / base-breakout-scan), or generic market-timing explanations.
---

# regime-scan

A **market-level** daily read, the layer above the four name-level scans. It answers three questions that move at different speeds:

1. **Trend**: the direction of the primary trend (slow: weeks–months)
2. **Breadth**: whether the trend is healthy or narrowing (the early-warning layer; divergences show up here first)
3. **Sentiment / Credit**: whether fear/greed is stretched and positioning fragile (fast: days)

**A turn tends to show up as internals deteriorating while the index still prints highs.** The scan encodes that two ways: it raises **divergence flags** (breadth not confirming, equal-weight lagging, credit rolling over, defensive rotation, vol-curve inversion), and it logs each US market day once to `state/history.csv`, so the **slope** of breadth / VIX / credit across days (the real turn signal) stays visible beyond today's number.

This formalizes the "regime gate + cohort dashboard" idea from `methodology.md` (2026-06-03) and the breadth/narrowing observations in `macro.md` into one daily action.

## What it pulls (all yfinance, ~516 tickers, one batched download)

- **Indices**: SPY, QQQ vs 50/200DMA + 200DMA slope; RSP/SPY (equal- vs cap-weight = the narrowing proxy)
- **Breadth**: % of the **S&P 500 universe** (`state/breadth_universe.txt`, ~500 names across all 11 sectors, regenerated quarterly by `build_universe.py`) above their 50/200DMA, plus 52-week new-highs − new-lows
- **Vol / sentiment**: ^VIX level + 5-day change; ^VIX/^VIX3M **term structure** (backwardation = acute stress)
- **Credit / rotation**: HYG/LQD (HY vs IG; credit tends to lead equities at turns); defensive (XLU/XLP/XLV) vs offensive (XLK/XLY/XLC) relative strength

**Dependencies** (auto-fetched by `uv run --with`): Python ≥ 3.10, `yfinance>=1.3,<2`, `pandas>=2` (which pulls in numpy transitively; the script imports only pandas/yfinance). No persistent venv.

`<SKILL_DIR>` below is the directory containing this `SKILL.md`.

## Run

```bash
# Standard daily run
uv run --with 'yfinance>=1.3,<2' --with 'pandas>=2' \
  python <SKILL_DIR>/scripts/scan.py

# Inspect the daily state log (no new scan); watch the slope of state/breadth/vix
... python <SKILL_DIR>/scripts/scan.py --show-history

# Machine-readable
... python <SKILL_DIR>/scripts/scan.py --format json

# Longer slope/relative-strength window (default 20 sessions ≈ 1 trading month)
... python <SKILL_DIR>/scripts/scan.py --lookback 30

# Signal-quality report + outcome grading (see "Signal quality & outcomes"). Re-run quarterly.
uv run --with 'yfinance>=1.3,<2' --with 'pandas>=2' --with 'numpy>=1.24,<3' \
  python <SKILL_DIR>/scripts/backtest_outcomes.py
```

## Parameters

| Flag | Default | Notes |
|---|---|---|
| `--lookback` | 20 | Sessions for the relative-strength windows (RSP/SPY, credit, defensive rotation). ~1 trading month. Raise for slower, less noisy reads. **The 200DMA trend-gate slope is NOT covered**: it stays fixed at 20d in code so tuning the RS window can't re-tune the trend gate (whose deadband is calibrated for 20d). |
| `--format` | markdown | `markdown` or `json`. |
| `--verbose` | — | Add the full 10-signal table. The daily read shows non-🟢 votes only. |
| `--show-history` | — | Print the daily state log and exit. |
| `--clear-history` | — | Wipe `history.csv` (no confirmation). |
| `--no-save` | — | Don't append this run to history. |
| `--save-stale` | — | Save even on a weekend / NYSE holiday. History rows key on the **data's session date** (SPY's last bar), so such a save rewrites the prior session's row with identical data; the default skip avoids that redundant write. |

## Refreshing the breadth universe

The breadth pool is the **live S&P 500 constituents**, pulled from Wikipedia (yfinance can't return index membership: `^GSPC` exposes no constituents attribute, and an ETF's `funds_data.top_holdings` caps at the top 10). It's a **quarterly snapshot**: breadth's whole signal is the *slope* of "% above 50DMA" across days, and churning the universe between runs would inject compositional noise into that slope. Freeze it within a quarter; regenerate at quarter boundaries.

**Fresh-clone bootstrap**: `state/breadth_universe.txt` is gitignored as a regenerable cache, so a fresh checkout doesn't have it; run the command below once before the first scan. Until then `scan.py` warns and skips the breadth signals rather than crashing, which guts the early-warning layer.

```bash
# Regenerate state/breadth_universe.txt  (+ a dated state/breadth_universe.<YYYY>-Q<N>.txt snapshot)
uv run --with 'pandas>=2' --with lxml --with requests \
  python <SKILL_DIR>/scripts/build_universe.py

# Preview the per-sector breakdown, write nothing
... python <SKILL_DIR>/scripts/build_universe.py --dry-run
```

The generator dash-normalizes class shares (`BRK.B`→`BRK-B`), prints a per-sector count, and backs the prior list up to `breadth_universe.bak.txt`. As a guard it **refuses to overwrite the live file if it parses fewer than 400 names** (a truncated fetch or a Wikipedia re-layout), so a bad fetch can't shrink the breadth pool the daily scan depends on. The original hand-curated 97-name list lives at `state/breadth_universe.legacy97.txt`. `scan.py` reads every non-`#` line of `breadth_universe.txt`, so you can hand-edit it (it ignores `#` header lines).

## How to read the output

**Data line**: first under the title, it names the session the read reflects. `rebuilt from 30m intraday bars` is the workaround for Yahoo's late daily bar succeeding (^VIX3M ran 0.4% off its settled close in the 2026-10-02 replay, everything else within 0.03%) and needs no mention. A `⚠️ Stale data` warning means the rebuild failed too: the whole read, and the history row it files under the earlier session, is a day old. Lead with that in plain words.

**State banner**: one of three, in escalation order (mirrors the methodology ladder):

- 🟢 **RISK-ON**: trend gate on + layers confirm + ≤1 divergence. *Trend healthy, hold per rules; new money can scale in on pullbacks.*
- 🟡 **CAUTION**: trend still up but ≥2 divergence flags (or breadth weakening). *Tighten trail stops, raise cash buffer; new money only on pullbacks, never chase 🔴.*
- 🔴 **RISK-OFF**: either the trend gate is off (SPY below / rolling 200DMA), **or** price is still above 200DMA but ≥4 internals broke (the "price hasn't dropped yet but internals are already rotting" late-stage top). *Cut gross exposure, let trail stops take over; don't bottom-fish an unconfirmed bounce.*

**Confirmed (2-day) line**: the state under a 2-run-day confirmation rule (`confirmed_state` in JSON). The raw daily label chatters at the score threshold (6 of the first 10 logged transitions were single-day whipsaws), so **conviction-funnel and premarket read this line for sizing; a first-day flip is "watch, don't act"**. The raw label above it remains the honest daily reading and is what history.csv logs; the confirmation costs one run-day of detection lag on genuine turns.

**Score**: sum of 10 per-signal votes (🟢 +1 / ⚪ 0 / 🔴 −1) across the four layers, with the 🟢/⚪/🔴 split in the banner. A blunt gauge; the **divergence flags and the trajectory matter more than the absolute score**.

**Non-🟢 votes**: only the ⚪/🔴 votes print, one line each with the raw reading — a 🟢 vote needs no explanation. The full 10-signal table is behind `--verbose`.

**⚠️ Turn warnings (divergence flags)**: the turn detector. Each fires *only while the uptrend is intact* (a broken internal under an already-broken tape is the bear, not a divergence):
- Breadth divergence: SPY near its 52w high but < 50% of names above their 50DMA
- Narrowing rally: RSP/SPY falling (mega-cap-only rally)
- Credit weakening: HYG/LQD rolling over while stocks hold
- Defensive rotation: defensives outrunning offensives **and the gap deepening vs 5 sessions ago** (2026-07-31 retune: as a pure level alarm it was on 83% of days, chronic and habituating; the score *vote* stays level-based and only the flag became a change alarm, so flag base rates before/after that date aren't comparable)
- Vol-curve inversion: VIX > VIX3M
- VIX 5-day spike

None alone is a sell; **2–3 stacking = de-risk**. This layer is the skill's founding requirement: catch sentiment *turning* before price confirms.

**State strip + Trend lines**: the trajectory layer, as glyphs instead of a table (2026-07-31 redesign). The strip is the last ~14 run-days of state (🟢🟢🟡🟢…, today last); below it, one sparkline per watched series (breadth, RSP/SPY, VIX, VIX/VIX3M term, credit, defensive−offensive) with the current value, a ↗/→/↘ arrow vs ~5 runs ago, and ⚠️ on any series whose divergence flag is firing. **Read the slope**: a one-day snapshot can't tell you if sentiment is turning; the multi-day drift can. The full numeric log stays in `--show-history` (which now ends with the same dashboard).

**Presenting the result**: the dashboard is already the readable form — relay it verbatim (keep the code block, or alignment breaks) rather than re-tabulating it, then add a short interpretation in the conversation's language: what the state is, which flag/series is the live concern, and what would change the call. Don't re-list every number the dashboard already encodes.

Write the interpretation for a reader with **no finance background**. Translate every term into everyday language the moment you use it — "breadth 60%" means "60% of the 500 big stocks are still in short-term uptrends, so fewer stocks are participating"; "defensive rotation" means "money moving into utilities/groceries/pharma, the stocks people hide in when nervous"; "VIX 17.9" means "the market's fear gauge; under 20 is calm". State plainly what, if anything, the reader should do — usually "nothing".

## How it ties to the journal rules

- The trend gate (SPY > rising 200DMA) is the mechanical **"market top"** from `methodology.md` 2026-06-03: it flips you out rather than predicting the top.
- The divergence flags are the **early warning**: 🟡 means *tighten trail stops / raise cash*, a step short of *dump everything*. Consistent with "don't predict the top — build a position that survives without predicting it."
- Breadth < ~60% + gate-off is the **kill switch** (`methodology.md` 2026-06-03): cut gross exposure.
- Pairs with `regime-scan` → `momentum-scan`: read the **market** here first (are we in? add or trim?), then read **names** there.

## Signal quality & outcomes (scripts/backtest_outcomes.py)

The script grades the signal itself in two separate halves: **signal quality** (readable now, even on weeks of data) and **SPY forward returns** (the grading framework is locked; its strata stay marked `⚠︎thin` until each holds ~8 independent 20-session windows, so conclusions accrue with the log rather than getting invented from it). Re-run quarterly alongside the sister scans' `backtest_outcomes.py`.

First report (2026-07-31, 36 readings; findings about the *signal*, not about returns):

1. **The daily label chatters at the state boundary.** 11 spells over 36 run-days, 36% of them a single run-day; 10 raw transitions, of which a 2-day confirmation rule removes **6 as whipsaws** at a 1-run-day detection lag (3-day: 8 removed, 2-day lag). The score oscillating around the RISK-ON/CAUTION threshold flips the label without the tape changing, and every flip reached downstream readers (funnel sizing, premarket framing) as a real regime change. **Fixed 2026-07-31**: the banner/JSON now carry a `Confirmed (2-day)` state (see "How to read the output"); funnel and premarket size off that line, and a first-day flip reads "watch, don't act". The raw daily label remains what history.csv logs.
2. **The Defensive-rotation flag was chronic: on 83% of days (longest streak 14).** An always-on warning can't be tested (no contrast group) and habituates the reader; premarket's 2026-06-09 miss (the flag was footnoted, then led the tape) is the same lesson from the other side. **Fixed 2026-07-31**: the flag now also requires the rotation to be *deepening* (> +0.5pp vs 5 sessions ago), a change alarm matching the skill's own "read the slope" doctrine; the score vote stays level-based. Flag base rates in the next report straddle this change; compare on-shares within one convention. Credit weakening (22%) and VIX spike (17%) were healthy and episodic already.
3. **Forward returns verify the pipeline, nothing more yet.** ~2 independent 20-session windows exist so far, all inside one regime; the by-state / by-score / by-flag / transition-day tables print with thin-markers, and you should read them as pipe-verification until a few genuine regime turns are on record.

## Known limitations

- **Breadth universe is the S&P 500 (~500 names), a quarterly snapshot**, smaller than the full ~4,000-name market, and it applies *today's* membership to past prices (a mild survivorship bias, standard for a forward-looking gauge; a real backtest would need point-in-time membership). Broad enough to smooth the breadth %, and self-refreshing via `build_universe.py`; edit `state/breadth_universe.txt` to retune.
- **Yahoo's daily bar can be late**: since 2026-09-02 the just-closed session's bar often isn't out by the evening run. All 21 evening runs from 09-02 to 10-02 were therefore filed under the previous session (rows key on SPY's last bar), honestly labeled but a day late. The scan now rebuilds that session from 30-minute intraday bars, ^VIX and the other indices included, and says so on the Data line.
- **No intraday / real-time**: the scan uses daily closes. A run during market hours sees a *partial* today-bar in the MAs/breadth (the script warns); the post-close run overwrites that row, since history keys on the data's session date rather than the wall clock.
- **Votes are mechanical**: thresholds follow convention, without optimization. Treat the output as a structured *dashboard to interpret* rather than a trade signal. The signal compounds across **days of history**; a single run is a snapshot.
- **Credit via HYG/LQD ETF ratio** is a proxy for the OAS spread: good for direction, coarser than actual HY option-adjusted spreads.

## Tests

```bash
cd <SKILL_DIR>/scripts && uv run --with 'yfinance>=1.3,<2' --with 'pandas>=2' \
  --with 'numpy>=1.24,<3' --with pytest pytest -q
```

Pure-logic tests (no network) cover the state machine, every divergence flag, the vote helper, the breadth/new-high-low math (`test_classify.py`), and the signal-report's spell/confirmation/flag-parsing/forward-window logic (`test_backtest_outcomes.py`).