Model Manager

---
name: model-manager
description: Test, validate, and add new AI models to the eval suite. Use when user asks to add new models, test model access, check pricing, or update models.yml.
---

# Model Manager

Test API access, validate configurations, and add new AI models to the AILANG eval suite.

## Quick Start

**Most common usage:**
```bash
# User says: "Can we add GPT-5.1 to the eval suite?"
# This skill will:
# 1. Test API access to GPT-5.1
# 2. Find the correct API model name
# 3. Look up pricing information
# 4. Update models.yml configuration
# 5. Run a test benchmark to verify
```

## When to Use This Skill

Invoke this skill when:
- User asks to "add a new model" to eval suite
- User mentions checking if a model is "accessible" or "available"
- User wants to "test API access" to a model
- User asks to "update models.yml" or "check pricing"
- User says "can we use [model name]?" for evaluations

## Available Scripts

### `scripts/test_model_access.sh <provider> <model-name>`
Test API access to a model and display authentication status.

**Usage:**
```bash
# Test OpenAI model
scripts/test_model_access.sh openai gpt-5.1

# Test Anthropic model
scripts/test_model_access.sh anthropic claude-sonnet-5-5

# Test Google Gemini via Vertex AI
scripts/test_model_access.sh google gemini-3-pro-preview-11-2025
# Test Ollama Cloud (flat-rate open-weight route; tag WITHOUT :cloud)
scripts/test_model_access.sh ollama-cloud glm-5.3-flash
```

**Output:**
```
Testing: openai/gpt-5.1
✓ OPENAI_API_KEY found
✓ API call successful
✓ Model: gpt-5.1-2025-11-13
✓ Tokens: 13 input, 10 output (10 reasoning)
Ready to add to models.yml
```

### `scripts/find_model_info.sh <model-keywords>`
Search for model information using web search and return API names + pricing.

**Usage:**
```bash
# Find GPT-5.1 info
scripts/find_model_info.sh "GPT-5.1 API model name pricing"

# Find Gemini 3 Pro info
scripts/find_model_info.sh "Gemini 3 Pro API documentation"
```

**Output:**
```
Searching for: GPT-5.1 API model name pricing
✓ Found API names:
  - gpt-5.1 (Thinking mode)
  - gpt-5.1-chat-latest (Instant mode)
✓ Pricing:
  Input: $1.25 per 1M tokens
  Output: $10.00 per 1M tokens
  Cached: $0.125 per 1M tokens
```

### `scripts/update_models_yml.sh <friendly-name> <api-name> <provider> <input-price> <output-price>`
Add a new model to models.yml configuration.

**Usage:**
```bash
# Add GPT-5.1
scripts/update_models_yml.sh \
  gpt5-1 \
  "gpt-5.1" \
  openai \
  0.00125 \
  0.01
```

**Output:**
```
Adding model to models.yml:
  Friendly name: gpt5-1
  API name: gpt-5.1
  Provider: openai
  Pricing: $0.00125 / $0.01 per 1K tokens

✓ Updated models.yml
✓ Validated YAML syntax
✓ Ready to test
```

### `scripts/verify_vertex_model.sh <model-name>`
Check if a Gemini model is available in Vertex AI.

**Usage:**
```bash
# Check if Gemini 3 Pro is available
scripts/verify_vertex_model.sh gemini-3-pro-preview-11-2025
```

**Output:**
```
Checking Vertex AI for: gemini-3-pro-preview-11-2025
✓ GCP project: multivac-internal-prod
✓ Access token obtained
✗ Model not found (404)
Recommendation: Monitor for availability, check again in 1-2 weeks
```

### `scripts/measure_ollama_credit_rate.sh <tag1> [tag2 ...]`
Ollama Cloud credit burn per token across models — see
[resources/ollama_cloud.md](resources/ollama_cloud.md).

### `scripts/run_test_benchmark.sh <model-name>`
Run a small test benchmark to verify model works end-to-end.

**Usage:**
```bash
# Test GPT-5.1 with fizzbuzz benchmark
scripts/run_test_benchmark.sh gpt5-1
```

**Output:**
```
Running test benchmark: fizzbuzz
Model: gpt5-1
✓ Benchmark completed
✓ Result: PASS (100%)
✓ Tokens: 245 input, 89 output
✓ Cost: $0.002
Model is ready for production use
```

## Workflow

### 1. Test API Access

**First, verify you can call the model:**

```bash
# Use test_model_access.sh
scripts/test_model_access.sh openai gpt-5.1
```

**Credentials live in `~/.config/ailang/secrets.env`** (ANTHROPIC/OPENAI/GOOGLE/OPENROUTER/
OLLAMA keys). An interactive shell does not source it — `. ~/.config/ailang/secrets.env`
before `eval-suite` or the access scripts. Without `ANTHROPIC_API_KEY`, standard mode falls
back to the Claude subscription OAuth token, which the provider serves for **Haiku only**:
Sonnet 5.5 / Opus 5.5 come back `429 rate_limit_error "Error"` with no rate-limit headers
(measured 2026-09-29). A 429 with no `anthropic-ratelimit-*` headers is that, not quota.

**What to check:**
- API key is set (OPENAI_API_KEY, ANTHROPIC_API_KEY, or gcloud auth)
- API call succeeds (not 401/403/404)
- Model returns expected structure
- Token usage is reported

**For Claude models — also probe the agent lane (claude CLI, subscription):**
```bash
env -u ANTHROPIC_API_KEY claude -p --model <full-id> --output-format json "Reply OK" | jq '.modelUsage|keys'
env -u ANTHROPIC_API_KEY claude -p --model claude-sonnet-9-zzz --output-format json "x"  # negative control: must fail
```
`modelUsage` must name the exact id. Use the FULL id in `agent_model_name`, never
`sonnet`/`opus`/`fable`: those aliases re-point at the newest model (2026-09-29 the harness's
CLI ran `sonnet`→claude-sonnet-5, a newer CLI →claude-sonnet-5-5), so an alias row silently
benchmarks a different model under its label. The agent harness runs the CLI found by
`executor.FindNativeBinary` — `~/.local/bin/claude` (self-updating) before any VSCode-bundled
copy, which lags and once rejected a new model as `unrecognized_model`.

**For Gemini models:**
- Uses Vertex AI (not public API)
- Requires `gcloud auth application-default login`
- Check availability with `verify_vertex_model.sh`

### 2. Find Model Information

**Search for official documentation:**

```bash
# Find API model name and pricing
scripts/find_model_info.sh "GPT-5.1 API documentation pricing"
```

**What to gather:**
- Exact API model name (e.g., `gpt-5.1` not `GPT-5.1`)
- Provider (openai, anthropic, google)
- Input price per 1K tokens
- Output price per 1K tokens
- Context limits (if relevant)
- Special features (adaptive reasoning, caching, etc.)

**Reference:** See [resources/provider_endpoints.md](resources/provider_endpoints.md)

### 2a. Reasoning-Model Check (REQUIRED before gating any verdict)

**The GLM-5.2 lesson (v0.30.0, 2026-07-19):** GLM-5.2 was rejected in June as "worse
than 5.1" — but the regression was OUR truncation: its always-on thinking phase
(28–32K tokens) shared a 32,768 `max_output_tokens` budget with content, and the
harness recorded neither `reason_tokens` nor `finish_reason`, so the guillotine was
invisible. With 64K headroom, 5.2 beats 5.1 on every axis. Kimi K3 nearly repeated
this (thinks by default, same 32K cap, unmeasured).

**Before writing any gate verdict for a new model:**

1. **Probe default thinking** — one small OpenRouter/API call, no reasoning params:
   ```bash
   # reasoning_tokens > 0 → the model thinks by default
   curl -s https://openrouter.ai/api/v1/chat/completions -H "Authorization: Bearer $OPENROUTER_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"model":"<api_name>","messages":[{"role":"user","content":"Prove there are infinitely many primes, then state the 10th prime."}],"max_tokens":3000}' \
     | jq '{reasoning: .usage.completion_tokens_details.reasoning_tokens, provider}'
   ```
2. **If it thinks: ground `max_output_tokens` in the DECLARED provider ceilings,
   not a guess.** Context window ≠ output cap ≠ thinking budget — a "1M context"
   model may declare no completion cap at all (Kimi K3 via Moonshot: undeclared →
   generation is bounded only by request `max_tokens` + remaining window).
   ```bash
   # Per-provider documented ceilings for this model (machine-readable):
   curl -s "https://openrouter.ai/api/v1/models/<api_name>/endpoints" \
     | jq -r '.data.endpoints[] | "\(.provider_name): ctx=\(.context_length) max_completion=\(.max_completion_tokens // "undeclared")"'
   ```
   Providers DIVERGE (glm-5.2: 32768 on DeepInfra vs 131072 on Z.AI first-party vs
   1M on others) — routing roulette means the effective cap varies per request
   unless you pin provider order. Set `max_output_tokens` from the first-party /
   dominant provider's declared ceiling, sized for thinking + content (never the
   fleet-default 32768), and record the provenance in the models.yml comment.
   Also check the vendor's own docs for the intended control surface: some models
   bound thinking by an **effort dial** (K3: Low/Standard/High/Max via
   `reasoning_effort` in models.yml), Anthropic by `budget_tokens`, not by
   output headroom. Reasoning bills as OUTPUT tokens — at K3's $15/M a chatty
   trace can cost more than the visible answer, so an eval that doesn't pin
   effort isn't reproducible; leave vendor-default unless deviating, and if you
   deviate, record it in models.yml (`reasoning_effort`).
3. **Leave default thinking ON.** A thinking-tuned model with thinking suppressed
   is not the model you're gating. The knob that is actually enforced is
   `max_tokens` headroom — OpenRouter third-party upstreams (Baidu, StreamLake)
   ignore `reasoning: {max_tokens: N}` (probed 2026-07-19); `reasoning_max_tokens`
   in models.yml is best-effort only.
4. **Read `finish_reason` + `reason_tokens` in every failure before concluding
   capability** (recorded on standard results since v0.30.0). A `finish=length`
   failure with huge `reason_tokens` is truncation, not weakness. Results banked
   before v0.30.0 cannot show this — never re-derive a verdict from them alone.
5. **Cost note:** thinking bills as output tokens. Budget/cost projections for a
   reasoning model must use `output + reasoning`, and expect ~2-10x the completion
   tokens of a non-thinking peer.

### 2b. Ollama Cloud models (flat-rate route)

Open-weight models at a flat subscription rate, as `motoko-cloud-*` rows. The row
conventions, quota semantics and credit-rate measurement are in
[resources/ollama_cloud.md](resources/ollama_cloud.md) — read it before adding one.

### 3. Update models.yml

> **⛔ NEVER add an OpenRouter row for an Anthropic, OpenAI or Google (Gemini) model**
> (Mark, 2026-09-29). Those vendors have direct lanes here (claude CLI subscription, codex
> subscription, Vertex). If the direct lane fails, fix or report the lane — do not add an
> `or-` twin to get a number. Open-weights releases (gemma, gpt-oss) are fine. Enforced by
> `TestModels_FirstPartyVendorsNeverViaOpenRouter`. Claude CLI rows pin the FULL id in
> `agent_model_name`, never `sonnet`/`opus`/`fable` (`TestModels_ClaudeCLIRowsNeverUseAnAlias`).

**Add the model configuration:**

```bash
# Add to models.yml
scripts/update_models_yml.sh \
  <friendly-name> \
  <api-name> \
  <provider> \
  <input-per-1k> \
  <output-per-1k>
```

**Naming conventions:**
- Friendly name: `gpt5-1`, `claude-sonnet-4-5`, `gemini-3-pro`
- API name: Exact string for API calls
- Use hyphens, lowercase

**Also update:**
- Model suites (`benchmark_suite`, `extended_suite`, `dev_models`)
- Add notes about special features
- Document agent CLI support (if available)

### 4. Run Test Benchmark

**Verify end-to-end:**

```bash
# Test with a simple benchmark
scripts/run_test_benchmark.sh <model-name>
```

**What to verify:**
- Benchmark completes successfully
- Results are reasonable (not garbage output)
- Token usage matches expectations
- Cost calculation works
- No errors in logs

### 5. Apply the Smoke-Test Gate (HARD RULE)

**Rule of thumb (project-wide):**
> **Rule out adding a model to our eval suite if it can't pass ALL the smoke tests.**

The smoke test is the **canonical `smoke` tier** — benchmarks tagged `tier: smoke`
in their YAML spec, selected with `--tier smoke` (NOT a hand-typed `--benchmarks`
list). These are the fundamental "can it speak AILANG at all" tests that the
established frontier tier passes cleanly. If a candidate fails them, the failure is
on the model, not the harness or the benchmark.

**The smoke tier is the source of truth — do NOT hardcode a benchmark list.**
Run `ailang eval-suite --tier smoke --dry-run` to see the current set (23 as of
2026-06-16, up from 17 — it grows, so always derive it, never trust this number).
It includes fizzbuzz, adt_option, gcd_lcm, nested_records, record_update,
recursion_fibonacci, type_safe_record_access, balanced_parens, etc. — all fundamental.

> **⚠️ csv_to_json_converter is `tier: core`, NOT `tier: smoke`.** An earlier
> version of this skill hardcoded `fizzbuzz,adt_option,csv_to_json_converter` as
> "the smoke set." That was wrong: csv_to_json is a **core-tier discriminator** that
> the *majority of frontier models fail* in standard mode (gpt5 base, gemini-3-pro,
> gemini-3-flash, sonnet-4-5, gpt5-mini all FAIL it; only the top tier —
> opus-4-6/4-7, sonnet-4-6, gemini-3-1-pro, gpt5-2-codex/gpt5-4 — pass). Gating OS
> models on csv_to_json means "be top-3-tier or be cut," which unfairly excludes
> viable models. Keep csv_to_json in `--tier core` runs for **ranking**, never as
> an include/exclude gate. (Empirically verified against eval baselines 2026-06-02.)

**Run smoke against a candidate (canonical tier):**

Standard mode accepts `--tier smoke` directly. **Agent mode requires an explicit
`--benchmarks` list** (deliberate cost guardrail), so derive it from the `tier:`
tags — never hardcode (the list drifts):

```bash
# Derive the smoke set from the tier tags (works for both modes, stays in sync)
SMOKE=$(grep -l 'tier: smoke' benchmarks/*.yml | xargs -n1 basename | sed 's/\.yml$//' | paste -sd, -)

# Standard mode:
ailang eval-suite --models <candidate>,claude-sonnet-4-6 --tier smoke \
  --langs ailang --output /tmp/smoke_<candidate> --parallel 2

# Agent mode (must pass the derived list explicitly):
ailang eval-suite --agent --models <candidate>,claude-sonnet-4-6 \
  --benchmarks "$SMOKE" --langs ailang --output /tmp/smoke_<candidate> --parallel 2

# Tabulate pass/fail (agent mode → results land under /agent, standard → /standard)
for f in /tmp/smoke_<candidate>/*/*.json; do
  name=$(basename "$f" .json | sed 's/_[0-9]*$//')
  jq -r --arg name "$name" '"\($name)\t\(if .compile_ok and .runtime_ok and .stdout_ok then "PASS" else "FAIL" end)\t\(.err_code // .error_category // "—")"' "$f"
done | column -ts $'\t'
```

**Decision tree** (N = number of benchmarks in the smoke tier, 23 as of 2026-06-16 — derive it, don't assume):
1. **claude-sonnet-4-6 fails any smoke-tier benchmark** — smoke tier is broken;
   fix the benchmark (or its `tier:` tag) before evaluating candidates.
2. **Candidate fails most of the tier** — CUT. Do not add to models.yml. Note the
   failure types in the cut commit message (WRONG_LANG, syntax, runtime,
   wrong-output) for future reference.
3. **Candidate fails 1–2 (near-clean)** — NEAR-MISS. Optionally keep with a
   "near-miss" comment block in models.yml (precedent: `motoko-or-gemma-4-26b`,
   `motoko-or-qwen3-5-35b-a3b`). Re-run periodically; if it starts passing the
   tier clean, that's a signal stdlib/prompt has improved. Note: agent-mode
   failures with `error_category: api_error` + "step budget exhausted" are a
   **harness step-budget cap, not a model gap** — don't count them as capability
   failures (bump the motoko v2 step budget instead).
4. **Candidate passes the tier clean** — it has cleared the FLOOR, nothing more.
   Smoke qualifies a model; it does NOT rank it (see step 5.5). Add the opt-in
   models.yml entry now (gate PASSED, promotion PENDING), then go to step 5.5 to
   decide whether it actually earns a suite slot / replaces an incumbent.

**Failure-mode taxonomy** (worth capturing in the cut commit message):

| Failure | Meaning | Likely cause |
|---------|---------|--------------|
| `WRONG_LANG` | Model produced Python/JS instead of AILANG | Prompt-following gap; small/MoE models lose plot at 23k-token system prompt |
| `syntax-error` (no WRONG_LANG) | Invented AILANG syntax (e.g. `let rec`, `\n.` lambda) | Model hasn't seen enough AILANG in training |
| `wrong-output` | Compiled and ran, wrong stdout | Spec-following gap, not language gap |
| `runtime-error` | Compiled, crashed at runtime | Logic bug |

**Reading smoke failures** (full precedents, incl. the 2026-05-04 OS-model sweeps, in
[resources/smoke_precedents.md](resources/smoke_precedents.md)):
- `api_error` is infrastructure (rate limit, timeout, harness) — re-run before counting it;
  "exceeded hard timeout" on an agent run is usually the benchmark's `timeout:`, not the model.
- `compile_error` = invented syntax; `WRONG_LANG` = produced Python/JS — genuine model gaps.
- Agent mode is a separate gate: a standard-mode failure may pass agent smoke, and vice versa.
- New OpenRouter vendor prefix → add it to `internal/ai/config.go::openrouterVendorPrefixes`.

### 5.5 Smoke is a FLOOR, not a RANKING — place the candidate on the ANCHORED ELO series (HARD RULE)

> **Passing smoke is necessary but NOT sufficient. Smoke says "this model can
> speak AILANG at all"; it does NOT say "this model is good enough to add" or
> "this model beats the incumbent." Those are RANKING questions, and smoke is
> saturated — every frontier-class model scores ~the same on it. Never make an
> add/keep/replace decision on smoke numbers.**

Why: the smoke tier is deliberately fundamental ("can it speak AILANG"), so any
viable model passes ~all of it. A smoke **tie is the expected outcome**, not a
signal — it carries zero ranking information. The discriminator is `--tier core`
(~26 benchmarks incl. `csv_to_json_converter`, the contract/state-machine tests),
plus `--tier frontier` when you want placement at the hard end.

#### ⛔ Run the CANDIDATE ALONE. Do not re-run incumbents or anchors.

**M-EVAL-ROLLING-ELO (landed 2026-08-27, PRs #939/#942) changed this.** ELO
ratings used to be incomparable across fits — `FitFromTrials` seeded every model
*and* benchmark at 1500 with no scale anchor, so the same rows produced different
absolute numbers in different pools (measured: or-glm-5-3-flash rated **2763** in
one pool vs **1995** in another over comparable rows). That is why the old
protocol re-ran candidate + incumbent + anchor together — a shared pool was the
only way to make numbers mean anything.

That is no longer true, and doing it now is pure waste:

- `internal/eval_harness/anchor_v1.json` freezes the fitted difficulties of the
  discriminating standard benchmarks. Standard-mode fits hold that panel fixed
  and let model ratings move ([`cmd/ailang/eval_elo.go:172`](../../../cmd/ailang/eval_elo.go)).
- So a candidate run **alone** is *placed* onto the same scale as every model
  ever measured. Anchored drift is **31.2** ELO vs **311.7** unanchored.
- **D3 retired full baselines as the default release measurement.** The full run
  is demoted to quarterly re-anchoring (`make eval-baseline FULL=true`).

```bash
# CORRECT — candidate only; the anchored fit places it against banked history.
ailang eval-suite --models <candidate> --tier core,frontier --langs ailang \
  --output /tmp/cf_<candidate> --parallel 4

ailang eval-elo /tmp/cf_<candidate> --json     # read-only ANCHORED placement fit
go run ./tools/eval-elo /tmp/cf_<candidate> --mode standard \
  --persist ~/.ailang/state/observatory.db              # bank it into the series
```

`ailang eval-elo` does NOT persist — it refuses `--persist` and tells you to use
`tools/eval-elo`. Persisting through the cmd path silently no-ops (it swallowed
`--persist` for weeks; see `project_agent_ratings_seeding_evalelo`).

Compare the resulting rating to the **banked** ratings already in
`observatory.db` (`LoadModelRatings`) — that is what the series is for.

> **⚠️ CHECK THE BANKED SERIES IS ACTUALLY ANCHORED BEFORE COMPARING.** An
> anchored placement and a pre-anchor banked rating are on DIFFERENT SCALES, and
> nothing in the output warns you. Measured 2026-09-01 on the dev box: every
> `model_ratings` row for `mode='standard'` was stamped **2026-08-03** — before
> the anchor landed (2026-08-28) — so those 19 values are unanchored, and
> `trial_history` held **agent-mode rows only** (739 rows, 3 models), meaning
> there were no banked standard trials to re-level them from. Reading Hy4's
> anchored 1915.5 against that table would have been exactly the 2763-vs-1995
> error the anchor exists to prevent.
>
> ```bash
> sqlite3 ~/.ailang/state/observatory.db \
>   "SELECT model_id, ROUND(rating,1), n_trials, substr(last_updated,1,10)
>      FROM model_ratings WHERE mode='standard' ORDER BY rating DESC;"
> sqlite3 ~/.ailang/state/observatory.db \
>   "SELECT mode, COUNT(*), COUNT(DISTINCT model_id) FROM trial_history GROUP BY mode;"
> ```
>
> **Update 2026-09-29:** anchored standard rows now exist — claude-opus-5-5 2871.9 (floor, 29/29),
> gpt6-sol 2288.7, gpt6-luna 2008.2 (all 2026-09-22, 29-benchmark core+frontier panel) and
> claude-sonnet-5-5 ≈2296 (28/29). The check below is still how you tell which rows qualify.
>
> If `last_updated` predates the anchor, or `trial_history` has no rows for the
> mode you are placing in, you have a placement but **no valid comparison set**.
> Say so plainly rather than ranking against stale numbers. Report the candidate's
> pass profile against the **anchored benchmark difficulties** (which the fit does
> give you) and treat the leaderboard position as unavailable until the series is
> re-fit. Do NOT "fix" this by re-running comparators — that is the anti-pattern
> above; the fix is banking standard-mode trials so the series can accumulate.

#### When you DO still co-run models

Three cases, and only these:

1. **Agent mode.** There is no agent-mode anchor yet — `eval_elo.go:171-172`
   anchors standard mode only, agent fits are unanchored. For an agent-mode
   ranking question the old same-pool rule still holds.
2. **The incumbent's banked rating predates a baseline-moving language change.**
   The anchor pins benchmark *difficulty*, not the harness or stdlib. A change
   that moves what models can do (e.g. the 2026-07-29 extension fix) invalidates
   older per-benchmark rates. Check the banked rating's version/date provenance
   first; if it is stale, re-run **just the incumbent** — never the whole panel.
3. **A paired/discordant analysis**, where `ailang eval-paired <on> <off>` needs
   both arms from the same run by construction.

When you genuinely do run several models, they must still go in **ONE**
`eval-suite` command — it overwrites its output directory (`.claude/rules/eval.md`).
That rule is about not clobbering results; it is not a reason to add models.

#### Deciding promote / replace

- **Match-or-beat the incumbent** to promote. A tie at higher cost keeps the
  incumbent; a tie at equal-or-lower cost favours the newer generation.
- **Close call on N=1** — core is ~26 single-shot runs and OS-model variance is
  real. Within 1–2 benchmarks (or overlapping ELO bands), escalate to N≥3 before
  deciding. Never flip an incumbent on a 1-benchmark N=1 delta.
- **Read `finish_reason` + `reason_tokens` on every failure** before calling it
  capability (§2a). A `finish=length` with large reasoning is truncation.

> **⚠️ Anti-pattern (2026-06-16, GLM-5.2 vs GLM-5.1):** GLM-5.2 cleared standard
> smoke at **22/23 — an exact tie with GLM-5.1** (both failed only
> `dense_operator_program`, which the anchor ALSO failed → a benchmark/harness
> issue, not a model gap). The first-pass conclusion was *"tie at +43% cost →
> keep GLM-5.1."* **That was WRONG.** A smoke tie is meaningless because smoke is
> saturated — it proves only that the candidate cleared the floor and QUALIFIES.
> When a candidate ties the incumbent on smoke, that is your cue to run core, NOT
> your answer.

> **⚠️ Anti-pattern (2026-09-01, Hy4 preview):** after Hy4 passed smoke 23/23, the
> placement run was launched as **six models × 31 benchmarks = 186 runs, ~$3.75** —
> candidate plus incumbent plus four comparators, on the pre-rolling-ELO reflex
> that a shared pool was needed. It was not: five of those six models already had
> banked anchored ratings, and the candidate alone costs **$0.23**. Killed at
> $0.24. **A comparator you re-run is a comparator you pay for twice.**

### 6. Document the Model

**Update relevant documentation:**
- Add model to this skill's resource guide
- Note any special parameters (e.g., `max_completion_tokens` for GPT-5.1)
- Document authentication requirements
- Add to teaching prompts if needed

### 7. Bank the placement — do NOT run a full baseline

Once the candidate has its core/frontier placement, persist it into the anchored
series so the next question can be answered from banked data instead of a re-run:

```bash
go run ./tools/eval-elo /tmp/cf_<candidate> --mode standard --persist ~/.ailang/state/observatory.db
```

`--persist` takes the DB path, not the results dir. `database is locked` means another
`ailang` process (coordinator, a running eval) holds observatory.db — retry once it is idle.

**`make eval-baseline FULL=true` is NOT part of adding a model.** D3 of
M-EVAL-ROLLING-ELO demoted the full baseline to a **quarterly re-anchoring**
event (and longitudinal spot-checks). Running one to place a new model costs
$5-25 + hours of wall clock to produce a number the anchored fit already gives
you for well under a dollar. If you think you need a full baseline, you almost
certainly need a linking run instead — re-read §5.5.

## Resources

### Provider Endpoints
See [resources/provider_endpoints.md](resources/provider_endpoints.md) for:
- API endpoint URLs for each provider
- Authentication methods
- How to test access manually
- Common errors and fixes

### Pricing Guide
See [resources/pricing_guide.md](resources/pricing_guide.md) for:
- How to find official pricing
- Price conversion (per 1M → per 1K)
- Cost calculation verification
- Caching and discounts

## Progressive Disclosure

This skill loads information progressively:

1. **Always loaded**: This SKILL.md file (workflow and script descriptions)
2. **Execute as needed**: Scripts in `scripts/` (testing, updating, verification)
3. **Load on demand**: Resources (detailed endpoint docs, pricing references)

## Notes

**Important:**
- Always test API access BEFORE updating models.yml
- Vertex AI (Gemini) requires gcloud auth, not API key
- GPT-5.1+ uses `max_completion_tokens` instead of `max_tokens`
- New models may not be available in all regions immediately
- Check for preview/beta status before adding to production suites

**Prerequisites:**
- API keys: `. ~/.config/ailang/secrets.env` (see §1 — shells do not source it)
- For Gemini: `gcloud` CLI installed and authenticated
- For Ollama Cloud: local ollama daemon running + `ollama signin` done (inference);
  `OLLAMA_API_KEY` set only for the /api/usage quota gauge
- For Gemini: GCP project set (`gcloud config set project PROJECT_ID`)
- `curl`, `python3`, and `jq` available in PATH

**Files modified by this skill:**
- `internal/modelreg/models.yml` - Model configurations (the embedded registry)
- Anthropic rows also need, per model: `internal/ai/reasoning_anthropic.go` (thinking style —
  `CanDisable:false` if `{type:"disabled"}` 400s), `internal/ai/anthropic/cache.go` (min
  cacheable prefix), `ceilingLimited` in `internal/eval_harness/models_headroom_test.go`, and
  `anthropicCacheReadMultiplier` in `internal/modelreg/resolve_test.go` if cache reads ≠ 0.1x
- `resources/provider_endpoints.md` - When adding a new provider (e.g. Ollama Cloud)
- (Optional) `prompts/vX.Y.Z.md` - Teaching prompts
- (Optional) `.claude/skills/model-manager/resources/` - Local model database