Model Manager

---
name: model-manager
description: Test, validate, and add new AI models to the eval suite. Use when user asks to add new models, test model access, check pricing, or update models.yml.
---

# Model Manager

Test API access, validate configurations, and add new AI models to the AILANG eval suite.

## Quick Start

**Most common usage:**
```bash
# User says: "Can we add GPT-5.1 to the eval suite?"
# This skill will:
# 1. Test API access to GPT-5.1
# 2. Find the correct API model name
# 3. Look up pricing information
# 4. Update models.yml configuration
# 5. Run a test benchmark to verify
```

## When to Use This Skill

Invoke this skill when:
- User asks to "add a new model" to eval suite
- User mentions checking if a model is "accessible" or "available"
- User wants to "test API access" to a model
- User asks to "update models.yml" or "check pricing"
- User says "can we use [model name]?" for evaluations

## Available Scripts

### `scripts/test_model_access.sh <provider> <model-name>`
Test API access to a model and display authentication status.

**Usage:**
```bash
# Test OpenAI model
scripts/test_model_access.sh openai gpt-5.1

# Test Anthropic model
scripts/test_model_access.sh anthropic Codex-sonnet-4-5-20250929

# Test Google Gemini via Vertex AI
scripts/test_model_access.sh google gemini-3-pro-preview-11-2025

# Test Ollama Cloud (flat-rate open-weight route; tag WITHOUT :cloud)
scripts/test_model_access.sh ollama-cloud glm-5.3-flash
```

**Output:**
```
Testing: openai/gpt-5.1
✓ OPENAI_API_KEY found
✓ API call successful
✓ Model: gpt-5.1-2025-11-13
✓ Tokens: 13 input, 10 output (10 reasoning)
Ready to add to models.yml
```

The `ollama-cloud` case additionally checks sign-in (`ollama signin` → POST
/api/me), catalogue membership, a reasoning-safe inference probe through the
local daemon, and a before/after quota snapshot from ollama.com/api/usage.

### `scripts/find_model_info.sh <model-keywords>`
Search for model information using web search and return API names + pricing.

**Usage:**
```bash
# Find GPT-5.1 info
scripts/find_model_info.sh "GPT-5.1 API model name pricing"

# Find Gemini 3 Pro info
scripts/find_model_info.sh "Gemini 3 Pro API documentation"
```

**Output:**
```
Searching for: GPT-5.1 API model name pricing
✓ Found API names:
  - gpt-5.1 (Thinking mode)
  - gpt-5.1-chat-latest (Instant mode)
✓ Pricing:
  Input: $1.25 per 1M tokens
  Output: $10.00 per 1M tokens
  Cached: $0.125 per 1M tokens
```

### `scripts/update_models_yml.sh <friendly-name> <api-name> <provider> <input-price> <output-price>`
Add a new model to models.yml configuration.

**Usage:**
```bash
# Add GPT-5.1
scripts/update_models_yml.sh \
  gpt5-1 \
  "gpt-5.1" \
  openai \
  0.00125 \
  0.01
```

**Output:**
```
Adding model to models.yml:
  Friendly name: gpt5-1
  API name: gpt-5.1
  Provider: openai
  Pricing: $0.00125 / $0.01 per 1K tokens

✓ Updated models.yml
✓ Validated YAML syntax
✓ Ready to test
```

### `scripts/verify_vertex_model.sh <model-name>`
Check if a Gemini model is available in Vertex AI.

**Usage:**
```bash
# Check if Gemini 3 Pro is available
scripts/verify_vertex_model.sh gemini-3-pro-preview-11-2025
```

**Output:**
```
Checking Vertex AI for: gemini-3-pro-preview-11-2025
✓ GCP project: multivac-internal-prod
✓ Access token obtained
✗ Model not found (404)
Recommendation: Monitor for availability, check again in 1-2 weeks
```

### `scripts/measure_ollama_credit_rate.sh <tag1> [tag2 ...]`
Empirical Ollama Cloud credit-rate comparison (V36/V46 method) — the only way to
answer "how many credits does model X cost vs model Y" without a published rate:

```bash
# Compare credit burn per token, e.g. the GLM-5.3 family
scripts/measure_ollama_credit_rate.sh glm-5.3-flash glm-5.3
```

**Output:** per-arm sessions-numerator delta over real tokens burned → units/M,
plus the cross-model ratio:
```
model                    tokens   credits  units/M
glm-5.3-flash            142486    +0.004   0.0281
glm-5.3                 140543    +0.011   0.0783

credit ratio: glm-5.3 costs 2.79x the credits per token of glm-5.3-flash
```

**Measured 2026-08-31 (three runs, one-shot shape):** glm-5.3-flash ≈ 0.03
units/M (page level: Medium), glm-5.3 ≈ 0.08–0.14 units/M (High) — ratio ≈ 3x,
consistent with the published ~3-4x-per-level ladder (V36: gpt-oss:20b 0.0069,
deepseek-v4-flash 0.029, kimi-k3 0.124). Precision is capped by the 3-decimal
numerator: prefer ≥140k tokens per arm, and re-run if an arm shows ≤2 ticks.

### `scripts/run_test_benchmark.sh <model-name>`
Run a small test benchmark to verify model works end-to-end.

**Usage:**
```bash
# Test GPT-5.1 with fizzbuzz benchmark
scripts/run_test_benchmark.sh gpt5-1
```

**Output:**
```
Running test benchmark: fizzbuzz
Model: gpt5-1
✓ Benchmark completed
✓ Result: PASS (100%)
✓ Tokens: 245 input, 89 output
✓ Cost: $0.002
Model is ready for production use
```

## Workflow

### 1. Test API Access

**First, verify you can call the model:**

```bash
# Use test_model_access.sh
scripts/test_model_access.sh openai gpt-5.1
```

**What to check:**
- API key is set (OPENAI_API_KEY, ANTHROPIC_API_KEY, or gcloud auth)
- API call succeeds (not 401/403/404)
- Model returns expected structure
- Token usage is reported

**For Gemini models:**
- Uses Vertex AI (not public API)
- Requires `gcloud auth application-default login`
- Check availability with `verify_vertex_model.sh`

### 2. Find Model Information

**Search for official documentation:**

```bash
# Find API model name and pricing
scripts/find_model_info.sh "GPT-5.1 API documentation pricing"
```

**What to gather:**
- Exact API model name (e.g., `gpt-5.1` not `GPT-5.1`)
- Provider (openai, anthropic, google)
- Input price per 1K tokens
- Output price per 1K tokens
- Context limits (if relevant)
- Special features (adaptive reasoning, caching, etc.)

**Reference:** See [resources/provider_endpoints.md](resources/provider_endpoints.md)

### 2a. Reasoning-Model Check (REQUIRED before gating any verdict)

**The GLM-5.2 lesson (v0.30.0, 2026-07-19):** GLM-5.2 was rejected in June as "worse
than 5.1" — but the regression was OUR truncation: its always-on thinking phase
(28–32K tokens) shared a 32,768 `max_output_tokens` budget with content, and the
harness recorded neither `reason_tokens` nor `finish_reason`, so the guillotine was
invisible. With 64K headroom, 5.2 beats 5.1 on every axis. Kimi K3 nearly repeated
this (thinks by default, same 32K cap, unmeasured).

**Before writing any gate verdict for a new model:**

1. **Probe default thinking** — one small OpenRouter/API call, no reasoning params:
   ```bash
   # reasoning_tokens > 0 → the model thinks by default
   curl -s https://openrouter.ai/api/v1/chat/completions -H "Authorization: Bearer $OPENROUTER_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"model":"<api_name>","messages":[{"role":"user","content":"Prove there are infinitely many primes, then state the 10th prime."}],"max_tokens":3000}' \
     | jq '{reasoning: .usage.completion_tokens_details.reasoning_tokens, provider}'
   ```
2. **If it thinks: ground `max_output_tokens` in the DECLARED provider ceilings,
   not a guess.** Context window ≠ output cap ≠ thinking budget — a "1M context"
   model may declare no completion cap at all (Kimi K3 via Moonshot: undeclared →
   generation is bounded only by request `max_tokens` + remaining window).
   ```bash
   # Per-provider documented ceilings for this model (machine-readable):
   curl -s "https://openrouter.ai/api/v1/models/<api_name>/endpoints" \
     | jq -r '.data.endpoints[] | "\(.provider_name): ctx=\(.context_length) max_completion=\(.max_completion_tokens // "undeclared")"'
   ```
   Providers DIVERGE (glm-5.2: 32768 on DeepInfra vs 131072 on Z.AI first-party vs
   1M on others) — routing roulette means the effective cap varies per request
   unless you pin provider order. Set `max_output_tokens` from the first-party /
   dominant provider's declared ceiling, sized for thinking + content (never the
   fleet-default 32768), and record the provenance in the models.yml comment.
   Also check the vendor's own docs for the intended control surface: some models
   bound thinking by an **effort dial** (K3: Low/Standard/High/Max via
   `reasoning_effort` in models.yml), Anthropic by `budget_tokens`, not by
   output headroom. Reasoning bills as OUTPUT tokens — at K3's $15/M a chatty
   trace can cost more than the visible answer, so an eval that doesn't pin
   effort isn't reproducible; leave vendor-default unless deviating, and if you
   deviate, record it in models.yml (`reasoning_effort`).
3. **Leave default thinking ON.** A thinking-tuned model with thinking suppressed
   is not the model you're gating. The knob that is actually enforced is
   `max_tokens` headroom — OpenRouter third-party upstreams (Baidu, StreamLake)
   ignore `reasoning: {max_tokens: N}` (probed 2026-07-19); `reasoning_max_tokens`
   in models.yml is best-effort only.
4. **Read `finish_reason` + `reason_tokens` in every failure before concluding
   capability** (recorded on standard results since v0.30.0). A `finish=length`
   failure with huge `reason_tokens` is truncation, not weakness. Results banked
   before v0.30.0 cannot show this — never re-derive a verdict from them alone.
5. **Cost note:** thinking bills as output tokens. Budget/cost projections for a
   reasoning model must use `output + reasoning`, and expect ~2-10x the completion
   tokens of a non-thinking peer.

### 2b. Ollama Cloud models (flat-rate route)

Ollama Cloud hosts open-weight models at a flat subscription rate, reachable as a
naming convention on an existing row shape — no new provider code. Canonical design:
[design_docs/planned/v0_34_0/m-ollama-cloud-provider.md](../../../design_docs/planned/v0_34_0/m-ollama-cloud-provider.md).

1. **Find the model** — `curl -s https://ollama.com/v1/models | jq -r '.data[].id'`
   (200 unauthenticated; ~19 models). The catalogue id is the TAG (e.g.
   `glm-5.3`, `deepseek-v4-pro:0813`); you request the cloud route by appending
   `:cloud` to the tag.
2. **Test access** — `scripts/test_model_access.sh ollama-cloud <tag>` (no
   `:cloud` — it appends it). Inference auth is the **device key** via
   `ollama signin` (POST localhost:11434/api/me); the daemon proxies
   `:cloud` models to ollama.com and loads nothing on the GPU (V21).
   `OLLAMA_API_KEY` is ONLY for the `GET https://ollama.com/api/usage` gauge —
   which the local daemon does NOT proxy (V24).
3. **Quota semantics (measured, V26/V9)** — `/api/usage` is a numerator with NO
   denominator: no limit/remaining/reset_at is published, so a pre-flight
   "refuse to start if quota low" gate is unbuildable. Metering: model weight ×
   tokens at usage levels 1–4 (`gpt-oss:20b` = 1, `deepseek-v4-pro` = 4);
   session limit resets 5h, weekly 7d; concurrency Free/Pro/Max = 1/3/10 (Max
   paused for new subs); over-concurrency requests QUEUE, not reject.
   `activity.cost` reads `0.00000` forever.
4. **models.yml row conventions (M-Ollama-Cloud-Provider, D1/D2/D6 — RATIFIED):**
   - api_name: `"<tag>:cloud"`, provider: `"ollama"` (same path as local rows),
     env_var: `""`, agent_cli: `motoko`, agent_model_name: `ollama/<tag>:cloud`
   - row key convention: `motoko-cloud-*` (unenforced by test but human-audited;
     the bank stores the ROW KEY, never api_name, so the key IS the route marker)
   - pricing: **IMPUTED from the OpenRouter twin's list price** (D1 — do NOT
     write 0/0; it maps to the false `free-local` provenance). Banks as
     `list-price-equivalent` = "the run went through a subscription lane and was
     never billed". Re-impute whenever the twin's rate drifts.
   - budgets: `max_tokens_per_bench: 3000000`, `hard_timeout_secs: 3600`
   - `default_thinking: "unknown"` until probed (same §2a rules apply — probe
     reasoning with max_tokens ≥ 2000; V25 showed reasoning models burn a
     small budget entirely on thinking)
5. **Scope (D2)** — these are executor/day-to-day rows, NOT banked-eval-rotation
   models unless Mark ratifies: an opaque resetting quota can starve a nightly
   rotation mid-run and bank a cohort with a hole in it.
6. **Concurrency** — cloud rows are EXEMPT from the single-GPU serial clamp (they
   touch no VRAM, D4), but ANY motoko row still serializes on its fixed backend
   port until motoko takes a per-run port.
7. **Quota check as part of the workflow** — snapshot `/api/usage` before and
   after any manual test (the script does this), and expect the numbers to move
   from unrelated traffic: anything else running on the flat route burns the
   same quota (e.g. a coordinator agent session on `glm-5.3-flash:cloud`).
8. **Credit-rate comparison** — before adding a cloud row, and whenever a cost
   question needs answering in Ollama terms:
   `scripts/measure_ollama_credit_rate.sh <tags…>` measures each model's
   units-per-M on the session numerator (V36 method) and prints the ratio
   (measured 2026-08-31: glm-5.3 costs ~3x glm-5.3-flash per token; both match
   their published page levels Medium/High). Rules: one-shot shape unless you
   say otherwise (V46 — agentic meters ~2x cheaper); ≥140k tokens per arm
   because the gauge rounds to 3 decimals; snapshots inside ONE script run;
   cross-check the ratio against the models' published page levels. This
   complements, never replaces, D1: banked dollars stay the OpenRouter twin's
   list price, the credits here are the flat-plan's own internal currency.

### 3. Update models.yml

**Add the model configuration:**

```bash
# Add to models.yml
scripts/update_models_yml.sh \
  <friendly-name> \
  <api-name> \
  <provider> \
  <input-per-1k> \
  <output-per-1k>
```

**Naming conventions:**
- Friendly name: `gpt5-1`, `Codex-sonnet-4-5`, `gemini-3-pro`
- API name: Exact string for API calls
- Use hyphens, lowercase

**Also update:**
- Model suites (`benchmark_suite`, `extended_suite`, `dev_models`)
- Add notes about special features
- Document agent CLI support (if available)

### 4. Run Test Benchmark

**Verify end-to-end:**

```bash
# Test with a simple benchmark
scripts/run_test_benchmark.sh <model-name>
```

**What to verify:**
- Benchmark completes successfully
- Results are reasonable (not garbage output)
- Token usage matches expectations
- Cost calculation works
- No errors in logs

### 5. Apply the Smoke-Test Gate (HARD RULE)

**Rule of thumb (project-wide):**
> **Rule out adding a model to our eval suite if it can't pass ALL the smoke tests.**

The smoke test is the **canonical `smoke` tier** — benchmarks tagged `tier: smoke`
in their YAML spec, selected with `--tier smoke` (NOT a hand-typed `--benchmarks`
list). These are the fundamental "can it speak AILANG at all" tests that the
established frontier tier passes cleanly. If a candidate fails them, the failure is
on the model, not the harness or the benchmark.

**The smoke tier is the source of truth — do NOT hardcode a benchmark list.**
Run `ailang eval-suite --tier smoke --dry-run` to see the current set (23 as of
2026-06-16, up from 17 — it grows, so always derive it, never trust this number).
It includes fizzbuzz, adt_option, gcd_lcm, nested_records, record_update,
recursion_fibonacci, type_safe_record_access, balanced_parens, etc. — all fundamental.

> **⚠️ csv_to_json_converter is `tier: core`, NOT `tier: smoke`.** An earlier
> version of this skill hardcoded `fizzbuzz,adt_option,csv_to_json_converter` as
> "the smoke set." That was wrong: csv_to_json is a **core-tier discriminator** that
> the *majority of frontier models fail* in standard mode (gpt5 base, gemini-3-pro,
> gemini-3-flash, sonnet-4-5, gpt5-mini all FAIL it; only the top tier —
> opus-4-6/4-7, sonnet-4-6, gemini-3-1-pro, gpt5-2-codex/gpt5-4 — pass). Gating OS
> models on csv_to_json means "be top-3-tier or be cut," which unfairly excludes
> viable models. Keep csv_to_json in `--tier core` runs for **ranking**, never as
> an include/exclude gate. (Empirically verified against eval baselines 2026-06-02.)

**Run smoke against a candidate (canonical tier):**

Standard mode accepts `--tier smoke` directly. **Agent mode requires an explicit
`--benchmarks` list** (deliberate cost guardrail), so derive it from the `tier:`
tags — never hardcode (the list drifts):

```bash
# Derive the smoke set from the tier tags (works for both modes, stays in sync)
SMOKE=$(grep -l 'tier: smoke' benchmarks/*.yml | xargs -n1 basename | sed 's/\.yml$//' | paste -sd, -)

# Standard mode:
ailang eval-suite --models <candidate>,Codex-sonnet-4-6 --tier smoke \
  --langs ailang --output /tmp/smoke_<candidate> --parallel 2

# Agent mode (must pass the derived list explicitly):
ailang eval-suite --agent --models <candidate>,Codex-sonnet-4-6 \
  --benchmarks "$SMOKE" --langs ailang --output /tmp/smoke_<candidate> --parallel 2

# Tabulate pass/fail (agent mode → results land under /agent, standard → /standard)
for f in /tmp/smoke_<candidate>/*/*.json; do
  name=$(basename "$f" .json | sed 's/_[0-9]*$//')
  jq -r --arg name "$name" '"\($name)\t\(if .compile_ok and .runtime_ok and .stdout_ok then "PASS" else "FAIL" end)\t\(.err_code // .error_category // "—")"' "$f"
done | column -ts $'\t'
```

**Decision tree** (N = number of benchmarks in the smoke tier, 23 as of 2026-06-16 — derive it, don't assume):
1. **Codex-sonnet-4-6 fails any smoke-tier benchmark** — smoke tier is broken;
   fix the benchmark (or its `tier:` tag) before evaluating candidates.
2. **Candidate fails most of the tier** — CUT. Do not add to models.yml. Note the
   failure types in the cut commit message (WRONG_LANG, syntax, runtime,
   wrong-output) for future reference.
3. **Candidate fails 1–2 (near-clean)** — NEAR-MISS. Optionally keep with a
   "near-miss" comment block in models.yml (precedent: `motoko-or-gemma-4-26b`,
   `motoko-or-qwen3-5-35b-a3b`). Re-run periodically; if it starts passing the
   tier clean, that's a signal stdlib/prompt has improved. Note: agent-mode
   failures with `error_category: api_error` + "step budget exhausted" are a
   **harness step-budget cap, not a model gap** — don't count them as capability
   failures (bump the motoko v2 step budget instead).
4. **Candidate passes the tier clean** — it has cleared the FLOOR, nothing more.
   Smoke qualifies a model; it does NOT rank it (see step 5.5). Add the opt-in
   models.yml entry now (gate PASSED, promotion PENDING), then go to step 5.5 to
   decide whether it actually earns a suite slot / replaces an incumbent.

**Failure-mode taxonomy** (worth capturing in the cut commit message):

| Failure | Meaning | Likely cause |
|---------|---------|--------------|
| `WRONG_LANG` | Model produced Python/JS instead of AILANG | Prompt-following gap; small/MoE models lose plot at 23k-token system prompt |
| `syntax-error` (no WRONG_LANG) | Invented AILANG syntax (e.g. `let rec`, `\n.` lambda) | Model hasn't seen enough AILANG in training |
| `wrong-output` | Compiled and ran, wrong stdout | Spec-following gap, not language gap |
| `runtime-error` | Compiled, crashed at runtime | Logic bug |

**2026-05-04 finding (precedent):** Tested 6 SOTA OS models (Gemma 4 26B, Qwen3
30B-A3B, Qwen3 235B-A22B, DeepSeek V4 Flash, Kimi K2.6, Qwen3 Coder Flash)
against this smoke set. Proprietary baselines passed 3/3; **zero OS models
passed all 3**. Most common failure: WRONG_LANG (model produced Python). Even
frontier-class OS models fall back on training-corpus patterns when given
AILANG's 23k-token teaching prompt — they've seen plenty of Python but very
little AILANG. Two near-misses (`or-gemma-4-26b`, `or-qwen3-coder-flash`)
retained on the watchlist; rest cut.

**Implication for stdlib/prompt work:** the smoke test doubles as a
language-improvement metric. Re-run it after stdlib changes or prompt
revisions; if the near-miss watchlist starts passing the third benchmark, the
language has become more "trainable-feel."

**Caveat — agent mode is a separate gate:** the smoke set above runs in
**standard** (single-shot API generation) mode. Models that fail standard mode
may still perform usefully in **agent** mode (`--agent` flag, opencode/pi
harnesses) where they get multi-turn iteration. If a candidate fails standard
smoke, run `ailang eval-suite --agent --models <candidate> ...` separately
before fully cutting it. Agent mode results don't override the standard-mode
gate but can justify adding the model under a different harness entry (e.g.
`opencode-<candidate>`, `pi-<candidate>`).

**2026-05-04 agent-mode smoke finding (precedent):** Tested 9 OS-via-OR
candidates through opencode harness. Cross-mode behaviour:

| Model | Standard | Agent | Δ |
|-------|---------:|------:|--:|
| **GLM 5** (z.ai) | not tested | **3/3** ✅ | — first OS model to pass |
| Gemma 4 26B | 2/3 | 2/3 | 0 (same near-miss) |
| DeepSeek V4 Flash | 0/3 | 2/3 | **+2** (agent unlock) |
| GLM 4.7 Flash | not tested | 2/3 | — near-miss |
| Kimi K2.6 | 1/3 | 1/3 | 0 |
| Qwen3 30B-A3B | 1/3 | 1/3 | 0 |
| Qwen3 Coder Flash | 2/3 | 1/3 | **-1** (agent regressed) |
| DeepSeek V4 Pro | not tested | 1/3 | Pro under-performed Flash |
| Qwen3 235B-A22B | 0/3 | 0/3 | 0 |

Key takeaways for the model-manager workflow:

1. **Agent mode is not a universal fix.** Most models that fail standard
   smoke also fail agent smoke. Multi-turn helps when the model can read
   compile errors and adjust; it hurts when the model interprets tool-call
   setup as the answer (Qwen3 Coder Flash regression).

2. **Pro tier ≠ better.** DeepSeek V4 Pro (1/3) under-performed V4 Flash
   (2/3) on AILANG smoke. The Pro reasoning/long-output overhead can hurt
   simple-task accuracy. Test both tiers when available.

3. **csv_to_json_converter is a `core`-tier DISCRIMINATOR, not a smoke gate.**
   Of the 27 benchmark runs (9 models × 3), csv_to_json was the single most-failed
   test — only GLM 5 passed it among OS candidates. ⚠️ **CORRECTION (2026-06-02):**
   this is exactly why it must NOT gate inclusion — it's failed by the *majority of
   frontier models* (gpt5 base, gemini-3-pro, gemini-3-flash, sonnet-4-5, gpt5-mini
   all FAIL; only opus-4-6/4-7, sonnet-4-6, gemini-3-1-pro, gpt5-2-codex/gpt5-4
   pass). It lives in `tier: core`, not `tier: smoke`. Use it as a high-signal
   **ranking/discriminator** metric in `--tier core` runs and as a language-
   improvement tracker — never as an OS-model include/exclude gate. The gate is
   `--tier smoke`.

4. **GLM 5 is genuinely cost-competitive frontier OS.** $0.60/$2.08 per 1M
   tokens, ~5–7× cheaper than Codex Sonnet 4.6 on input. Worth standing
   inclusion in eval rotation alongside frontier proprietary models.

5. **Vendor-prefix wiring is forward-compat infrastructure.** When adding
   models from a new vendor (e.g. `z-ai/`, `moonshotai/`, `microsoft/`,
   `minimax/`), add the prefix to
   `internal/ai/config.go::openrouterVendorPrefixes` so future ad-hoc
   `ailang run --ai vendor/model` invocations work without needing a
   models.yml entry.

6. **Per-benchmark timeouts can be tighter than agent-mode needs.** The
   `csv_to_json_converter.yml` spec has `timeout: 90s` baked in (set to
   match Codex Sonnet 4.6's ~43s typical solve time). OS models in agent
   mode routinely need 90–180s of iteration on csv_to_json — they CAN
   solve it but get killed by the timeout. Two follow-up models that
   demonstrated this on 2026-05-04:
     - **Kimi K2.6** (Moonshot): fizzbuzz✅ 119s, adt_option✅ 47s,
       csv_to_json❌ (timeout — initial run also had api_errors)
     - **MiniMax M2.7**: fizzbuzz✅ 46s, adt_option✅ 42s,
       csv_to_json❌ (timeout, not capability)
   Both are effectively 2/3 near-misses pending a benchmark timeout bump.
   When investigating a model that fails only csv_to_json with
   `error_category=api_error` and stderr saying "exceeded hard timeout
   (1m30s)", the failure is the benchmark spec, not the model.

7. **api_error vs syntax-error vs WRONG_LANG matters.** When tabulating
   smoke results, always check `error_category`:
   - `api_error` — infrastructure issue (rate limit, timeout, network).
     Re-run before counting against the model.
   - `compile_error` (no err_code) — syntax-error: model produced AILANG
     that doesn't parse. Genuine model gap.
   - `WRONG_LANG` — model produced Python/JS/etc. instead of AILANG.
     Genuine prompt-following gap.
   - `runtime_error` — compiled but crashed. Logic bug in generation.

### 5.5 Smoke is a FLOOR, not a RANKING — use `--tier core` to decide add/replace (HARD RULE)

> **Passing smoke is necessary but NOT sufficient. Smoke says "this model can
> speak AILANG at all"; it does NOT say "this model is good enough to add" or
> "this model beats the incumbent." Those are RANKING questions, and smoke is
> saturated — every frontier-class model scores ~the same on it. Never make an
> add/keep/replace decision on smoke numbers. Make it on `--tier core`.**

Why: the smoke tier is deliberately fundamental ("can it speak AILANG"), so any
viable model passes ~all of it. A smoke **tie is the expected outcome**, not a
signal — it carries zero ranking information. The discriminator is the **core
tier** (`--tier core`, ~26 benchmarks incl. `csv_to_json_converter`, the
contract/state-machine tests) where frontier models genuinely spread.

**Decision flow once a candidate PASSES smoke:**
1. **New vendor/family, no incumbent** — run `--tier core` head-to-head vs the
   `Codex-sonnet-4-6` anchor to size where it lands. Add to suites if it earns it.
2. **Replacing or competing with an incumbent** (e.g. GLM-5.2 vs GLM-5.1) — run
   `--tier core` for **candidate + incumbent + anchor in ONE command**, `--langs
   ailang`. **Only promote/replace if the candidate matches-or-beats the incumbent
   on core.** A core tie at higher cost → keep the incumbent. A clear core win →
   the cost bump may be justified.
3. **Close call on N=1** — core is ~26 single-shot runs; OS-model variance is real.
   If candidate and incumbent are within 1–2 benchmarks, escalate to N≥3 trials
   before deciding (don't flip an incumbent on a 1-benchmark N=1 delta).

```bash
# The discriminating run — candidate vs incumbent vs anchor, core tier, one command:
ailang eval-suite --models <candidate>,<incumbent>,Codex-sonnet-4-6 \
  --tier core --langs ailang --output /tmp/core_<candidate> --parallel 4
```

> **⚠️ Anti-pattern (2026-06-16, GLM-5.2 vs GLM-5.1):** GLM-5.2 (newest z-ai,
> reasoning model, 1M ctx, +43% price) cleared standard smoke at **22/23 — an
> exact tie with GLM-5.1** (both failed only `dense_operator_program`, which the
> `Codex-sonnet-4-6` anchor ALSO failed → a benchmark/harness issue, not a model
> gap). The first-pass conclusion was *"tie at +43% cost → keep GLM-5.1."* **That
> was WRONG.** A smoke tie is meaningless because smoke is saturated — it proves
> only that GLM-5.2 cleared the floor and QUALIFIES. The replacement decision had
> to be made on `--tier core`, where the two versions can actually separate. Rule:
> when a candidate ties the incumbent on smoke, that's your cue to run core, NOT
> your answer.

### 6. Document the Model

**Update relevant documentation:**
- Add model to this skill's resource guide
- Note any special parameters (e.g., `max_completion_tokens` for GPT-5.1)
- Document authentication requirements
- Add to teaching prompts if needed

### 7. Optional: Run Full Eval

**If model looks good:**

```bash
# Run small eval suite
ailang eval-suite --models <model-name> --benchmarks fizzbuzz,recursion_factorial

# Run full suite (expensive!)
make eval-baseline EVAL_VERSION=vX.Y.Z FULL=true
```

## Resources

### Provider Endpoints
See [resources/provider_endpoints.md](resources/provider_endpoints.md) for:
- API endpoint URLs for each provider
- Authentication methods
- How to test access manually
- Common errors and fixes

### Pricing Guide
See [resources/pricing_guide.md](resources/pricing_guide.md) for:
- How to find official pricing
- Price conversion (per 1M → per 1K)
- Cost calculation verification
- Caching and discounts

## Progressive Disclosure

This skill loads information progressively:

1. **Always loaded**: This SKILL.md file (workflow and script descriptions)
2. **Execute as needed**: Scripts in `scripts/` (testing, updating, verification)
3. **Load on demand**: Resources (detailed endpoint docs, pricing references)

## Notes

**Important:**
- Always test API access BEFORE updating models.yml
- Vertex AI (Gemini) requires gcloud auth, not API key
- GPT-5.1+ uses `max_completion_tokens` instead of `max_tokens`
- New models may not be available in all regions immediately
- Check for preview/beta status before adding to production suites

**Prerequisites:**
- API keys set in environment (OPENAI_API_KEY, ANTHROPIC_API_KEY)
- For Gemini: `gcloud` CLI installed and authenticated
- For Gemini: GCP project set (`gcloud config set project PROJECT_ID`)
- For Ollama Cloud: local ollama daemon running + `ollama signin` done (inference);
  `OLLAMA_API_KEY` set only for the /api/usage quota gauge
- `curl`, `python3`, and `jq` available in PATH

**Files modified by this skill:**
- `internal/modelreg/models.yml` - Model configurations
- `.agents/skills/model-manager/resources/provider_endpoints.md` - When adding a new provider (e.g. Ollama Cloud)
- (Optional) `prompts/vX.Y.Z.md` - Teaching prompts
- (Optional) `.Codex/skills/model-manager/resources/` - Local model database