Model Profile Analysis

---
name: model-profile-analysis
description: 'User shares eval profile JSON, asks to compare models, or discusses
  model routing decisions. Keywords: "profile,"...'
triggers:
- 'User shares eval profile JSON, asks to compare models, or discusses model routing
  decisions. Keywords: "profile,"...'
version: '1.0'
author: Exocortex
---

# Skill: Model Profile Analysis

## Trigger
User shares eval profile JSON, asks to compare models, or discusses model routing decisions. Keywords: "profile," "eval results," "compare models," "4B vs 14B," "what does the data show."

## Inputs Required
- **Profile JSON file(s)** — generated by eval_runner.py, located in uploads or at known paths
- **Known model profiles for comparison:**
  - Qwen3-4B: `qwen_qwen3-4b-2507.json`
  - Qwen3-14B: `unsloth_qwen3-14b.json`

## Procedure

### 1. Read the Profile
Load the JSON and identify which eval modules were run. Standard modules:
- `bst` — Belief State Tracker compliance
- `tool_reliability` — JSON validity, parameter accuracy, tool selection, error recovery
- `graph_compliance` — workflow graph adherence
- `pace_calibration` — PACE escalation framework
- `memory` — noise discrimination, reference rate, accuracy rate
- `context_sensitivity` — baseline performance, optimal injection, degradation threshold

### 2. Module-by-Module Analysis
For each module present, extract:
- **Raw scores** — the specific numbers
- **Delta from baseline or previous profile** — if comparing
- **Anomalies** — scores that are surprisingly high or low given the model size
- **Disabled domains** — for BST, which domains were disabled and why

### 3. Identify Capability Profile
Categorize the model by what it's good and bad at:
- **Precision operations** — tool reliability, JSON formatting, parameter accuracy
- **Strategic reasoning** — PACE calibration, graph compliance
- **Context utilization** — BST compliance improvement, memory accuracy
- **Noise filtering** — memory noise discrimination, context sensitivity degradation

The 4B/14B comparison established the template:
- 4B = precision tool operator (100% JSON, 80% params, 100% tool selection)
- 14B = strategic follower (perfect PACE, perfect graph, but tool reliability collapse)

New profiles should be positioned relative to these established patterns.

### 4. Routing Implications
Based on the capability profile, recommend:
- **What roles should this model fill?** — supervisor, specialist, utility
- **What tasks should it handle?** — tool-heavy, reasoning-heavy, classification
- **What scaffolding does it need?** — which Exocortex layers are critical for this model
- **What to disable?** — BST domains where enrichment hurts, features that degrade performance

### 5. Actionable Recommendations
Always end with specific next steps:
- Re-run recommendations (e.g., "re-run with thinking disabled to isolate variable")
- Profile configuration changes (e.g., "set `bst_domains_where_enrichment_hurts` to [bugfix, codegen]")
- Architecture implications (e.g., "route all tool-call tasks to 4B, plan execution to 14B")

## Output Format
Conversational analysis with specific numbers. Use inline metrics, not tables (tables are for specs, not analysis). When comparing two models, go metric by metric and state implications for each difference.

## Quality Checks
- [ ] Every claim about model capability cites a specific metric from the profile
- [ ] Anomalies are identified and hypotheses offered (e.g., thinking tokens interfering)
- [ ] Routing recommendations are specific to roles and task types
- [ ] Re-run recommendations include the exact command with flags
- [ ] Comparison positions new profiles relative to established 4B/14B patterns

## Anti-Patterns
- **Treating profile data as absolute.** Eval profiles are samples from a distribution. A single run can have noise. If a score is surprising, recommend a re-run before building architecture around it.
- **Ignoring runtime metadata.** Total runtime, API call count, and per-call timing reveal whether the right model was actually running. Check these first.
- **"Bigger is better" assumptions.** The 4B/14B comparison proved this wrong. 4B outperforms 14B on tool reliability by a wide margin. Always let the data lead.
- **Skipping the thinking tokens hypothesis.** For any model showing degraded tool reliability with extended reasoning, recommend a re-run with thinking disabled. This is now a known failure mode.