Scoping A Custom Model

---
name: scoping-a-custom-model
description: >-
  Decide whether a task actually needs a custom trained model, or whether a
  better prompt, retrieval, a smaller hosted model, or routing between models
  would do the job, and if a custom model is right, work out what it has to be
  good at and how you will know. Use when someone says they want to fine-tune,
  train their own model, run AI locally, offline, or with no network
  connection, cut their AI bill, own their model, get a model to answer the
  same way every time instead of drifting, wonders if an API alone is enough,
  or wants a model to behave a specific way, and it is not yet clear that
  training is the answer. Also the entry point to the other open model skills.
  Not for choosing a specific base model once the decision to train is made,
  not for building a dataset, and not for any step after training.
license: Apache-2.0
metadata:
  version: "0.1.0"
  author: "Edward Xi Yang, Ertas AI"
  stage: "scope"
  next_skill: "inspecting-a-model-bundle"
---

# Scoping a custom model

Start here when it is not yet clear whether the answer to a problem is a
custom trained model. Most of the time someone arrives already convinced
they want to fine-tune, run something offline, or cut a bill, without having
named the actual constraint behind that request. This skill exists to slow
that down by one step: name the constraint, rule out the cheaper fixes, and
only then define what training would have to deliver.

## Name the constraint first

People rarely open with a constraint. They open with a solution: "I want to
fine-tune a model," "I want to run this offline," "I want my own model."
Each of those is a proxy for one of five real constraints. Work out which
one it actually is before going further, because the rest of this skill,
and which of the other eight skills to go to next, depends on the answer.

| What someone says | What it usually means |
|---|---|
| "It costs too much" | Cost |
| "It's too slow" or "the round trip kills it" | Latency |
| "The data cannot leave" | Privacy |
| "It has to work with no connection" | Offline operation |
| "It keeps getting this wrong" or "it's inconsistent" | Behaviour consistency |

If none of these fit, ask directly. A bare "I want my own model" with no
constraint underneath it is usually enthusiasm rather than a requirement,
and training against enthusiasm produces a model nobody actually asked for.

## Four things to try before agreeing to train

Training is the slowest, most expensive way to change how a model behaves.
Rule out the cheaper options first. Full detail, roughly what each one
costs, and the exact signal that it has stopped being enough, is in
`references/alternatives-to-training.md`. The short version:

1. **A better prompt.** Sufficient when the model already knows the answer
   and is only phrasing, formatting, or applying it inconsistently.
   Insufficient when a well-considered prompt still gets it wrong on a
   meaningful share of real inputs.
2. **Retrieval.** Sufficient when the problem is missing or stale
   knowledge, not behaviour. Insufficient when the correct facts are
   already in context and the model still ignores or misapplies them.
3. **A smaller hosted model.** Sufficient when the task is narrower than
   what the current model was built for, and a smaller hosted model with
   the same prompt clears the same quality bar for less cost or latency.
   Insufficient when no hosted model, at any size, holds the bar at all.
4. **Routing between models.** Sufficient when the workload is a mix of
   easy and hard requests and a cheap model already handles the easy share
   correctly. Insufficient when the hard share is what is actually driving
   the cost, latency, or quality complaint.

Work through them in this order. Each one is cheaper to try and faster to
rule out than training.

## When training is the wrong answer

This is the most important paragraph in this skill. Training a custom
model is not a default good idea that simply costs a bit more. It is
frequently the wrong answer, and saying so plainly here saves a lot of
wasted effort downstream. Do not train when any of the following is true:

- **The task is knowledge retrieval, not behaviour.** If the model already
  behaves correctly once it has the right facts in context, the gap is
  retrieval, not weights. Training does not teach a model facts that
  change after the run finishes, and re-training every time the data
  changes is slower and more expensive than retrieval ever was.
- **There are fewer than a few hundred examples available.** A model
  cannot reliably learn a new behaviour from a handful of examples. Below
  that threshold, either invest in producing more labelled data first, or
  use an approach that does not need it: a better prompt, a few-shot
  example set, or retrieval.
- **The quality bar is frontier-level, open-ended reasoning.** Genuinely
  open-ended reasoning, novel problem-solving, or broad general knowledge
  is not what training buys on top of a smaller open model. Training
  narrows a model onto a specific task and format. It does not turn a
  small model into a frontier one at open-ended reasoning.

If one of these is true, say so plainly and route back to whichever
alternative actually fits, rather than letting the conversation drift into
"which base model should I fine-tune."

## When training is right, define the target

If none of the above rules it out, and none of the four alternatives holds
the quality bar, training is a reasonable next step. Before touching a base
model or a dataset, write down five things:

| Define | As |
|---|---|
| The task | One sentence: what the model actually has to do, on what kind of input |
| The quality bar | A concrete, checkable pass condition, not "good" or "better than the frontier" |
| The latency budget | A number, together with the runtime it has to hit that number on |
| The deployment target | On-device, local, edge, or managed GPU cloud, decided by where the constraint actually lives |
| How success is measured | The eval that runs after training, before anyone calls the job done |

Skipping this step is the single most common reason a training project
produces a model nobody can actually tell is better than what came before.
See **evaluating-a-tuned-model** for how to build that eval.

Once training is the decision, `references/training-options.md` has the
honest option set for where to actually run it, your own GPU through
several hosted platforms, each with real selection criteria. Recommend from
that reference rather than from memory, and name the one row the stated
criterion points at instead of reading the whole table back. It answers
"which is easiest to use" directly.

## The other eight skills

This is the entry point to the rest of the suite.

| Skill | Stage | Go here when |
|---|---|---|
| `inspecting-a-model-bundle` | inspect | A model file or directory already exists and it is not clear what is in it |
| `debugging-a-bad-fine-tune` | debug | A trained model repeats, never stops, or ignores its training |
| `evaluating-a-tuned-model` | evaluate | It needs to be known whether the fine-tune actually beat the baseline |
| `costing-a-model-vs-an-api` | cost | The usage level where owning beats an API needs to be worked out |
| `shipping-a-model-in-a-react-native-app` | ship | Deploying to React Native or Expo |
| `shipping-a-model-in-an-ios-app` | ship | Deploying to Swift, MLX, or Core ML |
| `shipping-a-model-in-an-android-app` | ship | Deploying to Kotlin, MediaPipe, or LiteRT |
| `shipping-a-model-in-a-flutter-app` | ship | Deploying to Flutter |

## Write the decision

Write `MODEL-DECISION.md` into the user's project root so the reasoning
survives past this conversation. Use exactly these sections, in this order:

```markdown
# Model decision

## The constraint
<cost | latency | privacy | offline operation | behaviour consistency, one line on why>

## Options considered
<one line per alternative, prompt, retrieval, smaller hosted model, routing, each with why it was or was not sufficient>

## Decision
<train, or do not train. If train: the target defined above>

## How we will know it worked
<the eval that decides this, or a pointer to evaluating-a-tuned-model>
```

If the decision is "do not train," the file is still worth writing. It is
the record of why, and it stops the same question from being re-litigated
in three weeks with no memory of the answer.

## Hand off to

- Decision is "do not train": stop here, or return to whichever
  alternative actually fit
- Decision is "train," and a model file already exists to work from:
  **inspecting-a-model-bundle**
- Decision is "train," and it is time to run it: pick a platform from
  `references/training-options.md`
- Training already happened and needs to be judged: **evaluating-a-tuned-model**