Debugging A Bad Fine Tune

---
name: debugging-a-bad-fine-tune
description: >-
  Diagnose a fine-tuned model that behaves badly at inference: output that never
  stops or repeats forever, replies that ignore the training data, garbled or
  nonsense tokens, a model that answers as the wrong persona, an adapter that
  will not load against its base, or quality that collapsed after quantisation.
  Works backwards from the symptom to the cause, most often a chat template
  mismatch between training and inference, a missing end-of-sequence token, a
  tokenizer mismatch, or overfitting on too small a dataset. Use when a trained
  model is loaded and producing wrong output. Not for models that fail to load
  at all for environment reasons, not for a crash while the model is being
  trained, and not for deciding whether the model is good enough, which is
  evaluating-a-tuned-model.
license: Apache-2.0
metadata:
  version: "0.1.0"
  author: "Edward Xi Yang, Ertas AI"
  stage: "debug"
  previous_skill: "inspecting-a-model-bundle"
  next_skill: "evaluating-a-tuned-model"
---

# Debugging a bad fine-tune

Use this when a trained model loads and runs but the output is wrong: it
never stops, it repeats, it ignores what it was trained on, it comes out
garbled, it answers as the wrong model, an adapter refuses to load, or it
got worse after quantisation. This works backwards from the symptom you can
actually observe to the specific cause, rather than starting from the
training pipeline and hoping to spot the bug by inspection.

It is not for a model that fails to load for environment reasons (wrong
runtime, missing dependency, out of memory), not for a crash during
training itself, and not for deciding whether a working fine-tune is
actually better than the base model. That last question is
**evaluating-a-tuned-model**.

## Start here

If `BUNDLE-REPORT.md` exists in the project root, read its **Shape**,
**Base model** and **Defects** sections before doing anything else.
Several of the causes below are things that skill already found and named:
a missing `generation_config.json`, a chat template that only exists in
`chat_template.jinja`, a default system prompt that still claims the base
model's identity, an adapter with no recorded base model. If the Defects
section already names one of these, that is very likely your answer, not a
separate thing to re-discover.

If `BUNDLE-REPORT.md` does not exist, that is not an error. Work directly
from the bundle: identify the shape (GGUF, adapter, or merged safetensors,
the same three shapes **inspecting-a-model-bundle** uses) from the file
listing, and re-derive base model and defect information as you go using
the checks below. You do not need to run that skill first.

### Re-run the defect check

From this skill's own directory:

```bash
python3 ../inspecting-a-model-bundle/scripts/check_defects.py /path/to/bundle
```

This is cheap to re-run even if `BUNDLE-REPORT.md` already has a Defects
section, since the bundle on disk is the current source of truth.

**If Python is unavailable, or the script errors:** work the checks below
by hand. Every check in this file is expressible as opening a JSON file and
reading a field, or running the serving tool's own inspection command
(`ollama show --modelfile <name>`, `llama-cli --verbose`). Nothing here
strictly depends on the script; it only saves typing.

## General triage order

Before branching into a specific symptom, these checks catch the largest
share of cases for the least effort, in order:

1. **Print the exact token IDs the serving engine builds for a sample
   turn** and diff them against what training produced. Not the string,
   the IDs. This single check catches a chat template mismatch, a missing
   generation prompt, a doubled BOS token, and a hand-rolled template bug
   all at once.
2. **Check for a repeated token at position 0 and 1.** A doubled BOS.
3. **Confirm `pad_token_id` is not the same value as `eos_token_id`.**
4. **Confirm the tokenizer's declared `eos_token` equals the token the
   chat template actually emits to close an assistant turn**, not
   whichever token the base model shipped with.
5. **Check what fraction of the training examples exceeded the trainer's
   max sequence length.** If any did, the label for their end-of-sequence
   token was very likely cut off, not just the visible text.
6. **Check the loss mask is not entirely `-100`** for any example that
   should have contributed to training.
7. **If serving through Ollama, run `ollama show --modelfile <name>`** and
   confirm the `TEMPLATE` block is not the bare `{{ .Prompt }}` fallback
   and that stop parameters are present.
8. **Confirm the adapter is actually active on the request being sent.**
   Compare a response against the adapter name with a response against the
   base model name; if they match, the adapter is not being applied.
9. **Check the quantisation level against the model's parameter count.**
   Under 3B and below Q4_K_M is an expected quality cliff, not a bug.
10. Only after all of the above, look at training hyperparameters:
    learning rate, LoRA rank and alpha, epoch count.

Full detail and the diagnostic command for each numbered check is in
`references/failure-catalogue.md`.

## Symptom table

| Symptom | Likely causes, in order |
|---|---|
| Generation never stops | pad token equals EOS token (FM-07); training examples ran past TRL's default `max_length` of 1024 and were truncated from the tail, taking the EOS with them (FM-19); a merge step rewrote `eos_token` back to the base model's raw EOS instead of the chat-tuned end-of-turn token (FM-08); imported into Ollama with no template detected, so no stop parameters exist (FM-37) |
| Output repeats itself | pad token equals EOS token (FM-07); wrong chat template applied at inference (FM-01); a hand-written template addresses the model with a role token it was never trained on (FM-36) |
| Model ignores what it was trained on | chat template mismatch between training and inference (FM-01); the adapter is loaded but not active, or merged on some target modules and not others (FM-16); a serving request is hitting the base model's name instead of the adapter's (FM-40); learning rate was left at a full-fine-tune default while training a LoRA, which under-trains it (FM-22) |
| Garbled or nonsense output | a doubled BOS token (FM-11); the chat template baked into the file is stale relative to the upstream base (FM-05); float16 numeric overflow during training or inference on older GPUs (FM-32) |
| Answers as the wrong model or persona | the chat template's default system prompt still names the base model, not the fine-tune (see `foreign-default-identity` in `BUNDLE-REPORT.md`'s Defects section, or check directly, below); a hand-written template uses a role token the model was never trained on (FM-36) |
| Adapter will not load against the base | vocabulary size or hidden dimension mismatch between the base being loaded now and the base the adapter was trained against (FM-15); `adapter_config.json`'s `base_model_name_or_path` points at the wrong repository or revision (FM-15); loaded with the wrong API, `get_peft_model()` on a checkpoint that needs `PeftModel.from_pretrained()` (FM-15) |
| Quality collapsed after quantisation | the model is under roughly 3B parameters and was quantised below Q4_K_M (FM-29); the file was requantised from an already-quantised source instead of a clean fp16 or bf16 original (FM-31); a LoRA was merged into an already-quantised base before conversion, corrupting the tensor layout on top of the quality loss (FM-18, FM-25) |
| Far worse than it looked during training | chat template mismatch, the single most common cause across this whole table (FM-01); catastrophic forgetting from training too many epochs on too narrow a dataset (FM-24); overfitting, training loss dropped well below the healthy range (FM-23) |

Every cause below has its full write-up, the diagnostic command, the fix,
and the upstream issue it was confirmed against in
`references/failure-catalogue.md`, grouped by category: chat template,
tokens and tokenizer, adapter and base, training hyperparameters,
quantisation, context length, and runtime-specific.

## Chat template mismatch, in more depth

This is the number one cause behind more than half of the rows in the
table above, so it has its own file:
`references/chat-template-debugging.md`. Go there for how to extract the
template actually baked into each of the three bundle shapes, how to
render it against a sample conversation, how to diff it against the base
model's template, and the specific way a hand-rolled template silently
drops the BOS token or the tool-calling turns.

## Checking the wrong-persona symptom directly

If `BUNDLE-REPORT.md` is not available, check for this by hand: open
`chat_template.jinja` and `tokenizer_config.json` if either exists, and
search for the base model's name as a bare identity claim ("You are
Qwen", "You are Llama") or an attribution clause ("created by Alibaba",
"created by Meta"). Either on its own is enough; the model does not need
to say the whole "you are X, created by Y" sentence for this to fire. It
only shows up when the caller sends no system message, which is why it is
easy to miss in testing. The fix is to replace the default system prompt
in the template, or to always pass an explicit system message at
inference.

## Write the findings

Once the symptom is identified, append a `## Debug findings` section to
`BUNDLE-REPORT.md` (create the file with just this section if it does not
already exist). Use exactly this structure:

```markdown
## Debug findings

### Symptom
<one line, in the words the user actually reported it>

### Checked, in order
<check -> result, one line each, following the triage order above>

### Cause
<the failure catalogue id and a one-line description, or "not identified
by the checks above" if nothing in the catalogue matched>

### Fix
<the fix, specific to what was found>

### Confidence
<high | medium | low>, one line on why
```

**If nothing in the catalogue matches:** say so plainly rather than
guessing at a fix. Report the exact evidence collected (the token IDs, the
config values, the template diff) as an "unknown, evidence attached"
result. A wrong guess here is worse than an honest unknown, because the
next person to look at this trusts a written finding more than they
re-check it.

## Hand off to

- The cause is a missing base model, a defect in the bundle itself, or the
  shape was never confirmed: **inspecting-a-model-bundle**
- The symptom is fixed and the next question is whether the fine-tune is
  actually better than the base model, not just working:
  **evaluating-a-tuned-model**
- The cause is a training hyperparameter or dataset problem that requires
  re-training: fix the training configuration per
  `references/failure-catalogue.md` and re-run the training job
- The retrain also means reconsidering where it runs, because the last run
  was slow, expensive, or awkward enough to be part of the problem:
  **scoping-a-custom-model**, whose `references/training-options.md` resolves
  a stated criterion to one platform rather than listing every option
- The retrain is worth doing only if the model earns its keep at the volume
  the app actually sees: **costing-a-model-vs-an-api**