Evaluation Chat

---
name: evaluation-chat
description: "Guide GSM8K evaluation, answer verification, and checkpoint chat
  or raw inference for train-llm-from-scratch."
disable-model-invocation: true
metadata:
  disco-role: operating
license: MIT
---

# evaluation-chat

Use this sub-skill when a future agent needs to evaluate or talk to checkpoints from the `train-llm-from-scratch` pipeline without re-opening source files.

## Use this for

- Building GSM8K evaluation commands for one checkpoint or a Base → SFT → DPO → PPO → GRPO stage table.
- Explaining greedy GSM8K accuracy, sample dumps, JSONL append files, and table rendering.
- Loading any stage checkpoint for inference and deciding chat-template mode versus raw continuation.
- Choosing chat sampling controls: greedy, temperature, top-p, top-k, max-new-token budget, and CPU/CUDA device.
- Explaining answer extraction, GSM8K correctness checks, and the verifier reward's small format bonus.

## Route elsewhere

- Training or resuming base checkpoints: `../model-pretraining/SKILL.md`.
- SFT, reward model, DPO/ORPO/KTO, PPO, or GRPO training: `../post-training-rlhf/SKILL.md`.
- Dataset preparation or validation before evaluation: `../data-preparation/SKILL.md`.
- Streamlit control-panel settings and jobs: `../configuration-ui/SKILL.md`.

## First decisions to make

1. Is the task evaluation, chat, or answer-format diagnosis?
2. Which checkpoint stage is being used: base/pretrained, SFT, DPO, PPO, GRPO, or reward checkpoint?
3. Is GSM8K data already locally available or is network/cache access allowed?
4. Should the output be deterministic (`--greedy`, required for headline eval) or sampled chat?
5. Which device is valid for the checkpoint and installed torch build: `cuda` or `cpu`?

## Reference map

- Read `references/evaluation.md` for evaluator flags, GSM8K scoring, stage tables, sample rows, and verifier behavior.
- Read `references/inference.md` for checkpoint loading, chat versus raw prompting, decoding, and chat CLI flags.
- Read `references/troubleshooting.md` when commands fail or outputs look malformed.

## Bundled helpers

- `scripts/build_eval_command.py` prints dry-run evaluation/table commands. It never loads a model or touches datasets.
- `scripts/build_chat_command.py` prints dry-run one-shot or REPL chat commands. It never loads a checkpoint.
- `scripts/check_answer_format.py` parses text with the same `<answer>`, `####`, and last-number priority used by the verifier family; it can optionally compare a gold number and estimate verifier reward.

Prefer the helpers when the user asks for commands. They make command construction explicit while avoiding accidental training, downloads, or checkpoint loads during planning.

## Operating guardrails

- Do not launch training from this sub-skill.
- Do not assume network access for GSM8K; confirm cache/data availability before running real evaluation.
- Use `--raw` for a base checkpoint; use default chat mode for instruction/post-training checkpoints.
- Use greedy decoding for comparable GSM8K numbers. Sampling is for interactive or qualitative chat, not the stage table.
- Do not treat the small verifier format bonus as proof of correctness; accuracy depends on the parsed numeric answer matching gold.