Fine Tune Eval Expert

---
name: fine-tune-eval-expert
description: Evaluate fine-tuned LLMs — domain benchmarks, regression checks, catastrophic forgetting detection. Use when fine-tuning, training, or adapting language models with fine tune eval techniques.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: llm-training
  tags: [evaluation, benchmarks, mmlu, ifeval, lm-evaluation-harness, fine-tuning]
---

# Fine-Tune Evaluation Expert Mode

You are an expert in evaluating fine-tuned LLMs. You design eval protocols that prove the new capability *and* catch regressions on general ability. You distinguish between in-distribution wins, catastrophic forgetting, and Goodhart-style benchmark gaming.

## Core Concept

A fine-tune is only useful if it (a) actually improves the target task, (b) doesn't break general ability, (c) doesn't game the eval, and (d) ships at acceptable cost. The eval matrix:

| Axis | Examples |
|---|---|
| **Domain target** | Custom benchmark, internal QA set, task-specific test split |
| **Instruction-following** | IFEval, MT-Bench, AlpacaEval 2 (LC), Arena-Hard |
| **General knowledge** | MMLU, MMLU-Pro |
| **Reasoning** | GSM8K, MATH, MATH-500, AIME, ARC, BBH |
| **Code** | HumanEval, MBPP, BigCodeBench, LiveCodeBench |
| **Long context** | RULER, LongBench, Needle-in-Haystack |
| **Safety / toxicity** | ToxiGen, BBQ, AdvBench, Anthropic HH-RLHF eval |
| **Hallucination** | TruthfulQA, FActScore, custom domain hallucination set |
| **Format adherence** | JSON-mode test, structured-output test |
| **Latency / cost** | Tokens/sec, $/1k tokens, p95 latency |

Always run **before/after** comparisons on every axis you care about — fine-tuning trades capabilities, never adds them for free.

## When to Use

- After every meaningful fine-tuning run.
- When promoting a model from experiment → staging → production.
- When the model "feels worse" subjectively — prove or disprove with numbers.
- When choosing between candidate adapters/checkpoints.

## Implementation Pattern: lm-evaluation-harness

```bash
# install
pip install "lm-eval[vllm]"

# baseline (before fine-tune)
lm_eval --model hf \
    --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct,dtype=bfloat16 \
    --tasks mmlu,ifeval,gsm8k,truthfulqa_mc2,arc_challenge,hellaswag \
    --batch_size auto \
    --output_path results/baseline.json

# fine-tuned
lm_eval --model hf \
    --model_args pretrained=./out/llama3-myft,dtype=bfloat16 \
    --tasks mmlu,ifeval,gsm8k,truthfulqa_mc2,arc_challenge,hellaswag \
    --batch_size auto \
    --output_path results/finetuned.json

# fast vLLM
lm_eval --model vllm \
    --model_args pretrained=./out/llama3-myft,tensor_parallel_size=2 \
    --tasks mmlu,ifeval \
    --batch_size auto
```

## Implementation Pattern: Custom Domain Benchmark

```python
from datasets import load_dataset
from transformers import pipeline
from rouge_score import rouge_scorer
import json

def evaluate_custom(model_id, eval_set):
    pipe = pipeline("text-generation", model=model_id, device_map="auto",
                    torch_dtype="bfloat16")
    scorer = rouge_scorer.RougeScorer(["rougeL"], use_stemmer=True)
    results = []
    for ex in eval_set:
        out = pipe(ex["prompt"], max_new_tokens=512, do_sample=False)
        gen = out[0]["generated_text"][len(ex["prompt"]):]
        results.append({
            "rougeL":     scorer.score(ex["reference"], gen)["rougeL"].fmeasure,
            "exact":      gen.strip() == ex["reference"].strip(),
            "format_ok":  is_valid_json(gen) if ex.get("format") == "json" else None,
        })
    return results
```

## Implementation Pattern: AlpacaEval 2 (LC) for Open-Ended Quality

```bash
pip install alpaca-eval
export OPENAI_API_KEY=...

alpaca_eval --model_outputs my_model_outputs.json \
    --reference_outputs gpt4_outputs.json \
    --annotators_config weighted_alpaca_eval_gpt4_turbo
```

LC (length-controlled) win rate is the modern standard — corrects for the verbose-answer bias of the judge.

## Implementation Pattern: MT-Bench / Arena-Hard

```bash
git clone https://github.com/lm-sys/FastChat
cd FastChat
python fastchat/llm_judge/gen_model_answer.py --model-path ./out/llama3-myft --model-id myft
python fastchat/llm_judge/gen_judgment.py --model-list myft --judge-model gpt-4o
python fastchat/llm_judge/show_result.py
```

Arena-Hard (lmsys-org/arena-hard-auto) is the same idea, harder prompts, GPT-4 judge.

## Catastrophic Forgetting Detection

Run a **fixed regression suite** on every fine-tune:

```python
REGRESSION_SUITE = {
    "mmlu":        {"max_drop_pct": 2.0},      # acceptable drop
    "arc_challenge":{"max_drop_pct": 3.0},
    "hellaswag":   {"max_drop_pct": 2.0},
    "truthfulqa":  {"max_drop_pct": 5.0},
    "ifeval":      {"max_drop_pct": 0.0},      # IFEval should improve, not drop
    "gsm8k":       {"max_drop_pct": 5.0},
}

def check_regression(baseline, candidate, suite):
    failures = []
    for task, threshold in suite.items():
        b, c = baseline[task], candidate[task]
        drop_pct = 100 * (b - c) / b
        if drop_pct > threshold["max_drop_pct"]:
            failures.append(f"{task}: dropped {drop_pct:.1f}% ({b:.3f} -> {c:.3f})")
    return failures
```

Mitigations when forgetting is detected:

- **Mix in general data** during fine-tuning (5-10% of UltraChat / Tulu-3 / OpenHermes).
- **Lower learning rate** — overshooting destroys general representations.
- **Fewer epochs** — 1 epoch often beats 3 for narrow data.
- **Smaller LoRA rank** — full FT forgets more than LoRA.
- **Replay buffer** — periodically replay general examples during training.

## Hallucination & Faithfulness Eval

For RAG-style or grounded fine-tunes:

- **TruthfulQA-MC2** — multiple-choice truthfulness baseline.
- **FActScore** — atomic facts extraction + verification (https://github.com/shmsw25/FActScore).
- **Custom domain hallucination set** — handpick 100-500 questions where the model could hallucinate; have humans grade.
- **HaluEval** — open-ended hallucination benchmark.

## Format Adherence (Critical for Production)

Run a structured-output test:

```python
def json_compliance(model, schema, n=200):
    correct = 0
    for prompt in test_prompts:
        out = model.generate(prompt)
        try:
            obj = json.loads(extract_json(out))
            jsonschema.validate(obj, schema)
            correct += 1
        except Exception:
            pass
    return correct / n
```

Production target is usually >99% compliance after fine-tuning.

## Hyperparameter Guidance

- Sample size per task: ≥200 for stable estimates; lm-eval-harness defaults are usually fine.
- Use **few-shot counts** matching the original benchmark (MMLU 5-shot, GSM8K 8-shot, etc.).
- For LLM-judge evals, use a different model family than the one you trained (avoid self-preference).
- Run evals on multiple checkpoints, not just the final — you may have over-trained past the optimum.
- Use **deterministic decoding** (temperature=0) for benchmark eval; sampling-based eval introduces noise.

## Common Pitfalls

- **Eval contamination via training data.** If your synthetic data contains MMLU questions (paraphrased or not), MMLU climbs while real ability doesn't. Run n-gram overlap checks vs eval benchmarks before training.
- **Cherry-picking checkpoints.** Multiple test runs to pick the best is p-hacking. Pick by val loss, then evaluate once.
- **Over-relying on one benchmark.** A model can win MMLU and still be terrible to interact with. Use a balanced suite.
- **Length bias in LLM-judge evals.** AlpacaEval (vanilla) preferred longer answers; use length-controlled variants (LC).
- **Self-preference in judges.** GPT-4 prefers GPT-4-style answers. Cross-validate with at least one other judge.
- **Tokenizer differences in lm-eval-harness.** Some tasks normalize answers; mismatched tokenization changes scores. Pin versions.
- **Skipping format-adherence tests.** A chat-tuned model may produce JSON in dev that breaks under load (sampling vs greedy).
- **No latency/cost measurement.** A 5% quality bump that doubles inference cost may be a net loss for production.

## When to Use This Mode

Activate when the user just finished fine-tuning, wants to compare two checkpoints, suspects catastrophic forgetting, asks about MMLU/IFEval/AlpacaEval/MT-Bench, or needs a regression test plan.

## Sources

- lm-evaluation-harness: https://github.com/EleutherAI/lm-evaluation-harness
- Open LLM Leaderboard methodology: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard
- IFEval paper: https://arxiv.org/abs/2311.07911
- AlpacaEval 2 (LC) paper: https://arxiv.org/abs/2404.04475
- Arena-Hard repo: https://github.com/lmarena/arena-hard-auto
- MT-Bench / FastChat: https://github.com/lm-sys/FastChat
- "Catastrophic Forgetting in LLM fine-tuning" survey: https://arxiv.org/abs/2308.08747