Evaluation

---
name: evaluation
description: "Enable built-in perplexity evaluation with `model.evaluate` and
  scaffolded adapter-based evaluation with persisted JSON results."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# Evaluation

Use this sub-skill for xTuring evaluation work that stays inside the package's current contract.

## When to use
- Score a causal model with `model.evaluate(...)` and get a single perplexity scalar.
- Wrap an evaluation backend behind `BaseEvalAdapter`.
- Persist a standardized `EvalRunResult` JSON artifact with `run_eval_adapter(...)` or `persist_eval_result(...)`.
- Inspect or normalize `EvalMetric` and `EvalRunResult` payloads.

## When not to use
- The dataset schema is still invalid or needs conversion.
- You need model selection, loading, generation config tuning, or inference debugging.
- You need training, serving, or API/UI behavior.
- You expect full lm-evaluation-harness execution; the current adapter is scaffold-only.

## Read in this order
1. `references/evaluation-workflows.md`
2. `references/api-reference.md`
3. `references/troubleshooting.md`

## Main entry points
- `CausalModel.evaluate(dataset, batch_size=1)`
- `BaseEvalAdapter`
- `LMEvalAdapter`
- `run_eval_adapter(...)`
- `persist_eval_result(...)`
- `EvalMetric`
- `EvalRunResult`

## Contract summary
- `model.evaluate(...)` returns one perplexity tensor, not a metric table.
- `run_eval_adapter(...)` returns `EvalRunResult` and fills timing fields when the adapter does not.
- `persist_eval_result(...)` writes the result as UTF-8 JSON with parent directories created as needed.
- `LMEvalAdapter` currently returns `status="planned"` and `metadata["integration_status"]="scaffold_only"`.

## Safe smoke check
Use `scripts/evaluation_scaffold_smoke.py` to verify the adapter and persistence contract without downloading a model.

If a dataset fails validation, fix it in the data sub-skill first; evaluation assumes an already-valid dataset object.