Evaluation Serving

---
name: evaluation-serving
description: "Evaluate LitGPT checkpoints with LM Evaluation Harness and serve
  LitGPT models through LitServe, including optional dependency checks, endpoint
  modes, result files, and safe preflight planning."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# LitGPT Evaluation and Serving

Use this sub-skill when an agent needs to run or plan `litgpt evaluate`, list LM Evaluation Harness tasks, interpret evaluation result files, start `litgpt serve`, or build Simple/Streaming/OpenAI-compatible request examples.

## Route First

- For checkpoint download, conversion, validation, LoRA merge, tokenizer/config layout, or `litgpt validate`, use `../checkpoint-conversion/` first.
- For local one-shot generation, chat, prompt styles, `LLM.load`, `LLM.generate`, or non-HTTP streaming, use `../inference-chat/`.
- For finetuning, pretraining, dataset formatting, recipe adaptation, or training-time eval settings, use `../training-data/`.

## Safe Workflow

1. Confirm the checkpoint is ready before any long operation; if layout is uncertain, route to `../checkpoint-conversion/`.
2. Run the bundled preflight checker instead of starting services or benchmarks:
   `python scripts/check_optional_eval_serve_deps.py --checkpoint-dir CHECKPOINT_DIR --mode both`.
3. For evaluation, install/confirm `lm_eval`, choose a small `--tasks` set, set `--limit` for smoke runs, and choose an explicit `--save_filepath` when results must be collected.
4. For serving, install/confirm `litserve`; also confirm `jinja2` and tokenizer chat-template readiness when using `--openai_spec true`.
5. Build request examples before starting a server:
   `python scripts/build_curl_examples.py --port 8000 --api-path /predict --mode simple` or `--mode openai --api-path /v1/chat/completions`.
6. Treat `litgpt serve` and full LM Harness runs as long-running/manual steps unless the user explicitly approves a bounded run.

## Quick Commands

- List tasks: `litgpt evaluate list`.
- Smoke evaluate: `litgpt evaluate CHECKPOINT_DIR --tasks hellaswag --batch_size 1 --limit 10 --out_dir eval-out --save_filepath eval-out/results.json`.
- Force reconversion after checkpoint changes: add `--force_conversion true`.
- Serve simple JSON API: `litgpt serve CHECKPOINT_DIR --port 8000 --api_path /predict`.
- Serve streaming simple API: `litgpt serve CHECKPOINT_DIR --stream true --api_path /predict`.
- Serve OpenAI-compatible API: `litgpt serve CHECKPOINT_DIR --openai_spec true --port 8000`.

## References

- `references/cli-reference.md` summarizes `litgpt evaluate` and `litgpt serve` options.
- `references/evaluation-recipes.md` covers LM Evaluation Harness task listing, conversion behavior, limits, seeds, and result files.
- `references/deployment-recipes.md` covers LitServe Simple, Stream, and OpenAI-compatible endpoints.
- `references/troubleshooting.md` maps common symptoms to safe fixes.
- `scripts/check_optional_eval_serve_deps.py` checks optional imports, batch-size syntax, port availability, and checkpoint readiness hints.
- `scripts/build_curl_examples.py` generates curl examples for simple, streaming, and OpenAI-compatible requests.