Inference And Serving

---
name: inference-and-serving
description: "Use this sub-skill for LlamaFactory inference, ChatModel usage,
  chat/webchat/api commands, OpenAI-compatible serving, backend selection,
  streaming, score evaluation, API auth, and batch vLLM prediction workflows."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# Inference and Serving

Use this sub-skill when an agent needs to run or debug LlamaFactory inference and serving flows for version `0.9.6.dev0` source behavior.

## Route Here For

- Using `ChatModel` from Python for `chat`, `stream_chat`, `get_scores`, and their async variants.
- Launching `llamafactory-cli chat`, `lmf chat`, `llamafactory-cli webchat`, `llamafactory-cli api`, or `lmf api`.
- Calling the OpenAI-compatible FastAPI server endpoints `/v1/models`, `/v1/chat/completions`, and `/v1/score/evaluation`.
- Choosing `infer_backend: huggingface`, `infer_backend: vllm`, or `infer_backend: sglang` for inference configs.
- Diagnosing streaming SSE, API key, score-vs-chat endpoint, and missing backend package failures.
- Planning safe batch prediction with vLLM and optional BLEU/ROUGE-style metric output.

## Boundaries

- Route model, tokenizer, quantization, adapter merge, and export failures to the `model-loading-and-export` sub-skill.
- Route template, prompt formatting, multimodal placeholder, dataset, and preprocessing issues to `data-and-templates` unless the failure is specific to API request payload handling.
- Route non-API LlamaBoard/Web UI install, browser, port, and operational issues to `webui-and-ops`.
- Keep runtime instructions self-contained; do not depend on repository checkout scripts or examples.

## Fast Path

1. Pick an inference config YAML with `model_name_or_path`, `template`, and `infer_backend`; the default architecture is v0 unless `USE_V1=1` is explicitly set.
2. For local interactive testing, run `llamafactory-cli chat CONFIG.yaml` or `lmf chat CONFIG.yaml`.
3. For OpenAI-compatible serving, run `API_PORT=8000 llamafactory-cli api CONFIG.yaml` and call `http://HOST:PORT/v1/...` endpoints.
4. Use `API_KEY` only when bearer-token auth is desired; clients must send `Authorization: Bearer $API_KEY` when it is set.
5. Prefer `infer_backend: huggingface` for broad compatibility and score evaluation; use `vllm` or `sglang` for faster generation when those optional packages and GPU resources are available.

## Bundled References

- `references/chat-and-api.md` covers `ChatModel`, CLI commands, API routes, request shapes, streaming, auth, env vars, and native verification candidates.
- `references/backend-selection.md` compares Hugging Face, vLLM, and SGLang inference backends and includes batch vLLM metric flow notes.
- `references/troubleshooting.md` maps common serving failures to likely causes and fixes.

## Bundled Scripts

- `scripts/openai_api_smoke.py` is a standalone OpenAI-compatible smoke client for model listing, chat, optional streaming, optional score evaluation, and API-key verification.
- `scripts/vllm_batch_infer.py` is a safe wrapper that prints or optionally runs a LlamaFactory batch vLLM command only when the caller supplies concrete model/config arguments.