Inference Deployment

---
name: inference-deployment
description: "Run and debug ms-swift inference, app, deployment, backend
  selection, OpenAI-compatible serving, multimodal requests, adapters, and
  result/logprob workflows."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# Inference and Deployment

Use this sub-skill when an agent needs to run `swift infer`, `swift app`, or `swift deploy`, choose an inference backend, call a local OpenAI-compatible server, use `swift.infer_engine` Python APIs, or debug serving/runtime failures.

## Route first

- Use `references/workflows.md` for concrete CLI, app, deploy, client, batch, multimodal, LoRA, logprob, and result-saving recipes.
- Use `references/api-reference.md` for `InferArguments`, `DeployArguments`, `AppArguments`, `InferRequest`, `RequestConfig`, engines, OpenAI endpoints, and Python client shapes.
- Use `references/backend-compatibility.md` before selecting `transformers`, `vllm`, `sglang`, or `lmdeploy`.
- Use `references/troubleshooting.md` for backend/model mismatches, optional dependency failures, QLoRA limitations, multimodal OOM, `vllm_limit_mm_per_prompt`, result/logprob surprises, adapter loading, and unreachable local servers.
- Use `scripts/build_inference_command.py` to print safe `swift infer`, `swift deploy`, or `swift app` command skeletons without launching models.
- Use `scripts/smoke_openai_client.py` to test a local OpenAI-compatible `/v1/chat/completions` server; it targets localhost by default and does not use external network resources.

## Use cases covered here

- Interactive text or multimodal CLI inference with `swift infer`.
- Dataset or batch inference with saved JSONL results and optional metrics/logprobs.
- Local Gradio-style inference UI with `swift app`, either self-hosting a model or connecting to an existing base URL.
- OpenAI-compatible serving with `swift deploy`, including `/v1/models`, `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, health/ping, and Swift client calls.
- Backend selection across `transformers`, `vllm`, `sglang`, and `lmdeploy`, including optional dependency caveats.
- Full checkpoint, LoRA adapter, merged LoRA, and multiple named LoRA serving plans.

## Reroute boundaries

- Training command construction, checkpoint creation, and LoRA training arguments belong to the training sub-skill.
- Dataset schema design and custom dataset registration belong to data/model customization.
- Export, merge-only packaging, quantization, and evaluation pipelines belong to export/evaluation.
- GRPO rollout, Ray, and Megatron serving/sampling internals belong to advanced RL/distributed workflows.