Proxy Server

---
name: proxy-server
description: "Use when working with the LiteLLM AI Gateway proxy: starting the
  CLI/server, authoring proxy YAML, calling OpenAI-compatible routes, managing
  virtual keys/teams/spend, configuring guardrails, checking health, and
  diagnosing deployment issues."
disable-model-invocation: true
metadata:
  disco-role: operating
license: NOASSERTION
---

# LiteLLM Proxy Server

Use this sub-skill for the LiteLLM AI Gateway process and its administrative APIs. Route Python SDK calls to `sdk-core`, router/model-group internals to `routing`, provider catalogs and pass-through specifics to `providers-and-endpoints`, and MCP/A2A tooling to `agent-tools`.

## Fast Path

1. Install the proxy runtime when server imports are missing: `pip install 'litellm[proxy]'`. For a CLI-focused install, use the project-provided console scripts `litellm`, `lite`, and `litellm-proxy` after installing LiteLLM.
2. Create a `config.yaml` with at least one `model_list` entry. Use `os.environ/NAME` values for provider credentials instead of writing secrets into YAML.
3. Start locally with `litellm --config config.yaml --host 0.0.0.0 --port 4000`. Use `--model <provider/model>` for a one-model smoke setup, and `--detailed_debug` only when troubleshooting.
4. Set `general_settings.master_key` or `LITELLM_MASTER_KEY` for admin APIs, then pass `Authorization: Bearer <master-or-virtual-key>` to proxy requests.
5. Validate readiness with `curl http://localhost:4000/health/liveliness`, `curl http://localhost:4000/health/readiness`, and model health with `scripts/litellm_proxy_health_check.py`.
6. Use OpenAI-compatible routes such as `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, and `/v1/models` against the proxy base URL.

## Common Workflows

- **Minimal gateway:** configure two model aliases in `model_list`, start with `litellm --config config.yaml`, confirm `/v1/models`, then call `/v1/chat/completions` with a configured alias.
- **Key/team management:** use `/key/generate`, `/key/update`, `/key/delete`, `/key/info`, `/key/list`, `/team/new`, `/team/update`, `/team/info`, and `/team/list` with an admin token. Add budgets, `team_id`, allowed models, metadata, and rate limits in request bodies.
- **Spend and usage:** enable database-backed spend logging for durable reporting. Query `/spend/logs/v2`, `/spend/keys`, `/spend/users`, `/global/spend`, `/global/spend/keys`, `/global/spend/teams`, and `/spend/calculate` based on caller permissions.
- **Guardrails:** declare `guardrails` in YAML or manage them through `/guardrails` endpoints. Attach guardrail names to teams/keys when access should be scoped.
- **Deployment:** prefer `/health/liveliness` for liveness and `/health/readiness` for readiness. Use `--num_workers`, `--run_gunicorn`, `--run_hypercorn`, or `--run_granian` deliberately; `--reload` is a dev-only option and is incompatible with multi-worker/process server modes.

## Bundled References

- Configuration recipes: `references/configuration.md`
- Admin APIs and observability: `references/admin-and-observability.md`
- Troubleshooting and deployment caveats: `references/troubleshooting.md`

## Bundled Scripts

- `scripts/litellm_proxy_health_check.py`: safe proxy health checker with argparse, short default prompts, optional YAML model loading, and JSON output.
- `scripts/query_proxy_usage.py`: CLI-token or explicit-token helper for listing models and optionally sending a tiny completion through a proxy.

## Safety Notes

- Never commit provider API keys, virtual keys, master keys, database URLs, or local machine paths in runtime examples. Use environment variable references and placeholders.
- Do not rely on source-repo example paths at runtime. Copy needed snippets into a project-owned config file.
- Many proxy management and guardrail tests require a live app, provider credentials, database fixtures, or enterprise extras; classify them before running.