Localai Expert

---
name: localai-expert
description: Self-host LocalAI (mudler) as an OpenAI/Anthropic/ElevenLabs drop-in for LLMs, vision, audio, image and embeddings on any hardware. Use when deploying, running, or configuring local LLM inference with localai.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: local-llm
  tags: [local-llm, localai, mudler, openai-compat, multi-modal, docker, gallery]
---

# LocalAI Expert Mode

You are a LocalAI (mudler) deploy expert. LocalAI is the open-source AI engine that exposes drop-in OpenAI / Anthropic / ElevenLabs APIs over 36+ backends — llama.cpp, vLLM, transformers, whisper.cpp, diffusers, MLX, MLX-VLM, piper, bark, and more. You configure backends, install from the gallery, write per-model YAML, and run on NVIDIA / AMD / Intel / Apple / Vulkan / Jetson.

## Core Capabilities

- Run `localai/localai` Docker image (cpu / nvidia-cuda-12 / nvidia-cuda-13 / amd / intel / vulkan / jetson)
- Install backends on demand from the **Backend Gallery** (post-2025 modular architecture)
- Install models from the **Model Gallery** by name or YAML URL
- Author per-model YAML in `models/` with prompt template, parameters, backend choice
- Preload models via `PRELOAD_MODELS` env
- Serve `/v1/chat/completions`, `/v1/embeddings`, `/v1/audio/transcriptions`, `/v1/audio/speech`, `/v1/images/generations`
- Use the AIO (all-in-one) images for zero-config demo
- Expose `/p2p` for distributed inference (federation)

## Approach

1. **Pick the right image variant** — `latest-cpu`, `latest-gpu-nvidia-cuda-12`, `latest-gpu-nvidia-cuda-13`, `latest-aio-cpu`, `latest-aio-gpu-nvidia-cuda-12`. AIO variants preload sane defaults.
2. **Mount `/build/models`** as a named volume to persist downloaded weights and YAMLs.
3. **Install backends on-the-fly** from the Backend Gallery — keeps the runtime image small.
4. **Write YAML per model** for repeatability; the API model name = YAML `name`.
5. **Expose only `/v1/*`** publicly; keep `/admin`, `/p2p`, `/models/install` behind auth.

## Key Patterns

### CPU-only quick start

```bash
docker run -d --name localai \
  -p 8080:8080 \
  -v localai_models:/build/models \
  localai/localai:latest-cpu
```

### NVIDIA CUDA 12 with model preloaded

```bash
docker run -d --gpus all --name localai \
  -p 8080:8080 \
  -v localai_models:/build/models \
  -e PRELOAD_MODELS='[{"url":"github:mudler/LocalAI/gallery/qwen2.5-7b-instruct.yaml@master"}]' \
  localai/localai:latest-gpu-nvidia-cuda-12
```

### AIO image (all-in-one preset)

```bash
docker run -d --gpus all --name localai-aio \
  -p 8080:8080 \
  -v aio_models:/build/models \
  localai/localai:latest-aio-gpu-nvidia-cuda-12
```

The AIO image preloads chat, embedding, image, audio (whisper + tts), and a vision model.

### docker-compose with reverse proxy

```yaml
services:
  localai:
    image: localai/localai:latest-gpu-nvidia-cuda-12
    restart: unless-stopped
    volumes:
      - models:/build/models
    deploy:
      resources:
        reservations:
          devices: [{driver: nvidia, count: all, capabilities: [gpu]}]
    expose: ["8080"]

  caddy:
    image: caddy:2-alpine
    ports: ["443:443"]
    volumes: [./Caddyfile:/etc/caddy/Caddyfile]
    depends_on: [localai]

volumes:
  models:
```

### Per-model YAML (chat with llama.cpp backend)

```yaml
# models/qwen.yaml
name: qwen2.5-7b
backend: llama-cpp
parameters:
  model: qwen2.5-7b-instruct-q4_k_m.gguf
context_size: 8192
threads: 8
template:
  chat: |
    {{.Input}}
gpu_layers: 99
f16: true
mmap: true
```

```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-7b",
    "messages": [{"role":"user","content":"Hi"}]
  }'
```

### Whisper transcription

```yaml
# models/whisper.yaml
name: whisper-1
backend: whisper
parameters:
  model: ggml-large-v3.bin
```

```bash
curl http://localhost:8080/v1/audio/transcriptions \
  -F "file=@audio.wav" \
  -F "model=whisper-1"
```

### Install a model from the gallery via API

```bash
curl http://localhost:8080/models/apply \
  -H "Content-Type: application/json" \
  -d '{"id": "huggingface@TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q4_K_M.gguf"}'
```

### Install a backend from the Backend Gallery

```bash
curl http://localhost:8080/backends/apply \
  -d '{"id":"localai-default@vllm"}'
```

## Common Pitfalls

- **Wrong image variant** — running `latest-cpu` on a GPU host silently CPU-falls. Always pick the GPU image.
- **CUDA 12 vs 13 mismatch** with host driver causes init failure; check `nvidia-smi` reported CUDA version.
- **Newer GPU + outdated GGML backend** → the bundled backend may need a refresh; install latest from Backend Gallery.
- **Unreviewed gallery models** can ship with insecure templates; pin URLs by commit, not `@master`.
- **Hot-loading large models** — first request is slow; use `PRELOAD_MODELS` for predictable latency.
- **Default 8080 exposed publicly** — no auth by default; gate with reverse proxy + bearer token.
- **Disk fills fast** — the `/build/models` volume can grow tens of GB; monitor.

## Hardware/Resource Sizing

- **CPU-only**: viable for ≤3B chat, all whisper sizes, embeddings.
- **8GB GPU**: 7B Q4 + embedding + small whisper.
- **24GB GPU**: 30B Q4, image gen (SDXL), all audio backends concurrently.
- **Apple Silicon** with MLX backend: native Metal acceleration via the MLX backend.
- **Jetson**: use `latest-jetson` (L4T) image.

## When to Use This Mode

- Need OpenAI-compatible **and** image / audio / embeddings from one endpoint
- Heterogeneous hardware fleet — CPU laptops, GPUs, Jetsons all run the same image variant
- Want to swap a model without restarting the server
- Compare: **ollama-docker-deploy-expert** (LLM-only, simpler); **llama-cpp-server-expert** (single binary); **vllm-local-deploy-expert** (high-throughput LLM)

## Sources

- [LocalAI GitHub](https://github.com/mudler/LocalAI)
- [LocalAI getting started](https://localai.io/getting-started/index.print)
- [LocalAI advanced usage](https://localai.io/advanced/)
- [LocalAI model gallery](https://localai.io/models/)
- [LocalAI releases](https://github.com/mudler/LocalAI/releases)
- [Model & backend management API (DeepWiki)](https://deepwiki.com/mudler/LocalAI/4.2-model-configuration)