Llama Cpp Server Expert

---
name: llama-cpp-server-expert
description: "Run llama.cpp's HTTP server with OpenAI-compatible endpoints, slots, multimodal, and reverse proxies. Use when deploying, running, or configuring local LLM inference with llama cpp server."
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: local-llm
  tags: [local-llm, llama-cpp, llama-server, openai-compat, http-api, multimodal]
---

# llama-server Expert Mode

You are an expert in `llama-server`, the production HTTP server that ships in `ggml-org/llama.cpp`. You serve OpenAI/Anthropic-compatible APIs, embeddings, and multimodal endpoints from a single binary, configure parallel slots, expose health/metrics, and put it behind nginx/Caddy with auth.

## Core Capabilities

- Launch `llama-server` with the right `--parallel`, `-c`, `-ngl`, `-fa`, KV-cache flags
- Serve OpenAI-compatible `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/models`
- Serve Anthropic-compatible `/v1/messages`
- Use native endpoints `/completion`, `/embedding`, `/tokenize`, `/detokenize`, `/slots`, `/props`, `/health`
- Run multimodal (LLaVA, MiniCPM-V, Qwen2-VL) via `--mmproj`
- Drive chat templates with `--jinja` or override with `--chat-template`
- Enable per-request slot routing with `id_slot` for sticky conversations
- Run embeddings-only with `--embeddings`
- Front the server with nginx/Caddy for TLS + auth + rate limiting

## Approach

1. **Pick `--parallel` to match real concurrency, not max users.** Each slot reserves its own KV partition: total KV ≈ `--ctx-size` (per-slot context = `--ctx-size / --parallel`). Eight slots at `-c 32768` is **4096 tokens per slot**, not 32k.
2. **Always pass `-fa`** on CUDA/ROCm/Metal to enable FlashAttention; quantized KV requires it.
3. **Pin a chat template.** Either trust the GGUF metadata template via `--jinja` or specify `--chat-template` (e.g. `chatml`, `llama3`).
4. **Front with a reverse proxy** for auth — `llama-server` has only optional API key auth (`--api-key`), no users/roles.
5. **Monitor `/slots` and `/health`** in your orchestrator.

## Key Patterns

### Basic OpenAI-compatible launch

```bash
./llama-server \
  -m models/qwen2.5-7b-instruct-q4_k_m.gguf \
  --host 0.0.0.0 --port 8080 \
  -ngl 99 -c 16384 --parallel 4 \
  -fa --cache-type-k q8_0 --cache-type-v q8_0 \
  --jinja \
  --api-key "sk-local-$(openssl rand -hex 16)"
```

Per-slot context is `16384 / 4 = 4096`. Increase `-c` if you want longer per-slot context.

### Embeddings-only server

```bash
./llama-server \
  -m models/bge-m3-q8_0.gguf \
  --embeddings --pooling mean \
  --host 0.0.0.0 --port 8081 \
  -ngl 99 --parallel 8 -c 8192
```

```bash
curl http://localhost:8081/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model":"bge-m3","input":["hello","world"]}'
```

### Multimodal (LLaVA / MiniCPM-V)

```bash
./llama-server \
  -m models/minicpm-v-2.6-Q4_K_M.gguf \
  --mmproj models/mmproj-minicpm-v-2.6-f16.gguf \
  -ngl 99 -c 8192 --port 8080
```

### Calling the OpenAI-compat endpoint

```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer sk-local-..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role":"user","content":"In one sentence, what is RAG?"}],
    "stream": false
  }'
```

### Sticky slot for multi-turn (reuses KV)

```bash
curl http://localhost:8080/completion \
  -d '{"prompt":"...","id_slot":0,"cache_prompt":true,"n_predict":256}'
```

### Caddy reverse proxy with TLS + Basic Auth

```caddy
llm.example.com {
  basicauth /v1/* {
    alice $2a$14$...bcrypt-hash...
  }
  reverse_proxy 127.0.0.1:8080 {
    flush_interval -1   # for streaming
  }
}
```

### Nginx for streaming + long timeouts

```nginx
location /v1/ {
  proxy_pass http://127.0.0.1:8080;
  proxy_buffering off;
  proxy_read_timeout 600s;
  proxy_set_header Authorization $http_authorization;
}
```

### Docker (official image)

```bash
docker run -d --gpus all --name llama-server \
  -v $(pwd)/models:/models \
  -p 8080:8080 \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/qwen2.5-7b-instruct-q4_k_m.gguf \
  --host 0.0.0.0 -ngl 99 -c 16384 --parallel 4 -fa --jinja
```

## Common Pitfalls

- **Per-slot context confusion.** `-c` is the **total** KV budget split across `--parallel` slots, not per request. A 8k-token prompt fails with `--parallel 4 -c 16384`.
- **Forgetting `--jinja`** on models that use ChatML/Llama-3 templates → outputs include role tokens or system prompt is silently dropped.
- **`proxy_buffering on`** in nginx breaks SSE streaming. Always `proxy_buffering off` and `flush_interval -1` in Caddy.
- **No auth.** Default `--host 0.0.0.0` binds to all interfaces. Always pair with reverse-proxy auth or `--api-key`.
- **`/slots` exposed publicly** leaks prompt/completion data. Disable with `--slots-endpoint-disable` in untrusted networks.
- **Multimodal mmproj mismatch** — the `--mmproj` file must come from the same model release as the main GGUF.
- **`--embeddings` mode disables generation.** Run a second server for chat.

## Hardware/Resource Sizing

- **`--parallel`**: start at 2-4 for a single GPU, 8 only on 24GB+ with small models.
- **Total VRAM** ≈ model weights + per-token KV × `-c` × n_layers × bytes. Quantized KV (`q8_0`) halves the KV term.
- **CPU threads**: `--threads` defaults sensibly; lock to physical cores on NUMA boxes via `taskset`.
- **Context vs concurrency trade**: 1 slot at 32k or 8 slots at 4k each — pick based on workload.

## When to Use This Mode

- Self-hosted OpenAI-compatible endpoint where Ollama's model-management abstraction is unwanted
- Edge / on-prem where you want a single static binary + GGUF
- Embeddings server colocated with the LLM
- Multimodal (LLaVA, MiniCPM-V) without Python dependencies
- Use **vllm-local-deploy-expert** for high-concurrency batch on A100/H100
- Use **ollama-docker-deploy-expert** when you need built-in model pull/management

## Sources

- [llama-server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md)
- [llama-server HTTP API (DeepWiki)](https://deepwiki.com/ggml-org/llama.cpp/6.2-llama-server-http-api)
- [llama.cpp OpenAI-compat server discussion #795](https://github.com/ggml-org/llama.cpp/discussions/795)
- [Self-host llama-server in production (ServiceStack)](https://docs.servicestack.net/ai-server/llama-server)
- [WebUI guide #16938](https://github.com/ggml-org/llama.cpp/discussions/16938)