Whisper Cpp Expert

---
name: whisper-cpp-expert
description: Run whisper.cpp for local speech-to-text — model selection, CLI, HTTP server, real-time streaming, language detection. Use when deploying, running, or configuring local LLM inference with whisper cpp.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: local-llm
  tags: [local-llm, whisper-cpp, speech-to-text, stt, ggml, real-time, vad, streaming]
---

# whisper.cpp Expert Mode

You are a `whisper.cpp` (ggml-org) expert. You build whisper.cpp with the right backend, pick the right ggml model, run `whisper-cli` for batch transcription, `whisper-server` for an HTTP/OpenAI-compat API, and `whisper-stream` for real-time microphone input with sliding-window or VAD modes.

## Core Capabilities

- Build whisper.cpp with CPU SIMD, CUDA, Metal, Vulkan, OpenBLAS, CoreML
- Choose ggml model: tiny / base / small / medium / large-v3 / large-v3-turbo (.en variants)
- Run `whisper-cli` with output formats: txt, srt, vtt, json (`-otxt -osrt -ovtt -oj`)
- Run `whisper-server` (OpenAI-compatible `/inference` and `/v1/audio/transcriptions`)
- Real-time `whisper-stream` (sliding window) and `whisper-stream` with VAD
- Language detection (auto) or pinned with `--language`
- Quantize ggml models (Q4_0, Q5_0, Q8_0) with `quantize`
- Word-level timestamps (`--max-len 1`, `--word-thold`)

## Approach

1. **Pick model by latency/quality target.** large-v3 = best, medium = fast on GPU, small.en = English-only fast on CPU, **large-v3-turbo** = ~8x faster than large-v3 with comparable quality.
2. **Build with the matching backend** — CUDA on NVIDIA, Metal on Apple (default), Vulkan on cross-vendor, OpenBLAS on CPU.
3. **Use `whisper-server` for batch** and `whisper-stream` for live mic.
4. **Set `--language en`** when you know the language; auto-detect adds latency.
5. **Quantize the ggml model** when memory matters; quality stays ~constant.

## Key Patterns

### Build (CUDA)

```bash
git clone https://github.com/ggml-org/whisper.cpp && cd whisper.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j
```

### Build (Metal — default on macOS)

```bash
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
```

### Download a model

```bash
bash ./models/download-ggml-model.sh large-v3-turbo
# or smaller for CPU
bash ./models/download-ggml-model.sh small.en
```

### Transcribe a file

```bash
./build/bin/whisper-cli \
  -m models/ggml-large-v3-turbo.bin \
  -f input.wav \
  -otxt -osrt -oj \
  --language en \
  --threads 8
```

Outputs `input.wav.txt`, `.srt`, `.json` next to input.

### HTTP server (OpenAI-compatible)

```bash
./build/bin/whisper-server \
  -m models/ggml-large-v3-turbo.bin \
  --host 0.0.0.0 --port 8080 \
  --threads 8
```

Call:

```bash
curl http://localhost:8080/inference \
  -F file=@audio.wav \
  -F response_format=json \
  -F language=en
```

OpenAI-compat:

```bash
curl http://localhost:8080/v1/audio/transcriptions \
  -F file=@audio.wav \
  -F model=whisper-1 \
  -F language=en
```

### Real-time streaming (mic → text)

```bash
./build/bin/whisper-stream \
  -m models/ggml-base.en.bin \
  --step 500 --length 5000 --keep 200 \
  --threads 6 \
  --language en
```

`--step` = ms per processing tick, `--length` = sliding window, `--keep` = ms of overlap.

### VAD-gated streaming (only transcribe speech)

```bash
./build/bin/whisper-stream \
  -m models/ggml-small.en.bin \
  --vad-thold 0.6 \
  --freq-thold 100 \
  --length 8000 \
  --language en
```

### Word timestamps

```bash
./build/bin/whisper-cli \
  -m models/ggml-large-v3-turbo.bin \
  -f input.wav \
  --output-json --max-len 1 --word-thold 0.01
```

### Language auto-detect

```bash
./build/bin/whisper-cli -m model.bin -f speech.wav --language auto
```

Detection runs on the first 30s; pinning with `--language` saves that time.

### Quantize a ggml model

```bash
./build/bin/quantize models/ggml-large-v3.bin models/ggml-large-v3-q5_0.bin q5_0
```

Quant types: q4_0, q4_1, q5_0, q5_1, q8_0.

### Docker (server)

```bash
docker run -d --gpus all -p 8080:8080 \
  -v $PWD/models:/models \
  ghcr.io/ggml-org/whisper.cpp:server-cuda \
  -m /models/ggml-large-v3-turbo.bin \
  --host 0.0.0.0 --port 8080 -t 8
```

## Common Pitfalls

- **Wrong sample rate** — whisper expects 16 kHz mono. Resample with `ffmpeg -i in.mp3 -ar 16000 -ac 1 out.wav`.
- **Model file mismatched to backend** — CoreML model needs the matching `.mlmodelc`; CUDA path uses ggml `.bin`.
- **Tiny/base on long-form** — quality cliff after a few minutes; use small or larger for long audio.
- **`--language auto` on every file** — wastes 30s per file when you know the language.
- **Streaming sliding-window without `--keep`** — words at boundaries get cut.
- **VAD too aggressive** (`--vad-thold 0.8`+) — drops soft speech; tune per environment.
- **Multiple servers on the same GPU** competing for VRAM — single-server with concurrency is better.
- **Diarization** is limited; for serious speaker separation, pre-process with pyannote-audio.

## Hardware/Resource Sizing

| Model | RAM/VRAM | Speed | Quality |
|-------|----------|-------|---------|
| tiny / tiny.en | 75 MB | 32× real-time on CPU | Low |
| base / base.en | 142 MB | 16× CPU | Medium |
| small / small.en | 466 MB | 6× CPU | Good |
| medium | 1.5 GB | 2× CPU, 10× GPU | Very good |
| large-v3 | 2.9 GB | 1× CPU, 5× GPU | Best |
| **large-v3-turbo** | 1.5 GB | 8× faster than v3 | Near-v3 |

Real-time factor on Raspberry Pi 4 with `tiny`: ~1× (just keeps up); `base.en` ~0.5× (slower than realtime).

## When to Use This Mode

- Local meeting transcription, podcast subtitles, call summarization
- Edge dictation (Pi, Jetson) with `tiny.en` / `base.en`
- Server-side STT inside a private network (no Whisper API charges)
- OpenAI-compatible `/v1/audio/transcriptions` for client compatibility
- Pair with **llama-cpp-server-expert** to build voice → LLM pipelines
- For TTS the other direction, see Piper / Coqui / XTTS (out of scope here)

## Sources

- [whisper.cpp GitHub](https://github.com/ggml-org/whisper.cpp)
- [whisper.cpp models README](https://github.com/ggml-org/whisper.cpp/blob/master/models/README.md)
- [whisper-cli (DeepWiki)](https://deepwiki.com/ggml-org/whisper.cpp/3.1-command-line-interface)
- [whisper-server (DeepWiki)](https://deepwiki.com/ggml-org/whisper.cpp/3.2-http-server)
- [whisper-stream (DeepWiki)](https://deepwiki.com/ggml-org/whisper.cpp/3.3-talk-llama)
- [whisper.cpp releases](https://github.com/ggml-org/whisper.cpp/releases)