Transcribe

---
name: transcribe
description: Transcribe audio files or YouTube videos using WhisperX (distil-large-v3.5) with optional speaker diarisation
---

# Transcribe — Audio to Text

Transcribe audio files using WhisperX with distil-large-v3.5, locally. Optional speaker diarisation via pyannote.

## When to use

When the user wants to transcribe an audio recording (voice memo, meeting, interview, lecture) or a YouTube video. Use when the user asks for a transcript, not merely because a message contains a media path or YouTube URL. For a digest request, use `$podcast-digest`.

## Codex execution

Read supporting files relative to this skill directory, not the shell working directory. For vault work resolve `VAULT_PATH` with `"${VAULT_PATH:?VAULT_PATH not set}/.claude/scripts/resolve-vault.sh"`; stop if resolution fails. All vault writes, including headers and verbatim bodies, use `locked-edit.sh`.

Use tracked Codex exec sessions for long local runs; if execution returns a session ID, poll that session to completion and inspect its exit code. Do not detach local transcription with shell `&`. Keep stderr in a scratch file for diagnostics and provide progress while waiting. Shell variables do not persist between exec calls: carry literal scratch paths forward or reload a saved job config.

Honour output, speaker names and save/discuss choices already supplied by the user. Ask only for missing decisions; optional speaker renaming need not block an explicitly requested save with generic labels. Choosing paid cloud transcription requires the user's authorisation; use the installed `../transcribecloud/SKILL.md` after reading it.

## Prerequisites

- Python venv with `whisperx` installed at `~/venvs/whisperx/`
- `ffmpeg` installed (for audio decoding; `ffprobe`, bundled with it, is used for duration probing)
- For YouTube URLs: `yt-dlp` installed
- For diarisation: HuggingFace token (via `HF_TOKEN` env var or `huggingface-cli login`). The user must have accepted the pyannote model licences at huggingface.co/pyannote/speaker-diarization-3.1, huggingface.co/pyannote/speaker-diarization-community-1, and huggingface.co/pyannote/segmentation-3.0. (The Phase 2 script pins `speaker-diarization-3.1`, which pulls `segmentation-3.0`; `community-1` is WhisperX ≥3.8's default and covers the fallback if the pin is ever removed.)

If any prerequisite is missing, tell the user what to install and stop. **Exception — venv exists but the import fails:** check for Python-version drift before concluding whisperx is missing. Compare `rg version ~/venvs/whisperx/pyvenv.cfg` against `~/venvs/whisperx/bin/python3 --version`: a mismatch means an OS Python upgrade broke the venv (packages sit stranded under the old `lib/pythonX.Y/site-packages`) — the fix is to recreate the venv and reinstall whisperx (compiled wheels are Python-version-specific), not to install anything new.

## Arguments

The free-text arguments following the skill invocation — path to the audio file or a YouTube URL, optionally followed by flags. If no path is provided, ask the user.

Flags (parsed from arguments):
- `--diarize` or `--diarise` — enable speaker diarisation
- `--speakers N` — exact number of speakers (implicitly enables diarisation)
- `--min-speakers N` / `--max-speakers N` — speaker count range (implicitly enables diarisation)
- `--raw` — skip LLM cleanup pass (save unprocessed WhisperX output)
- `--no-timestamps` — omit timestamps from output
- `--no-published` — skip the published-transcript check (Phase 0); go straight to WhisperX

## Workflow

### Phase 0: Check for a published transcript

A published transcript (podcast show site, Substack, official transcript page) is usually higher text-fidelity than any machine transcription, and free. Check for one *before* committing to a download + WhisperX run. "Published" does not entail "human-edited" — many are the publisher's own ASR output, which changes what you may claim in the header and how you resolve names and speakers (§15 "Names and speakers"), though it stays worth preferring over spending a WhisperX run. Skip this phase entirely if `--no-published` was passed, or if the source is a local audio file with no obvious online origin, or if the user explicitly said to transcribe the audio, or if diarisation was requested via flags (a published transcript cannot satisfy a diarisation request — a user-supplied transcript URL still wins per step 1).

1. **User-supplied transcript wins.** If the user provided a transcript URL or pointed at one ("the transcript is at X"), use it directly — go to step 4.
2. **Discover a candidate** (best-effort, for YouTube / named-podcast sources only):
   - YouTube, *if `yt-dlp` is installed* (it isn't verified until Phase 1 — check `command -v yt-dlp` first): scan the description for a transcript link — `yt-dlp --print "%(description)s" "URL"` — for links whose text or target contains "transcript" or points to the show's own site. (Description only — fetching pinned/top comments needs `--write-comments` and isn't worth it; treat comments as out of scope.)
   - Run **one** web search (if a web-search tool is available): `"<video/episode title>" transcript`. Prefer the official show/publisher page; treat third-party aggregator sites (usually auto-generated, not human-edited) as no better than WhisperX.
   - If neither `yt-dlp` nor a web-search/fetch tool is available, skip discovery and fall through to Phase 1 — rely only on a user-supplied URL (step 1).
3. If no credible candidate is found, say so in one line and fall through to Phase 1. Don't over-search — this is a quick best-effort check, not a research task.
4. **Fetch and validate the candidate.** Read §15 of `../_shared-rules-content.md` (relative to this SKILL.md) first — this skill doesn't load that file wholesale, so the extractor isn't in context until you read it. Extract the body to a file with the **§15 extractor**, which needs `curl`/`pandoc`/`python3`+`bs4`/`lxml` (its prereq check is the gate — if those are missing, skip the published path and fall through to Phase 1's WhisperX run after telling the user). The body lands at `<BODY_FILE>` — §15's printed `BODY=` path, a **deterministic function of the URL**, so you can reuse that exact path in step 6 (or re-derive it from the URL); there is no random temp name to carry across tool calls. Judge it on structure with **shell probes, not by reading the whole body into context** — `wc -w "<BODY_FILE>"`, `rg -n` for speaker labels / transcript markers, and `head`/`tail` for a bounded look. Accept as a genuine transcript if it has sustained episode-specific prose AND either (a) speaker turn-taking, (b) continuous single-speaker prose (lectures/monologues have no turns), or (c) the page explicitly labels itself a transcript. Reject show notes, a summary, an excerpt, or a paywalled stub. As a sanity floor, *if you can cheaply get duration* (`yt-dlp --print "%(duration)s" "URL"` for YouTube), expect ≳80–100 words per minute of audio — but don't reject a labelled transcript just for running light, and skip the floor when duration is unknown. If it's a summary or partial, discard it and fall through to Phase 1.
5. **Present the choice** (numbered list) and wait — don't auto-pick:
   1. **Use the published transcript** — higher text fidelity; but typically **no timestamps**, and speaker turns only where the source marks them (no diarisation). May be lightly cleaned of filler. **Don't assume human editing** — check for the auto-generated disclaimer (§15 "Names and speakers") and say which kind it is when presenting the choice: an auto-generated one carries WhisperX-grade name-garbling and speaker-mislabelling.
   2. **Run WhisperX** — machine transcription with timestamps and optional diarisation, at lower text fidelity.
6. **If the user picks the published transcript:** the body is already at `<BODY_FILE>` from step 4 (§15's printed `BODY=` path, reconstructable from the URL). Skip Phases 1–2 and the WhisperX-specific parts of Phase 3 (no download, no WhisperX, no diarisation, and **no** LLM cleanup pass — the body is appended verbatim per §14, whoever produced it). Use the agreed destination, or ask where to save if missing. Stage only the metadata header; assemble it with the existing `<BODY_FILE>` directly from disk using Phase 3 step 7's locked save procedure. The published body must not pass through model context or LLM cleanup. Adjust the header:
   - **Source:** the transcript URL; put the original audio / YouTube URL on a second `**Original media:**` line.
   - **Date acquired:** use this run’s acquisition date instead of **Date transcribed**.
   - **Model:** default `published transcript (provenance unverified)`; upgrade to `(source-provided, auto-generated)` on a disclaimer match, or to `(human-edited)` only where the page positively claims editing (§15's three-value ladder — the unverified value is the default precisely so a null cannot select the stronger claim). Either way the body is appended verbatim (`_shared-rules-content.md` §14); never clean it.
   - **Names and speaker labels** in the header, filename or report follow §15 "Names and speakers" — already in context from step 4.
   - Drop the **Diarisation** and **Cleanup** lines (or mark `n/a`); keep **Duration** only if known.

   Otherwise (user picks WhisperX, or no transcript was found/accepted), continue to Phase 1.

### Phase 1: Validate

1. **Determine source type.** If the argument is a YouTube URL (contains `youtube.com/watch` or `youtu.be/`), this is a YouTube source. Otherwise, confirm the audio file exists at the given path.
2. Check prerequisites are installed:
   ```bash
   ~/venvs/whisperx/bin/python3 -c "import whisperx" && command -v ffmpeg && command -v ffprobe
   ```
   If YouTube source, also check: `command -v yt-dlp`
3. If checks fail, report what's missing and stop.
4. **If YouTube source** (steps a-c are independent — run in parallel):
   a. Get the video title: `yt-dlp --print title "URL"`
   b. Download audio as WAV: `yt-dlp -f "bestaudio[abr<=64]/worstaudio" -x --audio-format wav -o "/tmp/yt_transcribe_%(id)s.%(ext)s" "URL"`
      - Whisper resamples to 16kHz mono internally, so a ≤64kbps source (typically YouTube format 249 @ 49kbps Opus) is plenty for transcription. Unconstrained `bestaudio` pulls format 401 (~100-250MB for typical videos, GB-scale for long ones) and is painfully slow on a slow link.
   c. Download auto-generated English captions (if available) for cross-referencing during cleanup:
      ```bash
      yt-dlp --write-auto-sub --sub-lang en --sub-format srt --skip-download -o "/tmp/yt_subs_%(id)s" "URL"
      ```
      The file lands at `/tmp/yt_subs_<id>.<lang>.srt` — glob `/tmp/yt_subs_<id>*` to locate the exact name for the Phase 3 cross-reference. If no English auto-captions are available (e.g. non-English video), retry with `--sub-lang` matching the video's language if known, or omit `--sub-lang` to get whatever is available. If still nothing, proceed without — this is a best-effort enhancement.
   d. Set `INPUT_FILE_PATH` to the downloaded WAV path.
   e. Store the YouTube URL and video title for use in the output metadata.
5. Get the audio duration:
   ```bash
   ffprobe -v quiet -show_entries format=duration -of csv=p=0 "INPUT_FILE_PATH"
   ```
6. Check for GPU:
   ```bash
   ~/venvs/whisperx/bin/python3 -c "import torch; print(torch.cuda.is_available())"
   ```
7. Determine if diarisation was requested (check for `--diarize`, `--diarise`, `--speakers`, `--min-speakers`, or `--max-speakers` in arguments — any of these enables diarisation).
8. If diarisation requested, verify HF token is available:
   ```bash
   ~/venvs/whisperx/bin/python3 -c "from huggingface_hub import HfApi; HfApi().whoami()" 2>/dev/null && echo "ok" || echo "no_token"
   ```
   If no token, tell the user to run `huggingface-cli login` or set `HF_TOKEN` and stop. Note `whoami()` only proves the token is valid — it does not prove the pyannote licence gates were accepted; a 403 later, inside diarisation, means a licence page from the Prerequisites list still needs accepting.
9. **Background decision:** If no GPU and (duration > 120s OR diarisation is enabled), tell the user the estimated wait and run as a **tracked Codex exec session**. Diarisation on CPU is slow (~2-3x audio length on top of transcription time).
   - **Cloud escape hatch:** when there's no GPU and the job is heavy — audio > ~30 min, or a batch of files — surface `$transcribecloud` (RunPod GPU; same WhisperX) as a faster **paid** alternative *before* committing to a long local grind, and let the user choose. It costs money and needs `runpodctl` + credits, so don't auto-switch: if those are absent, say so and fall back to the local exec session. (This is the reciprocal of `$transcribecloud`'s own "use instead of `$transcribe` when no local GPU and audio > 30 min" trigger.)

### Phase 2: Transcribe

Run via a tracked Codex exec session, polling until completion. **Redirect stderr to a temp file** (not /dev/null). On success, discard it. On failure, read it for diagnostics.

Set these variables before substituting into the script:
- `INPUT_FILE_PATH` — the audio file path
- `DIARIZE` — `True` or `False`
- `NUM_SPEAKERS` — integer or `None`
- `MIN_SPEAKERS` — integer or `None`
- `MAX_SPEAKERS` — integer or `None`

```bash
STDERR_TMP=$(mktemp)
~/venvs/whisperx/bin/python3 << 'PYEOF' 2>"$STDERR_TMP"
import os, json
os.environ["TQDM_DISABLE"] = "1"

import torch
import whisperx

device = "cuda" if torch.cuda.is_available() else "cpu"
compute_type = "float16" if device == "cuda" else "int8"
batch_size = 16 if device == "cuda" else 4
audio_file = os.environ["TRANSCRIBE_INPUT"]
diarize = DIARIZE
num_speakers = NUM_SPEAKERS
min_speakers = MIN_SPEAKERS
max_speakers = MAX_SPEAKERS

# 1. Transcribe
model = whisperx.load_model("distil-large-v3.5", device, compute_type=compute_type)
audio = whisperx.load_audio(audio_file)
result = model.transcribe(audio, batch_size=batch_size)
language = result["language"]

# distil-large-v3.5 is English-only — its output on other languages is unreliable.
if language != "en":
    raise SystemExit(
        f"Detected language '{language}', but distil-large-v3.5 is English-only. "
        "Re-run with a multilingual model (large-v3) or use $transcribecloud."
    )

# 2. Align (word-level timestamps)
model_a, metadata = whisperx.load_align_model(language_code=language, device=device)
result = whisperx.align(result["segments"], model_a, metadata, audio, device)

# 3. Diarize (optional)
if diarize:
    from whisperx.diarize import DiarizationPipeline
    try:
        diarize_model = DiarizationPipeline(
            model_name="pyannote/speaker-diarization-3.1",
            device=device,
            token=os.environ.get("HF_TOKEN"),  # whisperx 3.8.x param is `token` (NOT use_auth_token, which is the older pod-pinned API in $transcribecloud); None → falls back to the cached huggingface-cli login
        )
    except Exception as e:
        # Pipeline.from_pretrained returns None on auth/licence failure, so the
        # constructor blows up inside itself (typically AttributeError on .to()).
        raise RuntimeError(
            "Diarisation pipeline failed to load — token rejected or licence gates not accepted "
            "(see Prerequisites)"
        ) from e
    diarize_segments = diarize_model(
        audio,
        num_speakers=num_speakers,
        min_speakers=min_speakers,
        max_speakers=max_speakers,
    )
    result = whisperx.assign_word_speakers(diarize_segments, result)

out_path = "/tmp/transcribe_segments_" + os.path.splitext(os.path.basename(audio_file))[0] + ".json"
with open(out_path, "w") as f:
    json.dump({"segments": result["segments"], "language": language}, f)
print(out_path)
PYEOF
EXIT_CODE=$?
if [ $EXIT_CODE -ne 0 ]; then cat "$STDERR_TMP"; fi
rm -f "$STDERR_TMP"
exit $EXIT_CODE
```

Replace the boolean/integer placeholders with validated values. Export `TRANSCRIBE_INPUT` in the same exec call using proper shell quoting of the supplied path; never interpolate a path into Python source. Give concurrent runs distinct scratch/output paths instead of reusing a basename-derived JSON from an earlier run.

### Phase 3: Post-process and output

1. **Parse the segments JSON** from the file whose path Phase 2 printed (word-level JSON for a long recording is far too large for tool stdout — read it from the file, in chunks if needed).

2. **Format the transcript:**

   Determine the timestamp format from the audio duration: `[MM:SS]` for recordings under 1 hour, `[H:MM:SS]` for recordings 1 hour or longer. If `--no-timestamps` was passed, omit timestamps entirely.

   **Without diarisation:** Insert a paragraph break wherever the gap between one segment's end and the next segment's start exceeds 1.5 seconds. Concatenate segment text within each paragraph.

   **Monologue fallback.** For recordings longer than 5 minutes, after the initial split, check: is the paragraph count at least `duration_seconds / 120` (roughly one per two minutes)? If not, the source is a dense monologue without natural 1.5s pauses — recompute:

   1. Sort all segment-to-segment gaps in descending order.
   2. Let `N = min(floor(duration_seconds / 90), len(gaps_desc))` — target paragraph-break count (one break per ~90s), clamped to the number of available gaps.
   3. Use `gaps_desc[N-1]` as the new threshold and re-split. If there are no gaps at all (`len(gaps_desc) == 0`, e.g. a single segment), skip the fallback and keep the initial split.

   In your user-facing response, state the threshold actually used — either `(1.5s default)` or `(0.Xs — monologue fallback)` — so the choice is observable.

   Prefix each paragraph with a timestamp derived from the first segment's `start` field:
   ```
   [00:00] First paragraph of speech here.

   [01:23] Next paragraph after a gap.
   ```

   **With diarisation:** Iterate over the **words** array (not segments), since a single segment often contains multiple speakers. Before grouping, forward/back-fill any words with a missing or `None` speaker from their nearest non-None neighbour (pyannote occasionally leaves words unassigned; treating `None` as its own speaker produces a spurious extra speaker). Then group consecutive words by speaker. Start a new paragraph when the speaker label changes OR when the gap between consecutive words exceeds 1.5 seconds. Prefix each paragraph with a timestamp and speaker label in bold:
   ```
   [00:00] **Speaker 1:** First speaker's text here spanning one or more words.

   [00:45] **Speaker 2:** Second speaker responds with their text.

   [01:12] **Speaker 1:** First speaker continues after the other.
   ```
   The timestamp is derived from the `start` field of the first word in each paragraph group.

   Map raw labels (`SPEAKER_00`, `SPEAKER_01`, ...) to `Speaker 1`, `Speaker 2`, etc. in order of first appearance. The words array is found at `segments[].words[]`. Only `word` is guaranteed — `start`, `end`, `score` and `speaker` are all optional: alignment omits timing for tokens it can't align (numerals, symbols, out-of-vocabulary fragments), and speaker assignment skips any word without a `start`. Never index these keys directly. A word missing `start`/`end` keeps its text but contributes no timestamp and no gap: attach it to the paragraph in progress and take timing from the nearest neighbour that has it.

3. **LLM cleanup pass** (skip if `--raw` was passed):

   Review the assembled transcript text and fix transcription errors. Rules:
   - Fix non-words to their most likely intended word (e.g. "seduptresses" → "seductresses")
   - Fix obvious grammar/punctuation errors introduced by the transcription model (not the speaker's actual grammar — ESL patterns, filler words, etc. stay)
   - Mark genuinely unclear sections as `[inaudible]` rather than guessing
   - Preserve the speaker's actual words and meaning — don't rewrite, paraphrase, or "improve"
   - Preserve timestamps exactly
   - Preserve speaker labels exactly
   - Do NOT fix the speaker's actual speech patterns (um, uh, repeated words, broken sentences) — these are features, not bugs

   **YouTube caption cross-reference** (if YouTube auto-captions were downloaded in Phase 1 step 4c):
   Before starting the cleanup pass, extract a focused comparison from the SRT captions. Parse the SRT to plain text, then identify words/phrases where the WhisperX and YouTube outputs diverge — especially proper nouns (names, institutions, products, technical terms). Build a short divergence list (e.g. "WhisperX: 'Mercore' / YouTube: 'Mercor'") and use it as a reference during cleanup. Do not load the entire SRT into context — the useful signal is in the divergences, not the full text.

   Neither source is authoritative — both are machine-generated and make different errors. Where one system produces a recognisable proper noun and the other produces a garbled version, prefer the recognisable one. Where both agree on a spelling that looks wrong (e.g. both say "Tyler Cowan" when context suggests "Tyler Cowen"), apply your own judgement. Where they disagree and neither is clearly right, flag with `[?]` or choose the more plausible option based on context.

   **A third source outranks both, and it is usually already to hand:** human-written page text about the episode — the video description, the show notes, a chapter list, a linked resource list. Those are typed by a person; both machine transcripts are not. So for any name in the divergence list, grep that text for a distinctive fragment (match a stem — the garbling is phonetic) and **take the spelling it gives over either transcript's**, rather than adjudicating between two guesses or flagging `[?]`.

   Process the full transcript in a single pass. If the transcript exceeds ~50KB of text, split into ~20KB chunks with 2-paragraph overlap to preserve context at boundaries.

4. **Deliver** the formatted transcript as requested. For a long transcript requested as a file, save it and link the file rather than duplicating the full body in chat.

5. **If diarisation was used**, ask the user if they want to rename any speakers (e.g. "Speaker 1 is actually Dr Smith"). Apply renames before saving.

6. **Ask what the user wants to do next.** Present the options:
   1. Save the transcript — ask where (suggest a default: for local audio files, same directory with `.md` extension; for YouTube, suggest a logical location if one is apparent from context, otherwise ask). Also ask if the user wants a specific filename or is happy with the default (video title / audio filename).
   2. Discuss the transcript — e.g. summarise it, extract key points, ask questions about the content, pull out action items.
   3. Both — save first, then discuss.

   If the request already specifies save/discuss and destination, proceed with that choice. Otherwise wait for the missing decision.

7. **Save** the transcript as a markdown file (if the user chose to save). Use the path and filename agreed in step 6.

Stage the metadata header and transcript body in a unique scratch directory outside the vault. Preserve the body verbatim; do not run a spelling normaliser or rewrite the completed note through an editor. Read `../_shared-rules.md` §5 and `../_shared-rules-content.md` §14 before saving.

For a new vault note, assemble header + blank line + body into a staging file, then write that file in one locked append:

```bash
"$VAULT_PATH/.claude/scripts/locked-edit.sh" "<DESTINATION>" --append < "<STAGED_NOTE>"
```

Check for an existing destination before writing. Do not append a duplicate transcript or overwrite existing content; resolve a filename collision with the user unless the request already specifies replacement. For an authorised replacement use the locked wrapper's literal `--replace` mode, re-reading on exit 2 or 3. Outside the vault, write the staged bytes directly to the agreed destination. Check the write exit code and compare the saved body with the staged body byte-for-byte. Report success only after that comparison passes. The header's transcription date comes from `date +%F` for this run; a published transcript's acquisition date is not its original transcription date.

   ```markdown
   # Transcript: {video title or filename}

   **Source:** `{YouTube URL or filename/path to audio file}`
   **Date transcribed:** {YYYY-MM-DD}
   **Duration:** {MM:SS or H:MM:SS}
   **Model:** whisperx / distil-large-v3.5
   **Diarisation:** {yes (N speakers) | no}
   **Cleanup:** {yes | no (--raw)}

   ---
   ```

8. **If the user chose to discuss**, proceed with whatever they asked for (summary, Q&A, extraction, etc.).

## Notes

- WhisperX uses faster-whisper (CTranslate2 backend) which is faster than HF Transformers for inference.
- distil-large-v3.5 is **English-only**. For a known non-English source, choose multilingual `large-v3` before inference; do not rely on the English-only model to detect unsupported speech. Phase 2 also aborts if it returns a non-English language code. For other languages, run the multilingual `large-v3` or `$transcribecloud`. This is why the auto-caption step is best-effort rather than a non-English pathway.
- GPU (CUDA) is used automatically if available; falls back to CPU with int8 quantisation.
- Models are cached after first download.
- On GPU: transcription is near-instant; diarisation is also fast — an earlier ~20–30 min/hr claim was empirically disproven (a 46-min recording diarised in ~20s on a cloud RTX 4090; see `../transcribecloud/references/voice-references.md` for the historical source test).
- Diarisation accuracy is best with 2-3 speakers in clear audio. Specify `--speakers N` when you know the count.

## Skill Monitor

As you execute this skill, follow `../_skill-monitor.md` (relative to this SKILL.md): watch for gaps, and log observations at the end per that file.