Video Frames

---
name: Video Frames
description: Extract frames or the audio track from a local video file via ffmpeg. Pairs with openai-whisper for audio→text.
requires:
  bins:
    - ffmpeg
    - ffprobe
  env: []
---

# Video Frames

Use this skill whenever a local video file must become text (via audio +
Whisper) or selected JPG frames. The extension wraps `ffmpeg`/`ffprobe`
subprocesses with a path sandbox and per-call timeout.

## Use when

- "Transcribe this video"
- "Grab 12 frames from this clip"
- "What does this video say?" — chain through audio + whisper + summarize
- Prepare material from `bible_videos/` for agent kate to study

## Do not use when

- The file is not locally accessible (fetch first with `fetch_url` if
  allowed; operator decides the save path inside the sandbox)
- The clip is > 500 MB — split or downscale upstream
- You need real-time / streaming — this is batch only
- You need OCR on frames — that's a separate pipeline

## Tools

### `status`
No arguments. Returns ffmpeg + ffprobe versions, sandbox root, input and
frame limits.

### `probe`
- `path` (string, required) — local video file

Returns `{duration_secs, format, streams}` as JSON.

### `extract_frames`
- `path` (string, required)
- `output_dir` (string, required) — must lie under the sandbox root
- `count` (integer, optional, 1–1000, default 10) — evenly spaced over the whole clip
- `fps` (number, optional, 0.01–60) — overrides `count` with a fixed sample rate
- `width` (integer, optional, 16–4096) — resize keeping aspect ratio

Returns `{count_written, frames: [path, ...]}`.

### `extract_audio`
- `path` (string, required)
- `output_path` (string, required) — full path under the sandbox
- `codec` (string, optional, `mp3` default | `wav`)
- `mono` (boolean, optional, default true) — recommended for Whisper
- `sample_rate` (integer, optional, 8000–48000, default 16000) — matches Whisper

Returns `{output_path, bytes, codec, mono, sample_rate}`.

## Execution guidance

- For audio destined for `openai-whisper`, prefer **WAV mono 16 kHz** — this
  is the Whisper-native layout and avoids re-encoding downstream.
- Use `probe` first when the user's ask depends on duration (e.g.
  "every 30 seconds").
- Keep `count` modest (≤ 24 for most overview tasks). Thousand-frame dumps
  are rarely useful to an LLM.
- Error codes:
  - `-32030` ffmpeg missing → alert the operator
  - `-32032` ffmpeg failed → include the stderr preview in the reply so
    the user knows why (e.g. "no audio stream" for silent clips)
  - `-32033` timeout → suggest a shorter clip or a higher
    `VIDEO_FRAMES_TIMEOUT_SECS`
  - `-32034` io/sandbox → path was outside `VIDEO_FRAMES_OUTPUT_ROOT`;
    ask operator to use a sandboxed directory
- Wrap `extract_audio` → `transcribe_file` → `summarize_text` in a
  TaskFlow so a restart does not re-extract audio you already have.