Video Frame Extraction Analysis

---
license: Apache-2.0
name: video-frame-extraction-analysis
description: "Extract keyframes, detect scenes, and build CLIP-indexed temporal search over video content. Activate on: keyframe extraction, scene detection, video search, video indexing, temporal analysis. NOT for: video editing/rendering (video-processing-editing), video generation (ai-video-production-master)."
allowed-tools: Read,Write,Edit,Bash(python:*,pip:*,npm:*,npx:*)
category: AI & Machine Learning
tags:
  - video-analysis
  - keyframe-extraction
  - scene-detection
  - clip-indexing
  - temporal-search
pairs-with:
  - skill: video-processing-editing
    reason: Video editor handles rendering; this skill handles analysis and indexing
  - skill: multimodal-embedding-generator
    reason: CLIP/SigLIP embeddings power visual search over extracted frames
  - skill: audio-transcription-pipeline
    reason: Transcription aligns with visual frames for multimodal video search
---

# Video Frame Extraction & Analysis

Extract keyframes, detect scene boundaries, and build CLIP-indexed temporal search systems for video content analysis and retrieval.

## Activation Triggers

**Activate on**: "keyframe extraction", "scene detection", "video search", "video indexing", "find frame in video", "temporal search", "video content analysis", "scene boundary detection", "CLIP video search"

**NOT for**: Video editing, trimming, or rendering (video-processing-editing), video generation from text/images (ai-video-production-master), or face recognition in video (face-recognition-system-builder)

## Quick Start

1. **Decode video** — Use ffmpeg or decord for efficient frame extraction. Never decode all frames; sample intelligently.
2. **Detect scenes** — PySceneDetect for cut detection, or embedding-based scene boundary detection for gradual transitions.
3. **Extract keyframes** — One representative frame per scene, plus additional frames at fixed intervals within long scenes.
4. **Embed frames** — Run CLIP/SigLIP on keyframes to generate searchable visual embeddings.
5. **Index and search** — Store embeddings with timestamps in a vector DB. Query with text ("a person opening a door") or image.

## Core Capabilities

| Domain | Technologies | Notes |
|--------|-------------|-------|
| **Frame Extraction** | ffmpeg, decord, OpenCV, PyAV | decord is fastest for random access |
| **Scene Detection** | PySceneDetect, TransNetV2 | Cut detection + gradual transition detection |
| **Visual Embedding** | CLIP, SigLIP, InternVideo2 | Per-frame or pooled-scene embeddings |
| **Temporal Search** | Vector DB + timestamp metadata | "Find the frame where X happens" |
| **Shot Analysis** | Shot type classification, motion estimation | Wide/medium/close-up, camera movement |
| **OCR on Frames** | PaddleOCR, EasyOCR, Tesseract | Extract text from slides, titles, signage |

## Architecture Patterns

### Pattern 1: Scene-Based Keyframe Pipeline

```
Video File ──→ [Scene Detector] ──→ [Keyframe Selector] ──→ [CLIP Embed] ──→ [Vector DB]
    │                │                      │                      │              │
  input         PySceneDetect          1 frame per scene      SigLIP-large    store with
  mp4/mkv       threshold=27           at scene midpoint      384-dim         timestamp
                detect cuts            + every 5 sec in                       + scene_id
                + transitions          long scenes (>30s)                     + metadata
```

```python
# Scene detection + keyframe extraction
from scenedetect import detect, ContentDetector
import decord
import numpy as np

def extract_keyframes(video_path: str) -> list[dict]:
    """Extract one keyframe per scene with timestamps."""
    # Step 1: Detect scene boundaries
    scene_list = detect(video_path, ContentDetector(threshold=27))

    # Step 2: Extract keyframe at midpoint of each scene
    vr = decord.VideoReader(video_path)
    fps = vr.get_avg_fps()
    keyframes = []

    for i, scene in enumerate(scene_list):
        start_frame = scene[0].get_frames()
        end_frame = scene[1].get_frames()
        mid_frame = (start_frame + end_frame) // 2

        frame = vr[mid_frame].asnumpy()  # RGB numpy array
        timestamp = mid_frame / fps

        keyframes.append({
            "frame": frame,
            "frame_index": mid_frame,
            "timestamp_sec": timestamp,
            "scene_index": i,
            "scene_duration": (end_frame - start_frame) / fps,
        })

        # For long scenes (>30s), add extra keyframes every 5 sec
        scene_dur = (end_frame - start_frame) / fps
        if scene_dur > 30:
            for t in np.arange(start_frame + int(5*fps), end_frame, int(5*fps)):
                extra = vr[int(t)].asnumpy()
                keyframes.append({
                    "frame": extra,
                    "frame_index": int(t),
                    "timestamp_sec": int(t) / fps,
                    "scene_index": i,
                    "scene_duration": scene_dur,
                })

    return keyframes
```

### Pattern 2: CLIP-Indexed Video Search

```
Indexing (offline):
  Video ──→ [Extract Keyframes] ──→ [SigLIP Embed] ──→ [Vector DB]
                                                            │
                                                     metadata per frame:
                                                     video_id, timestamp,
                                                     scene_id, thumbnail_path

Querying (online):
  Text: "person walking through rain" ──→ [SigLIP Text Embed] ──→ [Vector Search]
                                                                        │
                                                                  top-k frames
                                                                  with timestamps
                                                                        │
                                                                  "video_3.mp4 @ 01:23:45"
                                                                  "video_7.mp4 @ 00:45:12"
```

### Pattern 3: Multimodal Video Index

```
Video ──┬──→ [Keyframes] ──→ [CLIP Embed] ──→ [Visual Index]──┐
        │                                                       │
        ├──→ [Audio Track] ──→ [Whisper] ──→ [Text Embed] ──→ [Text Index]──┤──→ [Fusion Search]
        │                                                       │
        └──→ [Frame OCR] ──→ [Text Extract] ──→ [Text Embed]──┘

Fusion search: query hits all three indexes, reciprocal rank fusion combines results
  "Explain the sales chart" →
    Visual: frame with chart → timestamp 15:30
    Audio: "our Q3 numbers show..." → timestamp 15:28
    OCR: "Q3 Revenue: $4.2M" → timestamp 15:30
    Fused result: 15:28-15:35 segment with high confidence
```

## Anti-Patterns

1. **Extracting every frame** — A 1-hour video at 30fps is 108,000 frames. Embedding all of them is wasteful and slow. Use scene detection to select keyframes.
2. **Fixed-interval sampling without scene awareness** — Sampling every 5 seconds misses fast cuts and over-samples static scenes. Scene-based extraction is always better.
3. **Ignoring temporal context** — A single frame without a timestamp is useless for video search. Always store timestamps and scene boundaries as metadata.
4. **Using OpenCV VideoCapture for random access** — OpenCV decodes sequentially; seeking to frame 50,000 reads all prior frames. Use decord for O(1) random access.
5. **No thumbnail generation** — Search results need visual previews. Generate thumbnails alongside embeddings, stored as small JPEGs referenced by path in metadata.

## Quality Checklist

- [ ] Scene detection threshold tuned for content type (27-35 for cuts, lower for gradual)
- [ ] Keyframes extracted at scene midpoints (not boundaries, which may be transitional)
- [ ] Long scenes (>30s) have additional keyframes at regular intervals
- [ ] CLIP/SigLIP embeddings normalized before storage
- [ ] Timestamps stored as metadata on every vector (video_id + seconds)
- [ ] Thumbnails generated and stored alongside embeddings
- [ ] Text search via transcription aligned with visual frames
- [ ] Random access via decord or PyAV (not sequential OpenCV reads)
- [ ] Search results return video_id + timestamp + thumbnail + confidence score
- [ ] Pipeline handles multiple video formats (MP4, MKV, WebM, MOV)