Carocut Media Audio

---
name: carocut-media-audio
description: 音频素材生成与获取。批量 Edge TTS 旁白生成（支持 storyboard pacing 字段驱动语速）、BGM/SFX 检索（BGM 节奏匹配 BPM 规则）、音频时长提取。包含 Edge voice 配置、速度调整规则、durations.json 格式规范（含 audio_visual_relation 说明）和关键的音频时序规则。
---

# Audio Assets

Generates voiceovers via TTS, retrieves background music and sound effects. Creates `durations.json` for timing synchronization in Remotion.

## Audio Types

| Type | Source | Output Directory |
|------|--------|------------------|
| Voiceover (VO) | Edge TTS | `{project_dir}/raws/audio/vo/` |
| Background Music (BGM) | Freesound | `{project_dir}/raws/audio/bgm/` |
| Sound Effects (SFX) | Freesound | `{project_dir}/raws/audio/sfx/` |

---

## Voiceover Generation

Generate speech audio from script using Edge TTS.

### TTS Batch Processing Workflow

Processes entire script.md and generates all voiceover files:
1. Parse `{project_dir}/manifests/script.md` for all `[VO_XXX]` markers and their text content
2. For each VO segment, invoke Edge TTS with the configured voice and speed
3. Save each result as `VO_XXX.wav` in `{project_dir}/raws/audio/vo/`
4. After all segments complete, extract durations to create `{project_dir}/manifests/durations.json`

### Voice Selection

完整 voice 列表和选择建议见 `references/edge-tts.md`。

| Input | Meaning | Recommendation |
|------|---------|----------------|
| `default` | Default alias | Maps to `zh-CN-XiaoxiaoNeural` |
| any valid edge-tts voice ID | Direct pass-through | 根据项目风格从 `references/edge-tts.md` 中选择 |

### Speed Adjustment Rules

- The default TTS voice may sound slow. Consider using speed `1.2` for more natural pacing.
- Speed range: `0.5` (half speed) to `2.0` (double speed)
- Recommended: `1.0` - `1.3` for narration, `0.8` - `1.0` for emotional/dramatic content

### TTS Speed and Storyboard Pacing (语速与节奏对应)

When generating TTS, read the corresponding shot's `pacing` field from the storyboard to determine speed:

| Pacing | TTS Speed | Description |
|--------|-----------|-------------|
| `slow` | 1.0 | 放慢语速，营造沉浸感和情感重量 |
| `medium` | 1.2 | 默认语速，自然叙述节奏（不变） |
| `fast` | 1.3 | 加快语速，营造紧迫感和兴奋感 |
| `pause` | N/A | 呼吸段（breathing shot），无旁白，不生成 TTS |

**Rules:**
- If the storyboard does not specify pacing for a shot, default to `medium` (speed 1.2)
- For `pause` pacing shots, skip TTS generation entirely — these are breathing shots with no voiceover
- The pacing value is per-shot; different VO segments in the same video may have different speeds
- After applying pacing-based speed, duration extraction must reflect the actual playback length at that speed

---

## Duration Extraction

Extract audio durations for timing synchronization. **This step is mandatory.**

### Output Format (durations.json)

```json
{
  "VO_001": 3450,
  "VO_002": 2800,
  "VO_003": 4200,
  "VO_004": 3100
}
```

All durations are in **milliseconds**. This file is critical for shot timing in Remotion.

### Duration Rules

- All values MUST be in milliseconds (integer)
- Every `[VO_XXX]` marker in script.md must have a corresponding entry
- Duration reflects the actual audio length at the configured speed (e.g., 1.2x)
- **Always re-extract durations after any audio modification**

### durations.json and Audio-Visual Relation

The `durations.json` format itself remains unchanged (simple `VO_XXX: milliseconds` mapping). However, the Builder's `timing.ts` module uses the storyboard's `audio_visual_relation` field in combination with durations to calculate actual audio start frames:

| `audio_visual_relation` | Builder Behavior |
|------------------------|------------------|
| `sync` | Audio starts at the same frame as the visual shot |
| `lead-visual` | Visual appears slightly before audio (audio offset positive) |
| `lead-audio` | Audio starts slightly before the visual cut (audio offset negative) |
| `counterpoint` | Audio timing is sync, but visual content is intentionally mismatched |

The `audioOffsetFrames` function in `timing.ts` reads these values to apply frame-level offsets. The Audio agent does **not** need to modify durations.json for this — it is handled downstream by the Builder.

> **重要依赖说明**：虽然 durations.json 格式简单（VO_XXX: 毫秒数），但 Builder 在计算实际音频起始帧时，会同时读取 storyboard.yaml 中对应 shot 的 `audio_visual_relation` 字段。Audio agent 只需确保 durations.json 中的时长准确；偏移量由 Builder 的 `timing.ts` → `audioOffsetFrames()` 函数自动计算。

---

## BGM and SFX Retrieval

Download royalty-free audio from Freesound.

### BGM Search Strategy

- Search for loops or long-form tracks matching the video mood
- Minimum duration: 60 seconds (to cover full video length)
- Keywords should describe mood, genre, and instrumentation

### BGM Rhythm Matching (BGM 节奏匹配)

Select BGM tempo based on the dominant `pacing` of each chapter/section in the storyboard:

| Chapter Pacing | BPM Range | Genre / Style |
|---------------|-----------|---------------|
| `slow` | 60-80 BPM | ambient, piano, soft strings, lo-fi |
| `medium` | 90-110 BPM | corporate, light electronic, acoustic |
| `fast` | 120-140 BPM | upbeat electronic, driving percussion, energetic |
| `breathing` shot | N/A | 环境音/氛围音，或与前后章节 BGM 的淡入淡出衔接 |

**Rules:**
- Determine a chapter's dominant pacing by the most frequent pacing value among its shots
- If a video has distinct chapters with different pacing, consider using multiple BGM tracks with crossfade transitions
- For breathing shots between chapters, use ambient/atmospheric sounds or fade the BGM to bridge the transition
- When searching Freesound, include BPM range in keywords (e.g., "ambient piano 70bpm", "upbeat electronic 130bpm")
- BGM should complement, not compete with, the voiceover. Lower BPM tracks naturally sit better under narration.

### SFX Search Strategy

- Search for short, punchy sound effects for transitions and UI feedback
- Maximum duration: typically 3-5 seconds
- Keywords should describe the exact sound needed

### Search Keywords Tips

| Need | Keywords |
|------|----------|
| BGM - Tech | `ambient electronic`, `minimal background`, `corporate` |
| BGM - Emotional | `piano emotional`, `cinematic orchestra`, `inspiring` |
| SFX - Transitions | `whoosh`, `swish`, `swoosh`, `transition` |
| SFX - UI | `notification`, `alert`, `click`, `pop` |
| SFX - Success | `success`, `complete`, `achievement`, `ding` |

### License Information

| License | Attribution | Commercial Use |
|---------|-------------|----------------|
| CC0 (Public Domain) | Not required | Yes |
| CC-BY (Attribution) | Required | Yes |

For CC-BY sounds, record attribution in `raws/audio/LICENSES.txt`.

---

## Output Structure

```
raws/
  audio/
    vo/
      VO_001.wav
      VO_002.wav
      ...
      durations.json       # Critical timing data
    bgm/
      ambient_loop.mp3
    sfx/
      whoosh.mp3
      notification.mp3
    LICENSES.txt           # Attribution for CC-BY sounds
```

---

## Workflow

1. **Generate Voiceovers**
   - Process manifests/script.md for all [VO_XXX] markers
   - All [VO_XXX] markers become VO_XXX.wav files
   - Extract durations to create durations.json

2. **Retrieve BGM**
   - Search for appropriate background music
   - Download to raws/audio/bgm/

3. **Retrieve SFX**
   - Search for transition sounds, UI sounds
   - Download to raws/audio/sfx/

4. **Verify and Report**
   - Check all files generated correctly
   - Report results to user

---

## Incremental Mode

当 dispatch context 中 mode: "incremental" 时：

### 输入
- amendment.affected_resources 中指定的 VO 段落 ID 列表（需要重新生成的语音片段）

### 执行规则
1. 读取 script.md，仅定位指定的 VO 段落
2. 仅对指定的 VO 段落重新生成 TTS 音频
3. 使用与原始生成相同的 character、tone、speed 配置
4. 替换 raws/audio/vo/ 中对应的 WAV 文件
5. **必须重新提取完整的 durations.json**（因为任何音频变化都会影响时序）
6. 返回处理摘要（重新生成了哪些 VO、新旧时长对比）

### 不执行
- 不重新生成未指定的 VO 音频
- 不重新检索 BGM/SFX（除非 amendment 中明确要求）

### 关键注意
- 即使只修改了一个 VO 片段，也必须重新提取整个 durations.json
- 时长变化可能影响下游的 Remotion 时序计算

---

## Audio Timing Rules

These rules are critical for correct Remotion synchronization:

| Rule | Detail |
|------|--------|
| Unit | All durations in **milliseconds** |
| Speed factor | Default 1.2x recommended; durations reflect actual playback length |
| Re-extraction | Mandatory after ANY audio file change |
| Completeness | Every [VO_XXX] in script.md must have a durations.json entry |
| Integer values | No floating point; round to nearest millisecond |

---

## Network Requirements

Edge TTS requires outbound internet access and the Python package `edge-tts`.

---

## User Communication

### Progress Report

```
音频素材生成进行中...

TTS 生成:
  进度: 15/45 条语音
  当前: VO_016
  音色: zh-CN-XiaoxiaoNeural
```

### Completion Report

```
音频素材生成完成。

语音配音:
  - 45 条语音文件
  - 总时长: 4分23秒
  - 音色: zh-CN-XiaoxiaoNeural
  - 时长数据: raws/audio/vo/durations.json

背景音乐:
  - 1 个循环音频 (2分30秒)

音效:
  - 3 个过渡音效
  - 2 个提示音

输出目录: raws/audio/
请确认后继续下一步。
```

---

## Error Handling

| Issue | Cause | Solution |
|-------|-------|----------|
| TTS timeout | Text too long | Split into shorter segments (max 200 chars) |
| Missing durations | Duration extraction not run | Run extraction after TTS generation |
| Audio quality poor | Wrong Edge voice for content | Try a different Edge voice ID |
| Freesound API error | Missing API key | Set FREESOUND_API_KEY |
| Network error | Internet unavailable or Edge TTS request failed | Check internet access and retry |

---

## Critical Notes

### Duration File

`durations.json` is **mandatory** for Remotion implementation. It contains timing data in milliseconds that drives:
- Shot duration calculations
- Audio layer positioning
- Voiceover synchronization

**Always re-extract durations after any audio modification.**

### Default Voice Speed

The default TTS voice may sound slow. Consider using speed `1.2` for more natural pacing. When the storyboard provides a `pacing` field, use the TTS Speed and Storyboard Pacing table above to determine the appropriate speed for each VO segment.