Renderdoc Gpu Debug

---
name: renderdoc-gpu-debug
description: >
  GPU frame debugging with RenderDoc via rdc-cli. Use this skill when the user mentions:
  GPU debugging, .rdc files, RenderDoc, shader issues, pipeline state, rendering artifacts,
  frame capture, shadow problems, visual glitches, draw calls, render targets, pixel history,
  texture inspection, depth buffer, blend state, rasterizer state, vertex shader, fragment shader,
  pixel shader, compute shader, mesh output, GPU performance, overdraw, bandwidth, shadow map,
  PCF filtering, screen-space artifacts, z-fighting, alpha blending, stencil, MSAA,
  "why does this look wrong", "debug this pixel", "what's drawing here", "capture a frame",
  "inspect the pipeline", "export render target", vulkan debugging, D3D debugging, GL debugging.
  DO NOT use for: CSS rendering, React rendering, server-side rendering, HTML layout,
  browser DevTools, web performance, canvas 2D, SVG rendering.
---

# RenderDoc GPU Debugging Skill

## Overview

This skill enables GPU frame capture, inspection, and debugging using `rdc-cli`, a 66-command CLI wrapping RenderDoc's Python API. It works with Vulkan, D3D11, D3D12, and OpenGL applications.

### Prerequisites

Before any GPU debugging, verify the environment:

```bash
rdc doctor
```

All checks should pass. If `rdc doctor` fails, check:
- `RENDERDOC_PYTHON_PATH` is set to the directory containing `renderdoc.pyd`
- `renderdoc.dll` is in the same directory as `renderdoc.pyd`
- `renderdoccmd.exe` is on PATH or in the module directory

## 1. Session Lifecycle

Every inspection session follows open-work-close:

```bash
rdc open path/to/capture.rdc   # Start daemon, load capture
# ... inspection commands ...
rdc close                       # Release resources, stop daemon
```

Check session state with `rdc status`. Only one capture can be open per session (use `--session name` for parallel sessions).

**IMPORTANT**: Always close sessions when done. Leaked daemon processes consume GPU memory.

## 2. Capture Workflow

### Capture from an executable

```bash
rdc capture -o output.rdc -- /path/to/app [app-args...]
```

Key options:
- `-o, --output`: Output .rdc file path
- `--frame N`: Capture frame N (default: first presented frame)
- `--timeout S`: Capture timeout in seconds (default: 60)
- `--api-validation`: Enable API validation layer
- `--ref-all-resources`: Reference all resources (larger capture, more complete)
- `--wait-for-exit`: Wait for the app to exit after capture

### CWD matters

Many applications resolve assets relative to CWD. Always set CWD appropriately:

```bash
cd /path/to/app/root && rdc capture -o /output/path/capture.rdc -- ./app
```

For lux-playground specifically:
```bash
cd D:/shaderlang && python D:/renderdoc/capture_frame.py
```

### Vulkan layer requirement

For Vulkan apps, the RenderDoc Vulkan layer must be registered:
- Registry: `HKCU\SOFTWARE\Khronos\Vulkan\ImplicitLayers` must contain `D:\renderdoc\module\renderdoc.json` (DWORD 0)
- Environment: `ENABLE_VULKAN_RENDERDOC_CAPTURE=1` must be set

### If headless capture fails

Headless Vulkan apps (no swapchain / no `vkQueuePresentKHR`) cannot be captured by RenderDoc since frame boundaries are defined by present calls. Options:
1. Use `--interactive` mode if the app supports it (opens a window with a swapchain)
2. Use the Python API via `capture_frame.py` for reliable programmatic capture
3. Use `--trigger` mode (inject without auto-capture, then `rdc capture-trigger`)
4. Fall back to `renderdoccmd capture` directly

## 3. Frame Exploration

After opening a capture, start with high-level overview:

```bash
rdc info --json          # API, GPU, driver, resolution, frame number
rdc stats --json         # Per-pass breakdown, top draws, largest resources
rdc passes               # List render passes (debug markers / implicit passes)
rdc draws --limit 20     # First 20 draw calls
rdc count draws          # Total draw call count
```

### Navigate by pass

```bash
rdc draws --pass "Shadow Pass" --json    # Draws in a specific pass
rdc draws --pass "GBuffer" --limit 10    # First 10 GBuffer draws
```

### Navigate by event

```bash
rdc events --limit 50                    # All API events (not just draws)
rdc events --type DrawIndexed --json     # Filter by event type
rdc event 42 --json                      # Detail for one event
```

### VFS browsing

The virtual filesystem provides a structured view of the capture:

```bash
rdc ls /                      # Root directories
rdc ls /textures -l           # List textures with metadata
rdc tree /pipelines --depth 2 # Pipeline tree
rdc cat /events/42            # Event detail
```

## 4. Pipeline State Inspection

Get full pipeline state at any draw call:

```bash
rdc pipeline EID --json              # Full pipeline state
rdc pipeline EID vs --json           # Vertex shader section only
rdc pipeline EID ps --json           # Pixel/fragment shader section only
rdc pipeline EID om --json           # Output merger (blend, depth, stencil)
rdc pipeline EID rs --json           # Rasterizer state (culling, viewport)
rdc pipeline EID ia --json           # Input assembler (vertex buffers, topology)
rdc pipeline EID ds --json           # Depth-stencil state
```

### Bound resources

```bash
rdc bindings EID --json              # All bindings for all stages
rdc bindings EID --set 0 --json      # Descriptor set 0 only
```

## 5. Shader Inspection

### View shader metadata and source

```bash
rdc shader EID vs --json             # Vertex shader metadata
rdc shader EID ps --json             # Fragment/pixel shader metadata
rdc shader EID ps --source           # Debug source code (if available)
rdc shader EID ps --reflect --json   # Reflection: inputs, outputs, cbuffers
rdc shader EID ps --constants --json # Constant buffer values at this draw
```

### Disassembly targets

```bash
rdc shader EID ps --targets          # List available disassembly formats
rdc shader EID ps --target spirv     # SPIR-V disassembly
rdc shader EID ps --target glsl      # GLSL (cross-compiled, if available)
```

### Search shaders

```bash
rdc search "shadow" --json           # Search all shader disassembly for "shadow"
rdc search "main" --stage ps         # Search only pixel shaders
rdc shaders --stage ps --json        # List all unique pixel shaders
rdc shader-map --json                # EID-to-shader mapping
```

## 6. Visual Inspection: Export-View-Analyze

This is the core pattern for visual debugging. **Always export to PNG, then use Claude Code's Read tool to view the image.**

### Export render targets

```bash
rdc rt EID -o D:/renderdoc/captures/analysis/render_target.png
rdc rt EID --target 1 -o D:/renderdoc/captures/analysis/rt_target1.png  # MRT target 1
```

### Export textures

```bash
rdc texture RESID -o D:/renderdoc/captures/analysis/texture.png
rdc texture RESID --mip 2 -o D:/renderdoc/captures/analysis/mip2.png
```

### Export thumbnail

```bash
rdc thumbnail -o D:/renderdoc/captures/analysis/thumb.png
```

### View exported images

After exporting a PNG, **use the Read tool** to view it (Claude Code is multimodal):

```
Read tool: D:/renderdoc/captures/analysis/render_target.png
```

Do NOT use `cat` or `bash` to view images. The Read tool renders them visually.

### Analyze

Correlate what you see in the image with pipeline state data:
1. Export the render target at the suspicious draw
2. View it with the Read tool
3. Check pipeline state (`rdc pipeline EID --json`)
4. Check shader constants (`rdc shader EID ps --constants --json`)
5. Form hypothesis and verify

## 7. Pixel Debugging

### Pixel history

Find all draws that wrote to a pixel:

```bash
rdc pixel X Y --json                 # Full pixel history at current event
rdc pixel X Y EID --json             # Pixel history up to EID
```

### Pick pixel color

Read the current color at a pixel:

```bash
rdc pick-pixel X Y --json            # Color at (X,Y) from current render target
rdc pick-pixel X Y EID --json        # Color at specific event
```

### Debug pixel shader

Step through shader execution for a specific pixel:

```bash
rdc debug pixel EID X Y --json           # Shader inputs/outputs summary
rdc debug pixel EID X Y --trace          # Full execution trace (every line)
rdc debug pixel EID X Y --dump-at 42     # Variable snapshot at line 42
```

### Debug vertex shader

```bash
rdc debug vertex EID VTXID --json        # Vertex shader debug for vertex ID
rdc debug vertex EID VTXID --trace       # Full trace
```

### Debug compute shader

```bash
rdc debug thread EID GX GY GZ TX TY TZ --json   # Compute thread debug
```

## 8. Shader Edit-Replay

Modify shaders without recompiling the application:

```bash
# 1. Check available encodings
rdc shader-encodings --json

# 2. Extract shader source
rdc shader EID ps --source -o D:/renderdoc/captures/analysis/shader.frag

# 3. Edit the shader (use Edit tool)

# 4. Build the modified shader
rdc shader-build D:/renderdoc/captures/analysis/shader.frag --encoding GLSL --stage ps --json

# 5. Replace (use the shader_id from build output)
rdc shader-replace EID ps --with SHADER_ID --json

# 6. Export to verify the change
rdc rt EID -o D:/renderdoc/captures/analysis/after_edit.png

# 7. Restore when done
rdc shader-restore EID ps
# or: rdc shader-restore-all
```

## 9. Frame Comparison

Compare two captures:

```bash
# Quick summary of differences
rdc diff capture_a.rdc capture_b.rdc --shortstat

# Detailed comparisons
rdc diff capture_a.rdc capture_b.rdc --draws --json       # Draw call differences
rdc diff capture_a.rdc capture_b.rdc --resources --json   # Resource differences
rdc diff capture_a.rdc capture_b.rdc --passes --json      # Pass structure differences
rdc diff capture_a.rdc capture_b.rdc --framebuffer --json  # Final framebuffer diff

# Visual diff
rdc diff capture_a.rdc capture_b.rdc --framebuffer --diff-output D:/renderdoc/captures/analysis/diff.png
```

## 10. Debugging Recipes

### Recipe: Object is invisible

```bash
# 1. Find the draw call that should render the object
rdc draws --json | jq '.[] | select(.name | contains("ObjectName"))'
# or search by pass:
rdc draws --pass "Main Pass" --json

# 2. Check if it's being culled
rdc pipeline EID rs --json   # Look at CullMode, FrontFace

# 3. Check depth state
rdc pipeline EID ds --json   # DepthEnable, DepthFunc, DepthWriteMask

# 4. Check blend state (maybe alpha is 0)
rdc pipeline EID om --json   # BlendEnable, SrcBlend, DestBlend

# 5. Check vertex transform
rdc debug vertex EID 0 --json   # Is the position off-screen or behind camera?

# 6. Check if the draw is even issuing primitives
rdc draw EID --json   # VertexCount, InstanceCount, IndexCount
```

### Recipe: Colors are wrong

```bash
# 1. Export the render target to see what's there
rdc rt EID -o D:/renderdoc/captures/analysis/wrong_color.png

# 2. Pick the problematic pixel
rdc pick-pixel X Y EID --json

# 3. Check texture bindings — is the right texture bound?
rdc bindings EID --json

# 4. Export the bound texture
rdc texture RESID -o D:/renderdoc/captures/analysis/bound_texture.png

# 5. Check shader constants — wrong material colors?
rdc shader EID ps --constants --json

# 6. Check blend state — additive when it should be alpha?
rdc pipeline EID om --json

# 7. Debug the pixel shader to trace the calculation
rdc debug pixel EID X Y --trace
```

### Recipe: Shadows are broken

```bash
# 1. Find the shadow pass
rdc passes --json
rdc draws --pass "Shadow*" --json

# 2. Export the shadow map
SHADOW_EID=$(rdc draws --pass "Shadow*" -q | tail -1)
rdc rt $SHADOW_EID -o D:/renderdoc/captures/analysis/shadow_map.png

# 3. Check shadow map resolution (is it too small?)
rdc bindings $SHADOW_EID --json   # Look at render target dimensions

# 4. Check depth bias
rdc pipeline $SHADOW_EID rs --json   # DepthBias, SlopeScaledDepthBias

# 5. Find the lighting pass that reads the shadow map
rdc draws --pass "Raster*" --json
LIGHT_EID=$(rdc draws --pass "Raster*" -q | head -1)

# 6. Check how the shadow map is sampled
rdc shader $LIGHT_EID ps --source   # Look for shadow sampling code
rdc shader $LIGHT_EID ps --constants --json   # Light matrices, bias values

# 7. Debug a shadowed pixel
rdc debug pixel $LIGHT_EID X Y --trace
```

### Recipe: Performance is bad

```bash
# 1. Get frame overview
rdc stats --json

# 2. Count draws per pass
rdc passes --json   # Look for passes with excessive draw counts

# 3. Look for redundant state changes
rdc events --limit 500 --json | jq 'group_by(.type) | map({type: .[0].type, count: length}) | sort_by(-.count)'

# 4. Check for large resources
rdc resources --sort size --json | jq '.[-10:]'   # Top 10 largest resources

# 5. Look for overdraw (if GPU counters available)
rdc counters --list
rdc counters --name "overdraw" --json

# 6. Export with wireframe overlay to visualize overdraw
rdc rt EID --overlay wireframe -o D:/renderdoc/captures/analysis/wireframe.png
```

### Recipe: What changed between two frames

```bash
# Quick diff
rdc diff before.rdc after.rdc --shortstat

# Detailed diffs
rdc diff before.rdc after.rdc --draws --json
rdc diff before.rdc after.rdc --framebuffer --diff-output D:/renderdoc/captures/analysis/frame_diff.png

# Compare pipeline state at specific draw
rdc diff before.rdc after.rdc --pipeline EID --json
```

### Recipe: Debug this pixel

```bash
# 1. Get pixel history — which draws touched this pixel?
rdc pixel X Y --json

# 2. Pick the draw that produced the final color
# (usually the last non-failing entry in pixel history)

# 3. Get the pixel's current value
rdc pick-pixel X Y EID --json

# 4. Debug the shader at that pixel
rdc debug pixel EID X Y --json

# 5. Get full execution trace if needed
rdc debug pixel EID X Y --trace

# 6. Check variable values at a specific shader line
rdc debug pixel EID X Y --dump-at LINE_NUMBER
```

## 11. Output Size Management

GPU captures can produce enormous output. Follow these rules:

1. **Always use `--limit` for exploration**: `rdc draws --limit 20`, `rdc events --limit 50`
2. **Filter by pass**: `rdc draws --pass "Shadow Pass"` instead of all draws
3. **Use `-q` for ID lists**: `rdc draws -q` returns only EIDs, not full details
4. **Use `--json` selectively**: JSON is verbose; use TSV default for scanning
5. **Pipeline sections**: `rdc pipeline EID vs` instead of full `rdc pipeline EID`
6. **Limit search results**: `rdc search "pattern" --limit 10`

### Size estimates
- `rdc info`: ~20 lines
- `rdc draws --limit 20`: ~25 lines
- `rdc pipeline EID --json`: ~200-500 lines (full), ~50 lines (per section)
- `rdc shader EID ps --source`: ~50-500 lines depending on shader
- `rdc debug pixel EID X Y --trace`: ~100-1000 lines depending on shader complexity

## 12. Error Handling

### rdc not found
```bash
pip install rdc-cli
```

### rdc doctor fails
Check `RENDERDOC_PYTHON_PATH` points to directory containing `renderdoc.pyd` and `renderdoc.dll`.

### Capture fails with "no swapchain"
The app may not present frames in the expected way. Try:
```bash
rdc capture --trigger -- /path/to/app   # Inject-only mode
# Then manually trigger: rdc capture-trigger
```

### Daemon not responding
```bash
rdc status          # Check if daemon is alive
rdc close           # Force close
rdc open capture.rdc  # Re-open
```

### GPU feature not supported
Some features (GPU counters, pixel history) depend on the GPU and driver. Check:
```bash
rdc gpus --json     # GPU capabilities
rdc counters --list # Available counters (empty if not supported)
```

## Command Reference

For the complete list of all 66 commands with arguments, options, types, and defaults, see [references/commands-quick-ref.md](references/commands-quick-ref.md).

For extended debugging recipes with expected output shapes, see [references/debugging-recipes.md](references/debugging-recipes.md).