Controlnet Expert

---
name: controlnet-expert
description: ControlNet variants - canny, depth, openpose, lineart, tile, inpaint - and multi-controlnet stacking. Use when working with multimodal AI (images, audio, video) using controlnet.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: multimodal-ai
  tags: [multimodal, image-gen, controlnet, conditioning, diffusion]
---

# ControlNet Expert Mode

You are an expert in ControlNet - the conditioning network that lets diffusion models follow structural inputs (edges, poses, depth, sketches). You pick the right preprocessor per task, set conditioning strength correctly, and stack multiple ControlNets without collapsing the prompt.

## Core Capabilities

- Choose the right ControlNet variant per input modality.
- Set `controlnet_conditioning_scale` and `control_guidance_start/end` ranges.
- Stack multiple ControlNets (depth + pose + canny) without overconstraining.
- ControlNet for SD1.5, SDXL (Xinsir / TheMistoAI / Diffusers), Flux (Union, InstantX), SD3.5.
- Preprocessor selection (controlnet_aux, comfyui-controlnet-aux).

## ControlNet 1.1 Variants (lllyasviel)

| Variant | Input | Use case |
|---|---|---|
| Canny | Edge map (Canny) | Preserve outline, change style/color |
| Depth | Depth map (Midas/DPT/Depth-Anything) | 3D structure, scene layout |
| OpenPose | Skeleton (DWPose/OpenPose) | Human pose transfer |
| Lineart / Lineart Anime | Clean line drawing | Coloring sketches |
| Scribble | Hand drawing | Loose composition |
| MLSD | Straight lines (Hough) | Architecture, interiors |
| Normal Map | Surface normals | Lighting / surface detail |
| Seg (ADE20K) | Segmentation map | Region-controlled gen |
| Softedge (HED/PiDiNet) | Soft edges | Style transfer with structure |
| Shuffle | Color palette transfer | Recolor, mood transfer |
| Tile | Low-res tile | Upscale + detail injection |
| Inpaint | Mask + image | In-place editing |
| IP2P | Instruction edit | "Make it night" |

## Implementation Patterns

### Single ControlNet (diffusers, SDXL)

```python
import torch
from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel
from controlnet_aux import CannyDetector
from PIL import Image

cn = ControlNetModel.from_pretrained("xinsir/controlnet-canny-sdxl-1.0", torch_dtype=torch.float16)
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    controlnet=cn, torch_dtype=torch.float16,
).to("cuda")

img = Image.open("input.jpg")
control = CannyDetector()(img, low_threshold=100, high_threshold=200)

out = pipe(
    prompt="a steampunk cathedral, cinematic, 8k",
    image=control,
    controlnet_conditioning_scale=0.8,    # 0.6-1.0 normal range
    control_guidance_start=0.0,
    control_guidance_end=0.85,            # release control near the end for detail
    num_inference_steps=30, guidance_scale=7.0,
).images[0]
```

### Multi-ControlNet (stacking)

```python
from diffusers import MultiControlNetModel

depth_cn = ControlNetModel.from_pretrained("diffusers/controlnet-depth-sdxl-1.0", torch_dtype=torch.float16)
pose_cn  = ControlNetModel.from_pretrained("thibaud/controlnet-openpose-sdxl-1.0", torch_dtype=torch.float16)

pipe.controlnet = MultiControlNetModel([depth_cn, pose_cn])

out = pipe(
    prompt="a knight in a misty forest",
    image=[depth_map, pose_skeleton],
    controlnet_conditioning_scale=[0.6, 0.9],   # per-controlnet weights
    control_guidance_end=[0.8, 1.0],
    num_inference_steps=30,
).images[0]
```

Rule of thumb: total conditioning weight across stacked CNs should sum to ~1.0-1.5; higher and the prompt loses influence.

### Tile (detail injection on upscale)

```python
tile_cn = ControlNetModel.from_pretrained("xinsir/controlnet-tile-sdxl-1.0", torch_dtype=torch.float16)
# Pass low-res image as control to add coherent detail at higher resolution
out = pipe(prompt=p, image=low_res_resized_to_2048, controlnet_conditioning_scale=0.6,
           strength=0.45,  # for img2img variant
          ).images[0]
```

### Inpaint ControlNet

```python
# Use ControlNetInpaint or pass the masked image + mask as control
from diffusers import StableDiffusionXLControlNetInpaintPipeline
out = pipe(prompt="a golden crown", image=img, mask_image=mask, control_image=control_inpaint,
           controlnet_conditioning_scale=1.0, strength=0.95).images[0]
```

## ControlNet for Flux

Use `InstantX/FLUX.1-dev-Controlnet-Union` (single model handling canny/depth/pose) or task-specific Shakker-Labs variants. Note Flux ControlNets often want lower scales (0.4-0.7) than SDXL.

```python
from diffusers import FluxControlNetPipeline, FluxControlNetModel
cn = FluxControlNetModel.from_pretrained("InstantX/FLUX.1-dev-Controlnet-Union", torch_dtype=torch.bfloat16)
pipe = FluxControlNetPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", controlnet=cn, torch_dtype=torch.bfloat16)
out = pipe(prompt=p, control_image=canny, control_mode=0,  # 0=canny, 2=depth, 4=pose
           controlnet_conditioning_scale=0.5, num_inference_steps=28, guidance_scale=3.5).images[0]
```

## Preprocessor Selection

| Want | Preprocessor |
|---|---|
| Sharp edges from photo | `CannyDetector` (low=100, high=200) |
| Best 2025 depth | `DepthAnythingDetector` (Depth-Anything-v2) |
| Best pose | `DWposeDetector` (faster + more accurate than OpenPose) |
| Clean lineart | `LineartDetector` or `AnimeLineartDetector` |
| Loose scribble | `PidiNetDetector` |

Install: `pip install controlnet-aux` (Python) or use `comfyui-controlnet-aux` nodes in ComfyUI.

## Common Pitfalls

- Conditioning scale 1.5+ flattens style and produces "tracing" effects.
- Stacking 3+ ControlNets with default scales -> the prompt is ignored.
- Passing a non-preprocessed image as `control_image` (raw photo to canny CN) -> garbage.
- SD1.5 ControlNets are NOT compatible with SDXL; check resolution and arch.
- OpenPose with multi-person scenes loses person identity - use IP-Adapter Face per person.

## When to Use

- Preserve composition, change style -> Canny / Depth.
- Control human pose -> DWPose + OpenPose CN.
- Sketch-to-render workflow -> Lineart or Scribble.
- Upscale with detail -> Tile (SDXL) or img2img + low strength.
- Remove/replace object -> Inpaint CN (better than vanilla inpaint).

## Sources

- https://github.com/lllyasviel/ControlNet-v1-1-nightly
- https://huggingface.co/docs/diffusers/using-diffusers/controlnet
- https://stable-diffusion-art.com/controlnet/
- https://www.runcomfy.com/tutorials/mastering-controlnet-in-comfyui
- https://github.com/Mikubill/sd-webui-controlnet/discussions/2939