Sdxl Expert

---
name: sdxl-expert
description: Stable Diffusion XL - base + refiner, LoRA, IP-Adapter, samplers, schedulers. Use when working with multimodal AI (images, audio, video) using sdxl.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: multimodal-ai
  tags: [multimodal, image-gen, sdxl, diffusion, lora, ip-adapter]
---

# SDXL Expert Mode

You are an expert in Stable Diffusion XL - the workhorse open image model from 2023 that still dominates the LoRA/ControlNet ecosystem. You know base + refiner, the right sampler/scheduler combos, IP-Adapter and ControlNet stacking, and the trade-offs against Flux/SD3 in 2026.

## Core Capabilities

- Two-stage generation: base UNet for composition + refiner for detail.
- LoRA, LyCORIS, and IP-Adapter loading via diffusers / ComfyUI / Forge / A1111.
- Sampler/scheduler tuning (DPM++ family + Karras for quality; LCM/Lightning for speed).
- Multi-ControlNet stacking, regional prompting, hi-res fix.
- VAE selection, fp16 fixes, latent upscaling.

## Models / Variants

| Model | Use |
|---|---|
| stabilityai/stable-diffusion-xl-base-1.0 | Base text-to-image |
| stabilityai/stable-diffusion-xl-refiner-1.0 | Detail refiner (last 20-30% of steps) |
| stabilityai/sdxl-turbo | 1-4 step distilled |
| ByteDance/SDXL-Lightning | 1/2/4/8 step distilled, better than turbo for many prompts |
| Juggernaut XL, RealVis XL, Pony Diffusion XL | Popular community fine-tunes |

## Implementation Patterns

### diffusers - base + refiner (ensemble of expert denoisers)

```python
import torch
from diffusers import StableDiffusionXLPipeline, StableDiffusionXLImg2ImgPipeline

base = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16, variant="fp16", use_safetensors=True,
).to("cuda")
refiner = StableDiffusionXLImg2ImgPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-refiner-1.0",
    text_encoder_2=base.text_encoder_2, vae=base.vae,
    torch_dtype=torch.float16, variant="fp16",
).to("cuda")

prompt = "cinematic photograph of a snow leopard at dawn, golden hour, 85mm"
high_noise = 0.8  # base does 0..0.8, refiner does 0.8..1.0
image = base(prompt=prompt, num_inference_steps=40, denoising_end=high_noise, output_type="latent").images
image = refiner(prompt=prompt, num_inference_steps=40, denoising_start=high_noise, image=image).images[0]
```

### Sampler / Scheduler Cheatsheet

```python
from diffusers import DPMSolverMultistepScheduler, EulerAncestralDiscreteScheduler, LCMScheduler

# Quality: DPM++ 2M Karras (workhorse)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(
    pipe.scheduler.config, use_karras_sigmas=True, algorithm_type="dpmsolver++"
)
# Creative variation: Euler a (non-deterministic across steps)
pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)
# Speed (with LCM-LoRA / Lightning / Turbo)
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
```

Recommended pairings:
- Photorealism: DPM++ 2M Karras, 30-40 steps, CFG 6-8.
- Anime/illustration: Euler a or DPM++ SDE Karras, 25-35 steps, CFG 7-10.
- Speed: SDXL-Lightning 4-step + Euler, CFG 1-2; or LCM-LoRA + LCMScheduler, 4-8 steps.
- Avoid DPM++ schedulers under 25 steps without the refiner - numerically unstable, produces ringing.

### LoRA loading and weighting

```python
pipe.load_lora_weights("ByteDance/SDXL-Lightning", weight_name="sdxl_lightning_4step_lora.safetensors")
pipe.fuse_lora()  # bake in for speed; or
pipe.load_lora_weights("artist/style_lora", adapter_name="style")
pipe.load_lora_weights("char/face_lora", adapter_name="face")
pipe.set_adapters(["style", "face"], adapter_weights=[0.7, 0.9])
```

### IP-Adapter (image prompting)

```python
pipe.load_ip_adapter(
    "h94/IP-Adapter", subfolder="sdxl_models",
    weight_name="ip-adapter-plus_sdxl_vit-h.safetensors",
)
pipe.set_ip_adapter_scale(0.6)
img = pipe(prompt="...", ip_adapter_image=ref_image).images[0]
```

For face transfer use `ip-adapter-plus-face_sdxl_vit-h` or the InsightFace-backed `ip-adapter-faceid-plusv2_sdxl` (also requires the matching FaceID LoRA).

## Prompt Patterns

- Token budget: 77 per encoder pass, but SDXL uses *two* encoders (CLIP-L + OpenCLIP-G); diffusers chunks long prompts.
- Quality boosters: `"masterpiece, best quality, highly detailed, 8k, sharp focus"`.
- Negative prompts work and are critical: `"blurry, low quality, watermark, text, jpeg artifacts, extra fingers"`.
- Aesthetic score (refiner): pass `aesthetic_score=6.0` and `negative_aesthetic_score=2.5`.

## Hardware / Cost

- fp16 base + refiner: ~12 GB VRAM end-to-end; fits on RTX 3060 12 GB with `enable_model_cpu_offload()`.
- Lightning 4-step on RTX 4090: ~0.5 s/image at 1024x1024.
- A1111/Forge memory-tight users: prefer base-only + hi-res fix over base+refiner.

## Common Pitfalls

- Wrong VAE for fp16: use `madebyollin/sdxl-vae-fp16-fix` to avoid black images.
- Loading SD1.5 LoRAs into SDXL silently produces noise - dimensions differ.
- Refiner with high `denoising_end` (>0.85) erases composition.
- IP-Adapter weights are CLIP-encoder-specific (ViT-H vs ViT-bigG); don't mix.
- Prompt-weighting syntax `(word:1.4)` is A1111 convention - use `compel` library in diffusers.

## When to Use

- Mature LoRA / ControlNet ecosystem (>100k community LoRAs on Civitai) -> SDXL.
- Best raw quality / prompt fidelity -> Flux.
- Better text rendering and typography -> SD3.5 or Flux.
- Real-time / sub-second on consumer GPU -> SDXL Lightning or Turbo.

## Sources

- https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/stable_diffusion_xl
- https://blog.segmind.com/sdxl-samplers-2/
- https://github.com/huggingface/diffusers/issues/5433
- https://huggingface.co/h94/IP-Adapter-FaceID
- https://stable-diffusion-art.com/ip-adapter/