<aside> 🛠

One ControlNet for 6 Conditioning Modes - SDXL Image Generation Guided by Any Control Image

Guided image generation from a control image (edges, depth, pose, etc.) and a text prompt using SDXL. Supports 6 conditioning modes in one model, canny/lineart, depth, openpose, hed/softedge, normal, and segmentation. Pair with an Annotator node to produce the control image from a source photo.

</aside>

What it does

ControlNet XL Union is an "all-in-one" ControlNet adapter for Stable Diffusion XL (SDXL). Instead of loading a separate ControlNet for every conditioning type (one for canny, one for depth, one for pose, etc.), this single model handles 6 different conditioning modes with the same network parameters. You give it a control image of the appropriate type, a canny edge map, a depth map, a pose skeleton, a HED soft-edge map, a surface normal map, or a semantic segmentation mask, and a text prompt, and SDXL generates a new image that follows the structure of the control image and the content/style of the text prompt. It's the workhorse for any SDXL-based image-to-image workflow that needs structural control but where you want flexibility to switch between control types without swapping models. Built by xinsir and trained on 10M+ carefully captioned images, with the same parameter count as the original ControlNet — no increase in compute or memory vs. running a single-condition ControlNet.

Problem it solves

• One model for many conditioning types – Switch between canny, depth, openpose, hed, normal, and segmentation without loading separate adapters — saves VRAM and pipeline complexity
• Composition-locked generation – Keep the structure, geometry, or pose of a reference photo while changing everything else (style, lighting, materials, era)
• Pose-driven character generation – Use openpose skeletons to place a character in any pose, then style with text
• Depth-guided scene restyling – Restyle a scene while preserving its 3D layout
• Semantic layout control – Use segmentation masks to lock in what goes where (sky, road, person) and let the prompt fill in how it looks
• Sketch-to-image – Turn line art (canny / HED / scribble-like maps) into rendered images
• Multi-condition fusion (built-in) – The model was trained to combine conditions, so you can stack control signals (e.g., openpose + depth) without manual tuning
• Stable foundation for the UI's control nodes – This is the model behind any "use a control image" SDXL flow in the studio

Input/Output

• Input:

  • Control image matching the selected conditioning type:
    • canny - edge maps from Annotator-CannyEdge
    • depth - depth maps from Annotator-DepthAnything
    • openpose - pose skeletons
    • hed - soft edge / HED maps
    • normal - surface normal maps
    • segment - semantic segmentation masks
  • Text prompt describing the desired image (style, content, mood)

  

• Options:

  • Conditioning type (canny / depth / openpose / hed / normal / segment) — must match the control image you supply
  • Strength of the ControlNet guidance (0.0 – 2.0, default 0.8)
    • 0.0 = ignore the control image · 2.0 = strictly follow it
    • 0.7 – 1.0 works well for most inputs; reduce if the control is too rigid
  • Number of diffusion steps (default 25)
    • 20 – 30 gives good quality; more steps add marginal quality at linear time cost
  • Random seed (default 1)
    • 1 picks a new random seed each run; set a fixed integer to reproduce the same image
  • Output image width (512 / 768 / 1024) — the control image is resized to match
  • Output image height (512 / 768 / 1024) — the control image is resized to match
  • Advanced — Negative prompt (leave empty for no negative guidance; e.g. "blurry, low quality, watermark, text")

• Output: Generated image

  • Up to 1024 × 1024 (native SDXL; bucket training supports any aspect ratio)
  • Style and content follow the text prompt; structure follows the control image

  

Accuracy & Speed

Technical Details