ControlNet
Relying on prompts alone often feels like opening a mystery box. ControlNet adds a layer of geometric structure — you give it a reference (a sketch, a camera angle), and generation follows it as closely as possible instead of being purely random.
A diffusion model on its own just draws randomly. With a ControlNet sketch as a guide, it draws randomly within that sketch 🤪. The skill to develop is learning how to direct these tools to work for you.
For those curious about the algorithms: ControlNet has a close relationship with edge detection — the author finds that topic fascinating and may go deeper on it in a future post.
What Can You Control?
Section titled “What Can You Control?”Common types include: edges / scribble, depth, pose, normals… Different ControlNet types require different reference images. Some need hand-drawn sketches; others take a photo and first convert it into a depth map — preprocessing often accounts for half the work in a real workflow.
Comfy Core doesn’t necessarily include all preprocessing nodes. Many people use community extensions, such as the commonly mentioned ControlNet aux preprocessor pack. Check the workflow instructions for what to install.
Where to Put the Models
Section titled “Where to Put the Models”Place downloaded ControlNet weights in ComfyUI/models/controlnet. Nodes like Load ControlNet will scan this directory.
Your base model stays in checkpoints. Some workflows also use a standalone Load VAE — if the workflow author recommends it, use the separate VAE; otherwise the one bundled in the checkpoint is fine (check your model’s documentation).
What You’re Tweaking in the Workflow
Section titled “What You’re Tweaking in the Workflow”The typical chain is: Load ControlNet → Apply ControlNet → pass the processed conditioning into the sampler.
On the Apply ControlNet node (or similar), you’ll commonly see:
| Parameter | Plain English |
|---|---|
| strength | How “hard” this ControlNet pushes. Too high and it rigidly follows the reference; too low and it might as well not be there. |
| start_percent / end_percent | At what point in the diffusion process this control kicks in / stops — like “sketch loosely in the first half, then lock in structure in the second half.” Very flexible. |
The official docs note: older workflows may contain Apply ControlNet (Old), which is a deprecated node hidden by default. Use the current Apply node for new projects.
Using Multiple ControlNets Together
Section titled “Using Multiple ControlNets Together”You can chain multiple ControlNets (e.g., sketch + depth), or check the official Mixing ControlNets guide for combination strategies. One rule: each ControlNet is competing for influence — stack too many weights and they’ll fight each other. Experiment slowly.
Summary
Section titled “Summary”Prompts control what to draw; ControlNet mostly controls the layout. Pick the right type first (scribble / depth / pose…), get the preprocessing right, then dial in strength and timing.