Controllable Generation
Why text prompts are not enough
A text-to-image model turns a sentence into a picture. But a sentence is a remarkably coarse way to control an image. You can ask for “a person dancing,” but you cannot easily say exactly where the left hand should be, the precise angle of the torso, which way the head tilts, or how the subject lines up with an existing photo. Words describe content well and spatial layout poorly.
Controllable generation is the family of techniques that add conditioning signals beyond the text prompt so you can steer structure and spatial layout directly — while still letting the base model supply texture, color, lighting, and style. The text says what; an extra control map says where.
The core theme: conditioning a generative model on signals other than text — edge maps, depth, human pose, segmentation masks, reference images, region masks — to gain fine-grained, reproducible control over the output.
ControlNet: structural conditioning
ControlNet (Zhang, Rao & Agrawala, 2023) is the landmark method for adding spatial control to a pretrained diffusion model such as Stable Diffusion. Its key insight is how to inject a new condition without breaking the powerful model you already have.
ControlNet makes a trainable copy of the U-Net's encoder blocks. The original encoder is frozen (its weights never change), so all the knowledge from large-scale pretraining is preserved. The trainable copy receives an extra conditioning image — a Canny edge map, a depth map, a human pose skeleton, a segmentation map, a scribble — and its outputs are added back into the frozen network to steer it.
Zero-convolutions are the trick. The trainable copy connects to the frozen network through 1×1 convolution layers whose weights and bias are initialized to zero. At the very first training step, a zero convolution outputs exactly zero, so it adds nothing — the model behaves identically to the untouched pretrained model. There is no random noise injected into a known-good network, so training is stable.
As training proceeds, gradients flow and these layers gradually learn how much and where the control signal should influence the result. Control is learned smoothly, starting from “do no harm.”
The division of labor is the important takeaway: the spatial map (pose / depth / edges) constrains the structure and layout, while the frozen base model + text prompt supply the texture, color, and style. You get a figure in exactly your pose, painted in exactly your requested style.
│ via zero-convolutions (init 0)
▼
text prompt ─▶ [ FROZEN base U-Net ] ─▶ denoise ─▶ image
Interactive: condition vs. content
Edit the control map on the left by dragging the skeleton's joints. The generated image on the right is built so its structure follows your skeleton, while the text/style prompt changes only its appearance. The control-strength slider scales how strongly the spatial condition is obeyed — exactly the conditioning-scale knob you set when running ControlNet.
+ trainable copy (ControlNet)
This visualization is deterministic and synthetic — a real ControlNet would denoise pixels — but it faithfully shows the principle: layout comes from the control map, appearance comes from the prompt, and the two are controlled independently.
Other ways to control beyond text
ControlNet handles structural conditioning, but it is one tool in a larger toolbox. Each method below adds a different kind of control signal.
| Method | Control signal | What it gives you |
|---|---|---|
| ControlNet | edge / depth / pose / segmentation map | precise spatial structure & layout |
| IP-Adapter (image prompt) | a reference image | borrow the style/subject of an image as a prompt |
| Inpainting / outpainting | a region mask | regenerate only inside (or extend beyond) a chosen region |
| img2img | an input image + denoising strength | keep the rough composition, restyle the rest |
| Regional prompting | per-region text prompts | different content in different parts of the canvas |
| LoRA | small fine-tuned weight deltas | a learned style or subject, toggled on per-generation |
| DreamBooth / Textual Inversion | a few images of a new subject | teach a new token (e.g. your dog) for personalization |
Personalization vs. structural control. DreamBooth and Textual Inversion answer “what subject?” — they bind a new token to a concept the base model never saw (your pet, a product, a person) from just a handful of example images. ControlNet answers “what layout?”. They are complementary and frequently stacked together.
Practical notes & pitfalls
- Preprocessing matters. ControlNet expects the right kind of map — a Canny ControlNet wants an edge map, a depth ControlNet wants a depth map. Feeding the wrong preprocessor gives weak or wrong control.
- Conditioning scale is a dial, not a switch. Too high and the output looks rigid and traced from the map; too low and the model ignores it. The slider in the demo above is exactly this knob.
- Control conflicts. If your text prompt and your control map disagree (a “sitting” prompt with a standing skeleton), the result is a compromise — the spatial map usually wins on layout.
- Methods compose. A typical pipeline might use a LoRA for style, a DreamBooth token for the subject, ControlNet for the pose, and inpainting to fix a hand — all at once.
Key Takeaways
- Text prompts give coarse control; controllable generation adds structural/spatial conditioning on top.
- ControlNet (Zhang et al., 2023) adds a trainable copy of the U-Net encoder that takes an extra condition (edge, depth, pose, segmentation, scribble) and feeds it back into the frozen base model.
- Zero-convolution layers (initialized to zero) make training start from the untouched pretrained model and learn control gradually, without disruption.
- The spatial map constrains structure; the base model + text prompt supply texture and style — controlled independently.
- Other control methods: IP-Adapter (image prompts), inpainting/outpainting (mask-based regional control), img2img strength, regional prompting, LoRA (style/subject), and DreamBooth / Textual Inversion (personalize a new token).
- The unifying idea is conditioning signals beyond text, and the methods compose for fine-grained, reproducible control.