Glossary
What is ControlNet in Stable Diffusion?
In short
ControlNet in Stable Diffusion is an extra neural network added to an image generation model. It makes the picture follow a given structure: a person’s pose, outlines, a depth map, or a floor plan.
Images, video, audio · ControlNet
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine a coloring book: the outlines are already drawn, and you choose the colors. However you color it, a cat stays a cat and a house stays a house, because the lines show where everything goes. ControlNet uses a similar idea. The computer gets such a page with lines or a stick figure, and it draws the picture exactly along it, not at random.
In plain words
A standard image model listens only to text, so the hero’s pose or the position of objects comes out random every time. ControlNet adds a second input: a map. It can be a sketch of outlines, a stick figure that shows a pose, a depth map, or a layout of where the sky, the road, and the buildings are. The model draws from the description but keeps the shape from the map. For example, an architect uploads a drawing of a facade and gets ten finish options with the windows and doors in their places. Usually a separate version of the module is trained for each type of map.
For business
Structural control makes generation usable for work where the exact shape matters: interior design, facade visualization, clothes on models in the right pose, storyboards for ads. Without it, a designer spends time going through random options; with it, the designer sets the composition at once. Costs: the module runs on top of an open model, so you need your own server or a cloud GPU, and an engineer builds a pipeline for the task. Each extra condition adds load on GPU memory. The risks are the same as for image generation in general: the base model’s license, similarity to other people’s work, and errors in details. Before you buy hardware, count how many images a day you need and which conditions you will really use.
For a specialist
Zhang, Rao, and Agrawala (2023) proposed ControlNet as an architecture that adds a spatial condition to a pretrained text-to-image diffusion model. The weights of the original model are frozen (locked copy), and its encoder blocks are copied into a trainable copy that receives the condition map. The copy connects to the main network through zero convolutions: convolution layers initialized with zeros. So at the start of training the module adds nothing and does not harm the base model. The authors tested conditions by edges, depth, segmentation, and human pose, alone and together, with and without text. They report stable training both on small sets (under 50,000 images) and large ones (over a million). Several modules are combined with weights (conditioning scale). The same scheme is used for inpainting and outpainting when the shape of an object must stay.
Example
An interior design studio photographs a client’s empty room and builds a depth map from the photo. A neural network with ControlNet draws five renovation options in which the walls, windows, and doorways match the real layout.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map