MMOOZZGG.com

Glossary

What is ControlNet in Stable Diffusion?

In short

ControlNet in Stable Diffusion is an extra neural network added to an image generation model. It makes the picture follow a given structure: a person’s pose, outlines, a depth map, or a floor plan.

Images, video, audio · ControlNet

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine a coloring book: the outlines are already drawn, and you choose the colors. However you color it, a cat stays a cat and a house stays a house, because the lines show where everything goes. ControlNet uses a similar idea. The computer gets such a page with lines or a stick figure, and it draws the picture exactly along it, not at random.

In plain words

A standard image model listens only to text, so the hero’s pose or the position of objects comes out random every time. ControlNet adds a second input: a map. It can be a sketch of outlines, a stick figure that shows a pose, a depth map, or a layout of where the sky, the road, and the buildings are. The model draws from the description but keeps the shape from the map. For example, an architect uploads a drawing of a facade and gets ten finish options with the windows and doors in their places. Usually a separate version of the module is trained for each type of map.

For business

Structural control makes generation usable for work where the exact shape matters: interior design, facade visualization, clothes on models in the right pose, storyboards for ads. Without it, a designer spends time going through random options; with it, the designer sets the composition at once. Costs: the module runs on top of an open model, so you need your own server or a cloud GPU, and an engineer builds a pipeline for the task. Each extra condition adds load on GPU memory. The risks are the same as for image generation in general: the base model’s license, similarity to other people’s work, and errors in details. Before you buy hardware, count how many images a day you need and which conditions you will really use.

For a specialist

Zhang, Rao, and Agrawala (2023) proposed ControlNet as an architecture that adds a spatial condition to a pretrained text-to-image diffusion model. The weights of the original model are frozen (locked copy), and its encoder blocks are copied into a trainable copy that receives the condition map. The copy connects to the main network through zero convolutions: convolution layers initialized with zeros. So at the start of training the module adds nothing and does not harm the base model. The authors tested conditions by edges, depth, segmentation, and human pose, alone and together, with and without text. They report stable training both on small sets (under 50,000 images) and large ones (over a million). Several modules are combined with weights (conditioning scale). The same scheme is used for inpainting and outpainting when the shape of an object must stay.

Example

An interior design studio photographs a client’s empty room and builds a depth map from the photo. A neural network with ControlNet draws five renovation options in which the walls, windows, and doorways match the real layout.

Related terms

Questions and answers

What is ControlNet for? To make a picture repeat a given shape: a pose, outlines, depth, or a floor plan. The text sets the style and content, and the map sets the composition.
Does ControlNet work in cloud services? The module was built for open diffusion models, so it usually runs on your own computer or server. Some online editors include similar features, such as uploading a pose or a sketch, but give them their own names.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us