Canny or edge
Carries forwardHigh-contrast outlines, object silhouettes, and major internal edges.
Watch forTexture and lighting can create noisy edges; every retained line may over-constrain the new image.
Technical reference
ControlNet: choose the condition that preserves the structure you care about
ControlNet is an architecture for adding image-based conditions to a compatible diffusion model. A condition can describe edges, depth, pose, lines, segmentation, or another spatial signal. It gives generation more structural guidance than a text prompt alone, but it does not copy every visual detail and it is not interchangeable across every base model.
Core idea
Prompt describes; condition constrains
A text prompt can say “a runner crossing a plaza,” but it may not hold the exact body pose or architectural lines. A ControlNet workflow normally creates or accepts a control map, sends it alongside the prompt to a compatible ControlNet and base model, and lets the generated image follow that map to a chosen degree.
The original paper describes a locked copy of pretrained model blocks, a trainable copy that learns the condition, and zero-initialized convolution layers that connect them. For a user, the practical consequence is simpler: the selected control adds a spatial instruction without replacing the text prompt.
Common conditions
Start from the part of the source that must survive
Depth
Carries forwardApproximate foreground, background, surface orientation, and spatial arrangement.
Watch forIt does not preserve identity, exact object boundaries, color, or fine material detail.
OpenPose
Carries forwardDetected body and sometimes hand or face landmark positions.
Watch forDetection can miss occluded joints; pose does not preserve the person, clothing, body shape, or anatomy quality.
Line art or scribble
Carries forwardChosen contours or a simplified drawing that defines the composition.
Watch forSparse or ambiguous lines leave more interpretation to the model; dense lines can lock unwanted detail.
Segmentation
Carries forwardBroad semantic regions such as sky, building, road, person, or vegetation.
Watch forRegion labels simplify boundaries and do not carry texture, exact geometry, or identity.
A practical setup checklist
- 1. Match the base family. Use a ControlNet trained or adapted for the exact model family in your workflow.
- 2. Inspect the preprocessor. Check the edge, depth, pose, or line map before generation; a bad map creates a bad constraint.
- 3. Write the visual change. Use the prompt for subject, material, setting, light, and style rather than restating only the control map.
- 4. Tune strength deliberately. Too little control can lose the structure; too much can make the result rigid or preserve unwanted artifacts.
- 5. Compare against the job. Check the required pose, outline, depth, or layout—not only whether the image looks polished.
ControlNet is not the same as Image to Image
Image to Image usually conditions generation on the source image as a whole. ControlNet extracts or accepts a more specific structural representation. That can make pose, edges, or depth easier to preserve while allowing appearance to change, but the result still depends on the base model, compatible control weights, preprocessing, prompt, and settings.
It also is not exact reconstruction
A depth map does not preserve a face. An edge map may preserve unwanted lines. Pose landmarks do not guarantee hands, clothing, or identity. Segment maps simplify boundaries. For measured geometry, exact products, or regulated imagery, treat the result as a generated interpretation and verify it independently.
If you only need the outcome, not ControlNet itself
Use a tool that owns the task you actually have. These workflows are less configurable and should not be described as ControlNet, but they may be enough when the goal is a visual transformation or focused edit.
Primary sources
Read the architecture and original examples
The paper is the primary source for the architecture and experiments. The authors’ repository shows the original Stable Diffusion 1.5 examples and preprocessors. A modern tool may use a different model family or editing method, so its own compatibility documentation still governs setup.