Diffusion Models

A diffusion model is a generative model that makes an image by starting from random noise and removing it a little at a time until a picture is left. In training it sees noise added to real pictures step by step and learns to predict what was added. To generate, it runs that process backwards, starting from random static and removing a little noise at a time until a picture is left. Most image generators of the last few years work this way or on a close variant, including the Stable Diffusion line, Black Forest Labs' Flux models and Google's Imagen.

How diffusion models learn to remove noise#

The forward direction needs no learning. Take a training image, add a little Gaussian noise, then a little more, across a schedule that in the original DDPM paper ran 1,000 steps, until the image cannot be told apart from static. The model's job is the reverse. Given a noisy image and how far along the schedule it is, predict the noise that was added so it can be subtracted. Trained on a very large set of images, the denoiser learns what natural pictures look like at every level of damage.

Generation starts from fresh static and applies the denoiser again and again. Current samplers do not need all 1,000 training steps; they compress the path into 20 to 50 steps. Distilled models (see distillation) get usable images in 1 to 8 steps, because a student was trained to skip ahead. The early steps settle the composition and the late steps fill in texture.

Latent diffusion, published by Rombach and colleagues at the end of 2021 and the basis of Stable Diffusion, made this practical on ordinary hardware. An autoencoder first compresses the image: a 512 by 512 pixel image with three colour channels becomes a 64 by 64 grid with four channels, 48 times less data. The denoiser works entirely in that compressed space, and a decoder turns the result back into pixels at the end.

Steering with a prompt#

A denoiser that drifts toward any plausible image is not much use; you want the one the prompt describes. A text encoder turns the prompt into vectors, and the denoising network reads them at every step through cross-attention, so each step is pulled toward "a fox in a snowy forest" rather than an arbitrary scene. The encoder matters. Early Stable Diffusion used CLIP. Google's Imagen paper found that making the language-model text encoder larger improved image quality and prompt matching more than making the image model larger. Stable Diffusion 3 uses two CLIP encoders and T5 together, and Black Forest Labs' FLUX.2 couples a 24B-parameter Mistral vision-language model with its image transformer to read the prompt.

Prompt conditioning on its own is weak, so samplers add classifier-free guidance. During training the prompt is dropped for a share of examples, with 10% or 20% working best in Ho and Salimans' experiments, so one network learns to denoise both with and without a prompt. At each generation step it makes both predictions and pushes the result away from the unprompted one: noise = unprompted + scale × (prompted − unprompted). A scale of 1 is the plain prompted model with no extra push. Higher scales follow the prompt more closely and lose variety, and pushed far they produce oversaturated, harsh images. Guidance runs the network twice per step, which is why guidance-distilled models that need only one pass are faster.

Editing a real image#

Because generation is guided denoising, you can start partway through instead of from pure static. Image-to-image adds noise to a real photo up to a chosen strength and denoises from there. The strength is how far up the noise schedule you jump: low keeps the composition and changes surface detail, high keeps little of the original. Inpainting adds noise only inside a masked region and, at every step, pastes the known pixels back, so the model regenerates the patch to fit what surrounds it. Swapping a sky or removing a bystander uses the same model as generating from scratch.

Instruction editing has since become the common interface. Google's Nano Banana models, built on Gemini, and OpenAI's GPT Image models take "make the jacket red, keep everything else" with no hand-drawn mask and find the region themselves. Their internals are not all published, and the masked-denoising picture still explains what such an edit preserves and what it redraws.

Choosing a model in October 2026#

The names change faster than the method. Black Forest Labs released FLUX.2 pro, flex and dev on 25 November 2025 and the smaller klein models in January 2026. OpenAI released GPT Image 2 on 21 April 2026. Google set its Imagen 4 API models, released on 24 June 2025, to shut down from 17 August 2026, and names a Nano Banana model as the replacement. Many of the newest models are flow-matching models, which learn a straighter path from noise to image than classic diffusion, but they still generate by repeated denoising conditioned on text.

Licences differ inside a single family. FLUX.2 klein 4B is under Apache 2.0, while klein 9B is under a non-commercial licence, so serving the larger one in a commercial product needs a separate licence from Black Forest Labs. The Stable Diffusion line, from SD 1.5 through SDXL and SD 3.5, still has a large community of LoRA adapters and ControlNets, which matters when the product needs a trained style or a fixed pose. These are foundation models: trained once at great expense, then adapted with adapters and editing tools rather than retrained.

Where the choice is made#

Pick by the constraint you cannot move. Open weights matter when you need your own adapters, local generation or control over the version you run, and then the licence decides which open model you can sell with. An API model makes sense when quality and instruction editing matter more than control, and then the provider's deprecation page matters, because a model you build on can be scheduled for shutdown within about fourteen months of release, as Imagen 4 was. Score the shortlist on your own prompts and your own brand rules, because a public ranking measures general preference and none of the models has seen your style guide.