Diffusion Models

A diffusion model is the engine behind most modern image generators, including the Stable Diffusion lineage, Google's Imagen, and Black Forest Labs' Flux. The core idea is almost paradoxical: to learn how to create images, the model first learns how to destroy them. It studies what it looks like to add noise to a picture, then runs that process in reverse, conjuring a clean image out of random static. Understanding that forward-and-reverse symmetry is the key to understanding why these models work and how you steer them.

Noising forward, denoising backward#

Training has two directions. The forward process takes a real training image and adds a little Gaussian noise, then a little more, across a schedule of roughly 1,000 steps until the image is indistinguishable from pure static. This direction is trivial and requires no learning. The model's actual job is the reverse: given a noisy image and a number telling it how far along the noise schedule it sits, predict the noise that was added so it can be subtracted. Train that denoiser on hundreds of millions of image-noise pairs and it learns what natural images look like at every level of corruption.

Generation then runs the reverse process from scratch. Start with a fresh field of random noise, apply the learned denoiser, and repeat. Production samplers do not need all 1,000 training steps; modern schedulers compress the trajectory into 20 to 50 denoising steps, and distilled models (see distillation) get usable images in 1 to 8 steps by training a student to skip ahead. Each step nudges the static a little closer to a coherent picture, which is why early steps decide composition and late steps refine texture.

One more engineering trick made this affordable. Latent diffusion, the 2021 idea behind Stable Diffusion, first compresses the image with an autoencoder, so a 512 by 512 pixel image becomes a 64 by 64 latent grid, roughly 48 times less data. The denoiser runs entirely in that compressed space and a decoder expands the result back to pixels at the end. That is why an SDXL-class model samples in seconds on one GPU instead of minutes.

Steering it with text#

A denoiser that wanders toward any plausible image is not useful; you want the one your prompt describes. That is text conditioning. The prompt is encoded into vectors by a text encoder and injected into the denoising network through cross-attention, so every step is pulled toward "a fox in a snowy forest" rather than an arbitrary scene. The encoder choice matters: early Stable Diffusion used CLIP text encoders, Imagen showed that a large language-trained encoder like T5 dramatically improves spelling and compositional accuracy, SD3 combines both, and Flux 2 goes further by using a 24B-parameter vision-language model as its encoder, which is a big reason recent models finally render legible text.

Conditioning alone is weak, so samplers add classifier-free guidance (CFG). During training, the prompt is randomly dropped about 10% of the time, so one network learns both conditional and unconditional denoising. At generation, each step runs both predictions and extrapolates away from the unconditional one: noise = uncond + scale times (cond minus uncond). A guidance scale around 1 ignores the prompt, 5 to 8 is the usual sweet spot, and 15 or more gives oversaturated, fried-looking images. The cost is real: CFG doubles the network evaluations per step, which is one of the things guidance-distilled models like Flux klein bake away.

Editing, not just creating#

Because generation is just guided denoising, you can start partway in instead of from pure noise. Image-to-image noises a real photo to, say, 60% strength and denoises from there, keeping composition while restyling content; the strength knob is literally how far up the noise schedule you jump. Inpainting masks a region, noises only that region, and at every denoising step pastes the known pixels back in, so the model regenerates the patch to fit its surroundings. That is how you swap a sky, remove a bystander, or extend a product shot, all with the same backbone.

In 2026 the frontier of editing is instruction-driven: Google's Nano Banana models (built on Gemini) and OpenAI's GPT Image line accept "make the jacket red, keep everything else" without a hand-drawn mask, localizing the edit themselves. Under the hood the mechanics vary, but the masked-denoising intuition still explains what is preserved and what gets resynthesized.

The model families in 2026#

The current landscape, worth knowing by name. The Stable Diffusion lineage (SD 1.5, SDXL, SD 3.5) remains the open-weight workhorse with the deepest ecosystem of LoRA fine-tunes and ControlNets. Flux 2 from Black Forest Labs, the team behind the original Stable Diffusion, shipped in late 2025 with five variants from the API-only max and pro down to klein, open-weight 4B and 9B models that generate in under a second on a consumer GPU; three Flux 2 variants sit in the text-to-image Arena top ten. Imagen 4 is Google's pixel-space family behind Gemini's image generation, and GPT Image 1.5 currently tops the Arena leaderboard with standout text rendering. Strictly speaking, the newest of these are flow matching models, which learn straighter noise-to-image paths than classic diffusion, but the mental model is identical: iterative denoising, conditioned on text.

These are foundation models in their own right: trained once at enormous cost, then adapted downstream with LoRAs, ControlNets, and editing interfaces rather than retrained.

Practical takeaways#

Think of a diffusion model as a noise remover you can aim with words. More steps buys quality up to a point, around 30 steps the returns flatten; guidance scale trades prompt adherence against naturalness, so start near 5 and tune; partial noising is the unlock for editing, with strength controlling how much survives. For production, pick by constraint: open weights and ecosystem (SD 3.5, Flux 2 dev or klein), top quality via API (Flux 2 pro, Imagen 4, GPT Image), or instruction-based editing (Nano Banana). And measure with your own prompts; leaderboard ELO does not know your brand style guide.