How Diffusion Models Actually Work

By Michael Samuel Naeem · September 27, 2025 · 5 min read

GANs dominated generative image headlines until diffusion models made "type a sentence, get a coherent picture" boringly reliable. The idea is almost absurdly simple: destroy an image with noise, teach a network to undo the destruction, repeat until undoing random noise produces new images.

Forward process: structured destruction

Given image x₀, add Gaussian noise over T timesteps:

q(x_t | x_{t-1}) = N(x_t; √(1-β_t) x_{t-1}, β_t I)

β_t is noise schedule — small per step, cumulative noise grows. After enough steps, x_T ≈ pure Gaussian noise. No learnable parameters in forward process — fixed Markov chain.

Closed form: sample x_t directly from x₀ at any t without iterating.

Reverse process: learn to denoise

Goal: learn p(x_{t-1} | x_t) — reverse one noise step. True reverse is intractable; train neural network ε_θ(x_t, t) predicting noise added at step t (or predict x₀ or score — equivalent reparameterizations).

Training loss (simplified DDPM):

L = E_{t, x_0, ε} [ || ε - ε_θ(√(ᾱ_t) x_0 + √(1-ᾱ_t) ε, t) ||² ]

Sample random timestep t, noise image, predict noise, MSE against actual ε. Surprisingly stable compared to GAN min-max.

Sampling at inference

Start x_T ~ N(0, I), iterate t = T down to 1:

# Conceptual DDPM sampling loop
x = torch.randn(batch, channels, height, width)  # pure noise
for t in reversed(range(T)):
    predicted_noise = model(x, t)
    x = denoise_step(x, predicted_noise, t, schedule)
return x  # generated image

T=1000 steps originally — slow. DDIM, DPM-Solver, consistency models reduce to 10–50 steps with quality tradeoffs.

Score matching connection

Noise prediction relates to score function ∇_x log p(x) — direction pointing toward higher data density. Denoising steps follow score toward plausible images. Unified view links diffusion to energy-based models and stochastic differential equations (score-based generative modeling).

U-Net architecture

Image (or latent) noise predictors typically use U-Net — encoder-decoder with skip connections, time embedding injected via additive or adaptive normalization:

# Pseudocode structure
class DenoiseUNet(nn.Module):
    def forward(self, x, t, context=None):
        t_emb = time_embedding(t)
        h = self.down_blocks(x, t_emb)
        if context is not None:  # text cross-attention
            h = self.attention(h, context)
        return self.up_blocks(h, t_emb)

Time t tells network how much noise remains — different behavior at t=900 vs t=10.

Latent diffusion (Stable Diffusion)

Pixel-space diffusion on 512×512×3 is expensive. LDM encodes images to 64×64×4 latents via VAE, diffuses in latent space, decodes with VAE decoder. Text prompts encoded by CLIP text encoder, fed via cross-attention layers.

Text → CLIP encoder → context vectors
Random latent z_T → U-Net denoise (conditioned on text) → z_0 → VAE decode → image

Same framework, smaller tensors — democratized local inference.

Classifier-free guidance

Training randomly drops text conditioning; inference blends conditional and unconditional noise predictions:

ε_guided = ε_uncond + w · (ε_cond - ε_uncond)

w > 1 sharpens prompt adherence — higher w can oversaturate or artifact.

Why diffusion won mindshare

• Stable training — no discriminator collapse
• Mode coverage — captures diverse data distribution
• Controllable conditioning — text, inpainting, ControlNet edges/depth
• Quality ceiling — competitive with best GANs on fidelity

Costs: sampling latency, compute for training large models, copyright/training data debates outside scope here.

Practical implications for builders

Fine-tune with LoRA on small GPU rather than full U-Net. Distilled schedulers for production latency. Evaluate with FID/CLIP score plus human review — metrics miss failure modes.

Understanding forward-reverse framing helps debug "blurry" (too few steps), "ignores prompt" (guidance too low), "burnt" (guidance too high).

Sampling schedulers and step count

Different schedulers trade quality for speed:

from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler

pipe = StableDiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0")
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)

image = pipe(
    prompt="A photorealistic portrait of a robot engineer",
    num_inference_steps=25,      # DPM++ sweet spot
    guidance_scale=7.5,          # CFG weight
    width=1024, height=1024,
).images[0]

Start with DPM++ at 25 steps. Reduce to 15–20 for latency-sensitive paths; increase to 40+ for quality-critical generation.

ControlNet and conditioning extensions

Extend base diffusion with structural control:

from diffusers import StableDiffusionControlNetPipeline, ControlNetModel

controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny")
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", controlnet=controlnet
)

# Canny edge image guides generation structure
image = pipe(
    prompt="modern office building",
    image=canny_edge_image,
    num_inference_steps=25,
).images[0]

ControlNet variants: Canny (edges), Depth (3D structure), OpenPose (human poses), Scribble (rough sketches). Enables precise layout control beyond text prompts.

Production deployment considerations

GENERATION_DEFAULTS = {
    "num_inference_steps": 25,
    "guidance_scale": 7.5,
    "width": 1024,
    "height": 1024,
    "negative_prompt": "blurry, low quality, distorted, watermark",
}

# Safety filter before returning to user
from diffusers.pipelines.stable_diffusion import StableDiffusionSafetyChecker
# NSFW detection — block or blur before CDN upload

Pin model checkpoint version in production container. Model upgrades change output style — communicate to users before upgrading.

Failure modes

• Too few steps (<15) — blurry, incomplete details
• guidance_scale too high (>12) — oversaturated, artifact-heavy
• guidance_scale too low (<4) — ignores prompt
• Unpinned model version — output style changes silently on checkpoint update
• No safety filter — NSFW content reaches users

Production checklist

• DPM++ scheduler at 25 steps for production default
• guidance_scale 7–8 for balanced prompt adherence
• Model checkpoint version pinned in deployment container
• Safety filter (NSFW detection) before CDN upload
• Negative prompt configured for quality baseline
• Identical request caching by (prompt, seed, model) hash

Resources

• Denoising Diffusion Probabilistic Models (Ho et al., 2020)
• Stable Diffusion paper (Rombach et al.)
• Hugging Face Diffusers library
• Lilian Weng — What are diffusion models?
• Annotated Diffusion Model (Alammar)