Home / Learn / AI, explained simply

How AI image generators work

New to this? Read from the top. Already know the basics? Jump to Going deeper for intermediate and advanced detail.

An image generator turning random noise step by step into a clear picture of a circuit board
Beginner

Type "a robot soldering a circuit board, watercolour" and seconds later you get an image that never existed. Most of today's image generators use a technique called diffusion. The core idea is surprisingly simple: learn to remove noise.

Learning by destroying images

During training, the model is shown millions of images with captions. Each image is gradually covered in random noise, step by step, until nothing is left but static. The model's job is to look at a noisy image (plus its caption) and predict the noise that was added, so it can be removed.

Generating by removing noise

To make a new image, the process runs in reverse. The generator starts from pure random noise and asks the model, "what noise is here, given this caption?" It removes a little of that predicted noise, then repeats, often tens of times. Each step makes the picture slightly clearer, until a detailed image emerges that matches the prompt.

Because every image starts from different random noise, the same prompt gives a different picture each time. Fixing the random seed repeats the same result.

How the text steers the picture

The prompt is turned into numbers by a text encoder, a model trained to link words and images. At every step, those numbers guide the denoising towards images that match the description. This is why specific prompts ("close-up, soft morning light, shallow depth of field") give more control than vague ones.

Why the mistakes happen

  • Hands and counting: the model learns what images look like in general, not the rules of anatomy, so fingers and the number of objects can go wrong, though newer models have improved a lot.
  • Text in images: older models produced garbled lettering. Newer ones handle short text much better, but long text is still error-prone.
  • Bias: models reflect patterns in their training data, which can show up as stereotypes unless prompts or the system correct for it.

Using them responsibly

Check each service's terms for commercial use, avoid imitating identifiable people or living artists' distinctive styles without permission, and label AI-generated images where it matters. Many services now add invisible watermarks or provenance information to their output.

Intermediate and advanced

Going deeper

The forward process

In the standard formulation, popularised by Ho and colleagues' 2020 paper on denoising diffusion probabilistic models (DDPM), a noisy image at step t is a weighted mix of the original image and Gaussian noise. The weight on the image shrinks as t grows, according to a "noise schedule". This toy example shows the signal fading on a few pixel values:

import math, random

random.seed(1)
image = [0.9, 0.1, 0.5, 0.7]          # a tiny "image" of 4 pixels

def noisy(x, signal_fraction):
    a = math.sqrt(signal_fraction)
    b = math.sqrt(1 - signal_fraction)
    return [round(a * p + b * random.gauss(0, 1), 2) for p in x]

for frac in (1.0, 0.7, 0.3, 0.01):
    print(f"signal {frac:.2f}: {noisy(image, frac)}")
signal 1.00: [0.9, 0.1, 0.5, 0.7]
signal 0.70: [0.15, 0.1, -0.14, -0.2]
signal 0.30: [0.66, 0.17, 0.73, -0.38]
signal 0.01: [0.09, -0.05, -1.45, 0.61]

By the last line the original values are lost in the noise. Training teaches a neural network to predict the noise term at any step, so it can be subtracted.

Latent diffusion

Denoising full-resolution images is expensive. Latent diffusion, described by Rombach and colleagues in 2022 and used by Stable Diffusion, first compresses images with an autoencoder into a much smaller "latent" representation, runs diffusion there, and decodes the result back to pixels. This made high-quality generation practical on consumer GPUs.

Classifier-free guidance

To follow prompts more closely, the model makes two predictions at each step: one with the prompt and one without. The final prediction exaggerates the difference between them. The "guidance scale" controls how much: higher values follow the prompt more literally but can look harsh or over-saturated, and lower values are more varied and natural.

Fewer steps, newer architectures

Samplers and distillation techniques have cut the number of denoising steps needed from hundreds to a few dozen or even a handful. Many recent models replace the original U-Net denoiser with transformers, and some use related approaches such as flow matching. The principle of iteratively turning noise into an image remains the same.

Control and editing

Because generation works step by step, it's easy to steer. Inpainting regenerates only a masked region. Image-to-image starts from a partly noised existing picture instead of pure noise. Control methods add extra inputs such as edge maps or poses, so the output follows a given layout.

Sources

The main references behind this guide. Numbers we worked out ourselves, such as the code outputs, come from running the code.