Skip to content
Stable Diffusion, Released on 22 August 2022, Makes a 512 by 512 Image From Random Noise in About 50 Denoising Steps, Using an Idea First Published in a 2015 Paper Inspired by Physics
AI & Technology

Stable Diffusion, Released on 22 August 2022, Makes a 512 by 512 Image From Random Noise in About 50 Denoising Steps, Using an Idea First Published in a 2015 Paper Inspired by Physics

Illustration by tuput

English

Six papers between 2015 and 2022 took one idea, learning to reverse a slow addition of noise, from toy data to text prompts. Stability AI's release put the result on a graphics card with a stated 6.9 GB of memory, and Hugging Face's guide describes the 50-step process.

· · 10 min read

Stability AI announced the public release of Stable Diffusion on 22 August 2022 and published the weights, the model card and the code. The company said the final memory use should be 6.9 GB of graphics memory. Hugging Face, which worked on the release, put out a guide the same day. It describes what happens when a user types a prompt: the model starts from random noise in a small grid of numbers, removes noise from it about 50 times while reading the prompt, and decodes the result into a 512 by 512 pixel picture.

The method behind those 50 steps appeared in a March 2015 paper that borrowed from physics. This article follows it through six papers and the release itself. Every figure below is the authors’ own, measured on their own hardware and datasets. The papers use different datasets and settings, so no score is compared across them.

Destroy a picture slowly, then learn to undo it

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan and Surya Ganguli submitted their paper to arXiv on 12 March 2015. Its title names nonequilibrium thermodynamics, the physics of systems that are changing rather than settled. The abstract says the essential idea, inspired by non-equilibrium statistical physics, is to destroy the structure in data slowly through a forward diffusion process and then learn a reverse process that restores it.

Take the running example from Hugging Face’s guide, a photograph of an astronaut riding a horse. Add a pinch of random noise to the photograph, then another pinch, and keep going until nothing is left but static. The forward direction needs no learning. A neural network is trained on the opposite job: given a slightly noisier version of an image, produce a slightly cleaner one. To make something new, start from pure static and apply the network repeatedly.

The 2015 authors trained models with thousands of layers or time steps. They tested them on a two-dimensional swiss roll (a spiral-shaped cloud of points), a binary sequence, handwritten digits from MNIST and small natural images from the CIFAR-10, bark and dead-leaves datasets.

Remove a little noise, a thousand times

Jonathan Ho, Ajay Jain and Pieter Abbeel of the University of California, Berkeley posted their paper on 19 June 2020. They write that, to the best of their knowledge, nobody had yet shown that diffusion models could generate high-quality samples, and that their results do. On the CIFAR-10 collection of small images, generated without any label, they report an Inception score of 9.46 and a Frechet Inception Distance (FID, a score where lower means generated images are statistically closer to real ones) of 3.17, which they call state of the art at the time. On 256 by 256 LSUN, a set of photographs of bedrooms, churches and cats, they report sample quality similar to ProgressiveGAN, an earlier generator.

The recipe is short. The paper uses 1,000 steps and adds noise in amounts that grow in even increments from 0.0001 to 0.02 per step. In training, the method picks an image, picks a random step, adds the matching amount of noise and asks the network to predict the noise that was added. Sampling starts from pure noise. For each of the 1,000 steps the network predicts the noise, the sampler subtracts it and, except at the last step, adds back a small fresh dose of randomness.

The network is a U-Net, an image model that shrinks a picture to a coarse summary and then expands it back, with self-attention layers at 16 by 16 resolution. The step number is fed in using the sinusoidal position embedding from the Transformer paper, a design also behind the language models explained in how a large language model is trained. The CIFAR-10 model has 35.7 million parameters, the adjustable numbers inside a network, and trained to completion in 10.6 hours on a TPU v3-8, a Google accelerator the authors compare to eight V100 GPUs. Producing a batch of 128 images at 256 by 256 took 300 seconds.

Faster sampling, and the same idea as an equation

A thousand sequential passes is slow. Jiaming Song, Chenlin Meng and Stefano Ermon, in a paper of 6 October 2020, measured it: about 20 hours to sample 50,000 images of 32 by 32 pixels from a DDPM on an Nvidia 2080 Ti graphics card, against under a minute for a GAN, the older generator type that makes an image in one pass. For 256 by 256 images they put the figure at nearly 1,000 hours.

Their fix, denoising diffusion implicit models or DDIM, keeps the training exactly as it was and changes only how samples are drawn. The authors report samples comparable to the 1,000-step models in 20 to 100 steps, a speed-up of 10 to 50 times, and note that time per sample grows in a straight line with the number of steps.

Seven weeks later, on 26 November 2020, Yang Song and five co-authors, among them Sohl-Dickstein from the 2015 paper, recast the process as a stochastic differential equation (an equation with random motion built in) running from data to noise, with a matching reverse equation running from noise back to data. In their words: “Creating noise from data is easy; creating data from noise is generative modeling.” They write that the framework encapsulates earlier score-based and diffusion approaches, and report generating 1024 by 1024 images from a score-based model for the first time.

Making the picture follow the words

An image generator has to follow a prompt, and diffusion needed a way to do that. The earlier method, classifier guidance, steers sampling using a separate image classifier. Jonathan Ho and Tim Salimans, in a paper posted on 26 July 2022 (a short version had appeared at a NeurIPS 2021 workshop), showed it can be done without one.

In classifier-free guidance a single network is trained on two tasks at once. For a fraction of training examples the label or prompt is randomly replaced by an empty placeholder, so the network learns to denoise both with and without it. At sampling time the two predictions are combined, and a weight w sets how far the result leans toward the prompt. On 64 by 64 ImageNet the authors tried dropping the condition with probability 0.1, 0.2 and 0.5, and found 0.5 consistently worse than the other two. Turning the weight up raises the quality of each image and lowers the variety across images.

The method costs time. Each sampling step needs two passes through the network, one with the prompt and one without, which the authors name as a possible disadvantage.

Stable Diffusion uses this method. The v1-4 model card says its last stage of training dropped the text conditioning 10 percent of the time, “to improve classifier-free guidance sampling.” Hugging Face’s guide gives the user-facing control as guidance_scale, with a default of 7.5 and values from 7 to 8.5 described as usually good. It warns that very large values give images that may look good but are less varied.

Working on a smaller canvas

Pixel-space diffusion is expensive. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser and Björn Ommer open their paper of 20 December 2021 by saying the most powerful diffusion models often take hundreds of GPU days to train. They cite 150 to 1,000 V100 days for an earlier model, and about five days on one A100 to produce 50,000 samples from such a model.

Their remedy is to run diffusion in a latent space, meaning a compressed numerical summary of the picture. A separate autoencoder, a network pair that shrinks an image and expands it again, is trained first. Diffusion then happens on the small summary, and the decoder turns the finished summary back into pixels. The authors also add cross-attention layers to the U-Net so that text or other inputs can steer it.

On a single NVIDIA A100, training for 2 million steps on ImageNet with the same number of parameters, they found a gap of 38 FID points between diffusion in pixel space and diffusion at a compression factor of 8. The paper attributes the small-factor results to leaving most of the compression of imperceptible detail to the diffusion model itself.

Stable Diffusion uses a factor of 8. The model card says it maps an image of height H and width W with three colour channels to a latent of H/8 by W/8 with four channels. For a 512 by 512 image that is a 64 by 64 grid, which Hugging Face’s guide calls a spatial compression ratio of 8 x 8 = 64. By plain division, the 786,432 numbers in the image become 16,384 numbers in the latent, a 48-fold drop in what the network handles at each step.

The prompt is turned into numbers by the text encoder of CLIP (the ViT-L/14 version), which the model card says is fixed and pretrained, so it is not trained along with the rest. Hugging Face describes the output as a 77 by 768 block of numbers that the U-Net reads through cross-attention.

What shipped on 22 August 2022

Stability AI’s announcement credits the legal, ethics and technology teams at Hugging Face and engineers at CoreWeave, a cloud provider. The model came under a Creative ML OpenRAIL-M licence, which the company describes as permitting commercial and non-commercial use while making the user responsible for ethical and legal use and requiring the licence to accompany any distribution. A safety classifier was included by default to remove unwanted outputs. The recommended weights were v1.4, and the company said NVIDIA chips were recommended, with support for AMD and Apple M1 and M2 chips to come. The page the company published shows the date as “Aug 22” without a year, and Hugging Face’s guide on the same release carries the date 22 August 2022.

The model card names Robin Rombach and Patrick Esser as developers and says Stable Diffusion v1 was trained on LAION-2B(en) and subsets of it. The LAION-5B paper of 16 October 2022 describes 5.85 billion image-text pairs filtered with CLIP, of which 2.32 billion have English text. For hardware, the card lists 32 x 8 A100 GPUs, and for its emissions estimate 150,000 hours on A100 PCIe 40GB cards in an Amazon Web Services region in the eastern United States, which it puts at 11,250 kg of CO2 equivalent. These are the card’s own estimates.

The steps matter in practice. In Hugging Face’s guide, the default is 50 denoising steps, and a run with 15 kept the same overall structure but had problems in the astronaut’s suit and the horse’s form.

What the model card says it cannot do

Hugging Face’s guide says later versions, 1.5, 2 and 2.1, run with minimal code changes. This article describes v1-4 and the papers up to 2022 only. The v1-4 card, written by Rombach and Esser, is specific about the limits of this version. The model does not achieve perfect photorealism and cannot render legible text. It does not perform well on compositional tasks such as “a red cube on top of a blue sphere”, and faces and people may not be generated properly.

The card says the model was trained mainly with English captions and works less well in other languages, and that the autoencoder is lossy. It also reports that the data was not deduplicated, so some images that appear many times in the training set are memorised, and it points to a public search tool for checking. The training set, it adds, contains adult material, and the card calls that dataset not fit for product use without additional safety mechanisms. Whether training on such data is lawful is being decided in courts, as three 2026 rulings on AI training show. The card lists impersonation and disinformation among misuses; the rules for AI fakes in elections are covered in this look at the FCC, the EU and India.

The 2020 sampler used 1,000 steps for every image it made. Hugging Face’s guide of 22 August 2022 calls 50 steps usually sufficient for a high-quality 512 by 512 image.

Share
Copied!

Sources & further reading

  1. Sohl-Dickstein, Weiss, Maheswaranathan and Ganguli: Deep Unsupervised Learning using Nonequilibrium Thermodynamics (arXiv:1503.03585, 12 March 2015)
  2. Ho, Jain and Abbeel: Denoising Diffusion Probabilistic Models (arXiv:2006.11239, 19 June 2020; NeurIPS 2020)
  3. Song, Meng and Ermon: Denoising Diffusion Implicit Models (arXiv:2010.02502, 6 October 2020; ICLR 2021)
  4. Song, Sohl-Dickstein, Kingma, Kumar, Ermon and Poole: Score-Based Generative Modeling through Stochastic Differential Equations (arXiv:2011.13456, 26 November 2020; ICLR 2021)
  5. Ho and Salimans: Classifier-Free Diffusion Guidance (arXiv:2207.12598, 26 July 2022)
  6. Rombach, Blattmann, Lorenz, Esser and Ommer: High-Resolution Image Synthesis with Latent Diffusion Models (arXiv:2112.10752, 20 December 2021; CVPR 2022)
  7. Stability AI: Stable Diffusion Public Release (22 August)
  8. CompVis and Stability AI: Stable Diffusion v1-4 model card (Hugging Face)
  9. Patil, Cuenca, Lambert and von Platen, Hugging Face: Stable Diffusion with Diffusers (22 August 2022)
  10. Schuhmann et al.: LAION-5B, an open large-scale dataset for training next generation image-text models (arXiv:2210.08402, 16 October 2022)

Researched and written with the help of AI tools and edited for accuracy. Provided for general information and discussion only, not professional advice. See our editorial standards and disclaimer. Spotted an error? Tell us.

#diffusion models#stable diffusion#generative ai#image generation#latent diffusion#classifier-free guidance#ddpm#stability ai

Enjoyed this? Get the next one.

One good read at a time, straight to your inbox. No spam, unsubscribe anytime.

More in AI & Technology
Perseverance Drove 456 Metres Across Jezero Crater on 8 and 10 December 2025 on Waypoints That an AI Model Drafted and NASA Engineers Checked
A rover that sits on average 225 million kilometres from Earth followed a path sketched by a language model that had studied orbital images, and the rover team's simulator signed it off first. Here is who did what, in whose words, and what the numbers do and do not show.
Young Workers in AI-Exposed Jobs Are 19% Below Where They Would Be, a Stanford Study of Millions of Payroll Records Finds, and Its Authors Say That Does Not Prove AI Did It
Across millions of ADP payroll records, young workers in the jobs AI handles best are falling behind their peers. Whether AI is the reason is the one thing the authors decline to claim.
The Academy's Rules for the 99th Oscars Require Human-Authored Screenplays and Human-Performed Acting Roles, While AI Tools Neither Help Nor Harm a Nomination
The rulebook text, paragraph by paragraph, with the April 2025 wording it builds on, the Academy's own comments, and the machine-learning tools that effects studios and the Academy's Sci-Tech awards already recognise.
← all articles