Diffusion Models

The previous sections described models that generate a sequence by choosing one token after another. Diffusion models take a different route.

They begin with a noisy representation and repeatedly revise it. At first, the representation contains almost no recognisable music. After several denoising steps, broad shapes begin to appear. Later steps refine rhythm, timbre, texture, and local detail.

The model does not remove real microphone noise from an existing recording. It reverses an artificial corruption process created for training.

Teaching the model with artificial corruption

Begin with a clean training representation called x0. This could be a waveform, spectrogram, or compressed latent.

The forward process adds a controlled amount of Gaussian noise:

x0 → x1 → x2 → ... → xT

Early versions remain close to the original. Later versions are increasingly corrupted. At a sufficiently noisy endpoint, xT resembles random noise much more than music.

Because the training system added the noise itself, it knows how much corruption was applied.

The schedule controls the difficulty

The model must learn from examples containing many levels of corruption. A nearly clean example requires small corrections. A heavily corrupted example requires the model to infer broad structure from weak evidence.

The noise schedule determines which corruption levels exist and how they are distributed across timesteps. A poor schedule can make useful learning or efficient sampling harder.

Later diffusion research introduced alternative schedules and samplers that reduce the number of model evaluations needed for generation.

What the model learns

Training does not always require running through the full corruption chain one step at a time.

A typical training step can:

1. Select a clean example.
2. Select a random diffusion timestep.
3. Add the amount of noise associated with that timestep.
4. Give the noisy representation and timestep to the model.
5. Ask the model to predict the added noise or another equivalent denoising target.
6. Compare the prediction with the known target.

The model repeats this task across many recordings and noise levels.

Generating from noise

Once training is complete, the model no longer needs a clean example.

Generation begins with random noise called xT. The model receives that noisy state, the current timestep, and any condition such as a text prompt. It predicts how the representation should change.

The sampler uses that prediction to create a slightly cleaner state. The process repeats:

xT → x(T-1) → ... → x1 → x0

The final x0 is decoded or played as the generated audio.

More steps are not automatically better

Early diffusion systems often used long sampling chains. Later samplers showed that useful outputs could be produced with fewer evaluations.

Reducing the step count can make generation faster, but an aggressive reduction may weaken detail, stability, or prompt adherence. The best setting depends on the trained model, sampler, representation, and task.

A model trained with one noise process can also support more than one sampling procedure.

Where can diffusion happen?

Diffusion is a method for modelling a distribution. It does not require one specific music representation.

An audio model can diffuse:

1. Raw waveform values
2. Spectrogram values
3. Compressed continuous latents
4. An intermediate representation inside a cascade

The choice changes the size of the problem, what information is represented directly, and which decoder is needed afterward.

Why generate in a latent space?

A waveform sampled at 44.1 kHz contains 44,100 sample values per channel for every second of audio. A compressed latent can represent the same duration with far fewer positions.

Diffusion in that latent space reduces the size of each model input and denoising update. The model can spend more capacity on musically useful variation instead of reconstructing every waveform sample directly.

The trade-off is that the autoencoder becomes part of the final quality limit. If the latent representation loses a transient, stereo cue, or high-frequency texture, the diffusion model cannot fully recover it later.

Real-world example: AudioLDM

AudioLDM performs diffusion in a compressed audio representation rather than directly in waveform space.

Its autoencoder represents mel-spectrogram information in a latent space. The diffusion model generates that latent, after which the latent is decoded to a mel-spectrogram and converted into a waveform.

AudioLDM also uses CLAP, a shared audio-language embedding model. During training, audio embeddings can provide the condition. During text-to-audio sampling, aligned text embeddings guide generation.

Real-world example: Noise2Music

Noise2Music uses a cascade of diffusion models for text-conditioned music generation.

A generator first creates an intermediate audio representation from the text condition. A cascader then generates higher-fidelity audio while conditioning on that intermediate result and, depending on the configuration, the text.

The system was designed to generate 30-second music clips. It demonstrates that diffusion can be combined with the hierarchical design discussed in Section 3.

Real-world example: Stable Audio

The Stable Audio research system uses a diffusion Transformer operating on a highly downsampled continuous audio latent.

Its conditioning includes text and timing information. Timing lets the model receive a requested duration and position within the training audio, rather than treating every clip as an unstructured fixed-length segment.

The paper reports generation of music lasting up to 4 minutes and 45 seconds using a latent rate of 21.5 positions per second.

Guiding the denoising path

Without a condition, the model learns to generate examples from its overall data distribution. With a condition, the model learns how the denoising direction changes for a prompt, class, melody, timing signal, or another input.

A prompt such as slow ambient strings with distant choir should make some denoising paths more likely than others. The condition does not select one finished recording. It influences the repeated updates that shape the result.

Directing the result

Diffusion explains how a noisy representation becomes music, but it does not yet explain how a user controls that process.

The next section, Conditioning and Control, follows text prompts, melody inputs, timing signals, audio references, and guidance scales through the model. It also examines what happens when several conditions point in different directions.