Lunar Boom Learning
Chapter 3
Chapter 3
How an AI Model Generates Music
From probability and token sequences to diffusion, control, and complete song structure
Chapter 1 showed how music becomes machine-readable representations, and Chapter 2 showed how those representations become training examples. This chapter follows the next step: how a trained model turns context, probability, noise, and control signals into new music.
What you’ll learn
- Explain how context becomes a probability distribution over possible musical events.
- Describe how autoregressive models build music through dependent token decisions.
- Explain how hierarchical systems separate long-range content from acoustic detail.
- Describe how diffusion models generate audio through repeated denoising.
- Compare text, melody, harmony, rhythm, lyrics, timing, and reference-audio conditioning.
- Distinguish generation, continuation, outpainting, inpainting, and variation.
- Explain why long-form coherence requires memory, repetition with variation, structural planning, transitions, and deliberate endings.
Questions to carry with you
How does a model decide what musical event or audio representation should come next?
How do autoregressive and diffusion models generate music differently?
Why do some systems separate semantic structure, coarse sound, and fine acoustic detail?
How do prompts, melodies, chords, lyrics, timing, and reference audio guide generation?
What happens when several conditioning signals conflict?
Why can a convincing short clip become repetitive or lose direction over several minutes?
How can a model preserve motifs, section identity, and song form across a complete track?
Sections
Section 3.1 · 12 min
Prediction as the Basic Learning Task
Learn how a music model uses context to assign probabilities to possible next tokens, compares its prediction with the real target, and improves through repeated prediction errors.
Section 3.2 · 13 min
Autoregressive Music Models
Learn how autoregressive models repeatedly predict, select, and append tokens, why every choice changes the next context, and where sequential generation becomes difficult.
Section 3.3 · 13 min
Hierarchical Generation
Learn how multi-stage music systems generate compressed structural tokens first, then add coarse and fine acoustic information until the final waveform can be reconstructed.
Section 3.4 · 13 min
Diffusion Models
Learn how diffusion models corrupt training examples with noise, learn to reverse that corruption, and generate music by gradually refining a noisy waveform, spectrogram, or latent representation.
Section 3.5 · 14 min
Conditioning and Control
Learn how text, melody, harmony, rhythm, lyrics, timing, and reference audio guide music generation, and how continuation and inpainting turn generation into editing.
Section 3.6 · 14 min
Why Long Songs Are Difficult
Learn why extending a convincing music clip into a complete song requires memory, repetition with variation, section planning, stable identity, convincing transitions, and a meaningful ending.
Chapter recap
Chapter takeaway
Chapter 3 follows the full generation path. It begins with one probabilistic prediction, turns that prediction into an autoregressive sequence, separates structure and sound through hierarchy, contrasts token generation with diffusion, adds user control through conditioning, and ends with the unresolved challenge of organising complete songs.
A generative model does not choose music with certainty. It assigns probabilities or denoising directions based on context and conditions.
Autoregressive models build music through a chain of dependent token choices, so early decisions influence everything that follows.
Hierarchical systems divide long-range content and acoustic detail across different representations or model stages.
Diffusion models generate by repeatedly refining noise in waveform, spectrogram, or latent space rather than selecting one left-to-right token path.
Conditioning can guide text, melody, harmony, rhythm, lyrics, timing, style, continuation, and editing, but conflicting controls must be resolved.
Generating several minutes of audio is not the same as generating a coherent song with recognisable sections, purposeful development, and a convincing ending.
Chapter 3 explained what happens during generation. Chapter 4, Training and Fine-Tuning a Music Model, examines how the model's parameters learn these behaviours, how objectives and optimisation shape the result, and how existing models can be adapted to new musical tasks.
Check your understanding
Chapter 3 Assessment
Review prediction, autoregressive generation, hierarchical modelling, diffusion, conditioning, editing, and long-form musical structure.