Hierarchical Generation

The previous section showed why generating every token in one long autoregressive stream can be difficult. A model may need to remember the shape of a song while also predicting tiny details in the sound.

Hierarchical generation divides those jobs across several levels.

A high-level stage can work with a short, compressed sequence that preserves broad content or structure. Later stages can use that sequence as a guide while adding timing, timbre, texture, and waveform detail.

Instead of asking one model to solve everything at once, the system moves from a rough musical representation toward a finished recording.

Why use more than one level?

A second of raw audio can contain tens of thousands of waveform samples. Even compressed codec representations may contain many token streams per second.

Long-range musical structure changes much more slowly. A phrase, motif, chord movement, or vocal line can extend across seconds or minutes.

A hierarchy lets the early stage work with fewer tokens. The model can focus on relationships across a longer span without also deciding every fine acoustic detail. Later stages receive the earlier result and fill in information at a higher resolution.

AudioLM: semantic, coarse, and fine stages

AudioLM uses three generation stages:

1. A semantic model generates tokens intended to preserve long-term structure.
2. A coarse acoustic model generates the first codec levels while following the semantic sequence.
3. A fine acoustic model predicts the remaining codec levels and improves local sound quality.

The complete acoustic token set is then passed through the SoundStream decoder to reconstruct the waveform.

Why can the fine stage work locally?

AudioLM assumes that fine acoustic details depend mainly on the coarse acoustic representation nearby. Once the broad content and rough sound are known, the fine stage does not need to solve the full long-range musical problem again.

The paper therefore processes fine acoustic tokens in short, non-overlapping chunks. This reduces the sequence length handled by the fine model and lets acoustic refinement scale separately from the total duration.

MusicLM adds text conditioning

MusicLM extends the hierarchical audio-language approach to text-to-music generation.

A text description is mapped into a shared music-language embedding space. The semantic stage then generates semantic audio tokens under that condition. Coarse and fine acoustic stages turn the semantic sequence into SoundStream tokens, which are decoded into 24 kHz audio.

The hierarchy separates two forms of control: the text guides what the music should be like, while the later acoustic stages determine how the generated result is rendered in sound.

Jukebox: several audio resolutions

Jukebox also uses a hierarchy, but its levels are not the same as AudioLM's semantic and acoustic split.

Jukebox trains audio autoencoders at several compression rates. The top-level codes are highly compressed and cover a longer span with fewer tokens. Middle and bottom levels carry progressively more detail.

An autoregressive prior generates the top-level codes. Upsampling models then generate the middle and bottom codes while conditioning on the coarser levels. A decoder reconstructs the final waveform.

Early-stage errors become inherited constraints

A later stage cannot freely replace the plan it receives. Its job is normally to add information while remaining compatible with the earlier sequence.

If the semantic stage loses the intended melody, the acoustic stages may render the wrong melody convincingly. If the top-level Jukebox prior creates an awkward structure, the upsamplers can add detail but may not repair the structure.

Hierarchy therefore changes the form of error accumulation discussed in Section 2. Instead of one token affecting later tokens in the same stream, an entire generated level can constrain every model below it.

The high-level representation is a bottleneck

A compressed level cannot preserve everything. This is intentional. The system needs a shorter sequence that keeps information useful for later stages.

The risk is that the bottleneck removes something the final task needs. A semantic representation may preserve broad content but weaken precise timbre. A highly compressed audio code may retain style and rhythm while losing small production details.

Choosing the representation is therefore part of the model design, not merely a storage decision.

Does every level need to be autoregressive?

No. A hierarchy describes how representations depend on one another. It does not require every stage to use the same generation method.

AudioLM generates its semantic, coarse acoustic, and fine acoustic tokens autoregressively. SoundStorm keeps the semantic conditioning stage but replaces AudioLM's two autoregressive acoustic stages with confidence-based parallel decoding.

This produced the same reported audio quality as the AudioLM acoustic generator while greatly reducing generation time in the paper's experiments.

From predicting tokens to refining noise

AudioLM, MusicLM, and Jukebox organise generation as a cascade of discrete-token predictions. Another family of models starts from noise or a corrupted representation and repeatedly refines it toward music.

The next section, Diffusion Models, explains this denoising process and why it can generate many parts of a representation in parallel rather than committing to one left-to-right token path.