Prediction as the Basic Learning Task

Chapter 1 showed that music can be represented as waveform samples, MIDI-style events, spectrograms, embeddings, or compressed audio tokens. Chapter 2 then showed how those representations are organised into training examples.

Now the model has something it can learn from.

For many music models, the basic task is surprisingly simple: look at the available context and predict what comes next.

The next item might be a note event, a time-shift event, one waveform sample, or an audio token selected from a neural codec codebook. The representation changes, but the learning pattern stays similar.

A small melody example

Suppose a training sequence contains:

C4, E4, G4, C5

The model receives the first three notes as context:

C4, E4, G4

Its task is to predict the fourth note. The correct target for this training step is C5.

The model does not normally answer with certainty. It may produce:

1. C5: 55%
2. A4: 20%
3. G4: 12%
4. E4: 8%
5. Other notes: 5%

Because the real target is C5, a high probability for C5 produces a smaller error than a low probability for C5.

Why not choose one answer immediately?

Music rarely has only one valid continuation. After a chord, several melodies may sound natural. A drum pattern may repeat, add a fill, or leave a rest. A singer may hold a vowel or begin the next word.

A probability distribution lets the model represent this uncertainty. Common continuations can receive high probability while less common but still plausible choices remain available.

The distribution also gives training a useful scale. Predicting the correct target with 60% probability is better than assigning it 2%, even when neither prediction is perfect.

Context can include more than earlier music

In an unconditional model, the earlier sequence may be the main context. In a conditioned model, extra information guides the probabilities.

For MusicGen, a prompt such as slow ambient piano with soft rain changes the distribution over upcoming audio tokens. Tokens associated with piano-like timbre, slow pacing, and quiet texture should become more likely than they would be under an unrelated prompt.

This is why Chapter 2, Section 3 placed so much emphasis on caption quality. Vague or unsupported captions teach weaker or incorrect relationships between language and sound.

Turning a whole sequence into many predictions

A sequence model can describe the probability of an entire sequence by breaking it into next-step probabilities:

P(x1, x2, ..., xT) = P(x1) × P(x2 | x1) × P(x3 | x1, x2) × ...

At position t, the model estimates:

P(xt | x1, ..., x(t-1), condition)

The full training track therefore creates many prediction tasks. Each position supplies a context and a target. This is far more efficient than treating the whole song as one indivisible answer.

From scores to probabilities

The model first produces one numerical score for every possible next token. These unnormalised scores are often called logits.

A softmax operation converts the logits into probabilities that add up to one. A token with a much larger logit receives a larger probability, but every candidate can retain some probability unless it is removed by a decoding rule.

During training, cross-entropy implementations commonly accept logits directly and compare them with the target token.

Learning from prediction error

After the model produces its scores, the training system reveals the real target from the dataset.

A common loss for token prediction is cross-entropy. It gives a larger penalty when the model assigns low probability to the correct target and a smaller penalty when the target already has high probability.

The training algorithm then adjusts the model's parameters in a direction that should reduce similar errors in the future. One correction is tiny. Repeated across many positions, examples, and training steps, these corrections shape the model's behaviour.

Choosing a token during generation

During generation, the correct next token is unknown because the model is creating a new sequence. A decoding rule must choose from the predicted distribution.

Greedy decoding selects the highest-probability token every time. Sampling draws from the distribution, so several likely tokens can be chosen across different runs.

MusicGen's official implementation supports greedy selection, temperature-based sampling, top-k sampling, and top-p sampling. These settings change how the distribution is used. They do not retrain the model.

Context changes the answer

The same three-note ending can lead to different predictions depending on what came before it.

A short context may show only C4, E4, G4. A longer context may reveal that the piece repeats an eight-bar theme, that the harmony recently moved to another key, or that the rhythm follows a call-and-response pattern.

Music Transformer was designed to improve access to long-range musical relationships. AudioLM also separated token roles so long-term structure and acoustic detail could both be modelled. Context is therefore not only about how much information is present. It is also about whether the architecture and representation make that information usable.

From one prediction to a full piece

One next-token prediction produces only one event. Music appears when the process repeats:

1. Predict a distribution.
2. Select a token.
3. Add it to the context.
4. Predict again.

The next section, Autoregressive Music Models, follows this loop from the first token to a complete musical sequence. It will also show why small early choices can reshape everything that follows.