Overfitting and Memorisation

Section 3 designed a small melody experiment with separate training, validation, and test data. Those divisions matter because a model can become extremely good at the examples it has already seen without learning behaviour that transfers to new music.

This section separates several problems that are often grouped together:

1. Overfitting: performance on training data improves more than performance on relevant unseen data.
2. Leakage: held-out information enters training or model selection.
3. Duplication: the same or nearly the same material appears multiple times.
4. Memorisation: the trained model retains information specific to individual training examples.
5. Replication: generated output reproduces a training example or part of one under a chosen similarity criterion.

These problems interact, but they are not interchangeable.

Reading training and validation curves

Training loss is calculated on examples used for optimisation. Validation loss is calculated on held-out examples without parameter updates.

A common overfitting pattern is:

1. Training and validation loss both fall.
2. Validation improvement slows.
3. Training loss continues falling.
4. Validation loss stops improving or begins rising.

The checkpoint near the best validation result may generalise better than the final checkpoint. The curve is evidence, not a complete diagnosis. Its interpretation depends on the loss calculation, data distributions, model mode, and uncertainty in the validation estimate.

Validation can fail before the model does

A validation score is trustworthy only when the validation process represents the intended use.

Common failures include:

1. Overlapping windows from one song across splits
2. Alternate performances or arrangements of one composition across splits
3. Duplicate files across partitions
4. Normalisation statistics calculated using all data
5. Validation prompts taken from training captions
6. A validation set that is too small or unrepresentative
7. Different preprocessing or masking in training and validation
8. Repeated tuning against the validation set until it becomes indirectly overfit

A rising validation loss may indicate model overfitting, but a broken validation design can create the same appearance.

Duplicates change how often the model sees an example

If one melody appears once and another appears fifty times through copies, excerpts, transpositions, remasters, or metadata variants, the second receives far more optimisation exposure.

Exact duplicates are the simplest case. Near duplicates can include:

1. The same audio under different filenames
2. A remastered or compressed version
3. A shortened excerpt
4. A MIDI transposition
5. Alternate performances of the same composition
6. A loop copied into several commercial packs
7. Several captions pointing to the same audio

Research in language modelling has shown a strong relationship between repeated exposure and regeneration of memorised sequences. Audio research has also found substantial duplication inside a commonly used training dataset and observed partial replication in trained diffusion models. These findings support duplicate auditing, but modality-specific measurements remain necessary.

Real-world example: MusicGen's memorisation experiment

The MusicGen researchers tested whether the model could continue extracts from its own training set. They sampled 20,000 training examples, supplied increasing lengths of original EnCodec-token prompts, and generated five seconds using greedy decoding.

They measured:

1. Exact matches: every generated token in the tested coarse codebook matched the source continuation.
2. Partial matches: at least 80 percent of the generated tokens matched the source continuation.

This is a strong targeted test of continuation from known training prefixes. It does not measure every possible form of musical similarity, and it examines one coarse codec stream rather than every detail in the waveform.

Music can match in several different ways

Two tracks can share one property while differing in others. A generated track might reproduce a melody but change its instrumentation, or preserve production texture while changing harmony.

Possible comparison levels include:

1. Exact token sequence
2. Pitch sequence
3. Transposition-invariant intervals
4. Rhythm and onset pattern
5. Chord progression
6. Melodic contour
7. Chroma or tonal content
8. Spectrogram texture
9. Audio fingerprint
10. Learned semantic embedding
11. Production-style embedding

No single metric captures every musically or legally relevant form of similarity.

Reducing overfitting

No single intervention prevents every kind of overfitting or memorisation. Useful options include:

1. More representative and less duplicated data
2. Smaller model or adapter capacity
3. Fewer training steps
4. Earlier checkpoint selection
5. Weight decay
6. Dropout
7. Data augmentation that preserves the task
8. Mixing broader rehearsal data during narrow fine-tuning
9. Freezing more pretrained parameters
10. Label smoothing for suitable classification objectives
11. Explicit replication tests during evaluation

Every intervention changes the learning problem. For example, aggressive pitch transposition may be suitable for key-invariant melody learning but unsuitable when absolute register is important.

From statistical risk to physical cost

Reducing overfitting may require more data, more model comparisons, several checkpoints, repeated seeds, and large nearest-neighbour indexes. Those safeguards consume storage, memory, compute, and engineering time.

The next section, Compute and Infrastructure, explains the hardware and systems required to train, evaluate, checkpoint, and reproduce music models at different scales.