Training From Scratch Versus Fine-Tuning

Section 1 explained the optimisation loop. The same loop can begin from two very different places.

A model can begin with randomly initialised parameters and learn the task from the available dataset. It can also begin with parameters that already encode useful musical and acoustic patterns.

The second option is usually called transfer learning or fine-tuning. Instead of relearning everything, the project adapts an existing model.

The practical decision is not simply train or do not train. There is a spectrum ranging from inference-only use to updating every model parameter.

Training from scratch

Training from scratch gives the project control over the model architecture, tokenizer, objective, context length, and conditioning design.

It also removes the behavioural assumptions inherited from a generator checkpoint. This can matter when the existing model uses an unsuitable representation or cannot support the required output.

The cost is that the new model must learn basic musical, acoustic, and conditioning relationships before it can specialise. That normally requires more data, more optimiser steps, more compute, and more failed experiments than adapting a capable pretrained model.

Continued pretraining

Continued pretraining keeps the same general task but changes the data distribution.

For a MusicGen-style language model, this could mean continuing codec-token prediction on a new licensed music collection while preserving the existing text-to-music architecture.

This approach can teach the model more about a domain without inventing a new output task. However, a narrow dataset can pull the model away from earlier capabilities if it dominates the updates.

Full fine-tuning

Full fine-tuning gives the adaptation process access to the complete pretrained model. It can change low-level, middle-level, and high-level representations.

This flexibility can be valuable when the new domain differs substantially from the original training distribution.

It also requires storing gradients and optimiser state for the full trainable model. A small or repetitive adaptation dataset can overfit, narrow the model, or damage capabilities that were useful before adaptation.

Parameter-efficient adaptation

Large pretrained models are expensive to duplicate and update for every new task.

Parameter-efficient methods keep the base checkpoint shared and store a much smaller task-specific update. This reduces trainable parameter count and can reduce optimiser memory and checkpoint storage.

The smaller update is still capable of overfitting or learning the wrong behaviour. Parameter efficiency describes how much is trained, not whether the adaptation is correct.

Adapter modules

The original adapter approach inserts small trainable layers into a pretrained network. The base parameters remain fixed, while each downstream task receives its own adapter parameters.

Adapters can be swapped while sharing one base model. They can also introduce extra operations during inference because the audio or token representation passes through the added modules.

Adapter capacity and placement matter. A module that is too small may not express the required change, while an unnecessarily large adapter reduces the efficiency advantage.

Low-rank adaptation

LoRA represents the adaptation to a large weight matrix through two smaller trainable matrices.

The original weight remains frozen. During the forward pass, the base transformation and the low-rank update are combined.

After training, the update can often be stored separately or merged into the base weight for deployment. The rank controls the capacity and parameter count of the update.

Adding a new conditioning module

A pretrained generator may already produce suitable music but lack the required input control.

Instead of retraining the complete model, a project can add an encoder, projection, cross-attention branch, or fusion module that translates the new input into features the generator can use.

The base model may remain frozen, or selected parts may be fine-tuned jointly with the new module. This strategy is useful when the desired change is mainly about how the model is controlled rather than what representation it generates.

Inference-only use

Many projects do not need adaptation.

Prompts, melodies, reference inputs, decoding settings, candidate ranking, editing tools, and post-processing may already provide enough control.

Inference-only use avoids training cost and preserves the released checkpoint exactly. It cannot teach the model a genuinely new output representation or reliably add behaviour that the model and its interface do not support.

Check compatibility before loading a checkpoint

A checkpoint contains parameter tensors with expected shapes and names. It is not a generic package that can initialise any new architecture.

Changing the model width, number of layers, conditioner, codec codebooks, vocabulary, or output representation can create incompatible parameters.

Meta's AudioCraft fine-tuning documentation warns that the configuration is not inherited automatically and that changed model layers require manual checkpoint handling.

Adaptation can weaken earlier abilities

Updating a model on a new task can reduce performance on capabilities learned before adaptation. This is commonly called catastrophic forgetting or catastrophic interference.

For music generation, a model adapted aggressively to one narrow collection might improve on that style while becoming less diverse, less responsive to other prompts, or less stable outside the adaptation domain.

The risk depends on the data, update size, strategy, and how much of the model is changed.

Test the decision on a small scale

Selecting a strategy is only the beginning. Before committing to a large training run, the project needs a small experiment with a clear hypothesis, controlled variables, a baseline, validation data, and stopping criteria.

The next section, Designing a Small Experiment, turns the selected adaptation strategy into a testable training plan.