Conditioning and Control

The previous sections explained how a model generates music. Conditioning explains how a user, dataset, or external signal directs that process.

Without conditioning, a model samples from the overall patterns it learned during training. With conditioning, it changes the probabilities or denoising updates so the output is more likely to follow a prompt, melody, chord sequence, rhythm, duration, lyric sheet, or reference recording.

Conditioning does not usually dictate every detail. It narrows the space of possible results.

Global and local conditioning

A global condition applies broadly to the complete output. A prompt such as warm acoustic folk with intimate vocals describes the overall result.

A local condition changes over time. A chord sequence may specify one chord for the first bar and another for the second. A drum condition may place individual hits at precise moments.

Global conditions are easier to attach to an entire generation. Local controls require alignment between the condition and the generated timeline.

Text prompts

Text is a flexible interface because it can combine many ideas in one description:

Slow cinematic piano with restrained strings, gradually becoming tense and percussive.

A text encoder converts the prompt into numerical features. The generative model then uses those features while predicting tokens or refining a latent.

Text works well for broad musical direction, but ordinary language is often ambiguous. Words such as dark, epic, or vintage do not correspond to one fixed acoustic result.

Melody conditioning

A melody condition gives the model information about pitch movement over time.

The model does not necessarily copy the original recording. It may preserve the melodic contour while changing the instruments, harmony, texture, or production.

MusicLM demonstrated that a hummed or whistled melody could be transformed according to a text description. MusicGen can also combine text with melodic features extracted from audio.

Harmony, rhythm, tempo, and key

Text can mention musical attributes, but specialised controls can express them more directly.

A chord condition can specify when harmonies should change. A rhythm condition can identify beat or drum events. Tempo can be represented as a BPM value, while key can be supplied as a symbolic label.

These controls are useful when a general prompt is not precise enough for arranging, accompaniment, or synchronisation tasks.

Lyrics conditioning

Lyrics provide the words that a singing model should produce. They do not automatically specify the melody, rhythm, pronunciation, voice, or placement of each syllable.

A system may receive unaligned lyrics, where only the text order is known, or aligned lyrics, where words, phonemes, or syllables are connected with time positions.

More precise alignment can improve control, but it requires more detailed training data.

Duration and timing

A request for 30 seconds of audio is different from a request for 90 seconds. The model needs to know not only what to generate, but how much temporal space is available.

Stable Audio introduced timing conditioning using the requested duration and the position of the sampled training segment inside its source recording. This lets the model generate variable-length outputs and learn that beginnings, middles, and endings may have different musical roles.

Reference audio

A reference recording can guide several different properties. One system may use it as a melody. Another may extract style, timbre, instrumentation, rhythm, or continuation context.

The phrase audio conditioning therefore does not describe one fixed task. The model architecture and feature extractor decide which information is preserved and which information is discarded.

When conditions conflict

Suppose the controls request:

1. A slow ambient texture
2. A drum pattern at 150 BPM
3. A gentle lullaby melody
4. Aggressive distorted production
5. A duration of eight seconds

The system cannot satisfy every interpretation equally. Some conditions may dominate, while others are weakened or ignored.

Conflict is especially likely when a global text prompt and a local symbolic condition describe incompatible musical directions.

Adjusting control strength

Diffusion and related models can use classifier-free guidance to strengthen a condition during generation.

The system compares a conditional prediction with an unconditional prediction and combines them. A larger guidance scale can increase prompt adherence, but it may reduce diversity or introduce artifacts.

MusicGen-Style extends this idea with separate text and audio conditions so their influence can be balanced.

Generation and editing

Conditioning is not limited to creating music from an empty starting point. Existing audio can become part of the condition.

A model can continue a recording, extend it before or after the known segment, replace a selected region, or create variations while preserving parts of the original.

These tasks differ mainly in which parts of the representation are fixed and which parts the model is allowed to generate.

Continuation

A continuation model treats the existing opening as context. The generated material should connect naturally with its rhythm, harmony, timbre, and production.

In an autoregressive model, the prompt tokens become the initial prefix. In a masked-token model, the known prefix remains visible while later positions are generated.

Continuation becomes harder when the supplied audio ends during an unstable chord, phrase, or transition.

Evaluating control

A sample can sound good while ignoring the requested condition. It can also follow the chord sequence accurately while sounding unnatural.

Control evaluation should therefore separate:

1. Overall audio quality
2. Adherence to the text
3. Melody similarity
4. Chord accuracy
5. Tempo accuracy
6. Lyric intelligibility
7. Duration accuracy
8. Boundary quality for edits
9. Similarity to a reference
10. Difference from the reference

The last two can conflict. A style-conditioned output should resemble the reference in useful ways without simply reproducing it.

Control becomes harder as the song grows

A model may follow a prompt, melody, and chord sequence for a short clip while losing direction over several minutes. The longer the output becomes, the more the system must manage repetition, development, transitions, memory, and ending structure.

The next section, Why Long Songs Are Difficult, explains why extending duration creates a different problem from generating a convincing short passage.