Learn how audio, captions, tags, lyrics, MIDI, and other fields are combined into training records for different AI music tasks.
About 10 minutes
Guiding question
What information belongs in one training example?
By the end, you’ll be able to
Identify the main parts of a music training example.
Match fields such as captions, tags, lyrics, and MIDI to the task a model is learning.
Explain the difference between an input, a target, conditioning data, metadata, and provenance.
A music dataset can contain thousands or millions of files, but a model learns from smaller units called training examples.
A training example is one organised record presented to the model. It might contain a ten-second audio clip and a written caption. It might contain a piano recording and the matching MIDI performance. It might contain a song, its lyrics, and labels for genre or artist.
There is no single correct format. The record should contain the information needed for the task the model is learning. A text-to-music model needs different fields from a transcription model, even when both use audio.
The main parts of a training record
Swipe sideways to view the full comparison
Part
What it means
Music example
Input
The information the model receives
A spectrogram given to a transcription model
Target
The answer the model is trained to predict
The matching MIDI note sequence
Conditioning data
Information used to guide what should be generated
A caption, melody, lyrics, artist label, or genre label
Metadata
Descriptive information used to organise, filter, split, or analyse records
Duration, source, performer, year, or recording format
Provenance
Information about where the record came from and how it may be used
Source URL, licence, contributor, and processing history
Conditioning data tells a generative model what kind of result it should create. It does not have to describe every detail. It gives the model useful direction.
For example:
A text caption can describe the mood, instruments, tempo, or style.
A melody can guide the tune while the model creates a new arrangement.
Lyrics can guide the words that should be sung.
Genre or artist labels can steer broad musical and vocal characteristics.
MusicGen supports text and melody conditioning. Jukebox used artist, genre, and unaligned lyrics as conditioning signals. The choice of condition depends on the type of control the model is designed to offer.
The task decides the record structure
Swipe sideways to view the full comparison
Task
Possible input or condition
Possible target
Useful fields
Text-to-music generation
Text caption
Audio or audio tokens
Audio, caption, duration, language, source
Music tagging
Audio clip
One or more labels
Audio, genre, instrument, mood, sound-event tags
Audio transcription
Audio recording
MIDI or note events
Audio, aligned MIDI, instrument, timing
Symbolic music generation
Earlier note events or a prompt sequence
Later note events
MIDI, tempo, time signature, instrument
Lyrics-conditioned song generation
Lyrics and optional labels
Sung music audio or audio tokens
Audio, lyrics, artist, genre, timing
Melody-conditioned music generation
Melody and optional text
Arranged music audio
Audio, melody, caption, tempo
In practice
Example: MusicCaps
MusicCaps contains 5,521 music examples. Each record includes the source video ID, the start and end time of the ten-second segment, AudioSet labels, an aspect list, and a free-text caption written by a musician. This format is useful for learning relationships between music audio and natural-language descriptions.
musiccaps-style-record.json · A simplified example inspired by the fields in MusicCaps.
Swipe sideways to view code
{
"audio": "clip-0042.wav",
"startSeconds": 30,
"endSeconds": 40,
"tags": ["piano", "female vocal", "pop"],
"caption": "A bright pop track with piano, a clear female vocal, and a steady drum beat."
}
In practice
Example: AudioSet
AudioSet was created for audio event recognition. Its records connect short audio excerpts with one or more labels from a sound-event ontology. A clip can contain several labels because real recordings often contain several sounds at once. This is different from a caption dataset because the target is a controlled set of categories rather than a free-form description.
Labels can describe a whole clip or a precise moment
Some datasets only say that a sound appears somewhere in the clip. These are often called weak labels because they do not state exactly when the event begins and ends.
Other records contain precise timing. DALI, for example, includes lyrics and vocal melody notes aligned with the audio at several levels of detail. MAESTRO aligns piano audio with MIDI note and pedal events.
Precise alignment supports tasks such as transcription, lyric timing, and note-level generation. It is also harder and more expensive to create accurately.
Examples in practice
Example: DALI
DALI contains 5,358 audio tracks with time-aligned lyrics and vocal melody notes. The annotations are available at four levels of detail. A record can therefore connect the recording with words, lines, note pitches, and timing. This makes it useful for tasks involving singing voice, lyrics, melody, and alignment.
Example: MAESTRO
MAESTRO pairs expressive piano audio with MIDI captured during the same performance. The official dataset includes key strike velocities, pedal positions, composer, title, and performance year. The audio and MIDI are aligned to about 3 milliseconds, which makes the records useful for transcription, symbolic composition, and turning MIDI performances back into audio.
A record is not always the same as a file
One audio file can produce several training examples. A long song may be divided into short segments so each record fits the model's context window. Each segment can inherit song-level metadata while receiving its own timing, caption, labels, or aligned events.
The reverse can also happen. One training example can refer to several files, such as an audio file, a MIDI file, a lyrics file, and a metadata record.
What matters is that the fields can be joined correctly and that the model receives the right information together.
A useful training record should answer five questions
1
What is the task?
State what the model should predict, generate, classify, or align.
2
What is the input?
Identify the audio, text, MIDI, lyrics, or other data the model receives.
3
What is the target?
Store the expected answer in a form the model can learn from.
4
Are the fields aligned and reliable?
Check that captions, notes, lyrics, and timing refer to the correct audio.
5
Can the record be traced?
Keep source, licence, consent, and processing information separate from the learning fields but linked to the record.
Interactive lesson
Build a Music Training Example
Try it: Select a task preset. Drag fields into Input, Condition, Target, Metadata, or Provenance. The builder will explain which fields are required, optional, misplaced, or unnecessary for that task.
A training example is one structured record. The model task decides which fields act as inputs, targets, or conditioning data. Metadata helps organise and analyse the record, while provenance tracks where it came from and how it may be used.
Check your understanding
Ready for a quick check?
Test whether you can identify the parts of a training record and choose fields for different music tasks.
Ready to continue?
Save this section to your account and continue from any device.