Designing a Small Experiment

Training a small model is most useful when the experiment answers one clear question.

Can a small Transformer generate convincing music? is too broad. It leaves the representation, dataset, model size, musical domain, output length, comparison method, and meaning of convincing undefined.

A more useful question is:

Does a two-layer Transformer predict the next event in sixteen-bar monophonic melodies better than a first-order Markov baseline when both use the same training data and tokenisation?

That question is narrow enough to implement, compare, diagnose, and repeat.

Begin with one model behaviour

Choose one output and one task.

A manageable first task might be:

1. Continue a monophonic melody
2. Predict the next symbolic event
3. Generate an eight-bar melody from a start token
4. Harmonise a supplied melody using chord labels
5. Fill a masked symbolic bar

Do not begin by combining text, lyrics, full instrumentation, expressive audio, long-form structure, and production quality. Every added capability introduces another representation, dataset, loss, failure mode, and evaluation problem.

Why start with symbolic music?

MIDI represents performance instructions and musical events rather than a recorded waveform. A symbolic model can work with pitch, onset, duration, velocity, program, tempo, control changes, and pedal events without learning microphones, rooms, mastering, codecs, or waveform reconstruction.

This reduces the output space and makes errors easier to inspect. A wrong pitch, invalid duration, repeated note, or missing stop token can be identified directly.

The compromise is that MIDI does not capture the complete sound of a recording. The final audio depends on the synthesiser or instrument used for playback.

Choose a dataset that matches the question

A dataset is suitable only when its content, permissions, representation, and grouping information support the experiment.

For a first model, prioritise:

1. Clear rights and permitted use
2. Consistent symbolic files
3. A narrow musical domain
4. Enough complete pieces or phrases for held-out evaluation
5. Stable performer, composer, work, or source identifiers
6. Documentation of preprocessing and exclusions

A larger dataset is not automatically better if its files are duplicated, weakly documented, or unrelated to the target task.

Decide what one token means

The same MIDI file can be represented in several ways.

A fixed-grid piano roll records active pitches at each time step. An event representation records changes such as note-on, note-off, time-shift, and velocity. A note-tuple representation might store pitch, onset, and duration together.

The representation determines vocabulary size, sequence length, timing precision, and which errors the model can make. It should be fixed before comparing model architectures.

Split pieces before creating training windows

A model should be evaluated on musical material that was not used for parameter updates or repeated experiment decisions.

If one song is cut into many overlapping windows and those windows are randomly split, nearly identical phrases can appear in training and validation. Validation loss may then measure recognition of the same work rather than generalisation to new music.

Assign complete works, source files, composers, performers, or other related groups to splits before extracting windows.

Create windows after the split

Long pieces are often divided into fixed-length token windows for batching.

A context length of 64 tokens does not necessarily equal 64 notes or a fixed number of bars. It depends on the tokenisation. A dense passage can consume more tokens than a sparse passage covering the same duration.

Record both token length and approximate musical duration so context comparisons remain interpretable.

Build a baseline before a neural model

A baseline shows whether the neural system adds value beyond a simpler procedure.

For melody generation, useful baselines include:

1. Sampling pitches from their training frequencies
2. A first-order Markov chain using the previous event
3. An n-gram model using a short event history
4. Copying or repeating the seed phrase

A complicated model that cannot beat a simple baseline on held-out prediction and musical evaluation has not justified its complexity.

Change one major factor at a time

When comparing an RNN and Transformer, keep the dataset, splits, tokenizer, context length, training budget, loss, generation prompts, and evaluation procedure fixed where possible.

If the Transformer receives more data, a longer context, and more optimisation steps, the experiment cannot identify which change produced the difference.

A controlled comparison may not make the systems identical, but it should document every meaningful difference.

Define success before listening to the result

A successful small experiment should meet several kinds of criteria.

Technical criteria check whether the pipeline works. Predictive criteria compare held-out loss with baselines. Musical criteria measure pitch, rhythm, repetition, and structure. Qualitative criteria use blinded human listening. Operational criteria measure training cost and generation speed.

Predefining the criteria reduces the temptation to select whichever metric happens to improve.

Listen without revealing the model name

Human evaluation can compare shuffled samples from the baseline, RNN, Transformer, and held-out data.

Use the same synthesiser, loudness treatment, duration, seed format, and playback interface for every model. Do not tell listeners which system produced each sample.

Ask focused questions such as rhythmic stability, melodic coherence, repetition, or preference. A single broad question such as Is this good? is difficult to interpret.

Record everything required to repeat the run

Random initialisation, data shuffling, dropout, sampling, and hardware operations can cause runs to differ.

Record:

1. Dataset manifest and checksums
2. Split group IDs
3. Tokenizer version
4. Model configuration
5. Optimiser and learning-rate schedule
6. Batch size and gradient accumulation
7. Random seeds
8. Software versions
9. Hardware type
10. Checkpoint selection rule
11. Generation prompts and sampling settings

PyTorch notes that complete reproducibility is not guaranteed across all releases, platforms, and devices, even when randomness is controlled.

Decide when the experiment should stop

A pilot should not run indefinitely.

Stop or revise when:

1. The baseline already satisfies the project need
2. The neural model does not beat the baseline
3. Training loss falls while validation worsens
4. Generated syntax remains invalid
5. The representation removes the musical property being studied
6. Results vary too strongly across seeds
7. Data leakage is discovered
8. Rights or provenance cannot be resolved
9. Compute cost exceeds the planned budget

A negative result can still be useful when it removes an unsupported direction.

When improvement becomes memorisation

A small experiment can produce falling training loss and attractive samples while learning the dataset too literally.

The next section, Overfitting and Memorisation, examines training-validation gaps, duplicate data, exact and approximate reproduction, model capacity, and the difference between learning a musical pattern and recalling a training example.