Dataset Balance and Bias

A dataset can contain millions of audio clips and still represent a narrow part of music.

Imagine a collection where half the hours come from English-language pop, a small group of artists appears thousands of times, and some regional genres have only a few short examples. The file count may look impressive, but the model keeps seeing the same kinds of patterns.

This is why dataset balance is about more than size. It asks who and what appears, how often they appear, how they are labelled, and which combinations are missing.

A model usually becomes more reliable on patterns that appear often and consistently. Rare, noisy, or poorly labelled categories may be harder to learn. The result can be weaker prompt following, less convincing generation, or outputs that drift toward more common styles.

The long tail

Many music datasets have a long-tail distribution. A small number of tags, artists, genres, or sound events appear very often, while many others appear only a few times.

AudioSet is a clear example. It contains a large ontology of sound events, but the number of examples varies widely between classes. MTG-Jamendo contains more than 55,000 tracks and 195 genre, instrument, and mood or theme tags. Its published tag distributions show that some labels are far more common than others.

A long tail is not automatically a mistake. It may reflect the available catalogue or the real use case. The problem begins when the dataset is described as broadly representative but rare categories do not contain enough varied examples for the model to learn them well.

Count examples and duration

File counts can be misleading. Ten full albums and 100 short clips do not contribute the same amount of audio. A five-minute track can also create many more training crops than a thirty-second clip.

A basic audit should therefore report both:

1. Number of source recordings
2. Total audio hours
3. Number of training segments after cropping
4. Unique artists and releases
5. Average and median duration

This helps reveal whether a category looks large only because a few long recordings were divided into many segments.

Watch artist and source concentration

A dataset can have many tracks but few independent creators. If one artist contributes 5,000 examples, the model may learn that artist's instrumentation, production, voice, or songwriting habits more strongly than the genre as a whole.

Useful concentration checks include:

1. Share of total hours from the largest artist
2. Share from the largest ten artists
3. Median tracks per artist
4. Number of unique releases
5. Share from the largest source platform or archive
6. Number of crops derived from each source recording

Artist and source limits can reduce domination, but they should be documented. Capping every artist at the same number can also remove useful variation from artists with broad catalogues.

Cultural coverage is a measurable issue

Recent music-generation research has examined which traditions appear in commonly used datasets. One large review reported that music from the Global North accounted for most of the analysed dataset hours and research attention. Another study found that only a small share of hours in the datasets it reviewed came from non-Western genres.

The exact percentage changes with the papers included and the way regions and genres are defined. That difference is itself important. Cultural categories are not simple technical labels, so an audit should publish its definitions rather than treating them as universal facts.

The shared finding is clearer than any single percentage: global musical traditions are not represented evenly in the datasets commonly used for generative music research.

Balance the labels as well as the audio

A category can have enough hours and still be poorly represented if the labels are inconsistent or incomplete.

For example, the same style may appear under several spellings, translations, or neighbouring genre terms. One platform may use a local name while another replaces it with a broad Western category. Some tracks may have detailed instrument tags while others have only one general genre label.

Before counting categories, preserve the original labels and map them to a documented vocabulary. Track unknown and missing values instead of silently treating them as negatives.

Production quality can become a hidden category

Suppose traditional recordings are mostly older archives or live performances, while mainstream pop arrives as recent studio masters. A model may begin to associate the traditional genre with noise, narrow frequency range, or lower loudness.

Cleaning every recording until it sounds similar can hide the problem without fixing representation. A better audit tracks recording era, source format, loudness, noise, live or studio setting, and codec quality. The dataset can then separate musical style from recording condition during sampling and evaluation.

Document the dataset, not just the final percentages

A balance report should explain:

1. Why the dataset was created
2. Which groups and dimensions were measured
3. How labels and regions were defined
4. Which sources and licences were used
5. Which records were removed
6. Which groups are missing or uncertain
7. How sampling weights are applied
8. Which evaluations should be reported by group

The Datasheets for Datasets framework was created to make this kind of information easier to communicate. A dataset cannot be audited properly when its collection and composition remain hidden.