Audio Tokens and Neural Codecs

Digital audio is already made of numbers. At a sample rate of 32 kHz, one second of mono audio contains 32,000 sample values. A three-minute track contains millions of them.

A model could try to generate those values one at a time, but the sequence would be extremely long. Neural audio codecs offer a more manageable route. They learn how to compress short sections of audio into a smaller set of codes, then reconstruct the sound from those codes.

When the codes are chosen from fixed codebooks, they can be stored as discrete IDs. These IDs are often called audio tokens. A music model can learn to predict the token sequence, then a decoder turns the completed sequence back into audio.

Quantization: turning smooth values into choices

The encoder first creates continuous values. Continuous means they can take many possible numerical values. A language model is easier to train when the possible outputs come from a fixed set, much like a vocabulary.

The quantizer solves this by replacing each latent vector with an entry from a learned codebook. Each codebook entry has an ID. The signal is now represented by a sequence of choices such as 18, 402, 77, and 911 rather than by a continuous stream of latent values.

This step is lossy. The selected entry is only an approximation of the original latent vector. A well-trained codec tries to keep the parts that matter most to perceived audio quality.

Why use several codebooks?

One codebook may capture a rough version of the signal but miss smaller details. SoundStream and EnCodec use residual vector quantization, often shortened to RVQ.

The first codebook approximates the latent vector. The system then measures what is still missing, called the residual. A second codebook approximates that leftover information. More codebooks can continue refining the result.

This creates a useful quality control. Using fewer codebooks gives a lower bitrate and a rougher reconstruction. Using more codebooks carries more information and can improve quality, but it also creates more tokens for a generative model to predict.

Bitrate: how much information is kept?

Bitrate measures how many bits are used each second. A lower bitrate stores or transmits less information. That gives stronger compression, but the decoder has less detail to work with.

At lower settings, a reconstruction may lose fine texture, sharp attacks, stereo detail, or clarity. It may also introduce artificial sounds known as compression artifacts. The exact result depends on the codec, the training data, and the type of audio.

Higher bitrate is not free. More codebooks or more frequent tokens increase storage and give a generative model more symbols to predict. Codec design is therefore a balance between compact sequences and convincing sound.

From codec tokens to a music model

Once audio has been turned into discrete IDs, generation begins to look more like sequence modelling. The model receives earlier tokens and predicts which tokens should come next.

The tokens are not musical notes. A single token does not usually mean piano, kick drum, or C major. Its meaning depends on the codebook, the surrounding tokens, and the decoder. The token streams collectively carry information about rhythm, pitch, timbre, voices, effects, and production.

After the music model finishes predicting the sequence, the codec decoder converts the tokens into a waveform.