Make some noise: Teaching the language of audio to an LLM using sound tokens
Shivam Mehta from KTH presents a method for teaching Large Language Models (LLMs) to understand and generate audio by treating it as a discrete language. The approach involves a two-step process: first, creating an ultra-low bitrate (0.293 kbps) audio representation using a causal variational autoencoder, and second, fine-tuning a Llama 7B model with these audio tokens using LoRA.
The central goal of this work is to extend the powerful emergent abilities of foundation models to the audio domain. This is achieved by reframing audio not as a continuous signal, but as a discrete “language” that can be learned by a Large Language Model (LLM) through the standard next-token prediction paradigm. The approach, inspired by the successes of models like SEED-Llama in the visual domain, is broken down into two primary steps: creating a quantized, low-bitrate audio representation and then fine-tuning an LLM to comprehend and generate these audio tokens.
Step 1: Causal Audio Tokenization at an Ultra-Low Bitrate
A significant challenge in processing audio is its high dimensionality. A 10-second clip at 44.1kHz contains over 440,000 data points, which is computationally prohibitive for standard transformer architectures.
-
Initial Compression: To address this, the first step leverages the pre-trained Variational Autoencoder (VAE) from Stability AI’s Stable Audio model. This VAE compresses a 10-second audio clip into a much more manageable continuous latent representation of 215 temporal steps and 64 dimensions.
-
Causal Quantization: To enable next-token prediction, this continuous representation must be discretized into a sequence of tokens. Inspired by SEED-Llama, a causal quantization process is introduced. A transformer with a causal attention mask processes the continuous latents, ensuring that the tokenization for any given time step only depends on previous steps. This embeds a left-to-right inductive bias, mirroring how LLMs process language.
“The idea was like what if we quantise these… representations and… they introduced the idea of having a coal left to right Q former contisation so that the current token only has access to the previous one which is also how LLMs sort of learn.”
-
Improved Reconstruction with Flow Matching: The initial tokenizer was trained to reconstruct the VAE latents using a Mean Square Error (MSE) loss. However, this was improved upon by replacing the decoder with a Conditional Flow Matching (CFM) model—a type of diffusion model. This probabilistic approach yielded superior reconstruction quality, particularly for out-of-distribution data, as validated by lower reconstruction errors and better FID-audio scores.
This tokenization process results in an ultra-low bitrate of 0.293 kbps, where one second of audio is represented by approximately 21 integer tokens from a codebook of 8,192.
Step 2: Fine-Tuning Llama with Audio Tokens
With a method for converting audio into a sequence of discrete tokens, a Llama 7B model was fine-tuned using LoRA for efficiency. The training was divided into two stages.
-
Stage 1: Pre-training for Association: The model was first trained on the Open-EQA dataset, which contains audio-caption pairs. The input was a mix of audio and text tokens, with their positions randomly swapped. The objective was simple next-token prediction, forcing the model to learn the relationship between audio events and their textual descriptions. This resulted in a base model capable of basic “autocomplete-style” audio captioning.
-
Stage 2: Instruction Fine-Tuning: The model was then fine-tuned on a larger, more diverse dataset (including WaveCaps and FMA) formatted for instruction following. Prompts were structured for both discriminative tasks (e.g., “Describe the sound in this audio clip”) and generative tasks (e.g., “Generate the sound of a bell”).
Results and Key Takeaways
The fine-tuned model demonstrates strong and promising capabilities in audio understanding and reasoning. It can identify instruments, describe how they work, and engage in “chitty-chatty” dialogue about audio content. On quantitative audio captioning benchmarks (FENCE, SPICE), it performs competitively with the LTU baseline model.
However, the model’s generative capabilities are still nascent and show clear signs of being undertrained. While it can generate some recognizable sounds like sirens or water dripping, it often falls into repetitive loops, a common behavior for undertrained generative models.
Key Implementation Details:
- To encourage the model to learn the new modality, the loss on audio tokens was weighted 10 times higher than the loss on text tokens.
- Z-loss was used to maintain training stability and prevent exploding logits.
In conclusion, this work successfully demonstrates that by tokenizing audio into a discrete, low-bitrate, causal sequence, an LLM can be fine-tuned to achieve a high level of audio comprehension. While true generative capabilities require further training and potentially more semantically rich tokenization, the approach validates the promise of treating audio as a language that LLMs can learn to speak.