Long-Form Speech Generation with Spoken Language Models

Long-Form Speech Generation with Spoken Language Models

Speakers: Se Jin Park, Student Researcher, Google DeepMind, KAIST; Julian Salazar, Google DeepMind; Aren Jansen, Google DeepMind; Keisuke Kinoshita, Google DeepMind; Yong Man Ro, KAIST; RJ Skerry-Ryan, Google DeepMind

Conference: ICML 2025

YouTube: https://slideslive.com/39044051

Overview

This article delves into the groundbreaking work presented by Se Jin Park and Julian Salazar on SpeechSSM, a novel approach to generating long-form, coherent, and expressive speech using spoken language models. Developed through a collaboration between Google DeepMind and KAIST, SpeechSSM represents a significant leap forward in the field, addressing the long-standing challenge of generating extended audio sequences that maintain contextual relevance, speaker identity, and naturalness. The talk highlights the limitations of existing speech generation methods, particularly their reliance on text intermediates and their struggle with the high temporal resolution and information density inherent in raw audio, which often leads to prohibitive computational costs and a degradation of quality over time.

The core innovation of SpeechSSM lies in its adoption of a decoder-only hybrid state space model architecture, specifically Griffin, enabling linear complexity and constant memory usage during inference. This allows for what the authors term "unbounded speech generation," where models can produce audio sequences lasting minutes or even hours without succumbing to memory constraints or a loss of coherence. The research emphasizes the importance of natively processing audio to capture crucial paralinguistic aspects—such as emotion, cadence, and speaker identity—that are often lost or difficult to model when relying on text-based representations.

The significance of SpeechSSM extends beyond mere technical prowess. It paves the way for the development of next-generation AI agents capable of continuous, real-time, full-duplex spoken interaction. Imagine conversational AI systems that can engage in hour-long dialogues without forgetting previous context, or virtual assistants that maintain a consistent vocal persona and emotional tone throughout extended interactions. By demonstrating the feasibility of long-form, textless speech generation, SpeechSSM moves the research community closer to a future where AI-driven spoken communication is indistinguishable from human interaction, offering profound implications for human-computer interfaces, content creation, and accessibility technologies.

Background

▶ Watch: Challenge of long-form speech generation in SLMs (0:00)

The landscape of language models has seen remarkable advancements, primarily driven by innovations in text-based architectures like Transformers. These models excel at understanding and generating human language, but their application to speech presents unique challenges. Spoken Language Models (SLMs) are autoregressive models designed to process speech audio directly, predicting the next segment or token of audio. This native audio processing offers a distinct advantage: the ability to capture rich paralinguistic information such as emotion, cadence, and speaker identity, which are fundamental to human communication and often difficult to convey or infer from text alone.

Despite their potential, SLMs have historically lagged behind their text-based counterparts, particularly in their ability to handle long sequences. Most existing SLMs focus on generating only a sentence or two, far short of the "long-form" goal of 16 minutes or more. This limitation stems from several inherent properties of speech and the computational demands of current modeling paradigms:

1. High Information Density: Speech audio is laden with confounding information, including background noise, room acoustics, and the unique characteristics of a speaker's voice, all of which can influence the underlying audio tokens.
2. High Temporal Resolution: Speech tokenization results in significantly longer sequences compared to text. A concept that might take one or two text tokens could require 10 to 15 speech tokens. For instance, a 10-minute audio segment can translate to approximately 15,000 speech tokens, causing sequence lengths to explode rapidly.
3. Transformer Limitations: The quadratic computational cost of self-attention mechanisms in standard Transformer architectures makes long-range modeling prohibitively expensive in terms of memory and compute. While text LMs have achieved impressive long-context reasoning (e.g., million-token contexts), their generation capabilities are often capped at much shorter lengths (e.g., 32,000 text tokens). For speech, the need is for continuous, real-time generation without pauses or restarts, demanding constant memory and infinite context.

Previous attempts to address these challenges have largely relied on workarounds. Many systems, such as Moshi or the team's prior work Spectron, resort to text intermediates, either generating text first and then converting it to speech, or embedding text internally. Other approaches involve using longer audio tokens to reduce the overall sequence length, but this can compromise granularity and expressive detail. The core problem remains: how to enable truly textless, unbounded speech generation that maintains quality and coherence over extended durations, mirroring the natural flow of human conversation.

Key Findings

▶ Watch: State Space Models (SSMs) for long sequences (4:50)

The research introduces SpeechSSM, the first textless spoken language model specifically engineered for generating coherent, expressive speech of unbounded length. The key findings and contributions are multifaceted:

1. Linear Time, Constant Space Modeling: SpeechSSM employs a decoder-only hybrid state space model (SSM) architecture, notably drawing inspiration from models like Griffin and Samba. This design is critical for achieving linear complexity in time and constant memory usage during inference, effectively circumventing the quadratic scaling issues of Transformers and enabling indefinite speech generation without memory exhaustion.
2. Decoupled Semantic and Acoustic Generation: The model architecture strategically separates the task of semantic modeling from acoustic generation. It utilizes a USMv2 tokenizer to convert speech into 32,000 semantic tokens at 25 Hz, allowing the core language model to focus purely on semantic coherence over time. These semantic tokens are then fed into SoundStorm, which, conditioned on a 3-second speaker prompt, generates the final acoustic waveform, ensuring the generated speech maintains the desired speaker characteristics.
3. Window Tokenizing and Decoding: To process arbitrarily long speech sequences with bounded computational resources, SpeechSSM introduces a simple yet highly effective technique called window tokenizing and decoding. Audio is divided into fixed-length, overlapping windows, processed independently, and then merged back into a continuous stream with boundary adjustments. This method allows the use of non-causal tokenizers and decoders on long inputs while keeping memory requirements constant.
4. Generative Length Extrapolation: A crucial finding was the importance of eliminating End-of-Sequence (EOS) bias to enable the model to extrapolate beyond its training length. This was achieved by:

• Completely removing explicit positional embeddings.
• Preventing implicit EOS cues by padding the last window with speech instead of silence.
• Trimming the last 10 seconds of training examples to avoid dataset-specific endings that could implicitly signal an end.

1. Novel Evaluation Benchmark and Metrics: Recognizing the inadequacy of existing metrics for long-form speech generation, the team proposed:

• The LibriSpeech long benchmark, created by re-processing existing LibriSpeech data into 4-minute segments for multi-minute continuation tasks (up to 4 and 60 minutes).
• New metrics: Semantic Similarity (cosine similarity between ground truth and generation), Side-by-Side Win Rates (using an LLM as a judge for fluency, coherence, logicality, and interestingness), Semantic Coherence over Length (SCL) (semantic similarity over 100-word segments), and Mean Opinion Score over Time (MOS-T) (naturalness of 5-second speech segments over time). These provide a more comprehensive assessment of long-form quality degradation.

1. Superior Performance and Efficiency: SpeechSSM significantly outperforms Transformer-based baselines across all proposed metrics. It achieves substantially lower perplexity and Gecko scores, and wins over 75% of side-by-side comparisons against strong SSM baselines. Critically, it maintains high MOS-T and SCL scores throughout 4-minute and even 60-minute generations, closely mirroring ground truth and demonstrating robust contextual coherence. Furthermore, it boasts remarkable efficiency, achieving over 120 times better throughput and 5 times faster decoding time for 10 minutes of audio compared to Transformer baselines on a single TPU.
2. Extrapolation Capability: A key result is that a SpeechSSM model trained on 4-minute audio sequences can generate coherent speech for up to 60 minutes with performance comparable to models trained on longer durations, underscoring its strong generative length extrapolation capabilities.

Technical Deep Dive

▶ Watch: SpeechSSM core design: linear-time, constant-space hybrid SSM (6:20)

The design of SpeechSSM is meticulously crafted to meet several critical desiderata for unbounded speech generation: constant memory during decoding, infinite context to capture distant dependencies, and generative length extrapolation.

Model Architecture: Hybrid State Space Models

At its core, SpeechSSM leverages a decoder-only hybrid state space model (SSM), specifically drawing on architectures like Griffin or Samba. SSMs represent a paradigm shift from Transformers for long-sequence modeling. While earlier sub-quadratic models existed, SSMs, particularly the S4 work on the long-range arena, demonstrated a unique ability to outperform Transformers on lengthy sequences, offering linear complexity with efficient forward passes. This choice directly addresses the computational and memory bottlenecks that plague Transformer-based models when dealing with the thousands of tokens generated by long-form speech. The linear scaling of SSMs ensures that as the generated sequence grows, the memory and computational cost per step remain constant, enabling truly "unbounded" generation.

Decoupling Semantic Modeling from Acoustic Generation

A key architectural decision in SpeechSSM is the explicit decoupling of semantic understanding and generation from the acoustic realization. This modularity allows each component to specialize and optimize for its specific task:

1. Semantic Modeling: The input speech is first processed by a USMv2 tokenizer. This tokenizer converts the raw audio into a sequence of 32,000 discrete semantic tokens at a rate of 25 Hz. By abstracting away low-level acoustic details, the core SpeechSSM language model can focus entirely on modeling semantic coherence, topicality, and the temporal flow of ideas. This abstraction reduces the "confounding information" that Julian Salazar mentioned, allowing the model to learn higher-level linguistic patterns more effectively.
2. Acoustic Generation: The decoded sequence of semantic tokens is then passed to SoundStorm, a neural audio codec. Crucially, SoundStorm also takes a 3-second speaker prompt as input. This prompt provides the necessary acoustic information—including timbre, pitch, and speaking style—to condition SoundStorm to generate speech that matches the characteristics of the desired speaker. This two-stage process ensures that the generated speech is not only semantically coherent but also acoustically natural and consistent in speaker identity.

Window Tokenizing and Decoding for Unbounded Input

To handle input audio of arbitrary length, especially for non-causal operations like tokenization that might require a broader context, SpeechSSM introduces window tokenizing and decoding. This technique is elegantly simple yet highly effective:

1. Windowing: The continuous audio stream is divided into fixed-length, overlapping windows.
2. Independent Processing: Each window is processed independently by the USMv2 tokenizer and subsequently by the SpeechSSM model. This ensures that the memory footprint of these operations remains bounded, regardless of the overall audio duration.
3. Overlap Adjustment: When merging the processed windows back into a continuous stream of tokens or raw audio, a clever overlap adjustment mechanism is employed at the boundaries. This smooths transitions and prevents artifacts that could arise from independent processing, ensuring a continuous and natural output. This approach allows the system to effectively "see" and process an unbounded sequence while operating within fixed memory constraints.

Facilitating Generative Length Extrapolation

Achieving high-quality generation beyond the lengths seen during training is paramount for long-form models. SpeechSSM incorporates several techniques to prevent End-of-Sequence (EOS) bias, which can implicitly signal the model to conclude generation prematurely or degrade quality:

1. Removal of Positional Embeddings: Explicit positional embeddings, common in Transformer architectures, can inadvertently encode information about the absolute position within a sequence, thus limiting extrapolation. SpeechSSM completely removes these, forcing the model to rely solely on relative dependencies and the state of the SSM.
2. Preventing Implicit EOS Cues:

• Speech Padding: Instead of padding the last window of training examples with silence (which could act as an implicit EOS signal), SpeechSSM pads with actual speech. This ensures that the model never associates silence at the end of a segment with the end of a coherent utterance.
• Trimming Training Data: The last 10 seconds of training examples are trimmed. This avoids dataset-specific endings (e.g., common phrases or sound effects that might consistently appear at the end of recordings in a dataset), further preventing the model from learning implicit cues that signal the end of a sequence.

These architectural and methodological innovations collectively allow SpeechSSM to generate expressive, coherent, and topically consistent speech for durations far exceeding typical SLM capabilities, all while maintaining computational efficiency.

Experimental Setup & Results

▶ Watch: SpeechSSM: Decoupling semantic and acoustic generation (6:40)

The evaluation of SpeechSSM focused on demonstrating its capability for long-form, coherent, and expressive speech generation, particularly its ability to extrapolate beyond training lengths.

Training Details

• Training Data: The primary training dataset consisted of 60,000 hours of audio from the LibriLight dataset. Additionally, for some demonstrations and samples, a larger dataset of 200,000 hours of extemporaneous speech (more expressive and informal) was used.
• Model Variants: Two main variants of SpeechSSM were trained: a 2 billion parameter model and a 9 billion parameter model.
• Initialization: Both SpeechSSM variants were initialized from Recurrent Gemma, a powerful language model pre-trained on an extensive 2 trillion text tokens. This provided a strong foundation for linguistic understanding.
• Hardware and Steps: Training was conducted on 16 TPUs for 100,000 steps, with a batch size processing 768,000 tokens per batch.

Baselines

To provide a fair comparison, the researchers included several baselines:

• Existing Transformer-based models: These were generally unable to generate intelligible speech beyond a minute, highlighting the fundamental challenge of long-range modeling for Transformers in this domain.
• SpeechTransformer: A custom Transformer baseline was trained using the same data and initialization as SpeechSSM. This allowed for a direct comparison, decoupling the effects of token types, initialization, and training data from the core architectural differences.
• SSM 2B and 7B baselines: These were used in side-by-side evaluations to compare against other State Space Model implementations.

Evaluation Task and Benchmark

A novel evaluation task was proposed: 10-second speech continuation, where the model is prompted with 10 seconds of audio and asked to generate completions up to 4 minutes and 60 minutes. To support this, the team created the LibriSpeech long benchmark by re-processing the existing LibriSpeech dataset, cutting it into 4-minute segments suitable for long-form evaluation.

Metrics

Recognizing the limitations of traditional, superficial metrics (like transcript perplexity, Sworger, and S-Blimp) for assessing long-form coherence and quality, the team introduced several new, more robust metrics:

• Semantic Similarity: Measures the cosine similarity between the ground truth and the generated transcript, capturing how well the core meaning is preserved.
• Side-by-Side Win Rates: Utilizes a Large Language Model (LLM) as a judge to compare generated transcripts side-by-side against a baseline or ground truth, evaluating fluency, coherence, logicality, and interestingness.
• Semantic Coherence over Length (SCL): Quantifies how semantic similarity changes over time. It measures the semantic similarity between the initial prompt and successive 100-word segments of the generation, revealing degradation patterns.
• Mean Opinion Score over Time (MOS-T): Assesses the naturalness of the generated speech over time. It measures the naturalness of 5-second speech segments extracted at regular intervals (e.g., each minute) throughout the generation.

Headline Results

The experimental results overwhelmingly demonstrated the superiority of SpeechSSM:

1. 4-Minute Continuation Task: SpeechSSM significantly outperformed other baseline models in traditional metrics like perplexity and Gecko.
2. Side-by-Side Evaluation: In direct comparisons against SSM 2 billion and 7 billion baselines, SpeechSSM won more than 75% of the time. While none of the models consistently won against ground truth (indicating the discriminative nature of the metric), this highlights SpeechSSM's strong relative performance.
3. Speaker Similarity: The model exhibited "particularly high" speaker similarity, attributed to the effective disentanglement of semantic and acoustic generation, allowing SoundStorm to faithfully replicate speaker characteristics from the prompt.
4. MOS-T Scores: SpeechSSM maintained very high MOS-T scores throughout the 4-minute generation, closely matching the ground truth. In stark contrast, existing baselines showed significant degradation in naturalness over time.
5. SCL Metric: The Semantic Coherence over Length metric revealed that while semantic similarity naturally degrades over very long generations (as topics tend to slowly diverge), SpeechSSM maintained high semantic similarity scores throughout a 60-minute generation, comparable to the ground truth line. Baseline models, especially beyond their training length (around 200-word segments), exhibited a sharp drop in coherence.
6. Extrapolation Capability: A crucial finding was that the SpeechSSM model trained on 4-minute audio performed similarly to models explicitly trained on longer durations when generating up to 60 minutes of speech. This powerfully validates the effectiveness of the extrapolation techniques (EOS bias removal).
7. Throughput and Decoding Time: For decoding 10 minutes of audio on a single TPU, SpeechSSM achieved more than 120 times better throughput and was over 5 times faster than the Transformer baseline. This efficiency is critical for real-world, unbounded generation.
8. Contextual Coherence in Transcripts: Analysis of generated transcripts showed SpeechSSM maintaining strong contextual coherence over long durations. For example, proper nouns introduced in the prompt, such as "Philip," "Prince Albert," and "Princess Maria," continued to appear and be referenced consistently throughout a 4-minute generation. Competitor models like SpiritLM showed rapid degradation, often referencing proper nouns only once before diverging. Even for 60-minute generations, SpeechSSM demonstrated the ability to maintain topical coherence, although issues with grammar and flow were observed, which the researchers expect to improve with model scaling and larger datasets.

These results unequivocally establish SpeechSSM as a leader in long-form speech generation, demonstrating both high quality and unprecedented efficiency.

Practical Implications

▶ Watch: Achieving length extrapolation by removing EOS bias (7:40)

The development of SpeechSSM carries significant practical implications for various domains, particularly in the realm of human-computer interaction and AI agent development. Its ability to generate coherent, expressive, and continuous speech for extended durations addresses fundamental limitations of prior systems.

1. Enabling True Conversational AI Agents: The most immediate impact is on conversational AI and virtual assistants. Current systems often struggle with maintaining context and speaker identity over long dialogues, frequently requiring "resets" or exhibiting a disjointed conversational flow. SpeechSSM's capacity for infinite context and constant memory decoding means that AI agents can continuously listen and generate speech in real-time, engaging in hour-long conversations without "forgetting" earlier parts of the discussion or running out of memory. This enables a more natural, fluid, and human-like interaction experience.
2. Enhanced Expressiveness and Naturalness: By natively processing audio and disentangling semantic from acoustic modeling, SpeechSSM can more effectively capture and reproduce paralinguistic information such as emotion, tone, and speaker characteristics. This leads to generated speech that is not only semantically correct but also emotionally resonant and acoustically natural, making AI voices more engaging and less robotic. This is crucial for applications requiring empathetic or persuasive communication, such as customer service, education, or entertainment.
3. Computational Efficiency for Production Systems: The adoption of hybrid State Space Models (SSMs) like Griffin, with their linear complexity and constant memory usage, translates directly into massive gains in efficiency. The reported 120x better throughput and 5x faster decoding time compared to Transformer baselines means that deploying long-form speech generation in real-time, at scale, becomes far more feasible and cost-effective. This reduces the infrastructure burden for companies building speech-enabled products.
4. Robustness to Context and Speaker Identity: The model's demonstrated ability to maintain contextual coherence (e.g., consistent reference to proper nouns) and speaker similarity over 60-minute generations is a game-changer. This robustness is critical for any application where long-term memory and consistent persona are important, from audiobook narration to AI companions.
5. New Possibilities for Content Creation and Accessibility: SpeechSSM could revolutionize the creation of audio content, enabling automated generation of podcasts, audiobooks, or even virtual characters with consistent voices and nuanced delivery. For accessibility, it could power more natural and continuous text-to-speech systems for individuals with communication challenges, or enable more sophisticated voice interfaces for assistive technologies.

Tradeoffs and Limitations

While SpeechSSM marks a significant advancement, it's important to acknowledge its current tradeoffs and limitations:

• Grammar and Flow in Very Long Generations: Although topical coherence is maintained, the 60-minute samples still exhibited some issues with grammar and overall flow. The researchers are optimistic that these will improve with model scaling and training on even larger datasets, but it indicates that perfect human-like fluency over extreme durations is still an active area of refinement.
• Audio-Only vs. Text-Audio Joint Models: A pertinent question raised during the Q&A was the benefit of an audio-only approach versus models that jointly leverage text and audio. The speakers highlighted that audio-only captures non-verbal cues (gasps, coughs, music) and avoids the inference slowdown associated with decoding text tokens in a synchronous setting. However, text provides a highly structured and robust representation of language. The trade-off is that while audio-only gains paralinguistic richness, it might require substantial data scaling to match the linguistic robustness of text-audio joint models. It's a design choice with different strengths and weaknesses depending on the application.
• Limited Explicit Control: The current model focuses on speech continuation and extrapolation. It does not yet offer fine-grained control over specific speech characteristics (e.g., changing emotional valence, speaking rate, or style) beyond what is implicitly learned from the speaker prompt. This would likely be an area for future research and task-specific training.

Despite these areas for future work, SpeechSSM represents a monumental step towards truly intelligent and natural spoken AI, moving beyond fragmented utterances to continuous, context-aware, and expressive verbal communication.

Key Takeaways

• SpeechSSM is the first textless spoken language model capable of generating coherent, expressive speech for unbounded durations (up to 60+ minutes).
• It leverages decoder-only hybrid State Space Models (SSMs) like Griffin for linear complexity, constant memory usage, and superior long-range modeling compared to Transformers, enabling efficient, indefinite generation.
• Key architectural innovations include decoupling semantic modeling (USMv2) from acoustic generation (SoundStorm with speaker prompt) and window tokenizing/decoding for processing unbounded inputs with bounded memory.
• Achieves robust generative length extrapolation by carefully removing EOS bias through techniques like omitting positional embeddings, speech padding, and trimming training data endings.
• Introduces a novel LibriSpeech long benchmark and advanced metrics (Semantic Similarity, Side-by-Side Win Rates, SCL, MOS-T) to accurately evaluate long-form speech quality and coherence over time.
• Demonstrates exceptional performance, outperforming Transformer baselines in quality, coherence, and efficiency (e.g., 120x throughput, 5x faster decoding for 10 minutes of audio on 1 TPU). It maintains high quality and coherence over 60-minute generations, even when trained on shorter sequences.
• Paves the way for next-generation real-time, continuous conversational AI agents by enabling natural, context-aware, and speaker-consistent spoken interactions without memory limitations or quality degradation.

About the Speaker(s)

The research presented on Long-Form Speech Generation with Spoken Language Models was a collaborative effort. Se Jin Park is a student researcher at Google DeepMind and is affiliated with KAIST, playing a key role in the design and implementation of SpeechSSM. Julian Salazar is a researcher at Google DeepMind, who presented the initial overview and context of the work. The extended team also includes Aren Jansen, Keisuke Kinoshita, Yong Man Ro (also affiliated with KAIST), and RJ Skerry-Ryan, all contributing to this significant advancement in spoken language models from both Google DeepMind and KAIST.