PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Training

PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Training

Speakers: Daiyaan Arfeen, Intern; Zhen Zhang; Xinwei Fu; Gregory Ganger; Yida Wang (collaborators at AWS)

Conference: MLSys 2025

YouTube: https://www.youtube.com/watch?v=None

Overview

The proliferation of large language models (LLMs) has necessitated increasingly sophisticated and scalable training techniques. Among these, pipeline parallelism has emerged as a crucial strategy for distributing the immense computational and memory demands of LLMs across multiple GPUs. However, a significant inefficiency inherent to pipeline parallelism is the phenomenon of pipeline bubbles – periods during which GPUs sit idle, waiting for data to propagate through the pipeline or for gradient synchronization. This talk introduces PipeFill, a novel system designed to reclaim this wasted GPU compute time by intelligently scheduling and executing independent "fill jobs" during these bubble periods. Developed by Daiyaan Arfeen and collaborators at AWS, PipeFill addresses a critical challenge in large-scale LLM training: maximizing GPU utilization to reduce training costs and accelerate development cycles.

PipeFill's core innovation lies in its ability to transparently manage memory and computation for these secondary jobs without impacting the performance of the primary LLM training task. By identifying specific types of workloads – namely, other DNN training or batch inference jobs – that are amenable to intermittent execution and possess adjustable memory footprints, PipeFill transforms idle GPU cycles into productive work. This approach promises substantial improvements in the overall efficiency of AI infrastructure, making large-scale model training more cost-effective and resource-efficient, particularly as LLMs continue to grow in size and complexity, demanding ever-larger distributed training setups.

Background

▶ Watch: Introduction to PipeFill and pipeline bubbles problem (0:00)

Training Large Language Models (LLMs) involves processing vast datasets with models comprising billions or even trillions of parameters. This scale often exceeds the memory and computational capacity of a single GPU, necessitating distributed training strategies. While data parallelism (DP) and tensor parallelism (TP) are fundamental techniques, they face limitations. Data parallelism replicates the entire model across multiple devices, processing different subsets of data simultaneously, but is bottlenecked by the memory capacity of individual GPUs for very large models. Tensor parallelism partitions individual layers or tensors across devices, allowing larger models, but its scalability can be constrained by communication overheads and the complexity of partitioning within layers. For models exceeding these limits, pipeline parallelism becomes indispensable.

Pipeline parallelism partitions the model's layers into sequential stages, with each stage assigned to a different GPU or group of GPUs. During a forward pass, a micro-batch of data progresses through these stages sequentially. To maximize throughput, multiple micro-batches are often processed concurrently in a pipelined fashion, allowing different stages to work on different micro-batches simultaneously. This creates an illusion of parallelism, effectively hiding some communication and computation latency.

However, a fundamental requirement of Deep Neural Network (DNN) training is gradient synchronization at the end of each mini-batch iteration. This means that after all micro-batches within a mini-batch have completed both their forward and backward passes through the entire pipeline, all accumulated gradients must be synchronized across devices before the model weights can be updated. This synchronization necessitates a "flush" of the pipeline, where no new micro-batches can enter until the current ones have fully exited and gradients are synchronized. These periods of forced idleness are known as pipeline bubbles. During a pipeline bubble, GPUs assigned to pipeline stages are completely idle, consuming power but performing no useful computation for the main training job.

The problem of pipeline bubbles is exacerbated when pipeline parallelism is combined with data parallelism, a common scenario for scaling LLM training to thousands of GPUs. As the number of data parallel replicas increases, the total number of samples per mini-batch iteration might remain constant, but the number of samples processed per replica decreases. For example, if a job uses 4 samples per mini-batch with one data parallel replica, it processes 4 samples before a gradient synchronization and pipeline flush. If scaled to two data parallel replicas, each replica processes only 2 samples per mini-batch before its own gradient synchronization. While this setup reduces the overall computation time for the main job by distributing the workload, it paradoxically increases the aggregate idle time across all GPUs. Each individual pipeline now flushes more frequently relative to its processing time, leading to a higher proportion of time spent in bubbles. This means that as LLM training scales out to meet increasing demands, the total amount of GPU time wasted in pipeline bubbles grows, leading to significant inefficiencies and higher operational costs. This inherent inefficiency is the core problem PipeFill aims to resolve.

Key Findings

▶ Watch: How data parallelism increases total pipeline bubbles (2:10)

PipeFill's primary contribution is demonstrating a practical and effective method for recovering GPU utilization during the idle periods (pipeline bubbles) inherent in large-scale pipeline-parallel LLM training. The system achieves this by running carefully selected "fill jobs" – other DNN training or batch inference tasks – within these bubbles. The key findings and contributions highlighted by the research include:

1. Zero Impact on Main Job Performance: A crucial aspect of PipeFill's design is its guarantee that the execution of fill jobs does not in any way slow down or interfere with the performance of the primary pipeline-parallel LLM training job. This ensures that the core training objective remains uncompromised, making the recovered GPU cycles a pure gain in efficiency. The initial results presented confirm this, showing that PipeFill maintains the main job's performance even when scaling from 1,000 to 8,000 GPUs.
2. Effective GPU Utilization Recovery: By strategically scheduling and executing fill jobs, PipeFill significantly improves overall GPU utilization. It transforms periods of complete idleness into productive computation, leading to a more efficient use of expensive GPU resources. While specific quantitative results on the overall utilization gain were not fully detailed in the provided transcript, the mechanism clearly points towards substantial improvements.
3. Intelligent Memory and Context Management: PipeFill addresses the complex challenges of operating within the highly constrained and dynamic environment of pipeline bubbles. It provides mechanisms for fill jobs to adapt their memory footprint to the limited GPU memory available during bubbles and efficiently manage context switching between the main job and fill jobs.
4. Optimized Fill Job Throughput: The system incorporates an optimization strategy to select the best configuration and partitioning of fill jobs, aiming to maximize their throughput. This ensures that the work performed by fill jobs during bubbles is as efficient as possible, leading to faster completion of these secondary tasks.

In essence, PipeFill provides a robust framework for transforming what was previously wasted GPU time into valuable computational cycles, thereby enhancing the economic efficiency and throughput of large-scale AI training infrastructure.

Technical Deep Dive

▶ Watch: Ideal characteristics for 'fill jobs' explained (4:20)

PipeFill is designed to tackle the fundamental challenge of utilizing GPUs during intermittent idle periods—pipeline bubbles—within pipeline-parallel (PP) LLM training. The system addresses several critical technical hurdles related to memory management, context switching, and workload scheduling to achieve this.

The core idea is to execute fill jobs (secondary, independent DNN training or batch inference tasks) during these bubbles. The selection of these job types is deliberate:

• Not Latency-Sensitive: Unlike online or real-time inference, these jobs can tolerate intermittent execution and do not require immediate responses, preventing increased latency for critical user-facing services.
• Adjustable GPU Memory Requirements: This is paramount because, even when the GPU compute units are idle during a bubble, a significant portion of the GPU's memory remains occupied by the primary LLM training job's model parameters, activations, and optimizer states. Fill jobs must therefore operate within the remaining, limited, and non-contiguous memory space.

PipeFill addresses these challenges through two primary mechanisms: flexible memory configuration for fill jobs and intelligent partitioning of fill job computation graphs across non-continuous bubble durations.

Memory Management for Fill Jobs

To operate within the constrained memory environment, PipeFill provides several "knobs" that can be adjusted to reduce a fill job's GPU memory footprint:

1. Micro-batch Size and Gradient Accumulation: For DNN training fill jobs, reducing the micro-batch size directly lowers the memory needed for activations. This can be combined with gradient accumulation, where gradients from multiple small micro-batches are accumulated before a single weight update. This technique effectively simulates a larger batch size while keeping the per-step memory footprint low. By increasing the number of gradient accumulation steps, the memory requirement for activations per forward/backward pass is reduced.
2. Parameter Offloading: Model parameters (weights and biases) can consume substantial GPU memory. PipeFill allows for parameter offloading, where portions of the fill job's model parameters are moved from GPU memory to CPU memory (or even host storage) and only loaded onto the GPU when needed for computation. This reduces the GPU-resident parameter footprint but introduces latency penalties for data transfer.
3. Activation Checkpointing: Also known as gradient checkpointing, this technique reduces activation memory usage during the backward pass. Instead of storing all intermediate activations from the forward pass, only a subset are checkpointed. Missing activations are recomputed during the backward pass. This trades off computation (recomputation) for memory savings. PipeFill can adjust the degree of checkpointing (e.g., how many layers are checkpointed) to balance memory and recomputation overhead.
4. Activation Offloading to CPU: Similar to parameter offloading, activations can also be offloaded to CPU memory. This is generally considered "very suboptimal" due to the high bandwidth requirements of activations and the latency of PCIe transfers, making it a last resort when GPU memory is extremely scarce.

By combining these techniques, PipeFill can dynamically configure fill jobs to fit into varying amounts of available GPU memory, making them adaptable to the specific memory constraints of different pipeline bubbles.

Context Switching and Non-Continuous Execution

Pipeline bubbles are not continuous, arbitrarily long periods; they are discrete, often short, intervals. PipeFill must enable fill jobs to execute piecewise across these non-contiguous durations. The solution involves greedy partitioning of the fill job's computation graph:

1. Computation Graph Representation: Each fill job, with a given memory configuration (e.g., specific micro-batch size, offloading strategy), has a defined computation graph (e.g., a sequence of layers for a DNN).
2. Offline Measurement: Before deployment, the system measures the characteristics of pipeline bubbles: their duration and the available GPU memory capacity. These measurements are crucial constraints for scheduling.
3. Greedy Partitioning Algorithm: For each possible fill job configuration, PipeFill greedily partitions its computation graph. The goal is to execute the largest possible contiguous chunk of the graph within a single bubble, subject to two main constraints:

• Bubble Duration Constraint: The execution time of the partitioned graph segment must not exceed the duration of the current pipeline bubble. It is critical that the fill job never "spills over" and encroaches on the main LLM training job's execution time, as this would slow down the primary task.
• GPU Memory Constraint: The memory footprint of the partitioned graph segment (including its model parameters, activations, and intermediate states, as configured by the memory knobs) must not exceed the available GPU memory during that bubble.

1. Iterative Partitioning: The algorithm continues this process, taking the next available bubble and scheduling the next segment of the fill job's graph, until the entire fill job's computation graph is partitioned. It aims to use the fewest number of bubbles possible, but acknowledges that 100% utilization of every bubble's duration might not always be achievable due to memory or computational dependencies. For example, if a bubble has a certain duration but only limited memory, only a smaller part of the computation graph might fit.
2. Optimization for Throughput: After partitioning the fill job's graph for various configurations, PipeFill then selects the combination of the best memory configuration (e.g., how much to offload, what micro-batch size) and the best partition strategy that yields the highest throughput for the fill job. This ensures that the maximum amount of useful work for the fill job is completed within the available bubble periods.

By implementing these sophisticated memory management and scheduling techniques, PipeFill transforms pipeline bubbles from periods of wasted resources into opportunities for accelerating secondary workloads, all without disrupting the critical path of LLM training.

Experimental Setup & Results

▶ Watch: Key challenges: memory, context switching, scheduling (5:20)

The transcript provides a brief introduction to the experimental results, indicating that PipeFill's primary achievement is ensuring the main LLM training job remains unaffected by the introduction of fill jobs. The speaker states, "PipeFill doesn't affect the main job," and elaborates that this was tested by scaling an LLM training job using data parallelism from 1,000 to 8,000 GPUs.

Beyond this crucial initial finding, specific quantitative results regarding the actual GPU utilization improvement, the throughput achieved by fill jobs, or detailed comparisons against baselines were not provided within the scope of the transcript. The discussion of experimental setup (datasets, specific LLM models, hardware types, metrics beyond "no effect") also remains largely unmentioned.

It is implied that the system measures bubble durations and GPU memory capacities offline to inform its greedy partitioning strategy, as discussed in the technical deep dive. However, the details of these measurements, the specific fill jobs used in experiments (e.g., model types, sizes), or the metrics for their performance (e.g., samples/second, training loss reduction) are not available in the provided material. The core takeaway from the presented results section is the fundamental validation that PipeFill can operate without imposing overheads on the primary LLM training task, a critical prerequisite for its practical adoption.

Practical Implications

▶ Watch: Partitioning computation graph for non-continuous bubbles (7:50)

PipeFill's approach to reclaiming idle GPU cycles during pipeline bubbles has profound practical implications for anyone involved in large-scale AI/ML development and infrastructure management, particularly in the context of LLMs.

For practitioners and model builders, PipeFill offers a significant boost in the effective utilization of expensive GPU resources. Training LLMs is notoriously costly and time-consuming. By filling bubbles, PipeFill effectively reduces the "true" cost per useful computation hour, allowing more models to be trained, more experiments to be run, or existing training jobs to complete faster if the fill jobs are related to the main task (e.g., hyperparameter tuning for the same LLM). This can accelerate research and development cycles, enabling faster iteration on model architectures and training methodologies. The ability to run "free" secondary tasks like smaller DNN training jobs or batch inference during primary LLM training means that computational resources can be stretched further, potentially making advanced AI development more accessible.

For infrastructure teams and deployers, PipeFill represents a powerful tool for optimizing cluster efficiency. In large GPU clusters, maximizing utilization is a constant challenge. PipeFill directly addresses a known source of inefficiency in a common LLM training paradigm. This can lead to:

• Reduced Operational Costs: By doing more with existing hardware, the total cost of ownership (TCO) for GPU clusters can be lowered. Less idle time means a better return on investment for capital expenditures on GPUs.
• Improved Throughput: The ability to concurrently run secondary workloads means that the overall throughput of the AI infrastructure increases. Jobs that might otherwise contend for dedicated GPU time can now be scheduled opportunistically.
• Enhanced Resource Scheduling: While the talk notes that advanced scheduling for fill jobs won't be covered, the framework itself enables more sophisticated resource management. Infrastructure teams can prioritize fill jobs based on importance, ensuring that valuable secondary tasks are completed whenever GPU cycles become available.

Tradeoffs and Limitations:

• Complexity: Integrating PipeFill introduces an additional layer of complexity to the training pipeline and resource management. Configuration of fill jobs, especially their memory "knobs," requires careful consideration.
• Fill Job Selection: Not all jobs are suitable as fill jobs. They must be non-latency-sensitive and have adjustable memory footprints. Real-time inference, for instance, is explicitly ruled out due to its latency requirements.
• Memory Constraints: The primary limitation for fill jobs is the available GPU memory. Even with offloading and checkpointing, some fill jobs might be too large or complex to fit into the remaining memory during bubbles. This might necessitate using smaller models or highly optimized fill tasks.
• Context Switching Overhead: While the talk emphasizes no impact on the main job, context switching between the main job and fill jobs inherently incurs some overhead. PipeFill's effectiveness depends on this overhead being negligible compared to the bubble duration and the computational gains from the fill jobs.
• Offline Profiling: The reliance on offline measurement of bubble durations and memory capacities suggests that the system might need recalibration if the main LLM training job's characteristics (e.g., micro-batch size, model architecture, number of stages) change significantly.

Despite these considerations, PipeFill offers a compelling solution to a pervasive problem in large-scale LLM training. By intelligently leveraging previously wasted compute cycles, it paves the way for more efficient, cost-effective, and accelerated AI development.

Key Takeaways

• Pipeline Bubbles are a Major Inefficiency: Large-scale LLM training using pipeline parallelism suffers from significant GPU idle time during "pipeline bubbles," especially when combined with data parallelism.
• PipeFill Recovers Idle GPU Time: PipeFill is a system designed to utilize these pipeline bubbles by running independent "fill jobs" (DNN training or batch inference) on the otherwise idle GPUs.
• No Impact on Main Job: A critical design principle and key finding is that PipeFill does not slow down or interfere with the performance of the primary LLM training job.
• Adaptive Memory Management: Fill jobs must operate with limited GPU memory. PipeFill uses techniques like gradient accumulation, parameter offloading, and activation checkpointing to adjust fill job memory footprints.
• Greedy Graph Partitioning: To handle non-continuous bubble durations, PipeFill greedily partitions fill job computation graphs, ensuring execution segments fit within bubble durations and available memory, optimizing for fill job throughput.
• Enhanced Infrastructure Efficiency: PipeFill significantly improves overall GPU utilization, leading to reduced training costs and accelerated development cycles for LLMs and other AI workloads.

About the Speaker(s)

Daiyaan Arfeen presented the PipeFill system, a project he worked on during his internship. The research and development of PipeFill were conducted in collaboration with Zhen Zhang, Xinwei Fu, Gregory Ganger, and Yida Wang, all affiliated with AWS. This indicates that PipeFill is a result of industry-led research focused on practical challenges in large-scale AI infrastructure.