DiffServe: Efficiently Serving Text-to-Image Diffusion Models with Query-Aware Model Scaling

DiffServe: Efficiently Serving Text-to-Image Diffusion Models with Query-Aware Model Scaling

Speakers: Sohaib Ahmad, Qizheng Yang, Haoliang Wang, Ramesh K. Sitaraman, Hui Guan (Presented by Shirong Yang)

Conference: MLSys 2025

YouTube: https://www.youtube.com/watch?v=None

Overview

The rapid advancement of text-to-image diffusion models has revolutionized content creation, but their computational intensity presents significant challenges for efficient serving in production environments. This talk introduces DiffServe, a novel system designed to optimize the serving of these models by dynamically adapting the model used for each query based on its perceived difficulty. Presented by Shirong Yang from UMass Amherst, this collaborative work with Adobe Research addresses the critical trade-off between image quality, generation latency, and computational cost that plagues current diffusion model serving infrastructures.

DiffServe's core innovation lies in its query-aware model scaling approach, which leverages a cascading architecture. Instead of committing to a single model for all requests, it first attempts to fulfill queries with a lightweight model. A specialized discriminator then evaluates the output quality, rerouting only the "hard" queries to a more computationally expensive, heavyweight model. This intelligent routing mechanism, coupled with a mixed-integer linear programming (MILP)-based resource allocator, allows DiffServe to significantly improve image quality and reduce service level objective (SLO) violations, demonstrating a sophisticated solution to a pressing problem in the ML systems space.

The importance of DiffServe stems from the observation that not all text-to-image prompts are equally challenging; a substantial fraction can be adequately handled by faster, less resource-intensive models without compromising user experience. By capitalizing on this insight, DiffServe offers a path towards more cost-effective and responsive AI services, making high-quality generative AI more accessible and sustainable for a wide range of applications.

Background

▶ Watch: Introduction: Computational cost of diffusion models (0:00)

The proliferation of text-to-image diffusion models has brought about unprecedented capabilities in image generation, but deploying these models at scale introduces substantial computational and economic burdens. The underlying denoising process in diffusion models is inherently iterative, often requiring dozens of steps to transform pure noise into a coherent image. This multi-step computation is the primary driver of high latency and significant GPU utilization, making efficient serving a complex task.

A critical aspect of this challenge is the inherent trade-off between image quality and generation speed. A diverse ecosystem of diffusion models exists, ranging from highly performant, high-quality models that are slow and expensive (e.g., Stable Diffusion 1.5) to faster, more lightweight models that may compromise on output fidelity (e.g., SD Turbo, SDXS). Traditional serving strategies often involve committing to a single model variant for all queries, which inevitably leads to suboptimal outcomes. Using a heavyweight model universally ensures high quality but incurs excessive costs and latency for simpler queries. Conversely, a lightweight model might be fast but would deliver unacceptable quality for complex prompts.

Prior work and existing systems have attempted to address these challenges. Some solutions focus on static model selection or basic dynamic resource allocation. For instance, systems like Clipper might serve all requests with a light or heavy model. More advanced systems like Proteus introduce dynamic resource allocation, adjusting the number of workers for different model variants based on overall demand. However, a significant limitation of these approaches is their lack of query awareness. They typically route queries randomly or based on aggregate load, failing to consider the individual difficulty of each prompt. This oversight means that easy queries might still be processed by expensive models, or hard queries might be assigned to lightweight models, leading to either wasted resources or degraded quality. The fundamental insight that "not all queries are equally hard" thus forms the bedrock for DiffServe's innovative approach, highlighting a gap in existing ML serving paradigms.

Key Findings

▶ Watch: Key observation: Not all queries are equally hard (3:00)

DiffServe's primary contribution is the development of a query-aware model scaling system that dynamically selects the most appropriate text-to-image diffusion model for each individual query. This approach fundamentally shifts from a "one-model-fits-all" paradigm to an adaptive, quality-and-efficiency-driven strategy. The key findings and contributions can be summarized as follows:

Firstly, DiffServe demonstrates that a significant portion of queries (observed to be 20-40% in experiments) are "easy" and can be effectively handled by lightweight diffusion models without a perceptible loss in quality compared to heavyweight alternatives. This observation underpins the system's efficiency gains.

Secondly, the introduction of a cascading model architecture is central to DiffServe. Queries are initially processed by a lightweight model, and the output is then evaluated by a specialized discriminator. This discriminator, trained to distinguish between "good enough" and "not good enough" images, makes a real-time decision: either return the lightweight model's output or reroute the query to a heavyweight model for a higher-quality generation. This intelligent gating ensures that expensive resources are only utilized when truly necessary.

Thirdly, DiffServe addresses the challenge of designing an effective discriminator for generative models. Unlike existing perceptual metrics (e.g., CLIP score, Pick score) which were found to perform worse than random routing and introduce overhead, DiffServe's discriminator is a custom-trained binary classifier. It learns to discern realistic, high-quality images from those that appear "fake" or low-quality, providing a robust confidence score for routing decisions. This specialized discriminator is shown to consistently outperform baselines in effectively guiding the cascading pipeline.

Finally, the system introduces a sophisticated mixed-integer linear programming (MILP)-based resource allocator. This component is critical for co-optimizing various system-level parameters—including the discriminator's confidence threshold, the number of workers allocated to lightweight and heavyweight models, and batch sizes. By searching a vast design space, the MILP allocator identifies Pareto-optimal configurations that deliver the best possible response quality for a given serving throughput, while adhering to latency SLOs. This dynamic, co-optimized resource management further enhances DiffServe's adaptability to varying workloads.

Through comprehensive evaluation, DiffServe significantly outperforms static baselines and state-of-the-art dynamic allocation systems like Proteus. It achieves up to a 24% improvement in image quality (lower FID score) and a substantial 19% to 70% reduction in SLO violation ratio, demonstrating its superior ability to balance quality and latency under diverse workload conditions.

Technical Deep Dive

▶ Watch: Core idea: Query-aware model scaling with cascading (4:50)

DiffServe's architecture is built upon several interconnected technical components that enable its query-aware model scaling and efficient resource management.

The core of DiffServe's data path is a cascading model pipeline. When a user submits a text prompt, it first arrives at a load balancer. This load balancer routes the query to a worker hosting a lightweight diffusion model (e.g., SD Turbo). This initial generation is designed to be fast, completing in approximately 0.1 seconds on an A100 GPU. Immediately following the lightweight model, a specialized discriminator evaluates the generated image. This discriminator produces a confidence score, indicating how likely the image is "good enough" based on a predefined threshold. If the confidence score surpasses this threshold, the image is deemed acceptable, and the result is returned directly to the user, saving significant computational resources and time. However, if the discriminator's confidence is low, the query is rerouted to a worker hosting a heavyweight diffusion model (e.g., SD 1.5). This second-stage model performs a more detailed, higher-quality generation, albeit with a higher latency (e.g., 1.8 seconds), and its output is then returned to the user. This selective routing ensures that the more expensive heavyweight model is only invoked when truly necessary.

A critical component of this cascading system is the discriminator. The design of this discriminator is non-trivial, especially for generative models where output quality can vary subtly. DiffServe's discriminator is a binary classifier trained using a real-versus-fake classification objective. During training, it learns to distinguish between high-quality "real" images (used as positive samples) and generated images (from both lightweight and heavyweight models, used as negative samples). This training approach enables the discriminator to develop an understanding of what constitutes a "realistic" or "high-quality" image in the context of diffusion model outputs. Crucially, during inference, the discriminator no longer requires real images; it simply evaluates the output of the lightweight diffusion model and produces a confidence score. This score guides the routing decision within the cascading pipeline, effectively estimating how realistic or high-quality the image appears. The authors emphasize that existing perceptual metrics like CLIP score or Pick score were found to be ineffective for this task, often performing worse than random routing and adding unnecessary computational overhead.

Beyond the data path, DiffServe incorporates a sophisticated control plane for dynamic resource allocation. This control plane consists of a controller and a model repository. Workers continuously send status updates to the controller, providing information about current loads and processing states. The controller utilizes this information, along with system-wide constraints and performance targets (e.g., latency SLOs), to make resource allocation decisions. These decisions involve co-optimizing several interacting parameters:

1. Confidence threshold for the discriminator: This threshold directly impacts the ratio of queries rerouted to the heavyweight model, influencing both overall quality and latency.
2. Number of workers allocated to lightweight and heavyweight models: This determines the serving capacity for each stage.
3. Batch size for model inference: Larger batch sizes can improve GPU utilization but may increase individual query latency.

To navigate the vast design space created by these interacting parameters, DiffServe employs a mixed-integer linear programming (MILP)-based resource allocator. The MILP formulation models the system's behavior and constraints, allowing the controller to search for Pareto-optimal configurations. These configurations represent the best possible trade-off between response quality (measured by FID score) and serving throughput, while strictly adhering to system constraints like latency deadlines. By dynamically reconfiguring these parameters, the MILP allocator enables DiffServe to adapt optimally to varying workloads and maintain high performance under diverse conditions. The controller then updates the load balancer with the new routing and allocation policies, ensuring queries are directed to appropriate workers running the correctly loaded models from the repository.

Experimental Setup & Results

▶ Watch: Challenge 2: Co-optimizing resource allocation parameters (9:00)

To validate the effectiveness of DiffServe, a comprehensive experimental evaluation was conducted against several baselines, focusing on key performance metrics under varying workload conditions.

Experimental Setup:

• Text Prompts: The experiments utilized text prompts drawn from the MS Coco dataset, a widely recognized benchmark for image generation tasks.
• Service Level Objective (SLO): A strict latency SLO of 5 seconds was set for each query, reflecting common requirements in real-time serving environments.
• Model Cascade:
• Lightweight Model: SD Turbo, chosen for its speed, running on an NVIDIA A100 GPU. It exhibited a rapid latency of just 0.1 seconds per query.
• Heavyweight Model: SD 1.5, selected for its higher quality output, also running on an A100 GPU. It had a higher latency of 1.8 seconds per query.
• Baselines:
• Clipper Light: A static baseline that uses only the lightweight model (SD Turbo) for all queries. It is not query-aware.
• Clipper Heavy: A static baseline that uses only the heavyweight model (SD 1.5) for all queries. It is also not query-aware.
• Proteus: A state-of-the-art dynamic resource allocation system. While it can tune model variants based on demand, it routes queries randomly to each variant, lacking query-awareness.
• DiffServe: The proposed method, combining both query-awareness (via the discriminator and cascading) and dynamic resource allocation (via the MILP optimizer).

Key Metrics:

• FID (Frechet Inception Distance): A standard metric for evaluating the quality of generated images. Lower FID values indicate better image quality and closer resemblance to real images.
• SLO Violation Ratio: Measures the percentage of queries that fail to meet the 5-second latency deadline. Lower values indicate better adherence to service level agreements.

Results:

The evaluation demonstrated DiffServe's superior performance across a spectrum of workloads.

1. Performance Under Median Workload:

• Static Baselines (Clipper Light, Clipper Heavy): As expected, these models struggled. Clipper Light achieved low latency (fewer SLO violations) but delivered poor image quality (high FID). Conversely, Clipper Heavy provided better image quality (lower FID) but suffered from frequent SLO violations due to its slower nature.
• Proteus: This dynamic system offered a middle ground, improving upon the static extremes by adapting resource allocation. However, its random routing strategy led to suboptimal performance, as it couldn't differentiate between easy and hard queries.
• DiffServe: Consistently occupied the bottom-left corner of the FID vs. SLO violation graph. This indicates that DiffServe simultaneously achieved both low FID (high image quality) and low SLO violation ratio (high latency adherence). This strong performance was attributed directly to its query-aware nature, allowing it to select the optimal model for each query based on its difficulty.

1. Adaptability to Varying Workloads:

• Experiments conducted under low load and high load conditions further reinforced DiffServe's robustness. The system maintained its superior trade-off between quality and latency across these diverse scenarios. This adaptability stems from its dynamic adjustment of the discriminator's confidence threshold and intelligent resource reallocation for workers, guided by the MILP optimizer.

1. Quantified Improvements:

• In summary, DiffServe was shown to improve image quality by up to 24% (relative to baselines, as indicated by a lower FID score).
• It also achieved a significant reduction in latency violation by 19% to 70%.

These results provide compelling evidence that DiffServe's combination of query-aware cascading and intelligent resource optimization offers a substantial leap forward in efficiently serving text-to-image diffusion models, delivering both high quality and low latency in a cost-effective manner.

Practical Implications

▶ Watch: DiffServe system architecture and workflow (10:50)

DiffServe presents significant practical implications for anyone involved in deploying, operating, or building applications on text-to-image diffusion models. Its core value proposition lies in enabling a more efficient and responsive serving infrastructure, directly impacting cost, user experience, and system reliability.

For practitioners and MLOps teams, DiffServe offers a blueprint for achieving a better balance between computational cost and service quality. By intelligently routing queries, organizations can significantly reduce the GPU hours spent on "easy" requests, leading to substantial cost savings in cloud infrastructure. This efficiency gain allows for either serving more requests with the same hardware budget or reducing the overall infrastructure footprint for a given workload. The ability to dynamically adapt to varying workloads, from low to high demand, also means more stable performance and fewer SLO violations, translating to a more reliable service for end-users.

Infrastructure teams will appreciate the system's ability to maximize resource utilization. The MILP-based resource allocator provides a principled way to co-optimize parameters like worker allocation and batch sizes, ensuring that expensive hardware like A100 GPUs are used effectively. This intelligent allocation mitigates the common problem of under-utilization during low demand or overload during peak times, leading to more predictable operational costs and performance.

For model builders and deployers, DiffServe highlights the importance of considering model cascades and quality discriminators in their deployment strategies. Instead of being forced to choose a single model variant, they can leverage a spectrum of models (e.g., distilled versions like SD Turbo alongside full versions like SD 1.5) to create a more resilient and performant service. The approach is generalizable; for example, as noted in the Q&A, even different "step" counts for the same model (e.g., Flux with fewer steps vs. more steps) can be integrated into the lightweight/heavyweight cascade.

However, implementing DiffServe also comes with its tradeoffs and limitations. The primary complexity lies in the development and maintenance of the discriminator model. Training a robust discriminator that accurately assesses image quality for routing decisions requires careful data curation and training strategies, which adds an overhead to the initial setup. Furthermore, while the MILP allocator provides optimal configurations, solving MILP problems can be computationally intensive, requiring careful design for real-time adaptability in highly dynamic environments. There's also the challenge, acknowledged in the Q&A, of accurately predicting query difficulty before running any diffusion model; DiffServe addresses this by running a lightweight model first, but a purely text-based pre-classifier remains an open research question.

Despite these complexities, the benefits of DiffServe—improved image quality, reduced latency violations, and enhanced resource efficiency—make a strong case for its adoption in production-grade diffusion model serving. It represents a significant step towards building more intelligent, adaptive, and cost-effective generative AI systems.

Key Takeaways

• Query-Aware Model Scaling: DiffServe introduces a novel approach that dynamically selects between lightweight and heavyweight diffusion models based on the perceived difficulty of each user query.
• Cascading Architecture: It employs a two-stage cascade where queries first pass through a lightweight model, and only "hard" queries are rerouted to a more powerful, heavyweight model after evaluation by a discriminator.
• Effective Discriminator: A custom-trained binary classifier, specifically designed to distinguish realistic from low-quality generated images, is crucial for accurate routing decisions, outperforming generic perceptual metrics.
• Optimized Resource Allocation: A mixed-integer linear programming (MILP)-based allocator co-optimizes system parameters like confidence thresholds, worker counts, and batch sizes to achieve Pareto-optimal configurations for quality and throughput.
• Significant Performance Gains: DiffServe improves image quality by up to 24% (lower FID) and reduces service level objective (SLO) violation ratios by 19% to 70% compared to state-of-the-art baselines.
• Cost Efficiency & Adaptability: By avoiding unnecessary use of expensive heavyweight models, DiffServe offers substantial cost savings and maintains robust performance across varying workload intensities.

About the Speaker(s)

The work on DiffServe is a collaborative effort by Sohaib Ahmad, Qizheng Yang, Haoliang Wang, Ramesh K. Sitaraman, and Hui Guan. The presentation at MLSys 2025 was delivered by Shirong Yang, a PhD student from UMass Amherst. This research represents a joint initiative between UMass Amherst and Adobe Research, bringing together academic rigor and industry insights to address critical challenges in AI/ML systems.