An AI Stack: From Scaling AI Workloads to Evaluating LLMs

An AI Stack: From Scaling AI Workloads to Evaluating LLMs

Speakers: Ion Stoica (Professor, UC Berkeley)

Conference: MLSys 2025

YouTube: https://www.youtube.com/watch?v=None

Overview

In this comprehensive talk at MLSys 2025, Professor Ion Stoica from UC Berkeley presented an insightful journey through the evolution of the AI/ML stack, focusing on three pivotal open-source projects he has been deeply involved with: Ray, vLLM, and Chatbot Arena. These projects collectively address critical challenges in scaling AI workloads, optimizing large language model (LLM) inference, and establishing reliable evaluation methodologies for the rapidly evolving field of generative AI. Stoica emphasized that these projects form a cohesive "AI stack," where Ray provides the foundational distributed compute framework, vLLM (and its sibling SGLang) delivers high-performance LLM serving, and Chatbot Arena offers a dynamic, human-preference-driven approach to model evaluation.

The talk underscored the relentless growth of AI demands, which consistently outpace single-node hardware capabilities, necessitating distributed and heterogeneous computing. Stoica highlighted how each project emerged in response to specific trends and challenges, offering simple yet powerful solutions that have garnered widespread adoption across the industry. Beyond detailing the technical innovations, the presentation also offered broader lessons on the importance of aligning with trends, embracing simplicity in design, and the necessity of continuous re-architecting in a fast-moving domain. This article delves into the technical underpinnings, impact, and future implications of these projects, offering a deep dive into the practical and theoretical advancements they represent.

Background

▶ Watch: Introduction to Ray, vLLM, Chatbot Arena, SGLang (0:00)

The landscape of AI/ML has undergone a dramatic transformation, characterized by an insatiable demand for computational resources and increasingly complex workloads. As early as 2016, when Ray was conceived, it was evident that AI's compute requirements were growing at an exponential rate—over four times a year—far outstripping the improvements in single-node memory capacity and bandwidth. This fundamental disparity necessitates a shift towards distributed and heterogeneous infrastructure. Concurrently, AI workloads themselves became more intricate, evolving from classic machine learning tasks to sophisticated pipelines involving data preprocessing, model training, hyperparameter tuning, reinforcement learning (RL) rollouts, and serving, often interleaving different stages. The challenge was that existing distributed systems were typically specialized, forcing developers to stitch together multiple frameworks, each with its own API, deployment semantics, and resource management strategies. This led to complex development, difficult deployment, inefficient resource utilization, and slow data movement due to inter-system serialization.

By 2023, the emergence of large language models (LLMs) introduced a new set of challenges, particularly in LLM serving. Unlike previous ML models, LLMs were general-purpose, leading to a paradigm shift from "one model per use case" to "one model for everything." This made LLM serving a pervasive and critical workload. However, serving LLMs proved extremely expensive. Their autoregressive nature, generating one token at a time, resulted in very low GPU utilization, as the massive parallelism of modern GPUs was underutilized. While batching multiple requests could increase utilization, it quickly ran into memory bottlenecks, primarily due to the KV cache (Key-Value cache), which stores intermediate attention states. Traditional memory management approaches, which allocated contiguous memory and reserved space for the maximum possible output length, led to significant waste through internal fragmentation, reservation overhead, and external fragmentation, severely limiting the number of concurrent requests.

Finally, the rapid proliferation of LLMs also exposed significant limitations in existing LLM evaluation methodologies. Traditional benchmarks, while useful, were static and prone to contamination (models being trained on test data), making them unreliable indicators of true performance. More critically, for interactive applications like chatbots, these benchmarks failed to capture human preference, which is paramount for user experience. Human evaluation, while ideal, was slow, tedious, and unscalable, making it impractical for the continuous development and iteration required in the LLM space. The need for a dynamic, scalable, and human-centric evaluation system became acutely apparent.

Key Findings

▶ Watch: AI compute demands necessitate distributed, heterogeneous infrastructure (2:00)

The projects presented by Professor Stoica — Ray, vLLM (and SGLang), and Chatbot Arena — have each delivered transformative key findings and contributions that address the aforementioned challenges across the AI stack.

Ray emerged as a unified compute framework for distributed applications, fundamentally changing how developers build complex AI workloads. Its core innovation lies in generalizing Python's procedural language constructs (functions and classes) to a distributed setting, offering unparalleled flexibility. By introducing tasks (remote functions) and actors (remote classes) alongside futures for asynchronous execution and a shared in-memory object store for efficient data passing by reference, Ray provided a single, coherent system. This eliminated the need to integrate disparate frameworks, drastically simplifying development and deployment. The impact is evident in its accelerating adoption, becoming a foundational component for AI infrastructure at companies like OpenAI and many "digital native" firms. Recent developments, such as Compiled Graphs, further enhance Ray's efficiency for fine-grained, GPU-centric workloads by reducing overhead and enabling direct GPU-to-GPU communication.

vLLM (and SGLang) made a profound impact on LLM inference efficiency by tackling the KV cache memory bottleneck. The central discovery, PagedAttention, drew inspiration from operating system virtual memory. By breaking the KV cache into fixed-size "blocks" and managing them non-contiguously with an indirection table, PagedAttention drastically reduced memory fragmentation and enabled on-demand allocation. This innovation led to a 2.5-5x improvement in LLM serving throughput and achieved memory utilization as high as 96.3%. Furthermore, PagedAttention facilitates prefix sharing, allowing multiple requests with common initial prompts to share physical KV cache blocks, further boosting efficiency in scenarios like parallel sampling or multi-turn conversations. RadixAttention, used in SGLang, refines prefix sharing by organizing requests in a Radix tree and employing LRU-based eviction, maximizing cache hits. These techniques are now widely adopted, becoming standard in virtually every open-source and commercial LLM inference engine.

Chatbot Arena revolutionized LLM evaluation by introducing a scalable, human-preference-based methodology. The key finding was the surprising effectiveness of LLM-as-a-judge, where powerful models like GPT-4 could grade other LLM outputs with high agreement (85%) compared to human-to-human agreement (81%). While acknowledging biases (position, verbosity, self-enhancement), this approach demonstrated a viable path to scale evaluation. More importantly, Chatbot Arena implemented a head-to-head, Elo-like rating system, similar to chess or tennis rankings. Users compare two anonymized LLM outputs for a given prompt, and their votes contribute to a dynamic leaderboard. This system provides a continuously updated, real-world benchmark that directly reflects human preference. The project's impact is significant, with millions of monthly users and votes, making it a go-to platform for labs to test models before release. Additionally, the accumulated data has enabled new applications like Prompt-to-Leaderboard, a model router that can achieve a 30-point rating improvement or a 2x cost reduction for the same score by intelligently selecting models.

Collectively, these projects underscore three overarching lessons: trends matter (aligning with shifts like heterogeneous compute, memory-bound inference, and human-centric evaluation), simple solutions matter (minimalist APIs, OS-inspired paging, Elo ratings), and plan for flexibility and rewriting (recognizing the fast pace of AI development necessitates continuous re-architecture).

Technical Deep Dive

▶ Watch: Challenges of combining disparate systems for AI pipelines (4:00)

The technical innovations underpinning Ray, vLLM/SGLang, and Chatbot Arena represent significant advancements in their respective domains.

Ray Core: A Unified Compute Framework

Ray's core design philosophy is to provide a flexible, general-purpose distributed computing framework by extending familiar Pythonic constructs. This is achieved through its "Fast Compute Model," which abstracts distributed operations:

• Tasks: Python functions decorated with @ray.remote can be executed asynchronously on remote nodes. Calling function_name.remote() immediately returns a future (an object reference) instead of the actual result, allowing the driver to continue execution. The result can later be retrieved using ray.get(). This enables implicit parallelism for independent tasks.
• Actors: Python classes decorated with @ray.remote can be instantiated as stateful, distributed services. Each actor runs in its own process, managing its internal state. Methods of an actor are invoked similarly to tasks, e.g., actor_instance.method.remote(), also returning futures. Actors are crucial for maintaining mutable state across distributed operations.
• Shared In-Memory Object Store: Ray employs a distributed, in-memory object store. This is a critical component that allows data to be passed by reference rather than by value. When a task or actor produces a result, it's stored in the object store, and only a reference to it is passed to subsequent tasks/actors. This minimizes data copies, especially for large objects, and is analogous to passing pointers in a single-node system. For example, if function F produces X on Node 1 and function G consumes X on Node 2, X is transferred once from Node 1's object store to Node 2's object store when G is scheduled, not twice (caller to F, then caller to G) as in a pure RPC model.
• Minimalist API: Ray Core exposes a remarkably small API, primarily comprising six functions (e.g., ray.init, ray.get, ray.put, ray.remote). This simplicity contributes to its ease of adoption.
• Heterogeneous Computing: Ray allows users to specify resource requirements (e.g., number of CPUs, GPUs, custom resources) for tasks and actors, enabling efficient scheduling on diverse hardware.

Compiled Graphs (New Ray Development):

To address the overheads of the classic Ray API for fine-grained tasks (1-10ms latency) and inefficient GPU-to-GPU communication, Ray introduced Compiled Graphs. This experimental feature is designed for scenarios where the computational graph is static or mostly static.

• Static DAG Definition: Users define a Directed Acyclic Graph (DAG) of tasks and actor methods using a bind mechanism (e.g., B.method.bind(A.method.bind(input))).
• Compile-time Optimization: During compilation, all actors are instantiated, and static buffers are allocated. This pre-computation eliminates dynamic resource allocation and control plane overhead during runtime.
• Super Function Execution: The compiled DAG can then be executed like a single "super function," passing inputs and awaiting outputs. This significantly reduces RPC calls and associated overheads.
• Direct GPU-to-GPU Communication: Crucially, Compiled Graphs enable direct, asynchronous GPU-to-GPU communication using protocols like NCCL. In the standard Ray model, a tensor transfer between GPUs on different machines would involve multiple copies (GPU0 -> Object Store0 -> Object Store1 -> GPU1). With Compiled Graphs, this can be optimized to a single, direct transfer between GPU memories, leveraging high-bandwidth interconnects.

vLLM & SGLang: PagedAttention for LLM Inference

The core technical innovation in vLLM is PagedAttention, a KV cache management technique inspired by virtual memory in operating systems.

• KV Cache Bottleneck: During LLM inference, the self-attention mechanism requires storing the "keys" and "values" for all previously generated tokens (the KV cache). This cache grows with the sequence length and is a major memory consumer, especially for large models and long contexts. Traditional approaches allocate contiguous memory for the entire sequence, leading to:
• Internal Fragmentation: Over-allocation for output sequences whose actual length is less than the reserved maximum.
• Reservation Overhead: Memory allocated for future tokens in a sequence sits idle while tokens are generated one by one.
• External Fragmentation: Gaps in memory that are too small for new, larger allocations, even if enough total memory is available.
• PagedAttention Mechanism:

1. KV Blocks: The KV cache is divided into fixed-size "blocks," each containing a fixed number of embeddings (e.g., 4 embeddings).
2. Logical vs. Physical: Each request sees a logically contiguous sequence of tokens. However, in physical GPU memory, these blocks can be non-contiguous.
3. Indirection Table: A block table (analogous to a page table) maps the logical block indices of a request to their physical locations in GPU memory. This allows for flexible, on-demand allocation of blocks as tokens are generated.
4. On-Demand Allocation: Memory blocks are allocated only when needed for new tokens, eliminating reservation overhead and internal fragmentation.
5. Prefix Sharing: PagedAttention naturally supports sharing of KV cache blocks across multiple requests that have common prefixes (e.g., a shared system prompt). If two requests share the first N tokens, they can point to the same physical blocks for those N tokens, significantly saving memory.

• Differences from OS Paging:
• Eviction: Unlike OS pages, evicting a single KV cache block is less beneficial as all prior tokens are needed to compute the next. PagedAttention often evicts entire prefixes or recomputes them.
• Swapping: Instead of swapping to CPU RAM (which is slow), PagedAttention (and similar systems) can recompute the KV cache for a prefix if needed, leveraging GPU parallelism, which can sometimes be faster than memory transfers.
• RadixAttention (SGLang): This technique further optimizes prefix sharing by organizing all active requests' prefixes into a Radix tree. Each node in the tree represents a shared prefix. When a new request arrives, it traverses the tree to find the longest shared prefix. When eviction is necessary, a Least Recently Used (LRU) policy is applied to the leaf nodes of the Radix tree, ensuring that less frequently accessed prefixes are removed first.

Chatbot Arena: Human-Centric LLM Evaluation

Chatbot Arena tackles the challenge of LLM evaluation through a scalable, human-preference-based approach.

• Head-to-Head Evaluation: The core mechanism involves presenting a user with a prompt and two anonymized, randomized responses from different LLMs. The user then votes on which response is better, if it's a tie, or if both are bad. This "best of two" format is chosen for its simplicity and ease for human evaluators.
• Elo-like Rating System: The votes are used to update an Elo-like rating system, similar to those used in chess or tennis. This allows for continuous, dynamic ranking of LLMs as new votes come in, reflecting real-world performance based on human preference. The Bradley-Terry model is used to derive scores from pairwise comparisons.
• LLM-as-a-Judge: An early, critical component of scaling evaluation was the use of powerful LLMs (e.g., GPT-4) to act as judges themselves. This involved providing the LLM with a prompt and two candidate answers and asking it to grade them. While effective and showing high agreement with humans, it was found to have biases:
• Position Bias: Preference for the answer in the first position.
• Verbosity Bias: Preference for longer answers.
• Self-Enhancement Bias: Preference for answers from models in the same "family" or by the same developer.
• Limited Reasoning: Less effective at grading complex tasks like math problems compared to humans.
• Prompt-to-Leaderboard: Leveraging the vast dataset of human votes, Chatbot Arena developed Prompt-to-Leaderboard. This system trains a model to predict Bradley-Terry coefficients (model scores) for a given prompt, even if that specific prompt has never been seen before. It achieves this by finding similar prompts in the historical data and using their associated votes as a proxy. This enables intelligent model routing, where a prompt can be directed to the best-performing (or most cost-effective) LLM for that specific type of query.

Experimental Setup & Results

▶ Watch: Ray's key idea: generalizing Python for distributed flexibility (5:30)

The effectiveness of Ray, vLLM, and Chatbot Arena is validated through a combination of performance benchmarks, adoption metrics, and systematic studies.

Ray's Impact and Performance:

• Adoption: Ray's growth is accelerating, as evidenced by GitHub stars, surpassing other popular distributed frameworks. It is widely adopted by companies like OpenAI (for training GPT models) and numerous "digital native" companies for their AI platform infrastructure.
• Workload Efficiency: For batch inference, Ray can achieve 10x or more cost reduction by efficiently orchestrating data loading, preprocessing (often on CPUs), and GPU-based inference, maximizing GPU utilization. It supports diverse workloads including fine-tuning, multi-modal search, and embedding computation.
• Overhead Challenges: The classic Ray API, while flexible, incurs high overhead for small tasks (1-10ms) due to numerous RPC calls and dynamic memory allocations. GPU operations also suffer from inefficient communication paths (e.g., GPU memory -> object store -> remote object store -> remote GPU memory).
• Compiled Graphs Performance Goals: The new Compiled Graphs feature aims to reduce overhead to less than 1 millisecond for tasks in the 1-10ms range, and significantly lower overhead for GPU-to-GPU communication by enabling direct NCCL transfers. While experimental, this promises to unlock new levels of efficiency for latency-sensitive, GPU-bound AI workloads.

vLLM's Throughput and Memory Efficiency:

• Initial LLM Serving Challenges: For a Llama 13B model on an NVIDIA A100 GPU (40GB memory), initial serving throughput was only a few requests per second. This highlighted the severe underutilization of GPUs and the memory bottleneck.
• Memory Overhead Analysis: Studies showed that conventional KV cache management led to substantial memory waste. A scenario assuming perfect knowledge of output length (Oracle) still resulted in significant "reservation overhead" and external fragmentation. Only a small fraction (e.g., the "green stuff" in the presented plots) of allocated memory was actively used for KV cache. This suggested that 3-4 times more requests could be served if memory were utilized efficiently.
• PagedAttention Results: By implementing PagedAttention, vLLM achieved a 2.5x to 5x improvement in serving throughput. Memory utilization for the KV cache reached an impressive 96.3%, significantly reducing wasted memory and allowing for a much higher number of concurrent requests on the same hardware.
• Industry Adoption: The techniques in vLLM (especially PagedAttention) and SGLang (RadixAttention) are now fundamental and have been implemented in virtually every open-source and commercial LLM inference engine, demonstrating their widespread impact and validation.

Chatbot Arena's Evaluation Metrics:

• Catalyst: The initial success of fine-tuned Llama models (e.g., using ShareGPT data) underscored the need for robust evaluation beyond static benchmarks.
• LLM-as-a-Judge Validation: A systematic study compared GPT-4's judgments with human evaluations. The findings showed that GPT-4 and human votes agreed in 85% of cases, while the agreement between two different human evaluators was 81%. This remarkable parity indicated that LLM-as-a-judge, despite its biases, could be a highly effective and scalable proxy for human preference.
• Platform Impact: Chatbot Arena boasts over 1 million monthly users and has accumulated over 3 million votes, providing an unparalleled dataset for LLM performance tracking. It has become a crucial testing ground for companies like Grok (xAI) to evaluate their models before public release.
• Prompt-to-Leaderboard Efficacy: Using the Prompt-to-Leaderboard system as a model router demonstrated significant practical benefits. It could improve the overall model rating by 30 points or reduce the cost for achieving the same score by 2x, showcasing the value of data-driven model selection.
• New Evaluation Domains: The platform continues to expand, introducing new evaluation arenas for tasks like web development, text-to-web, GitHub issue fixing, search, and image generation, reflecting the evolving capabilities of LLMs.

Practical Implications

▶ Watch: Why Ray chose Python and a procedural approach (6:00)

The projects discussed by Professor Stoica offer profound practical implications for practitioners, infrastructure teams, model builders, and deployers in the AI/ML ecosystem.

For Infrastructure Teams and Deployers:

• Ray provides a robust, unified foundation for managing diverse and complex AI workloads. This translates to simplified infrastructure management, as teams no longer need to deploy and maintain separate distributed systems for training, tuning, and serving. The ability to express entire end-to-end pipelines within a single Pythonic framework significantly reduces operational overhead and cognitive load.
• The advent of Ray's Compiled Graphs is particularly impactful for real-time AI systems and latency-sensitive applications. By drastically reducing overhead for small tasks and enabling efficient GPU-to-GPU communication, it allows for more fine-grained control over resource allocation and execution, crucial for maximizing performance on expensive GPU clusters. This directly addresses the challenge of moving from CPU-centric to GPU-centric compute, ensuring optimal utilization of the most expensive resources.
• vLLM and SGLang are game-changers for cost-effective LLM serving. By overcoming the KV cache memory bottleneck, these inference engines enable deployers to serve significantly more concurrent requests on existing hardware, leading to substantial cost reductions (e.g., 2.5-5x throughput improvement). Their techniques are now industry standards, meaning that leveraging these optimizations is critical for any organization deploying LLMs at scale. This allows for the deployment of larger, more capable models or serving a wider user base without proportional increases in GPU infrastructure.

For Model Builders and Developers:

• Ray empowers model builders with the flexibility to scale their custom models and algorithms without needing deep distributed systems expertise. Its general-purpose programming model means developers can iterate quickly on complex pipelines, from data ingestion and distributed training (Ray Train), hyperparameter tuning (Ray Tune), to serving (Ray Serve), all within a familiar Python environment. This accelerates the development cycle for advanced AI applications, including reinforcement learning and agentic systems.
• Chatbot Arena offers model builders an invaluable, dynamic feedback loop for evaluating their LLMs. Unlike static benchmarks, it provides real-world human preference data, which is essential for developing models that genuinely resonate with users. The Elo-like rating system offers a clear, objective measure of relative performance, fostering continuous improvement. The platform's ability to host models anonymously allows for unbiased evaluation, crucial for identifying true strengths and weaknesses before public release.
• The data derived from Chatbot Arena, particularly through Prompt-to-Leaderboard, provides tools for intelligent model routing. Developers can build systems that dynamically select the most appropriate (and cost-efficient) LLM for a given user prompt, optimizing for both performance and inference costs. This is critical for building adaptive and economically viable LLM-powered applications.

General Tradeoffs and Limitations:

• While Ray offers flexibility, its classic API can introduce overhead for extremely fine-grained tasks, necessitating advanced features like Compiled Graphs, which add complexity.
• LLM-as-a-judge, while powerful, is subject to biases that require careful consideration and mitigation strategies when interpreting results.
• The rapid pace of AI development means that even highly optimized systems like vLLM and Ray require continuous re-architecting and refactoring to keep up with new hardware, model architectures, and optimization techniques. This implies a significant ongoing engineering investment.
• The broader challenge of reliable AI remains. While these projects improve performance and evaluation, the fundamental issue of ensuring AI systems provide verifiable, unambiguously correct solutions is still largely unsolved, limiting their deployment in high-stakes, human-unsupervised scenarios.

In essence, these projects highlight that successful AI deployment requires a holistic approach, integrating efficient distributed computing, optimized inference, and robust human-centric evaluation. The future demands even greater vertical stack integration and cross-layer optimization to maximize resource utilization and build truly reliable AI systems.

Key Takeaways

• Distributed AI is Essential: The growth of AI workloads and model complexity necessitates distributed and heterogeneous computing infrastructure. Ray provides a flexible, unified framework for scaling diverse AI tasks, from training to serving, simplifying development and management.
• Memory-Efficient LLM Serving: The autoregressive nature of LLMs and the substantial KV cache footprint make memory management critical for cost-effective inference. PagedAttention (vLLM) and RadixAttention (SGLang), inspired by OS virtual memory, dramatically improve KV cache utilization and throughput by enabling on-demand allocation and prefix sharing.
• Human-Centric LLM Evaluation: Static benchmarks are insufficient for evaluating interactive LLMs. Chatbot Arena's Elo-like rating system, powered by human head-to-head comparisons and validated by LLM-as-a-judge, offers a dynamic, scalable, and human-preference-aligned approach to assess model performance.
• Simplicity Drives Impact: The success of these projects demonstrates that simple, well-understood ideas (e.g., Pythonic distribution, OS paging, Elo ratings) applied creatively to new domains can lead to profound impact and widespread adoption in the AI community.
• Continuous Re-architecture is Key: The fast-evolving AI landscape demands flexibility and a willingness to constantly re-architect and refactor systems. Projects like Ray and vLLM have undergone multiple significant rewrites to adapt to new trends, hardware, and optimization opportunities.
• Future of AI: Vertical Integration and Reliability: The next frontier involves deep vertical stack integration (from application to hardware) to maximize resource utilization, especially for expensive GPUs. Furthermore, building truly reliable AI systems with clear, verifiable specifications is paramount for broader adoption beyond human-in-the-loop applications.

About the Speaker(s)

Professor Ion Stoica is a distinguished faculty member at UC Berkeley, renowned for his extensive contributions to distributed systems and AI over the past 15 years. His work spans various levels of the technology stack and includes involvement in highly influential projects such as Apache Spark and Apache Mesos. In this talk, he specifically highlighted his pivotal roles in the development of Ray, a unified compute framework for distributed AI; vLLM and SGLang, cutting-edge inference engines for large language models; and Chatbot Arena, an innovative platform for human-preference-based LLM evaluation. Professor Stoica's research consistently focuses on addressing fundamental challenges in scalable and efficient computing for emerging AI workloads.