LMArena: An Open Platform for Crowdsourced AI Benchmarks

LMArena: An Open Platform for Crowdsourced AI Benchmarks

Speakers: Wei-Lin Chiang

Conference: MLSys 2025

YouTube: https://www.youtube.com/watch?v=None

Overview

In an era defined by the rapid evolution of generative AI, particularly large language models (LLMs), the traditional paradigms for evaluating AI performance are proving increasingly insufficient. Wei-Lin Chiang from UC Berkeley presented a compelling talk at MLSys 2025, introducing LMArena, an open, community-driven platform designed to address these pressing challenges. LMArena is an ambitious initiative aimed at crowdsourcing AI benchmarks and evaluating language models at scale through human feedback, providing a dynamic, real-world lens into model capabilities.

The core problem LMArena seeks to solve is the fundamental difficulty in answering the question: "Which AI is the best?" This seemingly simple query becomes incredibly complex in the context of open-ended generative models, where objective metrics often fall short. LMArena moves beyond static, specialized benchmarks by fostering a live, organic evaluation environment that continuously adapts to new models and real-world user interactions. By leveraging collective human intelligence, the platform generates dynamic leaderboards and granular performance insights, offering an invaluable resource for both researchers and practitioners navigating the fast-paced landscape of foundation models.

This talk illuminated not only the critical need for evolving evaluation methodologies but also the practical, scalable solutions LMArena brings to the table. From its innovative crowdsourcing mechanism and the use of the Bradley-Terry model for robust ranking, to advanced features like Prompt-to-Leaderboard and the expansion into multimodal evaluations such as Web Arena and Repo Chat, LMArena stands as a testament to the power of open collaboration in advancing AI assessment. The platform has garnered significant community support and industry recognition, solidifying its role as a pivotal tool in understanding and improving the next generation of AI systems.

Background

▶ Watch: Introduction to LMArena and crowdsourced AI benchmarks (0:00)

The landscape of AI evaluation has undergone a dramatic transformation with the rise of generative models, particularly large language models (LLMs). Historically, benchmarks like ImageNet provided clear, objective metrics such as accuracy for classification or object detection. These benchmarks were well-suited for specialized, static tasks with definitive ground truths. However, the advent of LLMs, trained on vast swathes of internet text and multimodal datasets, has ushered in a fundamentally subjective problem space where traditional evaluation methods falter.

The challenges in evaluating generative models are multifaceted and severe. Firstly, tasks are increasingly open-ended, often requiring real-world interaction and nuanced understanding rather than simple multiple-choice answers. Judging the quality of a generated piece of code, creative writing, or a complex response necessitates expert knowledge in diverse fields, making evaluation expensive and time-consuming. For instance, the GPQA benchmark, a high-quality dataset, cost over $120,000 to create for just 500 questions, highlighting the prohibitive expense of manual, expert-driven evaluation at scale.

Secondly, a critical issue is the pervasive risk of test leakage or data contamination. As models are trained on increasingly large and diverse datasets, often comprising the entire internet, it becomes nearly impossible to guarantee that benchmark questions or similar data points have not inadvertently been included in the training corpus. This contamination leads to an overestimation of model performance, as models are effectively tested on data they have already seen. The Llama 3 technical report, cited in the talk, revealed that many widely used benchmarks, including MMLU, exhibit high contamination scores, often exceeding 80%. This renders static benchmarks unreliable and undermines their utility in truly assessing a model's generalization capabilities.

Finally, the relentless pace of innovation in the ML/systems space exacerbates these issues. New models are released every week or two, quickly rendering static benchmarks obsolete. The cost and effort required to build and maintain high-quality, uncontaminated benchmarks that can keep pace with this rapid development are simply unsustainable. This confluence of subjectivity, contamination risk, and rapid model evolution creates an urgent need for dynamic, real-world, and scalable evaluation platforms—a gap that LMArena seeks to fill.

Key Findings

▶ Watch: Critique of static benchmarks: high contamination scores (8:00)

LMArena emerged as a direct response to the limitations of static benchmarks, offering a novel, community-driven approach to evaluating large language models. The platform's core innovation lies in its ability to harness crowdsourced human feedback to create a "live benchmark" that continuously evolves.

A primary finding is the effectiveness of anonymized pairwise comparisons in gathering reliable subjective feedback. Users are presented with a prompt of their choosing, and two anonymized models respond side-by-side. The user then votes on which response is superior, after which the model identities are revealed. This method ensures unbiased evaluation and captures real-world use cases directly from the community. The data collected from these interactions—the prompt, the two model responses, and the user's vote—forms the basis for all subsequent analysis.

The platform utilizes a Bradley-Terry model regression to compute an Arena Score for each model. This score quantifies the likelihood of a model being preferred by a user in a head-to-head battle, providing a robust, statistically sound ranking system. Unlike simple win/loss ratios, the Bradley-Terry model accounts for the relative strengths of opponents, yielding a more accurate and stable leaderboard.

Beyond a single, overall ranking, LMArena has demonstrated the ability to provide granular performance analysis. Users can categorize their prompts (e.g., math, coding, creative writing), allowing the platform to generate category-specific leaderboards. This reveals how model rankings shift across different domains, highlighting specific strengths and weaknesses. The Arena Explorer further enhances this by employing clustering algorithms on user prompts to uncover latent, second-level categories (e.g., web development vs. SQL programming within "coding"), providing even deeper insights into model behavior.

A significant technical advancement highlighted is the Prompt-to-Leaderboard technique. This involves training a meta-model on millions of historical comparison data points to predict a leaderboard specific to any given user prompt. This capability enables instant, personalized leaderboards tailored to individual use cases. Practical applications include intelligent LM routers that can direct queries to the best-performing model for a specific task in real-time. Furthermore, by integrating cost data, the platform can implement cost-constrained routers, maintaining high performance while achieving significant cost savings (e.g., 2-3x in observed scenarios).

The talk also underscored LMArena's successful expansion into new modalities, demonstrating the platform's versatility. Web Arena allows users to prompt models to build live websites from natural language, with human evaluators judging the generated sites after real-time code compilation and sandboxed execution. Text-to-Image evaluation provides human feedback for generative image models. Repo Chat enables users to interact with models by providing context from a GitHub repository, facilitating evaluation of advanced coding and understanding tasks.

Since its launch in April 2023, LMArena has accumulated impressive statistics: over 100 million user queries, approximately 3 million user votes, and a monthly user base of around 900,000 unique users. It has successfully ranked over 300 models and received strong support and recognition from frontier AI labs (OpenAI, Google, Meta, XAI) and industry leaders like Jeff Dean, Elon Musk, Sam Altman, and Demis Hassabis. These figures and endorsements validate LMArena's efficacy and impact as a crucial platform for next-generation AI evaluation.

Technical Deep Dive

▶ Watch: Introducing Chatbot Arena: a live, community-driven platform (10:00)

LMArena's technical architecture is built around a robust, scalable system designed to capture, process, and analyze massive volumes of human feedback for LLM evaluation. At its core is a sophisticated data collection and ranking methodology, complemented by advanced analytical tools and an extensible framework for multimodal AI.

Crowdsourced Evaluation Mechanism

The foundation of LMArena is its crowdsourced human feedback loop. When a user interacts with the platform, they submit a natural language prompt. The system then randomly selects two models from its extensive pool, anonymizes their identities (e.g., Model A and Model B), and generates responses. This anonymized pairwise comparison is crucial for mitigating bias, as users are forced to evaluate responses purely on merit before knowing the underlying model. After reviewing both responses, the user casts a vote indicating which model provided a better answer, or if they are comparable. This interaction generates a "battle" datum, consisting of the user's prompt (representing a real-world use case), the two sampled model responses, and the user's preference vote.

Leaderboard Generation with Bradley-Terry Model

To translate these discrete pairwise votes into a comprehensive ranking, LMArena employs the Bradley-Terry model. This statistical model is a well-established method for deriving a ranking from paired comparisons. For a set of items (models in this case), the Bradley-Terry model estimates a latent "strength" parameter for each item. The probability that item i beats item j in a pairwise comparison is modeled as a function of their respective strength parameters. Specifically, if $s_i$ and $s_j$ are the strengths of models $i$ and $j$, the probability that $i$ is preferred over $j$ is $P(i > j) = \frac{e^{s_i}}{e^{s_i} + e^{s_j}}$.

LMArena performs a regression based on all collected votes to estimate these strength parameters, which are then normalized and transformed into the Arena Score. This score directly represents the likelihood of a model being preferred by a user. The use of the Bradley-Terry model is vital because it accounts for the relative difficulty of battles (e.g., beating a strong model yields more "credit" than beating a weak one) and provides a more stable and accurate ranking than simple win/loss ratios, especially when dealing with an imbalanced number of comparisons per model. The leaderboard is continuously updated as new votes are submitted, making it a "live" benchmark.

Granular Analysis and Arena Explorer

To provide deeper insights, LMArena goes beyond a single overall leaderboard. It allows for category-specific leaderboards, where users can explicitly tag their prompts (e.g., "coding," "creative writing," "math"). The system then filters votes by these categories and recomputes Bradley-Terry rankings, revealing how models perform in different domains. This helps identify specific strengths and weaknesses.

The Arena Explorer further refines this granular analysis. It applies clustering algorithms to the vast collection of user prompts. By automatically grouping similar prompts, the Explorer uncovers latent categories and sub-categories without requiring manual tagging. For example, within a "programming" cluster, it might identify sub-clusters like "web development," "C++ programming," or "SQL queries." Users can then drill down into these clusters to see the distribution of questions and specific user prompts, understanding precisely what types of tasks models excel or struggle with. This data-driven approach allows for an unprecedented level of detail in model performance breakdown.

Prompt-to-Leaderboard Technique

One of the most innovative technical developments is the Prompt-to-Leaderboard technique. This involves training a separate machine learning model, a meta-model, on the entire dataset of millions of user prompts and their corresponding battle outcomes (which models won/lost). The objective of this meta-model is to predict a specific leaderboard—or more accurately, the relative Arena Scores—for any given new prompt.

This model learns the nuances of how different prompt characteristics influence model preferences. Once trained, it can instantly generate a personalized leaderboard tailored to a user's exact use case (i.e., their specific prompt). This capability unlocks several powerful applications:

• LM Router: In a production setting, an LM router can leverage this model to automatically route an incoming user query to the best-performing LLM for that specific query, optimizing user experience.
• Cost-Constrained Router: By incorporating the cost of inference for each model, the system can formulate an objective to maximize performance while adhering to a budget. The Prompt-to-Leaderboard model provides the performance estimates, allowing the router to select the most cost-effective model that still meets performance targets. The talk reported significant cost savings of 2-3x while maintaining equivalent performance using this technique.
• Model Weakness Analysis: Developers can use this tool to quickly identify specific types of prompts or domains where their model underperforms, facilitating targeted debugging and improvement.

Expansion to New Modalities

LMArena's design is extensible, allowing it to evaluate models beyond traditional text-based chatbots.

• Web Arena: This modality targets text-to-website generation models. Users provide a natural language prompt (e.g., "build a clone of Hacker News"). The models generate code (e.g., HTML, CSS, JavaScript) in real-time. This code is then compiled in the browser and executed within a sandboxed environment, presenting a live, interactive website to the user. Human evaluators then judge the quality and correctness of the generated website. This complex evaluation task requires sophisticated infrastructure to handle code generation, compilation, and secure execution.
• Text-to-Image: For generative image models, the platform collects human feedback on the quality, relevance, and aesthetic appeal of images generated from text prompts. The UI is being continuously improved to streamline this subjective evaluation.
• Repo Chat: This innovative modality allows users to provide a link to a GitHub repository. The model gains context from the codebase, enabling users to "chat" with it about issues, pull requests, or specific code sections. This evaluates a model's ability to understand and reason about complex codebases, perform advanced coding tasks, and assist with software development workflows. This requires significant engineering to ingest and process large code repositories and provide them as context to the LLMs.

The underlying infrastructure for LMArena must be highly scalable to handle millions of queries, concurrent model inferences from various providers, and the storage and processing of vast amounts of feedback data. While not explicitly detailed in the talk, the system likely employs cloud-native services, distributed databases, and high-throughput APIs to manage its operations and support a diverse range of models and modalities.

Experimental Setup & Results

▶ Watch: How Chatbot Arena works: anonymous model comparison (13:00)

LMArena, by its very nature, is a continuous, large-scale experiment in crowdsourced AI evaluation. Its "experimental setup" is the live platform itself, constantly collecting data from a global community of users.

Data Collection and Scale:

Since its launch in April 2023, LMArena has amassed an impressive dataset:

• User Queries: Over 100 million user queries.
• User Votes: Approximately 3 million pairwise comparison votes.
• Unique Users: Around 900,000 unique users monthly.
• Models Ranked: The platform has ranked over 300 models from various developers and research labs.
• Multilingual Support: Data is collected across many different languages, ensuring diverse evaluation.
• Category Distribution: Significant portions of the data now fall into specific categories, with coding and creative writing each accounting for over 10-20% of the total, indicating robust use across diverse tasks.

Baselines and Context:

The talk effectively frames LMArena's necessity by highlighting the shortcomings of existing static benchmarks. It cites the Llama 3 technical report, which found that most traditional benchmarks suffer from significant contamination. Specifically, widely used benchmarks like MMLU were shown to have contamination ratios exceeding 80%. This means a large proportion of questions in these benchmarks were likely present in the training data of the models being evaluated, leading to inflated scores and unreliable assessments of true generalization. This pervasive issue underscored the need for a "live benchmark" like LMArena, where every question is fresh and user-generated, intrinsically mitigating contamination risks.

Key Results and Headline Numbers:

• Overall Leaderboard: The primary output is the constantly updated leaderboard, which ranks models based on their Arena Score (derived from the Bradley-Terry model), indicating their likelihood of being preferred by users. This provides a real-time, community-validated ranking of state-of-the-art LLMs.
• Granular Performance: Category-specific leaderboards demonstrate how model performance shifts across different domains (e.g., a model might excel in creative writing but underperform in complex coding tasks). The Arena Explorer further refines this by algorithmically clustering prompts into fine-grained categories, allowing users to delve into specific sub-domains like "SQL queries" within "programming."
• Prompt-to-Leaderboard Efficacy: The new technique for training a model to predict a prompt-specific leaderboard showed significant practical benefits. When applied to LM routing, this approach can maintain the same level of performance while achieving 2-3x cost savings by intelligently selecting the most suitable and cost-efficient model for each query. This directly translates to substantial operational efficiencies for deploying LLMs at scale.
• Web Arena Engagement: The Web Arena platform, allowing users to prompt models to build websites, has already collected 200,000 votes from the community since its launch a few months prior to the talk. This rapid engagement highlights the demand for evaluating models on more complex, interactive tasks. Notably, models like Gemini 2.5 quickly reached the top of this specialized leaderboard, demonstrating the utility of Web Arena in identifying cutting-edge capabilities.

Hardware and Frameworks:

While the talk does not specify the exact hardware or software frameworks used to build and operate LMArena itself, it implicitly relies on a scalable cloud infrastructure capable of hosting numerous LLMs, handling high query volumes, performing real-time inference, and processing large-scale data for leaderboard computation. The models being evaluated would span a wide range of architectures and training setups, from proprietary frontier models (e.g., GPT-4, Claude, Gemini) to open-source alternatives (e.g., Llama, Mistral), utilizing various GPU/TPU hardware configurations in their respective inference environments.

Practical Implications

▶ Watch: Computing the leaderboard: Bradley-Terry model and Arena score (14:00)

LMArena's innovative approach to AI evaluation carries profound practical implications for a wide spectrum of stakeholders in the machine learning ecosystem. By providing a dynamic, real-world, and crowdsourced benchmark, it addresses critical needs for reliability, efficiency, and continuous improvement in the age of generative AI.

For Practitioners (Developers, Researchers):

• Reliable Model Comparison: Practitioners gain access to a continuously updated, real-world leaderboard that is less susceptible to data contamination. This enables more trustworthy comparisons between models, helping them select the most appropriate LLM for their specific applications.
• Identifying Strengths and Weaknesses: The granular, category-specific leaderboards and the Arena Explorer allow developers to pinpoint precisely where models excel and where they fall short. This is invaluable for understanding model behavior, guiding feature engineering, and selecting models that align with task requirements (e.g., a model strong in creative writing might not be the best for complex coding).
• Faster Feedback Loop: Model developers can quickly get real-world feedback on their new releases, accelerating the iteration cycle. This "live benchmark" approach means evaluations evolve as quickly as models do, providing relevant insights almost immediately.

For Infrastructure Teams (MLOps, Deployers):

• Optimized Model Routing: The Prompt-to-Leaderboard technique and the resulting LM Router are game-changers for MLOps and infrastructure teams. Instead of relying on static configurations, queries can be dynamically routed to the best-performing model for a given prompt, optimizing user experience and resource utilization.
• Cost Efficiency: The cost-constrained router directly addresses a major concern for deploying large models: inference cost. By allowing teams to define cost ceilings, the system can select models that deliver satisfactory performance while achieving significant cost savings (demonstrated at 2-3x). This is crucial for making LLM deployments economically viable at scale.
• Enhanced Monitoring: LMArena provides a framework for monitoring AI performance in real-world scenarios, identifying regressions or shifts in user preferences over time. This helps maintain the quality and reliability of deployed AI systems.

For Model Builders (AI Labs, Open-Source Contributors):

• Mitigating Test Leakage: By relying on fresh, user-generated prompts, LMArena inherently mitigates the pervasive problem of test leakage and data contamination that plagues static benchmarks. This provides a more honest assessment of a model's true generalization capabilities.
• Guiding Model Development: The detailed insights into model performance across various categories and prompt clusters offer clear directions for future research and development. Builders can focus on improving specific weaknesses identified by the community, leading to more robust and versatile models.
• Benchmarking New Modalities: The platform's expansion to modalities like Web Arena, Text-to-Image, and Repo Chat provides critical infrastructure for evaluating novel AI capabilities that extend beyond traditional chat, pushing the boundaries of what can be reliably benchmarked.

Tradeoffs and Limitations:

While LMArena offers substantial advantages, it also involves certain tradeoffs:

• Subjectivity of Human Feedback: While crowdsourcing scales human evaluation, it inherently introduces subjectivity and potential biases among annotators. However, the large volume of votes and the use of statistical models like Bradley-Terry help to average out individual biases.
• Dependency on Community Engagement: The platform's effectiveness relies on a continuously active and diverse user community providing prompts and votes. Sustaining this engagement is crucial.
• Latency for New Models: While "live," new models still need to accumulate a sufficient number of votes to achieve a stable and reliable Arena Score, which can take some time.
• Infrastructure Complexity: Operating a platform that hosts numerous LLMs from various providers, handles real-time inference, and processes millions of data points is a significant engineering challenge.

In summary, LMArena transforms AI evaluation from a static, academic exercise into a dynamic, practical, and community-driven process. Its implications range from improving model selection and deployment efficiency to accelerating research and development, ultimately fostering a more robust and reliable AI ecosystem.

Key Takeaways

• Dynamic, Crowdsourced Evaluation: LMArena provides a critical solution to the challenges of evaluating rapidly evolving generative AI models by leveraging crowdsourced human feedback, offering a "live benchmark" that constantly refreshes with real-world user queries.
• Mitigation of Test Contamination: By using fresh, user-generated prompts for every evaluation, LMArena inherently avoids the pervasive issue of data contamination that plagues traditional static benchmarks like MMLU (which can have over 80% contamination).
• Robust Ranking with Bradley-Terry: The platform employs the Bradley-Terry model to compute an Arena Score, providing a statistically sound and stable ranking of models based on their likelihood of being preferred by users in anonymized pairwise comparisons.
• Granular Performance Insights: LMArena offers detailed analysis beyond overall scores, including category-specific leaderboards and the Arena Explorer (using clustering algorithms) to identify specific model strengths and weaknesses across diverse domains and fine-grained task types.
• Prompt-to-Leaderboard for Intelligent Routing: The innovative Prompt-to-Leaderboard technique enables the system to predict a custom leaderboard for any given prompt, facilitating intelligent LM routing and cost-constrained routing that can achieve 2-3x cost savings while maintaining performance.
• Expansion to Multimodal AI: The platform is actively expanding to evaluate new modalities, including Web Arena (text-to-website generation), Text-to-Image, and Repo Chat (interactive codebase understanding), demonstrating its versatility and commitment to comprehensive AI assessment.

About the Speaker(s)

Wei-Lin Chiang is a researcher from UC Berkeley. He is a key figure behind the development of LMArena, an open platform designed for crowdsourced AI benchmarks and large-scale language model evaluation using human feedback. His work focuses on addressing the evolving challenges of AI assessment in the era of generative models, particularly concerning the subjectivity and rapid development cycle of these advanced systems. Chiang's presentation at MLSys 2025 highlighted his team's efforts in building LMArena into a community-driven platform that has garnered significant support from both the academic community and frontier AI labs.