Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards

Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards

Speakers: Yangsibo Huang, Milad Nasr, Anastasios Angelopoulos, Nicholas Carlini, Wei-Lin Chiang, Christopher A. Choquette Choo, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Ken Ziyu Liu, Ion Stoica, Florian Tramer, Chiyuan Zhang

Conference: ICML 2025

YouTube: https://slideslive.com/39044087

Overview

This talk, presented by Christopher Choquette Choo on behalf of lead author Yangsibo Huang and a large team of collaborators, delves into the vulnerabilities inherent in voting-based benchmarks, a rapidly growing paradigm for evaluating large language models (LLMs). While traditional static benchmarks like GSM8K and MMLU suffer from issues such as limited diversity, small example sets, and data contamination risks due to their fixed and published nature, voting-based platforms offer an interactive, online alternative where users directly compare and rank model responses. The most prominent example, Chatbot Arena (or LMSYS Arena), boasts over 3.5 million votes to date and relies heavily on model anonymity to ensure unbiased evaluation. This research uncovers critical methods for breaking this anonymity and subsequently manipulating model rankings, highlighting significant challenges for the integrity of these popular evaluation systems.

The talk systematically explores two primary adversarial strategies: de-identification attacks and re-ranking manipulation. It demonstrates how an adversary can effectively determine the identity of an anonymous model based on its unique response style, and then leverage this knowledge to strategically cast votes that alter a model's standing on the leaderboard. The research quantifies the effort required for such attacks and proposes a suite of potential mitigations, ranging from enhanced authentication to anomaly detection. This work is crucial for the AI/ML community as it addresses the pressing need to develop robust and trustworthy evaluation methodologies for LLMs, ensuring that leaderboards accurately reflect model performance rather than susceptibility to adversarial tactics.

Background

▶ Watch: Introduction to voting benchmarks and adversarial manipulation (0:00)

The landscape of AI model evaluation has traditionally relied on static benchmarks, collections of predefined questions or tasks against which models are tested. While these benchmarks, such as GSM8K for mathematical reasoning or MMLU for multi-task language understanding, have been foundational, they present inherent limitations. Their fixed nature often leads to a relatively small, curated set of examples, which can lack the diversity needed to comprehensively assess complex LLMs. More critically, the public availability of these benchmarks creates a significant risk of data contamination, where models inadvertently (or intentionally) train on the test data, leading to artificially inflated performance metrics that do not reflect true generalization capabilities.

In response to these challenges, voting-based benchmarks have emerged as a promising alternative. These platforms, exemplified by Chatbot Arena from LMSYS, shift the evaluation paradigm from static tests to dynamic, human-in-the-loop assessments. In Chatbot Arena, users are presented with a chat interface, provide a prompt, and receive two anonymous responses from different language models. The user then ranks these responses based on quality and content, and this vote is aggregated with all other user votes to calculate an Elo rating for each model, ultimately determining its rank on a public leaderboard. A cornerstone of this system is model anonymity, which is designed to prevent user biases (conscious or unconscious) from influencing the voting process and to deter intentional manipulation of rankings. The problem addressed by this research is precisely how this anonymity, and by extension the entire ranking system, can be compromised by sophisticated adversaries. The talk highlights that while these benchmarks offer significant improvements in evaluation diversity and interactivity, they introduce new attack surfaces that must be understood and mitigated to preserve their utility and trustworthiness.

Key Findings

▶ Watch: Models reveal unique styles enabling de-identification (1:10)

The research presents several critical findings related to the vulnerabilities of voting-based leaderboards, primarily focusing on de-identification and re-ranking attacks, and then proposing a framework for mitigations.

Firstly, the study definitively demonstrates that model anonymity can be broken. The core insight is that different language models possess unique "styles" in their responses. By collecting a corpus of responses and analyzing them using bag-of-words features and Principal Component Analysis (PCA), the researchers found that responses from distinct models tend to form relatively distinct clusters, indicating a quantifiable stylistic signature. Two methods for de-identification were explored: a simple direct query (asking the model its name, which surprisingly often works due to models following instructions) and a more robust supervised learning approach. Using a logistic regression classifier trained on responses from various models, the adversary can achieve high accuracy in identifying an anonymous model. A standout result is the ability to de-anonymize the trainer of a model with up to 95% accuracy, a significant breach of the anonymity principle. The effectiveness of this attack varies slightly across different prompt domains, with specialized areas like math and code yielding slightly higher de-anonymization accuracy.

Secondly, the research quantifies the feasibility of re-ranking attacks on leaderboards. By simulating an adversarial environment using historical LMSYS Arena data, the team determined the number of votes required to manipulate a model's rank. For high-ranked models, it takes approximately 500 votes to move a model up or down a single Elo spot. Lower-ranked models are significantly easier to move, requiring far fewer votes (as they have accumulated fewer total votes). However, since models are chosen randomly for head-to-head comparisons, an adversary needs to log roughly two orders of magnitude more interactions than the required votes to reliably encounter and vote on the target model. The study also found that the adversary's voting behavior (e.g., randomly voting when the target model isn't present, or attempting to hide their malicious intent) does not significantly impact the overall attack performance.

Finally, the talk introduces a cost model for attacks and proposes a range of mitigation strategies. The cost model breaks down the adversary's expenses into three main components: the account cost (obtaining accounts), the action cost (cost per vote/interaction), and the detector cost (training the de-identification model, estimated at roughly $500 in API costs). Proposed mitigations include standard authentication techniques like two-factor authentication (2FA) or requiring valid credentials to increase account cost. Rate limiting can force adversaries to acquire more accounts. More sophisticated defenses involve anomaly detection to identify malicious voting patterns, potentially by perturbing votes to make adversaries distinguishable. However, this introduces a security-utility tradeoff, as perturbed votes can lead to perturbed (less accurate) rankings. Increasing the action cost through measures like captchas or enforcing unique prompts for evaluation are also suggested. The key takeaway here is that while vulnerabilities are significant, a multi-faceted approach to defense can largely protect against these types of manipulations, though often with associated costs or compromises to user experience or ranking accuracy.

Technical Deep Dive

▶ Watch: Attack success: 95% accuracy in model de-anonymization (2:40)

The technical core of this research revolves around two sophisticated attack vectors: de-identification and re-ranking, each supported by specific methodologies, and a subsequent exploration of mitigation strategies.

The de-identification attack leverages the inherent stylistic differences between LLMs. The fundamental premise is that each model, due to its architecture, training data, and fine-tuning, develops a unique "voice" or pattern of generating text. To quantify this, the researchers employ bag-of-words (BoW) features. For each model response, a vector is created representing the frequency of words (or n-grams) present in the text. This transforms qualitative textual style into a quantitative numerical representation. To visualize these differences, Principal Component Analysis (PCA) is applied to these BoW feature vectors. PCA reduces the high-dimensional feature space to a lower dimension (e.g., 2D or 3D) while preserving as much variance as possible, allowing for the visualization of distinct clusters corresponding to different models.

Beyond visualization, two practical de-identification methods are proposed:

1. Direct Query: A surprisingly effective, albeit simple, method involves directly asking the anonymous model for its name. Given that LLMs are trained to follow instructions, many models will truthfully respond with their identity.
2. Supervised Learning Classifier: For a more robust and generalizable approach, a logistic regression classifier is trained. This classifier takes the BoW feature vector of a model's response as input and outputs a prediction of the model's identity. The training process involves collecting a diverse set of responses from various known models. For a specific target model, its responses form the "positive class," while responses from all other models form the "negative class." The classifier is trained to distinguish between the target model's style and the styles of other models. The accuracy of this classifier, noted to be up to 95% for identifying model trainers, underscores its efficacy. The research also investigates how different prompt domains (e.g., high-resource languages, low-resource languages, specialized areas like math and code) affect de-identification accuracy, finding that specialized domains often yield slightly better results due to potentially more distinct stylistic patterns.

The re-ranking attack strategy focuses on manipulating the Elo rating system used by platforms like LMSYS Arena. Elo is a method for calculating the relative skill levels of players in competitor-versus-competitor games, and it's adapted here for models. When a user votes, the Elo ratings of the two models involved are updated based on the outcome. An adversary's goal is to strategically cast votes to artificially inflate or deflate a target model's Elo score.

To quantify this, the researchers built a simulated environment using historical LMSYS Arena data. This simulation allows them to replay interactions and observe the impact of adversarial votes on Elo rankings. The simulation models the process where an adversary logs votes, specifically targeting a model to move it up or down one Elo spot. A key challenge for the adversary is that models are chosen randomly for comparison. This means that to achieve a certain number of votes for or against a target model, the adversary must engage in a significantly larger number of interactions (i.e., making many votes where the target model is not even present). The study quantifies this, showing that achieving the necessary votes often requires two orders of magnitude more interactions. The research also explored various adversary behaviors (e.g., voting randomly when the target is not present, or trying to hide their malicious intent) and found that these variations had minimal impact on the overall success of the attack, suggesting the attack is robust to such attempts at camouflage.

The proposed mitigations are structured around increasing the cost model for the adversary. This cost model has three components:

1. Account Cost: The expense associated with creating or acquiring valid user accounts. Mitigations include:

• Two-Factor Authentication (2FA): Requires a second verification step, making account creation more difficult and costly.
• Requiring Valid Credentials: Moving from simple email registration to more robust identity verification. This shifts the cost from zero to the cost of obtaining valid, unique credentials. However, it introduces a tradeoff: users might be de-anonymized in the process, potentially changing their voting habits.

1. Action Cost: The expense or effort required for each vote or interaction. Mitigations include:

• Rate Limiting: Restricting the number of votes per account over a given period, forcing adversaries to acquire more accounts.
• Captchas: Introducing challenges (e.g., "I'm not a robot" tests) before voting, increasing the manual effort per vote.
• Enforcing Unique Prompts: Requiring users to submit genuinely unique prompts for evaluation, making automated, repetitive voting more difficult.

1. Detector Cost: The expense of training and maintaining the de-identification classifier. Mitigations here are less direct but involve making it harder for the adversary to collect clean training data.

Finally, Anomaly Detection Techniques are discussed. These involve monitoring voting patterns for statistical anomalies that might indicate adversarial activity. One proposed method is to perturb votes, meaning the platform slightly alters the recorded vote or the Elo calculation for certain users. The idea is to make malicious voting behavior distinguishable from benign behavior. However, this introduces a direct security-utility tradeoff: while it can make the defense stronger, deliberately perturbing votes also leads to perturbed (less accurate) rankings for all models, potentially undermining the very purpose of the leaderboard. The challenge is to find a balance where adversarial manipulation is deterred without significantly degrading the quality of the evaluation.

Experimental Setup & Results

▶ Watch: Quantifying the risk of re-ranking models on leaderboards (2:50)

The experimental methodology employed in this research involved both real-world data simulation and controlled classifier training to demonstrate the feasibility and quantify the impact of adversarial attacks.

For the de-identification attack, the core experiment involved training a logistic regression classifier to identify anonymous models.

• Datasets: The researchers collected "a bunch of different responses for a bunch of different models on a set of prompts." While specific model names and exact dataset sizes were not detailed in the talk, the implication is a diverse collection of LLM outputs across various prompts. The talk did specify exploring different "domains of prompts," including high-resource languages, low-resource languages, and specialized areas like math and code, to observe variations in attack efficacy.
• Baselines: The simplest de-identification method (directly asking the model its name) served as a qualitative baseline, demonstrating that even rudimentary attacks can sometimes succeed. The logistic regression classifier then provided a more robust and quantitative measure.
• Metrics: The primary metric was de-anonymization accuracy, representing the percentage of times the classifier correctly identified the anonymous model.
• Headline Numbers: A significant result was achieving up to 95% accuracy in de-anonymizing the trainers of models. This high accuracy underscores the distinct stylistic fingerprints left by different model architectures and training methodologies.
• Ablations/Variations: The study observed "some slight deviations" in attack accuracy across different prompt domains. Specifically, specialized areas like math and code "tend to have a better attack accuracy," suggesting that these domains might elicit more distinctive or less ambiguous stylistic patterns from models, making them easier to identify.

For the re-ranking attack, the experiments were conducted within a simulated environment to precisely control and quantify adversarial actions.

• Datasets: The simulation was built using historical LMSYS Arena data. This real-world data provided a realistic distribution of votes, model pairings, and Elo score dynamics to ensure the simulation's fidelity.
• Methodology: An adversary was simulated to "log votes" with the specific goal of moving a target model up or down "exactly one spot" in its Elo ranking. The simulation tracked the number of votes cast and the number of total interactions (where the adversary might or might not encounter the target model).
• Metrics: Key metrics included:
• Number of votes required to achieve a one-spot rank change for a model.
• Number of interactions required to accumulate those votes.
• Headline Numbers: The results showed that re-ranking high-ranked models required approximately 500 votes to move them one spot. Lower-ranked models, having fewer total votes, were "much easier" to move, requiring significantly fewer votes. However, due to the random pairing of models in Chatbot Arena, the adversary needed to log "roughly two orders of magnitude more interactions" than the direct votes to achieve the desired effect. For example, to cast 500 targeted votes, the adversary might need to participate in tens of thousands of interactions.
• Ablations/Variations: The researchers also simulated different adversarial behaviors, such as voting randomly when the target model was not present or attempting to hide voting patterns. They found "there isn't much difference in performance across them," indicating that the attack's effectiveness is largely robust to such behavioral variations.

The mitigation cost model was also quantified, with the estimated cost of obtaining a detector (primarily through API calls for generating responses) being roughly $500. This figure helps frame the economic barrier for an adversary to launch a sophisticated de-identification attack.

Practical Implications

▶ Watch: Developing a cost model for the adversarial attack (3:40)

The findings of this research carry significant practical implications for various stakeholders in the AI/ML ecosystem, particularly as voting-based benchmarks continue to gain traction for evaluating large language models.

For practitioners and model builders, the primary implication is the newfound understanding that model anonymity on platforms like Chatbot Arena is not guaranteed. Models possess distinct stylistic signatures that can be exploited for de-identification, even without direct access to model internals. This means that performance on such leaderboards, while seemingly unbiased, could be susceptible to targeted manipulation. Model developers should be aware that their models' unique "voices" could be used against them, potentially leading to misattribution or strategic downranking by competitors. This could also influence how models are fine-tuned, potentially encouraging developers to consider making their models' outputs less stylistically distinct, though this could have other performance implications.

For infrastructure teams managing these evaluation platforms, such as the LMSYS Arena team, the research provides a clear roadmap of vulnerabilities and potential defenses. The revelation that de-identification attacks can achieve 95% accuracy and that re-ranking requires a quantifiable number of votes (e.g., 500 for high-ranked models) necessitates the implementation of robust security measures. The proposed cost model for attacks offers a framework for prioritizing and designing mitigations. Implementing measures like two-factor authentication (2FA) or stricter credential validation can significantly increase the "account cost" for adversaries, making large-scale attacks more expensive. Rate limiting can curb the volume of votes from individual malicious accounts. More advanced anomaly detection techniques are crucial for identifying unusual voting patterns that might indicate adversarial activity, though these come with the inherent security-utility tradeoff of potentially perturbing legitimate rankings.

The research also highlights critical tradeoffs and limitations inherent in securing these systems. Increasing account costs through 2FA, for instance, might de-anonymize users, potentially influencing their voting behavior and undermining the very goal of unbiased evaluation. Similarly, "perturbing votes" for anomaly detection, while effective in deterring adversaries, directly impacts the accuracy of the Elo rankings, creating a dilemma between security and the utility of the leaderboard. Infra teams must carefully balance these considerations, choosing mitigations that provide sufficient security without unduly compromising user experience or the integrity of the evaluation process.

Ultimately, this work underscores that the development of truly robust and adversarial-proof evaluation methodologies for LLMs is an ongoing and critical research direction. As AI systems become more powerful and their evaluation more central to their adoption, understanding and mitigating these sophisticated manipulation techniques will be paramount for maintaining trust and ensuring fair assessment of model capabilities.

Key Takeaways

• Voting-based benchmarks like Chatbot Arena are growing in popularity as an alternative to static benchmarks but introduce new, significant vulnerabilities.
• Model anonymity can be effectively broken using de-identification attacks that leverage unique stylistic patterns in model responses, achieving up to 95% accuracy in identifying model trainers via logistic regression classifiers on bag-of-words features.
• Leaderboard rankings can be manipulated: It requires approximately 500 votes to move a high-ranked model by one Elo spot, although due to random model selection, an adversary needs to log two orders of magnitude more interactions to achieve these targeted votes.
• Mitigation strategies exist but involve tradeoffs: Defenses like 2FA, rate limiting, and anomaly detection can increase the adversary's cost and effort. However, these often come with compromises to user anonymity, user experience, or the accuracy of the underlying ranking system (e.g., through vote perturbation).
• Robustness is a critical research direction: The findings emphasize the urgent need for further research into developing evaluation methodologies that are not only strong and effective but also resilient to adversarial manipulation and manipulation attempts.

About the Speaker(s)

The talk was presented by Christopher A. Choquette Choo, on behalf of the lead author Yangsibo Huang, who was unable to attend the conference. The research is a collaborative effort involving a large team of distinguished researchers from various institutions, including Milad Nasr, Anastasios Angelopoulos, Nicholas Carlini, Wei-Lin Chiang, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Ken Ziyu Liu, Ion Stoica, Florian Tramer, and Chiyuan Zhang. While specific individual bios were not provided in the transcript, the breadth of the author list suggests a highly interdisciplinary team with expertise spanning machine learning, systems, and security, reflecting the complex nature of the problem addressed.