Twitter Algorithmic Bias Bug Bounty Challenge Results

Rumman Chowdhury (Director, ML Ethics, Transparency and Accountability (META) · Twitter), Jutta Williams (Product Manager, META · Twitter), Kyra Yee (Research Engineer, META · Twitter), Irene Font Peradejordi (Twitter), Ariel Herbert-Voss (Research Scientist · OpenAI), Peiter "Mudge" Zatko (Twitter), Patrick Hall (Principal Scientist · bnh.ai)

AI Village @ DEF CON 29 · Day 3 · AI Village

Overview

This talk presents the groundbreaking results of Twitter's Algorithmic Bias Bug Bounty Challenge, the first-of-its-kind public initiative in the industry. Hosted at DEF CON 29's AI Village, this challenge invited a global community of researchers, developers, and enthusiasts to scrutinize Twitter’s saliency algorithm, also known as its image crop algorithm, for potential biases. The challenge specifically tasked participants with identifying representational and allocative harms, fostering a crucial dialogue around the ethical implications of machine learning systems.

Watch on YouTube

Visual summary for Twitter Algorithmic Bias Bug Bounty Challenge Results by Rumman Chowdhury, Jutta Williams, Kyra Yee, Irene Font Peradejordi, Ariel Herbert-Voss, Peiter "Mudge" Zatko, Patrick Hall
Visual summary for Twitter Algorithmic Bias Bug Bounty Challenge Results by Rumman Chowdhury, Jutta Williams, Kyra Yee, Irene Font Peradejordi, Ariel Herbert-Voss, Peiter "Mudge" Zatko, Patrick Hall

Key moments

  1. 0:00 Introducing Twitter's Algorithmic Bias Bounty Challenge
  2. 11:00 First place winner identifies stereotypical beauty standard bias
  3. 17:10 Honorable mention: bias against religious head coverings
  4. 25:50 Mudge on algorithms reflecting societal biases
  5. 34:00 Most generalizable: 'Invisible Borders' adversarial attack
  6. 57:30 Q&A: Discussing critical issues of data provenance in ML
  7. 62:40 Closing thoughts and call to action for future challenges

Twitter Algorithmic Bias Bug Bounty Challenge Results

Speakers: Rumman Chowdhury (Director, ML Ethics, Transparency and Accountability (META), Twitter); Jutta Williams (Product Manager, META, Twitter); Kyra Yee (Research Engineer, META, Twitter); Irene Font Peradejordi (Twitter); Ariel Herbert-Voss (Research Scientist, OpenAI); Peiter "Mudge" Zatko (Twitter); Patrick Hall (Principal Scientist, bnh.ai)

Conference: AI Village @ DEF CON 29

YouTube: https://www.youtube.com/watch?v=KLwdzLiJywI

Overview

This talk presents the groundbreaking results of Twitter's Algorithmic Bias Bug Bounty Challenge, the first-of-its-kind public initiative in the industry. Hosted at DEF CON 29's AI Village, this challenge invited a global community of researchers, developers, and enthusiasts to scrutinize Twitter’s saliency algorithm, also known as its image crop algorithm, for potential biases. The challenge specifically tasked participants with identifying representational and allocative harms, fostering a crucial dialogue around the ethical implications of machine learning systems.

Led by Dr. Rumman Chowdhury, Director of Twitter's Machine Learning Ethics, Transparency and Accountability (META) team, and supported by a diverse panel of internal experts and external judges, the initiative aimed to uncover unforeseen biases at an unprecedented pace. The challenge garnered widespread participation from five continents, attracting entries from individuals, startups, large corporations, and even high school students. Its success not only highlighted specific vulnerabilities within Twitter's algorithm but also established a significant precedent for how companies can proactively address algorithmic bias through open, collaborative, and incentivized mechanisms, akin to traditional security bug bounties.

The findings presented in this talk reveal a wide array of biases, ranging from linguistic and demographic preferences to the reinforcement of stereotypical beauty standards and susceptibility to subtle adversarial attacks. The insights gained underscore the complex interplay between societal biases, training data, and algorithmic outcomes, emphasizing the critical need for diverse perspectives in the design, evaluation, and continuous improvement of AI systems. This challenge represents a pivotal step towards more transparent and accountable machine learning, demonstrating a viable path for translating academic discussions on AI ethics into tangible, product-level improvements.

Background

▶ Watch: Introducing Twitter's Algorithmic Bias Bounty Challenge (0:00)

The proliferation of machine learning (ML) algorithms across social media platforms has brought immense utility but also significant ethical challenges. One such challenge lies in the potential for these algorithms to inadvertently perpetuate or amplify existing societal biases, leading to unintended harms. Twitter's image crop algorithm, designed to automatically crop images for optimal display in feeds, became a focal point for these concerns. Initially, internal research, spearheaded by Kyra Yee and her team, had already begun to address these issues, but the scope of potential biases remained vast and often unpredictable.

The core problem stems from the inherent nature of ML systems: they learn from data, and if that data reflects human biases, the models will inevitably encode and operationalize them. These biases can manifest as representational harms, where certain groups are unfairly or inaccurately portrayed, or allocative harms, where resources or opportunities are unfairly distributed. For an image cropping algorithm, this could mean consistently excluding certain demographics from cropped images, thereby reducing their visibility and engagement on the platform. The challenge specifically targeted Twitter's saliency algorithm, which attempts to identify the most "interesting" or "salient" part of an image to center the crop.

Recognizing the limitations of internal teams in identifying the full spectrum of potential biases, particularly those tied to diverse global cultures and experiences, Twitter's META team drew inspiration from the success of traditional security bug bounty programs. Jutta Williams, a product manager with a background in security engineering, highlighted how bug bounties revolutionized the security industry by democratizing vulnerability discovery. The rationale was simple: negative consequences of machine learning are often unpredictable by nature, requiring a multitude of voices and perspectives to uncover them in a safe and open environment. By opening up the challenge globally, Twitter aimed to accelerate the discovery process, tap into a broader range of expertise, and bring academic insights into commercial product development at a speed unachievable through internal efforts alone. This proactive approach sought to empower users and researchers to directly interrogate Twitter's algorithms, fostering a new model for responsible AI development and auditing.

Key Findings

▶ Watch: Honorable mention: bias against religious head coverings (17:10)

The Twitter Algorithmic Bias Bug Bounty Challenge yielded a rich tapestry of findings, highlighting diverse and often unexpected biases embedded within the saliency algorithm. The entries demonstrated the algorithm's propensity to favor certain demographics, linguistic styles, and even digital artifacts, often at the expense of marginalized groups or user intent.

The third-place winner, "Gazing at the Mother Tongue: Analyzing Twitter's Image Cropping Algorithm on Bilingual Memes," uncovered a significant linguistic bias. Researchers created memes containing both English and Arabic text, finding that the algorithm selected the English text over the Arabic at least 90% of the time, regardless of placement. This finding underscored the hegemony of English online and its potential to exacerbate existing social and economic inequalities for non-English speakers.

Second place was awarded to "HALT," which exposed biases against elderly and disabled individuals, as well as those with specific physical characteristics. By constructing a diverse dataset of faces, HALT demonstrated that the algorithm consistently assigned lower saliency scores to individuals with white or gray hair, leading to their exclusion from cropped images. This bias extended to individuals with albinism, often cropping them out entirely. A particularly creative validation method involved artificially whitening the hair of salient faces in images, which resulted in a drop in saliency for that individual in 58% of cases. Furthermore, HALT revealed that the algorithm often cropped out disabled individuals in wheelchairs, especially when seated lower than others in group photos. It also identified instances where the algorithm focused on light objects or logos instead of the face of an individual with very dark skin in uneven lighting conditions.

The first-place entry showcased a sophisticated analysis of how the algorithm encodes and reinforces stereotypical beauty standards, effectively demonstrating a feedback loop between societal perceptions and algorithmic outcomes. This entry used a counterfactual approach to understand what image manipulations would increase saliency scores. The results indicated a preference for faces with lighter skin tones, traditionally feminine characteristics, younger appearances, and thinner features. This finding suggests that the algorithm's training data, likely derived from eye-tracking studies, implicitly embeds and perpetuates these unrealistic beauty ideals.

Beyond the top winners, several honorable mentions further illustrated the breadth of biases:

  • An entry on military versus civilian attire revealed that individuals in camouflage were often cropped out. While the algorithm was technically "working as expected" (camouflage is designed to evade detection), this result highlighted a potential harm to a U.S. protected class (veterans) and the nuance between algorithmic function and human impact.
  • Another submission focused on religious head coverings, demonstrating through "citizen data science" that the algorithm overwhelmingly preferred individuals without head coverings, regardless of the specific religious garment (hijabs, turbans, yamakas). This simple yet impactful finding resonated with how many users experience bias on social media.
  • A "fan favorite" entry creatively demonstrated how the algorithm cropped manga, frequently focusing on the last panel, effectively spoiling the comic. This whimsical yet relevant finding highlighted potential implications for creative influencers and issues of content ownership and presentation.

Finally, two special recognitions were given:

  • The Most Innovative award went to an entry identifying bias in emoji skin tones. It found that lighter skin tone emojis, when added to images, were preferentially selected by the algorithm, revealing how even seemingly innocuous design choices can introduce bias at the system level.
  • The Most Generalizable award recognized an entry that used invisible borders to manipulate saliency scores. By adding small, human-imperceptible digital changes to images, the participant could control the algorithm's focus. This demonstrated a powerful adversarial machine learning technique, highlighting how subtle data manipulations can profoundly alter algorithmic outcomes and potentially be exploited for malicious purposes.

Collectively, these findings underscore the pervasive and multifaceted nature of algorithmic bias, emphasizing the critical need for diverse, external scrutiny of AI systems.

Technical Deep Dive

▶ Watch: Mudge on algorithms reflecting societal biases (25:50)

The Algorithmic Bias Bug Bounty Challenge was hosted on HackerOne, a popular vulnerability disclosure platform, with participants submitting their code and detailed methodologies via GitHub repositories. The core target was Twitter's saliency algorithm, which determines the most visually prominent or "interesting" region within an image for automatic cropping. This algorithm is crucial for how images are presented across different devices and feed layouts on the platform.

The methodologies employed by the winning entries showcased a range of innovative approaches to interrogating the algorithm:

The third-place entry, "Gazing at the Mother Tongue," utilized a straightforward but effective comparative methodology. Researchers took well-known English memes and translated their text into Arabic, carefully maintaining visual parameters like font size and image ratio. They then concatenated these English and Arabic versions side-by-side and fed them to the saliency algorithm. By measuring how often the crop centered on the English versus the Arabic text, they quantitatively demonstrated the linguistic bias, finding the English meme selected at least 90% of the time. This method effectively isolated the linguistic variable to demonstrate bias.

The second-place winner, "HALT," focused on constructing a representative dataset and using targeted image manipulation. They created a custom dataset encompassing a diverse range of faces, varying in skin tone, race, gender, and age. To validate their findings regarding hair color bias, they employed a clever counterfactual editing technique: they used image editing software to artificially whiten the hair of the most salient face in a picture. This manipulation resulted in a significant drop in the original face's saliency score in 58% of cases, causing the algorithm to shift its focus to another, non-white-haired face in the image. This method provided strong causal evidence for the hair color bias. Their analysis also extended to group photos, specifically examining how individuals in wheelchairs were often cropped out due to their lower seating position, indicating a spatial bias.

The first-place entry, which exposed the algorithm's preference for stereotypical beauty standards, employed a sophisticated generative adversarial network (GAN)-based approach. They adopted a counterfactual methodology, asking what specific changes to an image would result in a higher saliency score. To generate and modify test faces, they leveraged StyleGAN2-ADA, a state-of-the-art GAN model pre-trained on the Flickr-Faces HQ (FFHQ) dataset. This model is capable of generating high-quality, photorealistic images of faces from a vector of latent parameters. The researchers would randomly sample a latent vector to create an original face, then systematically perturb this latent vector through a random search in the latent space, maximizing for saliency while ensuring the resulting manipulated face remained visually similar to the original. A "noise parameter" was tuned to maintain this similarity. By qualitatively analyzing the differences between the original and the maximally perturbed faces, they could infer the characteristics favored by the saliency model (e.g., lighter skin, younger appearance, feminine features, thinner faces). While acknowledging limitations such as uncontrolled background perturbations, this method offered a highly intuitive way to dissect the algorithm's preferences.

The "Most Generalizable" honorable mention demonstrated an advanced adversarial attack technique. This entry showed that by adding small, invisible borders to images, the algorithm's saliency scores could be significantly manipulated. These digital changes, imperceptible to the human eye, were sufficient to redirect the algorithm's focus, highlighting the fragility of ML models to subtle data perturbations. This technique falls under the broader umbrella of adversarial machine learning, where inputs are crafted to intentionally mislead a model, and its generalizability suggests that similar vulnerabilities might exist in a wide range of image detection algorithms.

These diverse technical approaches, from simple comparative analysis to advanced GAN-based manipulation and adversarial attacks, collectively provided a comprehensive view of the saliency algorithm's biases and vulnerabilities, underscoring the power of external scrutiny in uncovering complex ML ethical issues.

Demo / Proof of Concept

▶ Watch: Q&A: Discussing critical issues of data provenance in ML (57:30)

While the talk itself did not feature a live demo, the entire algorithmic bias bug bounty challenge served as a large-scale, distributed Proof of Concept (PoC) for identifying and demonstrating biases in a real-world machine learning system. The submissions themselves were, in essence, detailed PoCs, each accompanied by code and descriptive documentation explaining the methodology and findings.

The presentation included visual examples from the winning entries to illustrate the biases. For instance, the second-place winner, HALT, showed images of individuals with white or gray hair being cropped out, as well as an albino individual whose face was excluded from the final crop. Another compelling example depicted a person in a wheelchair in a group photo, where the algorithm's crop conspicuously excluded the disabled individual. These visuals provided concrete evidence of the allocative harms identified.

The first-place winner's methodology, utilizing StyleGAN2-ADA, inherently involved generating and manipulating images to demonstrate the algorithm's preferences. The presentation displayed side-by-side comparisons of original generated faces and their "maximally perturbed" counterparts (e.g., darker vs. lighter skin, older vs. younger features, wider vs. thinner faces). These visual transformations served as direct demonstrations of the algorithm's encoded beauty standards.

Furthermore, the "citizen data science" approach highlighted by the honorable mention on religious head coverings represented a highly accessible form of PoC. The individual simply posted various photos of people wearing different religious head coverings (hijabs, turbans, yamakas) to Twitter and observed how the saliency algorithm cropped them, overwhelmingly preferring individuals without coverings. This simple, real-world experiment effectively demonstrated how the average user might experience algorithmic bias without needing technical expertise, making it a powerful and relatable "demo."

The "Most Generalizable" finding, involving invisible borders, also functioned as a profound PoC for adversarial attacks. Although not explicitly shown in the talk, the concept itself — that a human-imperceptible digital alteration could significantly change the algorithm's output — is a powerful demonstration of a model's vulnerability. This suggests that the challenge itself provided a platform for participants to develop and submit functioning exploits that reveal algorithmic weaknesses.

In summary, while there wasn't a single "demo" segment, the entire talk was a showcase of numerous, independently developed proofs of concept, each highlighting a specific facet of algorithmic bias within Twitter's image cropping system.

Defensive Implications

▶ Watch: Closing thoughts and call to action for future challenges (62:40)

The findings from Twitter’s Algorithmic Bias Bug Bounty Challenge carry profound defensive implications for both Twitter and the broader AI industry. The speed and diversity of bias discovery demonstrated the critical need for a paradigm shift in how machine learning systems are audited and secured.

For Twitter, the challenge provided invaluable, rapid feedback directly applicable to improving its saliency algorithm. The diverse findings necessitate a re-evaluation of training data, including its provenance, inclusivity, and representativeness, to mitigate biases related to language, demographics, age, disability, and cultural attire. The identification of a feedback loop reinforcing stereotypical beauty standards underscores the need for proactive measures to prevent algorithms from amplifying societal harms. Furthermore, insights from entries like the "invisible borders" highlight the importance of robustness testing against adversarial attacks and considering how subtle data manipulations can distort algorithmic outcomes. Jutta Williams emphasized the need for building infrastructure and technology stacks to not only identify but also process and act upon bias reports proactively, transforming academic findings into product changes.

For the broader AI community and other companies deploying ML systems, the challenge provides several key takeaways for enhancing defensive posture:

  1. Embrace Algorithmic Bias Bug Bounties: The most significant implication is the validation of public bug bounties as an effective mechanism for uncovering algorithmic biases. As Patrick Hall noted, "Just because you haven't found bugs in your enterprise AI machine learning offering certainly doesn't mean it doesn't have bugs." Companies should proactively invite external scrutiny to identify biases faster and more comprehensively than internal teams might.
  2. Prioritize Diverse Perspectives: The global participation and the discovery of non-Western/US-centric biases (e.g., Arabic language, religious head coverings) underscore that diverse teams and external contributors are essential for identifying the full spectrum of potential harms. AI design and auditing must move beyond homogenous perspectives.
  3. System-Level Evaluation is Crucial: Mudge's point about component interaction (e.g., emoji design choice impacting saliency) highlights that it's not enough to test individual ML modules in isolation. Comprehensive system-level testing is required before deployment to catch biases arising from the interaction of different components or user-generated content.
  4. Address Data Provenance and Transfer Learning Risks: The discussion around transfer learning and the origin of training data is a critical defensive concern. As Mudge and Patrick Hall articulated, reusing models or datasets (e.g., Flickr-Faces HQ, YouTube videos from 2009, Reddit outlinks) without understanding their provenance and inherent biases can lead to the widespread propagation of those biases. Developers must rigorously vet training data, confirm labels, and ensure representativeness, as "those data sets live forever."
  5. Beyond "Working as Expected": The camouflage example illustrates that an algorithm functioning "as expected" (e.g., camouflage evading detection) can still lead to undesirable and harmful outcomes for users. Defensive strategies must consider the human and societal impact, not just technical performance metrics.
  6. Integrate Responsible AI into Development Cycles: Patrick Hall advocated for systematic and programmatic incorporation of bias bug bounties and other risk mitigation techniques into AI/ML development cycles. This means moving beyond one-off audits to continuous, integrated ethical review.
  7. Empower Communities: As Ariel Herbert-Voss noted, opening up models empowers affected communities to push back and protect themselves from algorithms. Companies should consider mechanisms that allow users to report perceived biases, fostering a more responsive and accountable AI ecosystem.

In essence, the challenge provided a blueprint for building more resilient, fair, and trustworthy AI systems by democratizing the process of identifying and addressing algorithmic harms.

Key Takeaways

  • Algorithmic Bias Bug Bounties are Effective: Twitter's challenge demonstrated that public, incentivized bug bounties are a highly efficient and rapid method for discovering diverse algorithmic biases in real-world ML systems, far exceeding the pace of internal auditing.
  • Societal Biases are Deeply Embedded: The findings revealed that ML algorithms often encode and amplify existing societal biases related to demographics, language, age, disability, and beauty standards, creating harmful feedback loops that require proactive intervention.
  • Diverse Perspectives are Non-Negotiable: Global participation from a wide range of individuals and organizations was crucial for uncovering a broad spectrum of biases, particularly those relevant to non-Western cultures and marginalized communities, highlighting the need for inclusive AI development.
  • ML Models are Vulnerable to Adversarial Attacks: Even subtle, human-imperceptible data manipulations (like "invisible borders") can significantly alter algorithmic outcomes, underscoring the importance of robustness testing against adversarial machine learning techniques.
  • Data Provenance and System-Level Testing are Critical: The risks of transfer learning and reusing biased datasets, combined with biases arising from interactions between different system components, necessitate rigorous data provenance tracking and comprehensive system-level evaluation beyond isolated unit tests.
  • Proactive Responsible AI is Essential: Companies must move beyond academic discussions to systematically integrate bias detection, reporting infrastructure, and mitigation strategies into their AI development pipelines, ensuring that ethical considerations drive product changes.

About the Speaker(s)

The talk featured a diverse group of experts from Twitter, OpenAI, and the responsible AI community:

  • Dr. Rumman Chowdhury is the Director of the Machine Learning Ethics, Transparency and Accountability (META) team at Twitter, leading the charge on ethical AI development and auditing.
  • Jutta Williams serves as a Product Manager on the META team at Twitter. With a background as a security engineer, she played a key role in organizing the bias bounty, drawing parallels to traditional security vulnerability programs.
  • Kyra Yee is a Research Engineer on the META team at Twitter and was one of the authors of Twitter's original paper addressing image cropping concerns, which laid the groundwork for this challenge.
  • Irene Font Peradejordi is a Machine Learning Experience Researcher on Twitter's META team, focusing on how users perceive machine learning on the platform.
  • Ariel Herbert-Voss is a Security Research Scientist at OpenAI, where she also runs red team operations. She is a co-founder of the DEF CON AI Village.
  • Peiter "Mudge" Zatko is a notable figure in the security community and supports various groups at Twitter. He is a long-time DEF CON attendee and speaker.
  • Patrick Hall is a Principal Scientist at bnh.ai, a law firm specializing in cybersecurity, data privacy, and algorithmic discrimination. He previously helped run responsible machine learning at h2o.ai.

Reviews

Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT

Twitter actually did something right here. Rumman Chowdhury and team threw their saliency algorithm to the wolves in a first-of-its-kind bias bug bounty, proving what happens when you let hackers at production ML models. This is the foundational case study that bridged traditional infosec bug bounties with adversarial AI red teaming.

Heather Calloway (CISO) — MUST SEE

Twitter's Algorithmic Bias Bug Bounty operationalized AI ethics into a vulnerability management problem. The panel proves that traditional bug bounty mechanics can successfully uncover representational harms and model biases at scale. If your organization is deploying AI, this model represents the new standard of care for identifying downstream algorithmic risk.

→ Top-rated talks at AI Village @ DEF CON 29

All talks from AI Village @ DEF CON 29