Bugs Are Shallow: Finding Vulnerabilities in Top GitHub Projects

Laurence Tennant (Application Security Consultant · Include Security)

Security BSides London 2023 · Day 1 · Clappy Monkey Track

Overview

In this compelling talk from BSides London, Laurence Tennant, an Application Security Consultant at Include Security, challenges the long-held belief of "Linus's Law" through the lens of wide vulnerability hunting. Tennant presents a novel approach to discovering security flaws, moving beyond the traditional deep-dive penetration testing of individual applications to systematically search for a specific class of vulnerabilities across thousands of open-source projects. His research culminates in the discovery of a significant Mass Assignment vulnerability in Free Code Camp, one of GitHub's most-starred repositories, demonstrating that even highly visible projects can harbor critical, long-standing bugs.

Watch on YouTube

Visual summary for Bugs Are Shallow: Finding Vulnerabilities in Top GitHub Projects by Laurence Tennant
Visual summary for Bugs Are Shallow: Finding Vulnerabilities in Top GitHub Projects by Laurence Tennant

Key moments

  1. 0:00 Introduction: Deep vs. Wide Vulnerability Hunting
  2. 11:45 Discovering grep.app: The Ideal GitHub Search Tool
  3. 12:30 Explaining Mass Assignment with a Vulnerable Example
  4. 16:30 Exploiting Mass Assignment in Free Code Camp
  5. 20:30 Practical Takeaways for Attackers and Defenders
  6. 23:30 Q&A: Difficulty Informing Projects of Vulnerabilities
  7. 28:00 Q&A: Does Research Support or Disprove Linus's Law?

Bugs Are Shallow: Finding Vulnerabilities in Top GitHub Projects

Speakers: Laurence Tennant, Application Security Consultant, Include Security

Conference: BSides London

YouTube: https://www.youtube.com/watch?v=URbqivZAzFY

Overview

In this compelling talk from BSides London, Laurence Tennant, an Application Security Consultant at Include Security, challenges the long-held belief of "Linus's Law" through the lens of wide vulnerability hunting. Tennant presents a novel approach to discovering security flaws, moving beyond the traditional deep-dive penetration testing of individual applications to systematically search for a specific class of vulnerabilities across thousands of open-source projects. His research culminates in the discovery of a significant Mass Assignment vulnerability in Free Code Camp, one of GitHub's most-starred repositories, demonstrating that even highly visible projects can harbor critical, long-standing bugs.

The talk highlights the efficacy of specialized search tools and targeted vulnerability classes for uncovering widespread issues that might otherwise go unnoticed. Tennant meticulously details his methodology, from identifying suitable search engines to selecting Mass Assignment as the focus vulnerability, and ultimately demonstrating its exploitation. This research not only provides a practical guide for attackers seeking overlooked vulnerabilities but also offers crucial defensive implications for developers and security teams, emphasizing the importance of robust input validation and the adoption of readily available static analysis tools.

Tennant's work serves as a powerful reminder that popularity and community engagement do not inherently guarantee security. By demonstrating a critical flaw in a highly reputable project, he underscores the continuous need for vigilant security practices and encourages a shift in perspective towards more scalable and focused vulnerability discovery techniques. The findings suggest that while "eyeballs" are plentiful in open source, the right kind of security-focused eyeballs are still a scarce and valuable resource.

Background

▶ Watch: Introduction: Deep vs. Wide Vulnerability Hunting (0:00)

Traditionally, application security testing often involves a "deep" approach, where a security consultant meticulously examines a single project or a small group of related projects. This involves thoroughly understanding the business logic and exhaustively searching for various types of vulnerabilities. Laurence Tennant, however, sought to explore a different paradigm: wide vulnerability hunting. This strategy involves specializing in a single vulnerability class and then broadly searching for it across an extensive number of codebases. Tennant likens this to "industrial fishing" or "trolling," where a large net is cast to catch specific "good fish" (vulnerabilities) amidst a lot of "bycatch" (false positives or irrelevant findings). The vast ocean for this type of fishing, he explains, is GitHub, with its over 200 million open-source repositories.

A central theme underpinning this research is Linus's Law, coined by Linus Torvalds, which posits that "given enough eyeballs, all bugs are shallow." This implies that the larger and more active a project's community, the higher its code quality and the fewer security vulnerabilities it should possess. While this theory suggests open-source software is inherently more secure than closed-source, Tennant notes that empirical evidence is conflicted. One paper he found couldn't support Linus's Law, while another, focusing on Google projects, observed that highly-starred GitHub projects indeed exhibited higher code quality and faster bug fixes. This conflicting evidence motivated Tennant to test the law in practice.

To achieve his goal of wide vulnerability hunting, Tennant initially explored several methods for searching GitHub's colossal codebase. The most obvious starting point was GitHub Search. However, at the time of his research, it was severely limited: it didn't recognize symbols, lacked regular expression support, and crucially, offered no sorting by relevance. This meant sifting through millions of boilerplate or abandoned "my first project" repositories, which was inefficient for finding vulnerabilities in active, significant projects. While GitHub has since improved its search capabilities, adding symbol recognition and regex support, the core problem of relevance sorting for large projects persists.

Next, he considered Google BigQuery, a data warehouse service offering SQL querying over massive datasets, including a 3TB dataset of GitHub's data. This seemed promising, as it allowed for pattern matching and sorting by repository watch count, a proxy for popularity. However, BigQuery presented two major drawbacks: the GitHub dataset hadn't been updated since 2016, rendering it largely obsolete, and it proved prohibitively expensive, with a single query costing over $200 of his $300 free trial credit.

Tennant also contemplated writing a script to download the top 10,000 Python repositories and run local static analysis tools. He considered CodeQL and Semgrep. CodeQL, owned by GitHub, translates code into a queryable database, enabling powerful taint tracking analysis. While excellent, it has a steep learning curve and requires building C++ projects to generate the database. Semgrep, on the other hand, is praised for its great defaults and ease of custom rule creation, though it can produce many false positives without customization. Ultimately, he decided against this approach for the initial phase, seeking an "easier" path, though he now believes this might be the most effective strategy for future research.

The breakthrough came with the discovery of grep.app, a third-party GitHub searching tool. Unlike GitHub's native search, grep.app specifically indexes only the top half a million GitHub repositories. This critical feature solved the relevance problem, ensuring that any found code patterns were likely within active and important projects, eliminating the need to sift through irrelevant "craft and starter projects." With grep.app, Tennant finally had the powerful and targeted search engine he needed for wide vulnerability hunting.

Having found his search engine, Tennant needed to select a vulnerability class. He sought something that was easy to search for, often led to high-impact vulnerabilities, and, crucially, was a type that developers were less aware of. He explicitly chose to avoid well-known issues like SQL Injection, which modern developers are generally taught to protect against. Instead, he focused on vulnerabilities like unsafe serialization, XXE, and Mass Assignment, which can be equally high-impact but are often overlooked. He settled on Mass Assignment as his target due to its prevalence and the relative lack of awareness among developers, particularly within the Node.js ecosystem.

Key Findings

▶ Watch: Explaining Mass Assignment with a Vulnerable Example (12:30)

The pivotal discovery that enabled Laurence Tennant's wide vulnerability hunting was the third-party GitHub search tool, grep.app. This platform effectively solved the "relevance" problem inherent in standard GitHub search and the outdated/expensive issues of Google BigQuery. By indexing only the top half-million GitHub repositories, grep.app allowed Tennant to quickly identify code patterns within significant and active projects, significantly reducing noise and increasing the likelihood of impactful findings. This tool became the cornerstone of his research methodology, providing the targeted search capability he needed.

Tennant strategically focused his efforts on Mass Assignment vulnerabilities, specifically targeting the Node.js ecosystem. He noted that while Mass Assignment is relatively well-known and mitigated in frameworks like Ruby on Rails (due to past high-profile incidents and robust static analysis tools like Brakeman), it is "much less widely discussed in Node.js." This lack of pervasive awareness, combined with Node.js's diverse and rapidly evolving ecosystem, made it an ideal target for finding overlooked vulnerabilities that could have significant impact. He systematically identified various data persistence patterns across different Node.js frameworks, such as find_one_and_update, update attributes (in loopback), and numerous functions in AdonisJS, to craft targeted search queries for grep.app.

This focused approach quickly yielded a significant hit: a Mass Assignment vulnerability within Free Code Camp. Tennant was particularly excited by this finding because Free Code Camp is one of the most-starred repositories on GitHub, boasting approximately 380,000 stars. The vulnerability was located in a function called updateUserFlag, specifically on line 208, where a constant updates was directly assigned the request.body. This updates object was then passed straight into a user.updateAttributes function without any filtering or validation of the incoming properties. The affected endpoint was a PUT route at /update-user-flag.

Upon examining the user model in Free Code Camp's codebase, Tennant discovered several sensitive properties that were directly stored as part of the user object. While there was no explicit "admin" role, the model contained boolean flags like isBackendCert and isFullStackCert, which represented certifications earned by users. The impact of the Mass Assignment vulnerability was immediate and profound: an attacker could simply send a crafted PUT request to the /update-user-flag endpoint, setting these certification flags to true, along with isCheetah: false and isHonest: true (to maintain a semblance of legitimacy).

The result was striking: Tennant was able to instantly "earn" a complete suite of certifications, which he calculated would normally require over 6,000 hours of dedicated study. This demonstrated a clear circumvention of the platform's core educational value proposition. The vulnerability had existed in the Free Code Camp codebase for five years, despite its immense popularity and community engagement. This finding directly challenges the practical application of Linus's Law, suggesting that the sheer number of "eyeballs" on a project does not automatically guarantee the discovery and remediation of all bugs. Tennant posited that while Free Code Camp has many users, most are beginners learning to code, not experienced security contributors actively auditing the codebase. This highlights a critical distinction between passive user engagement and active security scrutiny.

Technical Deep Dive

▶ Watch: Exploiting Mass Assignment in Free Code Camp (16:30)

At its core, Mass Assignment is a vulnerability where an application automatically binds user-supplied input (e.g., from a web form or API request body) to an object's properties without proper filtering or validation. The OWASP definition describes it as occurring "when an application receives input data, such as an HTTP request, and uses that data to 'mass-assign' values to a data model or object, often without explicitly checking which attributes should be allowed to be set." This can lead to attackers modifying properties they should not have access to, potentially escalating privileges, bypassing security controls, or corrupting data.

Tennant illustrated this with a basic pseudo-code example of a vulnerable web application. Imagine an API POST /users/register route that calls a registerUserHandler. This handler then takes the request.body directly and saves it as a newUserModel into the database. If the UserModel has properties like username, email, and a default role of user, a developer might assume only username and email will be supplied. However, an attacker could include role: admin in their request body. Because the handler indiscriminately assigns the entire request.body to the UserModel, the attacker successfully registers as an administrator. The fundamental problem is a failure to whitelist or blacklist properties that can be modified by user input.

A famous historical example of Mass Assignment, cited by Tennant, involved Egor Homakov's exploit against Ruby on Rails about a decade ago. Homakov demonstrated how he could leverage a Mass Assignment vulnerability in the public key model to commit code to any repository on GitHub. The public key model had a user property. By manipulating this, Homakov was able to associate his own public key with the GitHub Rails user, effectively gaining unauthorized commit access to the Rails repository. This incident significantly raised awareness of Mass Assignment within the Ruby on Rails community, leading to robust built-in protections and static analysis tools like Brakeman that actively warn against such patterns. As a result, Ruby on Rails, once a primary target for this vulnerability, is now "probably one of the most secure when it comes to this particular vulnerability class," according to Tennant.

In contrast, Tennant identified the Node.js ecosystem as being far less mature in its understanding and mitigation of Mass Assignment. He attributed this to its highly diverse nature, with "about a billion" different frameworks and a tendency for developers to build components from scratch, rather than relying on a single, opinionated framework like Ruby on Rails or Django. This fragmentation means there hasn't been a systematic, community-wide effort to squash bug classes like Mass Assignment. Tennant's research involved identifying various common data persistence patterns in Node.js frameworks that could lead to Mass Assignment, such as:

  • Using find_one_and_update in database interactions.
  • updateAttributes functions in frameworks like LoopBack (though LoopBack often includes separate ACLs for property modification).
  • A multitude of potentially vulnerable functions (Tennant mentioned "about 10 different functions") in frameworks like AdonisJS.

The specific vulnerability discovered in Free Code Camp perfectly exemplified these patterns. The core issue resided in the updateUserFlag function, which handled PUT requests to the /update-user-flag API route. Within this function, line 208 was critical:

Here, the updates constant was directly assigned the entire request.body received from the client. This updates object was then passed without any sanitization or filtering to the user.updateAttributes method, effectively allowing any property present in the request body to overwrite corresponding properties in the user object within the database.

Upon inspecting the user model, Tennant found several sensitive boolean fields that indicated earned certifications, such as isBackendCert, isFullStackCert, isFrontEndCert, etc. These flags were simply stored as part of the user's profile. Crucially, there was no concept of an "admin" user in Free Code Camp, but the ability to arbitrarily grant oneself certifications constituted a significant logical flaw and a clear bypass of the platform's core functionality. The direct, unfiltered assignment of request.body to the user model's properties was the fundamental Mass Assignment flaw.

Demo / Proof of Concept

▶ Watch: Q&A: Difficulty Informing Projects of Vulnerabilities (23:30)

The demonstration of the Mass Assignment vulnerability in Free Code Camp was straightforward yet highly impactful. Laurence Tennant outlined the process, which involved crafting a malicious PUT request to the vulnerable endpoint.

The endpoint in question was /update-user-flag. To exploit the vulnerability, an attacker needed only to send an HTTP PUT request to this route with a JSON request body containing the desired sensitive properties set to true. Specifically, Tennant included properties such as isBackendCert: true and isFullStackCert: true. To make the profile appear more legitimate, he also included isCheetah: false and isHonest: true, reflecting the platform's internal flags for academic integrity.

Upon sending this crafted request, the application's updateUserFlag function, which directly assigned the request.body to the user model via user.updateAttributes, processed the input without validating or filtering the properties. As a result, the attacker's user profile was immediately updated with the new certification statuses.

The outcome was a clear and visible demonstration of the exploit's success. Tennant showed how his Free Code Camp profile instantly displayed a multitude of certifications, which he estimated would normally require "over 6,000 hours worth of studying" to legitimately earn. This included certifications signed by Quincy Larson, the head of Free Code Camp, lending an air of authenticity to the fraudulently obtained credentials.

Tennant reported the vulnerability to the Free Code Camp team, who were highly responsive and efficient in implementing a fix. The chosen solution was to implement an allow-listing approach. Instead of blacklisting potentially sensitive properties (which is prone to oversight), the fix involved explicitly defining a list of properties that are allowed to be modified by user input. Any property in the request.body that was not on this allow-list was simply filtered out before being passed to the user.updateAttributes function. This simple yet effective change directly addressed the Mass Assignment vulnerability, preventing unauthorized modification of sensitive user data.

Defensive Implications

▶ Watch: Q&A: Does Research Support or Disprove Linus's Law? (28:00)

The discovery of a Mass Assignment vulnerability in a highly-starred project like Free Code Camp offers several critical lessons for defenders. Preventing Mass Assignment vulnerabilities primarily revolves around rigorous input validation and careful handling of user-supplied data when updating object properties.

Laurence Tennant highlighted two primary methods for preventing Mass Assignment:

  1. Allow-listing Properties: This was the approach adopted by Free Code Camp to fix the vulnerability. Instead of attempting to blacklist properties that shouldn't be modified (which is inherently prone to missing new or overlooked sensitive fields), developers should explicitly define an allow-list of properties that are permitted to be changed by user input. Any property in the incoming request body that is not on this allow-list must be filtered out before the data is bound to the object model. This ensures that only intended and safe modifications occur.
  1. Specific API Routes for Property Modification: Tennant expressed a preference for a more robust architectural pattern: designing distinct API endpoints for modifying individual properties. For example, instead of a generic /user/update endpoint that accepts a full user object, one would implement /user/email to update only the email address, /user/username to update the username, and so forth. While this approach might require "a little bit more repetition" in terms of code, it inherently creates a safer pattern. Each endpoint would be responsible for validating and updating only one specific property, making it extremely difficult for an attacker to inject unintended property changes. This design choice forces developers to be explicit about what can be modified and provides a clear separation of concerns.

Beyond direct Mass Assignment prevention, Tennant also advocated for the broader adoption of static analysis tools within the open-source community. He specifically recommended that GitHub project maintainers enable CodeQL scanning as part of their Continuous Integration/Continuous Deployment (CI/CD) pipelines. GitHub has made this process incredibly easy, often requiring just "a two-button thing" in the project settings. While CodeQL might not have caught this specific Mass Assignment vulnerability at the time (though rules are constantly improving), enabling default rule sets, especially for languages like Python, can provide a "quick win" by catching various low-hanging fruit and common security issues. Tennant noted that many open-source projects, even popular ones, are not yet leveraging this readily available and powerful security feature.

These defensive strategies underscore the importance of shifting developer mindset from implicit trust in input to explicit validation and control. The Free Code Camp incident serves as a stark reminder that even well-intentioned code in widely used projects can harbor fundamental security flaws if these principles are not rigorously applied.

Key Takeaways

  • Linus's Law is Not Universal: The discovery of a five-year-old Mass Assignment vulnerability in Free Code Camp, one of GitHub's most-starred repositories, challenges the notion that "given enough eyeballs, all bugs are shallow." The type of "eyeballs" matters; a large user base of learners does not equate to a vigilant security auditing community.
  • Vulnerabilities Abound in Open Source: There are countless overlooked vulnerabilities, even in highly popular and widely used open-source projects. Wide vulnerability hunting across many codebases is an effective strategy for uncovering these persistent flaws.
  • Targeted Search Tools Are Crucial: Traditional GitHub search often lacks the relevance filtering needed for effective wide hunting. Tools like grep.app, which index only top-tier repositories, are invaluable for focusing efforts on impactful projects and minimizing noise.
  • Mass Assignment Remains a High-Impact Threat: Despite its historical prominence, Mass Assignment continues to be a prevalent and high-impact vulnerability, particularly in diverse and rapidly evolving ecosystems like Node.js, where awareness and systematic mitigation efforts lag behind more mature frameworks.
  • Defensive Best Practices are Essential: Developers must implement robust input validation to prevent Mass Assignment. This includes allow-listing (explicitly defining which properties can be modified) and, ideally, designing specific API routes for each modifiable property to enforce granular control.
  • Leverage Available Static Analysis: Open-source project maintainers should proactively enable tools like CodeQL scanning in their CI/CD pipelines. This provides an easy "quick win" for catching common security bugs and improving overall code quality, even if it doesn't solve all problems.

About the Speaker(s)

Laurence Tennant is an Application Security Consultant at Include Security. He first developed a passion for cybersecurity while participating in the Cyber Security Challenge UK online, where he engaged with various puzzles and challenges. This experience led him to connect with other security enthusiasts, forming the CTF (Capture The Flag) team "Crown," which for several years was recognized as the strongest CTF team in the UK. Tennant is also the co-founder of cryptoHack, a website dedicated to providing fun and educational challenges for learning cryptography. His professional work focuses on helping organizations secure their applications, often involving penetration testing and security consulting.

Reviews

Dr. Zero (Offensive Security Researcher) — SOLID

Tennant takes a baseball bat to Linus's Law and proves that 380,000 GitHub stars don't mean shit if none of those eyeballs belong to a security researcher. He hunted for mass assignment flaws across the Node.js ecosystem and popped Free Code Camp for a full certification bypass. Good, practical wide-net hunting, even if the bug class is older than some of the developers writing it.

Heather Calloway (CISO) — STRONG ACCEPT

This talk systematically dismantles the assumption that popular open-source projects are inherently secure. Tennant demonstrates how a basic vulnerability class can survive for half a decade in a top-tier GitHub repository. It’s a necessary reality check for Application Security leaders evaluating our static analysis coverage and open-source risk models.

→ Top-rated talks at Security BSides London 2023

All talks from Security BSides London 2023