The End of the AppSec Team

The End of the AppSec Team

Speakers: Justin Collins, Security Engineer, Gusto

Conference: LocoMocoSec

YouTube: https://www.youtube.com/watch?v=oEPPRdH2ykg

Overview

In his thought-provoking LocoMocoSec 2020 talk, "The End of the AppSec Team," Justin Collins, a seasoned security engineer and Product Security Lead at Gusto, challenges the conventional wisdom surrounding application security (AppSec) teams. Collins argues that the traditional model of a centralized AppSec team acting as a gatekeeper is fundamentally unsustainable and ultimately destined to fail. The talk is not a call for the abandonment of application security itself, but rather a radical re-imagining of how security responsibilities are structured and integrated within an organization.

Collins posits that the future of effective application security lies in three core principles: security must be distributed throughout the company, it must be extensively automated, and it must be democratized, making it accessible and actionable for every developer. Drawing on his extensive experience at companies like Twitter and Gusto, as well as insights from the broader DevOps movement, Collins outlines a vision where security is seamlessly woven into the daily work of development teams, rather than being a bottleneck managed by a specialized, often overwhelmed, security silo.

This article delves into Collins's critique of the current AppSec landscape, exploring the systemic issues that hinder traditional teams and presenting a compelling case for a transformative shift. It examines the parallels to the evolution of operations, quality assurance, and database administration teams, suggesting that AppSec is ripe for a similar revolution. The ultimate goal is to empower developers to build secure products by default, reducing the burden on dedicated security teams and fostering a culture of shared security ownership.

Background

▶ Watch: The End of the AppSec Team: Distributed, Automated, Democratized (0:00)

The premise of Collins's talk rests on a stark reality: AppSec teams are overwhelmingly outnumbered and consequently overwhelmed. He illustrates this with a common scenario: 96 developers producing a "tsunami of work" for a mere four AppSec engineers. This ratio, he notes, is often better than what many companies experience. The responsibilities heaped upon these small AppSec teams are vast and varied, ranging from reviewing projects and code, performing threat modeling, running and triaging security tools, managing pen testing and bug bounty results, handling security incidents, writing secure libraries, and attempting to cultivate security champions among developers. Each of these tasks is time-consuming and energy-draining, making successful execution across the board nearly impossible.

Collins describes this as a "grind," where AppSec teams are perpetually stuck reacting to immediate issues – a never-ending cycle of design reviews, code reviews (which are impossible to do comprehensively), CVEs, scan reports, and "quick questions" that often devolve into hours of work. This reactive posture prevents teams from engaging in "systemic impact" work that could proactively improve security posture. The natural inclination to solve this by "hiring more AppSec people" is quickly dismissed as unfeasible; there simply aren't enough qualified AppSec professionals globally, and scaling human resources will never keep pace with the exponential growth of development output. "People don't scale; machines scale," Collins asserts, but he cautions that simply "throwing some automation on it" is also not a complete solution. He references a 2012 talk he co-gave at Twitter on security automation, noting that while groundbreaking at the time, such efforts are now "ancient relics" compared to today's needs, and automation alone cannot solve every problem.

A core issue highlighted is the paradox of "security is everyone's job." While conceptually true, this often translates into a blame game where the security team is held accountable for all flaws, even if they didn't write the code. This pressure leads AppSec teams to adopt a "say no" posture, avoiding risk to prevent being paged at 2 AM. Collins emphasizes that for non-security personnel, "security is everyone's job" often feels like an additional, vague burden on top of an already full workday, easily delegated back to the named "security team." He draws parallels to the DevOps movement, where walls between developers and operations, QA, and DBA teams led to inefficiencies like invisibility, inconsistency, long feedback cycles, and a transfer of responsibility. The DevOps Handbook is cited, specifically its critique of "ineffective quality controls" that require external teams for tedious, manual tasks or approvals from distant, busy people lacking adequate context – a description that perfectly fits the traditional AppSec model.

Key Findings

▶ Watch: Why more people or just automation won't scale (9:30)

Justin Collins's central thesis is that the traditional, centralized AppSec team model is fundamentally broken and unsustainable. His key findings and proposed solutions pivot on a radical shift in how application security is perceived, practiced, and owned within an organization.

1. The Unsustainability of Centralized AppSec Teams: Collins argues that AppSec teams, as currently structured, are "destined to fail." The overwhelming ratio of developers to security personnel, coupled with an ever-expanding list of responsibilities, creates a perpetual "grind" that prevents systemic security improvements. Attempting to scale by hiring more AppSec engineers is futile due to a global talent shortage and the inherent inability of human-centric processes to keep pace with modern software development velocity.
2. The Imperative of Distribution, Automation, and Democratization: The future of effective application security must be built upon three pillars:

• Distributed: Security responsibilities and capabilities must be spread across the entire organization, not siloed within a single team. This includes outsourcing certain functions where appropriate.
• Automated: Manual security tasks must be replaced by robust automation to ensure scalability and speed. While automation is crucial, it is not a silver bullet and cannot solve all problems alone.
• Democratized: Security knowledge, tools, and feedback must be accessible and actionable for everyone, particularly developers. This means making security part of their daily work without it feeling like an "extra job."

1. Security as an Integrated Quality Control: Drawing from insights in the DevOps Handbook and industry reports, Collins highlights that high-performing organizations successfully embed security into daily work. This integration leads to tangible benefits: a 50% drop in production vulnerabilities and a 25% improvement in remediation time. This demonstrates that security is not an optional add-on but a critical quality control that, when integrated, enhances overall product quality and efficiency.
2. The Evolution of AppSec Roles: Just as operations, QA, and DBA teams have transformed or "disappeared" into the fabric of development (e.g., infrastructure as code, test-driven development), AppSec must undergo a similar metamorphosis. The work traditionally performed by AppSec teams needs to either:

• Disappear entirely: Through inherent platform security, secure-by-default frameworks, and automated updates.
• Be integrated into daily work: Becoming an organic part of the developer's workflow (e.g., security tests in CI/CD, peer reviews).
• Be made actionable by developers: Security reports and alerts must be clear enough for developers to triage and remediate without direct AppSec intervention.

1. Shifting Responsibility and Accountability: A fundamental finding is the need to shift the locus of security decision-making and accountability from the AppSec team to the development teams. AppSec's role evolves from saying "no" and rubber-stamping to enabling, educating, and providing risk assessments, empowering developers to make informed security choices and own the consequences.

Technical Deep Dive

▶ Watch: Why AppSec 'walls' are bad: Lessons from DevOps (18:30)

Collins's vision for the future of application security is deeply technical, proposing concrete ways for security to either disappear, integrate into daily work, or be made actionable by developers.

1. Making Security Disappear:

This category focuses on embedding security so deeply into the development ecosystem that explicit security intervention becomes unnecessary for many common issues.

• Automated Dependency Updates: Tools like dependabot automatically detect vulnerable dependencies, create pull requests to update them, and integrate with CI/CD for verification and merging. This eliminates manual tracking and patching of known vulnerabilities in third-party libraries. Collins referenced a tweet describing a bot finding, fixing, and merging a dependency vulnerability with "no humans involved."
• Deploying on Fully Patched Systems: Leveraging modern deployment paradigms like containers and immutable infrastructure ensures that every new deployment runs on a system with the latest security patches. This removes the burden of OS-level vulnerability management from AppSec.
• Security Built into Frameworks and Libraries: Frameworks should inherently protect against common vulnerabilities. Collins emphasizes secure by default APIs and libraries. Developers should not need to set special flags or take extra steps to enable security; insecure configurations should be the exception, requiring explicit opt-in. This includes robust input validation, output encoding, and secure session management built directly into the tools developers use daily.
• Secure by Default Environments: Cloud environments and internal platforms should be provisioned with secure baselines, least privilege access, and hardened configurations from the outset, minimizing the attack surface without developer intervention.

2. Integrating Security into Daily Work:

This involves embedding security directly into the developer's workflow, making it an organic part of their job rather than an external imposition.

• Security Checklists in Code Reviews: Instead of AppSec performing all code reviews, developers should peer-review each other's code with security considerations in mind. These checklists would guide developers to identify common security pitfalls. Collins argues that AppSec engineers are often not effective peer reviewers because they lack the deep context of the specific code being written, unlike other developers on the team.
• Developers Threat Modeling: Empowering development teams to conduct their own threat modeling for new features and architectures. This shifts the proactive identification of security risks closer to the source of design and implementation.
• Security Tests as Part of the Test Suite: Integrating security tests directly into the developer's existing test suite (test-driven development style). These could be unit tests, integration tests, or functional tests that specifically check for security properties (e.g., authorization bypasses, input validation failures). If a security test fails a year after being written, it's treated like any other failing test – a developer's responsibility to fix. This ensures security issues are caught early and fixed by the team that introduced them.
• Automated Security Scans: Static Application Security Testing (SAST) and Dynamic Application Security Testing (DAST) tools should be automated and run on-demand or as part of the CI/CD pipeline. The AppSec team might set up and curate these tools, but developers should be able to trigger them, review results, and take action. Collins, creator of Brakeman (a static analysis security tool for Ruby on Rails), implicitly understands the power of such integration.
• Vulnerabilities as Part of the Pull Request Flow: Security findings from automated tools should appear directly within the pull request (PR) workflow, providing immediate feedback to developers. This integrates vulnerability remediation into their natural development cycle.
• Actionable Security Alerts to Responsible Developers: Security alerts should be routed directly to the development teams responsible for the affected code, not solely to the AppSec team. These alerts must be "actionable," providing sufficient context and guidance for developers to understand and remediate the issue without requiring security team intervention. This tackles the "signal to noise" problem, pushing the responsibility for tool output quality onto vendors and AppSec teams curating them.

Collins acknowledges the challenge of signal-to-noise ratio in automated tools. He emphasizes that tool vendors have a responsibility to improve their products, making results more accurate and actionable, providing all necessary information for developers to understand and fix issues. His experience at Gusto involves building internal training to bridge the "knowledge gap" for developers, reinforcing that the security team's role shifts from gatekeeper to enabler and educator.

Demo / Proof of Concept

▶ Watch: Future AppSec: Disappear, integrate into daily work, or democratize (25:00)

Justin Collins explicitly stated that "this is a thinking talk, not a how-to talk." As such, there was no live demonstration or specific technical proof of concept presented during the session. The talk focused on a philosophical shift and strategic vision rather than a step-by-step implementation guide.

However, Collins did allude to his practical efforts in his current role as Product Security Lead at Gusto to put these ideas into practice. He mentioned building internal training programs for developers to address security knowledge gaps, and actively pushing back against the "rubber stamp" mentality where developers seek AppSec approval without owning the security decision. He also referenced his prior work on security automation at Twitter in 2012, acknowledging that while primitive by today's standards, it represented an early attempt to scale security through machines, aligning with the "automated" pillar of his current thesis. These examples serve as real-world applications of the principles discussed, even if not presented as a direct demo.

Defensive Implications

▶ Watch: Q&A: Addressing developer knowledge gaps and shifting responsibility (40:00)

The implications of Justin Collins's vision for defenders are profound, necessitating a fundamental re-evaluation of the AppSec team's structure, responsibilities, and strategic objectives.

1. Shift from Gatekeeper to Enabler: AppSec teams must transition from being a bottleneck that says "no" to an enabling force that empowers development teams. This means providing the right tools, secure-by-default frameworks, and clear, actionable guidance. The AppSec team's expertise should be leveraged to build and curate secure infrastructure and processes, rather than to individually review every piece of code.
2. Invest Heavily in Automation: Defenders must prioritize and invest in comprehensive automation across the entire software development lifecycle. This includes automated dependency updates, SAST/DAST tools integrated into CI/CD, automated vulnerability scanning, and automated deployment on patched systems. The goal is to eliminate manual, repetitive security tasks and free up AppSec resources for higher-value work.
3. Prioritize Developer Education and Training: Bridging the security knowledge gap among developers is critical. AppSec teams should develop effective, integrated training programs that teach developers how to identify, prevent, and remediate common vulnerabilities. This training should be practical, relevant to their daily work, and easily accessible, moving beyond generic awareness.
4. Embed Security into Development Workflows: Security should be integrated into every stage of the development process. This involves encouraging developers to perform their own threat modeling, incorporate security tests into their unit and integration test suites, and peer-review code for security issues. The feedback loop for security findings must be immediate and directly integrated into pull requests.
5. Democratize Security Information and Tools: Security metrics, logs, and tool outputs must be made accessible and understandable to development teams. This transparency empowers developers to take ownership of their product's security posture. AppSec teams should focus on curating tools and results to minimize noise and maximize actionability for developers.
6. Redefine AppSec Team Responsibilities: With many traditional tasks either disappearing or being integrated, the AppSec team's role evolves. They can focus on:

• Architecture & Design: Guiding the creation of secure system architectures and frameworks.
• Tooling & Platform Security: Building, maintaining, and curating security tools and platforms that empower developers.
• Incident Management (not just response): Potentially specializing in managing the process of security incidents, collaborating with dedicated incident response teams.
• Advanced Threat Research: Focusing on emerging threats and complex attack vectors that require specialized expertise.
• Strategic Risk Management: Providing high-level risk assessments and influencing organizational security strategy.

1. Challenge the "Rubber Stamp" Culture: AppSec professionals must push back against requests for simple "security approval." Instead, they should provide risk assessments and expert opinions, empowering development teams to make informed decisions and take ultimate responsibility for the security of their features and products. This fosters a culture of shared ownership and accountability, moving away from AppSec as the sole bearer of risk.

Key Takeaways

• Traditional AppSec teams are unsustainable: The current model of a small, centralized AppSec team overwhelmed by a "tsunami of work" from developers is fundamentally broken and cannot scale.
• The future of AppSec is Distributed, Automated, and Democratized: Security responsibilities must be spread across the organization, manual tasks replaced by automation, and security knowledge/tools made accessible to all developers.
• Integrate security into daily developer work: Security should not be an "extra job" but an organic part of the development workflow, similar to how quality and operations have been integrated (e.g., security tests in CI/CD, peer code reviews).
• Automation is crucial but not a silver bullet: While essential for scaling, automation alone cannot solve all security problems; it must be coupled with human enablement and responsibility.
• Developers must own security decisions and accountability: AppSec's role shifts from gatekeeper to enabler, providing tools, knowledge, and risk context, but developers ultimately own the security posture of their code.
• AppSec roles must evolve: AppSec teams should focus on building secure platforms, curating tools, educating developers, and strategic risk management, rather than reactive reviews and "rubber-stamping."

About the Speaker(s)

Justin Collins is a highly experienced security engineer who currently leads the Product Security team at Gusto. His career journey includes significant roles at various technology companies, starting at AT&T Interactive (which later became yp.com), followed by positions at Twitter and SurveyMonkey. Collins is also recognized for his contributions to the open-source community, having created Brakeman, a widely used static analysis security tool specifically designed for Ruby on Rails applications. He later developed a commercial version of Brakeman, which was eventually sold to Synopsys, where he also worked for a year. This period at Synopsys provided him a unique opportunity to step back from traditional AppSec team roles and reflect on the future of the discipline, ultimately informing the core ideas presented in this talk. Justin is active online under the handle "president beef."