Running an efficient bug bounty program and PSIRT function

Running an efficient bug bounty program and PSIRT function

Speakers: Garrett McNamara, Senior Manager of Product Security Adversary Management at ServiceNow; Jeff Guerra, Senior Security Engineer at OnePassword

Conference: BSides SF

YouTube: https://www.youtube.com/watch?v=diwvvtj4-Oc

Overview

In this insightful talk from BSides SF, Garrett McNamara of ServiceNow and Jeff Guerra of OnePassword (formerly GitHub) delve into the critical components of establishing and maintaining effective bug bounty programs and Product Security Incident Response Team (PSIRT) functions. Their presentation highlights the inevitability of security incidents and offers practical, experience-driven strategies to make these "fires" more manageable, rather than attempting the impossible task of eliminating them entirely. The speakers emphasize the importance of automation, robust incident response processes, meaningful metrics, and strategic engagement with the researcher community.

The core message revolves around building resilient security operations that not only react to vulnerabilities but also proactively integrate lessons learned into the Secure Software Development Life Cycle (SSDLC). This talk is essential for security managers, program owners, and engineers looking to optimize their vulnerability management, justify program investments, and foster a collaborative environment with external security researchers. It provides a roadmap for organizations at various stages of maturity, from those just starting a PSIRT to established programs seeking greater efficiency and impact.

Background

▶ Watch: Introduction to efficient bug bounty and PSIRT (0:00)

The landscape of software development is fraught with security challenges, making the occurrence of vulnerabilities and subsequent incidents a certainty rather than a possibility. Many organizations struggle with ad-hoc responses, inefficient processes, and a lack of clear metrics to demonstrate the value of their security efforts. This creates a significant need for structured and efficient approaches to vulnerability management and incident response.

Garrett McNamara underscored this challenge by recounting his experience at ServiceNow, where in 2022, there was no formal PSIRT team or program. This necessitated building a comprehensive incident response capability from the ground up. Similarly, Jeff Guerra drew upon his five years at GitHub, where he supported their extensive bug bounty and vulnerability management programs, gaining deep insights into scaling such initiatives. The speakers highlighted that while the security industry often focuses on finding vulnerabilities, equal importance must be placed on how organizations respond to, remediate, and learn from these findings. The talk implicitly addresses the growing complexity of software ecosystems, the increasing sophistication of attackers, and the constant pressure on security teams to do more with less, necessitating smart automation and clear strategic direction. A foundational resource mentioned is the First.org services framework, which provides a structured approach for organizations to build and mature their PSIRT capabilities, breaking down the daunting task into manageable, progressive levels.

Key Findings

▶ Watch: Streamlining with Slack chat ops automation (2:05)

The talk presented several critical findings for organizations aiming to optimize their bug bounty and PSIRT operations:

• Automation is Paramount: Leveraging tools like Slack, Tines, and custom applications for real-time alerts, automated triaging, and streamlined workflows significantly boosts efficiency and reduces manual overhead.
• Structured Incident Response: Adopting a formal framework, such as the First.org services framework, provides a clear, repeatable process for discovery, triage, remediation, disclosure, and education, guiding teams through complex incidents.
• Metrics for Justification and Storytelling: Effective metrics go beyond simple report counts; they should tell a compelling story about the program's value, impact, and return on investment (ROI) to leadership, ideally tracked through CISO-level OKRs (Objectives and Key Results).
• Cultivating Researcher Relationships: Actively engaging with and retaining security researchers through fair compensation, non-monetary rewards (swag, public recognition), and collaborative opportunities (beta feature access, patch review) is crucial for program success.
• PSIRT and Bug Bounty Integration: For many organizations, these functions are inseparable, benefiting from shared processes, internal triage capabilities, and coordinated incident response efforts.
• Repeatable Disclosure Process: A consistent, durable process for vulnerability disclosure, involving legal and communications stakeholders with clear Service Level Agreements (SLAs), is essential for managing expectations and maintaining trust.
• Bug Bounty as a Talent Pool: Successful researchers who consistently find advanced vulnerabilities against an organization's products can be a low-risk, high-value source for recruiting internal security talent.
• Program Maturity Beyond Public Scope: A program's maturity is not solely defined by whether it is public or private, but by its ability to effectively cover its asset scope, respond consistently to vulnerabilities, and adapt to evolving threats.
• Start Small, Stress-Test, Automate: The most effective approach for new programs is to begin with a small, manageable scope, rigorously test internal processes, and heavily invest in automation before scaling.

Technical Deep Dive

▶ Watch: Bots for triaging, assigning severity, and escalation (2:40)

The speakers provided a detailed look into the technical and procedural mechanisms that underpin efficient bug bounty and PSIRT operations.

Automation and Integrations:

A cornerstone of efficiency is automation, particularly through Slack chat op automation. Jeff Guerra, drawing on his GitHub experience, highlighted how custom internal Slack applications, often built using Slack webhooks, can provide automated real-time alerts for new reports, status updates, and even assist in triaging. These bots can kick off vulnerability tracking with pre-assigned severities into tracking systems, and facilitate easy escalation of issues. For more sophisticated orchestration, tools like Tines were mentioned, which enable complex security workflows. The talk emphasized that integrating bug bounty platforms' APIs with custom applications provides the most robust and tailored results.

Garrett McNamara detailed ServiceNow's bespoke solution, built on the ServiceNow platform, to address their specific PSIRT needs. This application features a two-way integration with their bounty vendor's API, allowing not only the ingestion of reports but also the pushing of status updates and changes back to the bounty platform. This deep integration extends to various internal ticket tracking systems, creating a centralized hub for all incident-related data. A unique technical integration is with a patch level monitoring system. For a given vulnerability, this system can assess patch adoption rates across ServiceNow's customer base, quantifying residual exposure for known issues. The custom application also serves as a critical knowledge repository, storing root cause analysis (RCA) notes, exploit code samples, snippets of vulnerable code, draft disclosure communications (e.g., for CVEs and security advisories), and identifying upstream improvement opportunities for product security.

Structured Incident Response with First.org Framework:

To guide incident response, ServiceNow adopted the First.org services framework. This framework categorizes incident response into distinct phases:

• Discovery: Confirming the validity and reproducibility of a reported vulnerability, ensuring it's not a false positive.
• Triage: Pinpointing the vulnerability's location in the codebase, assessing its true severity (which may differ from initial reports), determining if similar vulnerabilities exist elsewhere (variant hunting or data-driven variant analysis), and strategizing the most effective remediation to prevent bypasses.
• Remediation: Implementing patches, which might involve near-term fixes or multi-part solutions depending on complexity.
• Disclosures: Managing public communication, including CVEs and security advisories, based on organizational policy.
• Education: Establishing a feedback loop to the Secure Software Development Life Cycle (SSDLC) to prevent recurrence and improve overall security posture.

The speakers noted that complex incidents often require distinguishing the status of the "acute" reported issue from other related workstreams, such as variant hunting. They also suggested analyzing commit history to determine if vulnerabilities are "legacy" issues predating significant AppSec investment or newly introduced, and using an internal version of the OASP Top 10 to track the burning down of specific vulnerability classes.

Post-Incident Learning and SSDLC Feedback:

Post-incident, two types of reflection are crucial:

• Post Incident Review (PIR): Focuses on the response mechanism itself—how well teams collaborated to identify, confirm, and remediate the issue.
• Post-mortem: Dives deeper into systemic breakdowns—what parts of the system allowed the vulnerability to exist in the code. This process, ideally conducted with blameless approaches (as advocated by Google SRE), focuses on process and system failures rather than individual blame.

These reflections directly feed into the SSDLC. Tooling enhancements include analyzing and fine-tuning SAST rules based on vulnerability trends identified in post-mortems. Creating and publishing DAST templates for regression checking severe vulnerabilities ensures that fixed issues don't reappear. Education and enablement efforts involve sharing real-world vulnerability case studies with security champions programs and conducting internal write-ups of interesting or complex vulnerabilities (e.g., multi-stage RCE chains) to inspire internal research and improve organizational security hygiene.

Demo / Proof of Concept

▶ Watch: Utilizing FIRST.org services framework for PERTs (6:15)

This particular conference talk focused on strategic and operational advice for managing bug bounty and PSIRT functions rather than demonstrating a specific exploit or tool in real-time. The speakers did not present a live technical demonstration or proof of concept during their session. However, Jeff Guerra referenced a previous B-Sides San Francisco talk he gave in 2023, titled "Life of a Bug at GitHub," which provides a comprehensive walkthrough of the entire PSIRT and bug bounty program lifecycle, from initial report reception to final resolution. This prior talk serves as a conceptual demonstration of the processes and automations discussed.

Defensive Implications

▶ Watch: Incident response stages: Discovery, Triage, Remediation (6:40)

For security practitioners and organizations, the insights shared by McNamara and Guerra offer several actionable defensive implications:

1. Formalize PSIRT with a Framework: Organizations should adopt a recognized framework like First.org's services framework to structure their incident response. This provides a clear, repeatable process for discovery, triage, remediation, disclosure, and education, ensuring consistent and effective handling of vulnerabilities.
2. Automate Relentlessly: Invest in Slack chat op automation, integrate bug bounty platform APIs with internal systems, and leverage orchestration tools like Tines. This reduces manual effort, accelerates response times, provides real-time visibility, and enhances the overall efficiency of vulnerability tracking and escalation.
3. Integrate PSIRT and SSDLC: Establish strong feedback loops from incident response to the SSDLC. Use post-mortems to fine-tune SAST rules, develop DAST templates for regression testing, and educate development teams through security champions programs and internal case studies to prevent recurring vulnerability classes.
4. Develop Meaningful Metrics: Move beyond simple volume counts. Implement CISO-level OKRs that track the impact and ROI of the bug bounty program. Focus on metrics such as the ratio of critical/high severity reports, time-to-triage for different severity levels, and the burning down of specific OASP Top 10 vulnerability classes to demonstrate tangible value.
5. Prioritize Researcher Engagement and Retention: Treat security researchers as strategic partners. Beyond competitive monetary rewards, offer non-monetary rewards like unique swag, public recognition (blog posts, Hall of Fame), and opportunities for collaboration (e.g., access to beta features, reviewing patch snippets). A positive and repeatable researcher experience fosters loyalty and encourages higher-quality submissions.
6. Establish Durable Disclosure Processes: Create clear, repeatable playbooks for vulnerability disclosure that involve legal, communications, and product teams. Define internal SLAs for responding to researchers and ensure consistency in handling disclosure requests, building trust and predictability.
7. Consider Bug Bounty as a Recruitment Channel: Actively identify and engage top researchers who consistently find advanced vulnerabilities. Their deep product knowledge and proven technical skills make them excellent candidates for internal security roles, significantly reducing ramp-up time.
8. Start Small and Scale Strategically: For new programs, begin with a limited scope to stress-test processes, ensure clear ownership, and build a solid foundation. Only expand the asset scope or transition from a private program to a public program once internal capabilities are robust and consistent response times can be maintained.

Key Takeaways

• Embrace Automation for Efficiency: Leverage Slack, Tines, and API integrations to automate alerts, triaging, and tracking, significantly boosting the efficiency of your bug bounty and PSIRT operations.
• Implement a Structured Response Framework: Adopt frameworks like First.org's to guide your PSIRT through consistent phases of discovery, triage, remediation, disclosure, and education, ensuring a repeatable and effective incident response.
• Metrics Must Tell a Story: Beyond raw numbers, use CISO-level OKRs and trend analysis to communicate the program's value, justify investment, and drive improvements in the SSDLC.
• Cultivate Researcher Relationships: Foster a strong, collaborative relationship with the security researcher community through fair compensation, non-monetary incentives, and opportunities for deeper engagement (e.g., beta testing, patch review).
• Bug Bounty Programs are a Talent Pipeline: Recognize and utilize your top bug bounty researchers as a valuable, low-risk source for recruiting skilled security engineers for your internal team.
• Start Small, Stress-Test, and Scale: Begin with a manageable scope, rigorously test all internal processes, and ensure clear ownership before expanding your bug bounty program or transitioning from private to public.

About the Speaker(s)

Jeff Guerra is a Senior Security Engineer at OnePassword. Prior to this role, he spent five years at GitHub, where he played a pivotal role in supporting their bug bounty and vulnerability management programs. Jeff has a diverse background, having dabbled in application security (AppSec) and various vulnerability management initiatives. He describes himself as someone who loves chaos in his professional life but balances it with a passion for gardening on weekends, intentionally disconnecting from technology.

Garrett McNamara serves as the Senior Manager of Product Security Adversary Management at ServiceNow. His extensive role encompasses a wide array of focus areas, including offensive security, defensive security, risk management, bug bounty program oversight, mergers and acquisitions (M&A) security integration, and remediation efforts. Garrett was instrumental in establishing ServiceNow's PSIRT function from scratch in 2022, bringing a wealth of experience in building and maturing security programs within large organizations.