Monday Keynote: Perseverance, Growth and Success. The Sunburst Story from the CISO perspective.

Monday Keynote: Perseverance, Growth and Success. The Sunburst Story from the CISO perspective.

Speakers: Tim Brown, CISO, Team8

Conference: FIRSTCON

YouTube: https://www.youtube.com/watch?v=lki1Kprw1n4

Overview

Tim Brown, currently the CISO at Team8 and formerly the Chief Information Security Officer for SolarWinds, delivers a poignant and remarkably candid keynote address at FIRSTCON, recounting the harrowing experience of navigating the Sunburst supply chain attack in December 2020. This talk transcends a mere technical recounting of the incident, offering a deeply personal and organizational perspective on one of the most impactful cybersecurity events of the decade. Brown details the immediate chaos, the long-term strategic and operational shifts, and the unprecedented legal challenges faced by a CISO at the epicenter of a nation-state attack.

The talk is a masterclass in crisis management, emphasizing the critical role of pre-established trust, cross-functional collaboration, and radical transparency in the face of an existential threat. Brown’s narrative underscores that while the technical sophistication of the attack was profound, the human and organizational response ultimately determined the company's survival and recovery. His insights into managing public perception, internal morale, and personal well-being amidst relentless pressure provide invaluable lessons for cybersecurity leaders grappling with high-stakes incidents.

This article dissects Brown's account, exploring the technical intricacies of the Sunburst attack, SolarWinds' multi-faceted response, the profound defensive implications for the broader industry, and the personal toll and ultimate vindication of the CISO leading the charge. It highlights how an incident of this magnitude can serve as a catalyst for profound, positive change, ultimately leading to a more resilient and "secure by design" posture.

Background

▶ Watch: Receiving the incident notification: 'tainted code' (1:15)

SolarWinds, a company with approximately 200,000 customers, $1 billion in revenue, and a 25-year history in network observability and monitoring, found itself at the heart of a global crisis on December 12, 2020. The company's flagship Orion platform, a 20-year-old product with 45,000 to 50,000 customers on maintenance globally, was compromised. Tim Brown received a call informing him that SolarWinds had shipped tainted code that was actively being exploited, with public disclosure imminent on Sunday. This sudden notification launched SolarWinds into an incident response scenario that few organizations are prepared for.

Prior to the Sunburst incident, Brown had cultivated a robust incident response (IR) program by treating numerous smaller vulnerabilities and data sharing issues as "level zero" incidents. This frequent practice, though for minor events, proved invaluable. It fostered crucial relationships and built trust among the security team and key cross-functional stakeholders, including legal, the CEO, CFO, marketing, and HR. This pre-existing rapport meant that on day one of the Sunburst crisis, these teams were not meeting for the first time, but rather were a cohesive unit ready to collaborate under immense pressure. This foundation of trust, Brown emphasizes, is impossible to create in the heat of a critical incident and was paramount to their initial response.

The immediate investigation, aided by Mandiant's CTO, Charles Carmichael, quickly confirmed the reality of the threat within 30 minutes. SolarWinds was confronted with the chilling reality that 18,000 customers had downloaded a potentially compromised version of their software, although the full extent of the harm was not immediately known. This set the stage for a period of "controlled chaos," requiring rapid engagement with external partners like DLA Piper (legal counsel), CISA, and the FBI, and the deployment of CrowdStrike's B-team, quickly escalated to the A-team, to manage the unfolding crisis.

Key Findings

▶ Watch: How attackers compromised the SolarWinds build system (10:00)

The Sunburst attack was characterized by its exceptional sophistication and strategic targeting, attributed to the Russian SVR. Key findings from the incident and its aftermath reveal a multi-stage compromise:

1. Initial Reconnaissance via O365 Compromise: The attackers first compromised SolarWinds' O365 email and Teams environment. This was achieved through an illicit consent grant during a "friends and family" deployment involving approximately 25 people. Crucially, SolarWinds was running an E3 license for O365, which lacked access to the granular logs necessary to detect this initial breach. The attackers leveraged email communications to meticulously map out SolarWinds' internal build system, identifying automated emails related to the build process to understand its architecture and vulnerabilities.

1. Build System Supply Chain Injection: The core of the Sunburst attack was the compromise of SolarWinds' build system, rather than direct manipulation of source code repositories like GitHub. This strategic choice allowed the attackers to bypass standard code scanning and security checks. They corrupted a template for transient virtual machines used in the build process. A malicious service was injected into these VMs, designed to wait for a specific file to be built. Upon detection, it would swap out the legitimate code for the Sunburst payload, allow it to be compiled, built, and digitally signed by SolarWinds, then swap the code back. This ensured the malicious code was embedded within a trusted, signed product.

1. Targeted Impact and Payload Design: While 18,000 customers downloaded versions containing the Sunburst code, fewer than 100 ultimately progressed to the second and third stages of the attack. These later-stage compromises specifically targeted critical entities, including four US government agencies. The Sunburst payload itself was designed for stealth and persistence, mimicking SolarWinds' legitimate Orion Improvement Plan (OIP) protocol to blend in with network traffic and evade detection. It possessed command-and-control (C2) capabilities, allowing the attackers to issue instructions and potentially execute further actions. Brown emphasizes the chilling reality that the attackers chose a specific payload; it could have been far more destructive, akin to a Stuxnet-level industrial control system attack.

1. CISO Liability and Legal Precedent: A significant and unprecedented finding from the aftermath was the SEC's investigation and subsequent charges against Tim Brown personally and SolarWinds. This marked a pivotal moment, highlighting the personal liability faced by CISOs for cybersecurity incidents and related disclosures. Although the case was eventually dismissed, the initial charges and the multi-year legal battle, which incurred over $1 million in personal attorney costs for Brown, set a new precedent for CISO accountability and spurred industry-wide discussions about corporate support and indemnification for security leaders.

Technical Deep Dive

▶ Watch: The 'foxhole team' and company unity during crisis (28:50)

The technical ingenuity of the Sunburst attackers lay in their ability to exploit subtle weaknesses in SolarWinds' internal infrastructure and development processes. The attack began not with a direct assault on the product, but with a reconnaissance phase targeting SolarWinds' internal communications.

The initial breach of O365 email and Teams was a critical enabler. The attackers leveraged an illicit consent grant, a technique where a malicious application is granted permissions to access user data, typically bypassing traditional multi-factor authentication (MFA). SolarWinds' O365 E3 license proved insufficient to detect this, as it lacked the advanced logging and security features present in higher-tier licenses. By monitoring internal emails, particularly those related to automated build processes, the attackers gained intimate knowledge of SolarWinds' development pipeline, including the location and configuration of their build systems. This intelligence allowed them to craft an attack tailored to circumvent existing security controls.

The core of the supply chain attack focused on the build system, specifically targeting the transient virtual machines (VMs) used for compiling the Orion software. Instead of directly modifying source code in repositories like GitHub, which would have triggered alerts from static analysis and version control systems, the attackers implanted a malicious service within the VM templates. This service was designed to lie dormant until a specific compile-time event. When the legitimate Orion software component was being built, the malicious service would temporarily swap out SolarWinds' clean code for the Sunburst payload. This tainted code would then be compiled, digitally signed with SolarWinds' legitimate certificates, and packaged into the final product. Crucially, after compilation, the malicious service would swap the legitimate code back, leaving no trace in the source repository or the final build environment, making it incredibly difficult to detect. The discovery of this mechanism was aided by a fortuitous event: a crashed virtual machine that had a backup, allowing investigators to recover the raw Sunburst source code.

The Sunburst payload itself was meticulously crafted for stealth and persistence. It mimicked the legitimate Orion Improvement Plan (OIP) protocol, a telemetry collection mechanism used by SolarWinds, to communicate with its command-and-control (C2) servers. This allowed the malicious traffic to blend in with normal network activity, making it challenging for network defenders to distinguish between legitimate and malicious communications. The payload was designed to be highly selective, only activating in specific environments and targeting high-value victims, further reducing its footprint and increasing its chances of remaining undetected.

In response to the incident, SolarWinds undertook a monumental overhaul of its security posture, encapsulated in its "Secure by Design" program. This included:

• Halting new feature development for six months to focus entirely on security remediation.
• Implementing brand new build systems that were peer-reviewed and engineered to require collusion between multiple individuals for any changes, significantly increasing the difficulty of compromise.
• Hardening all infrastructure with enhanced security configurations and continuous monitoring.
• Deploying an unprecedented level of visibility with three different Endpoint Detection and Response (EDR) solutions and three distinct Security Operations Centers (SOCs).
• Establishing a full-time red team and a dedicated security audit team to continuously test and validate their defenses.
• Mandating UBI keys (YubiKeys) for all employees, enforcing strong multi-factor authentication across the organization.

These measures represent a significant investment and a paradigm shift towards building security in from the ground up, recognizing that exemplary security is the only path to recovery and sustained trust after such a profound breach.

Demo / Proof of Concept

▶ Watch: Understanding why SolarWinds became a supply chain target (33:50)

Tim Brown's talk focuses on the narrative and lessons learned from the SolarWinds Sunburst incident, rather than presenting a live technical demonstration or proof of concept. The "proof of concept" in this context was the Sunburst attack itself – a real-world, highly sophisticated supply chain compromise that demonstrated the efficacy of the Russian SVR's attack methods against a critical software vendor. The talk meticulously details how the attackers executed their plan, from initial reconnaissance to code injection and payload deployment, providing a retrospective analysis of their techniques and SolarWinds' subsequent response and recovery efforts.

Defensive Implications

▶ Watch: CISO contract advice and personal liability lessons (55:00)

The SolarWinds Sunburst attack served as a stark wake-up call, fundamentally reshaping defensive strategies across the cybersecurity landscape. Tim Brown's experience highlights several critical implications for defenders:

1. Supply Chain Security is Paramount: The incident unequivocally demonstrated that an organization's security is only as strong as its weakest link in the supply chain. Defenders must extend their security scrutiny beyond their own perimeters to include all third-party software vendors. This necessitates rigorous vendor risk management, demanding software attestation from providers, and understanding the security practices of every component integrated into their systems. Organizations can no longer assume that digitally signed software from a trusted vendor is inherently secure.

1. Build System Protection as Critical Infrastructure: The attack's focus on the build system, rather than source code, underscored a significant blind spot. Build environments are now recognized as critical infrastructure targets. Defenders must implement stringent controls around these systems, including multi-person approval processes, peer review for all changes, immutable infrastructure principles for build VMs, and continuous monitoring for anomalies. The goal is to make compromise difficult and detection immediate, potentially requiring collusion for any malicious activity to succeed.

1. Enhanced Visibility and Layered Detection: The failure to detect the initial O365 illicit consent grant due to insufficient logging (E3 license) emphasizes the need for comprehensive telemetry. Organizations must invest in higher-tier security licenses and advanced logging capabilities across all critical systems, especially cloud environments. The post-incident adoption of multiple EDRs and SOCs by SolarWinds illustrates the move towards a layered, redundant approach to detection and response, acknowledging that no single tool is foolproof against sophisticated adversaries.

1. Proactive and Resilient Security Posture: The "Secure by Design" philosophy, which prioritized security from the outset, became SolarWinds' guiding principle. Defenders must embed security into every stage of the software development lifecycle (SDLC) and operational processes. This includes adopting a full-time red team and a security audit team for continuous, aggressive testing. The shift in mindset from merely "being secure" to "being resilient" is crucial, recognizing that breaches are inevitable and focusing on rapid detection, containment, and recovery.

1. Robust Incident Response and Cross-Functional Trust: The success of SolarWinds' initial response hinged on pre-existing trust and collaboration between security, legal, executive leadership, and other departments. Organizations must regularly practice IR scenarios, including tabletop exercises that involve all relevant stakeholders, to build these relationships before a crisis hits. Clear roles and responsibilities, with the CISO acting as a key advisor to the CEO who ultimately leads the crisis, are essential for effective decision-making under pressure.

1. Addressing CISO Liability: The SEC charges against Tim Brown highlight the increasing personal liability faced by CISOs. Organizations must provide clear contractual definitions of the CISO's roles and responsibilities, robust indemnification, and legal support for their security leadership. CISOs, in turn, must ensure their security programs are well-documented and demonstrate due diligence, as these records can be critical in legal defense.

The Sunburst incident forced a re-evaluation of what constitutes adequate defense, pushing organizations towards more proactive, transparent, and resilient security architectures.

Key Takeaways

• Supply Chain Attacks are a Real and Present Danger: The Sunburst incident demonstrated that nation-state adversaries can deeply compromise trusted software vendors, turning them into attack vectors for critical targets. Organizations must critically assess and secure their software supply chains.
• Trust and Cross-Functional Collaboration are Paramount in Crisis: Pre-existing relationships and trust between security, legal, executive leadership, and other departments, forged through regular incident response practice, are invaluable for effective crisis management.
• Transparency Builds Trust and Aids Recovery: Despite legal risks, SolarWinds' decision to be aggressively transparent with customers and government agencies was crucial for regaining trust and facilitating collective defense, ultimately contributing to their recovery.
• Incident Response is a Marathon, Not a Sprint: Dealing with a major incident like Sunburst is a multi-year endeavor, encompassing immediate containment, long-term remediation, and protracted legal challenges. Contingency planning and managing team stress and recovery are critical for sustained effort.
• CISO Roles Carry Significant Personal and Legal Liability: The SEC charges against Tim Brown underscore the unprecedented personal risk for CISOs. Clear job descriptions, robust corporate legal support, and indemnification are essential considerations for security leaders.
• Shift to "Secure by Design" and Resilience: The post-incident overhaul at SolarWinds, focusing on peer-reviewed build systems, enhanced visibility (multiple EDRs/SOCs), and a full-time red team, exemplifies the necessity of building security in from the ground up and prioritizing resilience over an elusive goal of absolute security.

About the Speaker(s)

Tim Brown is a seasoned cybersecurity leader, currently serving as the CISO of Team8. Prior to this role, he held the critical position of Chief Information Security Officer at SolarWinds during the infamous Sunburst supply chain attack in December 2020. His keynote address draws directly from this deeply personal and professional experience, offering a rare, first-hand account of navigating a nation-state level cyber crisis.

Brown's talk is distinguished by its blend of technical insight, organizational strategy, and raw human perspective. He candidly shares the challenges of managing an incident of global scale, dealing with media scrutiny, unprecedented legal proceedings (including personal SEC charges), and the immense stress on himself and his team. His career demonstrates a profound commitment to improving cybersecurity practices, with a particular emphasis on fostering resilience, transparency, and a "secure by design" approach in software development and organizational security. His continued work in the industry, including speaking engagements like this one, reflects his dedication to sharing lessons learned to strengthen the collective defense of the cybersecurity community.