Tales from the Inside: An Incident Response Weekend

Sahan Fernando

CypherCon 9.0 (2026)

Overview

Sahan Fernando, Chief Information Security Officer for Rady Children's Health, presented "Tales from the Inside: An Incident Response Weekend" at CypherCon, offering a candid account of a real-world cybersecurity incident from January 2024. This talk focuses on a near-catastrophic breach that was narrowly averted through swift, instinctual, and well-coordinated incident response. Fernando's presentation provides a transparent look at the challenges, successes, and hard-learned lessons from an event that could have crippled a major pediatric healthcare system.

Watch on YouTube

Visual summary for Tales from the Inside: An Incident Response Weekend by Sahan Fernando
Visual summary for Tales from the Inside: An Incident Response Weekend by Sahan Fernando

Key moments

  1. 4:00 Ray Children's Health: Pediatric care challenges
  2. 6:00 Initial brute force attack and Microsoft Teams red herring
  3. 10:00 Detecting Kerberoasting and rapid incident response
  4. 16:00 MFA gap exploited; attacker registers personal phone
  5. 18:45 Operational fallout: PAX, autologons, and legacy systems
  6. 25:00 FBI alert: compromised accounts found on dark web
  7. 29:00 Foundational security, hygiene, and relationships are crucial

Tales from the Inside: An Incident Response Weekend

Speakers: Sahan Fernando, Chief Information Security Officer, Rady Children's Health

Conference: CypherCon

YouTube: https://www.youtube.com/watch?v=SFBt6oI_p0I

Overview

Sahan Fernando, Chief Information Security Officer for Rady Children's Health, presented "Tales from the Inside: An Incident Response Weekend" at CypherCon, offering a candid account of a real-world cybersecurity incident from January 2024. This talk focuses on a near-catastrophic breach that was narrowly averted through swift, instinctual, and well-coordinated incident response. Fernando's presentation provides a transparent look at the challenges, successes, and hard-learned lessons from an event that could have crippled a major pediatric healthcare system.

The talk provides invaluable insights into defending a large, integrated healthcare environment operating on thin margins, where legacy systems and critical patient care needs often clash with robust security implementations. Fernando's objective is to foster community discussion around "near-miss" incidents, highlighting the practical application of security hygiene and the critical importance of strong organizational relationships during high-stakes events. This detailed recounting serves as a practical guide for security professionals, particularly in healthcare, on navigating high-pressure incident response while maintaining operational continuity.

The incident involved an initial brute-force campaign targeting the healthcare sector, followed by a sophisticated Kerberoasting attack that breached internal systems. Fernando emphasizes how foundational security practices, quick decision-making, and pre-established credibility were instrumental in containing the threat before it resulted in a reportable data breach or widespread operational disruption, a fate that tragically befell a peer organization shortly after.

Background

▶ Watch: Ray Children's Health: Pediatric care challenges (4:00)

The incident's context is rooted in the unique operational and economic realities of the pediatric healthcare sector. Rady Children's Health, a large pediatric health system with over 900 beds, operates with thin financial margins. This often limits significant IT infrastructure investment, leading to prolonged use of legacy systems and a constant balance between efficiency and security. Fernando highlighted reliance on Active Directory Federation Services (ADFS) for authentication claims against Microsoft 365, a legacy deployment that complicated early detection. Pediatric care's specialized nature means service disruptions cannot easily be mitigated by diverting patients, amplifying outage impact.

The incident began on a Friday morning in January 2024 with a surge of help desk calls reporting user lockouts. Initially, the team was sidetracked by a red herring: a concurrent Microsoft Teams outage, underscoring how Occam's Razor can mislead. The true nature of the threat emerged as a massive, programmatic brute-force campaign specifically targeting healthcare. Attackers systematically attempted credential combinations, repeatedly hitting ADFS, causing widespread account lockouts and significant operational challenges.

Despite initial containment of the brute-force campaign by Friday afternoon, the threat actor had established a foothold. The organization’s legacy infrastructure, coupled with inherent complexities—including overprivileged service accounts, undocumented configurations, and a culture prioritizing speed (e.g., "autologons" for quick Epic access)—created fertile ground for a more advanced attack. This incident serves as a stark reminder of how persistent threats exploit even minor vulnerabilities in complex, mission-critical environments.

Key Findings

▶ Watch: Detecting Kerberoasting and rapid incident response (10:00)

The incident unfolded in two distinct, yet related, phases, revealing critical findings about the threat actor's methods and organizational vulnerabilities.

First, a brute-force campaign targeted Microsoft 365 accounts via Rady Children's Health's legacy ADFS environment. This widespread, programmatic attack against healthcare caused extensive account lockouts.

Second, and more critical, was the detection of Kerberoasting activity early Saturday morning. An alert from the Managed Security Service Provider (MSSP) flagged an internal account querying numerous Service Principal Names (SPNs) from Active Directory (AD). This is a hallmark of Kerberoasting, a technique to extract Kerberos service ticket hashes for offline cracking, enabling lateral movement and privilege escalation.

Investigation revealed the compromised account belonged to a social worker with access to the Electronic Medical Record (EMR) and sensitive patient encounters. The threat actor gained internal network access via the Citrix environment. A critical vulnerability: the social worker's account, authorized for Multi-Factor Authentication (MFA), had never enrolled. The attacker exploited this, registering their own device (a specific Samsung phone from Asia) as the MFA authenticator, bypassing a crucial security control.

A significant outcome was rapid containment, preventing data exfiltration or widespread damage. The team caught the attacker quickly, limiting activity primarily to basic AD querying. This meant no reportable breach under HIPAA, averting severe legal and reputational consequences.

However, response actions led to unexpected operational impacts. Broad password resets for over 300 active accounts and numerous service accounts caused disruptions on Sunday. Specifically, Kerberos armoring, enabled to mitigate future Kerberoasting, interacted negatively with legacy systems. After a 24-hour window, critical applications like PACS (picture archiving and communication system) and autologons (bedside machines directly authenticating to Windows) began failing. This was traced to an overprivileged Citrix service account also launching the PACS service, highlighting fragile interdependencies.

Finally, an FBI notification revealed several Rady Children's Health accounts, including the compromised one, were listed for sale on the dark web. This suggested the initial compromise was via info-stealing malware or the brute-force campaign, reinforcing the sophisticated nature of the threat. This intelligence allowed additional preventative measures against other potentially compromised accounts.

Technical Deep Dive

▶ Watch: MFA gap exploited; attacker registers personal phone (16:00)

The incident at Rady Children's Health demonstrated a multi-stage attack exploiting common enterprise vulnerabilities, particularly in environments with legacy components. The initial stage involved a brute-force attack targeting Microsoft 365. The organization's reliance on Active Directory Federation Services (ADFS) to relay authentication claims from Microsoft 365 (now Microsoft Entra ID) back to on-premises Active Directory meant brute-force attempts hit ADFS, causing user account lockouts. This programmatic nature indicated a sophisticated, targeted campaign.

The critical turning point was the detection of Kerberoasting. This attack technique exploits Kerberos authentication within Active Directory. An attacker, even with low privileges, can request service tickets for any Service Principal Name (SPN) registered in AD. The Domain Controller encrypts this ticket using the service account's NTLM hash. Attackers then take these tickets offline to crack the password hash. The initial alert was triggered by an internal account querying a bunch of SPNs from Active Directory, highly anomalous behavior for a standard user or application.

The compromised social worker account gained access via the Citrix environment. A critical security gap was identified: the account was authorized for Multi-Factor Authentication (MFA) but the user never completed enrollment. This allowed the threat actor, with compromised credentials, to log into Citrix and, during the authentication flow, register their own device for MFA. The artifact was a specific Samsung phone model, indicating an origin in Asia, a strong indicator of compromise. This highlights a common MFA deployment pitfall: authorization does not always equal enforcement.

Incident response involved technical containment and recovery. To mitigate Kerberoasting and potential lateral movement, the team performed scoped password resets for approximately 300 active users logged into Citrix, plus affected service accounts. This avoided a catastrophic full domain reset. The speaker specifically mentioned a 20-year-old domain with a service account from 2006, possessing local administrator privileges and an unchanged password—a prime Kerberoasting target.

As a proactive defense, Kerberos armoring (Kerberos Flexible Authentication Secure Tunneling or FAST) was enabled. This enhances protection against offline password cracking by encrypting the entire Kerberos exchange. However, its implementation led to unforeseen operational issues on Sunday. After 24 hours, as systems re-authenticated and received new Kerberos claims, critical applications like PACS and autologons (bedside Windows machines configured for direct desktop authentication) began failing. This was traced to an overprivileged Citrix service account that, unknowingly, launched the PACS service. The resolution involved changing this service account's logon type to "log on as local system," revealing deep, undocumented interdependencies in legacy environments.

Other hardening measures included:

  • MFA Audits: Checking all MFA registrations for suspicious activity.
  • Remote Enrollment Restrictions: Implementing a policy to stop remote MFA enrollment after 14 days.
  • Authentication Hardening: Moving away from knowledge-based authentication (NKA) for help desk password resets to video-only verification with ID.
  • EDR (Endpoint Detection and Response): The EDR system was crucial for detecting Kerberoasting, but its incomplete deployment across the Citrix environment due to server team concerns limited initial visibility.
  • Health ISAC and FBI Collaboration: Attempted sharing of Tactics, Techniques, and Procedures (TTPs) with Health ISAC, and later, coordination with the FBI for dark web intelligence about compromised accounts.

This incident underscores the complex interplay of legacy infrastructure, human factors (MFA enrollment), and the ripple effects of security changes in a highly integrated, patient-critical environment.

Demo / Proof of Concept

▶ Watch: FBI alert: compromised accounts found on dark web (25:00)

The talk did not include a live demonstration or proof of concept. Sahan Fernando's presentation was a narrative account of a real-world incident, focusing on the chronological unfolding of events, the challenges faced, and the lessons learned from the perspective of an incident response leader.

Defensive Implications

▶ Watch: Foundational security, hygiene, and relationships are crucial (29:00)

This incident offers significant defensive implications, especially for organizations in healthcare with similar challenges:

  1. Prioritize Foundational Security: Emphasize good hygiene, patching, robust account cleanup, and strong password policies. Address technical debt like overprivileged, ancient service accounts. Proactive efforts to eliminate attack paths are paramount.
  1. Enforce and Audit MFA Rigorously: Move beyond merely enabling MFA. Actively enforce enrollment and audit MFA registrations regularly. Implement policies like restricting remote enrollment and checking MFA logs for suspicious activity.
  1. Proactively Address Legacy Systems: Identify, document, and remediate legacy vulnerabilities. Use incidents as "political capital" to advocate for modernization, such as migrating to SaaS Identity Providers (IDPs) like Microsoft Entra ID and implementing phishing-resistant authentication.
  1. Enhance EDR Coverage: Ensure comprehensive EDR deployment across all critical assets, including VDI and remote access platforms. Collaborate with infrastructure teams to overcome performance barriers and ensure full telemetry.
  1. Strengthen Identity and Access Management (IAM): Implement granular access controls and the principle of least privilege. Break down monolithic service accounts. Transition to non-knowledge-based authentication (NKA) for helpdesk password resets.
  1. Develop Flexible Incident Response (IR) Playbooks: Playbooks should guide, not rigidly script, allowing responders to adapt. Regular drills build "muscle memory" for decision-making under pressure.
  1. Cultivate Strong Organizational Relationships: Pre-existing credibility and trust with operations and leadership enable faster, more effective incident response. Understanding business operations and communicating risks in context are vital for cooperation.
  1. Participate in Threat Intelligence Sharing: Share Tactics, Techniques, and Procedures (TTPs) with organizations like Health ISAC. Engage with law enforcement (e.g., FBI) for external intelligence, aiding in identifying compromised accounts.
  1. Document Critical Systems Thoroughly: Lack of documentation for the PACS service account caused significant recovery delays. Ensure thorough documentation of application dependencies, service account functions, and network configurations to reduce mean time to recovery.

Key Takeaways

  • Foundational security practices are paramount: Good hygiene, patching, and account management are often the most effective defenses.
  • MFA enforcement, not just enablement, is critical: Rigorous auditing of MFA registrations and proactive measures against bypasses are essential.
  • Legacy systems are high-risk: Technical debt significantly increases the attack surface and complicates incident response.
  • Relationships and credibility matter: Pre-established trust with leadership and operations enables faster, more effective incident response.
  • Incident response requires flexibility: Adaptability and instinctual decisions under pressure are crucial for containment.
  • Visibility is non-negotiable: Comprehensive EDR deployment across all critical environments is essential for early detection.

About the Speaker(s)

Sahan Fernando is the Chief Information Security Officer (CISO) for Rady Children's Health, a large pediatric health system. He oversees cybersecurity strategy and incident response for a complex, integrated healthcare environment. With a strong background as an IR commander and in various security roles, Fernando brings a pragmatic, mission-driven approach to cybersecurity, understanding the unique challenges and operational constraints within healthcare. This talk at CypherCon marked his first presentation to a non-healthcare-specific audience, reflecting his commitment to transparency and community knowledge sharing.

Reviews

Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT

Usually when a healthcare CISO takes the stage, you get 45 minutes of sanitized LinkedIn platitudes about 'resilience.' Fernando actually shares the dirty laundry on a near-miss, admitting to unenrolled MFA gaps, 18-year-old service accounts, and breaking hospital infrastructure mid-firefight.

Heather Calloway (CISO) — MUST SEE

Fernando provides an unvarnished, high-value autopsy of a near-miss at a pediatric health system. It’s a textbook case study on how a simple identity governance failure can escalate into a domain-level crisis, and it is required viewing for any security leader balancing legacy tech debt with operational uptime.

→ Top-rated talks at CypherCon 9.0 (2026)

All talks from CypherCon 9.0 (2026)