Firewalls and Fire Alarms: What to Do When Your Best Defenses Go Up in Smoke

Firewalls and Fire Alarms: What to Do When Your Best Defenses Go Up in Smoke

Speakers: Dr. Catherine J. Ullman, University of Buffalo

Conference: Blue Team Con Online

YouTube: https://www.youtube.com/watch?v=xhUUvj0u3zQ

Overview

In "Firewalls and Fire Alarms: What to Do When Your Best Defenses Go Up in Smoke," Dr. Catherine J. Ullman delivers a compelling and insightful presentation that transcends traditional cybersecurity discourse, drawing powerful parallels between fire safety and information security. Dr. Ullman, leveraging her unique background as both a seasoned cybersecurity professional and a volunteer firefighter for nearly 30 years, challenges the pervasive and often dangerous assumption that security controls are infallible. The talk's core message is a proactive embrace of resilience: acknowledging that controls will inevitably fail, and the true measure of an organization's security posture lies in its preparedness for such failures.

Dr. Ullman meticulously illustrates her points through two starkly contrasting yet thematically linked case studies: the catastrophic Iroquois Theater fire in Chicago in 1903, which claimed 602 lives despite being advertised as "fireproof," and the devastating Conti ransomware attack on the Irish Health Service Executive (HSE) in 2021. By dissecting the failures in prevention, detection, and containment in both scenarios, she provides a vivid framework for understanding common pitfalls in security strategies. This presentation is crucial for anyone involved in defending digital assets, from frontline security analysts to executive leadership, as it reframes the challenge from absolute prevention to intelligent, adaptable response.

The significance of Dr. Ullman's talk lies in its ability to cut through technical jargon and deliver fundamental truths about risk management. By demonstrating that a "fireproof" building can burn and a "secure" healthcare system can be crippled, she underscores the critical importance of continuous verification, robust incident response planning, and a culture of adaptability. This is not merely a technical discussion but a strategic imperative, urging organizations to move beyond a false sense of security and cultivate the mindset needed to improvise, adapt, and overcome when their best defenses inevitably go up in smoke.

Background

▶ Watch: Welcome and talk introduction by host (0:00)

The foundational premise of Dr. Ullman's talk is rooted in a critical re-evaluation of what terms like "fireproof" and "secure" truly imply. Drawing from her extensive experience in the fire service, she explains that in the physical world, "fireproof" doesn't mean impervious to fire; it signifies a material or structure capable of withstanding intense heat for a limited duration, delaying the spread of fire. This concept is vividly illustrated by the fact that solid objects don't actually burn directly; rather, they heat up and release vapors that then ignite. This fundamental misunderstanding of fire dynamics mirrors a common misconception in cybersecurity: that a system can be made entirely "hackproof" or "permanently secure."

Historically, the drive for "fireproof" construction emerged from tragic events like the Brooklyn Theater fire, which prompted a demand for safer public venues. This led to the Iroquois Theater in Chicago being marketed as "absolutely fireproof" upon its opening in 1903. However, a devastating fire during a matinee performance of "Mr. Bluebeard" on December 30, 1903, quickly exposed the fallacy of this claim. Over 600 people perished, primarily due to smoke inhalation and being trapped, despite the building's supposed state-of-the-art safety features. This catastrophic event serves as a historical benchmark for the dangers of a false sense of security—a theme Dr. Ullman expertly translates to the digital realm.

In the contemporary context, the Irish Health Service Executive (HSE) faced its own "fire" in May 2021 when the Conti ransomware group launched a highly disruptive attack. As Ireland's public healthcare organization, managing 4,000 locations, 54 acute hospitals, and 130,000 staff, HSE was classified as critical infrastructure. Despite having various infosec controls in place—including a security operations team, a cybersecurity solutions provider, an incident response (IR) provider, antivirus software, and firewalls—the organization was severely crippled. This incident highlighted the vulnerability of even seemingly protected entities when controls are misconfigured, unverified, or inadequately supported by comprehensive plans and training. Dr. Ullman argues that both the Iroquois Theater fire and the HSE ransomware attack underscore a universal truth: controls are merely safeguards designed to delay disaster, not eliminate the possibility of it. The inherent fallibility of any control necessitates a proactive approach focused on resilience and continuous adaptation.

Key Findings

▶ Watch: Dr. Ullman introduces topic and agenda (1:10)

Dr. Ullman's presentation distills several critical findings by interweaving the lessons from the Iroquois Theater fire and the HSE ransomware attack. The overarching conclusion is that no system or structure is truly "fireproof" or "secure" in an absolute sense; rather, all controls are safeguards that serve to delay disaster, not prevent it indefinitely. This foundational understanding underpins all subsequent findings regarding control failures and the imperative for resilience.

The talk categorizes controls into three primary types: Prevention, Detection, and Containment. Across both case studies, Dr. Ullman demonstrates how failures in each of these categories contributed to catastrophic outcomes.

Common Themes of Control Failure:

1. False Sense of Security: In both cases, a belief that existing measures ("fireproof" theater, infosec controls at HSE) were sufficient led to complacency. This assumption was not validated by rigorous testing or verification.
2. Controls Not Acting as Designed:

• Iroquois Theater: The asbestos curtain was mostly wood pulp and jammed; ventilators were nailed shut; sprinklers were non-existent; and exits were locked or obscured.
• HSE: Antivirus was in monitor mode in critical areas, not actively preventing threats; the network architecture was flat, hindering lateral movement detection.

1. Lack of Verification and Testing: Controls were assumed to be in place and functional but were never audited or tested effectively. The theater's building inspection was incomplete due to bribery. HSE lacked formal patch management and verification of AV efficacy.
2. Human Error and Organizational Failures:

• Iroquois Theater: Fire officials were paid off; staff lacked emergency training; management prioritized aesthetics (obscuring exits) and revenue (overcrowding) over safety.
• HSE: IT staff rebooted infected systems, unaware of proper incident response; inconsistent staff knowledge levels; lack of a formal communication plan or security oversight (no CISO).

1. Inadequate Design and Resources:

• Iroquois Theater: Poorly constructed fire escapes; lack of basic communication tools (phone to fire department); inadequate firefighting equipment (kill fire extinguishers).
• HSE: A completely flat network architecture; lack of a true Security Operations Center (SOC); insufficient tools for advanced detection and response (e.g., EDR, SOAR).

Key Learnings and Improvements Post-Incident:

The tragedies spurred significant advancements and policy changes in both domains:

• From the Iroquois Theater Fire: Led to the creation of the UL product safety label, mandating crash bars on exit doors, requiring backup power for exit lights, clear marking of pathways and exits, fire-resistant stage backdrops, direct connection of fire alarms to fire stations, ventilation standards, and strict maximum seating capacities.
• From the HSE Conti Ransomware Attack: Prompted the implementation of a 24/7 monitoring service, appointment of a CISO and CTTO, segregation of legacy systems, new control platforms, organization-wide security awareness training, strict restricted access based on least privilege, development of comprehensive incident response playbooks and effective communication plans, and the adoption of advanced security technologies like SIMs, UBAs, SOAR automation, and vulnerability scanning.

Dr. Ullman emphasizes that while technology adoption is crucial, the most impactful changes often revolve around process, communication, and a fundamental shift in mindset towards anticipating and preparing for control failures.

Technical Deep Dive

▶ Watch: Speaker's background and volunteer firefighter experience (2:18)

Dr. Ullman's technical deep dive is structured around the three categories of controls—prevention, detection, and containment—and how their failures, both in the Iroquois Theater fire and the HSE ransomware attack, reveal critical insights for modern cybersecurity.

Prevention Failures and Solutions

Iroquois Theater (Physical Prevention):

The theater was advertised as "fireproof," yet its construction was rushed. Critical preventative features were compromised:

• Flammable Materials: Despite regulations, highly flammable scenery was allowed in, reportedly with fire officials being paid off.
• Compromised Fire Curtain: The supposed asbestos curtain, a primary barrier to prevent fire spread from the stage to the audience, was largely made of wood pulp and became stuck halfway during the incident.
• Incomplete Inspection: Building codes and inspections were bypassed or ignored, demonstrating a systemic failure in regulatory enforcement and oversight.

HSE (Cyber Prevention):

HSE's preventative posture was severely lacking in fundamental areas:

• End-of-Life Systems: A staggering 30,000 end-of-life Windows 7 systems were still operational, presenting massive, unpatchable vulnerabilities.
• Lack of Patch Management: There was no formal patch maintenance project; patching occurred ad hoc, leaving critical systems exposed.
• Misconfigured Antivirus: While antivirus (AV) software was installed, in many instances, it was configured in monitor mode only, failing to actively stop or quarantine threats.

Technical Solutions for Prevention:

Dr. Ullman stresses the need for:

• Regular Verification and Auditing: Continuously checking that controls are not only present but correctly configured and operational.
• Offensive Security Testing: Employing pen testing and red teaming to actively challenge defenses and identify weaknesses before attackers do.
• Security Awareness Training: Educating users on best practices, including safe data handling and recognizing phishing attempts.
• Prioritizing Crown Jewels: Focusing initial and most rigorous control implementation and testing on the organization's most critical assets. This includes ensuring Access Control Lists (ACLs) are correctly set to prevent unauthorized access.

Detection Failures and Solutions

Iroquois Theater (Physical Detection):

The theater lacked even basic fire detection:

• No Fire Alarm System: Crucially, there was no mechanism to detect the fire early or alert occupants.
• Unconnected Sprinklers: Although planned, sprinklers were either not installed or not connected to a water source, rendering them useless for detection and suppression.

HSE (Cyber Detection):

HSE's detection capabilities were hampered by configuration and architectural issues:

• AV in Monitor Mode: As with prevention, AV in monitor mode meant that while threats might be observed, no automated action was taken to contain them.
• Insufficient Auto-Remediation: Over-reliance on tools that claim "auto-remediation" can be dangerous, as many leave behind remnants that allow attackers to persist or re-establish access.
• Lack of Continuous Monitoring: The absence of active, continuous monitoring meant that initial signs of compromise could be missed or misinterpreted.

Technical Solutions for Detection:

• Verified and Calibrated Detection Systems: Ensuring Intrusion Detection Systems (IDS), Security Information and Event Management (SIEM), and other monitoring tools are correctly configured and regularly tuned.
• Honeypots and Honey Tokens: Deploying these low-cost, high-value tools can provide early warnings of lateral movement and attacker presence, even for organizations with less mature security programs. They are particularly effective because any interaction with them signifies malicious activity.

Containment Failures and Solutions

Iroquois Theater (Physical Containment):

Containment efforts during the fire were almost universally ineffective:

• Stuck Fire Curtain: Failed to contain the fire to the stage.
• Inadequate Fireman Tools: The on-site house fireman had only six small "kill fire extinguishers," completely insufficient for a large stage fire.
• Nailed-Shut Ventilators: Designed to vent hot gases and smoke, these were nailed shut, trapping deadly smoke and heat within the theater.
• No Standpipe Connections: Firefighters couldn't connect hoses inside the building.
• No Sprinkler System: Absence of sprinklers meant no automated internal fire suppression.

HSE (Cyber Containment):

HSE's ability to contain the ransomware spread was severely compromised:

• Flat Network Architecture: The network was "completely flat," meaning there was no network segmentation. This allowed the Conti ransomware to move rapidly and unimpeded across 16 systems and data center servers, affecting 4,000 locations without initial detection of lateral movement.
• No Communication Plan: A lack of clear communication protocols meant that even when issues were identified (e.g., at 1 AM), the right people weren't immediately notified or didn't know who to call.
• AV in Monitor Mode: The inability to quarantine files directly contributed to the rapid spread.
• Untrained IT Staff: Operations staff, upon discovering attacker activity, simply rebooted systems, mistakenly believing this would eradicate the threat, thus losing valuable forensic data and allowing persistence.
• Unhandled Threats: The discovery of unhandled threats on 16 different systems highlighted the failure of existing tools to fully neutralize the attack.

Technical Solutions for Containment:

• Robust Incident Response (IR): Implementing comprehensive IR plans and playbooks that detail steps for isolation, eradication, and recovery.
• System Isolation: Training staff on how to safely isolate compromised systems (e.g., unplugging network cables).
• Network Segmentation: Architecting networks to prevent widespread lateral movement, limiting the blast radius of an attack.
• Effective Communication Plans: Establishing clear internal and external communication channels, including out-of-band communication mechanisms (e.g., satellite phones, dedicated secure chat apps) that function even if primary systems are down.
• Appropriate Tooling: Investing in Endpoint Detection and Response (EDR) solutions that offer advanced detection, response, and auto-remediation capabilities beyond basic AV.
• Consistent Configuration: Ensuring all security tools are consistently and correctly configured across the entire environment.

The deep dive reveals that both historical physical disasters and modern cyberattacks are often not due to a single point of failure but a cascading series of weaknesses in design, implementation, verification, and human response.

Demo / Proof of Concept

▶ Watch: Story 1: The tragic Iroquois Theater Fire (2:45)

The talk "Firewalls and Fire Alarms: What to Do When Your Best Defenses Go Up in Smoke" by Dr. Catherine J. Ullman focuses on theoretical frameworks, historical case studies, and real-world incident analysis rather than demonstrating specific technical exploits or defensive tools. Therefore, there was no live demo or proof of concept presented during this session. The insights were derived from the detailed examination of the Iroquois Theater fire and the HSE ransomware attack, along with the speaker's extensive experience.

Defensive Implications

▶ Watch: Story 2: Irish Health Service Conti ransomware attack (6:20)

Dr. Ullman's talk provides a critical framework for defenders, shifting the mindset from an unattainable goal of absolute prevention to a pragmatic embrace of resilience. The defensive implications are multi-faceted, touching upon strategy, technology, processes, and culture.

1. Embrace Resilience, Not Perfection: The fundamental implication is that organizations must accept that controls will fail. Instead of striving for 100% prevention, the focus should be on building systems and processes that can improvise, adapt, and overcome when incidents occur. This involves routine fire drills (cyber equivalents like tabletop exercises - TTXs) and resilience training that explores "what happens when your tools fail."

1. Layered Security is Paramount: No single control is sufficient. Defenders must implement multiple, overlapping layers of prevention, detection, and containment. If one layer is breached, others should still function to delay the attacker, providing time for response. This mirrors the concept of defense-in-depth, advocating for controls at every stage of a potential attack chain.

1. Continuous Verification and Testing of Controls: The most significant takeaway for defenders is the imperative to regularly verify, audit, and test all security controls. This goes beyond mere deployment; it means actively confirming that firewalls are configured correctly, access controls are enforced, antivirus is in active protection mode (not just monitor mode), and EDR tools are fully functional. This includes:

• Offensive Security Testing: Regular pen testing and red teaming engagements to simulate real-world attacks.
• Detection Testing: Verifying that Intrusion Detection Systems (IDS), SIEMs, honeypots, and honey tokens genuinely detect the threats they are designed to.
• Vulnerability Scanning: Continuously identifying and patching vulnerabilities, especially on end-of-life systems or those with known CVEs.

1. Develop and Drill Comprehensive Incident Response (IR) Plans: A well-defined and regularly practiced IR plan is non-negotiable. This plan must be:

• Documented and Accessible: Stored in multiple formats and locations, including out-of-band communication channels, so it's available even if primary systems are down.
• Comprehensive: Covering all phases from preparation to post-incident analysis.
• Drilled Regularly: Through TTXs involving all relevant stakeholders, from technical teams to executive management, simulating realistic scenarios (e.g., ransomware, data breach).
• Include Playbooks: Specific, step-by-step guides for common incident types.

1. Prioritize Effective Communication: Communication failures were central to both case studies. Defenders must establish clear, multi-level communication plans for incidents:

• Internal Communication: How security teams communicate with IT operations, legal, HR, and executive leadership.
• External Communication: Protocols for engaging with law enforcement, regulators, customers, and the media.
• Out-of-Band Methods: Ensuring communication can continue even if primary network infrastructure is compromised (e.g., VoIP systems going down).

1. Invest in Proper Architectural Design and Tooling:

• Network Segmentation: Moving away from flat networks to segmented architectures that limit lateral movement and contain breaches.
• Modern Security Tools: Deploying and properly configuring EDR, SIEM, User and Entity Behavior Analytics (UBA), and Security Orchestration, Automation, and Response (SOAR) platforms to enhance visibility, detection, and automated response.
• Least Privilege: Restricting access to systems and data to only what is absolutely necessary.

1. Cultivate a Security-Aware Culture: Security is everyone's responsibility.

• Broad Education: Training all employees, not just IT or security staff, on their role in identifying and reporting suspicious activity.
• Empowerment: Creating an environment where employees feel comfortable reporting "weird things" without fear of blame.
• Contextual Training: Teaching IT staff proper incident response actions (e.g., isolating a system by unplugging a network cable) instead of counterproductive measures (e.g., simple rebooting).

1. Understand the Business and Protect Crown Jewels: Defenders must understand how the business functions to identify its crown jewels—the most critical data, systems, and services. Security efforts should be prioritized to protect these assets, ensuring that controls are robust around them and that response plans specifically address their compromise.

By adopting these implications, organizations can move beyond a superficial sense of security and build a truly resilient defense capable of withstanding the inevitable challenges of the modern threat landscape.

Key Takeaways

• Controls Will Fail; Embrace Resilience: No defense is 100% "fireproof" or "secure." Organizations must anticipate control failures and prioritize preparedness and adaptability.
• Verify, Audit, and Test Regularly: Never assume controls are working as intended. Continuously verify their presence, configuration, and effectiveness through auditing, pen testing, red teaming, and detection testing.
• Implement Layered Security: Employ multiple, overlapping controls for prevention, detection, and containment to create defense-in-depth, delaying attackers and minimizing impact.
• Develop and Drill Incident Response Plans: Create comprehensive, documented IR plans and playbooks, including out-of-band communication methods, and practice them regularly with tabletop exercises (TTXs).
• Prioritize Effective Communication and Training: Establish clear communication channels for incidents across all organizational levels (internal and external) and provide consistent security awareness and incident response training to all staff.
• Understand Your Business and Protect Crown Jewels: Identify your most critical assets and ensure security strategies, controls, and response efforts are prioritized to safeguard them effectively.
• Be Adaptable and Prepare for Tool Failures: Conduct drills on "what if our tools fail" to identify alternative manual steps and contingency plans, fostering an "improvise, adapt, overcome" mindset.

About the Speaker(s)

Dr. Catherine J. Ullman is a distinguished voice in the cybersecurity community, known for her ability to bridge disparate fields to offer fresh perspectives on complex security challenges. She has been a staff member at the University of Buffalo for 26 years and brings a unique blend of experience as a volunteer firefighter for close to 30 years. This dual background allows her to draw insightful parallels between physical emergency response and cybersecurity incident management. Dr. Ullman is an active speaker at various conferences and is also known by her social media handle, "investigator check." She is the author of "The Active Defender," a book that explores defensive strategies from an offensive perspective, providing defenders with a deeper understanding of attacker methodologies to enhance their controls and overall security posture.