From Rehearsal to Reality: Building Effective Testing, Exercising and Incident Response Across CNI

From Rehearsal to Reality: Building Effective Testing, Exercising and Incident Response Across CNI

Speakers: Rosanna Chowdhury (Chair, CEO, UK Resilience Academy), Jess Colvin (Global Head of Cyber Operations, City Group), Blake Darché (Head of Threat Intel, Threat Response, and Customer Security Response Operations, Cloudflare), Eloise Hines (Deputy CISO, Bank of England)

Conference: CYBERUK

YouTube: https://www.youtube.com/watch?v=WQuofb2lCjc

Overview

In an era where cyber attacks on critical national infrastructure (CNI) are no longer a hypothetical threat but an inevitable reality, the CYBERUK panel discussion "From Rehearsal to Reality" offered a sobering yet actionable perspective on enhancing national cyber resilience. Chaired by Rosanna Chowdhury of the UK Resilience Academy, the panel – featuring leading experts from City Group, Cloudflare, and the Bank of England – underscored that effective preparedness is not built on elaborate paper plans, but through rigorous, continuous rehearsal under pressure. The core message resonated: organizations that excel in incident response are those that have repeatedly tested assumptions, exposed gaps, and built the confidence to act decisively when a crisis strikes.

The discussion moved beyond mere compliance, delving into the practicalities of achieving genuine operational readiness. Speakers emphasized that national cyber resilience is a collective endeavor, highlighting the critical need for interconnected systems to work seamlessly together. They explored specific strategies for strengthening the UK's ability to respond to and recover from cyber attacks, focusing on the power of collaboration, intelligence sharing, and the evolving role of technology, including artificial intelligence, in shaping future defense mechanisms. This article synthesizes the panel's insights, providing a detailed technical and strategic overview of how organizations, particularly within CNI, can transition from theoretical preparedness to robust, real-world resilience.

Background

▶ Watch: Panel introduction: Cyber attacks no longer hypothetical (0:00)

The premise of the panel discussion was stark: cyber attacks on critical national infrastructure are a matter of "when, not if." This growing threat landscape, as articulated by Blake Darché, has evolved significantly over the last decade, shifting from concerns about isolated malicious files to complex, multi-vector attacks leveraging numerous cloud providers. This evolution necessitates a fundamental re-evaluation of how organizations prepare and respond. The interconnectedness of modern digital ecosystems means that a breach in one entity can have cascading effects across an entire sector, making individual organizational resilience intrinsically linked to collective national resilience.

Jess Colvin highlighted that cyber defense is inherently a "team sport," requiring seamless collaboration, intelligence sharing, and the adoption of common best practices and standards across the ecosystem. Eloise Hines introduced the concept of "severe but plausible scenarios," emphasizing that realistic and believable exercises are crucial for engaging defenders and fostering genuine preparedness. The financial sector, often seen as a benchmark for its mature approach to exercising and recovery, provided a foundational context for many of the discussions. Its history of significant investment, stringent regulatory environments, and a deep understanding of systemic risk has driven advanced practices in incident response and resilience, offering valuable lessons for other CNI sectors. The panel explored how these established practices, coupled with emerging technologies and collaborative frameworks, can elevate the resilience posture of the entire nation.

Key Findings

▶ Watch: Blake Darché: Evolving threats and paramount value of threat sharing (3:25)

The panel discussion brought forth several critical findings and recommendations for enhancing cyber resilience:

1. The Imperative of Continuous Rehearsal: All panelists stressed that preparedness is forged through frequent, varied, and realistic exercises. Jess Colvin highlighted the importance of involving not just cyber teams, but also key decision-makers, boards, and senior leadership in company-wide cyber scenarios. Eloise Hines emphasized using scenarios based on attacks seen in the wild or near-misses to enhance plausibility and engagement. These rehearsals should test agility, flexibility, and muscle memory, moving beyond a "cookie-cutter approach" to address the diverse nature of cyber events.

1. Robust After-Action Processes: Post-incident reviews, or hot washes, are crucial. Jess Colvin advocated for a blame-free environment where teams can openly discuss successes and failures, leading to the productive update of runbooks, decision flows, and training. Blake Darché echoed this, emphasizing that after-action reports are vital for identifying vulnerabilities and preventing future incidents, suggesting that lessons learned should be integrated into playbooks for both human and AI tools.

1. Criticality of Contextualized Threat Information Sharing: Blake Darché argued for more automated and structured threat sharing, moving away from "PDF reports" towards AI-parsed, markdown files for faster dissemination. Jess Colvin noted that sharing is most effective through trusted forums like ISACs (Information Sharing and Analysis Centers), using protocols such as the Traffic Light Protocol (TLP) to manage information sensitivity. The financial sector's intricate nature means that a competitor's bad day often impacts the entire industry, making sharing a matter of collective protection rather than competition.

1. Policy and Regulation as Enablers: Eloise Hines explained that regulatory frameworks, like those at the Bank of England, encourage firms to identify important business services and define minimum viable versions for recovery from catastrophic outages. Policy's role is to frame crucial questions and foster a collective, "team sport" approach to resilience, recognizing that CNI incidents rarely impact a single firm.

1. Government's Role in Cross-Sector Collaboration: Blake Darché identified government and pseudo-government organizations (like ISACs) as ideal neutral intermediaries to facilitate cross-sector information sharing, overcoming inter-organizational politics and competitive hesitancy. Jess Colvin praised the NCSC (National Cyber Security Centre) for its valuable role in sharing specific threat information across the UK.

1. The Transformative Potential of AI: Blake Darché posited that AI will revolutionize the cybersecurity workforce, enabling faster information sharing and automated response. He envisions a future where AI agents, imbued with subject matter expertise, will be central to detection and response, shifting the focus from traditional software engineers to those who can effectively "put together the subject matter expertise into an AI agent for their organization."

1. Beyond the Cyber Team: A Whole-of-Organization Approach: Eloise Hines highlighted the need to view crisis management as a profession across the entire institution, drawing lessons from different types of crises (e.g., bank failure) and promoting broader skill sets like rapid information summarization and decision-tracking under uncertainty. Jess Colvin underscored that every human in an organization is an ingress point for cyber threats, necessitating universal baseline expectations, training, and easy reporting mechanisms for attacks like phishing.

Technical Deep Dive

▶ Watch: Eloise Hines: Engaging defenders with plausible, real-world scenarios (4:32)

The panel, while not presenting code or architectural diagrams, delved deeply into the operational and strategic "how-to" of building cyber resilience, offering insights that are technically critical for effective incident response and recovery.

Exercising Methodologies and Structures:

A significant portion of the discussion focused on the practical implementation of cyber exercises. Jess Colvin advocated for a multi-faceted approach, differentiating between company-wide cyber exercises involving key decision-makers and senior leadership, and smaller, more focused geographical or business-transaction-limited events. The goal is to test organizational agility and flexibility, recognizing that "no two cyber events are the same." She mentioned gamification of decision-making processes, where leaders practice responding to complex scenarios like dealing with untrusted third parties, vulnerability exploits, or DDoS attacks. A particularly impactful technique mentioned by Colvin is the use of no-notice exercises, which, while unpopular with teams, inject a crucial element of realism by simulating unexpected events, conflicting priorities, or the unavailability of key personnel.

Eloise Hines introduced the Bank of England's "Pizza in a Puzzle" initiative, a regular, short, and sharp exercise conducted every other month. The key is that the puzzle isn't known until participants enter the room, preventing pre-planning and truly testing immediate response capabilities and muscle memory. Blake Darché added that every incident, regardless of perceived severity (e.g., a lost laptop), should be treated as a crisis to continuously practice and refine the incident command structure and the role of the incident commander. This constant state of readiness ensures that when a major event occurs, the muscle memory is ingrained.

Recovery Practices:

A specific technical recovery practice discussed was bare metal restores. Jess Colvin noted this as a prominent topic across the industry, highlighting that while there are different approaches, the conversation and sharing of practices are vital for organizations to determine what works best for their specific environment. The concept of identifying minimum viable versions of important business services was also central, guiding recovery efforts to restore critical functions first in catastrophic scenarios.

Information Sharing Mechanisms:

The panel emphasized structured and efficient information sharing. Jess Colvin detailed the use of ISACs (Information Sharing and Analysis Centers) as trusted forums for sharing, often governed by explicit sharing agreements. The Traffic Light Protocol (TLP) was highlighted as an "important wrapper" for contextualizing the sensitivity and dissemination of shared intelligence, enabling organizations to understand how widely they can share specific pieces of information. Blake Darché envisioned a future where AI tools would replace "PDF reports" with structured markdown files, dramatically speeding up data parsing and information exchange. This shift from human-intensive, context-missing chats to automated, context-rich data transfer is seen as critical for keeping pace with the evolving threat landscape. The NCSC was specifically recognized for its role in sharing threat information with organizations across the UK, demonstrating a successful model of government-industry collaboration.

Third-Party Risk Management:

Blake Darché and Jess Colvin underscored the escalating challenge of third-party cyber events, which have seen a significant increase in the past two years. Colvin outlined a critical technical approach to managing this risk:

1. Inventory of Critical Third Parties: Understanding which partners are truly essential and on whom the organization is most reliant.
2. Connectivity Mapping: Detailed knowledge of the level and directionality of connectivity (ingress and egress points) between the organization's environment and third parties (e.g., email data exchange vs. direct network connections).
3. Data Exposure Assessment: A clear understanding of what confidential, PII, or market data resides with third parties.

This granular understanding is vital for rapid decision-making during an incident involving a third-party compromise.

Training and Workforce Development:

Beyond specific technical tools, the panel discussed the technical skills required for an adaptive workforce. Blake Darché emphasized the need for employees to be "naturally inquisitive" and capable of self-learning, rather than solely relying on formal training. He cited examples where incident responders learn about unknown or unapproved systems during a live incident, highlighting the continuous discovery nature of the field. Eloise Hines stressed the importance of fostering constant curiosity and creating opportunities for less experienced staff to participate in exercises and incident discussions, even if just to observe, to grow the "next generation" of responders. The concept of crisis management as a broader profession within an organization, sharing skills across different types of crises, was also a key technical insight into building organizational resilience beyond just the cyber team.

Demo / Proof of Concept

▶ Watch: Finance sector practices: Frequent exercises, engaging senior leadership (5:30)

As a panel discussion, the session did not include any live demonstrations or proofs of concept. The content focused on strategic insights, operational methodologies, and lessons learned from real-world experiences in cyber resilience.

Defensive Implications

▶ Watch: Importance of blame-free after-action reviews for continuous learning (6:40)

The insights from the "From Rehearsal to Reality" panel provide a clear roadmap for defenders seeking to bolster their cyber resilience:

1. Implement Diverse and Frequent Exercises:

• Go beyond compliance: Design exercises that are plausible, severe, and based on real-world attacks or near-misses, as advocated by Eloise Hines.
• Involve all stakeholders: Ensure senior leadership, boards, and non-cyber decision-makers participate, preparing them for the strategic and business decisions required during a crisis, as Jess Colvin advised.
• Vary scenarios: Conduct a spectrum of exercises, from company-wide crisis simulations to smaller, geographically or business-transaction-limited events, testing adaptability.
• Incorporate realism: Consider no-notice exercises to simulate unexpected conditions, or "Pizza in a Puzzle" sessions for regular, unscripted challenges.
• Engage third parties: Include critical partners in exercises to test collective response and recovery capabilities.

1. Establish a Robust After-Action and Continuous Improvement Cycle:

• Foster a blame-free environment: Encourage open discussion of what went well and what didn't, focusing on organizational learning and improvement.
• Prioritize ruthlessly: Categorize identified lessons into people, process, and technology gaps, then aggressively prioritize fixes for those offering the greatest risk reduction or resilience gain.
• Integrate lessons into playbooks: Update runbooks and decision flows, and explore how AI tools can embed these lessons for automated response, as Blake Darché suggested.
• Revisit past scenarios: Periodically re-test identified gaps to ensure they have been truly fixed and not just "identified."

1. Enhance Threat Intelligence Sharing:

• Actively participate in trusted forums: Leverage ISACs and similar communities, utilizing protocols like TLP (Traffic Light Protocol) for secure and contextualized sharing.
• Advocate for automated sharing: Push for structured, machine-readable intelligence formats, moving away from manual "PDF reports" towards AI-parsed data for faster, more effective dissemination.
• Collaborate with government: Work closely with national bodies like the NCSC to receive and contribute to critical threat intelligence.

1. Develop an Adaptive and Curious Workforce:

• Cultivate self-learners: Encourage employees to be "naturally inquisitive" and adaptable, as Blake Darché emphasized, recognizing that the threat landscape constantly introduces new challenges.
• Broaden crisis management skills: View crisis management as a cross-functional profession, sharing skills (e.g., rapid information summarization, decision tracking) across different types of institutional crises.
• Provide growth opportunities: Leaders should create spaces for less experienced staff to observe and participate in exercises and incident responses to build future capabilities.
• Baseline organizational awareness: Implement regular training for all employees on identifying and reporting cyber threats like phishing, making it easy for them to contribute to the organization's defense.

1. Strengthen Third-Party Resilience:

• Maintain a critical third-party inventory: Know your essential partners and dependencies.
• Map connectivity and data exposure: Understand the ingress/egress points and what confidential or PII data resides with third parties.
• Engage partners in planning: Develop shared resilience plans and ensure you know "who to call in a crisis" among your critical third parties, ideally having built trust and relationships beforehand.

1. Focus on Minimum Viable Recovery:

• Identify important business services: Clearly define the critical functions that must be restored first in a catastrophic event.
• Determine minimum viable states: Understand what a reduced, but functional, version of these services looks like and what resources are absolutely required to operate them.
• Practice bare metal restores: Regularly test fundamental recovery mechanisms to ensure their effectiveness.

By embracing these defensive implications, organizations can move beyond a reactive stance, building a proactive, adaptive, and collaborative resilience posture that can withstand the inevitable cyber challenges of the future.

Key Takeaways

• Act Now, Continuously: Cyber resilience is a continuous discipline, not a one-off project. It must be built proactively through ongoing investment in people, resources, and frequent rehearsal.
• Rehearse Reality, Not Just Plans: Test the actual response and recovery capabilities of people, processes, and technology under pressure, rather than merely validating paper plans. Include diverse scenarios and stakeholders.
• Share and Collaborate Systematically: Leverage trusted forums like ISACs, utilize protocols like TLP, and advocate for automated, structured threat intelligence sharing to enhance collective defense across sectors and with government.
• Empower an Adaptive Workforce: Foster a culture of natural curiosity, self-learning, and cross-functional crisis management. Provide opportunities for all staff, including less experienced individuals, to build critical incident response muscle memory.
• Master Third-Party Risk: Develop a clear inventory of critical third parties, understand their connectivity to your environment, and assess data exposure. Engage these partners in resilience planning and exercises.
• Leverage Emerging Tech (AI): Explore the potential of AI tools for automating threat intelligence parsing, accelerating information sharing, and building intelligent agents for faster, more efficient cyber response.

About the Speaker(s)

• Rosanna Chowdhury (Chair): CEO of the UK Resilience Academy, she expertly moderated the panel, guiding the discussion on moving from theoretical preparedness to practical operational readiness in cyber resilience.
• Jess Colvin: As the Global Head of Cyber Operations for City Group, Jess brings extensive experience in managing cyber threats across a vast financial enterprise. She is based in the UK and is an active member of the UK cyber community, also serving as a board member of the Financial Services ISAC, a global organization integral to information and threat sharing.
• Blake Darché: Head of Threat Intel, Threat Response, and Customer Security Response Operations at Cloudflare, Blake provides a unique perspective from a leading internet infrastructure and security company. His expertise lies in addressing the evolving threat landscape and advocating for automated, efficient threat information sharing.
• Eloise Hines: Deputy CISO at the Bank of England, Eloise offers insights from a critical national infrastructure operator within the financial sector. Her focus includes leveraging plausible but severe scenarios for exercising, defining important business services for recovery, and fostering a broader, collaborative approach to crisis management.