From Planning to Impact: Lessons from Poland’s National Cybersecurity Exercises with a Dedicated ...

From Planning to Impact: Lessons from Poland’s National Cybersecurity Exercises with a Dedicated ...

Speakers: Marcin Fronczak (Cybersecurity Foundation), Miroslaw Maj (Open CSIRT Foundation)

Conference: FIRSTCON

YouTube: https://www.youtube.com/watch?v=RWDlRCc0s-M

Overview

In this insightful FIRSTCON talk, Marcin Fronczak and Miroslaw Maj shared the extensive experience and lessons learned from over a decade of organizing national-scale cybersecurity exercises in Poland. The presentation delved into the comprehensive methodology and the pivotal role of their bespoke platform, Cyber Bastion, in conducting complex simulations for critical infrastructure sectors. This discussion is particularly relevant for national cybersecurity agencies, CERTs, and large organizations grappling with the challenges of preparing for sophisticated cyber threats across multiple stakeholders.

The speakers highlighted Poland's journey from pioneering its first national exercise in 2012 to developing a highly refined approach, deeply integrated with European Union directives and international cybersecurity frameworks. The talk underscored the necessity of a structured, adaptable, and technologically supported framework to effectively test national cyber resilience, identify systemic weaknesses, and foster coordinated response capabilities among diverse entities, including government ministries, essential service providers, and law enforcement agencies.

Background

▶ Watch: Introduction and Poland's pioneering cyber exercise history (0:00)

Poland's commitment to national cybersecurity exercises dates back to 2012, when the Cybersecurity Foundation organized its first national-scale exercise for a critical infrastructure sector. This early initiative was pioneering and laid the groundwork for a decade of continuous development in exercise methodologies. Initially, the approach was based on a simple four-phase model—identification, planning, execution, and evaluation—inspired by the 2011 Cyber Europe exercises organized by the European Union Agency for Network and Information Security (ENISA).

Over the years, as both Poland's capabilities and ENISA's guidelines evolved, the methodology became significantly more detailed and robust. The speakers referenced ENISA's updated guide from February of the current year, which proposes a more granular approach to exercise organization. The legal framework underpinning these exercises in Poland is the Krajowy System Cyberbezpieczeństwa (KCS), derived from the NIS Directive (Network and Information Security Directive), specifically NIS Directive version one during the period of the discussed exercises, now transitioning to NIS2. This directive mandates certain entities to participate, but Poland's legislation expanded this to include additional critical roles, such as law enforcement agencies, acknowledging their practical involvement in national incident response. The challenge inherent in these large-scale, multi-stakeholder exercises—involving hundreds of participants from numerous organizations—necessitated a sophisticated platform to manage the intricate planning, execution, and evaluation phases.

Key Findings

▶ Watch: Overcoming challenges of complex conditional injects (16:00)

The central finding presented by Fronczak and Maj is the indispensable role of a dedicated, purpose-built platform like Cyber Bastion for managing the immense complexity of national cybersecurity exercises. Their extensive experience revealed that generic tools like spreadsheets are woefully inadequate for orchestrating scenarios involving hundreds of participants, intricate injects, and real-time monitoring.

Key findings from their decade of experience include:

• Structured Planning is Paramount: Operational exercises require extensive preparation, ideally 32+ weeks, though their seasoned team can condense this to about half a year due to established processes. This includes formal agreements, scenario development, and numerous topic-oriented meetings to gather stakeholder input while maintaining scenario secrecy.
• The Power of Dry Runs: Multiple internal dry runs (where the exercise team simulates all roles) and subsequent dry runs with participants are crucial. This not only validates the scenario logic but also familiarizes participants with the platform, preventing technical issues from disrupting the actual exercise.
• Inject Management is Complex: Scenarios can involve over 100 injects (elementary scenario pieces), many of which are conditional injects. Managing these dynamically, based on participant actions or inactions, is a significant challenge that requires automated support.
• Data-Driven Evaluation: Moving beyond qualitative assessments, the ability to collect granular telemetry and provide quantitative measurements of participant performance is vital for producing concrete, actionable reports and fostering continuous improvement.
• AI for Efficiency: The recent integration of Bastion AI promises to drastically reduce the time and effort required for scenario, infrastructure, and threat generation, marking a significant leap in exercise preparation efficiency, albeit with the necessity for human review.
• Tailored Reporting: Providing individual, anonymized, and tailored reports to each participating organization allows them to understand their specific strengths and weaknesses, making the exercise outcomes directly applicable to their internal improvement processes.

Technical Deep Dive

▶ Watch: Cyber Bastion's scenario compression for efficient dry runs (22:00)

The core of Poland's national cybersecurity exercise capability is the Cyber Bastion platform, a sophisticated environment designed for safe, flexible, and realistic simulation of cyber threats. The platform's strong foundations are built upon several well-known and accepted cybersecurity frameworks:

• MITRE ATT&CK Framework: This framework is extensively used to create realistic attack campaigns. Tactics, Techniques, and Procedures (TTPs) from real-world incidents are mapped to offensive injects, allowing participants to train against contemporary adversary actions. This mapping also facilitates the definition of appropriate mitigations and detection strategies for specific techniques.
• NIST Cybersecurity Framework (NIST CSF): Cyber Bastion uses the NIST CSF's five core functions—Identify, Govern, Protect, Detect, Respond, and Recover—to present and calculate the efficiency of safeguards implemented within the platform. This provides a holistic view of an organization's security posture against simulated threats.
• Defense in Depth Model: Safeguards within the platform are symbolized by cards and categorized into specific layers, reflecting a Defense in Depth architecture. This allows for the analysis of security effectiveness across multiple defensive layers.
• CTI Feeds: To ensure scenarios are current and relevant, the platform integrates Cyber Threat Intelligence (CTI) feeds, allowing for the rapid incorporation of the latest attack methodologies and threat actor behaviors.
• DesInfo Framework: For scenarios involving hybrid attacks and disinformation techniques, Cyber Bastion utilizes a DesInfo Framework, which is analogous to MITRE ATT&CK but specifically tailored for the disinformation domain.

Role Management: The platform's role management is robust, mirroring the structure defined in Poland's national cyber security act. Roles include ministries, sectoral and national teams, and essential service providers. A key flexibility is that one participant can hold multiple roles, and one role can be assigned to multiple participants. Appropriate sets of permissions are granted based on roles, controlling access to injects, communication tools, playbooks, and simulated infrastructure.

Inject Management System: Inject management is a critical feature. For long, multi-day exercises, the number of injects can easily exceed 100. Cyber Bastion's scenario wizard simplifies the creation and management of complex injects, which can be informational, offensive, or even positive (e.g., budget increases). The platform specifically handles conditional injects, which are crucial for dynamic scenarios. For instance, if a national CSIRT is expected to produce a report but fails to do so during the exercise, a conditional inject allows the moderator to simulate that action, ensuring the scenario's logical progression and testing other participants' reactions. This capability moves beyond manual, error-prone methods like Excel spreadsheets used in the past.

Evaluation and Telemetry: Cyber Bastion provides ready-made measurement capabilities to quantify participant performance. It monitors communication schemes and content, ensuring adherence to legal regulations. During execution, a dedicated moderator dashboard tracks progress, communication, and participant actions. It can simulate communications using predefined email templates and trigger flags corresponding to simulated participant actions. Critically, the platform collects comprehensive telemetry throughout the exercise, including all communication between participants, actions taken, and adherence to official procedures. This data is then used to generate analytical reports.

Playbooks and Reporting: National regulations and required actions are implemented as playbooks within Cyber Bastion. This module allows for the analysis and comparison of expected actions against those actually performed by participants. After each exercise, an analytics module generates detailed reports. These reports include:

• Verification of actions taken by individual participants against playbooks.
• Participant resilience, measured by an HP factor.
• Comparison of participant decisions and strategies against recommended system actions (green indicating participant actions, black indicating recommended actions).
• Efficiency of safeguards against specific techniques, analyzed from the perspective of NIST CSF security functions.
• Analysis of which categories and layers of Defense in Depth safeguards were most frequently implemented.

Each participant receives an individual, tailored report to aid their internal improvement processes.

Scenario Compression: To facilitate repeated testing of multi-hour scenarios, Cyber Bastion includes a mechanism to compress the scenario's timeline. This allows a 6-hour scenario, for example, to be tested in approximately 70 minutes, significantly saving time while preserving the logical flow and conditions of the original scenario.

Demo / Proof of Concept

▶ Watch: Detailed post-exercise analytics and tailored participant reports (33:00)

The speakers provided a compelling demonstration of Bastion AI, the platform's newest functionality designed to streamline the preparation of exercises. This AI module is embedded directly into Cyber Bastion and significantly reduces the manual effort involved in scenario and infrastructure generation.

The demonstration showcased the following capabilities:

1. Scenario Generation: The speaker prompted Bastion AI to create a "simple exercise scenario" comprising five informational and 20 offensive injects. The AI, having been trained on platform principles, rapidly generated these injects, with each offensive inject already mapped to specific techniques defined in the MITRE ATT&CK Framework.
2. Scenario Description: Following the inject generation, the AI was asked to prepare a description of the scenario, leveraging CISA requirements, demonstrating its ability to adhere to external standards.
3. Infrastructure Generation: The AI was then tasked with preparing a "simple infrastructure" for the exercise. Initially, it generated basic components, and when prompted to include OT (Operational Technology) environment assets, it quickly adapted and added relevant industrial control system components.
4. Threat Definition: For the generated infrastructure, Bastion AI was asked to define threats on particular assets, including vulnerabilities, generic risks, and indicators of compromise (IoCs). The AI successfully populated these details for each asset, illustrating its capability to enrich the scenario with realistic threat intelligence.
5. Role Assignment: Finally, the AI was used to define specific roles and assign them to assets and injects. This ensures that appropriate attack campaigns are visible only to participants assigned to relevant roles, maintaining the integrity and focus of the exercise.

The speakers emphasized that while Bastion AI dramatically accelerates the initial drafting phase, the generated output serves as a "draft" that still requires human review and refinement to ensure accuracy and alignment with specific exercise objectives. This functionality represents a significant leap forward in reducing the time and complexity associated with designing sophisticated cybersecurity exercises.

Defensive Implications

▶ Watch: Automating scenario and infrastructure creation with Bastion AI (37:00)

The insights shared from Poland's national cybersecurity exercises and the capabilities of the Cyber Bastion platform offer several critical defensive implications for organizations and national entities:

1. Structured Exercise Design is Key: Defenders should adopt a structured methodology for designing and conducting exercises, moving beyond ad-hoc simulations. Leveraging frameworks like ENISA's guidelines ensures comprehensive coverage and effective evaluation.
2. Invest in Dedicated Tools: For organizations with significant operational complexity or those responsible for critical infrastructure, investing in or developing dedicated exercise management platforms is crucial. Generic tools cannot handle the scale, complexity, and dynamic nature of modern cyber threats and multi-stakeholder responses.
3. Framework-Driven Approach: Integrating established cybersecurity frameworks such as MITRE ATT&CK, NIST CSF, and Defense in Depth into exercise design provides a common language for threat modeling, control evaluation, and performance measurement. This allows for a more analytical and data-driven approach to improving security posture.
4. Mastering Conditional Scenarios: The ability to design and manage conditional injects is vital. Real-world incidents are dynamic, and exercise scenarios must reflect this. Defenders need to test how their teams adapt when expected responses don't occur or when unexpected events unfold.
5. Focus on Measurable Outcomes: Exercises should not just be about participation but about measurable improvement. Collecting comprehensive telemetry and generating tailored, quantitative reports (like those from Cyber Bastion) allows organizations to identify specific areas for improvement, validate security controls, and demonstrate tangible progress in resilience.
6. Continuous Preparation and Dry Runs: The emphasis on multiple dry runs—both internal and with participants—highlights the importance of preparation. Defenders should integrate regular dry runs into their exercise planning to validate scenarios, test technical platforms, and ensure all participants are familiar with their roles and tools.
7. Leverage AI for Efficiency (with Caution): Tools like Bastion AI can significantly reduce the burden of scenario and infrastructure generation, allowing security teams to focus more on strategic elements. However, human oversight remains critical to ensure the accuracy, relevance, and pedagogical value of AI-generated content.
8. Internalize National Regulations: For entities within regulated sectors, implementing national cybersecurity regulations as internal playbooks within an exercise platform (as done with KCS in Cyber Bastion) ensures that legal compliance is actively tested and reinforced during simulations.

Key Takeaways

• National-scale cybersecurity exercises demand a robust, structured methodology and a dedicated, sophisticated platform to manage their inherent complexity.
• The Cyber Bastion platform effectively integrates industry-standard frameworks like MITRE ATT&CK, NIST CSF, and Defense in Depth to create realistic and evaluable simulations.
• Effective management of conditional injects is crucial for dynamic scenarios, ensuring the exercise adapts to participant actions (or inactions) in real-time.
• Comprehensive telemetry collection and data-driven, tailored reporting are essential for providing actionable insights and driving continuous improvement in organizational and national cyber resilience.
• AI capabilities, such as Bastion AI, offer significant potential to automate and streamline scenario, infrastructure, and threat generation, drastically reducing exercise preparation time, though human validation remains vital.
• Extensive preparation, including multiple internal and participant dry runs, is non-negotiable for ensuring the smooth and effective execution of large-scale cybersecurity exercises.
• Adopting a "defense in depth" approach, both in security strategy and exercise design, is critical for identifying weaknesses across multiple layers of protection.

About the Speaker(s)

Marcin Fronczak is associated with the Cybersecurity Foundation. He has been a driving force behind Poland's national cybersecurity exercise initiatives, having organized the first national-scale critical infrastructure sector exercise in Poland in 2012. His extensive experience spans over a decade in developing and implementing methodologies for large-scale cyber exercises.

Miroslaw Maj is from the Open CSIRT Foundation. He is also deeply involved in the long-term development and execution of cybersecurity exercises in Poland, contributing to the evolution of the methodologies and the technical capabilities of platforms like Cyber Bastion. Together, they represent a wealth of practical knowledge in national cyber resilience and incident response training.