Tabletop Lessons from 3 Decades and 2 Continents

Tabletop Lessons from 3 Decades and 2 Continents

Speakers: Kenneth Van Wyk (KRvW Associates, LLC)

Conference: FIRSTCON

YouTube: https://www.youtube.com/watch?v=ERFAKdOPqZ0

Overview

In "Tabletop Lessons from 3 Decades and 2 Continents," Kenneth Van Wyk of KRvW Associates, LLC, presents a compelling analysis of the most common and egregious mistakes observed in incident response (IR) operations and tabletop exercises. Collaborating with Elliot Atkins (who was unable to attend), Van Wyk distills insights from their combined decades of experience, spanning over 150 facilitated exercises across diverse industries and geographies. The central thesis is that while organizations often possess strong technical capabilities, their incident response efforts are frequently undermined by non-technical, operational, and procedural failures.

The talk highlights that these recurring issues are not about a lack of technical skill but rather deficiencies in areas like process adherence, clear communication, decisive leadership, and robust action tracking. Van Wyk argues that these "simple" mistakes often lead to the most significant financial and reputational costs during a crisis. The presentation serves as a critical call to action for the security community, particularly those in day-to-day operational roles, to shift their focus from purely technical readiness to cultivating an ingrained operational discipline that treats incident response as an imperative.

This deep dive into the human and process elements of incident response is crucial because it addresses the often-overlooked vulnerabilities that persist even in technically sophisticated environments. By identifying these systemic weaknesses and proposing practical, experience-driven remediations, Van Wyk provides invaluable guidance for organizations aiming to achieve truly resilient and effective incident response capabilities. The talk emphasizes that continuous, focused practice and a disciplined operational mindset are far more impactful than voluminous documentation or advanced tools alone.

Background

▶ Watch: Introduction and co-presenter's unexpected absence (0:00)

The genesis of this talk stems from Kenneth Van Wyk's observation of repeated mistakes across various clients and industries during his extensive career in incident response and tabletop exercises. Recognizing these patterns, he reached out to the FIRST community for collaborators and found a kindred spirit in Elliot Atkins from the UK. Despite working a continent apart and with completely isolated datasets, their collective experience of facilitating well over 150 exercises revealed a striking commonality in the operational failures they witnessed.

Van Wyk explicitly states that their work is "not a scientific statistical study," but rather a distillation of their most common and egregious findings. He notes a significant shift from his early career expectations: over 20 years ago, he anticipated that organizations would primarily stumble on technical errors during incidents. However, he quickly discovered that clients generally possess "pretty darn good technical capabilities," often better than outsiders could provide, and they understand their business systems well. The real vulnerabilities, he found, lay elsewhere.

The persistent problem, according to Van Wyk, is not a lack of technical expertise or even a lack of desire to "do the correct things" from incident response teams. Instead, it resides in the operational execution, communication, and decision-making frameworks surrounding the technical response. These are the areas where teams, even with good intentions and strong technical skills, frequently fall short, leading to significant and often avoidable costs. The talk serves as a platform to share these collective observations and initiate a broader discussion within the community, with Van Wyk openly inviting feedback and alternative remediation ideas, emphasizing that their proposed solutions are grounded in opinion and experience, not scientific fact.

Key Findings

▶ Watch: Problem with overly complex incident response plans (7:40)

The core of Van Wyk's presentation centers on a series of critical, non-technical failures that consistently undermine incident response effectiveness. These findings, drawn from decades of experience, highlight the gap between technical capability and operational discipline:

1. Unusable Incident Response Plans (IRPs): Organizations frequently possess IRPs that are excessively long and verbose, often exceeding 100 pages, with some seen as long as 250 pages. While these documents serve a purpose in process design and regulatory compliance, they are "useless" for operational guidance during a crisis. Responders carrying freshly printed IRPs into a war room is a sign of this fundamental flaw, as such documents cannot provide the "break glass" level of concise, actionable instruction needed in an emergency.

1. Pervasive Procedural Errors: Simple yet critical procedural steps are routinely overlooked. Examples include failing to maintain a chain of custody for evidence, which renders collected data inadmissible for legal or forensic purposes ("you can't get that whipped cream back into the can"). Another common and costly error is the failure to contact cyber insurance providers early in an incident. Waiting until after the fact can result in denied claims, turning a potentially recoverable loss into a significant financial burden.

1. Lack of Business Prioritization in Triage: Incident response teams often approach triage from a purely technical perspective, neglecting to engage business owners or understand the broader business impact. This leads to misprioritization, where a "medium" technical incident might be a "huge business problem" (e.g., a denial-of-service attack impacting temperature controls for critical clinical trial samples in a pharmaceutical company, or an outage affecting power company operations). Without business input, IR efforts can miss the true target of mitigation.

1. Undefined Decision Authority: During high-stress incidents, teams frequently "spin their gears" attempting to determine who has the authority to make critical decisions. Questions like "Who pays the ransom?", "Who contacts law enforcement?", or "Who speaks to the media?" often lack clear, pre-defined answers, leading to delays and confusion. This ambiguity paralyzes response efforts and can result in incorrect or delayed actions.

1. Personnel Unavailability and Lack of Escalation: Key personnel, such as legal counsel or media relations, are often absent during tabletop exercises or real-world incidents. Organizations frequently lack well-defined escalation chains or deputies who can step in when primary contacts are unavailable. This absence of critical decision-makers or communicators can severely hamper incident resolution, as incidents "don't wait for us."

1. Failure to Decide: Teams engage in extensive discussions and debates about critical issues but then "move on without having ever made a decision." Van Wyk notes this happens "all the time" in exercises, where the facilitator must resist the urge to intervene and let the team experience the consequences of indecision. This paralysis is a significant operational flaw, as operations demand definitive action.

1. Inadequate Action Assignment and Tracking: Despite the critical nature of incident response, many teams fail to rigorously track assigned actions. Tasks are given without clear ownership, deadlines, or follow-up mechanisms. Van Wyk advocates for a "pedantic" approach, using simple tools like whiteboards to record: who is doing what, when it was assigned, how long it's expected to take, and when to check back. Without this discipline, tasks are forgotten, duplicated, or left incomplete.

1. Poor Operational Communications: Communications during an incident are often ambiguous, non-explicit, and lack the precision required for high-stakes operations. Misinterpretations ("I thought you meant this database, not that one") lead to costly errors. Van Wyk draws an analogy to air traffic control, where commands are explicit, concise, and acknowledged through repetition to ensure mutual understanding and correct execution.

These findings collectively underscore that while technical prowess is a prerequisite, it is the adherence to operational discipline, clear processes, and effective human coordination that ultimately determines the success or failure of an incident response effort.

Technical Deep Dive

▶ Watch: Prioritizing business impact during incident triage (14:30)

While Van Wyk explicitly states that the most significant and costly errors in incident response are rarely technical, the talk implicitly delves into the operational architecture and procedural protocols that should underpin an effective incident response capability. The "technical deep dive" here, therefore, focuses on the structural and process-oriented elements that, when absent or flawed, lead to the "non-technical" failures described.

The speaker highlights that the Incident Response Plan (IRP), while often bloated and operationally useless for the typical responder, serves a critical architectural purpose. It is the repository for the detailed design of the IR process, documenting how the organization intends to respond. The "useful audience" for a 250-page IRP, according to Van Wyk, is a "very small number of people," perhaps "five or six," who need to understand the intricate process flow and design. For operational teams, the architectural need is for highly distilled, actionable guidance, akin to "in case of fire, break glass" instructions. This implies a need for a layered documentation architecture: comprehensive backend process design for architects and auditors, and concise frontend playbooks or checklists for responders.

A central "protocol" proposed is the Incident Leader role. This individual, often called an Incident Commander or Incident Manager, is not necessarily the most senior person or the ultimate decision-maker, but rather the "orchestrator" or "symphony orchestra conductor." Their protocol involves:

• Process Orchestration: Ensuring the IR process flows smoothly, knowing "who's doing what."
• Concise Guidance: Providing clear, short directives to various stakeholders (e.g., "General Counsel, I need you to focus on these five issues right now").
• Decision Facilitation: Ensuring that decisions are actually made, rather than merely debated, by driving the discussion to a conclusion.
• Action Tracking Oversight: Maintaining awareness of all assigned actions and verifying their progress.

The lack of clear decision authority points to a flawed organizational "protocol" for crisis management. Critical decisions (e.g., ransom payment, law enforcement engagement, media communication) require a pre-defined architecture of who is empowered to decide, and how those decisions are escalated and communicated. This architectural element should be explicitly documented within the IRP design, even if not in the operational playbook.

Operational communications are presented as a critical "protocol." Van Wyk champions a highly disciplined communication model, drawing parallels to air traffic control. This "protocol" involves:

• Explicitness and Conciseness: Instructions must be unambiguous and to the point.
• Read-Backs/Verification: The recipient repeats the command to confirm correct understanding ("Roger. Clear to take off on runway 1-9."), ensuring mutual comprehension before execution. This is a fundamental error-prevention mechanism.

For action assignment and tracking, the "technical deep dive" reveals a preference for simple, low-tech solutions over complex systems. Van Wyk's "favorite incident response tool" for operations is a whiteboard and pen. This "system" supports a crucial "protocol":

• Visual Tracking: Writing down tasks, assignees, deadlines, and expected feedback times.
• Dynamic Updates: Easy to erase and revise action items.
• Record Keeping: Periodically taking photos with a camera phone to create an audit trail of actions.
• Accountability: The incident leader uses this system to track every action, set reminders (e.g., "Alexa, remind me in 30 minutes"), and verify completion.

The integration of business prioritization into the triage "architecture" is also crucial. This involves not treating incidents as purely IT problems but understanding the business processes each system supports. The "protocol" is to involve relevant business owners from the outset, allowing their input to drive the prioritization of response efforts. This mandates a pre-existing understanding of critical business assets and their dependencies on IT systems.

Finally, the discussion on training and exercises describes a "protocol" for building operational muscle memory. Inspired by Navy SEAL training, Van Wyk outlines a progression:

1. Train: Learn a new skill set.
2. Practice: Refine that skill set.
3. Drill/Exercise (Repeat): Engage in frequent, even small-scale (e.g., 30-minute lunch sessions weekly), repetitive exercises to make processes "second nature" and prevent errors. This "protocol" moves beyond mere knowledge to ingrained, disciplined execution, ensuring reliability and repeatability.

In essence, while the talk avoids discussing specific technical tools or exploits, it provides a robust framework for the design and execution of incident response, emphasizing that the "technical deep dive" for effective IR lies in mastering these operational architectures and human protocols.

Demo / Proof of Concept

▶ Watch: Key takeaway: Constant training makes response second nature (33:00)

This talk was a presentation of distilled findings and experience-driven recommendations rather than a demonstration of a specific tool or proof of concept. Kenneth Van Wyk's insights are derived from decades of facilitating tabletop exercises, which themselves serve as a simulated environment to "prove" the validity of the identified operational weaknesses and the effectiveness of proposed remediations. The "proof" lies in the recurring patterns of failure observed across numerous organizations in these simulated scenarios, and the subsequent improvements seen when the recommended operational disciplines are adopted.

Defensive Implications

▶ Watch: Q&A: Simplifying plans vs. regulatory demands (41:40)

The insights shared by Kenneth Van Wyk offer profound defensive implications, urging organizations to critically re-evaluate their incident response strategies beyond mere technical controls:

1. Revamp Incident Response Plans (IRPs): Defenders should develop a two-tiered IRP approach. A comprehensive, detailed document (potentially 100-250 pages) can satisfy design, audit, and regulatory requirements, potentially using appendices for extensive citations. However, for operational use, create concise, actionable playbooks or "break glass" checklists, ideally no more than a few pages, providing immediate, explicit guidance for responders. This ensures that the documentation serves its intended audience effectively.

1. Establish and Empower an Incident Leader: Designate a specific individual for the Incident Leader (or Commander/Manager) role. This person should be trained and empowered to orchestrate the response, drive decision-making, ensure clear communication, and rigorously track actions. Their focus is on process flow and operational discipline, not necessarily being the most senior or technically skilled individual. This role is crucial for preventing paralysis and ensuring coordinated action.

1. Pre-Define Decision Authorities and Escalation Paths: Proactively identify and document who holds decision-making authority for critical issues such as ransom payments, legal engagement, law enforcement notification, and public communications. Establish clear escalation chains for when primary personnel are unavailable. This planning prevents costly delays and ensures that appropriate stakeholders are engaged promptly.

1. Integrate Business Impact into Triage: Incorporate business owners and their perspectives into the incident triage and prioritization process from the very beginning. This requires understanding the business processes that IT systems support and assessing incidents based on their potential business impact, not just technical severity. This ensures that response efforts align with organizational priorities and protect critical assets (e.g., cold chain integrity, operational technology).

1. Implement Robust, Simple Action Tracking: Adopt a disciplined approach to tracking all assigned actions. Van Wyk advocates for simple, readily available tools like whiteboards to record who is doing what, when it was assigned, expected completion times, and follow-up schedules. Photos of whiteboards can serve as an audit trail. This low-tech, high-discipline approach ensures accountability and prevents tasks from being overlooked.

1. Standardize and Practice Clear Communications: Enforce a strict communication protocol during incidents, emphasizing explicit, concise language. Implement a read-back mechanism (like in air traffic control) where instructions are repeated by the recipient to confirm understanding before execution. This minimizes misinterpretation and ensures all team members are on the same page.

1. Prioritize Frequent, Targeted Exercises: Move beyond infrequent, large-scale annual exercises. Instead, implement a regimen of frequent, even short (e.g., weekly 30-minute lunch-break sessions), focused tabletop drills. The goal is to build muscle memory and make operational procedures "second nature," reducing the need for "second-guessing" during actual incidents. This continuous drilling fosters a reliable, repeatable, and disciplined process.

1. Cultivate an Operational Mindset: Foster a culture of operational discipline throughout the organization, extending beyond the IR team. Drawing inspiration from critical infrastructure safety briefings, embedding a mindset where process, discipline, and attention to detail are paramount can significantly enhance overall resilience. This "operational imperative" should permeate all aspects of security response.

1. Proactive Engagement with Cyber Insurance: Understand the terms and conditions of cyber insurance policies before an incident occurs. Crucially, establish a clear procedure for engaging the insurance provider immediately upon detection of a covered incident. This proactive step can prevent claims from being denied due to delayed notification, safeguarding potential financial recovery.

By addressing these non-technical, operational aspects with the same rigor applied to technical defenses, organizations can significantly enhance their overall incident response maturity and reduce the costly impact of security breaches.

Key Takeaways

• Non-Technical Failures are Paramount: The most common and costly mistakes in incident response are not technical, but rather procedural, communication, and decision-making errors.
• IR Plans Need Operational Utility: Comprehensive Incident Response Plans (IRPs) are often too verbose to be useful during a crisis; organizations need concise, actionable operational guides alongside detailed process documentation.
• Strong Incident Leadership is Essential: A designated Incident Leader (Commander/Manager) is crucial for orchestrating the response, ensuring decisions are made, and driving operational discipline, rather than solely relying on technical experts or senior management.
• Clear Decision Authority Prevents Paralysis: Pre-defining who makes critical decisions (e.g., ransom, legal, media) and establishing clear escalation paths are vital to avoid costly delays and indecision during an incident.
• Operational Discipline Through Frequent Exercises: Regular, even short (e.g., weekly 30-minute), focused exercises are indispensable for building "second nature" operational muscle memory and repeatable, reliable processes.
• Business Impact Drives Prioritization: Incident response must integrate business owners from the outset to ensure that triage and response efforts are aligned with critical business processes and their potential impact, not just technical severity.

About the Speaker(s)

Kenneth Van Wyk is a seasoned expert in incident response and security consulting, affiliated with KRvW Associates, LLC. With over two decades of experience, he has written and facilitated a vast number of tabletop exercises and participated in numerous incident response operations. Van Wyk's unique background, having been born on a British RAF Air Force base, hints at his deep appreciation for operational discipline. He is a passionate advocate for improving the human and procedural aspects of incident response, often challenging conventional approaches to IRPs and crisis management.

Elliot Atkins, Kenneth Van Wyk's co-presenter, also brings extensive experience in incident response operations and tabletop exercises from the UK. While unable to attend FIRSTCON due to a family medical emergency, his collaborative work with Van Wyk formed the foundation of this presentation. Atkins shares a common ground with Van Wyk in his appreciation for operational rigor, exemplified by his current project of restoring a British Tornado fighter jet. Together, Van Wyk and Atkins have collectively written and facilitated well over 150 exercises, providing a rich dataset for their shared observations.