Analysis without Paralysis: Mastering the Art of Investigation

Analysis without Paralysis: Mastering the Art of Investigation

Speakers: Terryn Valikodath, Senior Incident Response Consultant, Cisco Talos Incident Response

Conference: Blue Team Con Online

YouTube: https://www.youtube.com/watch?v=Pp48tnYX7JQ

Overview

In the dynamic and often overwhelming world of cybersecurity, incident investigations are a critical but frequently undertaught skill. Terryn Valikodath, a Senior Incident Response Consultant with Cisco Talos Incident Response, addresses this gap in his Blue Team Con Online talk, "Analysis without Paralysis: Mastering the Art of Investigation." Valikodath highlights the common pitfalls of ad-hoc investigations, such as analyst burnout, missed details, and an inability to set clear expectations. His presentation introduces the ADAPT methodology, a structured framework designed to guide security professionals through any investigation, from initial alert to final report, without succumbing to the complexity and pressure.

The core premise of Valikodath's talk is that while analysts are often proficient at interpreting data (analysis), they lack formal training in organizing and executing a comprehensive investigation. This distinction is crucial: analysis focuses on what the data means, while investigation orchestrates that analysis into a methodical pursuit of answers and objectives. By providing a repeatable, adaptable process, the ADAPT methodology aims to empower both junior and senior analysts to approach incidents with confidence, reduce stress, and produce actionable, defensible findings.

This talk is particularly vital for anyone involved in DFIR (Digital Forensics and Incident Response), from aspiring analysts to seasoned managers. It emphasizes the importance of a proactive, planned approach over reactive "jumping in," which often leads to inefficiencies and errors. Valikodath’s practical insights and structured methodology offer a refreshing perspective on how to navigate the chaos of an incident, transforming it into a clear, provable narrative that informs both technical remediation and strategic defensive improvements.

Background

▶ Watch: Introduction to "Analysis without Paralysis" and speaker background (4:00)

Valikodath begins by drawing a crucial distinction between Digital Forensics and Incident Response, noting that while often conjoined, they serve different primary objectives. Digital forensics typically focuses on a thorough, post-event investigation to understand precisely what happened, often with an emphasis on evidence preservation for legal or detailed root cause analysis. Incident Response, conversely, operates in a more time-critical environment where an incident might still be ongoing. The priority shifts to containment and eradication, meaning investigators may have less freedom to preserve evidence perfectly, and speed is paramount. Valikodath estimates his own work leans 60-70% towards incident response, highlighting the practical challenges of live systems and active threats.

He defines an investigation as "moving from the unknown to the known and then most importantly to the provable." Many analysts can identify suspicious activity, but the true challenge lies in substantiating those observations with clear, undeniable evidence. This concept echoes the scientific method – observation, question, hypothesis, answer, conclusion – which Valikodath suggests should serve as a guiding mindset for investigators, encouraging them to formulate questions before diving into data rather than simply "hoping to find something that looks bad."

Central to effective investigation is cultivating an investigative mindset. This involves three key components:

1. Knowing your evidence: Understanding how to interpret various data sources (e.g., Windows event logs, security console alerts) and, crucially, how to explain their significance to both technical and non-technical audiences.
2. Questioning everything: Maintaining objectivity while acknowledging personal biases (e.g., a bias towards ransomware investigations due to past experience). The goal is to ask "why is this malicious?" and be able to clearly articulate the reasons, moving beyond simple VirusTotal scores.
3. Knowing what's normal: This is perhaps the most critical skill. Understanding typical behavior across computers, networks, specific industries, and human actions allows investigators to quickly filter out noise and identify anomalous activity. For instance, knowing which processes are standard for Windows startup or typical network traffic patterns for an organization makes suspicious outliers immediately apparent.

Valikodath also briefly touches upon MITRE ATT&CK, recommending it as a vital resource for anyone entering or working in the industry. He describes it as a standardized language for discussing adversary tactics, techniques, and procedures (TTPs), which aids in consistent communication and understanding across security teams.

Key Findings

▶ Watch: The critical, untaught need for investigation skills (6:00)

The central contribution of Terryn Valikodath's talk is the introduction and detailed explanation of the ADAPT methodology for incident investigations. This framework is designed to provide a universal, structured approach that can be applied to any investigation, regardless of the specific threat or the initial level of information available. It's built to combat the "analysis paralysis" that often overwhelms security professionals facing complex or unfamiliar incidents.

The ADAPT methodology comprises five core, interconnected steps:

1. A – Approach: The planning phase, focusing on setting clear objectives, establishing documentation practices, and strategizing where and how to acquire evidence.
2. D – Discovery: The analytical phase, where investigators uncover and document "findings" – any activity that appears abnormal or suspicious.
3. A – Association: The synthesis phase, where individual findings are ordered chronologically, correlated across different evidence sources, and normalized using consistent terminology and timestamps.
4. P – Profile: The intelligence-gathering phase, comparing observed behaviors and indicators with known threat intelligence or past incidents to identify patterns and potential adversaries.
5. T – Timeline: The final output phase, creating a comprehensive, easy-to-digest chronological record of all events, including both malicious activities and defensive actions.

A significant finding is the emphasis on delaying direct evidence acquisition until a clear plan and objectives are established. Valikodath stresses that "jumping in" without a roadmap leads to burnout and mistakes. The ADAPT framework encourages iterative progress, acknowledging that investigators might loop back to earlier stages (e.g., from Association back to Discovery) as new information emerges. This flexible yet structured approach ensures thoroughness while maintaining efficiency, ultimately leading to more accurate findings and more effective incident response.

Technical Deep Dive

▶ Watch: Clarifying the difference between analysis and investigation (7:30)

The ADAPT methodology is the technical core of Valikodath's presentation, providing a detailed, step-by-step guide for conducting robust investigations.

A - Approach: The Plan

The Approach phase is entirely dedicated to planning before touching any evidence. It starts with setting a clear objective, which, while seemingly simple, is often overlooked. Objectives can range from "is this activity malicious?" (90% of investigations) to "contain the compromise" or "determine root cause." Valikodath stresses that objectives will evolve, but a starting point is crucial to guide the team.

Documentation is paramount. Valikodath recommends using templates (available on his GitHub) to provide a structured starting point for note-taking, preventing the blank-page dilemma. Notes should be taken in a consistent method (e.g., Word, SharePoint, Google Drive) with appropriate access controls. For junior team members or those less technically inclined, serving as a scribe is an excellent way to contribute and learn.

Knowing where to look for evidence comes with experience and is categorized into five key types:

1. Logs: Windows event logs, Linux logs, application-specific logs are fundamental.
2. Security Tools: Consoles from EDR systems, SIEMs, and other security products (e.g., Microsoft consoles).
3. Intelligence: Not just threat intelligence, but also internal intelligence like ServiceNow tickets or undocumented application configurations that might explain seemingly anomalous network connections.
4. Forensic Analysis: Traditional "deadbox" forensics, involving bit-for-bit copies of devices for deep, offline analysis.
5. Triage Tools: Tools run on live systems to quickly extract specific forensic artifacts, offering speed but impacting the live system. Valikodath notes these are not mutually exclusive; both can be used.

The approach also involves prioritizing evidence based on the investigative question and acquisition cost (e.g., pulling Windows event logs for login activity is faster than a full forensic image). The principle of "measure twice, cut once" is applied to evidence acquisition, emphasizing the need to practice and plan acquisition methods beforehand to ensure accuracy and completeness the first time.

D - Discovery: Uncovering Findings

The Discovery phase is where the actual analysis begins, uncovering findings – any activity that deviates from the norm or appears suspicious. Key practices include:

• Organizing by evidence source: This keeps notes structured and allows for easier cross-referencing later.
• Writing in a report format: Use full sentences for findings (e.g., "This computer reached out to this network address at this time") to naturally prompt the inclusion of critical context (what initiated it, what user was logged on).
• Considering the audience: While notes are often for peers, thinking about how to simplify explanations (e.g., for Mimikatz) can improve clarity.
• Using UTC timestamps: This is non-negotiable for consistency, preventing time zone headaches. Convert all timestamps to UTC at the point of discovery.
• Threat hunting: This is a natural extension of discovery, allowing investigators to proactively search for more malicious activity based on initial findings.
• Baselines: A deep understanding of what's normal for user behavior, network traffic patterns, and operating system/application activity is crucial for filtering out noise and quickly identifying suspicious events. Valikodath suggests the PEAK threat hunting framework for those new to the concept.

Valikodath provides a structured breakdown for documenting each finding:

1. Timestamp: The exact time of the event, ideally in UTC.
2. Source: Where the evidence was obtained (e.g., specific log file, EDR console query).
3. Event Description: An objective, straightforward sentence describing what occurred.
4. Context: Why the finding is considered suspicious or malicious (e.g., a workstation logging into a domain controller, a recently created account). This is where the investigator's "bias" or expertise adds value.
5. Code Snippet: Visual evidence such as a file hash, a screenshot of logs, or a command output, providing immediate context for anyone reviewing the notes.

It's important to document even if something isn't definitively malicious, treating it as suspicious until proven benign. Documenting gaps (e.g., "no external network connections observed") is also valuable, as it helps rule out possibilities and shows due diligence.

A - Association: Putting Pieces Together

In the Association phase, the focus shifts from individual findings to connecting the dots.

• Ordering Findings: All documented findings are arranged chronologically using their UTC timestamps.
• Connecting Events: Investigators look for chains of events across different evidence sources to build a coherent narrative. This helps identify what happened between key events (e.g., between a login and file exfiltration) and prompts more targeted searches.
• Capturing Indicators: Key artifacts like IP addresses, file hashes, and account names are extracted for later use.

Normalization is a critical sub-step:

• Standard Timestamps: Valikodath strongly recommends the ISO8601 format for consistent, unambiguous time representation.
• Consistent Terminology: Using a standardized vocabulary (e.g., "source IP" instead of "connected IP" or "source network address") across all notes prevents confusion and missed information. MITRE ATT&CK can aid in this standardization.
• Audience-centric explanation: Reviewing notes as if explaining to a non-expert helps identify areas needing more detail (e.g., explaining Kerberoasting).

P - Profile: Threat Intelligence and Research

The Profile phase aims to answer: "Have we or someone else seen this behavior before?" This involves comparing observed TTPs, IP addresses, and other indicators with known threat intelligence or previous incidents. It's not about exhaustive threat intelligence analysis but rather a due diligence step to:

• Identify known attack patterns.
• Uncover potentially missed details.
• Leverage external intelligence teams if available.

Indicators are categorized into four types:

1. Network: DNS entries, IP addresses, domain names.
2. File: File hashes, names, paths.
3. Identity: Account names, session cookies, tokens, private keys (Valikodath prefers "identity" over "account names" to encompass a broader range of credentials).
4. Behavioral: Patterns like brute-force attempts or specific lateral movement tools.

This phase helps contextualize the incident and provides valuable input for future defensive strategies.

T - Timeline: The Chronological Database

The final step, Timeline, consolidates all findings into an easily digestible chronological record, typically an Excel or CSV spreadsheet. This serves as the "database of incident information." Essential headers for the timeline include:

• Timestamp: UTC, preferably ISO8601.
• Origin: Where the event initiated (e.g., a specific workstation).
• Target: The system or resource affected (e.g., a domain controller).
• User/Identity: The account or identity involved.
• Evidence Source: Where the information was found.
• Event Type: A concise, 2-3 word description (e.g., "Brute force attempt," "Lateral movement").
• Description: A brief blurb from the detailed notes.
• MITRE ATT&CK: The associated technique or tactic.

The timeline is not just for malicious actions; it should also include defensive actions (e.g., system isolation, password resets, notifications). This provides a complete picture, demonstrating response effectiveness and helping to identify if actions had the desired impact (e.g., "no suspicious activity after password reset"). The timeline simplifies complex incident data, making it accessible to both technical and non-technical stakeholders and serving as a powerful tool for reporting and justifying security investments.

The entire ADAPT methodology is presented as a cyclical, rather than linear, process, allowing investigators to revisit earlier stages as new information comes to light.

Demo / Proof of Concept

▶ Watch: Understanding the core differences: DFIR vs. Incident Response (8:00)

Terryn Valikodath's presentation focuses on a methodological framework rather than demonstrating a specific tool or exploit. While there isn't a live "demo" in the traditional sense, the practical application of the ADAPT methodology is evident through the detailed breakdown of each step and the examples provided for structuring findings and timelines. Valikodath does offer tangible resources in the form of templates for documentation and note-taking, which are freely available on his GitHub, serving as a direct proof of concept for the structured approach he advocates.

Defensive Implications

▶ Watch: Defining investigation: From unknown to known and provable (10:00)

The ADAPT methodology offers profound defensive implications, transforming reactive incident chaos into a structured, proactive defense strategy.

1. Reduced Paralysis and Burnout: By providing a clear, repeatable framework, ADAPT significantly reduces the "analysis paralysis" that often leads to analyst burnout. Investigators know exactly what steps to take next, even in unfamiliar situations, fostering confidence and efficiency.
2. Clearer Objectives and Expectations: The emphasis on setting initial objectives, even if they evolve, allows security teams to align their efforts with organizational goals (e.g., containment, root cause). This clarity also enables better communication with management, setting realistic expectations for investigation timelines and outcomes.
3. Enhanced Reporting and Communication: The structured documentation, consistent terminology, and chronological timeline generated by ADAPT directly translate into robust, professional reports. These reports are easier to understand for both technical peers and non-technical stakeholders, fostering better decision-making and accountability.
4. Actionable Intelligence for Future Defense: The "Profile" phase, which involves comparing findings to known threat intelligence and past incidents, directly contributes to building an organizational threat intelligence database. This helps identify recurring adversaries, TTPs, and vulnerabilities, enabling more targeted and effective proactive defenses.
5. Justification for Security Investments: The comprehensive timelines, which include both malicious and defensive actions, provide irrefutable evidence of an incident's impact and the effectiveness of response measures. This data is invaluable for justifying budget requests for critical security tools like SIEMs, EDRs, and patching initiatives, by demonstrating the tangible benefits of these investments in mitigating real-world threats.
6. Improved Threat Hunting Capabilities: The methodology inherently encourages threat hunting, as investigators constantly seek to uncover more malicious activity based on initial findings and baselines. This proactive posture enhances an organization's ability to detect and respond to threats before they escalate.
7. Standardized Language and Processes: Adopting consistent timestamps (UTC, ISO8601) and terminology (potentially leveraging MITRE ATT&CK) across an organization streamlines investigations, reduces miscommunication, and facilitates collaboration among different teams or even external consultants.
8. Training and Skill Development: ADAPT provides a practical roadmap for training junior analysts, offering a structured way to learn investigative skills. The role of a "scribe" is explicitly highlighted as an entry point for learning, fostering growth within the security team.
9. Proving Malicious Intent: The focus on moving from "known to provable" ensures that findings are thoroughly substantiated, reducing false positives and providing a solid foundation for legal or disciplinary actions if necessary.

Key Takeaways

• Investigations are a distinct, learned skill: While analysis is taught, the systematic process of investigation often isn't, leading to burnout and missed details. The ADAPT methodology fills this critical gap.
• Proactive planning prevents paralysis: The "Approach" phase emphasizes setting clear objectives and planning evidence acquisition before diving into data, leading to more efficient and effective investigations.
• "Knowing what's normal" is paramount: Understanding typical system, network, and user behavior is the most effective way to quickly identify and filter out malicious or anomalous activity.
• Documentation is your memory and your report: Structured note-taking, consistent timestamps (UTC, ISO8601), and a "report format" for findings turn raw notes into a comprehensive, defensible final report with minimal extra effort.
• The timeline is an invaluable database: A well-constructed timeline, including both malicious and defensive actions, serves as a clear, chronological narrative that aids technical understanding, stakeholder communication, and justification of security investments.
• Continuously question and correlate: The ADAPT framework encourages constant questioning, cross-referencing findings across diverse evidence sources, and leveraging threat intelligence to build a complete and accurate picture of an incident.

About the Speaker(s)

Terryn Valikodath is a Senior Incident Response Consultant with Cisco Talos Incident Response. His role involves both proactive and reactive cybersecurity efforts, including handling true incidents for customers, developing incident response plans, conducting tabletop exercises, and performing threat hunting. Valikodath's unique journey into cybersecurity began with a background in forensic science and criminology, which he complemented by leveraging his early experiences jailbreaking iPhones and rooting devices. He transitioned through various IT roles, including IT support and system administration, before specializing in digital forensics and incident response. He is a prolific writer, maintaining a blog where he shares insights on incident response and "soft skills" relevant to the field. He can be found on Twitter as @Cybercoat and on other platforms as @ChocolateCoat or @ChocolateCoatForensics, where he also shares his investigative templates on GitHub.