Automation for digital forensics: Towards a classification model for the community

Automation for digital forensics: Towards a classification model for the community

Speakers: Gaëtan Michelet, Doctoral researcher, University of Lausanne

Conference: DFDS

YouTube: https://www.youtube.com/watch?v=4DNrEiKkugc

Overview

In the rapidly evolving field of digital forensics (DF), automation has become increasingly crucial for managing the growing volume and complexity of evidence. However, the term "automation" itself can encompass a vast spectrum of tools and methodologies, ranging from simple, deterministic parsers to sophisticated, AI-driven analytical systems. This talk, presented by Gaëtan Michelet at the DFDS conference, addresses this ambiguity by proposing a novel classification model specifically tailored for automated digital forensics tools and methods. Michelet, a PhD student with a strong interest in leveraging artificial intelligence (AI) for DF automation, particularly in the under-researched reporting phase, highlights the critical need for a framework that differentiates tools based on their capabilities, impact on investigation, and inherent risks.

The core problem identified by Michelet is that current definitions of automation are often too broad, failing to distinguish between highly disparate tools. For instance, a basic file system parser and an AI system designed to identify relevant evidence might both be classified as "automated," despite fundamental differences in their operational complexity, decision-making processes, and potential for error. This lack of clear differentiation poses challenges for both practitioners seeking to select appropriate tools and academics aiming to systematically evaluate and advance the field. The proposed classification model aims to provide a quick and insightful overview of these crucial elements, fostering more informed tool selection, risk assessment, and research direction within the DF community.

Michelet's work is part of his broader PhD research, which focuses on pushing forward the automation of the reporting phase in digital forensics—a phase he notes has received significantly less attention than acquisition, examination, and analysis. His previous contributions include proposing a definition for automation tailored to DF and exploring the use of Large Language Models (LLMs) for generating sections of forensic reports. This talk on classification builds upon that foundational work, providing a framework to understand and categorize the diverse landscape of automated solutions being developed and utilized in digital forensics today.

Background

▶ Watch: Speaker intro; problem: diverse automation needs classification (0:00)

The journey of a digital forensics investigation typically involves four main phases: acquisition, examination, analysis, and reporting. While significant strides have been made in automating the initial three phases—from automated data collection to sophisticated analytical tools—the reporting phase has largely remained a manual, human-intensive process. This disparity motivated Gaëtan Michelet's PhD research, which seeks to introduce greater automation into the generation of forensic reports, potentially leveraging AI and LLMs. His ongoing projects include fine-tuning LLMs to improve report generation accuracy and quality, indicating a future where AI plays a more integrated role in the entire forensic workflow.

Before delving into classification, Michelet first revisited a definition of automation previously proposed by his team, specifically tailored for digital forensics. This definition posits that an automated tool or method can be either software or hardware based, and its primary goals are to enhance efficiency, reliability, or transparency. Crucially, it must also aim to reduce or remove human engagement in a task. The key insight here is that even a slight reduction in human involvement qualifies a method as automated; the goal is not necessarily to fully remove the human from the loop.

However, this broad definition, while foundational, quickly reveals its limitations when applied to real-world scenarios. Michelet illustrated this with two contrasting examples: a basic file system parser and an hypothetical AI-driven evidence identifier. The file system parser simply follows documentation, presents files and metadata, involves minimal decision-making, carries low risk, but also has limited capabilities (e.g., struggles with new or corrupt file systems). In contrast, an AI tool using machine learning to identify relevant evidence makes complex decisions, entails higher risks (e.g., misidentification, tagging important data as irrelevant), but offers significantly greater capabilities and potential impact on an investigation. Both tools fit the proposed definition of automation, yet their operational characteristics, inherent risks, and investigative utility are vastly different.

This stark contrast underscored the need for a more granular classification system. Without it, practitioners lack crucial insight into the capabilities (and underlying impact) and the amount of risk associated with using a particular automated tool. Such insights are vital for making informed decisions in an investigation. While existing automation classification models have been developed since the 1990s, particularly in industrial and aerospace contexts, they often do not directly apply to the unique challenges of digital forensics. Consequently, researchers frequently adapt existing models to their specific domains, a strategy Michelet and his team also adopted, exploring various categories and model dimensionalities to arrive at a suitable framework for DF.

Key Findings

▶ Watch: The problem: different automation types need classification (3:15)

In developing a more nuanced classification model for digital forensics automation, Gaëtan Michelet and his team explored several potential categories, ultimately identifying four as particularly relevant: levels of automation, complexity, impact on the investigation, and decision-making process. Each category offered unique insights but also presented specific challenges for practical application.

The levels of automation category provides information about a tool's autonomy, indicating how much it can achieve without human intervention. This was deemed highly useful for understanding a tool's self-sufficiency. Complexity was considered under the assumption that more complex tools generally offer greater capabilities but also introduce higher risks. However, quantifying complexity objectively proved difficult, making it challenging to use as a primary classification axis. Similarly, the impact on the investigation is undeniably valuable for practitioners, as it directly addresses a tool's utility. Yet, like complexity, its quantification is difficult, and a tool's impact can be highly context-dependent, being very useful in one case but irrelevant in another. The decision-making process category, which focuses on the types of decisions a tool is capable of making, emerged as a highly promising avenue, directly addressing the core difference between deterministic parsers and intelligent systems.

Beyond individual categories, the team also considered the dimensionality of the model. A single-dimensional model is simple but lacks granularity, while adding more categories increases granularity but risks making the model overly complex and difficult to use. A model with too many dimensions (e.g., 25 possible classes from two categories with five classes each) could lose its usability, rendering it impractical for practitioners and undermining its purpose. The goal was to strike a balance between providing sufficient detail and maintaining usability.

Based on these considerations, the key finding and primary contribution of Michelet's work is the proposal of a two-dimensional classification model. This model leverages the Level of Automation and the Decision-Making Level as its two axes. This combination was chosen because it effectively captures the crucial distinctions between different automated tools—how much human effort they reduce and what kind of intelligence or process they employ—while remaining manageable and intuitive for users. This two-dimensional approach aims to provide the necessary granularity to differentiate tools like simple file system parsers from sophisticated AI-driven analysis engines, without overwhelming the user with excessive complexity.

Technical Deep Dive

▶ Watch: Explanation of 'Level of Automation' categories (7:55)

The proposed classification model is a two-dimensional matrix designed to categorize automated digital forensics tools and methods based on their Level of Automation (LoA) and Decision-Making Level (DML). This structure allows for a comprehensive yet intuitive understanding of a tool's operational characteristics and potential impact.

The Level of Automation dimension is broken down into five distinct classes:

1. Manual: At this level, there is no automation whatsoever; the practitioner performs all operations by hand. This serves as the baseline against which all automation is measured.
2. Assistance: Here, a few operations are automated, but the practitioner must actively launch the tool or initiate the automated process. An example might be a script that filters logs, but requires manual execution.
3. Autonomous Assistance: This level also involves a few automated operations, but critically, the tool is launched automatically once a specific condition is met. Michelet provided an example: after an image parsing module finishes, a subsequent module automatically extracts WhatsApp databases and displays the first 10 messages. The automation is event-driven.
4. Partial Task Automation: At this level, the tool achieves only a part of a complete forensic task. For instance, if the overarching task is "identifying all relevant textual documents," a tool that clusters documents by topic (e.g., using a clustering algorithm) would fall here. While it automates a significant step, a human still needs to review the clusters to determine relevance, meaning the complete task is not yet finished.
5. Full Task Automation: This represents the highest level of automation, where the tool achieves a complete forensic task. An example given is the classification of an executable file as malicious or not, which can be considered a self-contained forensic task.

Michelet acknowledged that distinguishing between advanced Autonomous Assistance and Partial Task Automation can sometimes be challenging. He recommended that if there is hesitation, the classification should lean towards Partial Task Automation, emphasizing that the key differentiator is the magnitude of workload undertaken by the tool.

The Decision-Making Level dimension, orthogonal to the LoA, also comprises five classes, describing the nature of decisions the tool can make:

1. No Decision: The tool simply follows a predefined, objective process without making any interpretive choices. A basic file system parser that strictly adheres to documentation is a prime example. It presents data as it is found, without evaluating its significance.
2. Objective Decision: Decisions at this level involve simple comparisons or logical operations. Filtering data by a specific date, for example, is an objective decision. The criteria are clear, and the outcome is deterministic.
3. Subjective Decision: This level involves decisions based on a set of rules that were manually created by practitioners, often combined with input data provided by the practitioners. An example is a tool that generates a list of keywords based on the investigated crime (input from practitioner) and then searches for those keywords using predefined rules. The "subjectivity" comes from the human-defined rules.
4. Learning Decision: Here, the tool leverages machine learning (ML) algorithms. Practitioners provide a dataset, and the ML algorithm automatically creates a set of rules for decision-making. These rules are often opaque to the human user. The clustering of textual documents based on a clustering algorithm falls into this category, as the algorithm learns how to group similar documents. Classifying an executable as malicious or not, based on a classification model trained on data, is another example.
5. Full Decision: This is similar to a Learning Decision, but with a critical difference: the human fully trusts the tool's output and does not control or intervene in the decision-making process or its interpretation. This implies a high degree of confidence in the tool's autonomous judgment, typically reserved for highly mature and validated AI systems.

Michelet illustrated the model with several examples:

• Manually selecting files created at a certain date would be Manual LoA and Objective Decision DML, as it involves simple comparisons performed by hand.
• A tool searching for keywords generated based on the crime investigated (with practitioner input) would be Assistance LoA (as it's a few operations, launched by the practitioner) and Subjective Decision DML (rules generated from practitioner input).
• Clustering textual documents by topic using an algorithm is Partial Task Automation LoA (as the practitioner still needs to review clusters for relevance) and Learning Decision DML (due to the ML algorithm).
• Classifying an executable as malicious or not using a machine learning model is Full Task Automation LoA (as it completes a forensic task) and Learning Decision DML.

The speaker emphasized that this is a "working model" that needs refinement, encouraging feedback from both academics and practitioners to improve its utility and accuracy. The model's strength lies in its ability to provide a structured way to evaluate the diverse landscape of automated tools, helping users understand not just what a tool does, but how it does it and the associated levels of autonomy and risk.

Demo / Proof of Concept

▶ Watch: Explanation of 'Decision-Making Level' categories (9:20)

This talk primarily focused on presenting a theoretical classification model rather than demonstrating a specific tool or proof of concept. The speaker illustrated the model's application through conceptual examples, such as a basic file system parser versus an AI-driven evidence identifier, and provided hypothetical scenarios for classifying different types of automated forensic tasks. The purpose was to introduce the framework and solicit feedback for its refinement, not to showcase a functional implementation. Therefore, no live demo or technical proof of concept was presented during the session.

Defensive Implications

▶ Watch: Future work, model refinement, and call for community feedback (12:20)

The classification model proposed by Gaëtan Michelet offers significant defensive implications for digital forensics practitioners, incident responders, and security analysts. By providing a structured framework to understand automated tools, defenders can make more informed decisions, manage risks more effectively, and optimize their investigative workflows.

Firstly, the model enables informed tool selection and deployment. When evaluating new automation tools, defenders can use the LoA and DML dimensions to quickly grasp a tool's capabilities and limitations. For instance, understanding if a tool performs "Partial Task Automation" versus "Full Task Automation" helps set realistic expectations about the level of human oversight still required. A tool with a "No Decision" DML, like a simple parser, is highly reliable for its specific function, while a "Learning Decision" tool, such as an AI-based malware classifier, might offer powerful insights but requires careful validation and understanding of its potential for false positives or negatives. This clarity helps defenders choose tools that align with their specific investigative needs and risk tolerance.

Secondly, the model facilitates robust risk assessment and trust calibration. Tools operating at higher Decision-Making Levels, especially "Learning Decision" or "Full Decision," inherently carry greater risks due to their complex, often opaque, decision-making processes. Defenders need to understand that relying on an AI tool to classify evidence (Learning Decision) is fundamentally different from using a deterministic filter (Objective Decision). The model prompts defenders to ask critical questions: How much trust can be placed in the tool's output? What are the potential consequences of an incorrect automated decision? This understanding allows for proportionate human review and validation, preventing over-reliance on automated outputs where accuracy is paramount. For example, an AI classifying an executable as malicious might require a human analyst to confirm before taking disruptive action.

Thirdly, the framework can guide resource allocation and human expertise focus. By identifying which parts of an investigation are truly "Full Task Automation" and which are "Partial Task Automation" or "Assistance," security teams can strategically allocate their human analysts. Tasks classified with lower LoA or DML require more direct human engagement. Conversely, where tools achieve "Full Task Automation," human effort can be redirected to more complex, less automatable challenges, or to validating the output of higher-level automated systems. This ensures that skilled analysts are focusing their expertise where it adds the most value, rather than on repetitive or deterministic tasks.

Fourthly, the model aids in standardizing communication and knowledge sharing within the DF community. By providing a common vocabulary, practitioners can discuss automated tools with greater precision. This shared understanding can accelerate tool development, foster better collaboration between researchers and practitioners, and improve training programs by clearly delineating the types of automation being taught and their associated operational considerations.

Finally, for defenders involved in developing custom automation or integrating commercial tools, the model offers a blueprint for designing and evaluating robust solutions. It encourages developers to clearly articulate where their tools fall on the LoA and DML scales, promoting transparency about a tool's capabilities, decision logic, and inherent risks. This proactive approach can lead to the development of more trustworthy and effective forensic automation tools that meet the specific needs of the defensive community. Ultimately, by providing clarity on the nature of automation, Michelet’s model empowers defenders to leverage technology more intelligently and securely in their fight against cyber threats.

Key Takeaways

• The Problem of Broad Definitions: Existing definitions of automation in digital forensics are too broad, failing to distinguish between simple, deterministic tools (e.g., file system parsers) and complex, AI-driven systems (e.g., AI-based evidence identifiers), despite vast differences in capabilities, impact, and risk.
• Need for a Classification Model: There is a critical need for a classification model that provides quick insights into the capabilities, investigative impact, and inherent risks of automated DF tools for both practitioners and academics.
• Proposed Two-Dimensional Model: The talk introduces a novel two-dimensional classification model for DF automation, utilizing Level of Automation (LoA) and Decision-Making Level (DML) as its axes, balancing granularity with usability.
• Levels of Automation (LoA): This dimension categorizes tools from Manual to Assistance, Autonomous Assistance, Partial Task Automation, and Full Task Automation, indicating the extent of human engagement reduction.
• Decision-Making Levels (DML): This dimension classifies decision processes from No Decision to Objective Decision, Subjective Decision, Learning Decision, and Full Decision, reflecting the complexity and nature of a tool's intelligence.
• Practical Application and Future Refinement: The model provides a practical framework for understanding and classifying diverse automated tools, with examples ranging from manual file selection to AI-driven malware classification. It is presented as a "working model" seeking community feedback for further refinement and future application in systematizing knowledge about DF automation.

About the Speaker(s)

Gaëtan Michelet is a doctoral researcher who began his PhD studies in September 2021 with the University of Lausanne and is currently affiliated with the University of Augsburg. He possesses a strong interest in the application of automation and artificial intelligence to digital forensics, particularly focusing on advancing the automation of the reporting phase, which he identifies as an area largely underserved by current automation efforts. His research includes proposing tailored definitions for digital forensics automation, exploring the use of Large Language Models (LLMs) for generating forensic report sections, and fine-tuning LLMs for improved performance. Michelet is actively engaged in contributing to the digital forensics community through his research and seeks to foster discussion and collaboration on topics related to automation.