Towards Interpretable Topic Modelling as a Tool for Hypothesis-Driven Forensic Communication Analysis

Towards Interpretable Topic Modelling as a Tool for Hypothesis-Driven Forensic Communication Analysis

Speakers: Jenny Felser, Doctoral researcher, University of Applied Sciences Mittweida

Conference: DFDS

YouTube: https://www.youtube.com/watch?v=Fvqcqdaxrw0

Overview

In the rapidly evolving landscape of digital forensics, investigators face an ever-increasing deluge of data, particularly from mobile communication platforms. Jenny Felser, a doctoral researcher at the University of Applied Sciences Mittweida, presented her PhD topic focusing on this critical challenge: developing interpretable hypothesis-driven topic modeling for forensic communication analysis. Her work addresses the overwhelming volume of instant messages and other digital communications that often contain only a small fraction of relevant evidence, making manual review impractical and inefficient.

The core problem her research tackles is the sheer scale of data. A single mobile phone can contain tens of thousands of messages, translating into thousands of pages of text. As Felser vividly illustrated, sifting through 50,000 messages is akin to reading all the Harry Potter books not once, but four times over – an impossible task when only a mere 40 pages might hold pertinent information. This talk outlines a novel approach to leverage machine learning, specifically topic modeling, to dynamically identify evidence relevant to a specific investigative hypothesis and present it in an interpretable manner, significantly streamlining the forensic process.

Felser’s research is crucial for digital forensic practitioners who need to quickly and accurately identify relevant communication patterns amidst noise. By integrating an investigator's specific knowledge and experience into the analytical pipeline, her proposed method aims to overcome the limitations of traditional keyword searches and generic classifiers, which often fail to capture the nuanced, case-specific context of criminal communications. The goal is to transform a daunting data analysis task into a focused, hypothesis-driven evidence discovery process, thereby enhancing the efficiency and effectiveness of digital forensic investigations.

Background

▶ Watch: The massive scale of mobile communication data (2:00)

The explosion of mobile communication has created both a boon and a bane for forensic investigators. While mobile devices often contain crucial evidence, the sheer volume of data—from instant messages on platforms like WhatsApp and Signal to SMS—presents an unprecedented challenge. Investigators are frequently confronted with hundreds of thousands of messages, the vast majority of which are irrelevant to the case at hand. Manually reviewing this volume is not only time-consuming but often unfeasible, leading to potential delays in justice or even missed evidence.

Traditional forensic approaches to communication analysis have significant limitations. A simple keyword search, while straightforward, often proves inadequate. Broad terms can return an overwhelming number of irrelevant messages, while overly specific terms might miss critical evidence due to variations in language, slang, or codified expressions. For instance, an investigator looking for evidence of online grooming might search for explicit terms, but perpetrators often use euphemisms like "play games" or discuss "meeting up" in seemingly innocuous contexts. Conversely, general terms like "money" or "meeting" could yield millions of results, making manual review impossible.

Pre-trained classifiers, which use machine learning to categorize content (e.g., identifying grooming content), also fall short in many forensic scenarios. While effective for well-defined, static categories, criminal communication is highly individual, heterogeneous, and constantly evolving. Each case presents unique circumstances, specific jargon, and preparatory actions that a generic classifier simply cannot anticipate or adapt to. For example, planning a complex offense might involve discussions about "floor plans" or "building materials" that, without context, appear entirely innocent.

The fundamental problem with existing topic modeling techniques, particularly unsupervised algorithms, is their lack of investigator knowledge. These models tend to extract dominant topics within a dataset, which are often benign "small talk" or everyday conversations, rather than the subtle, case-relevant patterns. Furthermore, the output of these models – often a probability distribution of single words – can be difficult for human investigators to interpret without additional context. This gap highlights the critical need for a method that can dynamically incorporate an investigator's specific hypotheses, background knowledge, and experience into the analytical process, guiding the topic model towards truly relevant information.

Key Findings

▶ Watch: Issues with unsupervised topic modeling (7:30)

Jenny Felser's preliminary experiments and proposed pipeline highlight several key findings regarding the integration of investigator knowledge into topic modeling for forensic analysis. The central tenet of her research is that hypothesis-driven topic modeling is essential to overcome the limitations of unsupervised approaches and address the massive data volume in digital forensics.

The research focuses on two primary goals: dynamically identifying evidence for a specific hypothesis and summarizing relevant communication in an interpretable way. This can only be achieved by incorporating the investigator's unique knowledge and experience. Felser's team conducted a series of preliminary experiments using a dataset from an international pre-data identification competition, specifically focusing on a scenario involving online grooming. This dataset included six conversations on the grooming topic and 100 "little conversations" simulating unrelated small talk.

The experiments explored different methods for providing prior knowledge to the topic model:

1. Characteristic Words (Seed Words): The simplest approach involved providing the topic model with a list of words characteristic of the expected topic. For the grooming scenario, initial seed words might include terms related to "secret message," "relationship," or "sexual images." While straightforward, this method proved challenging. Investigators often find it difficult to determine truly effective seed words, especially for nuanced or codified language. For instance, "play games" might be a euphemism for sexual activity, but it also has an innocent meaning, leading to ambiguity. Similarly, terms like "location" could be relevant but lack context.

1. Extractive Topics from Similar Messages: This approach involved identifying existing messages within the dataset that were known to be relevant to the hypothesis (e.g., actual grooming messages) and using them to inform the topic model. The idea was to "extract topics" from these known relevant messages. However, this method faced challenges similar to seed words; the initial representation of the topic from these messages might still be too broad or sensitive to the specific examples chosen, potentially capturing other dominant themes (e.g., general "female topic" instead of grooming).

1. Generating Artificial Messages with Large Language Models (LLMs): Leveraging the power of LLMs, the researchers attempted to generate artificial messages based on characteristic words or prompts describing the expected topic. The expectation was that these synthetic messages would provide a richer, more diverse representation of the hypothesis. While LLMs could generate messages describing "sex over" or "legal between adults," they often struggled to precisely capture the specific, sensitive context of online grooming due to the initial topic representation not being adequately refined. The output tended to be too generic or focused on adult consensual interactions rather than the predatory nature of grooming.

1. Topic Summaries: This method proved the most promising. Instead of individual words or messages, the researchers provided the topic model with a topic summary – a high-level description of what conversations about the hypothesis typically entail. For the grooming scenario, a summary might include characteristics like "exchanging contact information" or "making up meetings in the real world." This approach, leveraging algorithms that can summarize information about conversations, produced more meaningful and interpretable topic mixes. It was better able to capture the nuanced characteristics indicative of online grooming, offering a more robust hint for investigators.

These findings underscore the difficulty of effectively translating an investigator's abstract hypothesis into a format that a machine learning model can optimally utilize. While simple approaches like seed words are intuitive, they often lack the contextual depth required. The use of LLMs for generating artificial data showed potential but highlighted the need for careful prompt engineering and topic representation. Ultimately, the ability to provide a more conceptual "topic summary" appears to be a significant step forward in guiding topic models toward highly specific and relevant forensic evidence.

Technical Deep Dive

▶ Watch: Challenges with ambiguous and codified language (11:40)

The proposed technical pipeline for interpretable hypothesis-driven topic modeling aims to dynamically identify evidence by integrating an investigator's prior knowledge directly into the topic modeling process. This departs from traditional unsupervised topic models which, by design, are uninformed about specific investigative hypotheses and thus tend to extract the most dominant, often irrelevant, topics (e.g., small talk).

At the core of Felser's approach is the concept of seed-guided topic modeling. Instead of allowing the algorithm to discover topics purely from statistical co-occurrence of words, the investigator's hypothesis is used to "seed" or bias the model towards topics of interest. This guidance helps the model prioritize and extract specific, expected topics that might otherwise be overlooked due to their lower frequency compared to general conversation.

The talk explored several mechanisms for injecting this prior knowledge:

1. Characteristic Words: This involves providing a list of keywords or phrases that are highly indicative of the target hypothesis. For example, in a case involving a terrorist organization, seed words might include specific group names, locations, or code words. The challenge here lies in the ambiguity and dynamism of language. As Felser noted, "play games" can mean something entirely different in a grooming context than in an innocent one. Moreover, criminal communication often employs codified language or euphemisms (e.g., "floor plan" for planning an attack, "Windows XP" in a context unrelated to operating systems) that are difficult to capture with a simple list of keywords. The effectiveness of this method is highly dependent on the investigator's ability to accurately predict and supply these terms, which is often difficult without prior context or expertise in the specific criminal subculture.

1. Extractive Topics from Known Relevant Messages: This method attempts to learn the characteristics of the target topic directly from a small set of already identified, relevant messages. The idea is to extract latent topics from these "golden samples" and use them to guide the analysis of the larger dataset. While seemingly robust, this approach can be sensitive to the quality and representativeness of the initial sample. If the initial messages are too narrow or contain confounding information, the extracted topic might not accurately reflect the broader hypothesis.

1. LLM-Generated Artificial Messages: Recognizing the limitations of manually curated keywords and small message samples, the research explored using Large Language Models (LLMs) to generate synthetic messages that embody the characteristics of the target hypothesis. The process involves prompting an LLM with descriptive phrases or characteristic words related to the hypothesis (e.g., "generate artificial messages about online grooming involving minors"). The goal is to create a diverse set of synthetic data that can enrich the topic model's understanding of the target concept. However, this also presented challenges. LLMs, without careful calibration, might generate messages that are too generic, focusing on broader themes (e.g., "sex between adults") rather than the specific, illegal context of the hypothesis (e.g., "online grooming of minors"). This highlights the need for precise prompt engineering and a feedback loop to refine the LLM's output to ensure it accurately reflects the nuanced aspects of the forensic hypothesis.

1. Topic Summaries: This was identified as the most promising approach. Instead of raw words or messages, the input to the topic model is a more abstract, high-level summary of what a conversation related to the hypothesis typically involves. For instance, a summary for online grooming might describe "exchanging contact information, making up meetings in the real world, and discussing sensitive topics with minors." This approach leverages the ability of algorithms to distill key information from existing conversations or expert knowledge into a concise description. By providing a more conceptual representation, the topic model can better grasp the multi-faceted nature of the hypothesis, leading to more meaningful and interpretable topic extraction. This method moves beyond simple keyword matching to capture the behavioral patterns and communicative intent associated with a specific crime.

A critical challenge highlighted in the talk is the difficulty in finding suitable language services for actions. This refers to the problem of mapping abstract criminal actions or intentions (e.g., "planning an attack") to specific linguistic expressions, especially when those expressions are highly contextual or disguised.

Looking ahead, Felser's pipeline aims to incorporate two key extensions:

• A quantitative evidence score to rank topics based on their relevance to the hypothesis, providing investigators with a clear measure of importance.
• An interactive iteration process, allowing investigators to provide real-time feedback to the system, refining seed words, topic summaries, or LLM prompts to continuously improve the accuracy and relevance of the extracted topics. This human-in-the-loop approach is crucial for adapting the model to the unique characteristics of each case and overcoming the inherent limitations of purely automated methods.

Demo / Proof of Concept

▶ Watch: Generating artificial messages using LLMs (13:40)

While the talk did not feature a live software demonstration of a fully developed tool, it thoroughly described a "test pipeline" and preliminary experiments that served as a proof of concept for the proposed methodology. Jenny Felser detailed the experimental setup used to validate the different approaches for integrating investigator knowledge into topic modeling.

The primary dataset for these preliminary experiments was derived from an international pre-data identification competition, specifically focusing on a grooming scenario. This dataset was crucial as it provided realistic, albeit simulated, communication data. It consisted of six conversations explicitly identified as being related to the grooming topic, alongside a larger collection of 100 "little conversations" which represented typical, unrelated small talk. This structure allowed the researchers to test how effectively their methods could identify the target grooming conversations amidst a significant volume of irrelevant communication.

The "test pipeline" involved feeding this dataset into a topic model, with varying forms of prior knowledge provided as guidance. The goal was to observe how different input methods – characteristic words, extractive topics from known messages, LLM-generated artificial messages, and topic summaries – influenced the model's ability to extract the "expected topic" (online grooming).

For example, when using characteristic words, the researchers would input terms thought to be relevant to grooming. The evaluation then involved assessing whether the extracted topics indeed reflected grooming or if they were diluted by more dominant, irrelevant themes. The challenges encountered, such as the ambiguity of terms like "play games" or the difficulty in determining effective seed words, were direct observations from running this part of the pipeline.

Similarly, the use of LLMs to generate artificial messages was part of this experimental setup. The team would prompt an LLM to create synthetic conversations based on grooming characteristics and then incorporate these into the topic model's training or guidance. The observation that LLMs often produced messages too general (e.g., "sex over" between adults) rather than specific to the grooming context was a direct outcome of this proof-of-concept phase.

Crucially, the "topic summaries" approach, where algorithms leveraged information from conversations to describe what grooming entailed (e.g., "exchanging contact information and making up meetings in real world"), demonstrated the most promising results within this experimental framework. This showed that a more abstract, conceptual input could lead to better-defined and more interpretable topic extraction, directly supporting the core hypothesis of the research.

The preliminary experiments thus served as a vital demonstration of the feasibility and comparative effectiveness of different knowledge integration strategies, providing concrete evidence for the strengths and weaknesses of each approach within the context of forensic communication analysis.

Defensive Implications

▶ Watch: Interactive iteration for investigator feedback (17:50)

While Jenny Felser's talk primarily focuses on post-incident forensic analysis – helping investigators make sense of communication data after a crime has been committed – the implications extend indirectly to defensive strategies and broader cybersecurity practices. Understanding how sophisticated analytical techniques can be applied to communication data offers valuable insights for organizations and individuals looking to bolster their security posture.

Firstly, for organizations dealing with sensitive internal communications, Felser's research highlights the potential for advanced topic modeling to uncover hidden patterns, even when language is deliberately disguised or codified. This knowledge can inform the development of more robust monitoring and anomaly detection systems that go beyond simple keyword alerts. Instead of just flagging explicit terms, defensive systems could potentially be trained to identify subtle shifts in communication patterns or the emergence of new "topics" that, when combined with investigator-defined hypotheses, might indicate insider threats, corporate espionage, or other malicious activities. For example, if internal communications suddenly start discussing "floor plans" or "building materials" in an unusual context, such a system, guided by a hypothesis of sabotage, could flag it.

Secondly, the emphasis on interpretable results is critical. Defensive tools often suffer from a "black box" problem, providing alerts without clear reasoning. Felser's work, by aiming for interpretable topic models, suggests that future defensive analytics could offer not just an alert, but also a summary of why a particular communication pattern is deemed suspicious, empowering security teams to respond more effectively and with greater confidence.

Thirdly, the challenges encountered in integrating investigator knowledge – particularly with codified language and multi-window contexts – underscore the importance of security awareness training within organizations. If forensic analysts struggle to decipher disguised communication post-incident, it implies that malicious actors are already leveraging such techniques. Educating employees about the risks of using ambiguous language, especially when discussing sensitive topics, and promoting clear communication guidelines can reduce the attack surface.

Furthermore, insights from this research can influence data retention policies and data collection strategies. Knowing how forensic tools analyze communication can guide organizations in deciding what data to retain, for how long, and in what format, ensuring that if an incident occurs, the necessary data is available and in a usable state for such advanced analytical methods. It emphasizes the value of maintaining context around communications to aid both defensive monitoring and subsequent forensic investigations.

Finally, for cybersecurity professionals, understanding the cutting edge of forensic analysis techniques provides a holistic view of the security lifecycle. It informs how effectively incidents can be investigated, which in turn feeds back into improving preventative measures. By recognizing the sophistication with which communication data can be analyzed, defenders can better anticipate adversary tactics and design more resilient systems and policies.

Key Takeaways

• Overwhelming Data Volume: Digital forensic investigators face an insurmountable challenge in manually reviewing the immense volume of mobile communication data (e.g., 50,000 messages equating to 50,000 pages of text), necessitating automated, intelligent solutions.
• Limitations of Traditional Methods: Traditional keyword searches are too broad or too narrow, while generic pre-trained classifiers lack the adaptability for the unique, heterogeneous, and often disguised language found in individual criminal cases.
• Hypothesis-Driven Approach is Crucial: To overcome the limitations of unsupervised topic models (which tend to extract dominant, irrelevant "small talk" topics), integrating an investigator's specific hypothesis and prior knowledge is paramount for guiding the analysis towards relevant evidence.
• Challenges in Knowledge Integration: Providing investigator knowledge to topic models is complex; simple "seed words" often fail to capture nuanced or codified language, while LLM-generated artificial messages can be too generic without careful refinement.
• Topic Summaries Show Promise: Presenting prior knowledge as a topic summary (a high-level description of what conversations about a hypothesis entail) proved to be the most effective method, enabling the model to extract more meaningful and interpretable topic mixes indicative of specific criminal activities like online grooming.
• Future for Interpretability and Interaction: Future work aims to enhance the system with a quantitative evidence score for topic relevance and an interactive iteration process allowing investigators to provide real-time feedback, making the analysis more precise and user-friendly.

About the Speaker(s)

Jenny Felser is a Doctoral researcher at the University of Applied Sciences Mittweida, where she is affiliated with the communication forensic group as a research assistant. Her PhD topic, which she anticipates completing by the end of next year, centers on developing interpretable hypothesis-driven topic modeling for forensic communication analysis, specifically focusing on mobile communication data. Felser holds a Master's degree in Cybersecurity and a Bachelor's degree in Digital Forensic Sciences, with both academic pursuits concentrated on mobile communication analysis. Her research aims to support investigators in analyzing instant messages and other digital communications, particularly in complex scenarios like online grooming, by making the evidence discovery process more efficient and interpretable.