Fast Synthetic Data Generation for Case-Specific Entity Extraction in Criminal Investigations

Fast Synthetic Data Generation for Case-Specific Entity Extraction in Criminal Investigations

Speakers: Mads Skipanes, Doctoral researcher, National Criminal Investigation Service (Kripos)

Conference: DFDS

YouTube: https://www.youtube.com/watch?v=hyeRyLkscCY

Overview

In this insightful presentation at DFDS, Mads Skipanes, a doctoral researcher at the National Criminal Investigation Service (Kripos), unveiled a novel approach to addressing a persistent challenge in criminal investigations: the laborious and time-consuming process of information analysis. The talk, titled "Fast Synthetic Data Generation for Case-Specific Entity Extraction in Criminal Investigations," delves into Skipanes's PhD research, which aims to establish a framework for semi-automated information analysis, heavily leveraging advancements in natural language processing (NLP). At its core, the study proposes using Large Language Models (LLMs) to generate synthetic data, enabling the rapid fine-tuning of Named Entity Recognition (NER) models tailored to the unique linguistic and contextual nuances of individual criminal cases.

The significance of this research cannot be overstated. Criminal investigations generate an immense volume of unstructured text data—from interview transcripts and crime scene reports to digital forensics findings—all consolidated within what is often referred to as the "case file." Manually sifting through these documents to identify, extract, and connect critical entities (persons, locations, vehicles, events) is a monumental task that drains investigative resources and can impede the speed and effectiveness of justice. Skipanes's work directly confronts this bottleneck by demonstrating a methodology that promises to accelerate the information analysis pipeline, allowing investigators to focus on higher-level analytical tasks rather than manual data indexing. The ability to quickly adapt NER models to specific case contexts, whether a cybercrime incident or a homicide investigation, represents a crucial step towards more efficient and responsive law enforcement operations.

Background

▶ Watch: Introduction to PhD research and information analysis (0:00)

The foundational premise of Skipanes's PhD research revolves around developing a robust framework for semi-automated information analysis within criminal investigations. This endeavor sits at the intersection of digital forensics and natural language processing, aiming to bridge the gap between the raw, often chaotic, influx of investigative data and actionable intelligence. As Skipanes highlighted, investigators are confronted with a vast hub of information—interviews, crime scene forensics, digital artifacts—all contributing to the "case file," which predominantly consists of unstructured textual documents. The challenge lies in processing these documents to identify key entities and establish connections that can drive an investigation forward.

Prior to this specific study, Skipanes and his team had already undertaken several critical steps in his three-year PhD journey. They began by meticulously specifying and defining information analysis within the context of criminal investigations, subsequently bridging these definitions to computational methods. This included the identification and selection of key NLP techniques, with Named Entity Recognition (NER) emerging as a primary candidate for processing the extensive document lists. Furthermore, early work explored the application of semantic text similarity to address the dynamic and evolving information needs throughout an investigation. Unlike traditional keyword or string matching, semantic similarity allows investigators to find relevant information even when the exact phrasing differs, crucial for answering questions like "did someone hear about a red vehicle?" as information needs shift. Another related study focused on Retrieval Augmented Generation (RAG), with the similar goal of improving the effectiveness of identifying relevant information in ongoing investigations.

The impetus for this research stems from the inherent inefficiencies of current manual processes. Skipanes emphasized that the manual indexing of police documents is not only time-demanding but also resource-intensive. Roles such as "document reader," "indexer," and "indexing supervisor" exist solely to manage the deluge of incoming and generated documents. To illustrate the scale of this challenge, Skipanes cited data from two homicide cases: these involved 4,321 manually created entities, 13,379 manually generated links, and close to 100 investigators over the course of the investigations. Such figures underscore the urgent need for automation to alleviate the burden on human resources and accelerate the investigative timeline. To standardize the approach to entity identification, the team started by presenting an ontology used within the Norwegian criminal investigation service. This ontology serves as a standard for criminal cases, defining broad entity classes, though specific subclasses can vary significantly depending on the nature of the individual case. This established context provides the groundwork for developing targeted, efficient computational tools.

Key Findings

▶ Watch: Challenge: manual indexing and criminal investigation ontology (4:50)

The central objective of Skipanes's study was to ascertain the feasibility and effectiveness of using Large Language Models (LLMs) for generating synthetic data to fine-tune Named Entity Recognition (NER) models for case-specific entity extraction. The underlying hypothesis was that while a general NER model might perform well in a specific domain, such as cybercrime, its performance could degrade significantly when applied to a different case scenario, like a homicide investigation. The goal was to develop a fast methodology for adapting models to new domains as an investigation unfolds.

The experimental results yielded a nuanced but promising picture. The methodology involved using a confidential dataset of two homicide cases as inspiration. From this, "lookalike sentences" were extracted and fed into Claude, the chosen LLM, to generate synthetic data. This process resulted in the creation of 817 synthetic sentences and 1458 synthetic entities. A ScandiNER model, pre-trained for Scandinavian languages, was then fine-tuned using this synthetically generated dataset.

When evaluated on the synthetic dataset itself, the fine-tuned model demonstrated exceptionally strong performance, achieving a weighted average correctness of 94.2%. This metric, a combination of precision and recall, indicated that the model was highly effective at identifying and classifying entities within the data it was trained on. However, the true test came when the model was applied to a separate, unseen, and manually annotated set of 25 sentences from the confidential, real-world data. In this scenario, the model's performance experienced a noticeable drop, yielding a weighted average correctness of 81.6%.

Skipanes attributed this performance disparity primarily to two factors: the limited size of the real-world evaluation dataset and persistent challenges related to linguistic patterns. Despite efforts to generate "lookalike" linguistic patterns from the confidential data, a gap remained between the synthetic and real-world linguistic nuances. Nevertheless, the overarching conclusion of the study was positive: the method demonstrates that LLMs can indeed be effectively utilized for synthetic entity labeling data generation, subsequently enabling the training of lighter-weight NER models. This approach, Skipanes noted, facilitates an "easy and human-understandable transfer of label data" from analog police work to training data for entity extraction, thereby encouraging a more seamless integration of computational methods into traditional investigative practices. The team also emphasized their commitment to sharing their Norwegian dataset, acknowledging the broader challenge of data availability in this sensitive domain.

Technical Deep Dive

▶ Watch: Methodology: Generating synthetic data with Claude LLM (6:00)

The technical core of Skipanes's research addresses the critical need for adaptable Named Entity Recognition (NER) models in the dynamic landscape of criminal investigations. The fundamental problem is that a NER model, meticulously trained and optimized for a specific domain (e.g., cybercrime with its unique terminology for malware, IP addresses, and attack vectors), may not perform adequately when confronted with an entirely different domain, such as a homicide investigation involving names, locations, vehicles, and specific event sequences. The objective was to explore a "fast method" for fine-tuning these models in an ongoing investigation, making them "case-specific."

The scope of "criminal investigation" in this context is broad, encompassing all collected sources of information. The primary focus is on connecting disparate pieces of information across the multitude of documents generated throughout an investigation. To structure the entity extraction process, the team began by presenting an ontology employed by the Norwegian criminal investigation service. This ontology provides a standardized set of entity classes for criminal cases, though the subclasses can vary significantly based on the specific type of investigation. This hierarchical classification is crucial for defining what constitutes a relevant "entity" in a given context, providing the target labels for the NER model.

The motivation for this automation is compelling. As highlighted, two homicide cases alone involved 4,321 manually created entities and 13,379 manually generated links, with nearly 100 investigators involved. This sheer volume underscores the resource drain and the potential for human error or oversight inherent in manual processes. The proposed solution leverages Large Language Models (LLMs) for synthetic data generation, aiming to bypass the laborious and costly manual annotation process typically required for training or fine-tuning NER models.

The methodology unfolded as follows:

1. Inspiration Data: A confidential dataset derived from two actual homicide cases served as the source of real-world linguistic patterns and contextual information.
2. Lookalike Sentence Generation: Rather than directly using the confidential data for training, "lookalike sentences" were extracted or crafted based on the patterns observed in the confidential dataset. This step is critical for maintaining the privacy and confidentiality of sensitive investigative data while still capturing the domain-specific linguistic characteristics.
3. LLM-based Synthetic Data Generation: These "lookalike sentences" were provided as input to Claude, the chosen LLM. The LLM was then prompted to generate synthetic sentences and associated entity labels. This process yielded 817 synthetic sentences and 1458 synthetic entities, creating a new, ethically sound training dataset that mimics the characteristics of real investigative documents.
4. Model Fine-tuning: A pre-existing ScandiNER model was selected for fine-tuning. ScandiNER is an NER model previously fine-tuned for Scandinavian languages, making it a suitable baseline for processing Norwegian investigative texts. The synthetic dataset generated by Claude was then used to fine-tune this model, adapting its entity recognition capabilities to the specific entity classes and linguistic style relevant to criminal investigations.
5. Evaluation: Model performance was assessed using two distinct datasets:

• Synthetic Data: The model was initially evaluated on the synthetic data it was trained on, achieving a high weighted average correctness of 94.2%.
• Confidential, Unseen Data: To gauge real-world applicability, the fine-tuned model was then tested on a separate set of 25 manually annotated sentences from the original confidential dataset. This evaluation revealed a performance of 81.6% weighted average correctness.

The observed performance drop on the unseen confidential data, while present, is critical for understanding the limitations and future directions. Skipanes attributed this to the limited size of the evaluation data and the nuanced challenges of replicating truly authentic linguistic patterns. Despite this, the ability to achieve over 80% accuracy on unseen, real-world data with a relatively small amount of synthetic training data generated rapidly by an LLM represents a significant advancement for resource-constrained investigative environments. The methodology highlights a practical pathway for creating specialized NER models without extensive manual annotation, accelerating the deployment of AI tools in law enforcement.

Demo / Proof of Concept

▶ Watch: Conclusion: LLMs promising for synthetic entity labeling (8:15)

While the talk did not feature a live demonstration or a specific proof of concept tool, the speaker presented the methodology and results of their study as a compelling proof of concept for the proposed approach. The core of the demonstration was the quantitative evidence of the fine-tuned ScandiNER model's performance on both synthetic and real-world confidential data. The detailed explanation of how Claude (LLM) was used to generate 817 synthetic sentences and 1458 entities, followed by the fine-tuning of ScandiNER, served as a conceptual walkthrough of the system's operation. The presentation of the 94.2% weighted average correctness on synthetic data and the 81.6% weighted average correctness on unseen confidential data functioned as the empirical validation of the method's viability, illustrating its potential for practical application in criminal investigations.

Defensive Implications

▶ Watch: Future work: Information systems, knowledge graph generation (9:00)

The implications of this research, while not directly tied to traditional cybersecurity "defenses" against external threats, are profoundly "defensive" in the context of law enforcement operations. This work offers a powerful defense against the overwhelming tide of unstructured information that can hinder and delay criminal investigations. By providing a framework for semi-automated information analysis, Skipanes's methodology equips investigators with advanced tools to combat information overload and resource scarcity.

Firstly, the ability to rapidly fine-tune Named Entity Recognition (NER) models for case-specific entity extraction significantly enhances the efficiency of data processing. Instead of investigators manually sifting through thousands of documents to identify persons, locations, vehicles, or specific events, an AI-powered system can perform this initial triage with impressive speed. This frees up human investigators to focus on higher-level analytical tasks, critical thinking, and complex pattern recognition that still require human intuition and expertise. This is a defensive measure against time-consuming, repetitive manual labor.

Secondly, the use of Large Language Models (LLMs) for synthetic data generation addresses a critical bottleneck: the lack of readily available, annotated training data in sensitive domains like criminal investigations. Generating high-quality, domain-specific training data manually is prohibitively expensive, time-consuming, and often impossible due to confidentiality constraints. By enabling the quick creation of synthetic datasets, the proposed method provides a "defense" against data scarcity, allowing law enforcement agencies to deploy tailored AI solutions without compromising sensitive information or expending vast resources on manual annotation. This also allows for rapid adaptation of models to novel crime types or evolving linguistic patterns observed in new cases.

Finally, the research indirectly supports improved investigative agility and responsiveness. In fast-moving investigations, the ability to quickly adapt computational tools to new information needs or shifts in the case's focus can be crucial. The methodology's emphasis on "fast synthetic data generation" means that investigators are not beholden to static, general-purpose models but can instead deploy highly specific, dynamically tuned NER systems. This enhances the investigative team's capacity to "connect the dots" more effectively and rapidly, potentially leading to faster identification of suspects, recovery of evidence, and ultimately, swifter justice. The challenges highlighted (limited data, linguistic patterns) also serve as a "defensive" warning, urging practitioners to continuously refine their data generation and model evaluation strategies to ensure robust performance in real-world scenarios.

Key Takeaways

• LLMs for Synthetic Data: Large Language Models (LLMs) like Claude show significant promise for generating synthetic, labeled data for Named Entity Recognition (NER) in sensitive domains like criminal investigations, overcoming challenges of data scarcity and confidentiality.
• Case-Specific Adaptability: The methodology enables rapid fine-tuning of NER models, such as ScandiNER, to become "case-specific," addressing the limitation that general-purpose models may not perform well across different types of criminal cases (e.g., cybercrime vs. homicide).
• Performance Trade-offs: While the fine-tuned model achieved a high weighted average correctness of 94.2% on synthetic data, performance dropped to 81.6% on unseen, real-world confidential data, indicating the need for larger evaluation datasets and further refinement in replicating authentic linguistic patterns.
• Bridging Analog and Digital: The approach facilitates an intuitive transfer of knowledge from traditional "analog police work" (e.g., manual entity identification) into structured training data for computational models, streamlining the integration of AI into investigative processes.
• Addressing Information Overload: This research directly tackles the challenge of processing vast amounts of unstructured text in criminal case files, aiming to semi-automate the laborious task of information analysis and free up human investigators for higher-level tasks.
• Future Directions: Ongoing work includes improving information management systems in criminal investigations and automating knowledge graph generation for exploratory analysis, further enhancing the ability to connect complex information in an automated fashion.

About the Speaker(s)

Mads Skipanes is a Doctoral researcher affiliated with the National Criminal Investigation Service (Kripos). His extensive PhD research is dedicated to developing a comprehensive framework for semi-automated information analysis within the complex domain of criminal investigations. This work is deeply rooted in the field of natural language processing, aiming to leverage computational methods to enhance how law enforcement agencies process and connect information from vast, unstructured textual data sources. His studies, including the one presented, focus on improving efficiency and effectiveness in investigative processes through innovative applications of AI and machine learning.