Building Local Knowledge Graphs for OSINT

Building Local Knowledge Graphs for OSINT

Speakers: Dr. Donald Pellegrino, Desim

Conference: Recon Village

YouTube: https://www.youtube.com/watch?v=yIUfntW_TxY

Overview

In this insightful Recon Village talk, Dr. Donald Pellegrino, a seasoned expert with over two decades of research and development experience in information science, presented a robust methodology for Open Source Intelligence (OSINT) investigations. Titled "Building Local Knowledge Graphs for OSINT: Bypassing Rate Limits and Maintaining OBSAC," the presentation delved into a practical solution leveraging knowledge graph technology, specifically Resource Description Framework (RDF), to construct queryable, offline OSINT repositories. This approach addresses critical challenges faced by modern OSINT analysts, including overcoming API rate limits and preserving operational security (OBSAC) by minimizing direct interaction with external services.

Dr. Pellegrino, whose extensive background includes building systems for the Pentagon, US Department of Defense, Homeland Security, and various branches of the US military, introduced a systematic method that integrates traditional "good old-fashioned AI" principles like expert systems and knowledge bases with contemporary large language models (LLMs). The talk underscored the importance of creating a controlled, local environment for data collection and analysis, enabling investigators to conduct in-depth research without constantly revealing their investigative focus or being constrained by external service policies. This methodology promises to scale individual analytical efforts, foster collaboration, and enhance the scientific repeatability of OSINT workflows.

Background

▶ Watch: Leveraging good old-fashioned AI for modern analysis (2:40)

The landscape of OSINT investigations is fraught with challenges that can impede effective analysis. Dr. Pellegrino framed these issues around the classic CIA triad of information security: confidentiality, integrity, and availability. Each of these aspects presents unique hurdles in OSINT, with the advent of LLMs introducing new complexities.

Confidentiality is paramount in OSINT, yet traditional methods often compromise it. Every query to a web service or search engine, and now every LLM prompt, serves as an "indicator of interest," revealing the investigator's focus to data providers. This constant disclosure can lead to targeted countermeasures, alert subjects, or even legal ramifications. Bypassing this requires methods that collect data without explicitly signaling specific targets, necessitating bulk collection and local storage.

Integrity of data is another significant concern. Web content, primarily designed for human visual cognition, is often dynamic, relying on JavaScript, iframes, or infinite scrolling. Simple tools like wget or curl frequently fail to capture the full, active content of a webpage, leading to incomplete or inaccurate data. Reproducible work becomes difficult when the source material is inconsistent or dynamically rendered. Furthermore, LLMs introduce their own integrity risks, such as hallucinations or qualitative decisions by providers to filter or alter responses based on deemed "inappropriate" vocabulary, potentially hindering legitimate investigative queries.

Availability is a practical constraint. External services are subject to rate limits, downtime, or changes in terms of service, making investigators dependent on factors outside their control. Deadlines and critical investigations cannot afford such dependencies. The obvious solution, local caching, mitigates this by bringing data under the investigator's control. However, merely caching data is insufficient; it must be managed, indexed, and made queryable efficiently to scale.

Dr. Pellegrino highlighted that while local caching addresses availability and partially confidentiality (by reducing repeated queries), the systematic management of this local data and its integration with modern analytical tools like LLMs is where the true innovation lies. The talk proposes a structured approach to overcome these long-standing OSINT challenges, making investigations more secure, reliable, and scalable.

Key Findings

▶ Watch: Introducing case study: Analyzing Recon Village speakers (4:50)

The central findings of Dr. Pellegrino's talk revolve around a multi-faceted methodology that transforms how OSINT is conducted, moving from reactive querying of external services to proactive construction and leveraging of local, structured knowledge.

Firstly, the talk demonstrates that bypassing rate limits and enhancing OBSAC is achievable through systematic local caching and bulk data collection. By collecting more data than immediately necessary and burying specific targets within larger datasets, investigators can reduce their digital footprint and avoid revealing granular interests to external providers. This necessitates a robust local data management strategy.

Secondly, the presentation establishes knowledge graphs, specifically using RDF and ontologies, as a canonical framework for structuring heterogeneous OSINT data. This structured approach allows for the integration of diverse source materials – from web scrapes to expert-curated datasets – into a unified, queryable format. RDF's triple-store model (subject-predicate-object) provides a mathematically sound way to express relationships, enabling sophisticated link analysis and pattern discovery that is challenging with unstructured data. Ontologies, like the Friend-of-a-Friend (FOAF) framework, offer standardized schemas to normalize terms and facilitate interoperability across different analytical domains or even between analysts with varying expertise or languages.

Thirdly, Dr. Pellegrino showcased the transformative role of local LLMs in the OSINT pipeline. While external LLMs pose confidentiality and integrity risks (e.g., logging prompts, content filtering, hallucination), running LLMs locally empowers investigators with full control. These local models can be leveraged for highly efficient information extraction from unstructured web content, generating structured RDF data that aligns with pre-defined ontologies. This significantly reduces the manual effort traditionally required for data structuring and enhances the accuracy of extraction, as demonstrated by an LLM identifying speakers missed by a human analyst.

Finally, the talk introduced Graph Retrieval Augmented Generation (Graph RAG) as a sophisticated method to manage the context window limitations of LLMs. Instead of naive text search over documents, Graph RAG systematically feeds relevant sub-graphs from the local knowledge base into the LLM's context, ensuring that queries are grounded in precise, pre-analyzed data. This not only makes LLM-driven analysis more accurate and reliable but also enables complex graph analytics and network cluster identification directly through natural language queries, scaling analytical depth far beyond what traditional methods or basic RAG could achieve.

Technical Deep Dive

▶ Watch: Addressing key challenges: Rate limits and operational security (6:00)

The core of Dr. Pellegrino's methodology lies in a systematic, four-stage process: collecting sources, extracting information, building the knowledge graph, and leveraging the knowledge graph with LLMs.

1. Collecting Sources:

The initial step involves acquiring raw data while minimizing the investigator's footprint. Traditional tools like wget or curl are often insufficient for modern, dynamically rendered websites. The talk introduced a custom Rust implementation, leveraging an under-development Rust library called rd client, which aims to provide more robust web scraping capabilities. For highly dynamic content, browser automation tools like Playwright and Chromium were employed to ensure full page rendering and content loading, including JavaScript-generated elements and content behind tabs or infinite scrolls. To further enhance OBSAC, the custom crawler was designed to integrate with Tor infrastructure, although this aspect was noted as beyond the immediate scope of the talk. The vision is for LLMs to assist in writing and tuning these crawlers, allowing for highly customized and effective data collection based on specific investigative needs.

2. Extracting Information & Building the Knowledge Graph:

Once raw data is collected, the challenge shifts to transforming it into a structured, queryable format. This is where Resource Description Framework (RDF) and ontologies become central.

• RDF: RDF is a Worldwide Web Consortium (W3C) standard for expressing graph data in a canonical, mathematical way. It represents information as triples (subject-predicate-object), where each part is a URI. For example: John knows Jane. This structure allows for a universal encoding of relationships, making it possible to reduce most other data formats to RDF. The talk emphasized its utility in providing a common expressivity for multiple analysts or LLMs to distill salient points from diverse source materials. RDF data is typically stored in a triple store, which is a type of graph database.
• Ontologies: Ontologies are formal representations of knowledge within a specific domain, built upon RDF. They provide a schema or vocabulary to normalize terms and define relationships. Dr. Pellegrino specifically highlighted the Friend-of-a-Friend (FOAF) ontology, a standardized framework for encoding social networking data. By aligning extracted data to an ontology, investigators gain a well-thought-out data structure without having to invent one from scratch, facilitating data integration and interoperability. This allows for building sub-graphs that can connect to larger graphs or be sliced and diced for different analytical perspectives.
• LLM-Assisted Extraction: A key innovation is using local LLMs for information extraction directly into RDF. Instead of manual parsing or complex rule-based systems, an LLM (e.g., Gwen 3) can be prompted to identify entities and relationships from unstructured text (like an HTML page) and output them in RDF, aligned to a specified ontology. This dramatically reduces the effort and time required for data structuring.

3. Leveraging the Knowledge Graph with LLMs:

With a local knowledge graph constructed, the focus shifts to advanced querying and analysis using LLMs.

• SPARQL: As the knowledge graph is a database, it can be queried using SPARQL, the RDF query language. SPARQL allows for complex queries over structured graph data, similar to SQL for relational databases.
• LLM Querying: LLM queries are multi-component, consisting of a system prompt (configuring the LLM's behavior), a user prompt (the direct question), and a context window (the relevant information provided to the LLM). The challenge is that LLMs have limited context windows, making it difficult to analyze large volumes of source material directly.
• Retrieval Augmented Generation (RAG): RAG combines the LLM's pre-trained knowledge with external, specific documents. However, traditional RAG often relies on naive text search over documents, which can be inefficient and miss crucial relationships.
• Graph Retrieval Augmented Generation (Graph RAG): This is a more advanced technique where the local knowledge graph systematically manages the LLM's context window. Instead of feeding raw documents, Graph RAG identifies and extracts relevant sub-graphs or triples from the knowledge graph based on the query, introducing only the most pertinent, structured information into the LLM's context. This approach scales to large collections of source material, ensures accuracy, and enables complex graph analytics such as identifying common speakers, network clusters, and implicit connections.

Technical Stack:

The proof-of-concept utilized a custom Rust codebase for crawling. For LLM processing, Gwen 3 (a 30-billion parameter model) was used with VLM and required a GPU. Other local LLM toolkits like Llama CPP (behind LM Studio) were also mentioned as viable options for running models locally. The talk also noted that the presentation slides and some code were generated in collaboration with Claude, demonstrating LLM utility in development workflows.

Demo / Proof of Concept

▶ Watch: Decomposing local storage into portable, reusable containers (6:25)

Dr. Pellegrino presented a compelling case study centered on analyzing the speakers of the Recon Village conference itself, demonstrating the practical application of his methodology.

1. Initial Manual Collection & Its Flaws:

The first step involved a human analyst (Dr. Pellegrino) manually reviewing the Recon Village website to compile a list of speakers. While the website was visually engaging and modern, its dynamic nature posed immediate challenges. Speakers were listed in multiple sections (a main speaker list, and separate tabs for Friday and Saturday schedules that required manual scrolling to load). The manual effort, though seemingly straightforward, resulted in a significant oversight: four additional speakers were completely missed because they were on panels or listed as co-speakers in dynamically loaded sections. This highlighted the integrity risk of relying solely on human visual cognition and basic scraping tools.

2. Automated Collection with Enhanced Integrity:

To overcome the limitations of manual and basic scraping, a custom CLI tool called collect was developed in Rust. This tool leveraged the rd Rust client and Playwright/Chromium to ensure dynamic JavaScript content was fully rendered before extraction. This enabled a more accurate and complete capture of the raw source material, crucial for subsequent analysis.

3. LLM-Powered Information Extraction to RDF:

With the raw HTML content collected, Dr. Pellegrino then used a local LLM (Gwen 3) to extract the speaker information. By prompting the LLM to identify speaker names and their affiliations, the LLM not only successfully identified the four speakers missed by the human analyst but also structured this information. Crucially, the LLM was further instructed to express this extracted data in Resource Description Framework (RDF) format, aligned with the Friend-of-a-Friend (FOAF) ontology. This transformed unstructured HTML into a structured, machine-readable graph, representing relationships like "Person knows Person" or "Person worksFor Organization."

4. Data Enrichment and Integration:

The speaker data, now in RDF, was then expanded by integrating it with other sources:

• Wikipedia Data: Information on public companies (affiliations of some speakers) was collected from WikiData, which conveniently already publishes its data in RDF. This allowed for a seamless join based on company names to retrieve additional attributes, such as the founding dates of these companies. This process, performed locally, avoided informing Wikipedia of specific investigative interests.
• Manual Analyst Data: A colleague's manually curated dataset of private companies (not found on Wikipedia) was also integrated. By aligning this manually collected data to the same FOAF ontology and RDF format, it could be seamlessly combined with the automatically extracted and Wikipedia-sourced data.

5. Querying and Insights:

With the consolidated knowledge graph, SPARQL queries could be executed to derive complex insights. Examples included:

• Identifying "common speakers" or "network clusters."
• Characterizing the nature and age of companies affiliated with speakers (e.g., Tyson Foods, a 90-year-old company, Microsoft, Fortinet, Palo Alto).
• Hypothetically, comparing characteristics across different Defcon villages or tracking changes in technologies and companies over time as a function of speaker titles.

This proof-of-concept powerfully illustrated how local knowledge graphs, combined with LLM-driven extraction and standard ontologies, create a scalable, secure, and highly effective OSINT analysis pipeline, surpassing the limitations of manual effort and traditional scraping.

Defensive Implications

▶ Watch: Ensuring data integrity from human-centric web content (7:15)

The methodology presented by Dr. Pellegrino offers significant defensive implications for both OSINT practitioners and the organizations they investigate or protect.

For OSINT practitioners acting as defenders (e.g., threat intelligence analysts, incident responders, red teamers), this approach provides a robust framework to enhance their own operational security (OBSAC) and the integrity of their intelligence gathering. By building local knowledge graphs and leveraging local LLMs, analysts can:

• Reduce Digital Footprint: Minimize direct queries to external services, thereby reducing the chance of revealing investigative intent, targets, or patterns of interest to adversaries or data providers. This is crucial for maintaining stealth and avoiding premature alerting of subjects.
• Mitigate Rate Limits and Service Dependencies: Ensure continuous access to data, even if external sources impose rate limits, go offline, or change their policies. This guarantees the availability of crucial information during critical investigations.
• Improve Data Integrity and Reproducibility: The use of advanced scraping techniques (e.g., Playwright/Chromium) ensures more complete and accurate data collection from dynamic websites. Structuring this data into RDF with ontologies provides a canonical, verifiable format, making analyses scientifically repeatable and less prone to human error or LLM hallucination when controlled locally.
• Enable Advanced Analytics: The ability to perform sophisticated link analysis and graph analytics on integrated datasets allows defenders to identify hidden relationships, network clusters, and emerging threats more effectively than with traditional, siloed data analysis.
• Foster Collaboration: Standardized data formats (RDF, ontologies) facilitate seamless sharing and integration of work products among multiple analysts, even across different linguistic or domain expertise, enhancing collective intelligence capabilities.

For organizations and individuals being investigated, this talk serves as a critical reminder of the advanced capabilities available to sophisticated OSINT practitioners. Key takeaways include:

• Dynamic Content is Not a Full Shield: Relying on JavaScript-driven content to deter basic scraping is insufficient against advanced browser automation tools. Organizations should assume that all publicly accessible information, regardless of how it's rendered, can be collected.
• Interconnected Data is a Vulnerability: The ability to seamlessly integrate diverse public datasets (e.g., company websites, Wikipedia, social media) into a unified knowledge graph means that seemingly disparate pieces of information can be linked to reveal deeper insights, relationships, and vulnerabilities. Organizations should conduct internal OSINT assessments to understand what a determined investigator could piece together about their personnel, partners, and infrastructure from publicly available sources.
• LLMs Amplify OSINT: The power of LLMs to extract, summarize, and structure information from vast amounts of unstructured text significantly amplifies OSINT capabilities. Organizations should be aware that even subtle mentions or obscure data points can be automatically identified and incorporated into an investigator's knowledge graph.

In essence, Dr. Pellegrino's work equips defenders with a powerful methodology to conduct OSINT more securely and effectively, while simultaneously highlighting the evolving sophistication of OSINT techniques that organizations must be prepared to counter.

Key Takeaways

• Local Caching and Bulk Collection: Building local data repositories by systematically collecting more data than immediately needed is crucial for bypassing API rate limits and reducing the digital footprint of OSINT investigations, thereby enhancing operational security (OBSAC).
• Knowledge Graphs as Canonical Data Stores: Leveraging Resource Description Framework (RDF) and ontologies (like FOAF) provides a standardized, mathematically sound way to structure and integrate heterogeneous OSINT data, enabling robust link analysis and data integration across diverse sources and analytical efforts.
• Local LLMs for Secure and Efficient Extraction: Running large language models (LLMs) locally (e.g., Gwen 3 via VLM) enables highly efficient and accurate information extraction from unstructured web content, generating structured RDF data aligned with ontologies, all while maintaining OBSAC and avoiding external service biases or prompt logging.
• Graph Retrieval Augmented Generation (Graph RAG): This advanced technique systematically uses the local knowledge graph to populate an LLM's context window with precise, relevant sub-graphs, overcoming context limitations and enabling sophisticated graph analytics and reliable query responses grounded in structured data.
• Scientific Repeatability: The methodology promotes scientific repeatability by standardizing data representation, controlling LLM environments locally, and allowing for systematic experimentation with prompts and models, leading to more reliable and verifiable OSINT findings.
• Empowering the Investigator: By combining "good old-fashioned AI" principles with modern LLMs and knowledge graph technologies, investigators gain greater control over their tools, data, and analytical workflows, allowing them to scale their efforts and derive deeper insights securely and efficiently.

About the Speaker(s)

Dr. Donald Pellegrino is a distinguished expert with over 20 years of research and development experience in the field of information science. His extensive career includes building sophisticated systems for critical government entities such as the Pentagon, the US Department of Defense, Homeland Security, the US Army, and the US Navy, among many other organizations. Dr. Pellegrino's background is rooted in software development and "good old-fashioned artificial intelligence," with a deep understanding of expert systems, knowledge bases, and pre-LLM AI approaches. He is associated with Desim, an organization whose name is a portmanteau of "decision symbols," reflecting his work in rule-based expert systems and knowledge management. His work focuses on empowering investigators and analysts through innovative data integration and analytical methodologies.