From OSINT to Detection: Building an Agentic CTI Pipeline
Andrew Skatoff (Information Security Manager Sr · Federal Reserve)
RVAsec 2026 · Day 2 · Grand Ballroom F/G
Overview
In a landscape where cyber threats evolve at an unprecedented pace, security teams, particularly blue teams and threat hunters, are constantly overwhelmed by the sheer volume of incoming Cyber Threat Intelligence (CTI). Andrew Skatoff's RVAsec talk, "From OSINT to Detection: Building an Agentic CTI Pipeline," tackles this critical challenge head-on. Skatoff, an Information Security Manager Sr at the Federal Reserve, presented Huntable CTI Studio, an innovative, AI-assisted workbench designed to transform raw, unstructured CTI into validated and novel detection rules with remarkable efficiency.

Key moments
- 0:00 Introduction: The overwhelming CTI volume problem
- 4:00 Manual CTI analysis: The cognitive burden
- 6:30 Introducing Huntable CTI Studio workbench
- 10:40 Guiding principles for building AI automation
- 15:30 Scoring CTI with 'perfect discriminators'
- 29:30 Why LLMs are essential for understanding meaning
- 38:00 Sigma rule generation and novelty check
- 45:00 AI as an exoskeleton: Key takeaways
From OSINT to Detection: Building an Agentic CTI Pipeline
Speakers: Andrew Skatoff, Information Security Manager Sr, Federal Reserve
Conference: RVAsec
YouTube: https://www.youtube.com/watch?v=J11hMgmIAFU
Overview
In a landscape where cyber threats evolve at an unprecedented pace, security teams, particularly blue teams and threat hunters, are constantly overwhelmed by the sheer volume of incoming Cyber Threat Intelligence (CTI). Andrew Skatoff's RVAsec talk, "From OSINT to Detection: Building an Agentic CTI Pipeline," tackles this critical challenge head-on. Skatoff, an Information Security Manager Sr at the Federal Reserve, presented Huntable CTI Studio, an innovative, AI-assisted workbench designed to transform raw, unstructured CTI into validated and novel detection rules with remarkable efficiency.
The core problem addressed by this talk is the scalability of human analysis in the face of a "fire hose" of threat data. With thousands of articles, blogs, and Indicator of Compromise (IOC) dumps published monthly, only a tiny fraction contains actionable intelligence suitable for creating detection rules. Skatoff's work demonstrates how artificial intelligence, when applied judiciously and with strong guardrails, can act as a force multiplier, automating much of the laborious process of CTI consumption and rule generation.
This article delves into the architecture, design principles, and technical mechanisms of Huntable CTI Studio, exploring how it filters noise, extracts critical behaviors, generates Sigma rules, and ensures novelty, all while keeping the human analyst firmly in control. The insights shared are not just about a specific tool but offer a blueprint for leveraging Large Language Models (LLMs) and other AI techniques to enhance defensive capabilities in any security operation.
Background
▶ Watch: Introduction: The overwhelming CTI volume problem (0:00)
The genesis of Huntable CTI Studio lies in a ubiquitous problem faced by nearly every detection and threat hunting team: an overwhelming volume of CTI. Skatoff highlighted that teams are inundated with 30-40 feeds, vendor blogs, IOC dumps, and breach reports daily. Within this deluge, only a "handful of behaviors" are truly worth the effort of writing a detection rule. Manually sifting through this data is simply unsustainable. For instance, Threat Cluster reportedly tracked 30,000 articles in Q4 of the previous year, yet only about 1.5% scored high or critical for actionability. Skatoff's own tracking, using the tools he developed, found that approximately 93% of articles, even from reputable sources, lacked the actionable signal required for detection engineering.
The process for an analyst to convert CTI into a detection rule is arduous and highly manual. It begins with the difficult task of identifying a useful report, a challenge compounded by the lack of a standard format for documenting attacker techniques and tradecraft. Once a promising report is found, the analyst must then pinpoint sections with detection value. This is where significant cognitive work begins. As Skatoff illustrated with a real-world example of a ClickFix attack command line, a single line of text can conceal an entire kill chain. An expert threat hunter must interpret details like conhost launching headless shells, the creation of a scheduled task (MS task UI) for persistence, and beaconing to a stealthy domain. This interpretation requires deep knowledge of living off the land binaries (LOLBINs), Windows internals, specific telemetry sources, and existing detection coverage. Critically, none of this interpretive context is explicitly stated in the report; it resides solely "inside of the threat hunter's head."
This manual, skill-intensive, and time-consuming process for even a single command line, multiplied by thousands of reports weekly, creates an insurmountable backlog for many teams. The motivation behind Huntable CTI Studio was not to replace human analysts but to automate a significant fraction of these steps, thereby accelerating blue team velocity and making life considerably harder for attackers. The project also served as an opportunity for Skatoff to deepen his own understanding and application of rapidly evolving AI technologies.
Key Findings
▶ Watch: Introducing Huntable CTI Studio workbench (6:30)
The central finding presented by Andrew Skatoff is the successful development and demonstration of Huntable CTI Studio, an AI-assisted workbench that significantly streamlines the conversion of CTI into validated and novel detection rules. This prototype effectively addresses the critical challenge of CTI overload by automating the laborious steps that previously consumed valuable analyst time.
Key findings and contributions of Huntable CTI Studio include:
- Drastic Reduction in Noise: The pipeline, particularly its scoring and filtering mechanisms, can eliminate approximately 93% of irrelevant or non-actionable CTI articles before they reach resource-intensive processing stages. This is achieved through a combination of curated regex patterns and an ML classifier.
- Automated Extraction of Attacker Techniques: Specialized extractor agents, powered by Large Language Models (LLMs), can accurately identify and pull specific attacker techniques (e.g., command lines, process trees, registry changes) from filtered content. These agents are designed with "surgical prompts" to be precise and "fail closed," minimizing hallucinations.
- Automated Sigma Rule Generation and Validation: The system can generate Sigma rules from extracted techniques and validate their structure using the PySigma library. This ensures that the generated rules are syntactically correct and ready for deployment or further refinement.
- Novelty Detection: A crucial component is the similarity service, which prevents the creation of duplicate rules. It compares newly generated rules against existing ones (e.g., the Sigma HQ repository or an organization's internal rules) by analyzing their atomic behavioral conditions, rather than just metadata. This ensures that only truly novel detections are queued for analyst review.
- Human-in-the-Loop Control: Despite extensive automation, Skatoff emphasizes that the human analyst remains the "final gate." No rules are automatically deployed. The system is a "studio, not a push-button pipeline," providing a workbench for review, editing, and approval, thus maintaining human oversight and judgment.
- Auditable by Design: Every decision made by the system is fully traceable, providing evidence and detailed logs (including full chat logs from LLM interactions). This transparency is vital for debugging, building trust, and understanding the rationale behind generated outputs.
- Configurable and Adaptable: The system is designed to be configurable, not prescriptive. Users can tune various parameters, including hunt priorities, telemetry sources, and LLM hyperparameters, to align with their specific threat model and environment.
Ultimately, Huntable CTI Studio demonstrates that AI can serve as an "exoskeleton" for security analysts, empowering them to "lift more with the same effort" by significantly accelerating the CTI-to-detection lifecycle, rather than replacing their critical cognitive functions.
Technical Deep Dive
▶ Watch: Scoring CTI with 'perfect discriminators' (15:30)
Huntable CTI Studio is orchestrated as a multi-stage pipeline, leveraging a combination of traditional programming logic, machine learning, and advanced LLM techniques. The entire stack runs on standard web app Python, containerized with Docker, and uses LangGraph for orchestrating the agent workflow, which allows for complex loops and conditional logic, a significant advantage over linear frameworks like LangChain. It supports various LLM providers, including Anthropic, OpenAI, and local inference via LM Studio.
The pipeline comprises six main steps: Ingest, Scoring, Filter, Extract, Generate, and Q (Queue).
Ingest
The initial stage focuses on collecting CTI articles. Huntable CTI Studio comes pre-configured with subscriptions to approximately 35 high-quality sources, chosen for their likelihood of providing detailed attacker techniques. Users can specify URLs, check frequency, and historical lookback. For ad-hoc ingestion, a Chrome extension allows one-click article import, and direct URL or PDF imports are also supported.
Scoring and Filtering: The Quality Gates
This crucial two-stage process acts as the initial quality control, designed to be cheap before expensive, meaning it uses minimal resources to discard irrelevant content before engaging costly LLM operations or human review.
Scoring
The first quality gate uses a regex-based huntability score to rank articles. Skatoff deliberately chose regex over LLM inference for this stage to ensure cost-efficiency and determinism. The scoring system assigns points across five categories, with a maximum score of 100.
- Perfect Discriminators (up to 75 points): This category is weighted most heavily due to its high predictive value. It includes terms rarely found in marketing material but common in Windows attack techniques, such as
rundl32,httpsspelled backwards, orfrom base64 string. - Additional Patterns: Hundreds of further patterns accumulated through testing.
- Negative Indicators: Demerits are assigned for phrases like "best practices," "free trial," or marketing platitudes, signaling low detection value.
The scoring mechanism employs geometric decay, meaning the first match of a term contributes the most to the score, with subsequent matches contributing progressively less. This prevents score inflation from repetitive keyword usage. This stage alone is sufficient to eliminate roughly 93% of articles.
Filter
The second quality gate, the junk filter, sifts through high-scoring articles to remove irrelevant content before it reaches the LLM. This is critical for three reasons:
- Context Window Limits: LLMs have finite context windows; irrelevant text pushes valuable threat data out of scope, especially for local models running on GPUs.
- Signal-to-Noise Ratio Degradation: Burying actionable commands in marketing fluff forces the LLM to spread its attention thin, leading to missed details and inaccuracies.
- Distribution Shift: LLMs can imitate the "voice" of their input. Feeding them marketing copy can lead to less precise and "brochure-like" detection logic.
The filter operates as a hybrid two-tier system:
- Perfect Discriminator Protection: Chunks of text (roughly paragraph-sized, ~1000 characters) containing any perfect discriminator strings are automatically passed through, bypassing the ML model.
- Random Forest Classifier: For all other chunks, a Random Forest classifier evaluates 20 features to determine if the chunk is "huntable" or not. This model was trained on 800 human-annotated chunks from high-quality sources (e.g., Defer Report) and validated against hundreds more. The classifier has a configurable confidence threshold (default 60%). The UI allows analysts to provide feedback on chunk classifications, triggering retraining of the model in 10-15 seconds, and an evaluation system tracks performance over time.
Extract: Purpose-Built Agents
This stage employs specialized extractor agents, each designed to identify and pull specific types of artifacts. There are six primary types: general command lines, process trees (parent-child relationships), registry entries, services, scheduled tasks, and verbatim detection rules (e.g., existing hunt queries or Sigma rules embedded in reports). Each agent is aware of the others to ensure they "stay in their lane."
Skatoff emphasized why LLMs are essential for extraction, contrasting them with deterministic tools like regex or Named Entity Recognition (NER). While regex is excellent for matching spelling, it cannot cope with the "infinite spellings" of meaning. NER might perfectly parse a sentence like "windword executed rundl32.dll," but semantically, it's incorrect (a DLL cannot be a process). LLMs, designed for understanding meaning, can handle these nuances and the vast permutations of natural language.
The power of these extractor agents lies in their agent contracts, which are essentially very long, specific, and "surgical" prompts. These are not code but plain language instructions that define precisely what an agent is allowed to do, what constitutes a valid artifact (e.g., a real command line), and what it must refuse to extract. This "fail closed" philosophy is critical for minimizing hallucinations. These prompts are tunable, allowing users to tighten or loosen rules to adjust the agent's conservatism.
Generate: Sigma Rule Creation and Novelty Check
The generate step takes the extracted techniques and creates Sigma rules. Sigma is an open, generic signature format that is vendor-agnostic, allowing detection logic to be translated into various SIEM formats (Splunk, QRadar, etc.).
The generation process involves:
- Rule Generation: An LLM agent, guided by an elaborate prompt, drafts the Sigma rule.
- Validation: The rule is passed to PySigma, a validation library, to ensure it is correctly structured. The system attempts validation up to three times.
- Similarity Service (Novelty Check): This is a critical component that determines if the newly generated rule is truly novel or if coverage already exists. This service operates in three stages:
- Stage 0 (Coarse Check): The rule is reduced to its canonical form (stripping cosmetic differences) and hashed. If this fingerprint already exists, it's a verbatim duplicate and immediately dropped.
- Stage 1 (Candidate Selection): If not a verbatim duplicate, the service pulls candidate rules from repositories (e.g., Sigma HQ, internal rules) that belong to the same telemetry class (e.g., process creation rules are only compared against other process creation rules).
- Stage 2 (Atomic Comparison): The new rule is compared against the candidates by converting both into one or more atomic conditions (fields, values, and logical conditions). This allows for a precise, behavioral comparison, showing exactly which parts overlap. Skatoff highlights that this approach is preferred over embeddings for structured data like Sigma rules, as embeddings can blur the specific structural overlaps that an analyst needs to see. For example, a rule generated from an Interlock Rat article (PowerShell spawning PHP) was compared to an existing rule. They shared one atomic condition (
php.exeas the image), but the new rule had five unique conditions, resulting in a low similarity score of 0.08 (well below the default threshold of 0.5), thus flagging it as novel.
Q (Queue): The Human Studio
The final stage is the AI-assisted studio where human analysts review, refine, and approve the generated rules. The UI provides full traceability, showing the extracted observables and the exact source excerpt from the original article that formed the basis of the rule. This "sanity check" is crucial for human oversight. Analysts can directly edit rules or use an "enrich" button to invoke an LLM for polishing tasks, such as adding false positive guidance. A comparison feature ensures that LLM-assisted edits only refine the rule's presentation, not its underlying logic.
Demo / Proof of Concept
▶ Watch: Why LLMs are essential for understanding meaning (29:30)
While Skatoff opted for pre-recorded video segments rather than a live demonstration to avoid uncertainty, his presentation vividly walked through the functionality of Huntable CTI Studio. The demonstration highlighted several key aspects of the user experience and underlying mechanisms:
- Visual Scoring and Feedback: The UI presents ingested articles with sections color-coded based on their huntability score categories (e.g., perfect discriminators, additional patterns). This acts as a visual heat map, immediately drawing an analyst's attention to high-signal areas. Analysts can interact with these "chunks" of text, double-clicking to provide feedback (e.g., "this is a good chunk") to the underlying machine learning model. This direct feedback loop is crucial for continually refining the system's accuracy.
- Junk Filter Tuning and Evaluation: The demo showcased a dedicated "junk filter tuning" section. Here, analysts can review how the Random Forest classifier has categorized each chunk from an article. They can correct misclassifications, providing explicit feedback to the model. After collecting new annotations, the system can be retrained in approximately 10-15 seconds. An automated evaluation system then displays performance metrics, including a line chart tracking changes across different configuration versions and a detailed version comparison to understand specific article-level impacts of tuning. This empowers users to customize the filter's behavior to match their specific needs, for instance, deciding whether to include atomic IOCs in their detection scope.
- Agent Configuration and Testing: The UI provides a comprehensive interface for configuring each pipeline step and agent. Users can toggle agents on or off, experiment with LLM hyperparameters (e.g., setting
temperatureto zero for extractor agents to ensure minimal creativity and deterministic output), and select preferred LLM providers (Anthropic, OpenAI, LM Studio). Critically, the full agent contract (prompt) for each LLM-powered agent is visible and editable directly within the UI, allowing users to fine-tune the instructions. A "test" feature enables users to immediately evaluate the impact of prompt changes on a single article and agent, facilitating rapid iteration and debugging.
- Workflow Execution and Observability: Once configured, an article can be sent to the workflow with a "send to workflow" button. The system provides a live view and a status board, detailing the execution. Upon completion, a comprehensive observability trail is generated. This trail includes summary-level information, such as the number of Sigma rules created (e.g., 10 rules from one article), and full chat logs from all LLM interactions. This extreme transparency allows analysts to inspect every decision point and understand precisely how the system arrived at its conclusions.
- Sigma Rule Review and Enrichment: The "Sigma Q" section is where generated rules await human review. For each rule, the UI clearly displays the extracted observables and the exact source excerpt from the original article, providing crucial traceability for validating the rule's basis. Analysts can directly edit the rule or use an "enrich" button to invoke an LLM to perform polishing tasks, such as adding false positive guidance. A built-in comparison feature ensures that LLM-driven edits only modify metadata or descriptive text, confirming a 100% duplicate match on the underlying detection logic before approval.
The demo effectively showcased Huntable CTI Studio as a functional prototype, emphasizing its transparency, configurability, and the continuous human oversight embedded throughout the automated pipeline.
Defensive Implications
▶ Watch: AI as an exoskeleton: Key takeaways (45:00)
Huntable CTI Studio offers profound defensive implications, fundamentally shifting how blue teams can operate in a high-volume threat landscape:
- Accelerated Detection Velocity: By automating the initial 93% of CTI processing and rapidly converting actionable intelligence into validated Sigma rules, security teams can dramatically reduce the time from threat report publication to effective detection deployment. This acceleration provides a critical advantage in countering fast-moving adversaries.
- Strategic Resource Allocation: Analysts are freed from the monotonous, time-consuming toil of manual CTI review and rule drafting. This allows them to reallocate their expertise to higher-value tasks, such as complex threat hunting, refining detection logic, investigating alerts, and proactive security posture improvements.
- Proactive and Comprehensive Coverage: The system enables teams to keep pace with the influx of new attacker techniques. By efficiently processing more CTI, organizations can ensure their detection capabilities are more comprehensive and proactively address emerging threats reported in the wild.
- Reduced Duplication and Improved Rule Quality: The similarity service ensures that only novel detection rules are generated and queued. This prevents the creation of redundant rules, optimizing SIEM performance and reducing maintenance overhead. The "surgical prompts" and human-gated review process contribute to the precision and quality of the generated rules, minimizing false positives and maximizing detection efficacy.
- Tailored Defense: The configurable nature of Huntable CTI Studio—allowing customization of hunt priorities, telemetry sources, and LLM behavior—means that organizations can tailor their detection pipeline precisely to their unique threat model, technology stack, and compliance requirements. This ensures that the generated rules are highly relevant to their specific environment.
- Enhanced Transparency and Trust in AI: The "auditable by design" principle, providing full trace evidence and chat logs for every decision, fosters trust in AI-generated output. Defenders can scrutinize the rationale behind each rule, understand its origins, and debug effectively, moving past the "black box" perception often associated with AI.
- Empowerment through Exoskeleton AI: Skatoff's analogy of AI as an "exoskeleton" is key. It underscores that AI is not a replacement but an augmentation tool. It empowers analysts to achieve more with the same effort, enhancing their capabilities rather than diminishing their role. This perspective is vital for adoption and for ensuring that human judgment remains central to security operations.
Key Takeaways
- AI as an Exoskeleton for Analysts: Large Language Models and AI tools should be viewed as augmentations that empower security analysts to "lift more with the same effort" by automating repetitive tasks, rather than replacing human expertise.
- The Overwhelming Volume of CTI Necessitates Automation: The sheer scale of incoming CTI, with approximately 93% being non-actionable, makes manual processing unsustainable. Automation is critical for blue teams to keep pace with evolving threats.
- Surgical Prompts and Agent Contracts are Paramount: When using LLMs for extraction, precise, long, and specific "agent contracts" (prompts) are essential. These must define clear boundaries, specify what to extract, and mandate a "fail closed" approach to minimize hallucinations and ensure reliable output.
- Multi-Stage Filtering Optimizes Cost and Quality: Implementing cheap, deterministic filters (like regex scoring) before engaging expensive LLM operations is crucial for cost-efficiency. Furthermore, filtering out irrelevant content improves LLM accuracy by preserving context, enhancing signal-to-noise ratio, and preventing distribution shift.
- Behavioral Similarity Detection is Key for Novelty: To avoid alert fatigue and redundant effort, a robust similarity service that compares the atomic behavioral conditions of new rules against existing ones is vital. This ensures that only truly novel and high-value detections are queued for analyst review.
- Human Oversight and Auditability are Non-Negotiable: Despite extensive automation, a human must remain at the "final gate" for review, editing, and approval. Additionally, the system must be "auditable by design," providing full traceability and evidence for every decision to build trust and facilitate debugging.
About the Speaker(s)
Andrew Skatoff is an Information Security Manager Sr at the Federal Reserve, bringing a wealth of experience from a career spanning several decades in security roles within large financial institutions. A graduate of VCU, his professional journey includes significant positions at organizations such as the Dominion Energy Trading Floor and Capital One. More recently, Skatoff had the privilege of building the National Threat Hunting Program at the Federal Reserve, a testament to his expertise in proactive security measures.
Beyond his primary role, Andrew is actively involved in the security community, serving as a Teaching Assistant (TA) for SANS classes, where he helps educate the next generation of cybersecurity professionals. In his personal time, he is an avid tinkerer with new AI tools, reflecting his passion for innovation and continuous learning in the rapidly evolving tech landscape. He also enjoys playing indie folk covers and spending time outdoors with his wife and four children. Skatoff delivered this talk in his personal capacity, and the views expressed are his own.
Reviews
Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT
Usually, 'AI-powered CTI' means some vendor wrapped an OpenAI key in a React front-end and called it a revolution. Skatoff actually engineered a pipeline that respects the constraints of LLMs by stripping the marketing garbage out first using deterministic logic. This is how you build defensive automation that doesn't hallucinate its way into a self-inflicted SOC meltdown.
Heather Calloway (CISO) — STRONG ACCEPT
A highly pragmatic blueprint for using LLMs to solve the signal-to-noise crisis in cyber threat intelligence. This isn't vendor AI vaporware; it's a concrete pipeline architecture that uses cheap deterministic filtering to gate expensive LLM extraction, ultimately accelerating detection velocity without removing the human from the loop.