OpenTide: From Raw Intelligence to Structured Threat-Informed Detections

OpenTide: From Raw Intelligence to Structured Threat-Informed Detections

Speakers: Amine Besson (European Commission CSOC), Rémi Séguy (Lead of Threat Detection Engineering, European Commission CSOC)

Conference: FIRST CTI Conference

YouTube: https://www.youtube.com/watch?v=_fCcBK0bnIM

Overview

In the rapidly evolving landscape of cyber threats, the ability to translate raw threat intelligence into actionable, deployable detections is paramount for any security operations center (SOC). However, a significant and ever-widening gap exists between the continuous influx of threat intelligence and the capacity of detection engineering teams to operationalize it. This challenge forms the core problem addressed by OpenTide, an innovative open-source framework developed by Rémi Séguy, Lead of Threat Detection Engineering, and Amine Besson, from the detection engineering team at the European Commission Cyber Security Operations Center (CSOC).

OpenTide is engineered to bridge this critical operational gap, providing a structured, scalable, and platform-agnostic methodology for converting disparate threat intelligence into high-confidence detection rules. Born from years of practical experience within a demanding SOC environment, the framework emphasizes a schema-driven, Git-native, and multi-platform approach. This talk not only introduces OpenTide's foundational principles and architecture but also unveils AgentTide, a groundbreaking application of autonomous coding agents to automate the complex process of transforming intelligence reports into structured detection artifacts, dramatically reducing the manual toil previously required.

Background

▶ Watch: The widening gap between threat intel and detection (0:44)

The problem facing modern SOCs is multifaceted: threat intelligence (TI) production is accelerating at an unprecedented rate, while detection engineering struggles to keep pace. This creates an "operational gap" where organizations know more about attacks but lack the means to detect them effectively. Rémi Séguy highlighted four dimensions to this acceleration:

1. Volume: A continuous, ever-increasing stream of inputs, now augmented by AI findings, ensures that new intelligence never ceases.
2. Speed: The window between a vulnerability's disclosure and its active exploitation has shrunk dramatically, often from weeks to mere hours, demanding immediate defensive actions.
3. Complexity: Attacks are becoming more sophisticated, spanning multiple stages and involving intricate, iterative actions within each stage.
4. Team Capacity: Skilled detection engineers, capable of analyzing raw TI, understanding attack behaviors, defining detection objectives, writing quality rules, validating, tuning, and safely deploying them, are a scarce resource.

At the heart of this bottleneck is the manual CTI analysis process. Detection engineers, often aided by threat analysts, must meticulously read through reports, comprehend attacker operations, extract Tactics, Techniques, and Procedures (TTPs), and then determine "what and how to detect." This labor-intensive process can consume hours, even for relatively simple TTPs, hindering an organization's ability to achieve detection at scale.

While frameworks like MITRE ATT&CK are immensely valuable for classifying, describing, and sharing information about attacks, they often lack the granularity and context required for effective detection engineering. Séguy emphasized that relying solely on ATT&CK techniques is insufficient because:

• Lack of Granularity: A technique (e.g., "Remote Code Execution") does not specify the exact actions or procedures attackers take. Detection engineers need to detect specific actions within their unique context, not just broad categories.
• Insufficient Context: ATT&CK techniques provide limited information about severity, attack surface, or relevance to a specific environment, making prioritization challenging.
• Limited Chaining: While ATT&CK Flow offers improvements by mapping attack paths, it remains too high-level to directly drive granular detection engineering. Complex attacks require expressing sequences of behaviors.
• Inflexibility: SOCs need models that can be easily annotated and refined to fit their specific operational context and infrastructure.

OpenTide emerged as a direct response to these challenges, seeking to provide the necessary structure, automation, and contextualization to bridge the gap between intelligence and detection.

Key Findings

▶ Watch: OpenTide's three core objects: vector, objective, rule (9:10)

OpenTide represents a significant advancement in operationalizing threat intelligence into detection engineering. Its core contributions and key findings include:

1. Structured, Schema-Driven Intelligence to Detection Workflow: OpenTide provides a clear, top-down workflow that transforms raw threat intelligence into deployable detection rules through a series of structured objects: Threat Vectors, Detection Objectives, and Detection Rules. This structured approach, enforced by large JSON schemas, ensures validation, consistency, and a standardized language across the detection lifecycle.

1. Granular Behavioral Modeling Beyond ATT&CK: The framework introduces Threat Vectors to capture atomic threat contexts and Detection Objectives that crystallize specific detection goals, which are further broken down into signals (atomic detection ideas). This allows for a level of detail far below MITRE ATT&CK techniques, focusing on specific attacker behaviors and procedures crucial for high-fidelity detections, and enabling faster integration of novel TTPs.

1. Git-Native Operationalization: By embedding the entire detection lifecycle within a Git-native environment, OpenTide leverages familiar DevOps principles. This enables robust collaboration through branching, pull requests, and peer reviews, alongside inherent change tracking, auditing, and rollback capabilities, significantly improving the quality and maintainability of detection logic.

1. Multi-Platform and Multi-Tenancy Agnosticism: OpenTide is designed to be vendor-neutral, supporting deployment across diverse security platforms (e.g., Sentinel, Defender, Splunk, CrowdStrike) and even multiple instances of the same solution. This flexibility ensures that detection logic is portable and not locked into proprietary ecosystems.

1. Comprehensive Detection Coverage and Data Inventory Mapping: The framework integrates a Threat Surface vocabulary (e.g., VPN, OpenVPN) and asset management to contextualize threats and detections to an organization's specific infrastructure. This allows for a precise understanding of detection coverage, identifying gaps where data is unavailable or rules are not yet implemented for specific threats or assets.

1. Successful Integration of Autonomous Coding Agents (AgentTide): A pivotal finding is the ability to leverage autonomous coding agents to automate the generation of OpenTide's structured objects directly from unstructured intelligence reports. This significantly reduces the manual effort and time required, with AgentTide demonstrating the capability to generate high-quality threat vectors, detection objectives, and rules "as good or better compared to humans," even exhibiting self-correction and end-to-end generation capabilities.

Technical Deep Dive

▶ Watch: OpenTide's practical workflow from intel to detection (12:08)

OpenTide's architectural strength lies in its schema-driven design and Git-native implementation, which together provide a robust foundation for detection engineering at scale. The entire framework is built around three core objects: Threat Vector, Detection Objective, and Detection Rule, each playing a distinct role in the journey from raw intelligence to deployed detection.

The design is rooted in the principle of schema-driven development. Large JSON schemas underpin every object within OpenTide, ensuring structural validation, consistency, and a shared language across the detection engineering team. This not only standardizes the way threat intelligence is documented and interpreted but also facilitates automation and reduces errors.

Furthermore, OpenTide is inherently Git-native. This means that the entire lifecycle – from drafting new intelligence artifacts to deploying detection rules – leverages standard Git workflows. Detection engineers can work on independent branches, submit pull requests for peer review, track every change, and easily roll back to previous versions. This brings the rigor and collaborative benefits of modern software development to detection engineering, including built-in auditing and traceability.

The Core Objects: Threat Vector, Detection Objective, and Detection Rule

1. Threat Vector: This is an atomic container for any piece of threat context. The recommendation is to keep Threat Vectors atomic and chain them together to form complex attack paths, providing a more granular representation than high-level ATT&CK techniques. While OpenTide maps to MITRE ATT&CK for classification, it allows for much lower-level descriptions of specific procedures and behaviors. This flexibility enables SOCs to integrate novel TTPs much faster than ATT&CK's update cycles, maintaining agility against emerging threats. Each Threat Vector can include:

• A link to the Threat Surface vocabulary, which offers a hierarchical and granular classification of infrastructure components (e.g., "VPN" as a general surface, "OpenVPN" as a specific implementation).
• Chaining capabilities to link to other Threat Vectors, modeling attack flows.
• Additional metadata and detailed descriptions.

1. Detection Objective: This is where detection engineers "crystallize" what they want to detect. A Detection Objective bundles a set of related threats or behaviors and defines the desired detection capability. Crucially, each objective contains one or more signals. Signals are atomic detection ideas, independent of specific rule implementations. They serve two primary purposes:

• Data Requirements: OpenTide includes a data requirements framework that allows engineers to document whether the necessary log sources and data are available to detect a particular signal. This helps identify data gaps before rule development.
• Spec-Driven Development: Signals define the "what" of detection, enabling a specification-driven approach where the detection idea is clearly articulated before being translated into specific rules. Signals can also map to external detectors, such as built-in EDR capabilities, even if their internal logic is a "black box."

1. Detection Rule: This object represents the actual detection as code. OpenTide provides a system to plug in any platform-specific deployer and schema, making it highly adaptable. It supports a wide array of deployment targets, including Microsoft Sentinel, Microsoft Defender for Endpoint (MDE), Splunk, CrowdStrike, and Arfong Lab. This ensures that the detection logic can be deployed across a diverse security ecosystem without vendor lock-in.

The OpenTide Data Graph and Workflow

The interplay of these objects forms a comprehensive data graph. From a given Threat Vector, OpenTide can map to a specific Threat Surface and then to an asset within the organization's configuration. This detailed mapping allows the framework to answer critical questions about detection coverage: "For this specific threat vector targeting this asset on this surface, do we have a detection objective? Are there signals defined? Have we implemented a rule yet? Do we even have the necessary data?"

The practical workflow unfolds as follows:

1. Intelligence Ingestion: Raw threat intelligence is received.
2. Threat Vector Creation: Threat Vectors are either created or updated based on the intelligence, documenting specific behaviors and their context.
3. Detection Objective Definition: Detection engineers define Objectives, translating threat behaviors into concrete detection goals and outlining the necessary signals.
4. Detection Rule Development: Specific detection rules are written to implement the defined signals.
5. CI/CD Deployment: The entire process is integrated into a Continuous Integration/Continuous Deployment (CI/CD) pipeline. Changes to rules are pushed to a Git repository, triggering a pipeline that validates the code, deploys to staging environments, and eventually, after peer review and merge, to production. This pipeline can incorporate guardrails, additional validations, and business processes, ensuring safe and controlled deployment.

This layered approach, where Threat Vectors inform Objectives, and Objectives inform Rules, is crucial because "threats move too quick." Decoupling detection objectives from rapidly changing threat procedures provides a more stable layer for detection engineering, allowing for a few robust objectives to cover many transient threat procedures.

Demo / Proof of Concept

▶ Watch: OpenTide's human-centric design enables AI integration (21:00)

A significant portion of the talk was dedicated to AgentTide, a demonstration of how OpenTide's design principles facilitate the integration of cutting-edge AI for automation. The speakers acknowledged that when OpenTide was initially conceived, AI was not a primary consideration. However, the framework's foundation in detection as code and DevOps/CI/CD made it uniquely positioned to leverage the power of autonomous coding agents.

The core hypothesis behind AgentTide was whether coding agents could effectively transform unstructured intelligence reports into OpenTide's structured objects (Threat Vectors, Detection Objectives, Detection Rules) without the traditional human toil – reducing the process from hours to perhaps 15 minutes. The "TLDR" (Too Long; Didn't Read) answer presented was a resounding "yes."

The architecture of AgentTide involves a coding agent (e.g., based on models like Copilot, Claude, or a custom harness) that ingests "very loose world of intelligence" from reports, PDFs, or blog posts. This agent then processes the information and outputs structured OpenTide objects. A key component demonstrated was the agents.md file, an open-sourced system prompt that effectively instructs the AI agent on how to operate within the constraints and schemas of the OpenTide framework. This prompt acts as the operational guide, teaching the agent the necessary structure and workflow.

The results, publicly available on the ShareTide repository, showed that the generated objects were "as good or better compared to humans." The "better" aspect was attributed to the agents' tireless nature, as human engineers can only model so much in a day before fatigue sets in. Furthermore, the team observed "emerging behaviors" where the agent began to self-correct and steer its output towards the desired quality and structure, even generating end-to-end objects (threat vectors, objectives, and rules) when initially only tasked with threat vector generation.

The speakers clarified that these are not simple chatbots but autonomous coding agents designed to work for several hours, operating on a user's computer, accessing terminals, and leveraging internet access. This capability means the shape of the input intelligence (PDF, blog post, or even a loose article with links to crawl) becomes less critical; what matters is completeness. The agents can even incorporate ad hoc context or organizational knowledge defined in simple markdown files, making the generated detections highly relevant to the specific environment.

The success of AgentTide underscores a critical point: OpenTide was designed for people first. Its schema-driven approach provided a "quality of life" improvement for human engineers, and its top-down workflow (Intel -> Rules, or "tight flow") mirrored natural human thought processes. These design choices, combined with the inherent Git-native benefits of rollback, auditing, and traceability, inadvertently created the perfect environment for seamless AI integration.

Defensive Implications

▶ Watch: Q&A: Advanced NLP for TTP extraction accuracy (23:40)

OpenTide offers profound implications for defensive security operations, empowering SOCs to move beyond reactive measures towards more proactive, threat-informed detection strategies.

1. Accelerated Threat Intelligence Operationalization: By providing a structured framework and leveraging AI through AgentTide, organizations can drastically reduce the time it takes to convert raw threat intelligence into deployable detections. This speed is critical in an era where the time between disclosure and exploitation is measured in hours.
2. Enhanced Detection Quality and Consistency: The schema-driven approach ensures that all detection artifacts adhere to a consistent structure and quality standard. This reduces ambiguity and improves the reliability of detections. AgentTide further enhances this by producing high-quality, consistent outputs, mitigating human error and fatigue.
3. Granular and Contextualized Coverage: OpenTide’s emphasis on Threat Vectors and signals allows defenders to model specific attacker behaviors rather than broad techniques. Coupled with the Threat Surface vocabulary and asset mapping, SOCs gain a highly contextualized view of their detection coverage, understanding precisely which threats are covered on which assets and where gaps exist. This enables more informed prioritization of detection development efforts, focusing on "crown jewels" or critical attack surfaces.
4. Vendor Agnostic and Future-Proof Detection Engineering: The framework's multi-platform support frees organizations from vendor lock-in, allowing them to deploy detection logic across diverse security tools. This flexibility ensures that investments in detection engineering are portable and adaptable to evolving technology stacks.
5. Improved Collaboration and Auditing: The Git-native nature of OpenTide fosters a collaborative environment, enabling detection engineers to work together efficiently through pull requests and peer reviews. Furthermore, every change is tracked and auditable, providing a clear history of detection logic evolution and facilitating compliance requirements.
6. Reduced Manual Toil and Empowered Engineers: Automating the initial parsing and structuring of threat intelligence with AgentTide frees detection engineers from repetitive, time-consuming tasks. This allows them to focus on higher-value activities such as complex behavioral analysis, threat hunting, and refining detection logic, ultimately increasing job satisfaction and team efficiency.
7. Future-Ready Skill Development: The upcoming skills framework, particularly for languages like Kusto Query Language (KQL), promises to standardize and elevate the quality of detection rule development. By distilling patterns and anti-patterns from thousands of existing queries, this framework will equip engineers with validated, high-quality detection logic, transferable across various query languages.

Key Takeaways

• The widening gap between the accelerating volume and speed of threat intelligence and the capacity for detection engineering is a critical operational challenge for SOCs.
• OpenTide provides a structured, schema-driven, and Git-native framework that transforms raw threat intelligence into actionable, deployable detection rules.
• The framework introduces Threat Vectors, Detection Objectives (containing signals), and Detection Rules as core objects, enabling granular behavioral modeling beyond high-level ATT&CK techniques.
• AgentTide successfully leverages autonomous coding agents to automate the process of converting unstructured intelligence reports into OpenTide's structured objects, significantly reducing manual effort and improving consistency.
• OpenTide supports multi-platform and multi-tenancy deployments, ensuring vendor neutrality and adaptability, while also providing comprehensive detection coverage mapping against specific threat surfaces and assets.
• The framework's design, initially for human users, proved to be an ideal foundation for seamless integration with AI, highlighting the benefits of well-structured data and workflows for advanced automation.

About the Speaker(s)

Rémi Séguy is the Lead of Threat Detection Engineering at the European Commission Cyber Security Operations Center (CSOC). With extensive experience in the operational aspects of cybersecurity, Rémi has been instrumental in identifying the critical bottlenecks in the threat intelligence-to-detection lifecycle and leading the development of OpenTide to address these challenges. His work focuses on building scalable and effective detection capabilities for complex environments.

Amine Besson is a key member of the detection engineering team at the European Commission CSOC, working closely with Rémi Séguy. Amine plays a crucial role in maintaining and evolving the OpenTide framework, including the cutting-edge AgentTide project. His expertise contributes to the practical implementation and continuous improvement of the tools and processes that transform raw intelligence into structured, actionable detections. Together, their combined experience within a demanding operational environment has shaped OpenTide into a robust and highly practical solution for modern SOCs.