CTI-Transmute: Harmonizing Threat Intelligence in a Multi-Standard Ecosystem

CTI-Transmute: Harmonizing Threat Intelligence in a Multi-Standard Ecosystem

Speakers: Alexandre Dulaunoy, Security Researcher (CIRCL); Christian Studer, Software Engineer (CIRCL)

Conference: FIRST CTI Conference

YouTube: https://www.youtube.com/watch?v=Ntg_iEepaZY

Overview

The landscape of Cyber Threat Intelligence (CTI) is rich with diverse data formats, standards, and tools, each serving specific purposes and communities. While this variety fosters innovation, it also creates significant interoperability challenges, often compelling CTI teams to dedicate substantial resources to format translations and data mappings. "CTI-Transmute: Harmonizing Threat Intelligence in a Multi-Standard Ecosystem," presented by Alexandre Dulaunoy and Christian Studer from CIRCL, directly addresses this pervasive problem. The talk introduces CTI-Transmute, an open-source conversion layer and online service designed to seamlessly bridge the gap between disparate CTI formats, primarily focusing on MISP (Malware Information Sharing Platform) and STIX (Structured Threat Information Expression).

The core of CTI-Transmute lies in abstracting the complexities of format-specific engineering away from CTI tool developers and analysts. By providing a robust, API-driven solution, it aims to reduce the "friction" associated with multi-standard environments, allowing security professionals to focus on intelligence analysis and action rather than the mechanics of data exchange. This initiative is particularly crucial in an era where rapid, accurate, and context-rich threat intelligence sharing is paramount for effective cybersecurity defense.

Dulaunoy and Studer, both deeply involved in CTI standardization as co-chairs of the CTI TC STIX at OASIS, bring extensive experience to this problem. Their work on CTI-Transmute stems from years of grappling with the practicalities of CTI integration, particularly within the MISP ecosystem. The tool is not merely a syntactic translator but a semantic interpreter, recognizing that true CTI conversion requires preserving the original intent and fidelity of the intelligence, a nuanced "craft" rather than a straightforward scientific mapping.

Background

▶ Watch: Introduction and CTI format diversity problem (0:00)

The evolution of CTI has led to a proliferation of structured formats, each tailored to specific operational needs or community focuses. Formats like MISP are designed for operational sharing and community contribution, emphasizing flexibility and extensibility. In contrast, STIX serves as a common language for CTI products and automation pipelines, providing a more formal and machine-readable structure. Beyond these, specialized formats like Sigma for detection rules and YARA for malware signatures add further layers of complexity. While structured data is a vast improvement over unstructured documents like PDFs, the sheer number and variations of these formats create significant interoperability challenges.

A primary issue is the constant need for CTI teams to perform format translations and mappings, a repetitive and often error-prone task. This problem is compounded by several factors:

• Multiple Versions: Even within a single standard like STIX, multiple versions exist (e.g., STIX 1, often XML-based, versus STIX 2.x, JSON-based), each with its own nuances.
• Custom Extensions: Vendors and organizations frequently implement custom extensions or unique interpretations of standard formats, further complicating generic conversions.
• Semantic Gap: Fields with identical names across different formats may carry subtly different semantic meanings, leading to data misinterpretation during one-to-one translations.
• Fidelity and Trust: Analysts creating intelligence expect their data to retain its original meaning and context throughout any conversion process. Loss of fidelity or misrepresentation erodes trust in the converted intelligence.
• Producer Laxity: Some data producers may be "lax or weak" in how they convey information within a format, focusing only on specific aspects relevant to their immediate use case, ignoring broader contextual elements.

The direct trigger for CTI-Transmute's development was a specific request from CERT EU for STIX support within CIRCL's AIL framework (a tool for analyzing unstructured information). AIL already supported the full MISP standard format, but its lead developer, Aurélien, expressed strong aversion to directly integrating STIX, famously stating he "didn't want to touch STIX, not even with a rusty button." This developer-centric frustration highlighted a broader problem: engineers developing CTI tools should not be burdened with maintaining and understanding five or six different CTI formats. This insight led to the realization that a dedicated conversion layer, external to the core tools, was necessary to streamline CTI integration and reduce the friction experienced by both developers and analysts.

Key Findings

▶ Watch: CTI Transmute: A live API for conversion (5:45)

The central finding and contribution of Dulaunoy and Studer's talk is the introduction of CTI-Transmute as a pragmatic and comprehensive solution to the persistent problem of CTI format interoperability. This project is presented as both an open-source software and an online service (available at cti-transmute.org), acting as a dedicated conversion layer to facilitate the seamless exchange of threat intelligence between different standards.

Key findings and contributions include:

1. Delegation of Format-Specific Engineering: CTI-Transmute successfully isolates the complex logic of CTI format handling into a specialized, maintainable service. This allows developers of CTI tools (like AIL) to integrate CTI standards without directly grappling with their intricacies, dramatically reducing development overhead and maintenance burden.
2. Robust Bidirectional Conversion (MISP <-> STIX): The initial and primary focus of CTI-Transmute is the bidirectional conversion between MISP and STIX. This is powered by the battle-tested MiSTICS library, which has been operational within MISP for over six years. The library is not a simple field mapper but incorporates a deep "semantic logic," evident in its approximately 35,000 lines of Python code.
3. Semantic Preservation over Literal Translation: A crucial insight presented is that CTI conversion is "more craft than science," rejecting the notion of a complete mathematical isomorphism. Instead, CTI-Transmute prioritizes preserving the "analyst intent" and the semantic fidelity of the data. This involves sophisticated interpretation logic to handle semantic gaps, custom extensions, and the nuanced differences between data models (e.g., MISP's object templates/Galaxies mapped to STIX objects, and the handling of MISP's to_IDS flag vs. STIX's separated data elements).
4. Addressing Interoperability Challenges: The tool directly tackles issues like multiple STIX versions (acknowledging the past challenges with STIX 1 XML and focusing on JSON for STIX 2.x), custom vendor extensions, and the need for data fidelity. It aims to support a repository of extensions to facilitate their integration.
5. API-First Design with User-Friendly Interface: CTI-Transmute is built with an API-first approach, ensuring easy programmatic integration. Complementing this is a user-friendly web UI that provides interactive conversion, diffing capabilities, and secure sharing options for converted documents (public or private links). This dual approach caters to both automated pipelines and manual analyst workflows.

In essence, CTI-Transmute provides a much-needed abstraction layer, making CTI data more fluid and accessible across a multi-standard ecosystem, thereby empowering CTI analysts and developers to work more efficiently and effectively.

Technical Deep Dive

▶ Watch: Semantic gap and fidelity in CTI conversion (7:30)

CTI-Transmute's architecture is designed for flexibility, extensibility, and ease of use, built upon an API-first philosophy. The core system comprises a service layer that handles incoming CTI data, processes it through specialized conversion engines, and outputs the transformed data.

The primary conversion engine is powered by the MiSTICS library, which serves as the default and most developed converter. This library is the culmination of years of work, providing robust bidirectional conversion between MISP and STIX (specifically STIX 2.x JSON, with a future goal to deprecate STIX 1 XML parsing due to security and maintenance concerns). The complexity of this conversion is underscored by the fact that the MiSTICS library contains approximately 35,000 lines of Python code dedicated to its semantic logic, far beyond a simple field-to-field mapping.

Key technical aspects of the conversion process and data handling include:

• Semantic Interpretation: The talk emphasizes that CTI conversion is not a "mathematical isomorphism" but a "craft." This means the library doesn't just translate syntax; it interprets semantics. For example, it maps MISP's rich object templates and Galaxies (a framework for sharing contextual information about threat actors, malware, etc.) to semantically correct STIX objects. It also intelligently handles the differences in observability flags: MISP uses a single to_IDS flag, while STIX separates this into two distinct data elements, both of which are correctly translated by MiSTICS.
• Custom Extensions and Fidelity: CTI-Transmute is designed to support custom extensions from both MISP and STIX. If MISP data includes custom extensions, they can be exported to STIX and then accurately re-imported back into MISP, preserving fidelity. This is critical as many vendors and organizations rely on extensions to capture specific intelligence attributes not covered by standard fields. The project aims to build a repository of these extensions to further facilitate integration.
• Lossless Conversion Challenges: Achieving "lossless" conversion is a significant challenge. The speakers acknowledge that formats may express concepts that others do not fully support. The MiSTICS library strives to minimize data loss by intelligently mapping concepts and preserving the "analyst intent" as much as possible, rather than simply dropping unsupported fields. The difficulty of assigning a "coverage percentage" to the conversion process highlights this complexity, especially given the varying and often limited usage of features like the STIX patterning language.
• API Design: The CTI-Transmute service exposes a straightforward REST API with introspection points, including a Swagger definition. This allows developers to easily discover API endpoints and automatically generate client modules for integration. Tools supporting either STIX or MISP export can quickly gain support for the other format by simply pointing to the CTI-Transmute API.
• Web UI Functionality: The online service (cti-transmute.org) provides a user-friendly web UI that encapsulates the API's capabilities. It allows for interactive conversion, where users can paste STIX or MISP JSON, view the converted output, and perform operations like:
• Locking and Sharing: Documents can be made public or private and shared via unique links, facilitating collaboration.
• Editing and Diffing: Users can edit converted documents directly and compare different revisions or versions of the same document, which is useful for tracking updates from external sources like CISA reports.
• Configurable Options: For STIX to MISP conversions, users can specify MISP-specific parameters like sharing mechanisms (as STIX bundles do not inherently contain this information) and adapt semantic interpretations for bundles.
• Integration with MISP Modules (Roadmap): A significant future direction is the integration of MISP modules. With over 400 modules for enrichment, expansion, import, and export, plugging these into CTI-Transmute would allow for advanced workflows. For instance, a STIX file containing an IP address could be converted to MISP, enriched with passive DNS information via a MISP module, and then converted back to STIX, effectively adding additional intelligence to the original payload during the translation process. This transforms CTI-Transmute from a mere translator into an intelligent processing layer.

By combining a robust, semantically aware conversion engine with an accessible API and web interface, CTI-Transmute aims to become a foundational component in the CTI ecosystem, reducing friction and enhancing the utility of threat intelligence across diverse platforms.

Demo / Proof of Concept

▶ Watch: Demo: Converting CISA STIX reports live (21:00)

The presentation effectively showcased CTI-Transmute's capabilities through demonstrations of its live online service and existing integrations. The primary demonstration centered around the web interface available at cti-transmute.org, which allows users to perform interactive, real-time conversions between STIX and MISP formats.

Users can paste a STIX input (e.g., JSON from a CISA report) into one pane of the web UI. Upon conversion, the corresponding MISP output is displayed in a separate pane, formatted for clarity. The interface highlights key features that enhance usability and collaboration:

• Interactive Conversion: The core functionality allows for immediate conversion, demonstrating the speed and effectiveness of the underlying MiSTICS library.
• Input/Output Views: Users can easily toggle between the STIX input and MISP output views, ensuring transparency in the conversion process.
• Sharing and Privacy: Converted documents can be made public or private, and unique shareable links are generated. This is particularly useful for collaborating with partners who might use different CTI standards or for public dissemination of intelligence.
• Version Comparison and Editing: The platform supports comparing different versions of the same document, an invaluable feature for tracking updates from external CTI feeds (e.g., CISA frequently updates its STIX files). Users can upload an updated file and instantly see the differences. The ability to edit documents directly within the UI further enhances its utility for analysts.

A concrete example highlighted was the use of CISA reports as a regular test case. CISA publishes high-quality STIX files, which serve as excellent real-world data for fine-tuning the conversion library. This practical application demonstrates CTI-Transmute's ability to ingest and process complex, operationally relevant intelligence.

Beyond the web UI, the speakers detailed two critical existing integrations:

1. Rule Set: This platform for managing detection rules (like YARA rules) now leverages CTI-Transmute. Any rule within Rule Set can be automatically converted to MISP and STIX formats via the CTI-Transmute API. This integration acts as a continuous test case for the service, ensuring scalability and operational readiness, and provides immediate value by making detection rules accessible across different CTI ecosystems.
2. AIL Framework: The initial pain point that spurred CTI-Transmute's development, AIL, now integrates the service. By calling the CTI-Transmute API, AIL users can seamlessly handle STIX and MISP outputs without the AIL developer needing to directly implement complex STIX parsing and generation logic. This validates the core premise of abstracting format-specific engineering.

The demonstrations and integrations collectively illustrate CTI-Transmute's effectiveness as a "box input output" solution, allowing tools that support one CTI format to immediately gain support for others, significantly reducing integration overhead and broadening intelligence reach.

Defensive Implications

▶ Watch: Roadmap: CTI enrichment with MISP modules (26:00)

CTI-Transmute offers profound defensive implications by fundamentally streamlining how security operations centers (SOCs), threat intelligence teams, and incident response units interact with and leverage threat intelligence. Its core value proposition directly addresses several pain points that hinder effective defense:

1. Reduced Operational Friction: The most immediate benefit for defenders is the drastic reduction in the "friction" associated with multi-standard CTI environments. Instead of spending valuable time on manual conversions, developing custom scripts, or troubleshooting format parsing errors, analysts can rely on CTI-Transmute for automated, semantically-aware translation. This frees up resources to focus on actual intelligence analysis, threat hunting, and defensive actions.
2. Enhanced Interoperability and Sharing: CTI-Transmute acts as a universal translator, enabling organizations to consume and produce threat intelligence in their preferred formats while seamlessly exchanging it with partners, vendors, and communities that use different standards (e.g., a MISP-centric SOC can easily ingest STIX bundles from a government agency like CISA). This broadens the scope of intelligence accessible to defenders, leading to more comprehensive threat awareness.
3. Improved Data Fidelity and Trust: By prioritizing semantic preservation over mere syntactic translation, CTI-Transmute ensures that the original intent and context of the intelligence are maintained during conversion. This fidelity is crucial for defenders, as misinterpretations of CTI can lead to ineffective defenses or, worse, misdirected efforts. The ability to handle custom extensions also means that organization-specific context is not lost in translation.
4. Accelerated Integration of New Tools: For security engineering teams, CTI-Transmute simplifies the integration of new CTI sources or tools. If a new detection engine or analysis platform outputs data in a different format, CTI-Transmute can serve as an intermediary, quickly bridging the gap without requiring extensive custom development or modification of core tools. The integration with Rule Set demonstrates how detection rules can be instantly made available in multiple formats, accelerating their deployment.
5. Future-Proofing CTI Infrastructure: The open-source nature and API-first design of CTI-Transmute provide a flexible foundation for CTI infrastructures. As new CTI standards emerge or existing ones evolve, the conversion logic can be updated within CTI-Transmute, insulating downstream tools from constant changes. This reduces the technical debt associated with maintaining complex CTI pipelines.
6. Automated Enrichment Capabilities (Roadmap): The planned integration of MISP modules for enrichment is a game-changer for defensive operations. Imagine receiving a basic STIX indicator of compromise (IOC) – say, an IP address. CTI-Transmute could convert it to MISP, automatically trigger a passive DNS lookup via a MISP module, and then convert the enriched data back to STIX, now containing additional contextual information. This automated enrichment process provides defenders with more actionable intelligence without manual intervention, speeding up triage and response times.
7. Standardization of Conversion Logic: By centralizing the conversion logic, CTI-Transmute helps standardize how different formats are interpreted and translated across an organization or even an entire community. This consistency reduces variability and potential errors that can arise from multiple teams implementing their own conversion scripts.

In summary, CTI-Transmute empowers defenders by removing technical barriers to intelligence sharing and consumption, allowing them to leverage the full spectrum of available threat data more efficiently and effectively to strengthen their security posture.

Key Takeaways

• CTI-Transmute is a critical solution for CTI interoperability: It directly addresses the long-standing challenges of disparate CTI formats like MISP and STIX, reducing the operational burden on security teams.
• It offers an open-source, API-first conversion layer: Built on the robust MiSTICS library, CTI-Transmute provides a flexible API and a user-friendly web interface for bidirectional, semantically-aware conversions between MISP and STIX.
• CTI conversion is a "craft," not just a science: The tool emphasizes semantic preservation and analyst intent over literal translation, employing sophisticated logic (35,000 lines of Python) to handle nuances like custom extensions, semantic gaps, and differing data models.
• CTI-Transmute streamlines intelligence sharing and integration: By abstracting format-specific engineering, it allows tools like AIL and Rule Set to easily ingest and produce CTI in multiple standards, enhancing collaboration and data flow within the CTI ecosystem.
• Future plans include advanced enrichment and broader format support: The roadmap envisions integrating MISP modules for automated data enrichment (e.g., passive DNS lookups) and expanding support to other formats like CSV, further increasing its utility for defenders.
• It empowers defenders to focus on analysis, not format wrangling: By automating complex conversions, CTI-Transmute enables CTI analysts and SOC teams to dedicate more time to critical threat analysis, hunting, and response, ultimately strengthening an organization's defensive posture.

About the Speaker(s)

Alexandre Dulaunoy is a Security Researcher at CIRCL (Computer Incident Response Center Luxembourg). He is known for frequently presenting new tools and solutions at conferences like FIRST CTI. Alexandre, along with Christian Studer, co-chairs the CTI TC STIX at OASIS, highlighting his deep involvement and expertise in CTI standardization. His work often focuses on developing practical tools to solve real-world CTI challenges.

Christian Studer is a Software Engineer at CIRCL, working alongside Alexandre Dulaunoy. Christian has dedicated almost 10 years to working on MISP, primarily focusing on its integrations and the conversion of information between MISP and other tools and platforms. His extensive experience with MISP and his role as co-chair of the CTI TC STIX at OASIS underscore his profound understanding of CTI formats and the complexities of interoperability. His contributions, particularly to the underlying MiSTICS library, are fundamental to the success of CTI-Transmute.