Threat Modelling in Modern Systems

Threat Modelling in Modern Systems

Speakers: Donavan Cheah, Senior Cybersecurity Consultant, Cybersecurity MNC

Conference: DefCamp

YouTube: https://www.youtube.com/watch?v=JB4GqxF0zIA

Overview

In his rapid-fire "master class" at DefCamp, Donavan Cheah, a Senior Cybersecurity Consultant, delivered an insightful exploration of threat modeling in an increasingly complex technological landscape. The talk, titled "Threat Modelling in Modern Systems," provided a whirlwind tour of various methodologies, frameworks, and their application to contemporary challenges such as cloud environments and artificial intelligence (AI) systems. Cheah's presentation underscored the critical importance of proactive security measures, framing threat modeling not merely as a technical exercise but as a fundamental component of a holistic security program that informs and complements other security activities.

Cheah emphasized that threat modeling serves as a vital preventative measure, allowing organizations to identify potential vulnerabilities and attack vectors before they can be exploited. He distilled the essence of threat modeling into four core questions: "What are we working on?", "What can go wrong?", "What are we going to do about it?", and "Whether or not we do a good enough job." This foundational approach is crucial for bridging the gap between technical security concerns and broader business objectives, enabling effective communication with diverse stakeholders, including non-technical board members.

The talk is particularly relevant in today's rapidly evolving threat landscape, where traditional systems are being supplanted or augmented by cloud-native architectures and sophisticated AI applications. Cheah skillfully navigated the nuances of adapting established threat modeling frameworks like STRIDE and MITRE ATT&CK to these modern contexts, while also introducing more comprehensive methodologies like PASTA (Process for Attack Simulation and Threat Analysis). His insights into the probabilistic nature of AI systems and the inherent challenges they pose to deterministic threat modeling highlight the ongoing need for innovation and adaptation in cybersecurity practices.

Background

▶ Watch: Introduction and talk expectations (0:00)

Threat modeling, at its core, is a structured approach to identifying potential threats, vulnerabilities, and countermeasure requirements within a system or application. Donavan Cheah adopted the OWASP definition, describing it as "a structured representation of all the information that we can get about a system that affects its security." The fundamental rationale behind threat modeling is that "prevention is better than cure." It's an indispensable activity that informs and strengthens various other security functions, from Governance, Risk, and Compliance (GRC) to Red Teaming and Threat Hunting, by providing a clear understanding of potential threat scenarios.

A central theme Cheah addressed early on is the common misconception that a single, universally "best" threat modeling framework exists. He unequivocally stated, "there's no such thing" as one framework to rule them all. Instead, he presented a landscape of diverse frameworks, each designed to address specific pain points or perspectives:

• STRIDE: A developer-focused model from Microsoft, focusing on threats that cause a loss of cybersecurity properties (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege). STRIDE LM extends this with lateral movement.
• LINDDUN: Geared towards privacy concerns.
• MITRE ATT&CK: Concentrates on Tactics, Techniques, and Procedures (TTPs), mapping attacker behaviors across the attack chain (e.g., reconnaissance, initial access, execution).

Cheah referenced the FIRST (Forum of Incident Response and Security Teams) special interest group on threat modeling, which evaluated the applicability of various models across different industries. Their finding was that "all threat models are generally useful to at least some degree," but trade-offs exist, particularly in ease of use. STRIDE, for instance, is relatively easy to implement, while PASTA is "much more involved." Singapore's cybersecurity agency, as an example of a practical approach, combined STRIDE for high-level property loss identification with MITRE ATT&CK to detail how attackers achieve their objectives, demonstrating that mixing and matching frameworks can yield more comprehensive outcomes.

Key Findings

▶ Watch: No single best threat modeling framework (4:00)

Donavan Cheah's talk highlighted several pivotal findings crucial for understanding and implementing effective threat modeling in contemporary environments:

1. No Universal Framework: There is no single "best" threat modeling framework. Organizations must select and combine methodologies (e.g., STRIDE, MITRE ATT&CK, PASTA, LINDDUN) based on their specific system, context, and desired outcomes. This hybrid approach allows for addressing diverse concerns from developer-centric threats to TTPs and privacy.
2. Cloud Introduces Unique Challenges: Cloud environments fundamentally alter threat landscapes. Considerations like scalability, the shared responsibility model, and the amplified financial impact of attacks (e.g., a Denial of Service (DoS) leading to hefty bills) necessitate distinct threat modeling approaches, often leveraging resources like the CSA EGES 11 and MITRE Cloud Matrix.
3. AI Systems Are a Paradigm Shift: Threat modeling for AI systems (especially Large Language Models - LLMs) presents fundamental differences. Unlike traditional deterministic systems, AI often produces probabilistic outcomes, making it harder to predict and mitigate all threats. New attack vectors like LLM jailbreaking, data poisoning, and the inherent layers of abstraction require specialized frameworks like MITRE ATLAS and an understanding of OWASP Top 10 for GenAI.
4. AI Tools Augment, Not Replace, Human Expertise: While AI-powered tools (e.g., Threat Canvas) can assist with brainstorming and initial DFD generation, they are limited in identifying novel attack vectors or adapting to unusual system architectures. They elevate the "floor of competence" but cannot reach the "ceiling of competence" required for complex, bespoke systems, necessitating human refinement and contextualization.
5. PASTA Bridges Technical and Business Risk: The PASTA framework is particularly valuable for translating technical threats into business-relevant risks. Its seven-stage process forces a comprehensive, stakeholder-inclusive view, enabling effective communication with non-technical audiences and ensuring that security efforts align with business objectives and compliance requirements.
6. Threat Modeling Extends Beyond Security: In the context of AI, threat modeling practices are increasingly applicable to privacy and safety experts, not just security professionals. The "wrong" in an AI system can encompass ethical, societal, and functional failures beyond traditional cybersecurity breaches.
7. Early Integration and Continuous Adaptation are Key: Threat modeling should be initiated "as fast as possible" in the development lifecycle, ideally during design. It's an ongoing process that must adapt to evolving threats and system changes, leveraging frameworks like MITRE D3FEND to link identified TTPs to concrete defensive controls.

Technical Deep Dive

▶ Watch: LLMs for threat modeling: capabilities and limits (28:20)

Donavan Cheah's master class meticulously dissected various threat modeling frameworks and their applications, starting with foundational concepts and progressing to the complexities of modern systems.

The bedrock of many introductory threat modeling exercises is STRIDE, an acronym for Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, and Elevation of Privilege. This framework, originating from Microsoft, categorizes threats based on the cybersecurity properties they violate. For instance, "Spoofing" targets authentication, while "Denial of Service" impacts availability. Cheah noted that STRIDE LM expands this to include lateral movement, a crucial aspect in modern network attacks.

Complementing STRIDE, Cheah extensively featured MITRE ATT&CK, which focuses on Tactics, Techniques, and Procedures (TTPs). Unlike STRIDE's property-loss orientation, MITRE ATT&CK maps out the entire kill chain, detailing how attackers achieve their objectives, from reconnaissance and initial access to execution and command and control. This TTP-oriented view is particularly useful for red teams and threat hunters, providing concrete actions and observable artifacts. An example table presented showed how threats could be color-coded by attacker type (insider/external), action, location, and method, allowing for rapid identification of crucial mitigation details.

The discussion then shifted to the intricacies of cloud environments. Cheah highlighted that cloud introduces unique challenges compared to on-premise setups, such as scalability, the shared responsibility model, and the potential for greater financial impact from attacks (e.g., a DoS attack on a cloud environment leading to significant billing charges). He pointed to the Cloud Security Alliance's (CSA) EGES 11 (Eleven Top Pain Points) as a valuable resource for identifying cloud-specific issues. MITRE also offers specialized matrices for cloud, including IaaS and SaaS variants. Cheah encouraged using public resources like Azure's reference architectures for practical threat modeling exercises.

Effective threat modeling must culminate in risk management. Cheah outlined the four strategies for dealing with risk: Mitigate, Eliminate, Transfer, or Accept. He stressed the importance of documenting all identified threats and associated risks in a risk register. To bridge the gap between identified threats and practical defenses, Cheah introduced MITRE D3FEND. This framework, less mature than ATT&CK but highly useful, links ATT&CK TTPs to specific defensive controls. D3FEND categorizes controls into sections like isolate, harden, and protect, and uses a digital artifact ontology (e.g., network telemetry, event logs) to show how a TTP (like SMB Relay) can be mitigated by various countermeasures and what artifacts to look for.

A significant portion of the deep dive was dedicated to the PASTA (Process for Attack Simulation and Threat Analysis) framework. PASTA is a seven-stage, risk-centric methodology designed to simulate attacks and analyze threats, particularly in application environments. Its key strength lies in its ability to translate technical security details into business-relevant terms, making it ideal for communicating with diverse stakeholders. The seven stages are:

1. Define Objectives: Understand business objectives, security requirements, conduct Business Impact Assessments (BIAs), and define compliance requirements.
2. Define Technical Scope: Identify application boundaries, deployment environments, and crucial application dependencies (the "ingredient list" of the system).
3. Decompose Application: This traditional threat modeling step involves drawing Data Flow Diagrams (DFDs), identifying trust boundaries, users, and roles.
4. Analyze Threats: Feed in other frameworks like STRIDE and MITRE ATT&CK to identify attack scenarios, analyze incident/fraud cases, and ensure detectability through application logs and security events.
5. Analyze Vulnerabilities: Map identified threats to vulnerabilities, recognizing that mappings are not always one-to-one. This stage involves using use and abuse cases (e.g., persona non grata concept) and enumerating/scoring vulnerabilities (e.g., using CVSS or proprietary matrices).
6. Model Attacks: Engage pentesters to validate threats by identifying the attack surface and TTPs. This can involve attack trees to visualize complex attack paths.
7. Risk & Impact Analysis: Quantify risks, identify gaps in security controls, and prepare reports for board-level communication.

Finally, Cheah tackled the "unholy topic" of threat modeling AI systems. He acknowledged the common question: "Can GPT or LLMs generate threat models?" His answer was nuanced: they are useful for brainstorming and elevating the "floor of competence" by providing standard threats. However, they lack the "ceiling of competence" for novel attack vectors or unusual system architectures, often producing probabilistic outcomes rather than deterministic ones. He cited Threat Canvas as an example of an LLM-augmented tool for DFD generation, emphasizing that human refinement of AI-generated models is crucial.

AI systems introduce fundamental challenges:

• Probabilistic Outcomes: Unlike traditional systems with deterministic outputs, AI responses can vary, complicating predictable threat mitigation.
• Abstraction Layers: The complexity of AI models introduces new layers where threats can manifest.
• Novel Threats: Concepts like LLM jailbreaking, data poisoning, and model inversion are unique to AI. Cheah referenced the OWASP Top 10 for GenAI and MITRE ATLAS (TTPs for AI/ML systems) as essential resources.

He also discussed Project Guardrail, an approach where design-level threat modeling translates into questionnaires and best practices for developers. This "social engineering" encourages adherence to secure design patterns when full threat modeling isn't feasible for every change. Cheah concluded this section by presenting Ken's layered threat model for AI, which examines threats across foundational models, data operations, and cross-layer AI kill chains, highlighting the evolving nature of this field.

Demo / Proof of Concept

▶ Watch: Unique challenges of threat modeling AI systems (32:00)

The talk did not feature a live, interactive demonstration or proof of concept in the traditional sense of executing an exploit or showcasing a tool in real-time. Donavan Cheah's presentation was structured as a "master class," focusing on a high-level conceptual overview of numerous frameworks and methodologies rather than a hands-on technical demo.

However, Cheah did reference specific tools and resources that serve as practical examples for threat modeling. He mentioned Threat Canvas as an example of an LLM-augmented threat modeling tool capable of generating Data Flow Diagrams (DFDs) from text descriptions. He encouraged the audience to explore such tools independently, stating, "If you try to explore this for yourself, you can just go to Thread Canvas and you try to prompt the LLM there and you can see what kind of the FDC generates. I will leave this to you as an exercise." He also highlighted the utility of Azure's website and AWS for their comprehensive reference architectures, suggesting these as excellent starting points for individuals to practice drawing DFDs and identifying potential threats. While not a direct demonstration, these references served as practical pointers for attendees to engage with the concepts discussed.

Defensive Implications

▶ Watch: Q&A: How often to threat model & prioritize (47:00)

The insights shared by Donavan Cheah offer several critical defensive implications for cybersecurity practitioners navigating modern, complex systems:

1. Adopt Hybrid Threat Modeling Frameworks: Defenders should move beyond relying on a single framework. Combining methodologies like STRIDE (for high-level property loss) with MITRE ATT&CK (for TTPs) provides a more comprehensive view of potential threats. For structured, business-aligned risk assessment, PASTA is an invaluable framework, especially for communicating with non-technical stakeholders.
2. Integrate Threat Modeling Early and Continuously: Threat modeling should not be a late-stage activity. It must be integrated "as fast as possible" into the Software Development Life Cycle (SDLC), ideally during the design phase. This proactive approach allows for architectural and design changes that are far less costly than addressing issues in production. The frequency of threat modeling should also be dynamic, adapting to organizational size, maturity, and the rate of system changes.
3. Acknowledge and Address Cloud-Specific Risks: Cloud environments introduce unique security considerations. Defenders must be familiar with resources like the CSA EGES 11 and MITRE Cloud Matrix to identify and mitigate threats specific to cloud infrastructure, platforms, and software-as-a-service deployments. The financial implications of cloud attacks, particularly DoS, must be factored into risk assessments.
4. Leverage MITRE D3FEND for Control Implementation: To operationalize threat intelligence, security teams should use MITRE D3FEND to map identified ATT&CK TTPs to concrete defensive controls (e.g., isolate, harden, protect). This provides a structured way to implement mitigations and identify the necessary digital artifacts for detection and response.
5. Develop Specialized Threat Modeling for AI Systems: The probabilistic nature and inherent abstraction layers of AI systems demand a new approach. Defenders must familiarize themselves with emerging threat categories (e.g., OWASP Top 10 for GenAI, MITRE ATLAS for AI/ML TTPs) and consider that AI threat modeling extends to privacy and safety concerns, not just traditional security. Expect that some AI-specific threats (like certain prompt injections) may be impossible to mitigate completely.
6. Use AI Tools Wisely: While AI can aid in brainstorming and initial DFD generation, defenders must exercise critical judgment. AI tools elevate the "floor of competence" but cannot replace human expertise for novel architectures or complex, contextualized threats. Human review and refinement of AI-generated threat models are essential.
7. Bridge the Gap Between Security and Development Teams: Addressing the common challenge of disconnect between security engineers and developers is paramount. Threat modelers should act as facilitators, using collaborative techniques (e.g., workshops with post-it notes, gamification, time-boxing) to engage developers. The act of creating Data Flow Diagrams (DFDs) can itself foster mutual understanding of system components and interactions.
8. Prioritize Relevant Threats: Given resource constraints, not all identified threats can be addressed simultaneously. Defenders should prioritize threats based on their relevance to the organization, considering factors like targeted Advanced Persistent Threat (APT) groups, industry-specific attack trends, and known organizational weaknesses. Avoid simply copying exhaustive lists from generic sources.

Key Takeaways

• Threat modeling is far more expansive than just STRIDE: While STRIDE is a good starting point, a comprehensive approach requires integrating multiple frameworks like MITRE ATT&CK, PASTA, and LINDDUN, tailored to specific organizational and system needs.
• Cloud systems introduce distinct threat modeling considerations: Scalability, the shared responsibility model, and unique financial impacts necessitate specialized approaches, often guided by resources like CSA EGES 11 and MITRE Cloud Matrix.
• PASTA is crucial for business-aligned risk communication: Its seven-stage framework enables security professionals to translate complex technical threats into understandable business risks, facilitating effective communication with diverse stakeholders, including the board.
• AI tools can augment, but not replace, human threat modeling expertise: LLMs are valuable for brainstorming and initial threat identification, but human analysts remain indispensable for identifying novel attack vectors, addressing unique architectures, and refining probabilistic AI-generated outputs.
• AI systems fundamentally challenge traditional threat modeling paradigms: The probabilistic nature of AI, coupled with emerging threats like LLM jailbreaking and data poisoning, expands threat modeling to include privacy and safety concerns, requiring new frameworks like MITRE ATLAS and a re-evaluation of what constitutes "good enough" security.
• Effective threat modeling requires early integration, continuous adaptation, and collaborative effort: It must be embedded early in the SDLC, adapted for evolving threats, and fostered through strong collaboration between security and development teams to ensure practical, relevant, and timely risk mitigation.

About the Speaker(s)

Donavan Cheah is a Senior Cybersecurity Consultant at Thales, a global technology leader, where he works from the Singapore office. His professional experience includes a focus on securing complex AI systems, a topic he touched upon extensively in his DefCamp presentation. Beyond his corporate role, Donavan is deeply involved in the cybersecurity community. He co-leads the Threat Modeling Connect chapter in Singapore, an initiative dedicated to bringing together threat modeling practitioners from around the world to foster a community of practice. Additionally, he is a team member of Set Crack, a group of four individuals dedicated to researching cybersecurity within satellite systems. Donavan's appearance at DefCamp marked his first time speaking in Europe, an honor he expressed during his talk.