Machine learning for macOS malware detection
Biagio Montaruli (AI security researcher)
Objective for the We v4.0 (#OFTW) 2026 · Day 2 · Main Stage
Overview
This talk, presented by Biagio Montaruli, a senior AI security researcher at Dynatrace, delves into the critical and evolving landscape of macOS malware detection. Montaruli presents a dual approach: first, showcasing robust solutions leveraging classical machine learning techniques for enhanced detection, and second, introducing agentic AI as a novel paradigm for automated and explainable malware analysis. The presentation highlights the urgent need for sophisticated detection mechanisms, given the escalating threat to macOS, which is predicted to become a dominant operating system in enterprise environments.

Key moments
- 4:00 Gaps in macOS malware detection state-of-the-art
- 7:00 Extracting novel Mac-specific features for detection
- 10:00 Significant detection rate improvement using XGboost
- 12:00 Mac-specific features crucial for generalization (15% drop)
- 14:00 New research: Generative AI for macOS malware detection
- 16:00 MacSkill project: Designing an analyst-friendly AI agent
- 26:00 Future of LLMs in EDR solutions (Q&A)
Machine learning for macOS malware detection
Speakers: Biagio Montaruli, Senior AI security researcher, Dynatrace
Conference: Objective for the We
YouTube: https://www.youtube.com/watch?v=dkek3pB3dBI
Overview
This talk, presented by Biagio Montaruli, a senior AI security researcher at Dynatrace, delves into the critical and evolving landscape of macOS malware detection. Montaruli presents a dual approach: first, showcasing robust solutions leveraging classical machine learning techniques for enhanced detection, and second, introducing agentic AI as a novel paradigm for automated and explainable malware analysis. The presentation highlights the urgent need for sophisticated detection mechanisms, given the escalating threat to macOS, which is predicted to become a dominant operating system in enterprise environments.
Montaruli's research addresses significant gaps in prior work, particularly the lack of comprehensive datasets and the oversight of macOS-specific features in detection models. By developing a novel dataset and integrating unique macOS characteristics, his classical ML approach achieves substantial improvements in detection rates and generalization capabilities. The latter part of the talk unveils MacSkill, an innovative agentic AI project designed to bridge large language models (LLMs) with specialized security tools, offering a risk-based score and a fully explainable reasoning chain for analysts. This talk is crucial for security professionals, researchers, and developers seeking to understand and combat the growing sophistication of macOS-targeted threats through advanced AI methodologies.
Background
▶ Watch: Gaps in macOS malware detection state-of-the-art (4:00)
The motivation for this research stems from the alarming surge in macOS malware, a trend that has been consistently observed over recent years. Montaruli cites the Mulo Club Lab's 2025 report, which indicates a 67% increase in macOS malware variants and a 17% rise in stealers, impacting over 80 countries. This escalation is further compounded by projections from JF, predicting macOS to become one of the most widely adopted operating systems in enterprises by 2030. These statistics underscore a pressing demand for more robust and effective detection solutions, a gap that conventional methods often struggle to fill. Machine learning, with its capacity to automatically extract and learn malicious patterns from data, presents a compelling answer to this challenge.
Montaruli's journey into macOS malware detection during his PhD revealed several critical limitations in the existing state-of-the-art. Firstly, there was a notable lack of a comprehensive, state-of-the-art dataset. Previous research relied on datasets with a very limited number of samples, hindering the development of truly effective machine learning solutions. Secondly, prior detectors predominantly utilized generic features, often inspired by the Windows malware domain, while largely overlooking crucial macOS-specific features. These include vital indicators such as entitlements, certificates embedded within binaries, and common persistence techniques like Launch Agents often leveraged by macOS malware. These domain-specific attributes, as Montaruli later demonstrates, are semantically rich and crucial for accurate detection. Lastly, Montaruli observed a deficiency in evaluating the generalization capabilities of existing detectors, particularly their performance against temporally new samples or previously unseen malware families. This oversight is directly related to the critical concept of concept drift, where a model's performance degrades over time when confronted with evolving threats. Addressing these gaps formed the foundational objectives of his research.
Key Findings
▶ Watch: Significant detection rate improvement using XGboost (10:00)
Montaruli's research yielded significant findings across two distinct yet complementary fronts: classical machine learning for macOS malware detection and the nascent field of agentic AI in this domain.
For the classical machine learning approach, a primary contribution was the creation of a comprehensive dataset comprising over 40,000 samples, including both goodware and malware sourced from diverse origins like personal MacBooks, open-source GitHub applications, the Objective-C data set, MalwareBazaar, VirusShare, and VirusTotal. This extensive dataset served as the bedrock for training robust models. A pivotal innovation was the incorporation of novel macOS-specific features, such as macOS APIs, entitlements, persistence techniques, and the status of the certificate chain, alongside more generic features. These domain-specific features proved to be highly discriminative.
Utilizing tree-based machine learning models like XGBoost, known for their scalability and explainability, Montaruli's solution demonstrated superior performance. When compared against the state-of-the-art, the XGBoost model achieved a remarkable 16% increase in detection rate. Further analysis into feature importance revealed that macOS-specific attributes were among the top 10 most influential features. Specifically, the presence of a certificate and its validation by Apple were identified as the top two most important features in the multi-architecture benchmarking dataset, a finding that aligns logically with security analyst practices. A crucial experiment highlighted the indispensable role of these macOS-specific features: their removal led to a performance drop of more than 15%, unequivocally demonstrating their semantic richness and ability to generalize over new malware samples and families. Overall, Montaruli's solution outperformed the state-of-the-art by over 50% in detection rate and 37% in F1 score when evaluated on a dataset of temporally new samples. This groundbreaking research has been published in a paper available on ACM and arXiv, with the dataset and code openly released, and was presented at SSCS in Bangalore.
The second key finding introduces the concept of agentic AI for macOS malware detection through the MacSkill project. This represents a forward-looking approach to bridge large language models (LLMs) with specialized security analysis tools. An agent is defined as one or more LLMs connected to a set of tools that can be invoked to expand the LLM's knowledge. Montaruli outlines how a security analyst could prompt an LLM with a binary path, which then leverages specific tools to extract critical features like entitlements, signature information, security-sensitive APIs, and indicators of compromise (IOCs). This process follows a REACT paradigm (Reasoning and Acting) involving iterative steps of observing findings, planning tool calls, acting by executing tools, and updating hypotheses to arrive at a final verdict. MacSkill is designed with three core goals: supporting multiple LLM frameworks (OpenAI, local LLMs via Llama, and Apple Foundational Model API), being tightly coupled with macOS (leveraging native command-line tools and the mechi Python framework for feature extraction), and being analyst-friendly by providing a risk-based score and, crucially, a fully explainable reasoning chain. This project, slated for open-source release, represents a significant step towards more transparent and automated malware analysis.
Technical Deep Dive
▶ Watch: Mac-specific features crucial for generalization (15% drop) (12:00)
Montaruli's classical machine learning methodology for macOS malware detection follows a well-defined pipeline, starting with data collection and culminating in model deployment.
The initial and fundamental step was the creation of a comprehensive dataset. Recognizing the limitations of prior work, Montaruli assembled over 40,000 binary samples. This dataset was carefully balanced, comprising both goodware (collected from his personal MacBook, various applications, and open-source macOS projects on GitHub) and malware (sourced from reputable repositories such as the Objective-C data set, MalwareBazaar, VirusShare, and VirusTotal). VirusTotal, in particular, was instrumental for verifying the labels of the collected samples, ensuring high data quality.
Following data collection, the feature extraction phase converts raw binaries into a numeric representation suitable for machine learning models. A feature is defined as a numeric characteristic of the input sample. Montaruli's work innovated by combining generic features (e.g., total binary size, number of sections, number of segments, entropy)—often inspired by Windows malware analysis—with a crucial set of novel macOS-specific features. These domain-specific features include:
- macOS APIs: Identifying calls to specific macOS application programming interfaces.
- Entitlements: Parsing the permissions and capabilities declared by the application, which can indicate malicious intent (e.g., access to sensitive user data, system resources).
- Persistence Techniques: Detecting mechanisms like Launch Agents, which malware commonly uses to maintain execution across reboots.
- Certificate Chain Status: Analyzing the presence and validity of the code signing certificate, especially whether it has been validated by Apple. This is a strong indicator of legitimacy or compromise.
The rationale behind these specific features is that they capture the unique behavioral aspects and structural properties of macOS applications that differentiate goodware from malware more effectively than generic features alone.
The model training phase primarily focused on tree-based machine learning models, such as XGBoost, Random Forest, and Decision Trees. These models were chosen for several key advantages:
- Scalability: They can be efficiently trained and deployed, even in parallel.
- Explainability: A critical property in security, as these models allow for easy extraction of feature importance, enabling analysts to understand which features contributed most to a classification decision. This transparency is vital for trust and actionable intelligence.
Evaluation was rigorously conducted, with a strong emphasis on assessing the models' generalization capabilities against concept drift. For this, a new, balanced dataset of 9,000 temporally new samples was collected from VirusTotal. The primary metrics used were:
- True Positive Rate (TPR): The detection rate, typically calculated at a specific false positive rate (FPR), in this case, 1% FPR. This measures the model's ability to correctly identify malicious samples.
- F1 Score: A harmonic mean of precision and recall, providing a balanced measure that accounts for both detection accuracy and the reduction of false positives.
Finally, in the deployment phase, a new binary undergoes feature extraction, and this numeric representation is fed into the trained model, which then provides a malicious or benign verdict.
Montaruli's more recent work introduces MacSkill, an agentic AI solution built upon the REACT paradigm (Reasoning and Acting). This architecture aims to marry the interpretative power of LLMs with the analytical depth of specialized security tools.
The core of an agentic AI system is an LLM (or multiple LLMs) connected to a set of tools. These tools are crucial for expanding the LLM's knowledge beyond its training data, enabling it to interact with the real-world (in this case, a macOS binary).
MacSkill's tools are categorized into two main sets:
- macOS Native Tools: These leverage command-line utilities readily available on the macOS operating system, such as
codesignfor analyzing digital signatures. mechiPython Framework: This custom framework, based on Montaruli's prior research, is designed for efficient feature extraction specific to macOS binaries, including entitlements, security-sensitive APIs, and other indicators of compromise.
The REACT loop orchestrates the agent's operation:
- Observe: The agent reads current findings about the binary.
- Plan: Based on observations, the LLM devises a plan, deciding which tools to call.
- Act: The chosen tools are executed, and their results are obtained.
- Update: The LLM updates its internal knowledge and hypothesis based on the tool's output.
This iterative process continues until the agent can confidently arrive at a verdict.
MacSkill is designed to be highly flexible, supporting various LLM frameworks, including cloud-based solutions like OpenAI, local LLMs runnable via Llama, and critically, the Apple Foundational Model API, which allows for leveraging embedded, free-to-use models directly on the Mac, significantly reducing operational costs. The output of MacSkill is tailored for security analysts, providing not only a risk-based score but also a fully explainable reasoning chain, detailing why a particular verdict was reached (e.g., "this sample is malware because of these indicators"). This focus on explainability is paramount for analyst trust and effective incident response.
Demo / Proof of Concept
▶ Watch: MacSkill project: Designing an analyst-friendly AI agent (16:00)
While the talk did not include a live demonstration of the MacSkill agentic AI solution, Montaruli provided a detailed architectural overview and announced its imminent open-source release on GitHub. The conceptual framework, outlining the integration of large language models with specialized macOS analysis tools and a REACT-based reasoning loop, serves as a compelling proof of concept for how advanced AI can be applied to enhance and explain macOS malware analysis. The project's design goals, including multi-framework support, tight coupling with macOS native tools, and analyst-friendly outputs, clearly articulate its intended functionality and potential impact.
Defensive Implications
▶ Watch: Future of LLMs in EDR solutions (Q&A) (26:00)
Montaruli's research offers several critical implications for defenders tasked with securing macOS environments against evolving threats:
- Prioritize macOS-Specific Features for Detection: The most significant takeaway is the validated importance of macOS-specific features for robust malware detection. Defenders should ensure their security solutions actively analyze and prioritize indicators such as entitlements, the presence and Apple validation of certificate chains, and common persistence techniques (e.g., Launch Agents). These features were shown to be semantically rich and highly discriminative, leading to substantial improvements in detection rates and generalization capabilities. Relying solely on generic, Windows-inspired features is insufficient.
- Embrace Machine Learning for Automated Detection: Given the rapid increase in macOS malware variants, traditional signature-based detection is becoming increasingly inadequate. Defenders should integrate machine learning-based detectors that are trained on comprehensive, up-to-date datasets. Emphasis should be placed on models that demonstrate strong generalization capabilities against temporally new samples to combat concept drift and detect novel threats effectively. Tree-based models, as highlighted, offer a balance of performance and crucial explainability.
- Leverage Explainable AI for Analyst Empowerment: The explainability of ML models and the reasoning chains provided by agentic AI solutions like MacSkill are invaluable. Defenders should seek out and implement tools that not only provide a verdict but also articulate why that verdict was reached. This transparency empowers security analysts, improves their understanding of malware behavior, accelerates incident response, and builds trust in automated systems.
- Explore Agentic AI for Enhanced Analysis and Threat Hunting: The introduction of agentic AI presents a new frontier for automated malware analysis. Defenders can explore integrating such solutions to:
- Automate binary analysis: Reducing the manual effort required to dissect new or suspicious macOS applications.
- Accelerate threat hunting: Quickly extracting and correlating indicators of compromise (IOCs) from binaries.
- Gain deeper insights: By leveraging LLMs connected to specialized tools, analysts can gain richer, context-aware understanding of potential threats.
- Consider Cost-Effective LLM Deployment Strategies: For organizations concerned about the operational costs associated with cloud-based LLM APIs, Montaruli's work on MacSkill highlights viable alternatives. Utilizing local LLMs (e.g., via Llama) or leveraging the Apple Foundational Model API (which offers free, embedded models directly on macOS) can provide powerful AI capabilities while mitigating expenses. This democratizes access to advanced AI for security teams of all sizes.
- Contribute to and Adopt Open-Source Security AI: The open-sourcing of projects like MacSkill provides a valuable resource for the security community. Defenders should consider contributing to or adopting such open-source initiatives to collectively improve and evolve macOS security tooling. This collaborative approach fosters innovation and strengthens the overall defensive posture against common adversaries.
Key Takeaways
- Escalating macOS Threats: The significant increase in macOS malware variants and its growing enterprise adoption underscore an urgent need for advanced, automated detection solutions.
- Criticality of macOS-Specific Features: Incorporating domain-specific features like entitlements, certificate chain validation, and persistence mechanisms (e.g., Launch Agents) is paramount for effective and generalizable macOS malware detection, significantly outperforming generic features.
- Classical ML Excellence: Classical machine learning, particularly tree-based models like XGBoost, can achieve high detection rates (16% increase over state-of-the-art) and valuable explainability when trained on comprehensive, feature-rich datasets and rigorously evaluated for generalization.
- Agentic AI for Explainable Analysis: Agentic AI, which combines large language models with specialized security tools (following the REACT paradigm), offers a promising new approach for automated, in-depth, and fully explainable macOS malware analysis.
- MacSkill: An Open-Source AI Agent: The MacSkill project introduces an innovative, open-source AI skill designed to analyze macOS binaries, providing a risk-based score and a detailed reasoning chain while supporting multiple LLM frameworks, including cost-effective local and Apple Foundational Models.
- Actionable Intelligence for Defenders: Defenders should integrate domain-specific features into their detection strategies, explore AI-driven analysis tools for enhanced threat hunting, and prioritize solutions that offer explainable outputs to empower security analysts.
About the Speaker(s)
Biagio Montaruli is a Senior AI security researcher currently working at Dynatrace, based in Bari. He earned his PhD in AI security from URO, a research center in the south of France in Kodazour, in collaboration with SAP lab France. During his doctoral studies, Montaruli began his impactful work on machine learning for malware detection, the results of which form a core part of this presentation. He is also a former attendee and student scholar of Objective for the We, highlighting his deep connection to the macOS security community.
Reviews
Dr. Zero (Offensive Security Researcher) — SOLID
Competent applied ML work that fills a real gap in macOS malware research, but the 'agentic AI' half is vaporware dressed up as innovation. The classical ML contribution is legitimate — novel dataset, domain-specific features that actually matter, published and open-sourced. The MacSkill stuff is a GitHub repo announcement, not research.
Heather Calloway (CISO) — SOLID
Solid defender-focused research on improving macOS malware detection through ML feature engineering. The macOS-specific feature insight is real and useful. The agentic AI portion is too early-stage to evaluate seriously—it's a roadmap, not a result.