“How much does my CEO earn?” – Avoid Data Security Pitfalls in the Era of AI

Michael Landolt (Microsoft), Umberto Annino (Microsoft)

Swiss Cyber Storm 2024 · Day 1 · Panorama

Overview

In an era increasingly defined by artificial intelligence, the landscape of data security is undergoing a profound transformation. This talk, delivered by Michael Landolt and Umberto Annino of Microsoft at Swiss Cyber Storm, delves into the critical challenges of data protection and compliance in an AI-driven world, encapsulated by the provocative question: "How much does my CEO earn?" The seemingly innocuous query serves as a potent illustration of how generative AI (GenAI) tools, such as Microsoft Co-pilot, can inadvertently expose sensitive information if fundamental data governance principles are neglected.

Watch on YouTube

Visual summary for “How much does my CEO earn?” – Avoid Data Security Pitfalls in the Era of AI by Michael Landolt, Umberto Annino
Visual summary for “How much does my CEO earn?” – Avoid Data Security Pitfalls in the Era of AI by Michael Landolt, Umberto Annino

Key moments

  1. 0:40 AI and the CEO's salary: A data protection problem
  2. 3:59 Microsoft's 'Security is Job Number One' initiative
  3. 5:59 Microsoft's three core security principles explained
  4. 11:30 Co-pilot's 'Pandora's Box' for overshared data
  5. 12:50 Why data governance is now essential for AI
  6. 16:20 Microsoft Purview: Data security, classification, and DLP
  7. 21:00 Co-pilot respects data labels and user access
  8. 28:30 Q&A: Identity & Access Management with AI and Purview

“How much does my CEO earn?” – Avoid Data Security Pitfalls in the Era of AI

Speakers: Michael Landolt, Principal Program Manager, Microsoft; Umberto Annino, Senior Cloud Security Specialist, Microsoft

Conference: Swiss Cyber Storm

YouTube: https://www.youtube.com/watch?v=fhiD78MZdQQ

Overview

In an era increasingly defined by artificial intelligence, the landscape of data security is undergoing a profound transformation. This talk, delivered by Michael Landolt and Umberto Annino of Microsoft at Swiss Cyber Storm, delves into the critical challenges of data protection and compliance in an AI-driven world, encapsulated by the provocative question: "How much does my CEO earn?" The seemingly innocuous query serves as a potent illustration of how generative AI (GenAI) tools, such as Microsoft Co-pilot, can inadvertently expose sensitive information if fundamental data governance principles are neglected.

The presentation outlines Microsoft's own journey in bolstering its security posture—an initiative driven by a recognition of evolving cyber threats and the imperative to embed security deeply within its operations and product offerings. Building upon this foundation, the speakers highlight how AI acts as a powerful amplifier for existing data security vulnerabilities, transforming once-obscure data into readily accessible insights. The core message is clear: while AI offers immense capabilities, its safe and responsible adoption hinges on robust data governance and security controls.

This article explores the speakers' insights, detailing the shift in cybercriminal tactics, Microsoft's strategic response, and the specific tooling and methodologies, primarily leveraging Microsoft Purview, that organizations must adopt to navigate the complexities of data protection in the age of AI. It underscores that the "boring" work of data classification and access control is now more critical than ever, serving as the bedrock for preventing AI from becoming a Pandora's Box for confidential information.

Background

▶ Watch: AI and the CEO's salary: A data protection problem (0:40)

The talk opens by contextualizing the escalating threat landscape that necessitated a fundamental shift in Microsoft's approach to security. Michael Landolt highlights the stark realities presented in Microsoft's annual threat intelligence reports, which consistently show increasing numbers of ransomware attacks, stolen passwords, and denial-of-service incidents. A significant observation from their November report indicated a "significant shift in cyber criminals' tactics," driven by an ever-expanding volume of telemetry. Microsoft now monitors an astounding 78 trillion signals a day, a substantial increase from 65 trillion previously, and tracks over 1,500 threat groups, including 650 nation-state actors and 300 cybercriminal organizations. This dynamic and aggressive threat environment demanded an unparalleled response.

Microsoft's strategic pivot was encapsulated in CEO Satya Nadella's declaration of "Security Priority One." This wasn't merely a slogan but a deep-seated organizational commitment: every Microsoft employee now has security goals integrated into their performance system, impacting bonuses. The company has made an unprecedented investment, dedicating $20 billion USD over the next five years to security through its Secure Future Initiative. This initiative aims to make Microsoft more resilient and transparent about its security practices. Furthermore, the company has significantly expanded its security workforce, appointing 34,000 security engineers and engaging an additional 150,000 partners to enhance its security networks and cloud security.

This massive investment is guided by three core principles: Secure by Design, Secure by Default, and Secure Operations. "Secure by Design" dictates that security takes precedence over market demands in development decisions. "Secure by Default" ensures that all security capabilities are enabled out-of-the-box, shifting the burden from the CISO to justify enablement to justifying disablement with compensating controls. "Secure Operations" focuses on continuous monitoring of internal systems, understanding client environments, and anticipating future threat landscapes. Internally, Microsoft has aggressively tackled complexity by switching off 730,000 non-compliant applications and eliminating 5.75 million inactive tenants, unifying its tooling across multi-cloud (Azure, Google, AWS) and multi-platform (Android, Apple, Windows, Linux) environments. This internal transformation, leveraging automated threat intelligence and AI for security, positions Microsoft to offer these solutions to its customers.

Umberto Annino then elaborates on the critical problem that AI exacerbates: the long-standing neglect of data governance. For over two decades, he has evangelized the fundamental principle that "you don't even have to consider starting doing security if you don't know your assets." This includes knowing what data assets exist, where they are, and how critical they are. However, data governance has historically been perceived as a "boring" topic. The advent of AI tools like Co-pilot drastically changed this perception. When Microsoft provisioned Co-pilot to early pilot customers, their immediate queries were not about productivity but about discovering confidential information, such as "What's the salary of my CEO?" or "What's the most confidential stuff I shouldn't even be seeing?" Co-pilot, acting as a "cognitive-enabled search engine," effortlessly retrieved this data, exposing the widespread issue of over-shared and inadequately protected information. Customers' subsequent request—to make Co-pilot ignore certain data without fixing underlying access controls—underscored the urgency of revisiting data governance as a non-negotiable prerequisite for safe AI adoption.

Key Findings

▶ Watch: Microsoft's three core security principles explained (5:59)

The talk reveals several pivotal findings regarding data security in the age of AI, fundamentally altering the perception of data governance:

  1. AI as a Data Governance Amplifier: Generative AI tools, exemplified by Microsoft Co-pilot, fundamentally change how users interact with and access information. They act as "cognitive-enabled search engines," capable of synthesizing and retrieving data from vast, disparate sources. This capability, while powerful for productivity, also serves as a potent amplifier for pre-existing data governance weaknesses. If data is over-shared or lacks proper access controls, AI will make it trivially accessible.
  2. Immediate Exploitation of AI for Confidential Data Access: Early Co-pilot customers quickly leveraged the tool to query for sensitive information, such as executive salaries or highly confidential documents. This demonstrated that the human inclination to seek out privileged information remains, and AI significantly lowers the technical barrier to doing so.
  3. Customer Misconception: Blame the AI, Not the Governance: Initial customer reactions to Co-pilot exposing sensitive data were to ask Microsoft to "make Co-pilot ignore some data" rather than to address their own inadequate data access controls and governance policies. This highlights a critical misconception that AI itself is the problem, rather than a symptom-exacerbator of neglected data hygiene.
  4. Data Governance is No Longer "Boring": The direct and undeniable impact of AI on data accessibility has transformed data governance from a perceived administrative burden into an urgent, strategic imperative. The ability of AI to instantly surface confidential information provides a compelling, rational argument for investing in data classification, protection, and lifecycle management.
  5. Shared Responsibility Model for AI: Microsoft reiterates a shared responsibility model for AI adoption, analogous to cloud services. Microsoft provides the AI tools, platforms, and inherent safety measures (e.g., preventing prompts for illegal activities). However, customers retain full responsibility for governing the use of their data with these tools, ensuring proper access controls, classification, and compliance.
  6. Microsoft Purview as the Central Solution: Microsoft Purview is presented as the comprehensive portfolio of data security, data governance, and risk and compliance management tools designed to address these challenges. It aims to empower customers to understand, classify, protect, and manage their data effectively across multi-cloud and multi-platform environments.
  7. Co-pilot Respects Existing Controls: Crucially, Co-pilot acts as an "agent of your employee or user." This means Co-pilot cannot access more data than the user querying it can. It also respects sensitivity labels (e.g., "confidential"), automatically inheriting and applying them to any generated output (e.g., a summary or presentation created from sensitive sources). This underscores the vital importance of robust Identity and Access Management (IAM) and data classification policies already being in place.

Technical Deep Dive

▶ Watch: Why data governance is now essential for AI (12:50)

The technical core of the solution lies within Microsoft Purview, a comprehensive suite designed to address data security, data governance, and risk and compliance posture management. The speakers emphasize that many of these capabilities are already available to M365 customers, often without full awareness.

The Purview portfolio includes:

  • Information Protection (MIP): This is the foundational element, providing data inventory and classification services. It operates out-of-the-box with M365, automatically processing and classifying data within the environment. It can also integrate with other sources, including multi-cloud services (AWS, Google) and various Software-as-a-Service (SaaS) platforms, providing a unified view and reports on data classification. This capability allows organizations to understand what sensitive data they possess and where it resides, a crucial first step in any data governance strategy.
  • Data Loss Prevention (DLP): DLP policies are extended to prevent users from exfiltrating or sharing confidential and critical data with generative AI tools. This directly addresses concerns like developers pasting sensitive source code into public GenAI services. The concept is not new, but its scope must now explicitly include AI interactions to prevent data leakage.
  • Insider Risk Management: This feature monitors employee activity based on defined triggers. For instance, if an employee submits a resignation, their activity might be monitored for up to 30 days. Triggers can include suspicious behaviors like downloading large amounts of data, sending emails to personal accounts, saving files to USB drives, or printing extensive documents. Each suspicious activity contributes to a risk score for the individual.
  • Adaptive Protection: Directly linked to Insider Risk Management, Adaptive Protection allows organizations to automatically enforce security controls based on an employee's risk score. If a user's risk score exceeds a defined threshold, the system can automatically limit their data exfiltration capabilities, such as restricting attachment sending or preventing access to customer data. As the employee refrains from risky activities, their score decreases, and restrictions are lifted.
  • Data Map and Data Catalog: These tools provide an inventory for more structured data, extending classification and governance capabilities across different services and platforms. This helps organizations maintain a comprehensive understanding of their data landscape.
  • Co-pilot and Data Sensitivity: A critical technical aspect is how Co-pilot interacts with classified data. Co-pilot operates as the user's agent, meaning it cannot access any data that the user themselves is not authorized to see. This underscores the paramount importance of robust Identity and Access Management (IAM) and strong access controls. Furthermore, Co-pilot respects sensitivity labels applied to data. If a document is labeled "confidential," Co-pilot will not only respect that label when processing the data but will also automatically apply the "confidential" label and associated protections to any output it generates from that data (e.g., a summary, presentation, or report). This automatic inheritance of protection is a powerful mechanism for maintaining data integrity and compliance.
  • eDiscovery and Audit: All user activities, including interactions with Co-pilot and other GenAI tools, are logged and traceable within the system. This comprehensive auditing capability supports eDiscovery cases and provides accountability for data handling.
  • Communication Compliance: This feature allows organizations to filter and limit communication capabilities, such as restricting the use of swear words in internal or external chats, or preventing specific types of information sharing.
  • Data Lifecycle Management (DLM) and Records Management: These tools enable organizations to define retention and archiving policies for data, ensuring compliance with regulatory requirements and efficient data storage.

Beyond these data-centric tools, Microsoft also embeds safety measures into all its AI services. These inherent safety filters prevent AI models from generating harmful or inappropriate content, such as instructions for building dangerous devices. These are distinct from the data security tools but form an integral part of Microsoft's overall responsible AI strategy.

Regarding Identity and Access Management (IAM), the speakers reiterate Microsoft's commitment to Zero Trust principles. Co-pilot rigorously respects existing access controls, whether managed through Azure Active Directory (now Microsoft Entra ID) or other integrated IAM solutions. If an organization's IAM is adequately configured, Co-pilot will adhere to those permissions. Furthermore, Co-pilot for Security can leverage security observations and indicators to trigger automated actions, such as deactivating a compromised device or user, demonstrating the deep integration of AI into adaptive security responses.

The availability of these Purview features varies based on Microsoft 365 licensing. While Business Premium includes a foundational set of features, E3 subscriptions offer standard capabilities, and E5 subscriptions unlock advanced automation and sophisticated features like Insider Risk Management.

Demo / Proof of Concept

▶ Watch: Microsoft Purview: Data security, classification, and DLP (16:20)

While the talk did not feature a live demonstration or a specific proof of concept, the speakers extensively described the functionality and interaction of Microsoft Purview tools with generative AI, particularly Microsoft Co-pilot. Umberto Annino used the anecdote of early Co-pilot customers immediately searching for confidential information like "CEO's salary" to illustrate the inherent capability of AI to expose data governance weaknesses. He explained how Co-pilot acts as an agent of the user, respecting existing access controls and inheriting sensitivity labels, which conceptually demonstrates its operational behavior without a direct visual walkthrough. The discussion focused on the theoretical and practical application of these tools in a real-world enterprise context rather than a step-by-step technical showcase.

Defensive Implications

▶ Watch: Q&A: Identity & Access Management with AI and Purview (28:30)

The insights from this talk carry significant defensive implications for organizations grappling with the secure adoption of AI:

  1. Prioritize Data Governance Immediately: The most critical takeaway is that robust data governance is no longer optional or "boring"; it is a prerequisite for safely leveraging AI. Organizations must invest in understanding, classifying, and protecting their data before widespread AI deployment.
  2. Implement Comprehensive Data Inventory and Classification: Utilize tools like Microsoft Purview Information Protection to gain a clear inventory of all data assets, their locations (across multi-cloud, multi-platform, SaaS), and their sensitivity levels. This forms the bedrock for all subsequent security controls.
  3. Extend Data Loss Prevention (DLP) to AI Interactions: Update and extend existing DLP policies to explicitly cover interactions with generative AI tools. This prevents sensitive data (e.g., intellectual property, customer PII, source code) from being inadvertently or maliciously pasted into AI prompts, which could lead to exfiltration or compromise.
  4. Strengthen Identity and Access Management (IAM): Since AI tools like Co-pilot respect existing user permissions, a strong Zero Trust-based IAM strategy is paramount. Ensure that users only have access to the data they absolutely need to perform their job functions (least privilege). Regularly audit and review access rights.
  5. Leverage Insider Risk Management and Adaptive Protection: Implement solutions like Microsoft Purview Insider Risk Management to monitor for anomalous employee behavior, especially around data exfiltration, and combine this with Adaptive Protection to automatically enforce dynamic restrictions when risk thresholds are met. This is crucial for mitigating insider threats, which are exacerbated by AI's ability to quickly aggregate sensitive data.
  6. Embrace Sensitivity Labeling and Inheritance: Actively apply sensitivity labels to data. Crucially, leverage AI tools that respect these labels and automatically inherit them when generating new content from sensitive sources. This ensures that generated summaries or presentations maintain the same level of protection as their source material.
  7. Monitor and Audit AI Interactions: Implement comprehensive auditing for all user interactions with AI tools. This provides traceability for eDiscovery, compliance, and incident response, allowing security teams to understand how AI is being used and to detect potential misuse.
  8. Educate Users on Responsible AI Use: Develop and enforce clear policies for the responsible use of generative AI. Educate employees about the risks of sharing sensitive data with AI, the importance of respecting data classifications, and their role in maintaining data security.
  9. Consider Licensing for Advanced Capabilities: Understand that advanced data security and governance capabilities (e.g., advanced automation, Insider Risk Management) may require higher-tier licenses (e.g., Microsoft 365 E5). Plan licensing strategies to ensure access to the necessary tools.
  10. Align with the Shared Responsibility Model: Recognize that while AI providers offer platforms and inherent safety features, the ultimate responsibility for data governance, classification, and secure usage lies with the customer.

Key Takeaways

  • AI amplifies existing data governance weaknesses: Generative AI tools act as "cognitive search engines," making previously hard-to-find confidential data trivially accessible if not properly protected.
  • Data governance is now a critical, non-negotiable priority: The era of AI has transformed data governance from a "boring" administrative task into an urgent, strategic imperative for enterprise security.
  • Microsoft's "Security Priority One" sets a new industry benchmark: Microsoft's massive internal investment ($20B over 5 years), "Secure by Design/Default" principles, and security goals for all employees demonstrate the organizational commitment required.
  • The Shared Responsibility Model applies to AI: While AI providers offer tools and inherent safety measures, organizations are fully responsible for governing their data's use with these tools, including access controls and classification.
  • Microsoft Purview offers a comprehensive suite for data security: Tools like Information Protection, Data Loss Prevention, and Insider Risk Management within Purview are essential for inventorying, classifying, protecting, and monitoring data in an AI-driven environment.
  • Robust IAM and data classification are paramount: AI tools like Co-pilot respect existing access controls and sensitivity labels, meaning the effectiveness of AI security hinges on foundational Identity and Access Management and accurate data classification.

About the Speaker(s)

Michael Landolt is a Principal Program Manager at Microsoft. His background includes significant experience as a CISO, giving him a deep understanding of the challenges security professionals face in real-world enterprise environments. This experience informs his current role, where he contributes to Microsoft's security strategy and initiatives, including the "Secure Future Initiative."

Umberto Annino is a Senior Cloud Security Specialist at Microsoft. With over 20 years of experience in cyber security and information security consultancy, he describes himself as a "security evangelist." His long career has been dedicated to advocating for fundamental security principles, particularly the importance of knowing and classifying data assets, a message that has gained renewed urgency in the age of AI. He joined Microsoft two years prior to this talk, bringing his extensive expertise to the company's cloud security offerings.

Reviews

Dr. Zero (Offensive Security Researcher) — HARD PASS

This is a 45-minute commercial for Microsoft Purview and M365 E5 licenses masquerading as an AI security talk. The underlying premise—that LLMs expose your broken enterprise access controls—is true, but it doesn't take a whole slot to tell people to fix their SharePoint ACLs.

Heather Calloway (CISO) — SOLID

This is a Microsoft vendor pitch that accidentally tells the most important truth about enterprise AI. I would skip the Purview commercial, but the core premise that Copilot weaponizes your existing bad data permissions is mandatory reading for any executive pushing a rapid GenAI rollout.

→ Top-rated talks at Swiss Cyber Storm 2024

All talks from Swiss Cyber Storm 2024