Better Together? – on the Resilience of Human-technology Teams

Verena Zimmermann (Professor · ETH Zurich)

Swiss Cyber Storm 2025 · Day 1 · Scenario

Overview

In a compelling presentation at Swiss Cyber Storm, Professor Verena Zimmermann from ETH Zurich explored the critical question of how human and artificial intelligence (AI) teams can collaborate effectively, particularly within the high-stakes domain of cybersecurity. As the leader of the Security Privacy and Society group at ETH, Professor Zimmermann’s work focuses on human-centered cybersecurity, a field that delves into the pervasive human factors influencing security, extending beyond mere phishing training to encompass developers and cybersecurity professionals themselves. Her mission, as she articulated, is to transform "human risk into resilience" by enhancing the synergy between humans and technology.

Watch on YouTube · Slides

Visual summary for Better Together? – on the Resilience of Human-technology Teams by Verena Zimmermann
Visual summary for Better Together? – on the Resilience of Human-technology Teams by Verena Zimmermann

Key moments

  1. 0:00 Introduction: Human-centered cybersecurity and AI collaboration
  2. 4:15 Paradoxical finding: Human-AI teams often perform worse
  3. 5:15 First insight: Identify tasks where humans excel for AI augmentation
  4. 9:20 Understanding 'Calibrated Trust' in human-AI interaction
  5. 14:00 Study results: Transparency decreased trust, but reliance persisted
  6. 16:20 Why this paradox? Trust builds like with a new employee
  7. 18:50 Key recommendations for resilient human-technology teams
  8. 25:00 Q&A: Keeping human in loop even when AI performs better

Better Together? – on the Resilience of Human-technology Teams

Speakers: Verena Zimmermann, Professor, ETH Zurich

Conference: Swiss Cyber Storm

YouTube: https://www.youtube.com/watch?v=jgL2vriVg2M

Overview

In a compelling presentation at Swiss Cyber Storm, Professor Verena Zimmermann from ETH Zurich explored the critical question of how human and artificial intelligence (AI) teams can collaborate effectively, particularly within the high-stakes domain of cybersecurity. As the leader of the Security Privacy and Society group at ETH, Professor Zimmermann’s work focuses on human-centered cybersecurity, a field that delves into the pervasive human factors influencing security, extending beyond mere phishing training to encompass developers and cybersecurity professionals themselves. Her mission, as she articulated, is to transform "human risk into resilience" by enhancing the synergy between humans and technology.

The talk centered on the specific example of cybersecurity experts collaborating with AI tools, a timely and pertinent topic given the persistent workforce gap and the escalating complexity of the cyber threat landscape. While the intuitive appeal of leveraging AI to augment human capabilities and alleviate workload is strong, Professor Zimmermann’s research, along with a broader systematic review, reveals a nuanced and often paradoxical reality. The presentation meticulously unpacked the factors that dictate whether such partnerships truly lead to improved performance, emphasizing the delicate balance required to design effective and resilient human-technology teams.

Ultimately, the research presented challenged conventional assumptions about AI's role, demonstrating that simply adding AI support does not automatically lead to better outcomes. Instead, success hinges on a sophisticated understanding of task allocation, the design of explainable AI (XAI), and the intricate process of building calibrated trust over time. This article delves into the core insights, technical underpinnings, and practical implications of Professor Zimmermann's findings, offering a roadmap for organizations striving to optimize human-AI collaboration in cybersecurity.

Background

▶ Watch: Introduction: Human-centered cybersecurity and AI collaboration (0:00)

The premise for investigating human-AI collaboration in cybersecurity stems from a clear industry challenge: cybersecurity is an increasingly complex, technology-driven field demanding deep domain-specific knowledge, yet the pool of skilled cybersecurity professionals is a scarce resource. This workforce gap naturally leads to the exploration of advanced technologies like AI to support, relieve, or even potentially replace human experts, thereby decreasing workload and ideally enhancing overall team performance. Headlines often tout AI's superiority – "better at everything," "faster, cheaper, safer" – suggesting a straightforward path to augmentation or automation.

However, the reality is far more intricate. Professor Zimmermann cited a systematic review of over 100 experimental studies on human-AI combinations, which presented a surprising and paradoxical finding: the combined performance of humans and AI was often worse than the best individual performance of either humans or AI alone. This counter-intuitive result necessitated a deeper look. The review revealed a critical distinction: when humans outperformed AI in specific tasks, integrating AI as a support tool led to performance gains, effectively augmenting human expertise. Conversely, when AI already outperformed humans, their collaboration often resulted in performance losses, suggesting that human involvement could hinder the AI's superior capabilities.

This foundational insight led to the first crucial principle for successful human-AI teams: the necessity to pick the right tasks. AI excels at tasks requiring speed, precision, repetitiveness, and data-driven decisions. Humans, on the other hand, are superior in areas demanding creativity, content creation, emotional and social intelligence, strategy, experience, contextualization of information, and the ability to handle uncertainty and adapt flexibly to unforeseen circumstances. Therefore, effective collaboration is not about replacing humans with AI, but rather about intelligence augmentation – strategically applying AI to bolster human strengths in specific, well-defined contexts. This understanding forms the bedrock upon which the subsequent research by Professor Zimmermann's team was built, exploring how to design this collaboration for success in a high-stakes environment like cybersecurity.

Key Findings

▶ Watch: First insight: Identify tasks where humans excel for AI augmentation (5:15)

Professor Zimmermann's team, spearheaded by researcher Anala, embarked on a multi-study approach to uncover the factors critical for successful human-AI collaboration. Their journey began with an interview study involving cybersecurity professionals, aiming to understand their current work environment, the potential and limitations of AI integration, and their requirements. Professionals expressed a clear desire for AI to improve workflows, particularly by relieving them of extensive or repetitive tasks, and even assisting with more discretionary duties. However, a significant caveat emerged: the AI autonomy level they would grant to AI tools depended heavily on a complex risk-benefit analysis.

This analysis, the interviews revealed, was influenced by two primary factors: the task characteristics (e.g., capability fit, urgency, criticality) and trust in the AI system. Trust emerged as a recurring theme, with professionals stating that higher trust directly correlated with a willingness to grant higher autonomy levels to AI. Furthermore, trust itself was found to be largely dependent on transparency – the ability for professionals to understand "what's going on and how that AI model comes to a certain decision."

The research underscored the importance of calibrated trust, meaning trust that accurately aligns with the AI's actual capabilities. Over-trust, where professionals trust the tool more than its true ability, can lead to dangerous outcomes and over-reliance. Conversely, under-trust results in the AI system being underutilized or disused, rendering it unhelpful. It was also crucial to differentiate between trust (an attitudinal component, self-reported belief) and reliance (a behavioral component, the extent to which one follows AI recommendations).

To empirically test the role of transparency, the team conducted an online experimental study with over 100 cybersecurity professionals, focusing on the critical task of malicious domain blocking. They designed two dashboards: one incorporating explainable AI (XAI) to enhance transparency, and one without. The study measured trust, reliance, performance, and task load. The results were surprising and counter-intuitive:

  1. Decreased Trust with Transparency: In the transparent condition, trust in the AI system decreased after interaction. This indicated that simply providing explanations did not automatically foster appropriate trust.
  2. Persistent Over-Reliance: Despite the decrease in reported trust, professionals in the transparent condition still relied heavily on the AI's recommendations, exhibiting a trend towards over-reliance. This revealed a disconnect between attitude (trust) and behavior (reliance).
  3. No Performance or Task Load Improvement: The provided explanations did not improve overall performance in recognizing false AI classifications, nor did they significantly increase task load.

These unexpected findings led to a crucial working hypothesis: transparency is necessary, but its effectiveness in building trust depends critically on the alignment between the AI model and the expert's mental model. If the AI's reasoning aligns with the expert's established understanding of relevant features and their importance, trust might increase. If there's a mismatch, trust could decrease. This suggests that the quality and relevance of the explanation, rather than just its presence, are paramount.

Building on these insights, the team is currently working on a final study, recognizing that trust is not a one-off assessment but a process built over time and with experience. Experts often approach AI like a new employee, starting with low-stakes tasks and gradually granting more responsibility as confidence grows. The new system being developed will feature adaptive AI autonomy levels, allowing experts to adjust the AI's involvement based on their comfort level and the specific task, fostering gradual trust building and tailored collaboration.

Technical Deep Dive

▶ Watch: Study results: Transparency decreased trust, but reliance persisted (14:00)

The technical core of Professor Zimmermann's research lies in understanding and designing the interaction between human experts and AI systems, particularly through the lens of AI autonomy and transparency. The talk outlined a clear hierarchy of AI autonomy levels, illustrating the spectrum of control and decision-making responsibility:

  1. Manual: Human performs all tasks without AI assistance.
  2. Information Acquisition: AI assists by gathering and presenting information.
  3. Analysis/Rating: AI not only acquires but also analyzes and rates information.
  4. Recommendation (Human Approval): AI provides a specific recommendation, but human approval is required for execution.
  5. Autonomous (Human Veto): AI takes a decision that is executed unless a human vetoes it within a specified timeframe.
  6. Fully Autonomous: AI acts completely independently without human oversight.

The research posited that the appropriate AI autonomy level is determined by a risk-benefit tradeoff, which considers two main axes: task characteristics (e.g., capability fit between human and AI, urgency, criticality) and trust in the AI system. Trust, in turn, is heavily influenced by transparency.

To investigate transparency's role, the team developed an experimental setup for the task of malicious domain blocking. This involved designing two distinct dashboards: one incorporating explainable AI (XAI) and another serving as a non-transparent control. The XAI dashboard was structured into four key components:

  • Part A: AI Decision: Clearly stated the AI's classification (e.g., "block" or "allow" a domain).
  • Part B: Explanation (XAI): This was the core transparency component. It explained how the AI model arrived at its decision by detailing which features of the URL (e.g., domain age, presence of certain keywords, TLD reputation) were considered, how they were weighted, and their relative importance to the overall classification. This aimed to provide insight into the AI's internal logic.
  • Part C: Descriptive Feature List: A comprehensive list of all features the AI model used as a basis for its decision or recommendation, providing raw data for the professional's review.
  • Part D: Human Decision Task: The interface where the cybersecurity professional made their final decision to block or allow the domain, based on the provided information.

The study employed a between-subject design with over 100 cybersecurity professionals, meaning one group interacted with the transparent dashboard and the other with the non-transparent version. The specific type of XAI provided was feature-based, showing feature importance and weighting – a common form of local interpretability for machine learning models.

The surprising finding that transparency decreased trust but increased reliance led to the hypothesis that the match between the AI model and the expert's mental model is the crucial factor. Experts, through years of experience, develop an intuitive understanding of what constitutes a malicious domain and which features are most indicative. If the AI's explanations align with this established mental model, trust may increase. If the AI highlights features or weightings that contradict the expert's intuition, it can lead to confusion, skepticism, and a decrease in trust, even if the AI's decision is technically correct. This highlights a critical challenge for XAI: it's not enough to simply show the inner workings; the explanations must be interpretable and align with human domain expertise to build genuine, calibrated trust.

The ongoing research addresses this by developing an AI system with adaptive autonomy levels. This system would allow experts to dynamically adjust the AI's role, starting with lower-risk functions like information acquisition and gradually increasing to recommendations or even veto-based autonomy as trust and understanding are built through repeated interaction and proven performance. This reflects a more human-centric design approach, mirroring how humans build trust with new team members.

Demo / Proof of Concept

▶ Watch: Why this paradox? Trust builds like with a new employee (16:20)

While the talk did not feature a live, product-style demonstration of a deployable tool, the research itself was built around a sophisticated proof-of-concept system designed for experimental purposes. This system took the form of two distinct dashboards tailored for the task of malicious domain blocking.

These dashboards served as the core instrument for the online study, allowing Professor Zimmermann's team to rigorously test the impact of explainable AI (XAI) on human-AI collaboration. The transparent dashboard, described in the "Technical Deep Dive" section, presented AI-generated decisions alongside detailed explanations. These explanations elucidated how the AI arrived at its conclusion by highlighting specific URL features (e.g., domain age, suspicious keywords) and their respective weights in the decision-making process. The non-transparent dashboard, conversely, presented only the AI's final decision without this explanatory context.

The design of these dashboards, complete with their four key parts (AI decision, explanation, feature list, and human decision prompt), effectively created a controlled environment to simulate real-world cybersecurity tasks. This allowed the researchers to measure critical variables like trust, reliance, performance, and task load under different conditions of AI transparency. In essence, the dashboards functioned as a robust experimental platform, a proof of concept for investigating human-AI interaction dynamics rather than a direct product demo. The insights gained from this meticulously crafted experimental setup directly inform the ongoing development of AI systems with adaptive autonomy, aiming for more effective and trustworthy human-AI partnerships in practice.

Defensive Implications

▶ Watch: Q&A: Keeping human in loop even when AI performs better (25:00)

The findings presented by Professor Zimmermann carry profound implications for cybersecurity defenders and organizations seeking to integrate AI into their operations. The research moves beyond the simplistic notion of AI as a universal panacea, instead offering a nuanced framework for strategic implementation:

  1. Strategic Task Allocation for Intelligence Augmentation: Defenders must meticulously identify tasks where AI truly augments human intelligence, rather than aiming for full automation where human strengths are paramount. AI should handle repetitive, high-volume, data-driven tasks like initial log analysis, threat intelligence correlation, and identifying known attack patterns. Humans, conversely, should focus on strategic decision-making, contextualizing incidents, adapting to novel threats, and exercising creativity in incident response. Misallocating tasks can lead to performance degradation, as demonstrated by the studies where AI-superior tasks saw human involvement decrease overall effectiveness.
  1. Cultivating Calibrated Trust in AI Tools: The concept of calibrated trust is critical. Security teams must avoid both over-reliance (blindly trusting AI, potentially missing false negatives or novel attacks) and under-reliance (disregarding AI recommendations, leading to missed efficiencies). This requires comprehensive training for security professionals on the specific capabilities, limitations, and potential biases of their AI tools. Regular validation of AI outputs against human expertise is essential to build and maintain appropriate trust levels.
  1. Demanding Effective Explainable AI (XAI): When procuring or developing AI solutions, organizations should prioritize XAI that provides meaningful and actionable explanations. It's not enough for an AI to simply state its decision; it must articulate why in a way that resonates with the expert's existing mental model. If the XAI's explanations contradict or fail to align with the security analyst's understanding of a threat, it can erode trust. Therefore, XAI outputs should be designed to be interpretable and justifiable within the context of established cybersecurity principles and expert intuition.
  1. Phased Integration with Adaptive Autonomy: The research suggests a phased approach to AI integration, particularly for critical tasks. Organizations should consider AI tools that offer adaptive autonomy levels. Instead of immediately deploying fully autonomous systems, defenders can start with AI functioning at lower autonomy levels, such as providing information or recommendations. This allows security professionals to gradually build experience and trust, akin to mentoring a new team member. As confidence in the AI's performance and alignment with expert judgment grows, autonomy can be incrementally increased, ensuring human oversight remains robust until trust is fully earned.
  1. Maintaining Human Oversight for Resilience: Even in scenarios where AI demonstrably outperforms humans, the talk emphasized the enduring necessity of keeping the "human in the loop." Cybersecurity is a high-stakes, constantly evolving domain. Humans possess unique capabilities for adapting to unforeseen circumstances, handling ambiguity, and responding to zero-day exploits that current AI models may not yet comprehend. Maintaining human oversight provides a crucial layer of resilience, ensuring that a human expert can intervene, override, or take over in cases of AI failure, novel attacks, or ambiguous situations, thereby preventing potentially catastrophic outcomes in critical infrastructure protection.

By adopting these defensive implications, organizations can move beyond simplistic AI adoption to foster truly resilient and effective human-technology teams, leveraging AI's strengths while safeguarding against its limitations in the dynamic realm of cybersecurity.

Key Takeaways

  • Intelligence Augmentation is Key: AI should be strategically applied to augment human strengths in tasks where humans excel (e.g., creativity, context, adaptation), rather than attempting to replace human expertise where AI alone is superior.
  • Transparency is Necessary but Not Sufficient: While explainable AI (XAI) is crucial for building trust, its effectiveness hinges on the alignment between the AI model's reasoning and the cybersecurity expert's mental model. A mismatch can paradoxically decrease trust.
  • Trust Builds Over Time and Experience: Trust in AI is not a static attribute but a dynamic process, developed through repeated interactions and proven performance, much like trust in a human colleague.
  • Calibrated Trust Prevents Dangerous Outcomes: Fostering calibrated trust—where trust matches the AI's actual capabilities—is essential to prevent both over-reliance (leading to dangerous security gaps) and under-reliance (leading to disuse and missed efficiencies).
  • Adaptive AI Autonomy Facilitates Trust: Designing AI systems with adaptive autonomy levels allows human experts to gradually increase AI's responsibility as trust is built, enabling a more comfortable and effective integration into security workflows.
  • Human Oversight Remains Critical: In high-stakes cybersecurity environments, maintaining human oversight is paramount for overall system resilience, enabling adaptation to novel threats and ensuring a fallback in cases of AI failure or unforeseen circumstances.

About the Speaker(s)

Professor Verena Zimmermann is a distinguished academic holding a professorship at ETH Zurich, one of the world's leading universities for science and technology. At ETH, she leads the Security Privacy and Society group, where her research is dedicated to human-centered cybersecurity. Her work delves into the intricate human factors that permeate all facets of cybersecurity, extending far beyond traditional end-user training or simple usability considerations. Professor Zimmermann's focus encompasses the psychological foundations of human interaction with technology, exploring how individuals process information, make decisions, and how cybersecurity solutions can be better aligned with these inherent human characteristics. Her overarching mission, as she articulates, is to "turn the human risk into resilience," by enabling humans to contribute positively to security outcomes across various roles, including developers and seasoned cybersecurity professionals.

Reviews

Dr. Zero (Offensive Security Researcher) — SOLID

Every vendor is pitching an AI copilot right now, promising that 'explainable AI' will magically turn tier-1 analysts into seasoned threat hunters. Zimmermann actually ran the empirical human-factors research on this, and the results prove that slapping XAI on a dashboard often decreases trust while increasing blind reliance. It is a necessary reality check for anyone managing a SOC.

Heather Calloway (CISO) — STRONG ACCEPT

Every vendor is selling AI to solve the SOC headcount crisis. This research proves that dropping an AI copilot onto an analyst's desk often degrades performance while increasing blind reliance—a finding that should immediately change how we evaluate these tools in procurement.

→ Top-rated talks at Swiss Cyber Storm 2025

All talks from Swiss Cyber Storm 2025