Artificial Intelligence 🤝 Natural Stupidity

Artificial Intelligence 🤝 Natural Stupidity

Speakers: Brandon Sherman, Staff or Senior Staff Engineer

Conference: fwd:cloudsec

YouTube: https://www.youtube.com/watch?v=7bx9UfDHf_k

Overview

In "Artificial Intelligence 🤝 Natural Stupidity," Brandon Sherman, a Staff or Senior Staff Engineer, presents a compelling argument that while artificial intelligence (AI) has the power to magnify human brilliance, it equally magnifies human fallibility. The talk delves into the inherent human tendency to make mistakes, drawing parallels with lessons learned from the aviation industry to highlight the dangers of over-reliance on automation. Sherman's central thesis is that traditional "human in the loop" security models are inadequate when dealing with the scale and complexity of AI outputs, as humans are predisposed to cognitive traps and impaired judgment when automation fails.

Sherman challenges the conventional wisdom of simply training humans to be more vigilant, instead advocating for a fundamental shift in how we design systems that incorporate AI. He argues that instead of tasking humans with catching AI's errors, we should leverage AI's strengths (like pattern recognition) to create safeguards that prevent humans from making catastrophic mistakes, even when they are having a "dumb moment." This talk is crucial for anyone involved in designing, implementing, or securing systems that integrate AI, offering practical insights into building resilient and human-error-tolerant architectures in an increasingly AI-driven world.

Background

▶ Watch: Introduction: Artificial Intelligence 🤝 Natural Stupidity (0:00)

The premise of Brandon Sherman's talk rests on a fundamental understanding of the human condition: humans are inherently prone to error. Drawing on figures from literature and science, Sherman quotes author Kurt Vonnegut, "This is my principal objection to life. I think it's too easy when alive to make perfectly horrible mistakes," and even Charles Darwin, who in his diary confessed, "I am very poorly today and very stupid and hate everybody and everything... One lives only to make blunders." This innate fallibility is a constant, even for the most brilliant minds. Sherman cites a conversation with a Swiss air traffic controller who revealed that their rigorous training standards still account for "one in 10,000 commands is going to be a mistake." If highly trained professionals in safety-critical roles make errors at this rate, the average person, with broader responsibilities and less specialized training, is likely to make many more.

Humans are tool-users, a trait that has set us apart from most other species. However, Sherman distinguishes between traditional tools like a crowbar, which provide immediate haptic feedback and intuitive understanding of their physical properties, and abstract tools like Artificial Intelligence. With a crowbar, one instinctively understands its weight and potential for impact. AI, however, lacks this physical presence, making users more susceptible to cognitive traps. As an accelerant, AI can amplify human actions: in the right hands, it can be an F1 car; in the wrong hands, a forest fire. The critical distinction is that with AI, users cannot "hold" the tool, making them more vulnerable to errors of judgment.

The talk introduces the concept of "children of the magenta line," a term coined by American Airlines chief pilot Warren Vanderberg in 1997, derived from observations in the aviation industry. This describes pilots who become overly reliant on automated navigation systems, specifically the magenta line displayed on their screens, which dictates the flight path. This over-reliance leads to an "automation-dependent pilot" who loses situational awareness, often preceded by task saturation. The aviation industry, renowned for its safety record, offers invaluable lessons because its culture is built on meticulously investigating failures to prevent recurrence.

Sherman emphasizes that the paradox of automation dictates that when automation fails, human responses are not only slower but also less appropriate than if they had never relied on automation at all. Humans don't "rise to the level of their aspirations" but "fall to the level of their training." If that training primarily involves pressing "yes" to approve automated actions, then humans will continue to press "yes," even when "no" is the correct response. This phenomenon has real-world consequences, such as the widely discussed incidents at AWS where engineers, asked to authorize a tool command, deleted production databases, leading to multi-hour outages. These incidents underscore the danger of expecting humans to catch automation's mistakes, especially when constantly bombarded with more and more AI-generated output.

Key Findings

▶ Watch: Darwin's take on stupidity; AI magnifies mistakes (2:00)

The core findings of Sherman's talk revolve around the inherent tension between human fallibility and the pervasive integration of AI into critical systems. He posits that:

1. AI Magnifies Both Brilliance and Stupidity: AI is a powerful accelerant. While it can enhance human ingenuity and productivity, it simultaneously amplifies the impact of human errors. The same individual can leverage AI for groundbreaking work one moment and commit a catastrophic blunder the next.
2. Humans Are Fundamentally Error-Prone: Even highly trained professionals, like air traffic controllers, are prone to making mistakes (e.g., one in 10,000 commands). Expecting humans to be infallible, especially in complex, high-pressure environments, is an unrealistic and dangerous assumption.
3. AI Creates New Cognitive Traps: Unlike physical tools that provide intuitive feedback, AI operates in an abstract domain, making it harder for users to develop a proprioceptive understanding. This lack of intuitive grasp makes users more susceptible to cognitive biases and over-reliance, as exemplified by the "children of the magenta line" phenomenon in aviation.
4. "Human in the Loop" for Error Catching is Broken: The traditional model of relying on humans to review and correct AI-generated output or automated actions is fundamentally flawed. The paradox of automation demonstrates that humans become less adept at a task when automation takes over, making their responses slower and less appropriate when the automation inevitably fails.
5. The Solution Lies in System Design, Not Human Vigilance: Instead of attempting to train humans to be perfectly vigilant (a futile effort), the focus should shift to designing systems that anticipate human error. These systems should incorporate defense-in-depth mechanisms and safeguards that allow humans to make mistakes without catastrophic consequences.
6. AI Can Be Leveraged as a Safeguard Against Human Error: Rather than solely using AI to perform primary tasks, its strengths in pattern recognition and anomaly detection can be harnessed to monitor human actions. An AI agent can act as a "copilot," flagging suspicious human inputs or deviations from established patterns, providing a crucial layer of preventative security.
7. Deterministic AI for Critical Workflows: For tasks requiring high precision, repeatability, and auditability, AI should be used to build deterministic workflows. This approach ensures that the AI's actions are predictable and testable, allowing humans to review outputs that are based on a transparent set of rules, thereby mitigating the risks associated with non-deterministic AI outputs (like hallucinations).

These findings collectively advocate for a paradigm shift in how we approach security and operations in an AI-integrated world, moving away from human-centric error detection to system-centric error prevention.

Technical Deep Dive

▶ Watch: AI is inescapable; learn to use it effectively (3:40)

The technical core of Sherman's argument is rooted in the lessons drawn from the aviation industry, particularly the concept of "children of the magenta line." This term, coined by American Airlines chief pilot Warren Vanderberg in 1997, describes pilots who become automation-dependent. This dependency leads to a loss of situational awareness, often exacerbated by task saturation, where pilots are overwhelmed by multiple demands and inadvertently delegate too much cognitive load to automation.

A pivotal example is the American Airlines Flight 965 crash in 1995. During this incident, pilots received confusing guidance from air traffic control and were attempting to program a shortcut into their flight management system. Distracted by duty time limitations, terrain awareness, and other factors, they inadvertently input the wrong waypoint. The autopilot, faithfully following the magenta line on the navigation screen, began steering the aircraft directly towards a mountainside. By the time the pilots recognized the error, they had only about 13 seconds to react. The plane crashed, resulting in 159 fatalities and four survivors. The critical takeaway was not that the navigation system was inherently flawed, but that its near-perfect reliability for years had led pilots to implicitly trust it without maintaining adequate independent situational awareness.

The response to such incidents in aviation was not primarily to make the automation infallible, but to implement additional safety systems that would act as independent safeguards. For instance, enhanced ground proximity warning systems (EGPWS, though not explicitly named, the context implies this type of system) were made mandatory in 2000, designed to provide earlier warnings of impending terrain collision, even if the primary navigation system was being incorrectly followed. This approach acknowledges that humans will have "bad days" and make mistakes, and the system must be designed to prevent those mistakes from becoming catastrophic.

Sherman extends this principle to the realm of information security and AI. He highlights that we've seen similar failures in cybersecurity, such as the decades-long, largely unsuccessful effort to train users to detect phishing emails by checking padlocks, inspecting links, or looking for spelling mistakes. The effective solution wasn't better human training, but rather hardware-backed multi-factor authentication (MFA) like UbiKeys or Passkeys. These technologies are designed so that even if a human falls for a phishing attempt and attempts to authenticate on a fake website, the hardware key will not authenticate to the incorrect domain, making it impossible for the human to complete the mistake. This is a real-world example of building a "no-loan zone" where human error is safely absorbed.

The talk underscores the paradox of automation, where humans, when routinely supported by automation, become less skilled and less capable of responding effectively when the automation fails. This means that relying on a "human in the loop" to catch AI's mistakes is inherently flawed because the human's ability to detect and correct errors is degraded. Sherman illustrates this with a simple demo: he rapidly clicks through slides, relying on a pre-programmed 5-second transition delay as a safeguard. Even though he intended to stop on a specific slide, the machine's deterministic action (the fixed transition time) prevented him from overshooting, demonstrating how a machine can prevent human error even when the human is actively trying to perform the action.

Instead of asking humans to be vigilant, Sherman proposes leveraging AI's strengths in pattern recognition and anomaly detection to act as a safeguard against human error. He suggests using AI to flag suspicious human inputs or deviations from established patterns. Examples include:

• An AI agent reviewing requests for IAM permissions: If a human quickly approves a request, the AI could flag that the requesting user is logging in from a non-work location, or that similar requests from that user's team have a 0% approval rate, prompting a closer human look.
• Similar to email clients flagging missing attachments when the word "attached" is used, an AI could provide real-time "on-the-shoulder" feedback to humans performing critical actions.

Furthermore, Sherman advocates for using AI to build deterministic and repeatable workflows. This is crucial for tasks where accuracy, auditability, and predictability are paramount. He mentions Temporal as a workflow engine that can facilitate this. While LLMs themselves are often non-deterministic, they can be used to build deterministic systems. He provides examples of internal tools developed by his security team:

• Deputy: A tool for SBOMs (Software Bill of Materials) and dependency management through curated policies. Deputy can deterministically pin and bump GitHub Actions to specific SHA-1 hashes. Instead of an LLM directly creating a pull request (PR) that might hallucinate, Deputy uses a deterministic workflow to parse YAML files, make API calls to GitHub for the latest SHA, update the YAML, and then create a PR. This ensures that the human reviewing the PR is looking at a predictable, well-tested output from a rule-based system, rather than an arbitrary LLM generation.
• Camper: A tool designed to make non-deterministic things deterministic. This is particularly useful for managing complex security campaigns, such as the deployment of IMDSv2 (Instance Metadata Service Version 2), by bringing predictability and structure to inherently variable processes.

These examples demonstrate a shift from "trusting AI" to "designing with AI" in a way that acknowledges and mitigates both AI's and human's inherent limitations.

Demo / Proof of Concept

▶ Watch: AI as an accelerant: powerful tool, minimal safeguards (4:30)

Brandon Sherman presented a simple yet highly effective in-talk demonstration to illustrate the concept of a machine safeguarding against human error. He explained that he had duplicated several of his presentation slides and intended to click through them rapidly, attempting to stop precisely on a specific slide. Despite his legitimate intent and effort, he acknowledged that he had "done this and I actually did this in training before and the cool... okay, not a surprise, right? I was trying to talk and do my my slides and also try and stop."

The crucial element of this demonstration was a 5-second long transition that he had intentionally built into his slides as a safeguard. He explained, "I can hit next slide as much as I want on this transition and it's going to ignore me cuz that's what it's been told to do." This live, relatable example vividly showed how a deterministic machine process (the fixed slide transition) could override and prevent a human's potential mistake (overshooting the intended slide), even when the human was actively trying to perform the action correctly. It perfectly encapsulated the idea that even with legitimate effort, humans can err, and well-designed systems can prevent those errors from having consequences. This small, interactive proof of concept reinforced the talk's central message: leverage computers for what they are good at – deterministic, rule-based execution – to protect humans from their own fallibility.

Defensive Implications

▶ Watch: Lessons from aviation: safety, NTSB, and navigation (6:30)

The defensive implications of Brandon Sherman's talk are profound, urging a fundamental re-evaluation of security strategies in an AI-driven landscape. The primary directive is clear: do not rely on humans to be the sole "in the loop" mechanism for catching mistakes, whether those mistakes originate from AI or from other humans interacting with complex systems.

1. Shift from Human Training to Systemic Safeguards: Instead of continuously investing in training humans to be more vigilant (which the talk argues is largely ineffective, akin to decades of failed phishing awareness campaigns), security teams must focus on building defense-in-depth into their systems. These safeguards should be designed to absorb human errors safely, preventing them from leading to catastrophic outcomes. This means engineering environments where a "bad day" or a "moment of dumb" by an engineer doesn't translate into a multi-hour outage or a security breach.
2. Leverage AI for Anomaly Detection of Human Actions: Rather than using AI primarily for offensive or primary task execution, defenders should harness AI's strengths in pattern recognition and anomaly detection to monitor human behavior and inputs. An AI agent can act as a "shoulder angel," identifying deviations from normal patterns or suspicious human actions. Examples include:

• Flagging unusual login locations for users requesting critical permissions.
• Notifying if a human is approving access requests at an unusually high rate or for teams they don't typically interact with.
• Detecting if a human is interacting with a tool in a way that deviates from established, safe operating procedures.

1. Prioritize Deterministic AI for Critical Workflows: For security operations that demand precision, auditability, and repeatability (e.g., patching, policy enforcement, dependency management), leverage AI to build deterministic workflows. Tools like Deputy and Camper demonstrate this principle by ensuring that actions like pinning GitHub Actions to specific SHA-1 hashes follow a predictable, testable sequence of steps. This allows humans to review outputs that are generated by a clear set of rules, reducing the risk of AI hallucinations or unexpected behavior in critical infrastructure changes.
2. Implement "No-Loan Zones": Design systems where critical actions are protected by mechanisms that prevent human error from having devastating consequences. The example of hardware MFA (UbiKeys, Passkeys) preventing authentication to phishing sites is a prime illustration. Similarly, systems should be designed so that even if an engineer mistakenly authorizes a dangerous command, an underlying safeguard (like the slide transition demo) prevents the command from executing or limits its blast radius. This includes robust authorization layers, granular access controls, and automated rollbacks for changes.
3. Build Resilience Against the Paradox of Automation: Recognize that humans will become less proficient at tasks when automation handles them. Therefore, systems must be resilient enough to function even when human intervention is slow or inappropriate during an automation failure. This requires thorough incident response planning that accounts for degraded human performance and robust automated recovery mechanisms.
4. Question AI Trustworthiness: While AI can be "almost always right," defenders must operate under the assumption that it will make mistakes. The article strongly suggests that trusting AI output without independent verification is as dangerous as blindly following the "magenta line." The defensive posture should be one of informed skepticism, using AI's strengths to monitor and enforce, rather than to dictate actions without oversight.

In essence, the defensive strategy advocated by Sherman is to embrace human fallibility and design systems that are robust enough to cope with it, using AI as an intelligent layer of protection against both human and AI-generated errors, rather than as a primary source of truth to be blindly followed.

Key Takeaways

• AI amplifies both human brilliance and human stupidity: While AI can enhance our capabilities, it also magnifies the impact of our inherent fallibility.
• Humans are fundamentally error-prone: Even highly trained professionals make mistakes, and systems must be designed to account for this universal human trait rather than expecting perfect vigilance.
• Over-reliance on automation ("children of the magenta line") leads to cognitive traps: Blindly trusting AI or automated systems can degrade human situational awareness and lead to catastrophic errors when the automation inevitably fails.
• "Human in the loop" for catching AI's mistakes is often ineffective: Due to the paradox of automation, humans become less adept at tasks handled by automation, making them slower and less appropriate at correcting errors when automation fails.
• Leverage AI to safeguard against human errors: Instead of relying on humans to review AI output, use AI's strengths in pattern recognition and anomaly detection to flag suspicious human inputs or deviations, acting as an intelligent safety net.
• Prioritize deterministic AI for critical workflows: For security and operational tasks requiring precision and auditability, use AI to build predictable, rule-based systems (like Deputy and Camper) that generate testable outputs, allowing humans to review known outcomes rather than non-deterministic AI generations.

About the Speaker(s)

Brandon Sherman is described as a Staff or Senior Staff Engineer. His unique qualification for discussing human error and AI is his self-proclaimed status as "an idiot" who has "done all manner of really dumb things." This personal experience, he argues, makes him qualified to understand the nature of being dumb and how it intersects with moments of brilliance. He mentions that he is moving to Australia soon. His presentation style is energetic, reflective of "the speed of a pint of cold brew."