From Security to Safety: Navigating the Ethics of AI as Red Teamers and Penetration Testers

From Security to Safety: Navigating the Ethics of AI as Red Teamers and Penetration Testers

Speaker: Jeremy Miller (OffSec)

Conference: NorthSec 2025 — May 15–16, 2025, Marché Bonsecours, Montreal

Watch on YouTube: https://www.youtube.com/watch?v=MDep0OwUZoU

Reading time: ~7 minutes

TL;DR

Jeremy Miller argues that security practitioners — particularly red teamers and penetration testers — are better equipped than they realize to take on AI safety work, despite that domain's grounding in normative ethics rather than objective technical facts. Drawing on philosophy of epistemology, he reframes security work as fundamentally a knowledge-seeking practice, not a systems-manipulation practice, and shows why that epistemic orientation is exactly what AI safety challenges require. The discomfort red teamers feel when encountering AI safety is not a deficit; it is a signal that they are encountering genuine moral reasoning for the first time — and they already have the tools to engage with it.

Introduction

The field of AI safety has a vocabulary problem for security practitioners. When companies like Microsoft, OpenAI, and Google DeepMind ask red teams to evaluate their AI systems, they are not asking whether the system can be compromised in the traditional sense. They are asking whether it is fair, reliable, trustworthy, and safe for a diverse range of users — questions that belong to the domain of normative ethics, not descriptive security. For professionals trained to ask "can you get root?" the shift to "is this fair?" feels epistemically foreign and somewhat alarming.

Jeremy Miller's talk at NorthSec 2025 addresses that discomfort head-on. Miller, who works at OffSec and studied philosophy before discovering security, occupies an unusual position: he has spent years at the intersection of both worlds. The talk is structured as a philosophical argument in semi-formal form, beginning with the objection that security professionals are poorly positioned to make moral judgments, and systematically dismantling that objection.

The Problem: Security Is Descriptive, Safety Is Normative

Miller opens with what he calls the standard argument security practitioners make — either explicitly or implicitly — when asked to red team AI:

1. Traditional penetration testing deals with objective, descriptive facts. Can you get root? Yes or no. Can you exploit this LFI? Yes or no.
2. Moral judgments — of the kind embedded in AI safety concerns — are normative. They involve the word "should" or "ought." They are not straightforwardly true or false in the way technical facts are.
3. Many AI safety concerns (fairness, reliability, trust, justice) are moral in nature.
4. Therefore, security practitioners are ill-equipped to evaluate them.

Miller does not dispute the first three premises — he concedes they are roughly correct ninety percent of the time. His argument is that the conclusion does not follow from the premises. And the reason it does not follow has everything to do with the nature of knowledge, not the nature of morality.

▶ Watch: The Formal Argument (4:00)

David Hume and the Is-Ought Problem

To frame the philosophical stakes, Miller introduces David Hume's famous observation — the is-ought problem — from the eighteenth century. Hume noted that no sequence of purely descriptive statements about the world can logically produce a statement containing "should" or "ought." Moral judgments, in Hume's view, originate from sentiment and social consensus rather than from rational observation of facts. They represent what people collectively determine contributes to human flourishing and what causes harm.

The security practitioner who says "I can't evaluate AI fairness because fairness is subjective" is channeling Hume, whether they know it or not. Miller takes Hume's observation seriously — he does not attempt to refute it. Instead, he argues that even if moral judgments are subjective in Hume's sense, they are still knowable through systematic inquiry. And systematic inquiry under uncertainty is precisely what security professionals do for a living.

▶ Watch: David Hume and the Is-Ought Problem (8:00)

Security Is an Epistemological Practice

The central reframe of the talk hinges on a claim that initially sounds counterintuitive: security is not really about systems and networks. It is about knowledge.

Miller's argument is elegant. Consider a web application vulnerable to cross-site scripting. If you knew the exact string that would trigger the alert, and you had full access to the source code, there would be nothing for you to do — you would simply press Enter. All the actual work of exploitation — reconnaissance, fuzzing, reading documentation, chaining vulnerabilities — is the work of acquiring knowledge about a system you do not yet fully understand. The exploitation is trivial once the knowledge is complete. It is the knowledge-seeking that constitutes the practice.

He contrasts two self-images available to security practitioners: the rogue (the hooded figure typically depicted in Google image searches for "hacker") and the explorer. His claim is that security practitioners are fundamentally explorers — people whose professional value lies in their ability to map unknown territory, not in their ability to execute known attacks.

▶ Watch: Security as Epistemology (10:00)

Three Epistemic Virtues for AI Safety Work

Having established that security is an epistemological practice, Miller identifies three virtues he argues are already present in good security practitioners and directly applicable to AI safety challenges.

Curiosity — the Socratic willingness to acknowledge ignorance and pursue what is unknown — is the foundational disposition of good security work. An analyst who picks up an unknown binary and attempts to understand it is already practicing the same cognitive posture required to ask "who might be harmed by this AI system and how?"

Humility — epistemic humility about the limits of one's own knowledge — is structurally built into security work. No penetration tester assumes they have found all vulnerabilities. AI safety work requires the same orientation: acknowledging that a tested AI system may have failure modes that did not manifest during the evaluation.

Systematic skepticism — the discipline of treating every apparent fact as a hypothesis pending verification — is the practitioner's most reliable tool in both domains. In security, it prevents false negatives from becoming missed vulnerabilities. In AI safety, it prevents premature confidence that a system is fair or trustworthy.

Miller's argument is ultimately that the discomfort red teamers feel when confronted with AI safety is not evidence of inadequacy. It is evidence that they are being asked to reason explicitly about things they have always reasoned about implicitly — and that the skills they need to succeed are already developed, just not yet named.

▶ Watch: Three Epistemic Virtues (14:00)

Notable Quotes

"Security is not really about systems and networks. It is really about knowledge."

"All the work that you do in security is trying to figure out what don't I know and how am I going to get that knowledge so that I know enough about the program to exploit it."

"The discomfort is a signal. It means you are engaging with genuine moral reasoning for the first time — and you already have the tools."

"We are not rogues. We are explorers."

Key Takeaways

• The is-ought problem is real but not disqualifying. Hume's observation that you cannot derive "ought" from "is" applies to AI safety — but it does not mean normative questions cannot be engaged with rigorously.
• Security work is already epistemological. The core activity of a penetration tester is knowledge acquisition under uncertainty, not technical execution. That skill transfers directly to AI safety evaluation.
• AI red teaming is definitionally different from traditional red teaming. When major AI labs use the term, they mean evaluation of fairness, reliability, trust, and safety — not vulnerability exploitation. Understanding the distinction matters before engaging with the work.
• Curiosity, humility, and skepticism are the relevant virtues. Security practitioners who have cultivated these professionally are better prepared for AI safety work than they typically realize.
• The field is shifting rapidly. Microsoft's own data showed a sharp increase in safety-related features tested by their AI red team, with security features holding roughly flat — the trend lines suggest safety will be a dominant concern for offensive security practitioners going forward.
• Philosophy has practical value. The ability to identify when an argument's conclusion does not follow from its premises — a basic logical skill — is directly applicable to evaluating claims made about AI systems.