Building Anthropic: Safety-First AI at the Frontier

Building Anthropic: Safety-First AI at the Frontier

Speakers: Ben Mann, Co-Founder, Anthropic

Conference: CS153 Infra @ Scale 2025

YouTube: https://www.youtube.com/watch?v=UdxSCFmUk9o

Overview

In this insightful talk from CS153 Infra @ Scale 2025, Ben Mann, Co-Founder of Anthropic, provides a comprehensive look into the formidable engineering and safety challenges inherent in building state-of-the-art AI systems at an unprecedented scale. Mann, a key figure in the development of GPT-3 at OpenAI before co-founding Anthropic, shares his unique perspective on the journey from early AI research to the current frontier of large language models, emphasizing the critical role of safety as a foundational principle. The discussion delves into the technical breakthroughs that have enabled explosive growth in AI capabilities and the complex infrastructure required to support it.

The talk highlights Anthropic's commitment to a "safety-first" approach, framing it not just as an ethical imperative but also as a commercial differentiator that fosters a "race to the top" among AI developers. Mann articulates how the once-skeptical industry has come to accept the power of scaling laws and the immense engineering effort required to harness them effectively. He underscores the necessity of a tightly integrated research and engineering culture to navigate the complexities of training and deploying models like Claude 3.5 Sonnet, which push the boundaries of existing computing infrastructure.

This article dissects Mann's presentation, exploring the historical context of AI development, the technical intricacies of training and aligning frontier models, and the proactive measures Anthropic implements to ensure responsible scaling. It offers valuable insights for anyone involved in AI development, infrastructure management, or cybersecurity, particularly concerning the defensive implications of increasingly powerful and autonomous AI systems. The talk ultimately makes a compelling case for sustained investment in AI safety research and engineering, positioning it as central to the beneficial deployment of artificial general intelligence.

Background

▶ Watch: Anthropic's explosive growth and current scale (0:00)

Ben Mann's journey into AI began in undergrad, where he pivoted from mechanical engineering to computer science, drawn by the imaginative potential of AI. His early exposure to concepts like expert systems and multi-layer perceptrons predated the modern AI boom. A pivotal moment arrived in 2015 with ImageNet, which demonstrated the practical applicability of deep learning techniques on tasks previously requiring human judgment, trainable on a single GPU. This spurred Mann to self-teach in AI, leading him to OpenAI in 2017, influenced by Nick Bostrom's "Superintelligence" and its call for existential grappling with AI's impact.

At OpenAI, Mann became a first author on the GPT-3 paper, focusing on data engineering and analysis, and conducting architecture experiments. This experience solidified his belief in scaling laws—the empirical observation that increasing model size, data, and compute leads to predictable improvements in performance. This belief was contrary to widespread skepticism at the time, epitomized by Google's T5 paper, which concluded that even its 11-billion parameter model showed "no returns to scale" and was "undeployable." Many in the infrastructure community, accustomed to performance plateaus in traditional computing systems (like CPU performance after the '90s boom), doubted that exponential growth would continue in AI.

Mann challenged this premise, arguing that observed plateaus in other computing metrics, such as memory bandwidth or interconnect speeds, were often due to a lack of investment rather than fundamental physical limits. He cited the resurgence of innovation in interconnects following Nvidia's acquisition of Mellanox and the remarkable memory bandwidth of Apple's M-series chips as evidence. He likened the skepticism to the "Banister effect," where the perceived impossibility of breaking the 4-minute mile vanished once it was achieved. For Mann, the resistance also stemmed from a tendency to assign a "special" quality to human cognitive ability, underestimating how scaling could unlock complex capabilities like reasoning through improved elicitation. Four years ago, Mann and seven others departed OpenAI to found Anthropic, driven by a mission to embed safety as a core, non-negotiable principle in AI development, aiming to create a "race to the top" for responsible AI.

Key Findings

▶ Watch: GPT-2 reveals path to AGI via scaling (2:59)

The talk revealed several critical findings and principles guiding Anthropic's approach to frontier AI development:

• Robustness of Scaling Laws: Mann's experience with GPT-3 confirmed that scaling laws hold across many orders of magnitude. This predictability transforms AI development from an "art" of trial-and-error into a more scientific endeavor, allowing for small, cheap experiments to predict the efficacy of large, expensive training runs. This understanding is fundamental to Anthropic's strategy, enabling confidence in massive compute investments.
• Safety as a Commercial Differentiator: Anthropic found that breakthroughs in AI safety could be "commercially valuable," creating a "race to the top" where other companies are compelled to match their commitments to responsible AI. This demonstrates that prioritizing safety is not merely an ethical overhead but can drive market leadership and trust.
• Integrated Research and Engineering: A key lesson from the GPT-3 era and Anthropic's subsequent success is the necessity of a "really tight collaboration between researchers and engineers." Unlike older models where engineers might be "lackeys," Anthropic fosters cohesive teams that steer projects together, treating them as "engineering Mega projects" that require integrated resource allocation and strategic decision-making.
• Constitutional AI (RLAIF) for Scalable Alignment: Anthropic pioneered Reinforcement Learning from AI Feedback (RLAIF), also known as Constitutional AI, as a significant advancement over Reinforcement Learning from Human Feedback (RHF). RLAIF uses a set of natural language principles to guide the model to critique and revise its own outputs, enabling a more steerable, repeatable, and scientifically amenable alignment process without direct human supervision. This is only possible with models above a certain capability threshold.
• Mechanistic Interpretability as the Ultimate Audit Tool: Mann highlighted mechanistic interpretability as a promising frontier for AI safety. The ability to "peer into the mind of the model" and understand how internal concepts are formed, not just its token outputs, offers the potential to audit models for undesirable behaviors like resource stockpiling or shutdown resistance. This early-stage field is considered crucial for ensuring control over extremely powerful AI.
• The "Elicitation Overhang": A significant concern is the "elicitation overhang," meaning models may possess latent capabilities that are not yet discovered or fully understood because the right prompt or interaction method has not been found. Examples like Chain of Thought prompting or simply adding "try very very very very hard" dramatically improving outputs illustrate this challenge, underscoring the dynamic nature of evaluating AI capabilities.

Technical Deep Dive

▶ Watch: Leading GPT-3 project and confirming scaling laws (4:00)

Building Anthropic's frontier models, such as Claude 3.5 Sonnet, involves overcoming immense technical hurdles that transcend traditional software engineering. The scale of these endeavors demands treating them as "engineering Mega projects," requiring deep, cohesive collaboration between researchers and engineers.

In the GPT-3 era, the primary challenge was pushing the limits of what was thought possible with scaling laws. This involved meticulous data engineering, architectural experimentation, and constant monitoring of training runs. The shift from an "art" to a "science" meant understanding how hyper-parameters, data collection, and dataset quality influenced model performance at scale.

Today, with models exponentially larger (roughly 10 orders of magnitude higher than early Claude models in terms of user base and internal complexity), the engineering problems have ballooned:

• Compartmentalization and Compute Multipliers: With hundreds of people working on these models, Anthropic employs techniques borrowed from intelligence organizations and CPU developers to compartmentalize knowledge. This ensures that valuable "compute multipliers"—proprietary techniques that enhance capabilities for a given compute budget—do not leak, while still enabling a cohesive final product.
• Compute Orchestration at Extreme Scale: Anthropic relies on major cloud providers like Amazon and Google, pushing their Kubernetes clusters "out of spec" with extremely high node counts. Workloads are unique, demanding high reliability for massively distributed jobs where a single machine failure must allow for quick restarts without significant progress loss. Cloud storage for snapshots and efficient data transmission to training machines are also critical.
• Reinforcement Learning (RL) Complexity: The increasing reliance on RL for model alignment introduces further complexity. This involves managing stateful environments where agents interact, and ensuring these agents have the most recent model weights, requiring highly efficient update mechanisms. Mann humorously recalled a bug at OpenAI where a flipped negative sign in a reward model caused the model to become "more and more evil" during training, highlighting the subtlety and fragility of these systems.
• Observability and Monitoring: Training runs are continuously monitored with hundreds of diagnostics. Developers watch loss curves for health indicators, noting frequent loss spikes that often require rolling back and restarting without any code changes—a process Mann likened to having a "patient on life support." While standard engineering practices like on-call rotations and follow-the-sun models have been adopted, the process remains intensely demanding, often requiring engineers to "babysit the run."

The evolution of model alignment techniques is central to Anthropic's technical approach:

• Reinforcement Learning from Human Feedback (RHF): Initially pioneered at OpenAI, RHF involves humans submitting preferences on model outputs. These preferences train a preference model (the "teacher"), which then provides rewards to train the primary model (the "student") through reinforcement learning.
• Reinforcement Learning from AI Feedback (RLAIF) / Constitutional AI: Anthropic's innovation, RLAIF, is a more scalable and steerable approach. It defines a set of natural language principles (e.g., "be kind," "don't write cyber security attacks"). The model then critiques its own outputs against these principles and revises them, all in a "completely enclosed process with no humans in the loop." This method is "much more steerable," "repeatable," and "amenable to science," allowing for precise tuning, but it requires models to be above a certain capability threshold to effectively self-critique and revise.
• Multi-Turn Coherence: Early Claude models focused heavily on achieving multi-turn coherence from the start, collecting human feedback on dialogue-like interactions. This, combined with improving model quality, allowed Claude to maintain character and act as a "helpful, harmless assistant" long before broader public release.

Finally, the long-term vision includes mechanistic interpretability. This involves developing techniques to "peer into the mind of the model" to understand its internal representations and decision-making processes, rather than just observing its inputs and outputs. The goal is to audit models for emergent, undesirable behaviors such as "resource stockpiling or shutdown resistance," which are critical for controlling future, more powerful AI systems.

Demo / Proof of Concept

▶ Watch: Debating AI scaling laws vs. computing plateaus (6:00)

While the talk did not feature a live technical demonstration in the conventional sense, Ben Mann's narrative effectively served as a retrospective "proof of concept" for Anthropic's foundational principles and model capabilities.

Mann recounted the early days of Claude, specifically a "friends and family" version released in Slack prior to the public launch of ChatGPT. This early iteration, though "pretty basic" and lacking significant optimizations for speed or comfort, was described as "extraordinarily coherent." Testers, including the interviewer, noted a stark difference from earlier models like GPT-2.5, which could be "chaotic" and struggle with sustained, coherent conversation. Claude, by contrast, demonstrated an ability to "maintain character" and behave as a "helpful, harmless assistant."

This early internal release, operating within a constrained environment, served as a crucial validation point for Anthropic's focus on multi-turn human feedback and system prompting. The coherence observed was attributed to both the inherent quality improvement of scaled models and the deliberate design of training runs that incorporated multi-turn human feedback from the outset. This "proving ground" phase within the chat experience allowed Anthropic to rapidly iterate and integrate feedback, validating their approach to alignment and character consistency before exposing these capabilities to the wider world through more stable API offerings. The decision to initially withhold a public release, despite the model's capabilities, further underscored their commitment to safety, providing "six more months to work on safety" before the broader acceleration of the AI landscape.

Defensive Implications

▶ Watch: Investment, not fundamental limits, caused past plateaus (7:59)

The insights shared by Ben Mann carry profound implications for defenders, whether they are securing AI systems, integrating them into operations, or preparing for future AI capabilities. Anthropic's "defense-in-depth" mindset, borrowed from traditional security, suggests that safety is not a single feature but a multi-layered approach.

For Organizations Deploying LLMs:

• Scrutinize Provider Safety Policies: Organizations should prioritize AI providers with robust Responsible Scaling Policies (RSPs) and clear AI Safety Levels (ASLs). Anthropic, for instance, commits to specific mitigations at each ASL, such as two-party control for code commits at ASL3 to mitigate insider threat risks when models reach certain biological threat capabilities. Understanding these commitments is crucial for risk assessment.
• Be Aware of Elicitation Overhang: The concept of an "elicitation overhang" means that models may possess latent, unrevealed capabilities. Defenders must assume that even if a model doesn't currently exhibit malicious behavior, a novel prompting technique (like Chain of Thought) could unlock it. Continuous red teaming and vigilance are essential.
• Implement Defense-in-Depth for Usage: Beyond provider-side safety, organizations must apply their own defense-in-depth. This includes:
• Prompt Shielding: Employing online classifiers to detect and prevent users from attempting "bad stuff" or eliciting harmful outputs.
• Careful Fine-tuning and Context Provision: Understanding that providing more context (like uploading PDFs, a feature Anthropic rolled out from chat to API) can improve model performance and steerability, but also potentially expose sensitive information if not handled carefully.
• Distinguish API vs. Chat Risk Profiles: Recognize that API offerings, designed for stability and business continuity, evolve slower than chat experiences. This means API deployments might lag in new safety features but offer greater predictability, while chat interfaces allow for rapid iteration and testing of new safety mechanisms.
• Prepare for CBRN and Cyber Risks: Mann explicitly mentions CBRN (chemical, biological, radiological, and nuclear) risks and automated cyber attacks as primary concerns at higher ASLs. Defenders need to consider how increasingly capable models could accelerate human researchers in these domains or even directly orchestrate attacks.

For the Broader AI Community and Researchers:

• Prioritize AI Safety Research: The talk emphasizes that AI safety is a "super hard problem" requiring "more brains." Academic settings, in particular, are highlighted as crucial for contributing to evaluations (e.g., OS World benchmarks for desktop manipulation capabilities) without needing massive resources.
• Invest in Mechanistic Interpretability: This field is presented as the "greatest hope" for auditing and controlling powerful AI. Defenders should support and contribute to research that aims to "peer into the mind of the model" to detect undesirable internal states or motivations like resource stockpiling or shutdown resistance, before they manifest externally.
• Recognize AI Safety as an Engineering Challenge: Mann makes it clear that safety is not just a research problem but a significant engineering challenge, particularly with systems like RLAIF. Software engineers are vital in building the robust pipelines, observability tools, and control mechanisms necessary for safe AI development.
• Collaborate with Safety Institutes: Anthropic's collaboration with the United States and UK AI Safety Institutes underscores the importance of multi-stakeholder efforts to address global AI risks.

Ultimately, Mann's discussion on Anthropic's Long-Term Benefit Trust (LTBT), a governance mechanism capable of shutting down the company if it deviates from its net-positive mission for humanity, provides a glimpse into the extreme measures considered necessary for ensuring long-term safety. This proactive, multi-faceted approach to safety is a critical blueprint for all stakeholders navigating the rapidly evolving AI landscape.

Key Takeaways

• Scaling Laws are Foundational but Demand Engineering Mastery: The predictable performance gains from scaling models, data, and compute are robust, but realizing them requires massive, tightly integrated engineering efforts, transforming AI development into "engineering Mega projects."
• Safety is a Strategic Imperative, Not Just an Ethical One: Anthropic's experience demonstrates that a "safety-first" approach can drive commercial value and create a "race to the top," compelling other AI developers to prioritize safety commitments.
• Constitutional AI (RLAIF) is Key to Scalable Alignment: By enabling models to self-critique and revise based on natural language principles, RLAIF offers a more steerable, repeatable, and scalable method for aligning powerful AI systems than traditional human feedback.
• Robust Observability and Collaboration are Non-Negotiable: Managing frontier model training runs involves pushing cloud infrastructure "out of spec," requiring continuous, sophisticated observability, on-call rotations, and deep collaboration between researchers and engineers to handle daily breakage and subtle bugs (like the "evil" reward model).
• Mechanistic Interpretability Offers Hope for Control: Peering into the internal workings of models through mechanistic interpretability is considered the most promising long-term path to auditing, understanding, and ultimately controlling extremely powerful AI systems, preventing undesirable emergent behaviors.
• The Frontier Remains Data Center-Bound, but Risks Evolve: While local models will continue to advance, the cutting edge of AI capabilities will always reside on data center-level compute. However, the "elicitation overhang" means latent capabilities, including risks like CBRN and cyber attacks, can be unlocked by novel prompting, necessitating continuous vigilance and robust defensive strategies.

About the Speaker(s)

Ben Mann is a Co-Founder of Anthropic, a leading AI safety and research company. His academic journey began with an initial interest in mechanical engineering and robotics, but he quickly found his passion in computer science, specifically the AI track, at Columbia University. Despite not starting to code at a young age, Mann quickly fell in love with the field. He worked at Google, gaining foundational experience, before joining OpenAI in 2017. At OpenAI, he played a crucial role as one of the first authors on the groundbreaking GPT-3 paper, where he led data engineering efforts and conducted extensive analysis and architecture experiments. Mann's conviction in the power of scaling laws and the critical importance of AI safety, influenced by works like Nick Bostrom's "Superintelligence," led him and seven other colleagues to depart OpenAI and co-found Anthropic four years ago, with a mission to build safe and beneficial AI systems at the frontier. He is known for his deep engineering insights without holding a formal research background like a masters or PhD, having largely self-taught in the field of AI.