Closing Keynote: Fireside Chat with Will Eatherton

Closing Keynote: Fireside Chat with Will Eatherton

Speakers: Will Eatherton, SVP, Head of Cisco Networking Engineering, Cisco

Conference: Hardwear.io USA

YouTube: https://www.youtube.com/watch?v=5pEZqXF-Qqg

Overview

In a revealing fireside chat at Hardwear.io USA, Will Eatherton, SVP and Head of Cisco Networking Engineering, offered a compelling look into the increasingly critical world of hardware reliability and security, particularly in the age of hyperscale data centers and rapidly expanding AI infrastructure. Eatherton, whose teams are responsible for building the very hardware that AI workloads are now pushing to its limits, provided a unique perspective from the intersection of cutting-edge technology deployment and fundamental engineering challenges. The discussion highlighted how traditional approaches to hardware design, testing, and vulnerability management are no longer sufficient, driven by unprecedented scale, speed, and the emergent capabilities of AI in discovering complex exploits.

The talk underscored a fundamental shift in the strategic importance of hardware reliability, moving from a "leisurely walk" to a high-stakes sprint where even minor failures can result in massive financial and operational disruptions for hyperscalers and AI training environments. Cisco, as a key partner in initiatives like Project Glasswing, is actively leveraging advanced AI models such as Mythos to probe for vulnerabilities, revealing a new frontier in defensive and offensive security. Eatherton's insights reveal a future where the lines between hardware and software security blur, demanding a "shift-left" methodology and a re-evaluation of long-held engineering practices to safeguard critical infrastructure.

Background

▶ Watch: Introduction: AI workloads breaking hardware, Cisco's role (0:00)

Will Eatherton's 25-year tenure at Cisco, encompassing roles acquired through Skyport System and other ventures, positions him as a seasoned veteran in the complex landscape of data center, web, and service provider infrastructure. His experience dates back to the early 2000s, designing ASIC architectures and Network Processing Units (NPUs) featuring hundreds of 32-bit RISC processors to churn packets through multi-terabit fabrics. This history with intricate, multi-chassis hardware systems underscores that complexity in itself is not a novel challenge for the industry.

However, Eatherton emphasized that the current moment feels strategically different. The convergence of two major factors—the rise of hyperscalers and the explosive buildout of AI infrastructure—is uniquely stressing the system. Historically, new high-end products would undergo a gradual ramp-up: a few units for lab testing, followed by incremental purchases across diverse customers and geographies. This allowed manufacturers to identify and address "spot issues" over time, maintaining a low Annual Failure Rate (AFR). Today, hyperscalers acquire hundreds, then tens or hundreds of thousands of devices almost immediately, deploying them in concentrated geographic areas. This accelerated ramp means that issues traditionally discovered and remediated slowly now manifest at catastrophic scale, with potential career-ending implications for engineers and significant financial losses for customers. For instance, Eatherton cited that 66% of Meta's training job interruptions are hardware failures, highlighting the fragility and high cost of unreliability in AI workloads.

In response to this escalated challenge, Cisco has developed a comprehensive defensive design framework built around six core pillars: fault tolerance, reliability, observability, foreseeable misuse, design margin, and qualification. This proactive approach aims to find and mitigate issues much earlier in the product lifecycle, shifting from reactive fixes to preventative engineering. The necessity of this shift is further amplified by the high stakes of AI training, where even "small amounts of impacts can actually cause big... impacts that can cost tens or hundreds of millions of dollars."

Key Findings

▶ Watch: Observability is the most challenging defensive design pillar (5:15)

The talk unveiled several critical findings regarding the evolving landscape of hardware security and reliability:

1. Hardware Reliability's Strategic Apex: The current era, driven by hyperscalers and AI infrastructure, marks an unprecedented strategic criticality for hardware reliability. The rapid, at-scale deployment by hyperscalers transforms traditional "spot issues" into widespread disasters, with failure rates previously deemed acceptable (e.g., less than 1% AFR) becoming devastating when concentrated.
2. Observability as the Weakest Link: Among Cisco's six defensive design pillars, observability is identified as the most challenging and often "underweighted" area. This stems from the increasing complexity of technology, making it harder to monitor and diagnose issues like silent data corruption, and the fragmentation of the industry with open-source network operating systems like Sonic, which can limit end-to-end diagnostic capabilities.
3. Simplicity in High-Impact Decisions: Counterintuitively, some of the highest-impact defensive design decisions are not exotic but rather disciplined adherence to fundamental engineering principles. Examples include precisely orienting capacitors to reduce dust accumulation and corrosion, or accounting for the physical warping effects of gravity on increasingly heavy (up to 70 lbs) modular line cards, which can compromise 100-gig, 200-gig, and 400-gig serdes performance.
4. AI's Transformative Role in Vulnerability Discovery: The last three to six months have seen a significant shift where advanced AI models, specifically Project Glasswing partners utilizing Mythos, are not just identifying potential vulnerabilities but are generating potential exploits and exploit chains. This capability allows AI to combine lower-severity issues into high-impact attacks, fundamentally changing the threat landscape.
5. Accelerated Software Release Cycles: The speed and efficacy of AI in finding software vulnerabilities are forcing manufacturers like Cisco to accelerate their software release cadences. What was traditionally a six-month release cycle, with customers upgrading every one to two years, is now transitioning to quarterly releases, with critical infrastructure customers expected to adopt quarterly upgrades.
6. Hardware Security Lagging but Promising: While AI's impact on software vulnerability discovery is well underway, hardware security is "behind." However, early experiments at Cisco, involving feeding AI models extensive internal design data (schematics, FPGA designs, firmware), are yielding "interesting early results" in identifying hardware vulnerabilities and proposing exploitation methods.
7. Blurring Hardware-Software Boundary: AI models are demonstrating the ability to traverse the traditional hardware-software security boundary. They can propose integrated attacks that combine specific hardware tools (e.g., Screamer FPGA hacker boards) with custom firmware and precise attachment points on a device to achieve exploitation, effectively collapsing previously distinct threat models.
8. Democratization of Attack Capabilities: While nation-states have historically held an advantage in advanced attack capabilities, AI's ability to generate sophisticated exploits threatens to democratize these tools. This empowers "smaller-scale attackers" to create optimized FPGA attacks or other complex exploits without needing deep, specialized engineering understanding.
9. Increased Visibility for Engineering Debt: AI-driven vulnerability analysis shines a harsh light on "engineering debt," particularly in legacy products or those acquired years ago that have seen reduced staffing and attention. Models can reveal a disproportionately higher number of vulnerabilities in such products compared to well-maintained, high-codebase products, making previously invisible issues undeniable.

Technical Deep Dive

▶ Watch: AI models now proposing hardware exploits and attack strategies (17:40)

Cisco's defensive design framework of six pillars serves as the bedrock for addressing hardware reliability. The most challenging pillar, observability, highlights the difficulty in monitoring complex systems for issues like silent data corruption. Eatherton noted that integrating diagnostics is harder today, especially with the prevalence of open-source network operating systems like Sonic, which is popular with hyperscalers like Microsoft and Google. Unlike proprietary stacks where Cisco controlled the entire vertical from chip to telemetry, Sonic environments introduce integration gaps, demanding earlier and more comprehensive planning for potential misuses.

The discussion revealed that seemingly simple physical design choices carry significant weight at scale. For instance, the orientation of capacitors relative to airflow can dramatically reduce dust accumulation and corrosion, preventing intermittent shorts that are notoriously difficult to debug. Similarly, the increasing weight of modular line cards—some now reaching 70 pounds—introduces physical stresses. Gravity can cause minute warping over time, impacting the precise alignment and signal integrity of high-speed 100-gig, 200-gig, and 400-gig serdes interfaces, leading to performance degradation or failure, often only discovered after mass production.

Eatherton also touched upon "silly-sounding" but crucial design decisions driven by extreme operating conditions. As rack power densities soar (hundreds of thousands of watts per rack, with megawatts projected within two to three years), applying traditional conformal coatings can exacerbate thermal issues. This pushes the team to explore exotic solutions like nano-coatings or even liquid cooling, introducing new complexities related to plumbing, power, and network cable interfaces.

The most profound technical shift discussed is the application of AI, specifically Mythos (a Project Glasswing partner model), to vulnerability discovery. On the software side, AI is already proving highly effective. Eatherton cited experiments where AI refactored 500,000 lines of C code into 100,000 lines of Rust. While the code generation is "unbelievably fast," the challenge remains in the time-consuming testing phase. Crucially, AI is moving beyond mere code analysis to generating potential exploits. This involves exploit chaining, where lower-severity vulnerabilities are combined into a high-impact attack, complete with proposed implementations.

For hardware, this application is in its "early experiments" phase. Cisco feeds AI models extensive internal design state, including schematics, design specifications, FPGA designs, and firmware code. The AI then proposes potential vulnerabilities and exploits. Examples include suggestions to use a JTAG whisper glitching device from a side channel, proposing embedded software for an external device, or combining a Screamer FPGA hacker board with specific firmware attached to a management interface to compromise a system. While many proposals are initially "false," a significant number are "interesting," demonstrating AI's ability to synthesize complex attack strategies across hardware and software domains. The challenge for hardware lies in providing sufficient context to the AI models, as hardware designs are less standardized and openly documented than software libraries and CVEs.

Demo / Proof of Concept

▶ Watch: Mythos exposes vulnerabilities in "soft middle" legacy products (27:00)

While the talk did not feature a live, public demonstration or proof of concept, Will Eatherton extensively detailed internal experiments and early results from Cisco's use of AI, specifically Mythos, for hardware vulnerability discovery. These internal "demos" involve feeding the AI models comprehensive design state, including schematics, design specifications, FPGA designs, and firmware code. The AI then generates potential vulnerabilities and, crucially, proposes concrete exploitation strategies.

Eatherton described instances where the AI would suggest using specific attack hardware, such as a JTAG whisper glitching device for side-channel attacks, or the integration of a Screamer FPGA hacker board with custom embedded software, detailing how to attach it to a management interface and run a sequence of steps to exploit a device. These proposals, while still requiring human expert validation due to "tons of hallucinations," represent a significant leap in automated vulnerability research, demonstrating the AI's capacity to bridge the hardware-software divide in attack methodologies.

Defensive Implications

▶ Watch: Blind spots: Evolving system complexity and new technologies (34:50)

The insights shared by Will Eatherton carry significant implications for hardware defenders:

1. Accelerated Patching Cadence: The increased velocity of vulnerability discovery driven by AI necessitates a dramatic acceleration in software release and patching cycles. Cisco is moving towards quarterly software releases for critical infrastructure, expecting customers, particularly in sectors like finance, to adopt equally rapid upgrade schedules. This paradigm shift demands robust, automated testing and deployment pipelines to keep pace.
2. Shift-Left Security for Hardware: AI tools enable a more effective "shift-left" approach to security. By integrating AI-driven vulnerability analysis into the early design and architecture phases, organizations can identify and remediate "easy stuff" before products are even released. This frees up limited human pentesting teams to focus on more complex, novel attack vectors, making products more resilient by the time they reach customers.
3. Democratization of Advanced Attacks: Defenders must contend with the reality that AI will democratize advanced attack capabilities. While nation-states have historically possessed the resources for sophisticated hardware exploitation, AI can empower "smaller-scale attackers" to generate optimized FPGA files or complex exploits without requiring deep specialized knowledge. This broadens the threat landscape significantly.
4. Focus on the "Harness," Not Just the Model: Eatherton stressed that for defenders, the true advantage lies not just in access to powerful AI models like Mythos, but in the "harness" and "experimentation" surrounding them. Developing robust methodologies for feeding models with comprehensive system data, asking targeted questions, and interpreting results is paramount to effectively leveraging AI for defense.
5. Addressing Engineering Debt: AI-driven vulnerability analysis provides unprecedented visibility into "engineering debt"—security weaknesses in legacy products or those neglected over time. This increased transparency will likely put more pressure on engineering managers and business units to allocate resources for maintaining and securing older products, which previously might have been overlooked due to perceived low return on investment.
6. Embracing Transparency (with caveats): While acknowledging the economic challenges, Eatherton expressed a belief that greater transparency in hardware design, akin to open-source software, could ultimately benefit security. Publicly available well-done projects (like the historical Spark processor) can influence industry best practices and foster collective learning, even if direct IP reuse is limited.
7. Anticipating New Attack Vectors: As hardware architectures evolve (e.g., co-packaged optics, less modular fixed systems), some traditional attack surfaces like PCB traces or modular interfaces might shrink. However, new components and interfaces (e.g., liquid cooling systems, advanced optics technologies) will inevitably introduce novel vulnerabilities, requiring defenders to constantly anticipate and adapt to emerging blind spots. Management interfaces, in particular, remain a persistent area of concern.

Key Takeaways

• AI Accelerates Hardware Security Challenges: The combination of hyperscale deployment and AI workloads has made hardware reliability strategically critical, demanding a fundamental shift from traditional, slower development and remediation cycles.
• Observability is Paramount and Problematic: Despite its importance, achieving comprehensive observability in complex, heterogenous hardware and software stacks remains a significant challenge, hindering early detection of critical issues like silent data corruption.
• AI Blurs Hardware-Software Security: Advanced AI models like Mythos are capable of generating sophisticated, cross-domain exploits that bridge hardware and software, forcing a unified approach to security rather than treating them as separate disciplines.
• Shift-Left Security is Imperative: AI enables moving vulnerability discovery earlier into the design and architecture phases, transforming security testing from a post-design activity to an integrated, automated process, thereby enhancing overall product resilience.
• Democratization of Attack Capabilities: AI tools threaten to lower the bar for sophisticated hardware attacks, empowering a broader range of adversaries beyond nation-states, requiring defenders to innovate continuously.
• "Harness" and Experimentation Over Model Alone: For defenders, the effectiveness of AI in security hinges less on the specific model and more on the robust methodologies, contextual data, and iterative experimentation employed to leverage these tools.

About the Speaker(s)

Will Eatherton serves as the Senior Vice President and Head of Cisco Networking Engineering at Cisco. With over 25 years of experience at the company, including through two acquisitions, he leads the engineering organization responsible for building Cisco's data center, web, and service provider infrastructure. His teams oversee a vast portfolio, including products like the Cisco 8000 series, Silicon One, Nexus, and the company's entire data center AI infrastructure. Eatherton's background in silicon ASIC architecture and design, including his early work on NPUs with hundreds of 32-bit RISC processors, provides him with deep expertise in complex hardware. His current role places him at a unique vantage point, overseeing the shipment of hardware that is both foundational to and increasingly challenged by the demands of modern AI workloads and hyperscale environments.