From protecting a country to security guard in a bookshop

From protecting a country to security guard in a bookshop

Speakers: Ian Levy (Keynote speaker)

Conference: DistrictCon

YouTube: https://www.youtube.com/watch?v=RDqXQ4nCOIE

Overview

In his DistrictCon keynote, "From protecting a country to security guard in a bookshop," Ian Levy delivers a provocative and deeply critical "rant" about the state of the cybersecurity industry. Drawing on over 25 years at GCHQ, where he established the UK's National Cyber Security Centre (NCSC), and his current role as a Distinguished Engineer at Amazon, Levy argues that the fundamental incentives, language, and approaches to security are "fundamentally broken." His talk is a call to action, challenging the audience to rethink ingrained assumptions about risk, user behavior, and the true impact of technology in a rapidly evolving geopolitical landscape.

Levy's unique perspective, having navigated both national security and hyperscale commercial operations, provides a stark contrast between two worlds with vastly different scales, reward windows, and risk calculi. He contends that the security industry often fails to apply scientific rigor, communicate effectively, or design systems with human factors in mind, leading to persistent and avoidable vulnerabilities. The talk underscores that the impact of security failures in the private sector, particularly for global giants like Amazon, can now rival or even exceed that of national governments, demanding a radical shift in how we approach defense.

This article dissects Levy's arguments, exploring his key findings, technical deep dives, and defensive implications. It highlights his urgent message: if we continue down the current path, technology will become increasingly unsafe, driven by misaligned incentives, a commoditized threat landscape, and a dangerous anthropomorphism of both adversaries and artificial intelligence systems.

Background

▶ Watch: Keynote speaker Ian Levy begins his talk (5:44)

Ian Levy's career trajectory provides the essential context for his critical perspective. For over two decades, he served at GCHQ, the UK's signals intelligence and security agency, akin to the NSA. His most significant contribution was spearheading the creation of the National Cyber Security Centre (NCSC) in 2016, GCHQ's defensive arm. The NCSC was a pioneering initiative, consolidating all UK cyber defense efforts into a single agency, serving everyone from top-secret military intelligence systems to local governments, healthcare, education, businesses, charities, and the general public. This broad mandate forced Levy and his team to confront the realities of cybersecurity across an entire nation of 66 million people.

During his tenure at NCSC, Levy identified several core problems that continue to plague the industry. He termed the pervasive obscurity in security communication the "magic amulet problem," likening it to medieval quackery where complex, fear-inducing language prevents rational risk management. This led to a "throw everything at it" mentality, exemplified by his satirical "Airgap.im" product from 2010 – a 19-inch rack unit with nothing in the middle, yet marketed with exaggerated claims of 100% protection against all threats.

Another major fallacy he confronted was the notion that "users are dumb" and the "weakest link." Levy passionately argues this is "utter bollocks," illustrating his point with the B7 bomber analogy. In World War II, many B7 bombers crashed upon landing, initially blamed on pilot error. However, it was later discovered that the identical, closely spaced toggle switches for landing gear and flaps, in a dark, raining, high-stress environment, were the true cause. This led to shape coding in aviation, where critical controls are distinctively shaped (e.g., a wheel-shaped lever for landing gear in an Airbus A380). Levy challenges the industry: "Why is the IT equivalent of shape coding… why do we still present people with emails with embedded links and go, it's your job to make sure that link is cool before you click it?"

Upon moving to Amazon, Levy experienced a profound shift, realizing the "fallacy of scale." Amazon's operational scale dwarfs that of many governments; it has more physical hosts than people in the UK, and a single security monitoring system generates more data in a month than the entire UK government holds. This scale fundamentally alters the "calculus" of what's possible, from formal methods applied to entire systems to 10-year log retention and daily "log dives." He also noted differences in incentives: government often prioritizes ministerial reward within election cycles, limiting long-term investment, whereas Amazon's "customer obsession" fosters a longer-term risk calculus. Crucially, Levy came to understand that the impact of failure for a company like Amazon, which underpins critical infrastructure for dozens of allied governments, can now be "arguably higher" than that of a national government.

Key Findings

▶ Watch: Why blaming users for security failures is wrong (18:30)

Ian Levy's talk presents several critical findings that challenge the foundational assumptions of the cybersecurity industry:

1. Broken Incentives: The market currently rewards companies for not doing security properly. Levy cites examples like Amazon's share price rising after a major AWS outage (DynamoDB in US East-1, October 19, 2025, which took down a third of the internet) and Microsoft's share price increasing after disclosing "Midnight Blizzard" and the subsequent CSRB report. This indicates that financial markets do not penalize security failures sufficiently, disincentivizing proactive investment. He extends this to private equity firms like Toma Bravo, arguing they were truly responsible for SolarWinds by prioritizing cost-cutting over security.
2. Dysfunctional Language and Risk Communication: The industry's use of jargon, fear-mongering, and complex terminology ("flaming turd pile vulnerability") alienates non-experts and prevents effective risk management. Levy argues that risk is a "state of mind," not a science, and illustrates this with the UK's Chief Medical Officer's alcohol consumption guidelines vs. David Spiegelhalter's "equivalent risk" comparisons (e.g., daily wine vs. twice-weekly bacon sandwich or watching Downton Abbey). The way we frame risk (e.g., "mortgage-backed securities" vs. "toxic debt") profoundly influences perception and decision-making.
3. Lack of Scientific Approach: Unlike aviation, which achieved systemic harm reduction through rigorous engineering and data analysis (showing declining accident rates and increased survival), cybersecurity remains stuck in vulnerability management, a "fundamentally different thing and is dumb as shit." We focus on patching individual bugs rather than designing inherently safer systems.
4. Users are Not the Weakest Link; Design is: The pervasive belief that users are "stupid" and the primary source of security failures is a cop-out. Levy, using the B7 bomber example, argues that poor system design and user interfaces (e.g., asking users to parse email headers to spot phishing links) are the root cause. We must apply "shape coding" principles to digital interactions.
5. Scale Changes Everything: Hyperscale environments (like Amazon) enable capabilities (e.g., formal methods for entire systems, 10-year log retention, real-time analytics across vast networks) that are impossible for smaller entities or governments. This shifts the calculus of what's achievable in security.
6. Technology Encodes Values and Geopolitics: The choice of technology is inseparable from geopolitical power struggles. Standards bodies (3GPP, O-RAN, ITU) are "incredibly fragile" and "set up to be gained," often dominated by specific national interests (e.g., Chinese chairs and standard essential patents in 5G). The concept of "balkanized technology" is already here, with different nations adopting different foundational technologies (e.g., China's Long Arch 64 and native Linux variants vs. Western tech), eliminating "mutually assured destruction" in critical national infrastructure (CNI) and financial systems (Swift vs. CIPS).
7. Commoditization of Threat and Supply Chain Fragility: The rise of CNO (Computer Network Operations) and vulnerability research as a service (e.g., NSO Group, Candiru) means sophisticated exploits are available for purchase, changing the incentive structure for researchers (selling to brokers vs. responsible disclosure). Recent supply chain attacks like XZ Utils demonstrate a chilling level of long-term, sophisticated infiltration, raising questions about the defensibility of open-source ecosystems.
8. Dangerous Anthropomorphism: We falsely attribute human characteristics and motivations to adversaries (assuming they share our constraints/incentives), users (assuming they can perform complex security tasks), and increasingly, AI systems (assuming "emergent behavior" equates to "intent" or that agents "understand consequence" rather than just "context"). This leads to critical security blind spots, equating prompt injection to SQL injection as a fundamental design flaw.

Technical Deep Dive

▶ Watch: Aviation safety: a model for systemic harm reduction (28:00)

Ian Levy's "rant" is rich with technical examples, illustrating his broader points about the broken state of cybersecurity.

The Fallacy of Scale and Formal Methods: Levy highlights Amazon's scale, managing more physical hosts than the entire population of the UK. One security monitoring system at Amazon generates more data in a month than the UK government holds in total. This immense scale transforms what's technically feasible. At GCHQ, Levy invested £5 million annually into a formal methods research institute for five years, ultimately proving a cryptographic protocol and finding a single bug. In contrast, Amazon, with its vast compute resources and scientific talent, formally proves entire systems. This demonstrates how scale can fundamentally shift the economics and practicality of rigorous security techniques.

The Persistence of Fundamental Vulnerabilities: Levy traces the stack buffer overflow from its first documented description as a security problem in 1972 in the GCOS system (a user-mode to kernel-mode exploit) through to WannaCry in 2017. He points to the specific eight characters in the Windows SMB code responsible for the WannaCry vulnerability—a cast from a 32-bit to a 16-bit integer, leading to an overflow. His outrage is palpable: "How is it okay that we've built a world in which eight characters can fuck the entire world?" This illustrates the industry's failure to move beyond basic, decades-old vulnerability classes.

Memory Safety: CHERI vs. Amazon Nitro: To address the persistent memory safety issues, Levy invested £72 million at GCHQ in CHERI (Capability Hardware Enhanced RISC Instructions), a project by Cambridge University that builds instruction set architecture (ISA) extensions for various CPUs to provide spatial and temporal memory safety guarantees. Despite the significant investment, "nobody uses it." He contrasts this with Amazon's approach: the Nitro hypervisor. While not directly CHERI, Nitro achieves similar security guarantees for virtualization by being formally proven and written in Rust, a memory-safe language. Levy admits he "completely misunderstood incentives" in his CHERI investment, as Amazon achieved a similar outcome at scale, but for "much less cost," underscoring how economic and practical incentives drive adoption.

Active Cyber Defense in the UK: Levy details successful Active Cyber Defense (ACD) initiatives at the NCSC. The Protective DNS service for the entire country significantly reduced the UK's exposure to global phishing attacks from 1.2% to 0.2%. For HMRC (the UK tax collector), which was the second most-phished brand globally, ACD reduced its phishing ranking to 150th. This was achieved by "understanding the criminal incentive" and deploying real-time analytics across the UK telecoms industry, even detecting and thwarting attempts by foreign governments (e.g., India) to hijack the protective DNS. The NCSC also published its ACD reports, detailing what worked and what didn't, a rare level of transparency for a government agency.

Geopolitics, Standards, and Huawei: Levy delves into the intricate relationship between technology, standards, and geopolitics, using Huawei as a case study. He recounts how, in 2017-2018, Russian actors compromised 1,800 edge routers in a UK national carrier by exploiting Telnet from the internet with single-factor authentication and compromised passwords. They established GRE tunnels to siphon traffic, including HTTPS (port 443), which Levy dryly suspects was just "to fuck with us" rather than implying SSL decryption capabilities. This highlights the prevalence of basic, easily preventable vulnerabilities even in critical infrastructure.

He then exposes the geopolitical control over technical standards. For 3GPP, O-RAN, and ITU (key telecommunications standards bodies), Levy presents figures showing the proportion of Chinese chairs in technical working groups (e.g., 50% for O-RAN). Chinese companies also own approximately 25% of all Standard Essential Patents (SEPs) for 5G. This means, as Levy states, "You have to deal with Chinese technology if you want to use a mobile phone. You have no choice." He debunks the "single Chinese base station" fallacy, arguing that if your network's security is predicated on the absence of any foreign-made component, your network is fundamentally "built wrong." He also points out the hypocrisy of national-origin discrimination, noting that US companies like Ericsson Redback are legally prohibited from knowing the nationality of their coders (due to US code 1324 B1), making "nationality of the company" a useless discriminator for supply chain trust.

The UK's response involved a detailed telco security analysis (publicly available) that identified 140 different ways to "pwn a telco network," with only one involving a "bad manufacturer." This led to a change in law, empowering Ofcom (the UK telco regulator) to fine companies up to 10% of global turnover for inadequate security, regardless of service outages (previously, fines only applied for outages affecting over 10,000 people, maxing out at £500,000). This drastic shift in incentives led to tangible results, such as the 2023 ban on leasing SS7 global titles, a critical step to prevent international signaling system abuse for geolocation and other malicious activities, a problem that dates back to 1972.

Supply Chain Attacks and Processor Architectures: Levy uses XZ Utils as a prime example of the new reality of supply chain attacks, where sophisticated actors (likely China) invested "three years of effort" to build a legend and inject a backdoor. He questions how to defend open-source ecosystems against such sophisticated, long-term compromises. He also points to the emerging "balkanization of technology" through processor architectures like RISC-V (open-source, but with diverse international contributors) and Long Arch 64 (China's proprietary ISA). He reminds the audience of CVE-2012-17 (the Intel CISRE bug), a 20-year-old documentation error that allowed privilege escalation by mismanaging non-canonical addresses in privileged mode. He asks: "What makes you think you're going to find the same in RISC-V? What makes you think you're going to find the same in Long Arch 64?" implying that hidden flaws in these increasingly divergent architectures will be difficult to detect, further complicating global security.

Anthropomorphizing AI: Finally, Levy warns against anthropomorphizing AI systems. Citing Sarah Mercer from the Alan Turing Institute, he emphasizes that "agents understand context, not consequence." He argues that "emergent behavior in an AI system is not the same as intent" and that current issues like prompt injection are fundamentally similar to old SQL injection and URL encoding bugs—they are design flaws, not signs of sentient malice. This highlights a critical oversight in the rush to adopt new technologies without addressing foundational security principles.

Demo / Proof of Concept

▶ Watch: Market incentives are broken for cybersecurity investment (45:00)

As a keynote address focused on critical analysis and a "rant" against industry shortcomings, Ian Levy's talk did not include a live demonstration or proof of concept. Instead, his arguments were supported by real-world examples, historical context, and specific technical references from his extensive career.

Defensive Implications

▶ Watch: The dangers of anthropomorphizing AI systems (54:00)

Ian Levy's "rant" provides crucial insights for defenders, urging a fundamental shift from reactive vulnerability management to proactive, systemic harm reduction and a critical re-evaluation of established practices:

1. Prioritize Systemic Harm Reduction over Vulnerability Management: Stop chasing every individual bug. Instead, emulate the aviation industry's approach: design systems to be inherently safer, reducing both the likelihood of incidents and the impact when they do occur. This means investing in foundational security, not just patching.
2. Demand Better Design and User-Centric Security: Acknowledge that "users are not dumb; design is dumb." Implement "shape coding" principles in digital interfaces. This means eliminating ambiguous links in emails, providing clear visual cues for critical actions, and designing systems that make the secure choice the easiest and most intuitive choice. Do not offload complex security decisions onto end-users.
3. Challenge and Reshape Incentives: The current market incentives for security are broken. Defenders must advocate for regulatory frameworks (like the UK's Ofcom fines) or industry standards that genuinely penalize poor security and reward robust defense. This requires transparency and holding companies accountable for their security posture, not just their uptime.
4. Embrace Formal Methods and Memory Safety at Scale: For critical systems, leverage the power of modern compute and expertise to implement formal methods, as Amazon does with its Nitro hypervisor. Invest in memory-safe languages like Rust for new development, recognizing that foundational vulnerabilities like buffer overflows are still causing global-scale damage decades later.
5. Understand the Geopolitical Dimension of Technology: Recognize that "technology encodes values." Scrutinize supply chains deeply, going beyond national origin of companies to understand the true composition and control of critical components. Actively participate in and influence standards bodies (3GPP, O-RAN, ITU) to ensure they are not dominated by adversarial interests and to promote secure-by-design principles.
6. Address Basic Hygiene and Operational Security: The Russian Telnet compromise of UK national carrier routers (using single-factor authentication and default passwords) is a stark reminder that many breaches stem from fundamental operational security failures. Defenders must continuously enforce basic hygiene: disable unsecure protocols (Telnet, SSH without MFA), enforce strong authentication, and regularly audit network configurations, especially for internet-facing management planes.
7. Rethink Supply Chain Security for Open Source: The XZ Utils attack demonstrates a new level of sophistication and patience in supply chain compromise. Defenders need to explore new models for vetting open-source components, understanding their true provenance, and potentially investing in mechanisms to verify the integrity of critical dependencies beyond traditional scanning.
8. Be Wary of Anthropomorphism in AI Security: When dealing with AI systems, remember that agents understand context, not consequence, and emergent behavior is not intent. Approach prompt injection and other AI-specific vulnerabilities as fundamental design flaws, akin to SQL injection, rather than attributing them to the AI's "intelligence" or "will." Focus on securing the underlying data, models, and interaction mechanisms.
9. Advocate for Transparency and Data-Driven Security: The NCSC's publication of its Active Cyber Defense reports, detailing successes and failures, is a model for transparency. Defenders should push for more data-driven approaches and shared intelligence within their sectors to collectively improve defenses.

Key Takeaways

• The cybersecurity industry's language and incentives are fundamentally broken, leading to fear, mismanaged risk, and a lack of accountability.
• Users are not the weakest link; poor system design and user interfaces are the primary culprits. We must adopt "shape coding" for digital security.
• The scale of modern tech giants like Amazon fundamentally changes the calculus of what's possible in security, from formal methods to data analysis, and their impact can exceed that of national governments.
• Technology is inseparable from geopolitics; standards bodies are fragile, and the "balkanization" of technology (e.g., processor architectures, financial systems) is eliminating mutually assured destruction.
• The commoditization of threat (CNO as a service) and sophisticated supply chain attacks (XZ Utils) are making defense increasingly challenging.
• Market incentives are perverse, often rewarding companies financially after major security incidents, rather than penalizing them. This must be fixed for national security in its broadest sense.
• Anthropomorphizing adversaries, users, and especially AI systems (equating emergent behavior with intent) creates dangerous security blind spots.

About the Speaker(s)

Ian Levy is a distinguished figure in the cybersecurity world, known for his candid and often provocative insights. He spent over 25 years at GCHQ, the UK's Signals Intelligence and Security Agency. During his tenure, he was instrumental in establishing and leading the National Cyber Security Centre (NCSC), GCHQ's defensive arm, which consolidated the UK's cyber defense efforts for the entire nation. Currently, Ian serves as a Distinguished Engineer at Amazon, bringing his unique perspective from national security to the challenges of hyperscale cloud operations. Hailing from Yorkshire, England, he is characterized by his sarcastic humor and direct, no-holds-barred communication style. Levy is a mathematician and computer scientist by training, with a deep understanding of the technical intricacies of cybersecurity.