Bugs In Blocks

Bugs In Blocks

Speakers: Karsten Nohl, SRLabs

Conference: HITB Phuket

YouTube: https://www.youtube.com/watch?v=Rsoyb1heQE4

Overview

In this insightful presentation at HITB Phuket, Karsten Nohl of SRLabs, alongside researchers Lou and Gabri, delved into the often-overlooked yet critically important domain of blockchain security. The talk, titled "Bugs In Blocks," aimed to demystify the complexities of hacking third-generation blockchain systems, specifically those built on the Substrate framework, such as Polkadot and Kusama. Nohl highlighted a significant gap in the security community's engagement with blockchain technology, attributing it to a perceived high barrier to entry. The core message was an urgent call to action: despite its uncertain future as a global critical infrastructure, blockchain technology is a fascinating and puzzle-rich field for hackers, with tangible and often disastrous real-world consequences when vulnerabilities are exploited.

The presentation transcended typical discussions of financial scams or traditional infrastructure attacks, focusing squarely on vulnerabilities inherent to blockchain logic and design. Nohl underscored that even seemingly minor software bugs can lead to catastrophic outcomes, exemplified by a single integer bug that rendered a $65 million blockchain ungovernable. Crucially, the talk emphasized that malicious actors can profit not just through direct theft, but also by short-selling tokens when vulnerabilities diminish a blockchain's viability – turning availability issues or crashes into lucrative opportunities. To empower security researchers, SRLabs unveiled a practical toolchain, including a custom Substrate harness and a fuzzer management tool called Ziggy, designed to lower the entry threshold and enable immediate bug hunting.

Background

▶ Watch: Introduction to blockchain security and motivation (0:00)

The evolution of blockchain technology has progressed through distinct generations, each addressing limitations of its predecessor. The first generation, epitomized by Bitcoin, introduced a decentralized virtual ledger for peer-to-peer monetary transactions. Its simplicity contributed to its robust security. The second generation, pioneered by Ethereum, expanded this concept by introducing smart contracts, allowing for complex, self-executing agreements to live on the blockchain. While revolutionary, Ethereum's generic, "everything for everyone" approach led to scalability challenges.

The third generation of blockchains, the focus of Nohl's talk, emerged to tackle these scalability issues by enabling custom-built, application-specific blockchain systems. Projects leverage foundational frameworks like Substrate to create hundreds of specialized blockchains, designed to interface within a broader ecosystem. This modular, customized approach, while offering immense scalability, introduces a new layer of complexity. Each custom blockchain, or "runtime," built from pallets (reusable code modules or libraries), presents a unique attack surface. This intricate, distributed architecture, coupled with the financial incentives tied to blockchain stability, means that bugs—even those seemingly innocuous in traditional systems—can have profound and often devastating financial impacts, extending beyond direct token theft to market manipulation via short-selling. The problem is exacerbated by the relatively small number of security professionals actively auditing these complex systems.

Key Findings

▶ Watch: Understanding third-generation blockchains and Substrate (9:20)

SRLabs' extensive research into various blockchain projects has uncovered a significant number of vulnerabilities, with Karsten Nohl stating that his team has found "well over a 100 bugs" across dozens of analyzed blockchains. A critical finding is that the vast majority of these vulnerabilities fall into five distinct categories, listed in order of frequency:

1. Wrongly Set Weight / Exponential Complexity: These bugs arise from miscalculating the computational cost (or "weight") of executing certain instructions. If an instruction's weight is vastly underestimated, an attacker can submit a transaction that consumes an excessive amount of processing time, causing the blockchain to "choke" and miss block production, leading to a denial of service (DoS).
2. Diverting Program Flow: This category encompasses vulnerabilities that manipulate the blockchain's intended logic. This can range from arithmetic errors like integer overflows or underflows to subtle race conditions or reordering attacks that allow for unauthorized transaction manipulation or even token theft.
3. Crashing Nodes: Unlike traditional systems where a crash might be considered a fail-safe, a node crash in a distributed blockchain system can halt block production, severely impacting the chain's availability and viability. These often stem from unhandled errors or unexpected states that cause the program to panic! (a Rust-specific term).
4. Abusing Configuration Settings: Developers often misconfigure standard parameters, sometimes inadvertently leaving development-time settings in production. These misconfigurations can be exploited to slow down or bloat the chain, diminishing its performance and credibility.
5. Permanent Choking / Storage Bloating: These are particularly insidious bugs that allow an attacker to fill the blockchain's scarce, permanent storage with arbitrary data at an artificially low cost. Such attacks are difficult to recover from, causing long-term, persistent damage to the chain's operational integrity and value.

Nohl estimates that, on average, security researchers can expect to find between two to three bugs per blockchain project analyzed, with fuzzing alone often yielding one to two vulnerabilities. This highlights the prevalence of these issues and the urgent need for more security scrutiny in the blockchain space.

Technical Deep Dive

▶ Watch: Example: Identity validation race condition via transaction tips (24:30)

The technical core of "Bugs In Blocks" lies in the detailed exploration of the five bug categories, each illustrated with real-world examples from Substrate-based runtimes, primarily programmed in Rust.

1. Wrongly Set Weight / Exponential Complexity

Blockchain operations consume computational resources, and in systems like Substrate, each instruction has a pre-computed "weight" that determines its cost. If this weight is set too low for a given instruction, an attacker can craft a transaction that appears cheap but is computationally expensive, leading to a Denial of Service (DoS).

• Underpriced Instruction: A common vulnerability involves a single instruction whose weight is significantly underestimated. An attacker can submit a transaction containing this instruction, causing nodes to spend an inordinate amount of time processing it. This can lead to nodes missing the deadline for producing the next block, effectively halting the chain. Nohl emphasized that this is "shockingly common" and "completely voidable" through proper benchmarking during development.
• Exponential Complexity in sudo Calls: A more sophisticated variant involves functions that, when chained, exhibit exponential complexity. Nohl presented an example where a sudo function (akin to root privileges in Linux) was designed such that chaining multiple calls would cause a security function to branch twice in each iteration. For instance, just 40 or 41 chained calls could lead to a trillion iterations for a simple privilege check. This exponential explosion in computation would overwhelm nodes, preventing them from generating the next block and causing the chain to choke, even before reaching the actual privilege check. The attacker here incurs minimal initial cost but crashes the entire chain.

2. Diverting Program Flow

These bugs manipulate the blockchain's logic, often with severe financial consequences, including direct token theft or causing system instability.

• Integer Overflow in Proof Calculation: This class of bug exploits arithmetic operations that exceed the maximum value a data type can hold, causing it to "wrap around" (e.g., a 32-bit unsigned integer overflowing back to zero). Nohl described a bug where a security function, intended to verify a proof, involved summing values. An attacker could supply inputs that quickly summed up to exceed 2^32, causing an integer overflow. This would lead the node to perform an incorrect, extremely long computation, causing it to miss its block production window and halt the chain.
• Mishandling checked_sub for Underflows: Rust provides functions like checked_add, checked_sub, etc., which prevent arithmetic overflows/underflows by returning an error instead of wrapping around. While intended as a security measure, Nohl showed how improperly handling this error could still lead to a DoS. If a checked_sub operation, for instance, detects an underflow and the program simply stops execution (e.g., by panicking), the blockchain node will fail to compute the next block, effectively stopping the chain. The devil, Nohl noted, is in gracefully handling such exceptions without halting the entire system.
• Logic Bug: Identity Spoofing via Transaction Priority: This intricate logic bug involves exploiting the distributed and asynchronous nature of blockchain transaction processing. In a system where a registrar validates identities (similar to Twitter's blue checkmark), an identity is submitted for verification in one block. The registrar then processes this and issues an approval in a later block. An attacker can use transaction priority (by "tipping" or paying higher transaction fees) to sneak a different identity into a block before the registrar's approval for the original identity is processed. The registrar, operating on the older state, would then inadvertently stamp the attacker's fraudulent identity as valid, allowing the hacker to impersonate legitimate entities and potentially steal tokens.

3. Crashing Nodes (Panics)

In Rust, a panic! signals an unrecoverable error, causing the program to stop. While acceptable in many applications for "fail-safe" behavior, in a blockchain, a panic in a critical path means the node stops producing blocks, disrupting the entire chain.

• unwrap() on Illegitimate Values: A simple yet common example involves using unwrap() on an option or result type when the underlying value might not be legitimate. If unwrap() is called on a None or Err variant, it causes a panic. Programmers from other domains often use unwrap() for convenience, but in a blockchain context, this can lead to nodes crashing if unexpected input or state is encountered, again halting block production.

4. Abusing Configuration Settings

Even standard, seemingly benign configuration parameters can be exploited if not set correctly.

• Low existential_deposit: Many blockchains require a minimum "existential deposit" for an account to be considered active, preventing the creation of an infinite number of dust accounts. During development, this is often set to zero or a very small value for ease of testing. If developers forget to raise this to a sufficiently high value in production, an attacker can continuously create new accounts with tiny amounts of tokens. This bloats the chain's state, increases storage requirements, and slows down transaction processing, subtly degrading the chain's performance and credibility without an outright crash.

5. Permanent Choking / Storage Bloating

These are perhaps the most malicious types of attacks, as their impact is long-lasting and difficult to reverse.

• Underpriced Storage Writes: Blockchain storage is a finite and extremely valuable resource, as data is replicated across all nodes and stored forever. Nohl explained a bug where an attacker could exploit an underpriced function (e.g., one intended for registering a "bridge") that had an unintended side effect of writing a significant amount of data to the chain's storage. While storing a gigabyte on a system like Ethereum could cost "millions of dollars," this vulnerability allowed an attacker to achieve the same effect for a mere "$25,000." The goal is not to store the attacker's own data, but to fill up the chain's storage with "garbage" or irrelevant information, making it unusable or extremely expensive for legitimate operations. Recovering from such an attack is exceptionally difficult, often requiring a complete restart or a complex hard fork, leading to a "long-term persistency of the impact."

Demo / Proof of Concept

▶ Watch: Introduction to fuzzing for blockchain bugs (34:00)

A central component of the talk was a pre-recorded demo showcasing SRLabs' fuzzing tools designed to find these vulnerabilities in Substrate runtimes. Fuzzing is an automated testing technique that involves feeding a program with unexpected, invalid, or semi-random data inputs to discover bugs, crashes, or security vulnerabilities.

Fuzzing Fundamentals

Nohl explained that modern fuzzers are "coverage-guided," meaning they don't just use pure randomness. Instead, they start with a corpus of legitimate inputs, then intelligently mutate them. If a mutation leads to a new code path being executed, that mutated input is added to the corpus, allowing the fuzzer to explore deeper into the program's logic and discover "corner cases" that might trigger bugs like overflows or excessive computation.

SRLabs' Toolchain

To facilitate blockchain fuzzing, SRLabs developed two key open-source components:

1. Ziggy: This is a fuzzer management tool that orchestrates multiple best-in-class fuzzers, specifically AFL++ and Honggfuzz. Ziggy allows these fuzzers to run in parallel, sharing their findings and corpus to accelerate bug discovery.
2. Substrate Harness: A custom-built harness is essential for fuzzing complex systems like blockchain runtimes. This harness sets up a realistic execution environment for the fuzzer, including creating user accounts, assigning base balances, and configuring the runtime in a way that allows the fuzzer to send legitimate-looking transactions and interact with dispatchable methods. Without such a harness, the fuzzer would likely fail to trigger any meaningful code paths.

Demo Scenario

The demo targeted the Node Template, a standard template runtime included in the Substrate repository. For demonstration purposes, a deliberate integer overflow bug was inserted into the template's template_pallet module. This module contained two functions: do_something, which takes a u32 (32-bit unsigned integer) and stores it, and cause_error, which retrieves the stored value, adds one to it, and then stores the new value. The inserted bug modified cause_error to perform a simple new = old + 1 without checked_add, allowing for an overflow.

Fuzzing in Action

The fuzzer was launched on a server, targeting the modified runtime. Crucially, the fuzzer compiles the binaries with overflow checks enabled, which is the default behavior in Rust's debug mode. This means that instead of silently wrapping around, an arithmetic overflow or underflow would cause the program to panic!. This panic condition is precisely what the fuzzer is designed to detect.

Within approximately 20 to 30 minutes, the fuzzer successfully identified the crash. The output clearly showed:

• A generated input file, which, when executed, first called do_something with a value close to u32::MAX.
• Subsequently, it called cause_error.
• The program then panicked with the message "attempt to add with overflow," pointing directly to the exact line (pallet_template/lib.rs:101) where the overflow occurred.

This demonstration effectively illustrated how SRLabs' tools can quickly and automatically uncover critical vulnerabilities in Substrate-based blockchains, significantly lowering the barrier for security researchers to contribute to blockchain security.

Defensive Implications

▶ Watch: Q&A: Deeper dive into storage bloating vulnerability (50:10)

The insights from "Bugs In Blocks" offer crucial guidance for both blockchain developers and security practitioners seeking to secure these complex systems.

For Blockchain Developers:

• Rigorously Benchmark and Set Weights: Developers must invest in thorough benchmarking of all runtime instructions to accurately determine their computational cost. Underpricing any operation can lead to catastrophic denial-of-service attacks. This is a fundamental responsibility that, if neglected, creates a glaring vulnerability.
• Implement Robust Error Handling: Avoid using unwrap() in production code where None or Err states are possible. Instead, use match, if let, or ? operator for explicit error handling. Crucially, when using Rust's checked_ arithmetic functions (e.g., checked_add, checked_sub), ensure that the None case (indicating an overflow or underflow) is handled gracefully without causing a panic! that would halt the entire chain. A blockchain must be resilient to unexpected inputs and states.
• Scrutinize Configuration Settings: Pay meticulous attention to default and development-time configuration parameters, especially before deploying to a live network. Parameters like existential_deposit must be set to values that prevent abuse, such as infinite account creation and chain bloating.
• Analyze Function Side Effects and Storage Costs: Developers must be fully aware of all side effects of their functions, particularly those that interact with the blockchain's permanent storage. Any function that writes to storage should accurately reflect that cost, and potentially include mechanisms like deposits for long-term storage that are only reimbursed upon removal. This prevents storage bloating attacks where an attacker can fill the chain with garbage at an artificially low price.
• Consider Transaction Ordering and Priority: Design logic to be resilient against race conditions or reordering attacks that exploit transaction priority mechanisms. If the order of operations is critical (e.g., in identity validation), ensure that the system can detect and prevent malicious reordering, or that the logic is robust enough to handle non-sequential processing without critical flaws.

For Security Researchers:

• Embrace Fuzzing: The talk unequivocally demonstrated that fuzzing is an accessible and highly effective method for discovering critical bugs in blockchain runtimes. Researchers, even those new to blockchain, should leverage tools like SRLabs' Ziggy and the Substrate harness to begin their security analysis.
• Focus on the Runtime: The runtime component of third-generation blockchains presents a rich attack surface. Understanding how pallets interact and how custom logic is integrated is key to identifying vulnerabilities.
• Expect to Find Bugs: Karsten Nohl's assertion that one can expect to find 1-2 bugs per project with fuzzing alone should encourage more security professionals to enter this field. The "low-hanging fruit" is still plentiful.

Key Takeaways

• Blockchain vulnerabilities extend beyond direct theft; even seemingly minor bugs like crashes or slowdowns can be financially exploited through market mechanisms like short-selling.
• Third-generation blockchains, particularly those built on Substrate, introduce significant complexity due to their customizability and distributed nature, making them highly susceptible to specific classes of bugs.
• Common vulnerabilities include wrongly set weights leading to DoS, integer overflows/underflows causing logic errors or crashes, logic flaws enabling impersonation, unhandled panics, and exploitable configuration errors like low existential_deposit.
• Storage bloating is a particularly insidious attack that allows an adversary to fill the blockchain's permanent storage cheaply, causing long-term, difficult-to-recover damage to the chain's viability.
• Fuzzing is an accessible and highly effective automated technique for discovering these vulnerabilities, even for security researchers new to blockchain.
• SRLabs provides open-source tools, including the Ziggy fuzzer management system and a custom Substrate harness, to significantly lower the entry barrier for security auditing of blockchain runtimes.

About the Speaker(s)

Karsten Nohl is a prominent security researcher from SRLabs, specializing in critical infrastructure security, including Telco networks, banks, stock exchanges, and utilities. Based in Bangkok, he has a long-standing association with the HITB conference series. In this talk, Nohl highlighted his personal interest in blockchain as a "super interesting technology" that generates "hacking puzzles."

The research presented in "Bugs In Blocks" was a collaborative effort, primarily driven by Lou and Gabri from SRLabs. Lou presented the technical demo of the fuzzing tools and joined for the Q&A session, providing additional technical depth and clarification on the storage bloating vulnerability.