Neither: The space between bug and feature

Neither: The space between bug and feature

Speakers: Daniel Ridge, Hacker

Conference: DistrictCon

YouTube: https://www.youtube.com/watch?v=OzI4ngdbajI

Overview

In his thought-provoking DistrictCon keynote, Daniel Ridge, a seasoned hacker with a deep history in computing, challenges the conventional binary understanding of software behaviors as either "bugs" or "features." Ridge introduces the concept of "neither"—system functionalities or characteristics that exist in the interstitial space between these two definitions. These "neithers" are often intentional design choices from the perspective of a specific stakeholder (e.g., a manufacturer, an integrator, or even a legal department), yet they are not documented or intended for the end-user, appearing as unexpected or exploitable behaviors.

Ridge argues that by focusing solely on offense (exploiting bugs) or defense (patching vulnerabilities), the security community often overlooks the rich context and design intent that give rise to these "neither" states. Drawing on his extensive experience from building early Linux kernels and Beowulf clusters at NASA to leading computer crime forensics, he illustrates how understanding the historical, economic, and operational drivers behind system design can reveal "inevitable conclusions" about system behavior, rather than relying on "happy accidents" of discovery. This talk serves as a philosophical and technical call to broaden our analytical lens, urging security professionals to delve into the often-ignored documentation and supply chain dynamics to uncover these hidden functionalities.

The presentation uses historical examples from early personal computing, dot matrix printers, and modems to demonstrate how these "neithers" emerge from the very processes of manufacturing, testing, and regulatory compliance. Ridge emphasizes that recognizing these deeper motivations allows for a more systematic approach to identifying potential vulnerabilities and understanding system resilience. His core message is that by adopting the perspectives of all customers in a system's lifecycle—not just the end-user—we can unlock a profound understanding of how computers truly operate, moving beyond the "shadow" of the shell to appreciate the full "gorgeous NeXT cube" of computing's potential and pitfalls.

Background

▶ Watch: From HPC hacking to NASA Computer Crime Unit (4:30)

Daniel Ridge's journey into the "neither" space is deeply rooted in a career spanning the evolution of modern computing. Starting his career around 1992, Ridge was among those who built their first Linux kernel from FTP sites, a time when responsible disclosure was not yet a concept, simply because "it just didn't make any sense." His early work at NASA included pioneering efforts like the Beowulf cluster project—building high-performance computing from commodity Linux systems through pure hacking and willpower. This experience exposed him to the raw, unadorned interaction with computing hardware and software, where traditional boundaries were constantly blurred.

Ridge transitioned from HPC hacking to co-founding the NASA computer crime unit. This unit was tasked with responding to intrusions into NASA systems and supercomputers, a necessity because the FBI at the time lacked the expertise to perform forensics on systems like a Cray computer. This role provided firsthand insight into both transnational hacking and the curious case of intruders accidentally landing on the front end of powerful supercomputers without realizing their full potential. This period, including a personal experience with a JPL website defacement that targeted his boss but asked for Ridge's termination, highlighted the fluid nature of "offense" and "defense" in a nascent digital landscape.

His "basement" work following 9/11 and the Afghan wars further cemented his understanding of long-term, deep-seated system behaviors. Emerging from this period, Ridge engaged with his community by getting elected as an ANC Commissioner in Washington D.C., an experience that rekindled his appreciation for the city's rich, often overlooked, technological history. He proudly recounts DC's legacy, from the Navy Yard's role in nuclear weapons to the Smithsonian as the nation's "garage" and Alexander Graham Bell's first optical wireless telephone call from the Franklin School. This historical context for DC, a "neither city nor state," serves as a metaphor for the interfacial spaces he explores in computing—the seams where unexpected behaviors reside.

Philosophically, Ridge invokes Plato's Allegory of the Cave, suggesting that the security community often fixates on the "shadow" (the shell or superficial interface) of a system, rather than appreciating the "gorgeous NeXT cube" (the full, physical, and intended system). He critiques the industry's tendency to accept the framing of problems at face value, particularly the assumption that we (security professionals) are the primary customers of the systems we analyze. Instead, he argues, the true customers—those who "capitalistically handed some cash in exchange for it"—often have very different needs and expectations, which fundamentally shape what constitutes a "bug" or a "feature." This perspective forms the bedrock for understanding the "neither" space, where behaviors are rational and intentional for some party, even if obscure or exploitable to others.

Key Findings

▶ Watch: Plato's Cave and the true customer of computing systems (13:40)

Daniel Ridge's exploration of the "neither" space yields several critical findings that challenge conventional security paradigms and offer a fresh analytical framework:

• The "Neither" as Intentional Design: The core finding is that many behaviors perceived as vulnerabilities or quirks are, in fact, intentional "features" from the perspective of a specific, non-end-user stakeholder. These often arise from manufacturing, testing, regulatory, or legal requirements that dictate system design.
• Documentation as a Primary Source: Ridge repeatedly emphasizes that many "neithers" are openly documented in product manuals, BIOS listings, or technical specifications. Security researchers, often biased towards reverse engineering binaries, frequently overlook these readily available textual resources. He notes that with advancements in tools like AI-driven text analysis, parsing these extensive documents becomes increasingly feasible and efficient (46:00).
• Manufacturing and Calibration Interfaces: A significant source of "neithers" stems from the need to manufacture and test devices at scale. Systems require reliable ways to be calibrated, diagnosed, and flashed before shipping. These factory interfaces, often accessed via physical ports (e.g., keyboard port) or specific command sequences, persist in shipped products and can be leveraged for unexpected control (20:00).
• Technical Debt and Design Serialization: "Neithers" often exhibit remarkable stability and endurance across product generations. Manufacturers frequently leverage existing factory lines, diagnostic tools, or design patterns (technical debt) for new models, preserving these underlying behaviors even as external features evolve (24:00). This creates predictable patterns for discovery.
• Customer-Driven Functionality: The identity of the actual customer profoundly influences what constitutes a "feature." For example, a motherboard vendor's needs differ from an end-user's, leading to different diagnostic implementations. Similarly, a legal department's need for copyright protection influenced IBM's decision to publish BIOS source code (28:00).
• Supply Chain and IP Movement: The movement of intellectual property, licensing agreements, and the necessity for second-source components (e.g., AMD 8088 in IBM PCs due to IBM's insistence on a second source) can expose internal architectural details. Ridge describes IP movement as "the flea on the rat" that reveals communication points and where technical debt is arbitrated (30:00, 49:00).
• Modern Relevance of "Neithers": This concept is not confined to vintage hardware. Ridge points to the RTL-SDR as a contemporary example. These inexpensive devices are primarily TV demodulators whose debugging interfaces, designed for factory testing, are repurposed by users to extract raw I/Q data, essentially turning a manufacturing "neither" into a widespread hacking tool (50:00).
• Ethics and New Domains: Ridge concludes by advocating for the creation of "new domains" in security research, spaces unconstrained by existing ethical frameworks like responsible disclosure. He suggests that while ethics are crucial, they are often cynical, born from crisis, and monotonic (additive only). To truly innovate and understand, the community needs the freedom to explore beyond established boundaries, fostering new "scandals" that can precipitate new ethics (54:00).

Technical Deep Dive

▶ Watch: IBM PC's documented manufacturing backdoor via keyboard (18:00)

Ridge illustrates the "neither" concept through three distinct historical examples, demonstrating how manufacturer and customer perspectives shape system behavior in unexpected ways.

Neither 1: Input – The IBM Portable PC's Hidden Boot Mode

Ridge begins by showcasing his personal 1985 IBM Portable PC, visually demonstrating its ability to boot directly to a BASIC prompt without any floppy disk, hard drive, or external connections, except for a keyboard (17:00). This machine is functionally equivalent to the 1983 IBM XT, an evolution of the original IBM 5150 PC, known for its open architecture and schematics included in the manual.

The "neither" here is a manufacturing-specific boot mode. For IBM, the primary "customer" for this functionality was its internal factory testing unit. To ensure quality control on millions of machines, a reliable method was needed to test the board before assembly, especially given the base configuration often shipped without floppy drives or video adapters (20:00). The only guaranteed interface present in every configuration was the keyboard port.

The mechanism involves grounding a specific pin on the keyboard port, which triggers a special BIOS routine. Instead of the standard keyboard boot byte (AA), sending a different byte (65) to the keyboard controller initiates a "manufacturing boot." This routine allows loading and executing arbitrary code directly from the keyboard port, before the Power-On Self-Test (POST) and even before the CRT controller is initialized (22:00).

Crucially, this behavior was not a secret. IBM, partly to secure copyright protection for its software (software copyright became a thing in 1980), included the full BIOS source code listing with comments in the technical manuals until approximately 1987. Ridge highlights comments like "jump to manufacturing boot" and explicit details about data segments and far calls, making the functionality completely transparent to anyone who read the manual (22:00-24:00).

This "neither" persisted across the IBM PC family (5150, PC/XT, PC/AT 5170, Portable 5155) due to technical debt—the factory had invested in the diagnostic tools and processes for this method. However, it was absent in machines like the PCjr (which used a cartridge slot for diagnostics) or the Convertible PC (which always included a floppy drive, enabling floppy-based diagnostics). This illustrates how the presence of other guaranteed interfaces (cartridge, floppy) altered the diagnostic approach and thus the "neither" behavior (26:00).

Neither 2: Output – The Dymo Label Printer's Reverse Feed

For his second example, Ridge recalls the historical challenge of securing syslog outputs, where clever attackers would "roll that printer backwards and overprint lines" to erase their tracks (32:00). While he didn't bring a vintage Epson FX80 (known for its ESC J rewind command) or MX80 printer, he found a similar "neither" in a modern Dymo label printer.

The "neither" was the printer's ability to reverse-feed paper, an undocumented behavior not explicitly listed as a feature. After a month of fuzzing with "miles of register tape," Ridge discovered the mechanism was tied to an internal timer (32:00-34:00). The Dymo manual hinted at an "automatic reverse feed" after a "full form feed," intended for tearing off labels at a convenient point, but provided no user control (34:00).

The discovery came accidentally, as often happens in fuzzing: if the printer receives no print data for 10.5 seconds between lines, it presumes a form feed occurred and automatically performs a reverse feed. This allows an attacker to "back yourself up the syslog one inch every 10.5 seconds" (36:00). From Dymo's perspective, this was a "rational choice" for user convenience, not a bug, but also not a documented feature for manipulation.

Interestingly, fuzzing around this precise timer threshold (e.g., 10.422 seconds) revealed an actual bug: memory disclosure onto the thermal tape and hard lockups (38:00). This highlights how exploring the "neither" space can inadvertently uncover traditional vulnerabilities, especially those related to watchdog timers and race conditions that are difficult to observe directly. Ridge uses this to illustrate a broader point about the security community's historical focus on systems they "hated" (like Sun Microsystems machines, which were common on desks and thus often crashed by frustrated users) over those they didn't extensively use (like DEC machines with broader support and better documentation) (38:00-40:00). This self-imposed limitation, he argues, prevented critical engagement with mobile systems for many years.

Neither 3: Communication – The Rockwell RCV 336 Modem's Arbitrary Code Execution

The final "neither" involves the Rockwell RCV 336 modem chipset, prevalent from around 1994-1996, and used in modems from various manufacturers like Gateway and US Robotics (41:00). Rockwell's customers were not end-users, but rather the companies integrating their chipsets into final products.

The "neither" is the modem's capacity for arbitrary code execution via an in-band command sequence: +++ followed by at (42:00). This sequence, which allows the modem to pause and accept commands, could be leveraged to load and execute a firmware loader module (FLM). While the manual presented the FLM as a tool for flashing firmware, Ridge points out that the FLM itself is a "programming algorithm for the flash memory," meaning any program could be loaded into the modem's memory and executed (43:00-44:00). Examples of these programs were even included on the driver disks that users typically discarded.

Like the IBM PC example, this functionality was documented in early versions of the Rockwell manual, though later versions "de-emphasized" it (42:00). The ability to load well-formed programs made this a challenging "neither" to fuzz blindly, but a straightforward one to exploit by reading the documentation (44:00).

The "customer" for this "neither" was the modem manufacturer's need for flexibility. Being first to market with a new standard (like V.34) often meant taping out a chip before the standard was fully ratified. The ability to load custom firmware or programming algorithms allowed manufacturers to apply "late breaking changes to the standards ratification," some of which were "almost certainly adversarial by your friends on the standards committee who knew that your chip couldn't do that" (47:00-48:00). This flexibility was a crucial feature for Rockwell and its integrators, enabling them to adapt to evolving standards and maintain market competitiveness.

Demo / Proof of Concept

▶ Watch: Strategies for discovering 'neither' bugs nor features (45:30)

While Daniel Ridge's talk is rich with theoretical and historical examples, he did provide a tangible demonstration of one of his "neithers." He brought his personal, vintage 1985 IBM Portable PC onto the stage (17:00). He powered on the machine and showed the audience that, despite having no floppy disks in its drives, no hard drive, and no other peripherals connected beyond the keyboard, it successfully booted directly into a BASIC prompt (18:00). This live demonstration visually confirmed the existence of a functional computing environment accessible without any external media, serving as a direct, physical example of the "neither" behavior in the context of early IBM PCs.

For the other examples, Ridge described the proof-of-concept work he performed. He recounted "fuzzing three of these [Dymo label printers] for like a month" and going through "miles of register tape and little loops" to discover the reverse-feed mechanism and the associated memory disclosure bug. He humorously noted, "I have the receipts" (32:00). Similarly, for the Rockwell modem, he described the in-band arbitrary code execution capability and its documentation in the manuals, implying that such a PoC is discoverable through careful reading rather than live demonstration. While the Dymo and Rockwell examples were not live-demoed, Ridge's detailed explanation and personal anecdotes lend strong credibility to their existence and his findings.

Defensive Implications

▶ Watch: Hacking as creating new domains beyond existing ethics (53:40)

Ridge's exploration of the "neither" space offers profound implications for defenders, urging a shift in perspective beyond traditional bug hunting and patching:

• Prioritize Documentation Review: Defenders should move beyond solely analyzing binaries and network traffic. Comprehensive review of product manuals, datasheets, source code listings, and historical documentation (including those for deprecated systems or components) can uncover hidden diagnostic modes, undocumented commands, or design rationales that create exploitable "neithers" (45:00). Tools leveraging AI for text analysis can make this task more efficient.
• Understand the Full Supply Chain and Customer Ecosystem: Security assessments must consider the needs and perspectives of all stakeholders involved in a product's lifecycle—from chip designers and manufacturers to integrators, legal departments, and certification bodies. These diverse "customers" often drive design choices that introduce "neithers" for reasons like calibration, customization, or regulatory compliance (47:00).
• Assume Factory/Diagnostic Modes Exist: Every device, particularly embedded systems and IoT, must be manufactured and tested. Defenders should assume the presence of factory modes, debugging interfaces, and internal diagnostic capabilities that are often more privileged and less secured than end-user interfaces. These represent a significant, often overlooked, attack surface.
• Leverage Technical Debt: Recognize that manufacturers often carry forward existing design patterns and factory processes (technical debt) across product generations. This means that if a "neither" was found in an older product from a vendor, similar behaviors might persist in newer, seemingly unrelated products (24:00).
• Scrutinize Trust Proxies: Systems often make assumptions about trust based on seemingly innocuous signals (e.g., a specific byte from a keyboard, a timing interval, or even voice demodulation capability as a proxy for trust in an Alexa device). Defenders should identify these "trust proxies" and assess their robustness against adversarial manipulation (49:00).
• Track IP Movement and M&A Activity: The flow of intellectual property through the supply chain, including licensing agreements and mergers/acquisitions (e.g., Broadcom's "accretion disc" of acquired companies), can reveal where design decisions were made, where technical debt is arbitrated, and where vulnerabilities might reside (46:00, 49:00). These transitions often bring together disparate design philosophies, creating new "neithers."
• Proactive, Design-Centric Security: Instead of reacting to discovered bugs, adopt a more proactive, design-centric approach. By understanding the "inevitable conclusions" of system design choices driven by various customer needs, defenders can anticipate and mitigate potential "neithers" before they are exploited. This requires a deeper understanding of engineering trade-offs and business motivations.
• Broaden Ethical Frameworks: Ridge suggests that the security community's ethical frameworks (like responsible disclosure) are often reactive and self-limiting. While essential, defenders should also foster environments that allow for the exploration of new domains and unconventional research, potentially leading to new insights and the development of more comprehensive security paradigms (54:00).

Key Takeaways

• The "Neither" Space is Critical: Many system behaviors are neither traditional bugs nor end-user features, but rather intentional functionalities for other stakeholders (e.g., manufacturing, legal, or integration). Understanding this "neither" space is crucial for comprehensive security analysis.
• Documentation is a Goldmine: Product manuals, BIOS listings, and historical technical documents are often overlooked resources that explicitly detail "neither" functionalities, providing "inevitable conclusions" about system behavior that may be missed by purely binary analysis.
• Customer Perspective Shapes Design: The true "customer" (e.g., internal factory unit, motherboard vendor, or legal department) and their specific needs dictate design choices that create "neithers." Identifying these primary customers is key to understanding why certain behaviors exist and persist.
• Technical Debt and Supply Chain Reveal Patterns: Legacy design choices and technical debt carried forward across product generations, as well as the dynamics of IP movement and mergers, create enduring and predictable "neither" behaviors that can be systematically uncovered.
• Modern Systems Exhibit "Neithers": The concept of "neither" is not limited to vintage hardware. Contemporary devices like RTL-SDRs leverage factory debugging interfaces as their primary functionality, demonstrating the continued relevance of this design paradigm in modern computing.
• Expand Analytical and Ethical Horizons: Moving beyond a binary offense/defense mindset and embracing a broader, more philosophical approach to system analysis—including the courage to explore new, unconventional domains—can lead to deeper insights and more robust security strategies.

About the Speaker(s)

Daniel Ridge, who describes himself as a "hacker" speaking in his personal capacity, brings a wealth of experience from a long and varied career in computing. His journey began around 1992, at the dawn of the public internet, where he built his first Linux kernel and contributed to the pioneering Beowulf cluster project at NASA, which aimed to create high-performance computing from commodity Linux systems.

Ridge's expertise extended into the nascent field of cybersecurity, where he played a pivotal role in establishing the NASA computer crime unit. This unit was tasked with performing forensics on complex systems like Cray supercomputers, a capability the FBI lacked at the time. His early experiences included dealing with "transnational hackers" and even a personal "website defacement" at JPL.

Following the events of 9/11, Ridge spent an extended period engaged in "basement" work related to the Afghan wars. More recently, he re-engaged with the public sphere by getting elected as an ANC Commissioner in Washington D.C., an experience that re-ignited his passion for the city's rich, often overlooked, technological history.

Throughout his talk, Ridge demonstrates a deep, analytical, and philosophical understanding of computing, drawing connections between historical context, system design, and the often-unseen motivations behind what we perceive as bugs or features. He challenges conventional thinking, advocating for a holistic approach to security that values documentation, understands diverse customer perspectives, and acknowledges the ethical implications of technological advancement.