Reflections on TTBR & Everest

Reflections on TTBR & Everest

Speakers: Bowen, Blaze, Clark, Hoke, Mulligan

Conference: Voting Village

YouTube: https://www.youtube.com/watch?v=eRW2YjY0QuM

Overview

This panel discussion, "Reflections on TTBR & Everest," brings together key figures involved in two groundbreaking, independent security reviews of electronic voting systems: the California Top-to-Bottom Review (TTBR) and the Ohio Everest study. The talk delves into the historical context, technical challenges, and political battles that defined these pioneering efforts to scrutinize the security and integrity of election technology in the mid-2000s. Led by cybersecurity expert Matt Blaze, with contributions from Candace Hoke, Sandy Clark, Deirdre Mulligan, and a nod to the visionary leadership of then-California Secretary of State Deborah Bowen, the discussion provides a candid look at how technologists gained unprecedented access to voting system source code and hardware, and the profound vulnerabilities they uncovered.

The significance of these studies cannot be overstated. They represented the first official opportunities for independent technologists to conduct privileged reviews of voting systems, with the critical promise of publishing their unedited findings. This was a radical departure from the opaque, vendor-controlled environment that previously characterized election technology. The panelists collectively highlight the visionary nature of the TTBR, which established a model for independent scrutiny that remains relevant today for all public-purpose technology, from automated decision-making to generative AI. Their experiences underscore the enduring challenges of securing complex systems, the vital role of scientific integrity in public policy, and the foundational shift required to acknowledge voting machines as what they truly are: vulnerable computers.

Background

▶ Watch: Introduction to TTBR and initial concerns about findings (0:00)

The genesis of the TTBR and Everest studies lies in a critical period for election technology in the United States. Following the controversies of the 2000 presidential election, the Help America Vote Act (HAVA) of 2002 provided federal funding to states to upgrade their voting equipment, leading to a rapid deployment of new electronic systems. At this time, a prevailing, yet deeply flawed, belief among many election officials nationwide was that "election machinery was not computers," and therefore, computer science and cybersecurity principles were irrelevant to their security. This created a dangerous blind spot, fostering an environment where voting systems were developed and deployed with minimal independent technical scrutiny.

The California Top-to-Bottom Review (TTBR), initiated by then-Secretary of State Deborah Bowen, marked a pivotal moment. It was the first official opportunity for technologists to gain privileged access, including to the source code, of several voting systems used in California. Crucially, the agreement stipulated that researchers could publish their results unedited and without vendor approval. Matt Blaze, a professor at UC Berkeley, was tasked with leading the review of the SEVOIA voting system, assembling a team of graduate students for an intensive eight-week period. The initial concern was whether sufficient vulnerabilities would be found within such a tight timeframe, but the expectation was that early, rushed-to-production systems would yield significant findings.

Inspired by the success and methodology of the TTBR, Ohio's Secretary of State subsequently commissioned a similar review, known as the Everest study. Matt Blaze again led a team, this time focusing on the ES&S hardware predominantly used in Ohio. The Everest team benefited from lessons learned during the TTBR, securing access to both the hardware and software for testing in their lab. Combined, these two studies covered the election equipment used by approximately 90% of American voters, providing an unprecedented glimpse into the state of election security.

However, the Ohio Everest study faced significant political hurdles that highlighted the importance of the TTBR's commitment to scientific integrity. Candace Hoke, then a law professor and head of the Center for Election Integrity at Cleveland State University, served as a crucial go-between and legal representative. She revealed that Ohio's Secretary of State demanded final editorial control over all scientific reports, insisted on NDAs for scientists to prevent disclosure of changes, and only wanted vulnerabilities reported if a quick and complete fix was available, with findings suppressed from public view. Hoke courageously fought against these demands, advocating for the scholars' right to publish every vulnerability found, even if the specific exploit code was relegated to an appendix for election officials. This battle cost the team 12 weeks of valuable working time and ultimately led to Hoke's dismissal for "insufficient loyalty," but it preserved the scientific integrity of the Everest report.

The broader context also included a significant investment from the National Science Foundation (NSF), which funded the ACCURATE center. This interdisciplinary group of researchers, several of whom were involved in these reviews, had already been studying the accuracy, security, reliability, integrity, and privacy of voting systems. This pre-existing network of expertise and the foundational research, including concepts like software independence and risk-limiting audits, proved invaluable when the opportunity for these reviews arose. The legal landscape was also challenging, with vendor contracts typically prohibiting reverse engineering, security testing, or even sharing system documentation, further entrenching the opacity of the election technology ecosystem.

Key Findings

▶ Watch: Ohio SoS's "shocking" different expectations for the study (3:50)

The TTBR and Everest studies unearthed a shocking array of vulnerabilities, demonstrating that early electronic voting systems were far from secure. The most striking discovery, as articulated by Sandy Clark from the Everest team, was that "everything we thought of worked" in terms of compromising the systems to influence election outcomes. Even without being "lead hackers," the team found it alarmingly easy to achieve their objectives.

Among the key findings were:

• Abundant, "Low-Hanging Fruit" Vulnerabilities: The systems, rushed into production following HAVA, contained numerous "stupid bugs" that had gone unnoticed because no independent parties were allowed to scrutinize them. These included easily discoverable flaws that could be exploited by individuals with moderate technical skills.
• Malware Injection and Persistence: Researchers successfully demonstrated the ability to install their own malware onto voting machines. Critically, this malware could be designed to "reinfect the systems that are going to create the next election," indicating a potential for long-term, systemic compromise.
• Hardcoded and Unchangeable Passwords: Basic security hygiene was frequently absent. The teams found instances of hardcoded administrative passwords, with one egregious example where the code comment explicitly stated, "We hard-coded this so that hackers couldn't use it." Furthermore, documentation errors meant that even when election officials tried to change default administrative passwords, the instructions provided by vendors were incorrect or non-functional, leaving systems exposed.
• Outdated and Unpatched Operating Systems: Many systems were running severely outdated and unpatched operating systems, such as Windows CE, Windows 2000, and very early versions of Linux. These systems, some deployed as early as 2012, had no provisions for regular security updates or maintenance, leaving them vulnerable to a host of known exploits.
• Lack of Independent Ballot Design and Verification: The review revealed a concerning reliance on vendors for critical election processes. Many counties, unable to decipher complex documentation, had vendors design their ballots. This vendor-centric approach raised serious questions about the integrity and transparency of ballot creation.
• Physical and Software-Based Miscalibration: For touchscreen devices, even a "little tiny bit of miscalibration" could lead to a voter selecting one candidate but the system recording a vote for another. This highlighted vulnerabilities in the user interface and underlying software that could silently alter election results.
• Time and Resource Constraints: Despite the significant findings, the teams were severely limited by time. The California review had an eight-week window, and the Ohio team lost 12 weeks due to political negotiations. Panelists noted they often spent 48 straight hours typing reports, limited by their "typing speed" in documenting the sheer volume of vulnerabilities they uncovered. This suggests the true extent of vulnerabilities was likely even greater than what was reported.
• Architectural Flaws: Beyond individual bugs, the fundamental architecture of these systems was problematic. The highly distributed nature of US elections, with varying state, precinct, and county regulations, coupled with the use of "off-the-rack computers tweaked a little bit," meant there was no coherent, security-first design philosophy.

These findings collectively painted a stark picture: the electronic voting systems of the era were not merely imperfect; they were fundamentally insecure, built without adequate consideration for cybersecurity principles and deployed without robust, independent oversight.

Technical Deep Dive

▶ Watch: Ohio SoS demanded report editing, NDAs, and suppressed vulnerabilities (4:50)

The technical investigations undertaken during the TTBR and Everest studies provided unprecedented insights into the inner workings of electronic voting systems, revealing critical flaws across hardware, software, and operational procedures.

For the California TTBR, the core technical activity involved source code review. Matt Blaze's team gained privileged access to the source code of the SEVOIA voting system, allowing them to directly inspect the underlying logic and identify programming errors, security weaknesses, and potential backdoors. This level of access was revolutionary at the time, as vendor contracts typically prohibited such deep analysis.

The Ohio Everest study expanded upon this by securing access to both hardware and software, specifically focusing on ES&S hardware. This allowed for a more comprehensive approach, enabling physical analysis and interaction with the devices themselves, beyond just code inspection. The team investigated the entire election process, from the initial ballot design systems to the scanners and tallying machines, encompassing approximately 700,000 lines of code.

Key technical vulnerabilities and observations included:

• Malware Capabilities: The teams demonstrated the ability to introduce custom malware into the voting systems. This malware could perform various malicious functions, such as altering vote counts, manipulating audit logs, or even propagating to other connected systems used in future elections. This highlighted a severe lack of integrity controls and secure boot mechanisms.
• Credential Management Failures: A particularly alarming finding was the prevalence of hardcoded administrative passwords. These static credentials, often embedded directly into the software, could be easily extracted using basic tools like the strings command. One instance was noted where the developer's comment paradoxically stated, "We hard-coded this so that hackers couldn't use it," indicating a profound misunderstanding of security principles. Compounding this, the provided documentation for changing administrative passwords was often flawed, rendering attempts by election officials to secure their systems ineffective. This meant that default, easily guessable passwords remained active, creating wide-open access points.
• Operating System Obsolescence: Many of the systems reviewed were built upon severely outdated and unpatched commercial operating systems, including Windows CE, Windows 2000, and early versions of Linux. These operating systems, some dating back to the late 1990s or early 2000s, had long passed their end-of-life support and were riddled with publicly known vulnerabilities (CVEs). Crucially, the vendor contracts and system designs lacked any provisions for ongoing security updates or patching, leaving these critical machines exposed to a multitude of cyber threats that would be trivial to exploit.
• System Architecture and Interconnectivity: The panel highlighted that the backend systems responsible for ballot creation and vote tallying were often just standard desktop PCs. This integration of commodity hardware and software into a critical infrastructure, without specialized hardening or security considerations, introduced significant risk. The distributed nature of election administration in the U.S.—with different states, precincts, and counties often mixing and matching vendors and configurations—further complicated efforts to secure the ecosystem, creating a patchwork of varying security postures.
• Ballot Design and Calibration Weaknesses: The process of ballot design itself presented vulnerabilities. Counties often relied on vendors to create ballots due to the complexity of the systems and inadequate documentation. This dependence raised concerns about the potential for malicious or erroneous ballot designs. Furthermore, for Direct Recording Electronic (DRE) machines with touchscreens, even minor miscalibration could lead to a voter's intended selection being recorded incorrectly, subtly shifting vote totals without immediate detection.
• Software Independence as a Solution: A key architectural concept discussed was software independence. This principle acknowledges that software will inherently contain bugs and vulnerabilities, and therefore, election systems should be designed so that an undetected change or error in the software cannot cause an incorrect election outcome without being caught by an independent means. This typically involves a voter-verifiable paper record that can be audited, such as through risk-limiting audits. The idea is to shift the security burden from achieving perfect, bug-free software (which is deemed "unachievable") to creating an architectural model that allows for external verification, thus making the overall election system more trustworthy and resilient, even if the underlying software is flawed. This concept has since been incorporated into standards like the Voluntary Voting System Guidelines (VVSG) 2.0.

The detailed technical analysis performed by these teams exposed not just individual bugs, but systemic failures in design, implementation, and maintenance that fundamentally undermined the security and trustworthiness of electronic voting in the United States.

Demo / Proof of Concept

▶ Watch: Shocking ease of exploiting voting systems: "Everything worked." (7:30)

While the "Reflections on TTBR & Everest" panel itself was a discussion and not a live demonstration, the original TTBR and Everest studies extensively involved proof-of-concept (PoC) exploits and practical demonstrations of vulnerabilities. The researchers didn't just identify theoretical weaknesses; they actively exploited them to prove their impact.

Sandy Clark vividly recalled that for the Everest project, "everything we thought of worked" when attempting to manipulate the systems. This wasn't abstract; it meant they successfully demonstrated how a malicious actor could "put my own malware on this" and even get that malware to "go back and reinfect the systems that are going to create the next election." These were practical demonstrations of how an attacker could achieve their goal of getting a particular person elected.

The panel also mentioned that for the Ohio Everest report, while every vulnerability found was published, the "actual PC exploit" was placed in an appendix that was shared only with election officials. This indicates that the teams had working exploit code and could demonstrate the practical implications of their findings directly to the stakeholders responsible for election security. The goal was to tangibly illustrate the risks, such as how easily hardcoded passwords could be found and exploited, or how a miscalibrated touchscreen could alter votes.

Therefore, while no live demo was performed during this specific conference talk, the underlying work of the TTBR and Everest studies was deeply rooted in hands-on, exploit-driven security testing, forming the basis of their compelling and impactful findings.

Defensive Implications

▶ Watch: The crucial role of NSF funding for interdisciplinary research (8:10)

The reflections on the TTBR and Everest studies offer a wealth of defensive implications for securing election systems and, by extension, any public-purpose technology. The core message is a fundamental shift in mindset and approach.

1. Acknowledge Voting Systems as Computers: The most critical defensive implication is to unequivocally recognize that voting systems are complex computers, not simple, isolated "machines." This means they are inherently susceptible to the same vulnerabilities as any other software-driven system, requiring continuous security patches, updates, and rigorous cybersecurity practices. The term "voting computers" should gain more currency to reflect this reality.

1. Mandate Independent, Transparent Security Reviews: The TTBR model, with its commitment to privileged access for independent technologists and the unedited publication of findings, is a visionary standard that must be adopted. This transparency builds public trust and holds vendors accountable. This approach should extend beyond voting systems to all automated decision-making, generative AI, and technology used in public services. Governments must establish contracts with universities and independent experts that allow for tough, unbiased scrutiny.

1. Prioritize Software Independence in Architecture: Given the inevitability of software bugs and potential malicious actors, defenders must prioritize software independence in election system design. This architectural principle ensures that an undetected software error or compromise cannot solely determine the election outcome. Implementing voter-verifiable paper trails and conducting risk-limiting audits (RLAs) are crucial components of this strategy, allowing for independent verification of results even if the underlying software is compromised. This "architecture, not just code" approach reduces the stringent burden on software perfection.

1. Implement Robust Patching and Maintenance Regimes: The discovery of systems running Windows CE, Windows 2000, and early Linux versions without updates highlights a severe defensive failure. Election officials must demand and implement provisions for regular security updates, patching, and lifecycle management for all election technology. Systems must not be allowed to operate on unsupported or vulnerable operating systems.

1. Enforce Secure Configuration Management and Credential Hygiene: The presence of hardcoded passwords and unchangeable default administrative credentials is a basic security flaw. Defenders must enforce strict policies for changing default passwords, ensure accurate and functional documentation for configuration changes, and conduct regular audits to verify secure configurations. Tools should be designed to prevent hardcoding of sensitive information.

1. Re-evaluate Vendor Contracts and Control: The "vendor capture system" where vendors dictate terms, prohibit testing, and control documentation is detrimental to security. Governments and election officials must reform contracts to explicitly allow for comprehensive security testing (including red-teaming), reverse engineering, documentation sharing, and source code escrow. The goal should be to move towards a voter and administration-driven system design rather than solely vendor-driven solutions, potentially exploring open-source models or government-built systems where feasible.

1. Foster Interdisciplinary Collaboration: Election system design and security must account for all values: security, privacy, accuracy, accessibility, and civil rights. Defenders need to facilitate dialogue and collaboration between different stakeholder communities to ensure systems are robust and trustworthy for all users, avoiding situations where vendor choices pit different values against each other.

1. Professionalize Ballot Design and Verification: Relying on vendors to design ballots introduces a single point of failure and potential for manipulation. Election officials should develop in-house expertise or independent, verifiable processes for ballot creation and ensure robust validation mechanisms to prevent miscalibration or erroneous ballot configurations.

1. Continuous and Evolving Security Testing: While the "dumb bugs" found in early systems might be mostly fixed, the "subtle bugs" remain. Defenders must recognize that the "chase the last bug" approach is obsolete. Security testing needs to be continuous, sophisticated, and adaptive, acknowledging that motivated adversaries are not constrained by academic timelines or resources. Investment in ongoing, advanced security research and testing is paramount.

By adopting these defensive postures, election systems can evolve from vulnerable "machinery" to resilient "voting computers" that can withstand sophisticated attacks and maintain public trust.

Key Takeaways

• The California Top-to-Bottom Review (TTBR) and Ohio Everest study were foundational, visionary efforts that provided the first independent, privileged access to electronic voting system source code and hardware, uncovering pervasive and critical vulnerabilities.
• Early electronic voting systems were plagued by "low-hanging fruit" vulnerabilities, including hardcoded administrative passwords, unpatched legacy operating systems (e.g., Windows CE, Windows 2000), and basic capabilities for malware injection and persistence, often due to a lack of security-first design and independent scrutiny.
• The integrity of scientific research in election security is paramount; attempts by political entities to control findings, mandate NDAs, or suppress vulnerability reports (as seen in Ohio) must be vigorously resisted to ensure public trust and accurate risk assessment.
• A fundamental shift in mindset is required: voting systems are complex computers, not simple mechanical "machines." This demands continuous security patching, updates, robust configuration management, and the application of modern cybersecurity principles throughout their lifecycle.
• Software independence, an architectural approach where an undetected software error or compromise cannot cause an incorrect election outcome without being caught by an independent means (e.g., voter-verifiable paper trails and risk-limiting audits), is crucial for building trustworthy elections given the inherent fallibility of software.
• The prevalent "vendor capture" model, characterized by restrictive contracts, lack of transparency, and vendor-driven design, is a major impediment to secure and transparent election technology. A move towards independent oversight, government/voter-driven system development, and open standards is essential.

About the Speaker(s)

The panel comprised several distinguished experts who played critical roles in the TTBR and Everest studies:

• Matt Blaze: A professor at UC Berkeley (then at the University of Pennsylvania), Blaze led the source code review of the SEVOIA voting system for the California Top-to-Bottom Review (TTBR) and subsequently led the review of ES&S hardware for the Ohio Everest study. He is a renowned cybersecurity expert who has significantly contributed to the understanding of election security.
• Candace Hoke: A law professor and head of the Center for Election Integrity at Cleveland State University, Candace Hoke was instrumental in the Ohio Everest study. She served as a public monitor of Kyoga Election Reform and played a crucial legal role in fighting to maintain the scientific integrity of the Everest report against political pressure, ultimately leading to her dismissal.
• Sandy Clark: At the time of the Everest project, Sandy Clark was a graduate student working under Matt Blaze. She was directly involved in the technical work of uncovering vulnerabilities in the Ohio systems and experienced firsthand the challenges of negotiating the right to publish findings.
• Deirdre Mulligan: A professor at Berkeley's School of Information (and previously at the law school during the TTBR), Deirdre Mulligan is a lawyer by training. She emphasized the visionary nature of Secretary Bowen's work and the critical importance of public investment in interdisciplinary research, particularly from organizations like the National Science Foundation (NSF).
• Deborah Bowen (Secretary Bowen): As the former California Secretary of State, Deborah Bowen initiated the groundbreaking Top-to-Bottom Review. Her commitment to scientific integrity, allowing technologists privileged access to systems and ensuring the unedited publication of their findings, was described by the panelists as visionary and set a crucial standard for future election security efforts.