Everything Everywhere All at Once…in 2038

Everything Everywhere All at Once…in 2038

Speakers: Pedro Umbelino (Liaison)

Conference: FIRSTCON

YouTube: https://www.youtube.com/watch?v=MrzTR3glliA

Overview

In his compelling FIRSTCON talk, "Everything Everywhere All at Once…in 2038," Pedro Umbelino, a liaison and security researcher, delivered a stark warning about the impending 2038 Unix epoch rollover and the earlier 2036 NTP rollover. Speaking on behalf of himself and his co-researcher Trey Darly, Umbelino articulated that these are not mere bugs but a systemic class of vulnerabilities poised to disrupt critical infrastructure, security systems, and daily life on a global scale. The presentation underscored the profound and widespread implications of time representation failures, arguing that the lessons from Y2K are insufficient for the unprecedented challenge ahead.

The talk serves as an urgent call to action for the security community, policymakers, and organizations worldwide. Umbelino highlighted that while the "apocalypse" date of January 19, 2038, seems distant, the underlying vulnerabilities can be exploited today through time manipulation, making the threat immediate. He passionately argued for a coordinated, multi-faceted approach involving extensive inventory, rigorous testing, demanding vendor accountability, and implementing harm minimization strategies, emphasizing that 12 years is a dangerously short timeframe to address a problem of this magnitude and complexity.

Umbelino’s presentation resonated with a sense of critical urgency, contrasting the simple Y2K fix with the deep-seated architectural challenges of 2038. He detailed how 32-bit time representations are embedded across countless systems, from industrial control devices and smart TVs to core internet infrastructure and critical services like multi-factor authentication and TLS. The talk aimed to dispel complacency, urging attendees to become "that curious human who refuses to shut up in a meeting" and initiate proactive measures before widespread failures become an unavoidable reality.

Background

▶ Watch: Understanding timestamp rollovers and NTP's 2036 issue (0:45)

The core of the "2038 problem" stems from how computers traditionally represent time. Many systems, particularly those built on the C programming language, store time as a 32-bit signed integer representing the number of seconds since a specific reference date, known as an epoch. For the Unix epoch, this date is January 1, 1970, 00:00:00 UTC. A 32-bit signed integer can store values ranging from approximately -2.147 billion to +2.147 billion. When counting seconds from the Unix epoch, the maximum positive value will be reached on January 19, 2038, at 03:14:07 UTC. At this precise moment, the integer will "roll over" and become a large negative number, effectively teleporting affected systems back to December 13, 1901.

This isn't an isolated issue. A similar, earlier rollover event is anticipated for NTP (Network Time Protocol). NTP, which often uses a 32-bit timestamp counting seconds from January 1, 1900, faces its own rollover in 2036. As Umbelino vividly described, "parts of the internet backbone will literally restart to 1900 if it's not fixed." The pervasive reliance on NTP as a "sacred source of time" means that a lie from an upstream server will propagate downstream, affecting all dependent systems.

The speaker drew a crucial distinction between the 2038 problem and the Y2K bug. Y2K was primarily a display and calculation issue related to two-digit year representations, which, while requiring significant effort, was relatively straightforward to explain and fix. In 1999, roughly 50 million systems were connected. The 2038 problem, however, is a fundamental architectural flaw, a "class of vulnerabilities" rooted in the time_t structure of the C language, making it deeply systemic. Today, over 30 billion systems are connected—600 times more than during Y2K—and this number will only grow. The complexity of 2038, requiring a "crash course on signed integers, unsigned integers, epochs, and rollovers," makes it far harder to convey to non-technical stakeholders, leading to potential complacency.

The analogy of the IPv4 to IPv6 transition was used to illustrate the immense challenge of large-scale systemic changes. Despite IPv6 being first defined in 1995, it took 30 years for Google to report over 50% client adoption. This highlights that 12 years is a profoundly insufficient timeframe for a problem affecting virtually every digital system, many of which are deeply embedded, proprietary, or offline. Moreover, Umbelino warned that 32-bit systems are still being put into production today, and even 64-bit operating systems can harbor vulnerable 32-bit userland programs, drivers, or SDKs that rely on the problematic time_t representation.

Key Findings

▶ Watch: Comparing 2038's scale and complexity to Y2K (6:15)

Pedro Umbelino's research, conducted with Trey Darly, uncovered a disturbing prevalence of vulnerable systems and a systemic lack of preparedness for the impending time rollovers. A significant finding concerns NTP: despite a fix being available in the stable branch of NTPD since 2014, their scans revealed that approximately 40% of NTP servers publicly reporting their version will suffer from the 2036 rollover if not upgraded. Furthermore, an analysis of 2.5 million NTP clients showed that 39% are still speaking version three of the protocol, meaning even if an upstream server is updated, these clients will still roll over.

The core message regarding 2038 is that it is not a single bug but an "entire class of vulnerabilities." This systemic issue is tied to the C programming language's time_t structure, impacting virtually every aspect of computing: operating systems (even userland programs on Windows via old MSVC runtimes), programming languages, standards, protocols, database timestamps, and file timestamps. The problem's reach extends to critical functions like multi-factor authentication, cryptography, and TLS certificates, which rely heavily on accurate time.

Umbelino provided concrete examples of observed failures:

• Automatic Tank Gauges (ATGs): Two different ATGs from different vendors, using distinct technology stacks, failed identically when time-tested. Their authentication systems generated tokens that were always expired, locking out users. This demonstrated a common failure mode across disparate ICS/OT devices.
• Smart TVs: Half of Umbelino's personal smart TVs crashed during testing. Further analysis using sinkhole data from infected devices revealed that out of 127,000 infected smart TVs observed in one week, an estimated 6% would roll over and crash due to running the same vulnerable OS versions he had tested. This highlights the scale of the problem in consumer electronics.
• Smartwatches: A "very expensive" smartwatch entered an unrecoverable boot loop after being time-tested, underscoring the impact on dedicated time-keeping devices.

To quantify the scale of the problem, Umbelino and Darly employed various scanning and heuristic methods:

• NTP Server Scanning: From a sample of 123,000 NTP servers, 50,000 were identified as vulnerable to the 2038 rollover, indicating they might survive 2036 only to fail a couple of years later.
• SNMP Scanning: Out of 15 million SNMP servers, 600,000 reported version/architecture information. From 400,000 parsable instances, half were running 32-bit systems. For a high-confidence subset of 20,000 servers where Linux kernel versions could be extracted, 13,000 were determined to be vulnerable to the 2038 rollover.
• Heuristic Approach (Debian Popularity Contest): By correlating historical data on architecture popularity from sources like the Debian popularity contest, they developed a probabilistic model. This model estimated that around 8% of Debian Apache servers, 16% of overall Unix Apache servers, and 8% of Nginx servers are running on 32-bit systems, inferring potential 2038 vulnerability. This method, while acknowledging biases and limitations, provides directional insight into the scale of the problem where direct observation is impossible.
• Sectoral Impact: Applying these heuristics, the research indicated that sectors like consumer goods (almost 25% vulnerable systems) and education (high absolute count) are particularly exposed.

Crucially, Umbelino emphasized that these **2038 vulnerabilities can be triggered *today***. Insecure NTP and easily spoofed GPS signals allow for time manipulation, which can prematurely trigger rollover-like failures. This transforms a future problem into an immediate security threat, as tracked threat actors are already employing such tactics to deprecate systems. The "doubt is zero" that the rollover will happen; the "fear is uncertainty" about what will fail, how hard it will fail, and where those failures will occur.

Technical Deep Dive

▶ Watch: Approaches to discovering and measuring 2038 vulnerabilities at scale (14:00)

The technical foundation of the 2038 problem lies in the inherent limitations of 32-bit integer representation for time. The Unix epoch, established on January 1, 1970, defines the starting point from which seconds are counted. Many systems, especially those developed using the C programming language, utilize the time_t data type, which is commonly implemented as a signed 32-bit integer. This data type can hold values up to 2,147,483,647 (2^31 - 1). Counting seconds from 1970, this maximum value will be exhausted on January 19, 2038, at 03:14:07 UTC. Upon exceeding this limit, the value "rolls over" to a negative number, specifically -2,147,483,648, causing systems to interpret the time as December 13, 1901, 20:45:52 UTC. This abrupt jump back in time can cause catastrophic failures in any process relying on accurate, forward-moving time.

A distinct but related issue affects NTP (Network Time Protocol). While NTP often uses a 64-bit timestamp for precision, its primary timestamp format, particularly in older implementations, can also suffer from a 32-bit rollover. NTP's epoch begins on January 1, 1900. A 32-bit representation of seconds from this epoch will roll over in 2036. The speaker noted that even if an NTP server is upgraded to a modern version (e.g., NTPv4) that uses 64-bit timestamps, older NTPv3 clients will still interpret time using their 32-bit structures and thus remain vulnerable. This highlights a critical protocol versioning issue where client-side remediation is just as crucial as server-side.

The problem's reach extends far beyond "old" 32-bit kernels. Even modern 64-bit operating systems can host vulnerable components. This includes userland programs, drivers, SDKs, databases, file formats, and protocol glue that might internally use 32-bit time_t structures. For instance, the speaker mentioned that even old C runtimes like MSVC on Windows platforms can utilize the Unix time_t, rendering Windows applications vulnerable despite the underlying OS architecture. This makes the 2038 problem a pervasive software supply chain challenge.

To assess the scale of vulnerability, the research employed several methodologies:

• Server-Side Fingerprinting: For protocols like NTP and SNMP, specific configurations allow for the extraction of operating system versions and architectures. By correlating this information with known vulnerabilities and the system's build date, a "high confidence probe" can be developed to infer susceptibility to 2038. For example, from 123,000 NTP servers, 50,000 were inferred to be vulnerable. Similarly, out of 20,000 SNMP servers providing detailed Linux kernel information, 13,000 were identified as likely to roll over.
• Client-Side Data Analysis (Sinkholes): For devices not typically exposed to direct scanning, such as smart TVs, the researchers leveraged sinkhole data from infected systems. By identifying the OS and build information of these compromised devices and comparing it to their own lab tests, they could assess the vulnerability of large populations of client devices at scale. This indirect approach is vital for understanding risks in pervasive, often hidden, embedded systems.
• Heuristic Correlation: Recognizing the limitations of direct scanning, a probabilistic approach was developed. This involved correlating data from sources like the Debian popularity contest, which tracks architecture usage over time, with the likely build dates of systems. This allowed for an estimation of how probable it is for a system (e.g., an Apache web server) to be running on a 32-bit architecture, even without direct observation. While acknowledged as "gross accuracy" and "directionally correct," this method provides valuable insights into the scale of the problem in various industry sectors. The speaker noted that while banner versions are typically unreliable for vulnerability scanning due to backports, for the 2038 problem, an old banner strongly correlates with an old, unpatched OS that hasn't changed its fundamental time handling.

The speaker also highlighted that the systemic nature of the problem means a simple "patch" is insufficient. Resolving 2038 requires entire operating system changes, moving to 64-bit time_t representations, which entails significant re-engineering, recompilation, and re-deployment across vast, interconnected ecosystems. This massive undertaking is further complicated by the fact that time manipulation via insecure NTP or spoofed GPS signals can trigger these rollover vulnerabilities today, turning a future threat into an immediate exploit vector.

Demo / Proof of Concept

▶ Watch: 2038 vulnerabilities possible today via time manipulation attacks (27:00)

While the talk did not feature a live, in-person demonstration of a system failing due to a time rollover, Pedro Umbelino provided compelling accounts of personal and research-driven proof-of-concept (PoC) testing that confirmed the widespread nature and severe impact of these vulnerabilities. His anecdotes served as vivid "demos" of the problem's reality.

Umbelino recounted his initial skepticism about the 2038 problem, which transformed into alarm after a lightning talk by his colleague Trey Darly. Inspired, he began testing devices in his own environment:

• Automatic Tank Gauges (ATGs): As a security researcher familiar with ICS/OT devices, Umbelino tested two different ATGs. Crucially, these devices were from different vendors, utilized different technology stacks, and were built with different programming languages. Despite these distinctions, both failed in the same manner: their authentication systems generated tokens that were perpetually expired, effectively locking out users. This demonstrated a common failure mode across diverse industrial control systems.
• Smart TVs: Expanding his testing to consumer electronics, Umbelino found that half of his personal smart TVs broke when their internal clocks were advanced past the 2038 threshold. This indicated a pervasive vulnerability in widely deployed devices that are increasingly integrated into critical environments like airports and hospitals.
• Smartwatch: Perhaps the most striking personal failure involved a "very expensive" smartwatch, whose sole purpose is to tell time. When subjected to the time manipulation test, the device entered an unrecoverable boot loop, rendering it completely inoperable.

Beyond personal testing, Umbelino described a large-scale, indirect "demo" using sinkhole data. His company operates one of the world's largest sinkhole operations, capturing telemetry from infected devices. By analyzing data from approximately 127,000 infected smart TVs observed in a single week, he was able to assess that 6% of these would crash due to the 2038 rollover. This assessment was made possible because these TVs were running the same vulnerable operating systems that he had previously identified and tested in his home lab. This illustrated how large-scale data analysis can confirm the real-world prevalence of vulnerabilities observed in controlled PoC environments.

Finally, Umbelino mentioned his experience with time manipulation, stating, "I actually had to do that to test my car for example." This implied the use of techniques like spoofing NTP packets or GPS signals to trigger time-related failures, underscoring that the 2038 problem is not just a future date but a present-day exploit vector. These combined experiences served as powerful, real-world demonstrations of the problem's immediacy, breadth, and severity.

Defensive Implications

▶ Watch: Discussing susceptibility of physical Stratum 0/1 NTP devices (36:30)

Addressing the 2038 problem and its precursor, the 2036 NTP rollover, requires a proactive and multi-layered defensive strategy that extends beyond traditional patching. Pedro Umbelino outlined several critical implications and recommended actions for defenders:

1. Comprehensive Inventory and Dependency Mapping: The first step is to "know your systems" and "know your dependencies." Organizations must identify every system, application, and device that touches time or controls time-sensitive processes. This includes not just obvious servers but also embedded devices, ICS/OT systems, smart devices, and proprietary protocols. If a system relies on a 32-bit time representation anywhere in its stack, it poses a risk.

1. Rigorous Testing and Validation: Scanning provides triage, but "only tests are proof of safety." Defenders must conduct forward-in-time testing on their systems, simulating the 2038 rollover to identify specific failure modes. This should involve isolated test environments to avoid real-world disruption. The speaker specifically noted the difficulty of testing offline or deeply embedded systems, stressing that for complex systems, "there's no other way than test this in STO (System Test & Operations)."

1. Demand Proof and Accountability from Vendors: Organizations, especially government entities and critical infrastructure operators, wield significant purchasing power. They must use this leverage to "demand post-2038 certification" from vendors. This includes asking for a temporal Bill of Materials (TBOM) and requiring vendors to prove, not just claim, that their systems are compliant. For critical systems, particularly those bought with public money, source code and build environment escrow should be mandated at purchase. This provides an option to act if a vendor becomes unresponsive or goes out of business.

1. Triage by Tempo and Harm Minimization: Given the "bounded capacity" to fix everything, organizations must prioritize. Umbelino proposed a triage by tempo approach, focusing on systems where failure has the most immediate and severe consequences:

• Life Safety: Medical devices, emergency communications, air traffic control.
• Command and Control: Systems that could lead to geopolitical instability (e.g., deterrent chains).
• Critical Infrastructure: Power, water, transport, telecoms.
• Economic Systems: Financial services.

For systems that cannot be fully remediated, the focus shifts to harm minimization. This involves design principles to manage failure gracefully:

• Digitalize/Simplify: Remove digital components if they introduce more risk than benefit.
• Hybrid Analog-Digital: Decouple time coordination from critical control paths, ensuring core functionality degrades gracefully rather than failing entirely.
• Human Legible Override: Implement manual fail-back mechanisms at points where silent failures could occur, allowing human intervention.
• Graceful Degradation: Design for staged fallbacks, where systems progressively lose functionality rather than crashing. These strategies, common in aviation and nuclear power, should be applied more broadly.

1. Secure Time Sources and Own Your Clocks: The fact that 2038 vulnerabilities can be triggered today through time manipulation (e.g., insecure NTP, spoofed GPS) highlights the need for robust time synchronization security. Organizations must "own your own clocks," reducing reliance on easily manipulable external sources. This implies implementing secure NTP configurations, potentially using hardened, dedicated stratum 0/1 devices, and monitoring for time anomalies.

1. Integrate into Cybersecurity Exercises: Time-related failures and attacks must be incorporated into existing cybersecurity tabletop exercises. Organizations should ask, "How is time involved?" and "Are you testing for this rollover or for an attack that manipulates time?" This ensures that response plans account for time-dependent failures.

1. Coordination and Information Sharing: The problem transcends organizational and national boundaries. Regional and international collaboration is crucial for sharing risk assessments and best practices. The speaker emphasized that "no nation fixes this alone" and even suggested open communication with adversaries on such systemic issues to prevent global destabilization.

In essence, the defensive posture for 2038 is not about applying a last-minute patch in 2037. It demands systemic changes, proactive testing, strong procurement policies, and a shift in mindset towards designing for resilience and graceful degradation, starting immediately.

Key Takeaways

• Systemic Vulnerability, Not a Bug: The 2038 Unix epoch rollover (and 2036 NTP rollover) is a fundamental, systemic class of vulnerabilities stemming from 32-bit time representations, not a simple software bug. It is a predictable event with a fixed date, meaning "doubt is zero."
• Vastly Larger Than Y2K: With over 30 billion connected systems (600 times more than Y2K), the scope and complexity of the 2038 problem are unprecedented, affecting critical infrastructure, security systems, and daily life across all sectors.
• Immediate Threat through Time Manipulation: While the rollover date is in the future, the underlying vulnerabilities can be exploited today through time manipulation (e.g., insecure NTP, spoofed GPS), posing immediate security and operational risks.
• Pervasive Impact Across All Layers: Vulnerable 32-bit time representations are deeply embedded in operating systems, userland applications, drivers, SDKs, databases, file formats, and protocols, affecting not just old 32-bit kernels but also components on modern 64-bit systems.
• Actionable Steps: Inventory, Test, Demand Proof: Organizations must immediately inventory all time-sensitive systems, conduct rigorous forward-in-time testing, and demand post-2038 certification, temporal Bills of Materials, and source code escrow from vendors.
• Harm Minimization is Crucial: For systems that cannot be fully remediated, prioritize based on impact (life safety, critical infrastructure) and implement harm minimization strategies such as hybrid analog-digital designs, human legible overrides, and graceful degradation to manage inevitable failures.

About the Speaker(s)

Pedro Umbelino is a liaison and security researcher who delivered this talk at FIRSTCON. He has a background in security research, previously focusing on vulnerabilities in industrial control systems, such as Automatic Tank Gauges (ATGs) used in gas stations. His personal experience testing various devices, from ATGs and smart TVs to smartwatches, underscored his deep engagement with the 2038 problem. Umbelino is actively involved in efforts to address time security issues, urging others to join the "time security s." and the "Apocalypse Project."

Trey Darly is Pedro Umbelino's co-researcher on the 2038 problem. Although not present to deliver the talk, he is described as having been involved in the security community for many years and has been "thinking and working on this problem for many years now." Umbelino acknowledged Darly's unique eloquence and the significant joint effort behind the research presented. Darly's initial "elevator pitch" for 2038 at a previous conference was instrumental in sparking Umbelino's deeper investigation into the issue.