"That's my perspective from 30 years of doing this": An Interview Study on Practices, Experiences, and Challenges of Updating Cryptographic Code

"That's my perspective from 30 years of doing this": An Interview Study on Practices, Experiences, and Challenges of Updating Cryptographic Code

Speakers: Alexander Krause

Conference: USENIX Security

YouTube: https://www.youtube.com/watch?v=zkMcZE6_BlU

Overview

Alexander Krause from SISPA presented a seminal study at USENIX Security, delving into the critical yet often overlooked domain of cryptographic code updates. Titled "That's my perspective from 30 years of doing this," the research highlights the profound complexity and long-term commitment required to maintain secure cryptographic implementations in software. This work addresses a significant gap in software security research by investigating the human and organizational factors that influence how developers manage these updates in practice, moving beyond the traditional focus on cryptographic algorithm design or implementation flaws.

The talk underscores the unique challenges associated with cryptographic code, which, unlike typical software, can lose its security guarantees over time even when functioning correctly. Historical precedents like the decade-long deprecation of SHA-1 serve as stark reminders of this reality. With the ongoing, unprecedented transition to Post-Quantum Cryptography (PQC), understanding the practical difficulties developers face in updating crypto becomes paramount. The study, based on 21 unstructured interviews with experienced developers, offers invaluable insights into awareness, motivation, processes, and challenges, providing a crucial empirical foundation for improving cryptographic security practices across the industry.

This research matters because, despite the common adage "never implement your own crypto," many products still rely on outdated and insecure cryptographic code. If implementing crypto is hard, updating it presents an even more complex tapestry of technical, organizational, and human challenges. By dissecting these real-world experiences, Krause and his colleagues provide actionable recommendations for developers, library maintainers, and standards organizations, aiming to streamline secure cryptographic transitions and mitigate the pervasive risks associated with neglected or poorly managed crypto updates.

Background

▶ Watch: Introduction: The critical challenge of updating crypto code (0:00)

The necessity for cryptographic code updates stems from a fundamental characteristic of cryptography itself: its security guarantees are not static. Unlike other software components that might only require updates for bug fixes or new features, cryptographic algorithms can become vulnerable over time due to advancements in cryptanalysis, increases in computational power, or the discovery of previously unknown weaknesses. This means that a perfectly implemented cryptographic system can become insecure without any change to its code, simply because the underlying mathematical problem it relies on has been compromised or weakened. The transition away from SHA-1, a cryptographic hash function found to be practically vulnerable to collision attacks, exemplifies this, taking over a decade for widespread deprecation, leaving systems exposed for extended periods.

Currently, the industry is on the cusp of the largest cryptographic transition in history: the adoption of Post-Quantum Cryptography (PQC). This transition is driven by the looming threat of quantum computers, which are expected to be capable of breaking many of the public-key cryptographic algorithms widely used today, such as RSA and ECC. While the advice "never implement your own crypto" is well-established, prior research consistently indicates that a significant number of products still contain outdated and insecure cryptographic code, often due to custom implementations or neglected library dependencies. This highlights a critical gap: if implementing secure crypto is arduous, the process of updating these implementations, often under pressure and with evolving knowledge, is even less understood and critically underexplored.

The study presented by Alexander Krause directly addresses this gap. It posits that the challenges of cryptographic updates are not purely technical but deeply intertwined with human and organizational factors. To understand these complexities, the research team conducted 21 unstructured interviews with experienced developers. These participants boasted diverse backgrounds, ranging from 1 to 35 years in development and up to 37 years in security, encompassing roles such as software engineers, CTOs, security researchers, and students across various projects, companies, and open-source initiatives. The participants were broadly categorized into two groups: those who implemented cryptographic libraries themselves and those who integrated existing implementations into their products. This broad recruitment strategy ensured a comprehensive perspective on the entire crypto update ecosystem, allowing the researchers to investigate four key aspects: how developers become aware of needed updates, their motivations for updating, the actual processes they follow, and the myriad challenges they encounter throughout.

Key Findings

▶ Watch: Identified six-phase process for cryptographic code updates (3:00)

The research uncovered a structured, albeit often improvised, six-phase crypto update process followed by developers, alongside a comprehensive list of 13 distinct challenges that permeate this process. This framework provides a critical lens through which to understand the practicalities of maintaining cryptographic security.

The identified six phases of a cryptographic update are:

1. Triggers: Developers become aware of needed updates through various channels, including internal colleagues, GitHub issues, and CVE (Common Vulnerabilities and Exposures) trackers. These triggers often signal a vulnerability or an impending deprecation.
2. Objectives: The motivation for updating varies, ranging from fixing critical vulnerabilities to implementing preventive security improvements or meeting new regulatory or industry standards.
3. Planning: This phase involves assessing the urgency of the update, evaluating its potential impact, and deciding on the most appropriate technical and deployment approach.
4. Implementation: This is where the actual code changes occur. While some updates might be simple library upgrades, others could involve significant refactoring or even algorithm transitions. Most participants indicated handling this independently.
5. Quality Assurance: Given the high stakes, this phase often involves rigorous testing and code review. Many participants reported applying extra scrutiny, sometimes even "six-eye rules," specifically for cryptographic changes.
6. Rollout: The deployment strategy can range from emergency patches for critical vulnerabilities to phased deployments for less urgent updates, allowing for careful monitoring and risk mitigation.

A crucial overarching finding was that most participants lacked structured processes for managing these critical updates, often resorting to improvisation. This ad-hoc approach amplifies risk and inefficiency.

The study identified 13 distinct challenges that span these phases, with several emerging as particularly significant:

• Backward Compatibility: This was a major challenge, as developers consistently struggled with supporting legacy systems, old hardware, or outdated protocol versions while attempting to integrate newer, more secure cryptographic implementations. This issue is particularly acute in embedded environments, where devices might operate for 10 to 15 years without updates.
• Expertise Gaps: Many developers felt overwhelmed by the need to understand complex cryptographic concepts without dedicated in-house expertise. This leads to reliance on informal peer reviews or external consultants, slowing down critical updates.
• Documentation Problems: Participants consistently highlighted issues with documentation. Academic papers were often too complex and theoretical, while practical guidance and developer-friendly explanations were frequently missing.
• Lack of Structured Processes: The absence of established, repeatable workflows meant developers were essentially improvising their approach to these critical security updates, leading to inconsistencies and potential oversights.
• Trust in Third-Party Crypto: Concerns about introducing backdoors or unknown vulnerabilities when adopting or updating third-party cryptographic libraries also created hesitation.

Participants consistently emphasized that cryptographic updates are fundamentally different from regular software updates. The stakes are considerably higher, as one participant starkly put it: "If you don't update, for example, your PDF library, you may have a logo positioned wrongly, but if you don't update a crypto library, you may have your data leaked." This heightened risk often leads to stricter review processes, such as the "six-eye rule" mentioned, where multiple individuals review changes. The technical complexity also differs, requiring not just coding proficiency but a deep understanding of the underlying cryptographic principles.

Two direct quotes from participants powerfully illustrate these recurring themes:

• P3's observation, "Following the right people," captured the challenge of staying informed. Developers heavily rely on personal networks and informal social channels for critical security information. However, identifying trustworthy experts and filtering useful guidance from noise requires significant effort, leading to potential gaps or delays in crucial updates.
• P9's experience, "It has taken 10 years, maybe half a year of real work, but always as a side project," highlighted the severe impact of a lack of dedicated resources and structured processes. Cryptographic updates are frequently deprioritized in favor of other tasks, stretching what should be focused work into years of sporadic, inefficient effort, leaving software exposed to risk for far longer than necessary.

In summary, the key findings reveal a landscape where developers struggle with reliable information, lack formal processes, and face significant technical and organizational hurdles in keeping their cryptographic code secure.

Technical Deep Dive

▶ Watch: Why crypto updates are fundamentally different and high-stakes (4:40)

The technical challenges in updating cryptographic code extend far beyond simple version bumps, intertwining deeply with system architecture, legacy constraints, and the inherent complexity of cryptographic principles. The study illuminates several critical areas where technical hurdles significantly impede secure updates.

At the forefront is the pervasive issue of backward compatibility. This isn't merely about maintaining API compatibility but often involves supporting legacy systems, particularly in embedded environments, where devices can operate for 10 to 15 years without updates. These systems might rely on older hardware, specific protocol versions, or even custom cryptographic implementations that are difficult to modify or replace. Migrating from an older, potentially insecure algorithm (like a weaker hash function or a smaller key size) to a modern, robust one often means maintaining dual implementations or complex compatibility layers, which introduce their own attack surface and maintenance overhead. The architectural decision to support such legacy components can effectively prolong the lifespan of insecure configurations within an ecosystem.

The complexity of cryptographic concepts themselves poses a significant technical barrier. Updating crypto isn't just about changing lines of code; it requires understanding the mathematical underpinnings and security properties of the algorithms involved. For instance, transitioning from an insecure padding scheme to a secure one, or correctly implementing a new mode of operation for a symmetric cipher, demands a deep comprehension of cryptographic primitives and their interactions. Developers often lack this specialized expertise, leading to potential misconfigurations, subtle vulnerabilities, or an inability to properly evaluate the security implications of an update. This gap necessitates not just coding skills but the analytical rigor of a cryptographer.

Documentation shortcomings exacerbate these technical difficulties. While academic papers provide the theoretical foundation, they are often too dense and abstract for practitioners. Conversely, practical guidance, implementation examples, and clear migration paths for specific libraries or standards are frequently missing. This forces developers to reverse-engineer best practices or consult informal sources, increasing the likelihood of technical errors. For instance, understanding the precise implications of a NIST (National Institute of Standards and Technology) recommendation for a key derivation function, and how to correctly implement it in a specific programming language or framework, can be a daunting task without clear, actionable documentation.

The transition to Post-Quantum Cryptography (PQC) is poised to amplify all these technical challenges. PQC algorithms introduce entirely new mathematical structures and security assumptions. Developers have limited experience with these new primitives, and the standards are still evolving (e.g., NIST PQC standards are now being finalized, but implementation details and best practices will continue to mature). This means that initial PQC implementations will likely require multiple subsequent updates as standards solidify and new cryptanalysis emerges. The issues of backward compatibility, expertise gaps, and documentation scarcity will be particularly acute during this unprecedented transition, demanding careful technical planning and the development of robust, future-proof architectures.

The concept of crypto agility emerges as a critical technical design goal. This involves building systems with the foresight to easily swap out cryptographic algorithms and parameters without requiring a complete architectural overhaul. Achieving this requires careful interface design, abstraction layers for cryptographic primitives, and configurable security policies. While often discussed in theory, the study reveals that many existing systems lack this inherent agility, making updates a disruptive and technically challenging endeavor. The "six-eye rule" for crypto changes, reported by some participants, while an organizational process, underscores the technical criticality: such strict review is deemed necessary because even minor changes can have profound security implications that are difficult for a single individual to fully grasp.

Finally, trust in third-party crypto libraries presents a unique technical dilemma. While the advice is to use well-vetted libraries, developers still express concerns about introducing backdoors or vulnerabilities through these dependencies. This necessitates technical due diligence, including reviewing the library's design, its audit history, and the reputation of its maintainers. Without this, integrating an "updated" library might inadvertently introduce new, complex technical risks.

Demo / Proof of Concept

▶ Watch: Participant quotes on information sources and delayed updates (6:00)

The presented work is an interview-based qualitative study, focusing on the human and organizational aspects of cryptographic code updates rather than a technical exploit or a software demonstration. Therefore, the talk did not feature a traditional demo or proof of concept in the sense of executable code or a system showcase.

Instead, the "output" of this research takes the form of empirical findings derived from the detailed analysis of developer experiences. The researchers have made their "application package" available online, which includes their interview guide, code book, and other methodological materials. This serves as a foundational resource for future research in this under-explored area, allowing other researchers to build upon their methodology and findings.

Defensive Implications

▶ Watch: Recommendations for stakeholders and PQC transition relevance (8:30)

The findings of this study provide crucial insights for security defenders, offering actionable strategies to mitigate the significant risks associated with cryptographic code updates. Understanding the real-world challenges developers face is the first step towards building more resilient and secure systems.

For Developers and Engineering Teams:

1. Embrace Crypto Agility from Design: Defenders should advocate for and implement crypto agility during the initial design phases of any system. This means building in mechanisms that allow cryptographic algorithms, key sizes, and protocols to be easily updated or swapped out without extensive refactoring. This foresight can drastically reduce the cost and complexity of future updates, especially in light of the PQC transition.
2. Treat Crypto as Permanent Technical Debt: Cryptographic code should not be viewed as a "set-and-forget" component. Teams must recognize it as permanent technical debt that requires continuous attention, regular audits, and dedicated resources for maintenance and updates. This shifts the mindset from reactive patching to proactive security management.
3. Invest in Specialized Expertise: Organizations should invest in training, continuous learning, and potentially hiring dedicated cryptographic experts. Given the complexity and high stakes, relying on general-purpose developers to navigate intricate cryptographic principles is a significant risk. If in-house expertise isn't feasible, establishing relationships with external cryptographic consultants is a vital alternative.
4. Establish Structured Update Processes: Defenders must push for the creation and adoption of standardized, repeatable, and auditable workflows for crypto updates. This includes clear trigger mechanisms (e.g., monitoring CVEs, security advisories, and industry standards), defined planning phases, rigorous quality assurance (including specific test cases for crypto changes), and well-communicated rollout strategies. This moves away from ad-hoc improvisation.
5. Demand Better Documentation and Tooling: Developers should actively request and contribute to the creation of practical, developer-friendly documentation from library maintainers and standards organizations. This includes clear migration guides, actionable change logs, and concrete implementation examples. Furthermore, advocating for enhanced developer tooling, such as improved APIs, automated update detection (e.g., similar to Dependabot but specifically for crypto deprecations and vulnerabilities), and guided refactoring helpers, can significantly reduce manual effort and error.
6. Implement Enhanced Review Processes: The "six-eye rule" or similar rigorous peer review processes for cryptographic changes should become standard practice. The high stakes warrant extra scrutiny from multiple qualified individuals to catch subtle vulnerabilities or misconfigurations.
7. Proactive PQC Planning: With NIST PQC standards now finalized, organizations must initiate extensive planning and coordination for the migration to quantum-resistant algorithms. This involves identifying all cryptographic assets, assessing their quantum-vulnerability, piloting PQC implementations, and developing phased migration strategies to avoid last-minute, rushed transitions.

For Organizational Leadership and Management:

1. Prioritize Crypto Updates: Management must recognize the critical importance of cryptographic updates and allocate dedicated resources (time, budget, personnel) to them. These updates should not be relegated to "side projects" or deprioritized in favor of new features.
2. Address Legacy System Challenges: Organizations must develop clear strategies for handling legacy systems, especially those in embedded environments that cannot be easily updated. This might involve isolation, clear deprecation paths, or end-of-life planning for systems relying on insecure crypto.
3. Foster a Security-Conscious Culture: Leadership should cultivate an organizational culture that values proactive security, continuous learning, and open communication about cryptographic risks and challenges.

By adopting these defensive strategies, organizations can move from a reactive and ad-hoc approach to cryptographic updates to a proactive, structured, and secure posture, significantly reducing their exposure to evolving cryptographic threats.

Key Takeaways

• Cryptographic updates are fundamentally different and significantly more complex than regular software maintenance. They demand specialized expertise, rigorous review processes, and a deep understanding of evolving cryptographic principles, moving beyond routine engineering tasks due to their high stakes.
• Most developers and teams lack structured, repeatable processes for managing cryptographic updates, leading to widespread improvisation. This ad-hoc approach amplifies the risk of oversights, delays, and prolonged exposure to vulnerabilities, with updates often being deprioritized or handled as "side projects."
• Three core challenges repeatedly emerge: navigating complex backward compatibility requirements (especially with legacy systems and embedded devices), significant expertise gaps within teams (leading to reliance on informal reviews), and information scarcity (struggling to find authoritative, practical guidance amid academic papers and social media noise).
• The looming Post-Quantum Cryptography (PQC) transition will profoundly amplify these existing challenges. With new algorithms, evolving standards, and limited developer experience, the PQC migration will require multiple updates and extensive planning, making the current difficulties with crypto updates a critical precursor.
• Addressing these challenges requires a multi-faceted approach. Key recommendations include building crypto agility into initial designs, investing in education and dedicated cryptographic expertise, demanding better and more practical documentation and tooling (e.g., automated update detection), and establishing standardized, auditable workflows for crypto updates.
• Organizations must treat cryptographic code as permanent technical debt, requiring continuous attention and dedicated resources. This proactive mindset, supported by strong leadership and a security-conscious culture, is essential for maintaining long-term cryptographic security.

About the Speaker(s)

Alexander Krause is a researcher from SISPA (CISPA Helmholtz Center for Information Security), where he focuses on the practical challenges and human factors involved in maintaining software security, particularly concerning cryptographic implementations. His work, including this presented study, delves into how developers actually update cryptographic code in real-world scenarios, exploring the organizational, social, and technical obstacles they encounter. Krause's research aims to bridge the gap between theoretical cryptographic security and its practical application, providing empirical foundations and actionable recommendations for industry stakeholders. He collaborated with colleagues Aljot Kawa, Yan Clemer, Olivivera, and Sasha Fal on this significant research.