There I FIPS'd It - misadventures in federal cryptography

There I FIPS'd It - misadventures in federal cryptography

Speakers: Natalie Somersall, Principal Field Engineer, Public Sector, Chainguard

Conference: DistrictCon

YouTube: https://www.youtube.com/watch?v=QxhtUAVpdXU

Overview

In "There I FIPS'd It - misadventures in federal cryptography," Natalie Somersall, a Principal Field Engineer for the Public Sector at Chainguard, delves into the complex and often misunderstood world of Federal Information Processing Standards (FIPS). The talk critically examines the challenges organizations face in achieving and maintaining FIPS compliance, particularly in modern, distributed computing environments. Somersall highlights that while FIPS aims to provide a secure "paved path" for cryptographic implementations, real-world application often falls short due to a combination of outdated assumptions, technical complexity, and a pervasive lack of understanding among developers, security professionals, and even authorizing officials.

The presentation is a crucial wake-up call for anyone operating within the federal space or dealing with sensitive data that requires stringent cryptographic assurance. Somersall's insights are particularly timely given the recent revisions to standards like NIST 853, FedRAMP 20X, CMMC 2.0, and FIPS 140-3. These updates, coupled with an influx of new companies and cloud-native architectures into the federal ecosystem, create a fertile ground for missteps. The talk provides a candid look at common pitfalls, from superficial compliance checks to outright dangerous workarounds, emphasizing the critical need for deep technical understanding, rigorous validation, and transparent communication regarding an organization's cryptographic posture.

Background

▶ Watch: Introduction to federal cryptography challenges (0:00)

The landscape of federal cryptography is governed by a complex web of standards, primarily maintained by the National Institute of Standards and Technology (NIST). NIST publishes various standards, including Federal Information Processing Standards (FIPS), which dictate requirements for cryptographic modules used by federal agencies. The most pertinent FIPS standards for this discussion are FIPS 140, which specifies security requirements for cryptographic modules, and its sister standard, FIPS 890B, which addresses randomness and entropy. The current iteration, FIPS 140-3, is a critical update to the standard, reflecting advancements in cryptography and computing.

The core concept behind FIPS 140 is to provide a "secure paved path" for implementing cryptography. Rather than individual organizations developing their own cryptographic primitives, FIPS mandates the use of modules that have been independently peer-reviewed and validated to function as expected. This validation is performed under the Cryptographic Module Validation Program (CMVP). The initial FIPS 140 was published in 1994, a time before the widespread adoption of open-source software and easily shareable libraries. Back then, cryptographic modules were typically proprietary, licensed components. The spirit of the law was to ensure that the tricky business of cryptography was handled by trusted, verified components, thereby reducing the likelihood of implementation errors.

However, the real-world application of FIPS compliance is far from straightforward. The talk emphasizes that the Authorizing Official (AO), the individual responsible for accepting risk on a system and signing off on its operational authorization (e.g., FedRAMP authorized), may not be a cryptographer or even a developer. Their role is to understand and accept risk, making clear communication about cryptographic posture paramount. The traditional "Alice and Bob" cryptographic diagram, where two parties need to communicate securely without eavesdropping, simplifies a reality that is far more complex in modern architectures. Today's systems involve thousands of microservices, dozens of cloud regions, and intricate mesh networks, making a single control point for cryptography an outdated notion. The security of such a system is only as strong as its weakest link, a principle often overlooked when focusing solely on perimeter or storage encryption without considering internal data flows.

Key Findings

▶ Watch: Understanding FIPS validated modules and modern complexity (4:00)

Natalie Somersall's talk meticulously uncovers several pervasive "misadventures" and common pitfalls in achieving genuine FIPS compliance, moving beyond superficial checkbox exercises.

1. Misunderstanding the Scope of Protection: A significant issue is where organizations "draw the line" for FIPS compliance. Historically, it was common to place a FIPS-mode VPN at the perimeter and use FIPS-enabled storage. While this covers encryption in transit (VPN) and at rest, it often leaves internal communications between application components (databases, caches, microservices) unencrypted. This creates a glaring gap in the threat model, protecting only against external threats or physical theft of drives, but failing to secure the application's internal data flows. The least acceptable reason for this oversight, Somersall notes, is a lack of understanding or documentation of how data moves within the system.

1. "Compliant but Not Validated" – The DIY Temptation: FIPS 140-3 itself is a concise 11-page document, but it extensively references hundreds of pages of ISO documents, NIST special publications, RFCs, and implementation guidance. This intricate web of specifications often tempts organizations to "write something to spec" themselves, believing they can achieve compliance by adhering to the standards. However, Somersall strongly cautions against this DIY cryptography. The time, cost, and expertise required to not only write such a module correctly but also get it independently validated through CMVP and then maintain that validation over time, far outweigh the typical five-to-six-figure annual cost of licensed, pre-validated modules. DIY solutions are almost invariably not validated, even if they attempt to follow the specification, rendering them non-compliant from a FIPS perspective.

1. Disabling Validation Altogether: Perhaps the most egregious and alarmingly common misadventure is the complete disabling of cryptographic validation, particularly certificate validation. Somersall highlights that this practice, often seen in code as InsecureSkipVerify (Go) or ssl.verify_mode = OpenSSL::SSL::VERIFY_NONE (Ruby/Python OpenSSL bindings), essentially tells the system to trust everyone. While often used as a debugging step, it frequently remains enabled in production environments. This creates a false sense of security where HTTPS or other secure protocols appear to be active, but the underlying trust mechanism is entirely bypassed. Such systems are vulnerable to man-in-the-middle attacks, rendering any encryption effectively useless. Somersall notes that this can often be found through basic code review, dorking in Git repositories (GitLab, GitHub), or static analysis tools, but sometimes "all you need is grep."

1. Creative Ways to Fool Compliance Checks: Somersall exposes a darker side of compliance: deliberate attempts to circumvent or mislead auditors. These "silly" manipulations, though fortunately not frequent, include:

• Editing OpenSSL configuration files to only offer allowed ciphers, creating a facade of compliance.
• Forcing files to falsify FIPS mode, without actual underlying cryptographic validation.
• Using process profiling tools to omit certain files, preventing comprehensive FIPS validation.
• "FIPS mode checkboxes that do nothing but look beautiful on a screenshot."
• Renaming module providers to include "FIPS" in their name without actual validation.
• A particularly illustrative example is mounting a file containing the number 1 to /proc/sys/crypto/fips_enabled inside a container (e.g., on a Fedora host with Rocky Linux containers). This makes the container think it has an OpenSSL FIPS provider, despite the host environment lacking one, creating a completely false sense of security.

1. Language and Environment Specific Challenges: FIPS compliance is further complicated by the diverse ecosystems of programming languages and deployment environments.

• Java JVM: Often doesn't use the host's cryptography, requiring specific configuration and validated modules within the JVM itself.
• Go: Offers numerous cryptographic options, but their FIPS validation status and support vary widely.
• Rust: The standard library intentionally avoids implementing cryptography to preserve portability, meaning developers must explicitly integrate a separate, validated cryptographic implementation.
• Containers: As a "funny way to package a Linux process," containers require special attention. Validated sources of entropy must be available either from the host and passed in, or bundled directly within the container. Similarly, validated cryptographic modules must be present inside the container, not just on the host. The typical lack of shells or OpenSSL tools within minimal container images makes verification challenging, often necessitating API-level checks or sidecar patterns.

These findings collectively paint a picture of FIPS compliance as a deeply technical and nuanced challenge, one that demands more than just superficial adherence to a checklist.

Technical Deep Dive

▶ Watch: The definitive FIPS verification: CMVP certificates (10:00)

Achieving FIPS compliance is a journey into the specifics of cryptographic algorithms, module validation, and operational configurations. The talk provides critical technical details for navigating this landscape.

The foundation of FIPS compliance rests on FIPS 140-3, which, despite being only 11 pages long, is a gateway to a vast ecosystem of cryptographic specifications. It references hundreds of pages of ISO documents, NIST Special Publications, RFCs, and detailed implementation guidance. This layered documentation underscores the complexity of cryptographic implementation, emphasizing why DIY cryptography is almost always a perilous path. Even if one were to meticulously follow all referenced specifications, the lack of independent CMVP validation means the implementation does not meet FIPS requirements.

A crucial aspect of FIPS compliance is understanding approved versus non-approved algorithms. FIPS 140-3 specifies a strict list of algorithms that are permitted for security-related functions. Somersall highlights several common algorithms that are not allowed in FIPS mode:

• RC4
• MD5 (except for non-security purposes like integrity checks, e.g., S3 object hashes)
• ChaCha20 and its associated curves
• Curve25519 for key exchange

Conversely, algorithms like AES-256 are generally approved. When a system operates in FIPS mode, it must disable or refuse to use non-approved algorithms for security functions. This is a common reason why enabling FIPS mode can "break" an application, as it might rely on a disallowed algorithm. An important nuance is that non-approved algorithms can be used for non-security purposes, but this must be explicitly declared and audited. Many FIPS-validated modules will simply omit these algorithms to avoid the audit paperwork, which is a sensible approach.

To practically assess a system's cryptographic posture, Somersall recommends several tools:

• testssl.sh: An open-source command-line tool for checking TLS/SSL cipher suites and configurations of any TLS-enabled service. Its output is easily parsable, allowing comparison of offered ciphers against the list of FIPS-approved ciphers. It also checks many other security aspects of TLS.
• OpenSSL s_client: A powerful command-line utility for interacting with TLS/SSL servers. By echoing a GET command into OpenSSL s_client, specifying the host, port, and a particular cipher, one can test for expected failures (e.g., connection failing with RC4 or ChaCha20) and successes (e.g., connecting with AES-256). This provides definitive proof of which ciphers are supported.
• Nmap SSL enumerate ciphers script: For network scanning, Nmap offers a script that can enumerate SSL/TLS ciphers supported by a service, similar to testssl.sh but integrated into the Nmap framework.

However, Somersall cautions that merely enumerating ciphers from one point in a mesh network doesn't provide a complete picture of the entire system's compliance. The definitive proof of FIPS compliance is the CMVP certificate. This certificate, available on the CMVP website, confirms that a cryptographic module has been validated. When requesting or reviewing a CMVP certificate, it's crucial to check:

• That it is validated and not expired.
• The list of validated algorithms associated with it.
• Any caveats. Common caveats include "only FIPS validated when operated in FIPS mode" (meaning the module needs to be explicitly configured) or "no assurance of minimum key strength" (meaning the application must ensure it provides sufficiently strong keys to the module).

The problem of disabling validation is technically manifested through specific code snippets. In Go, this might appear as InsecureSkipVerify = true in an http.Transport configuration. In other languages or frameworks, it could be setting a verify_mode to VERIFY_NONE or similar. These settings instruct the cryptographic library to bypass certificate chain validation, effectively making the connection vulnerable to man-in-the-middle attacks, even if the data is encrypted. Such instances are typically found through code review, static analysis, or simply grep for these specific strings within a codebase.

Finally, the talk touches on the specific cryptographic architectures of different programming languages and containerized environments. Java's JVM often uses its own cryptographic providers (like JCE), which may not leverage the host's FIPS-validated libraries. Go offers multiple cryptographic implementations, each with varying degrees of FIPS support. Rust's standard library deliberately avoids implementing cryptography for portability, necessitating external, validated crates. Containers, being isolated Linux processes, require that validated cryptographic modules and sources of entropy (like /dev/random or /dev/urandom) are available inside the container, not just on the host. Verifying FIPS compliance within a minimal container often means interacting directly with the library's API or using a sidecar container, as traditional shell tools might be absent. The infamous /proc/sys/crypto/fips_enabled trick, where mounting a file with '1' inside a container makes it believe FIPS is enabled, highlights a critical, low-level manipulation that bypasses actual cryptographic validation.

The session concludes the technical deep dive with a cautionary tale about Artificial Intelligence (AI) for cryptography. Somersall demonstrates asking an AI to implement an RSA algorithm that is FIPS 140-2 compliant. The AI generated code with a prime exponent e=1, which renders the encryption completely ineffective, and other dangerously small prime values (3 and 17) for p and q. This vividly illustrates that while AI can generate code, it lacks the cryptographic understanding and validation rigor required for secure, compliant implementations, reinforcing the need for human expertise.

Demo / Proof of Concept

▶ Watch: Dangers of disabling validation and 'FIPS theater' (14:00)

While Natalie Somersall's talk doesn't feature a live, interactive demonstration in the traditional sense, it effectively presents several practical methods and specific examples that serve as powerful proofs of concept for both detecting and exploiting FIPS compliance weaknesses. These insights are invaluable for auditors, developers, and defenders.

For detecting non-compliant cryptographic implementations, Somersall outlines a clear methodology:

1. Code Review and Dorking: The easiest and most definitive way to identify DIY crypto or disabled validation is through code review. This allows direct inspection of how cryptographic libraries are being used and if validation is being bypassed (e.g., looking for InsecureSkipVerify in Go or similar constructs). When source code isn't available, Somersall suggests dorking in public or internal Git repositories (like GitLab or GitHub) for common strings associated with disabled validation or non-approved algorithms.
2. Network Enumeration Tools:

• testssl.sh: This open-source tool can be run against any TLS service to enumerate supported cipher suites. The output can then be compared against the list of FIPS-approved ciphers. If RC4, MD5, ChaCha20, or Curve25519 are offered for security purposes, it's a clear indicator of non-compliance.
• OpenSSL s_client: For a more targeted and definitive check, OpenSSL s_client can be used. By attempting to establish a connection to a service while explicitly requesting a known non-approved cipher (e.g., RC4-SHA or CHACHA20-POLY1305), and observing a connection failure, one can confirm the system's FIPS mode operation. Conversely, a successful connection with an approved cipher like AES256-SHA confirms proper function. This also serves as a "poem proof" (proof of concept for a screenshot).
• Nmap SSL enumerate ciphers script: This Nmap script offers similar functionality to testssl.sh for network-level cipher enumeration.

For demonstrating manipulative compliance failures, Somersall provides a striking example:

• Falsifying FIPS Mode in Containers: The talk describes a highly deceptive, yet "silly," method to make a Linux container appear FIPS-compliant when it is not. On a Fedora host running Rocky Linux containers (a Red Hat rebuild), which lack Red Hat validated FIPS providers, a malicious actor could simply mount a file containing the number 1 at /proc/sys/crypto/fips_enabled inside the container. This simple act tricks the container's OpenSSL provider into believing it is operating in FIPS mode, even though no actual FIPS-validated cryptographic module is present or active. This specific example serves as a powerful proof of concept for how easily compliance checks can be fooled without genuine underlying security, highlighting the danger of superficial auditing.

The AI-generated RSA implementation with e=1 is another conceptual demonstration, illustrating how relying on generative AI for cryptographic code can lead to fundamentally insecure results, where the "encryption" effectively does nothing. These practical examples, whether through detection methods or direct manipulation, underscore the talk's central thesis: FIPS compliance is not about checkboxes, but about deep, verifiable technical assurance.

Defensive Implications

▶ Watch: Best practices, future hope, and AI cryptography failures (20:00)

The insights from "There I FIPS'd It" offer crucial guidance for defenders navigating the complexities of federal cryptography. The primary defensive implication is a shift from superficial compliance to genuine cryptographic assurance.

1. Understand Your Data Flows: The most fundamental defense is to know your data flows. Documenting how data moves between services, components, and storage within your application is paramount. This enables you to identify all points where encryption is required (in transit and at rest) and ensures that FIPS-validated cryptography is applied comprehensively, not just at the perimeter or storage layer. This addresses the "weakest link" problem.

1. Demand and Verify CMVP Certificates: Always ask your vendors for their CMVP certificates. Don't just take their word for it. Crucially, don't just obtain the certificate; read and understand its caveats. Verify that the certificate is current, covers the specific algorithms you need, and note any operational requirements (e.g., "only validated when operated in FIPS mode"). This ensures that you are using a genuinely validated module and configuring it correctly.

1. Avoid DIY Cryptography: Unless you are a dedicated cryptographic research institution with the resources for independent validation, do not attempt to implement FIPS-compliant cryptography yourself. The cost, time, and expertise required for correct implementation and CMVP validation are prohibitive. It is almost always more secure and cost-effective to pay for a licensed, FIPS-validated module or leverage FIPS-validated services from cloud providers (e.g., AWS, Azure, GCP).

1. Never Disable Validation in Production: The practice of disabling certificate validation (e.g., InsecureSkipVerify) is a critical security vulnerability. While it might be a debugging shortcut, it must never be left enabled in production. Implement rigorous code reviews, static analysis (SAST), and runtime checks to ensure such configurations are never deployed. If a system requires bypassing validation, it indicates a fundamental trust issue that needs to be addressed at an architectural level (e.g., proper certificate management, trusted root CAs), not through disabling security controls.

1. Be Wary of Superficial Compliance: Understand that compliance checks can be fooled. Relying solely on screenshots or simple configuration flags is insufficient. Implement deeper technical verification methods using tools like testssl.sh, OpenSSL s_client, and Nmap to confirm actual cipher suite negotiation and FIPS mode enforcement. Be especially vigilant for deceptive practices in containerized environments, like the /proc/sys/crypto/fips_enabled trick.

1. Leverage Cloud Provider FIPS Services: Cloud providers (AWS, Azure, Google Cloud) increasingly offer FIPS-validated services and infrastructure. By offloading the responsibility for underlying cryptographic infrastructure to these providers, organizations can reduce their own burden of compliance. However, even with cloud services, you still need to understand your application's data flows and ensure you are correctly utilizing the FIPS-validated components offered.

1. Security != Compliance (But Strive for Both): Somersall emphasizes that "security and compliance aren't tightly coupled." There are many excellent, secure open-source cryptographic libraries that are not FIPS validated. If FIPS validation is not a strict requirement for your specific use case, using a well-vetted, secure open-source library is often preferable to a poorly implemented or deceptively compliant FIPS solution. However, for federal use cases, FIPS validation is a non-negotiable requirement. The goal should be to achieve both genuine security and documented FIPS compliance.

1. Budget for Expertise and Second Opinions: Cryptography is hard. FIPS compliance is harder. Budget adequately for people, time, and money to ensure proper implementation and validation. This includes hiring or consulting with cryptographic experts and always seeking a second opinion.

1. Trust But Verify Your Vendors: Approach all vendors with a "trust but verify" mindset. Ask specific, technical questions about their FIPS validation, how their modules operate, and how they handle non-approved algorithms. Don't be afraid to challenge their assertions and request proof.

1. Consult Your Authorizing Official (AO): When in doubt, or when faced with difficult decisions regarding exceptions or interpretations, the definitive answer always comes from your Authorizing Official. They are the risk owner, and their guidance is paramount for navigating compliance ambiguities.

Key Takeaways

• FIPS Compliance is Complex, Not Just a Checklist: Modern architectures and new standards like FIPS 140-3 demand a deep technical understanding, moving beyond superficial "checkbox" compliance.
• Validate Beyond the Perimeter: Ensure FIPS-validated encryption covers all data flows within your application, not just perimeter VPNs or storage, to avoid critical internal "weak links."
• Avoid DIY Cryptography: Developing and validating your own FIPS-compliant cryptographic modules is almost always cost-prohibitive and technically risky; leverage validated commercial or cloud solutions.
• Never Disable Certificate Validation: Bypassing InsecureSkipVerify or similar controls creates a false sense of security and exposes systems to man-in-the-middle attacks.
• Demand and Verify CMVP Certificates: Always request and meticulously review CMVP certificates for cryptographic modules, paying close attention to validity, supported algorithms, and operational caveats.
• Consult Your Authorizing Official: For any ambiguities, exceptions, or critical decisions regarding FIPS compliance, the Authorizing Official (AO) is the ultimate arbiter of accepted risk.

About the Speaker(s)

Natalie Somersall is a Principal Field Engineer for the Public Sector at Chainguard. Her professional expertise spans critical areas of cybersecurity, including application security, supply chain threat modeling, and software factory implementations, specifically within the public sector domain. With a deep understanding of federal requirements and modern architectural challenges, Somersall brings a pragmatic and insightful perspective to the often-complex world of federal cryptography and compliance. Her experience working with various organizations allows her to identify common pitfalls and offer actionable advice for navigating stringent standards like FIPS.