Long Live Short Lived Credentials - Auto-rotating Secrets At Scale

Long Live Short Lived Credentials - Auto-rotating Secrets At Scale

Speakers: Dwayne McDaniel

Conference: BSidesSF 2024

YouTube: https://www.youtube.com/watch?v=BF3mF9lF4o0

Overview

In an era where digital security incidents are increasingly common, the management of credentials has emerged as a critical vulnerability. Dwayne McDaniel, a Developer Advocate at G Guardian, delivered a compelling talk at BSidesSF 2024 titled "Long Live Short Lived Credentials - Auto-rotating Secrets At Scale," addressing the pervasive and growing problem of leaked and hardcoded secrets. The presentation underscored that leaked credentials are now the number one root cause of breaches, a stark reality highlighted by incidents like the Cloudflare and Uber breaches, which involved attackers leveraging compromised credentials to gain unauthorized access.

McDaniel's talk serves as a crucial wake-up call for organizations grappling with an ever-expanding attack surface. He argues that the traditional methods of credential management, inherited from a bygone era of computing, are no longer fit for purpose, especially given the exponential growth of machine-to-machine communication. The core of his proposed solution revolves around a two-pronged strategy: centralizing all secrets into a robust secret manager and implementing automated rotation policies. This approach aims to significantly reduce the window of opportunity for attackers and mitigate the impact of inevitable credential leaks.

The significance of this topic cannot be overstated. As G Guardian's research reveals, millions of hardcoded secrets are publicly exposed on platforms like GitHub annually, with a staggering 90% remaining valid days after discovery. This talk provides not only a sobering assessment of the current landscape but also a practical, step-by-step roadmap for organizations to mature their secret management practices. McDaniel also casts a vision for the future, advocating for a paradigm shift towards credential-less workload identities using frameworks like SPIFFE/Spire, ultimately striving for a more secure and resilient operational environment where security incidents stemming from static credentials become a relic of the past.

Background

▶ Watch: G Guardian report: 12.8 million hardcoded secrets on GitHub public in 2023, 9... (03:00)

The landscape of cybersecurity has fundamentally shifted, with leaked credentials now identified as the primary vector for breaches. Dwayne McDaniel opened his talk by citing reports from Sophos, IBM, and CISA, all converging on the conclusion that compromised credentials are the number one root cause of security incidents. This isn't a new phenomenon, but rather an escalating crisis. Attackers consistently follow established playbooks, such as those outlined by MITRE ATT&CK or the Lapsus Playbook, where the acquisition of credentials is a foundational step. Real-world examples underscore this severity: the Uber breach, for instance, began with the purchase of super admin credentials for a mere $20 on the dark web, demonstrating the low barrier to entry for high-impact attacks. Similarly, a complex situation at Cloudflare necessitated the rotation of 5,000 secrets after an attacker gained access via Okta credentials.

G Guardian, McDaniel's employer, conducts extensive research into the prevalence of hardcoded secrets. By analyzing every new commit to GitHub public repositories—a staggering 1.13 billion commits last year—they've uncovered a deeply troubling trend. In 2023, G Guardian identified 12.8 million unique hardcoded secrets exposed publicly. This number represents a significant increase from the 10 million found in 2022, indicating a growing problem rather than a shrinking one. Even more alarming is the statistic that 90% of all valid secrets discovered remained active and usable five days after their initial exposure. This persistence provides attackers with ample time to exploit compromised credentials, highlighting the critical need for more dynamic secret management.

McDaniel delved into the historical context of how organizations arrived at this precarious state. Early computing, characterized by paper tapes and batch functions, relied on physical security—a guard with a gun, as Grace Hopper famously quipped. The advent of networked machines introduced usernames and passwords to control access and resource usage. The rise of the internet and the World Wide Web simply extended this model, securing web traffic with TLS/SSL. However, the modern era of DevOps and cloud-native architectures has introduced an unprecedented scale of machine-to-machine interaction. In 2022, CyberArk reported a 45:1 ratio of machine resources to humans, a number that has undoubtedly grown since. This exponential increase in machine identities, each requiring some form of authentication, has overwhelmed the traditional, human-centric credential management model. The speaker emphasized that while this model was a necessary evolution, it is no longer sustainable for securing machine workloads. The challenge now is to transition to a more appropriate and automated system without disrupting existing infrastructure, acknowledging that a complete "rip and replace" is often impractical.

Key Findings

▶ Watch: Introduction of core solutions: eliminate credentials (IAM roles), MFA, and a... (07:00)

Dwayne McDaniel's presentation illuminated several critical findings regarding the state of secret management and its impact on cybersecurity:

• Leaked Credentials as the Primary Breach Vector: Multiple authoritative sources, including Sophos, IBM, and CISA, now identify leaked or compromised credentials as the number one root cause of security breaches. This highlights a fundamental shift in the attack landscape, moving beyond traditional perimeter defenses to focus on identity.
• Explosive Growth of Hardcoded Secrets: G Guardian's analysis of public GitHub repositories revealed an alarming trend. In 2023 alone, 12.8 million unique hardcoded secrets were discovered in new commits. This figure represents a substantial increase from the 10 million found in 2022, indicating that the problem is escalating rapidly.
• Persistent Vulnerability of Leaked Secrets: A critical finding from G Guardian's research is that 90% of all valid secrets found publicly remained active and exploitable five days after their initial discovery. This extended window of vulnerability provides attackers with ample time to leverage compromised credentials, significantly increasing the risk of a successful breach.
• Outdated Credential Management for Machines: The current paradigm of using human-centric credentials (like usernames and passwords or API keys) for machine-to-machine communication is fundamentally broken. With a reported 45:1 ratio of machine resources to humans, the scale and complexity of managing these identities manually or with long-lived static credentials are unsustainable and inherently insecure.
• A Two-Part Strategic Solution: McDaniel proposed a clear, actionable strategy to address these challenges:

1. Centralize Secrets: Consolidate all organizational secrets into a single, cohesive secret manager. This provides a secure, encrypted repository and a standardized method for access.
2. Automate Rotation: Implement auto-rotation policies for all secrets. This proactive measure ensures that credentials are regularly refreshed, drastically reducing the lifespan of any potentially leaked secret.

• The Vision for Credential-less Workload Identities: Looking beyond immediate fixes, the talk emphasized the ultimate goal of moving away from traditional credentials for machine workloads entirely. Frameworks like SPIFFE (Secure Production Identity Framework For Everyone) and its implementation Spire offer a path toward generating short-lived, runtime-specific identities, thereby eliminating the need for static, long-lived secrets for machines.

These findings collectively paint a picture of a critical security challenge but also offer a clear, progressive pathway toward more robust and automated secret management practices.

Technical Deep Dive

▶ Watch: Definition and key features of a secret manager: encryption, runtime access, ... (11:00)

The core of Dwayne McDaniel's technical discussion centered on a two-part plan for addressing the pervasive issue of leaked and long-lived credentials: centralizing secrets and implementing auto-rotation. This approach is designed to be incremental, acknowledging that a complete overhaul is often impractical.

Part 1: Gather All Secrets into One Cohesive Secret Manager

McDaniel defined a secret broadly as "anything that unlocks or locks data, crypts or encrypts data, or gives access to another system." This encompasses traditional credentials, API keys, database connection strings, and more. The primary goal is to move these secrets out of hardcoded locations (e.g., source code, .env files, plain text config files) and into a dedicated secret manager.

A secret manager is not merely a storage solution; it's a critical component of a secure infrastructure. Its key functionalities include:

• Secure Storage and Encryption: Secrets are stored in an encrypted format, inaccessible in plain text unless explicitly retrieved through authorized means.
• Safe Runtime Access: It provides a secure mechanism for applications and services to retrieve secrets at runtime, ensuring they are only exposed when absolutely necessary.
• Secure Transport: Secrets are transported securely across the network to the requesting service.
• Environment Agnostic: A truly effective secret manager must operate consistently across all environments—development, staging, production, and across different cloud providers or on-premises infrastructure. McDaniel stressed that having multiple, disparate secret management solutions across environments (e.g., "five versions of Vault systems") undermines the entire strategy.
• Centralized Reporting and Logging: Comprehensive logs are essential for auditing, incident response, and understanding secret access patterns. "If you don't have logs, you ain't got nothing," McDaniel stated, emphasizing their importance.
• Developer Friendliness: The solution must be easy enough for developers to adopt without significant friction, avoiding resistance that could lead to workarounds.

For organizations operating in a monocloud environment, the path is relatively straightforward. Major cloud providers offer native solutions:

• AWS Secret Manager
• Google Cloud Secret Manager
• Azure Key Vault

McDaniel noted a personal preference for Azure Key Vault's documentation but clarified that all three are robust options for single-cloud deployments.

For the vast majority of organizations operating in multi-cloud, hybrid, or on-premises environments, the choice is more complex. McDaniel listed several prominent solutions:

• HashiCorp Vault: Historically open-source, now under a Business Source License (BSL) and owned by IBM.
• Akeyless
• CyberArk
• Doppler

These solutions provide the necessary capabilities to manage secrets across diverse infrastructures. The speaker provided a simplified example of how a secret might be retrieved from a system like Vault: vault read secret/data/path/to/secret. While this might not appear "easier" than hardcoding, it is significantly more secure, forcing an attacker to compromise the secret manager itself rather than simply finding a hardcoded string.

Part 2: Set an Auto-rotation Policy

Implementing auto-rotation is the second critical step, but McDaniel cautioned against trying to "boil the ocean." The best approach is to start small, "Bird by bird," focusing on manageable chunks.

Planning for Rotation:

• Ownership: Identify who owns identity management within the organization, as this team will be central to the initiative.
• Maturity Assessment: Understand the organization's current secret management maturity model. G Guardian offers a model that outlines a path from no security to fully automated secret management.
• Developer Buy-in: Engage developers early. New approaches must integrate with existing workflows to avoid resistance.
• Secret Discovery: A crucial prerequisite is knowing where existing secrets reside.
• Greenfield Projects: Easiest starting point, as new secrets can be integrated into the manager from inception.
• Crown Jewels: High-value secrets must be identified and prioritized.
• Legacy Applications: Secrets in older systems may be less valuable but still require attention.
• Zombie Secrets: These are unknown, potentially active secrets residing in Shadow IT or forgotten corners of the infrastructure. Tools are essential for finding them.

Tools for Secret Discovery:

McDaniel mentioned several popular open-source tools for finding hardcoded credentials:

• gitleaks
• trufflehog
• detect-secrets

These tools can scan codebases and repositories to identify exposed secrets, providing a critical inventory for migration.

The Auto-rotation Process (General Steps):

Regardless of the specific implementation, the auto-rotation process typically follows these steps:

1. Create a New Secret: Generate a fresh, strong credential. Some scripts include an optional testing phase for the new secret.
2. Swap in Secret Manager: Replace the old secret with the new one within the secret manager. This is where the centralization aspect becomes critical, as the manager acts as the single source of truth.
3. Test in Production: Crucially, verify that the new secret works correctly in production environments without causing service disruptions.
4. Cleanup/Enumeration: The old secret should not simply disappear. It needs to be logged and potentially revoked or archived for auditing and forensic purposes. This step helps track "zombie secrets" and understand their lifecycle.

For monocloud users, cloud providers often offer pre-written scripts and services, such as AWS Secret Manager rotation Lambdas, which automate much of this process.

For multi-cloud/hybrid environments, the process involves more custom scripting:

• API-Driven Rotation: Identify which external services (e.g., CircleCI, Slack) offer APIs for secret rotation. Most modern services do.
• Custom Scripts: Build scripts that interact with the secret manager and the external service's API to perform the rotation steps.
• G Guardian's Brimstone: McDaniel mentioned an open-source collaboration called Brimstone, which aims to marry secret discovery with auto-rotation into a single system. While acknowledging the potential for multiple points of failure in such a complex system, he emphasized that it's still a significant improvement over manual processes.

The Future: Credential-less Workload Identities

McDaniel concluded by looking towards the ultimate solution: eliminating credentials for machine resources entirely. This paradigm shift is embodied by frameworks like SPIFFE (Secure Production Identity Framework For Everyone) and its implementation, Spire.

• SPIFFE is an open-source framework that defines an "opinionated set of how you would do this"—a set of protocols and ideas for establishing trust between software systems.
• Spire is the actual API and server implementation of SPIFFE.
• How it works: Spire provides microservices with their own short-lived certificates (SHT/JOT) at runtime. These identities are generated only when needed and live only for as long as the workload requires them.
• Authentication: A centralized, yet federated, server system manages these certificates, allowing services to authenticate each other across different environments and platforms (e.g., Kubernetes, other machine resources). This eliminates the need for static, long-lived API keys or passwords for machine identities, significantly reducing the attack surface and simplifying identity management at scale.

This vision represents a fundamental departure from the current credential-heavy model, offering a robust and scalable solution for securing modern, distributed architectures.

Demo / Proof of Concept

▶ Watch: The four-step process for implementing auto-rotation: create, test, swap, tes... (18:00)

While Dwayne McDaniel's presentation was rich with technical detail and actionable strategies, it did not include a live demonstration or a step-by-step proof of concept during the talk itself. He explicitly stated, "I could have spent this entire time walking you through scripts and real life examples of how to do this but my hope is for the people on YouTube and the people in the room that the rest of this talk will be like oh I didn't even know that existed now I can go actually have this conversation internally."

Instead of a live demo, McDaniel focused on outlining the conceptual framework and general steps involved in secret management and auto-rotation. He referenced the availability of pre-written scripts for monocloud environments, specifically mentioning AWS Secret Manager rotation Lambdas as an example of readily available, open-source solutions that implement the described auto-rotation logic. Furthermore, he highlighted G Guardian's open-source collaboration, Brimstone, which aims to integrate secret discovery with auto-rotation, serving as a practical example of how such a system could be built and deployed in a complex, multi-cloud environment. The talk's intent was to inspire and inform about the existence and feasibility of these solutions, rather than to provide a direct, hands-on tutorial.

Defensive Implications

▶ Watch: Discussion of SPIFFE/SPIRE for workload identity and the long-term goal of cr... (23:00)

The insights shared by Dwayne McDaniel offer critical defensive implications for organizations looking to bolster their security posture against the pervasive threat of leaked credentials.

Firstly, the immediate priority for defenders must be to eliminate hardcoded credentials wherever possible. For human users, this means transitioning away from static credentials and towards IAM roles and robust Multi-Factor Authentication (MFA). McDaniel strongly advocated for the use of physical security keys like YubiKey or Titan Key for human authentication, emphasizing their affordability and effectiveness in preventing credential theft.

Secondly, the foundational defensive strategy for machine identities is the centralization of all secrets into a dedicated secret manager. This move is non-negotiable for modern security. Organizations should evaluate and implement solutions like AWS Secret Manager, Google Cloud Secret Manager, Azure Key Vault for monocloud environments, or HashiCorp Vault, Akeyless, CyberArk, or Doppler for multi-cloud and hybrid infrastructures. The chosen secret manager must offer centralized logging and reporting capabilities to provide visibility into secret access and usage, which is crucial for auditing and incident response.

Thirdly, implementing auto-rotation policies for all secrets is a vital proactive defense. Defenders should adopt a "Bird by bird" approach, starting with new "Greenfield" projects where secrets can be integrated into the rotation pipeline from inception. Following this, focus should shift to "crown jewel" secrets—those that protect the most critical assets. Organizations must leverage secret scanning tools such as gitleaks, trufflehog, and detect-secrets to discover existing hardcoded and "zombie" secrets across their codebases and infrastructure. Once identified, these secrets should be migrated to the secret manager and placed under an auto-rotation schedule. For multi-cloud environments, this involves identifying and utilizing the API-driven rotation capabilities of various services (e.g., CircleCI, Slack) and potentially building custom automation scripts or leveraging open-source projects like G Guardian's Brimstone.

Finally, a long-term strategic defensive shift involves moving towards credential-less workload identities. Defenders should explore and plan for the adoption of frameworks like SPIFFE and its implementation Spire. By issuing short-lived, runtime-generated identities (e.g., SHT/JOT certificates) to microservices and other machine workloads, organizations can eliminate the need for static, long-lived credentials, thereby drastically reducing the attack surface and the impact of any potential compromise. This represents a paradigm shift from managing secrets to managing identities, offering a more scalable and resilient security model for distributed systems. The overall message is to prioritize, plan, and incrementally mature secret management practices to ensure a good night's sleep for security teams.

Key Takeaways

• Leaked credentials are the leading cause of breaches: Reports from Sophos, IBM, and CISA confirm that compromised credentials are the number one root cause of security incidents, highlighting an urgent need for improved management strategies.
• The scale of hardcoded secrets is immense and growing: G Guardian's research found 12.8 million unique hardcoded secrets in public GitHub commits in 2023, with 90% of valid secrets remaining active for at least five days, providing attackers ample time for exploitation.
• Centralized secret management is foundational: Consolidating all secrets into a dedicated secret manager (e.g., AWS Secret Manager, Azure Key Vault, HashiCorp Vault) is the critical first step to secure storage, controlled access, and comprehensive logging across all environments.
• Automated secret rotation is essential for reducing risk: Implementing auto-rotation policies, starting with new projects and critical "crown jewel" secrets, significantly shortens the lifespan of compromised credentials and reduces the attack window.
• Discovery tools are crucial for existing secrets: Tools like gitleaks, trufflehog, and detect-secrets are indispensable for identifying hardcoded and "zombie" secrets that need to be migrated and rotated.
• The future lies in credential-less workload identities: Adopting frameworks like SPIFFE and its implementation Spire allows for the generation of short-lived, runtime-specific identities for machines, ultimately eliminating the need for static, long-lived credentials for automated workloads.

About the Speaker(s)

Dwayne McDaniel is a Developer Advocate at G Guardian, a role he has held since 2016. Based in Chicago, he is deeply involved in the security community and is the host of the "Security Repo Podcast," where he engages with various experts in the field, including guests like Jason Haddix and Tanya Janca. Beyond his professional endeavors, Dwayne is an enthusiast of rock and roll. His expertise lies in bridging the gap between development and security, advocating for practical and scalable solutions to modern cybersecurity challenges.