Exposed Credentials – How Attackers Find & Exploit Secrets in Source Code

Exposed Credentials – How Attackers Find & Exploit Secrets in Source Code

Speakers: Mackenzie Jackson, Developer Advocate, GitGuardian

Conference: Hacktivity

YouTube: https://www.youtube.com/watch?v=swl4LZReYVU

Overview

In an era defined by distributed systems and cloud infrastructure, digital authentication credentials have become the indispensable "crown jewels" of modern organizations. Mackenzie Jackson, a Developer Advocate at GitGuardian, delivered a compelling and highly technical presentation at Hacktivity, shedding light on the alarming prevalence of exposed secrets within source code and the sophisticated methods attackers employ to discover and exploit them. The talk meticulously deconstructs the lifecycle of a leaked secret, from its accidental inclusion in public or private repositories to its ultimate exploitation in real-world breaches.

Jackson’s presentation transcends a mere theoretical discussion, offering concrete evidence and live demonstrations that underscore the urgency of the problem. He highlights that secrets—ranging from API keys and credential pairs to security certificates—are not just theoretical vulnerabilities but active targets for initial access, privilege escalation, and lateral movement within an organization's network. The sheer volume of newly exposed secrets year over year, as revealed by GitGuardian's extensive research, paints a stark picture of an escalating security challenge that demands immediate and comprehensive attention from developers and security professionals alike.

The significance of this talk lies in its practical approach, illustrating exactly how attackers leverage publicly available tools and insights into development practices to compromise systems. By examining real-world incidents involving major corporations like Toyota, Twitch, and HashiCorp, Jackson demonstrates that no organization is immune. The talk serves as a critical wake-up call, emphasizing that the problem of secret sprawl is pervasive across public code, private repositories, and even seemingly secure compiled applications, necessitating a multi-faceted defensive strategy grounded in proactive detection, robust secret management, and continuous developer education.

Background

▶ Watch: Defining digital authentication credentials (secrets) (1:00)

The landscape of modern application development is characterized by a high degree of modularity and integration. Applications are no longer monolithic, but rather intricate compositions of hundreds of different services, often leveraging third-party platforms like Stripe for payment processing or Okta for authentication. Each interaction between these services, and between an application and its underlying infrastructure (e.g., cloud platforms, Version Control Systems, testing environments), necessitates the use of secrets. These secrets, designed for programmatic use, are the keys that unlock access to data and infrastructure, making them critical assets.

The proliferation of these interconnected services and the secrets they require creates an exponentially expanding attack surface. Every service, every integration point, represents a potential entry point for an attacker if its corresponding secret is compromised. The core problem, as Jackson articulates, stems from the inherent tension between developer agility and security best practices. Developers, in their quest for rapid iteration and testing, frequently hardcode credentials directly into source code. While often intended as temporary measures, these secrets inevitably become embedded in Git history, persisting even after attempts to remove them. This historical record, a fundamental feature of version control, becomes a significant liability when secrets are inadvertently committed.

Prior work, particularly GitGuardian's annual "State of Secrets Sprawl" report, provides a quantitative foundation for understanding the scale of this issue. These reports consistently demonstrate a year-over-year increase in publicly exposed secrets. Furthermore, high-profile breaches attributed to groups like Lapsus$ illustrate how attackers actively seek out and exploit these vulnerabilities, sometimes resorting to direct means like buying access from insiders or leveraging credentials found on the dark web. The talk also implicitly builds upon the general understanding that "non-human readable" code, such as compiled binaries or container images, is often mistakenly equated with "secure," a dangerous misconception that attackers readily exploit.

Key Findings

▶ Watch: Live demo: committing a honey token to GitHub (12:00)

The talk presented several alarming findings, primarily drawn from GitGuardian's extensive research into secret sprawl:

• Massive Public Exposure: In the past year alone, GitGuardian scanned over a billion public commits on GitHub and identified a staggering 10 million secrets committed publicly. This represents a nearly 70% increase from previous years, highlighting a rapidly accelerating problem.
• Validated, Actionable Secrets: Crucially, many of these identified secrets were not merely high-entropy strings but valid, exploitable credentials. For instance, 20% of the found secrets were for cloud provider keys, translating to approximately 2 million valid cloud provider keys that could grant access to cloud services. Validation was performed by attempting to authenticate with the respective cloud providers.
• Diverse Secret Types: Beyond cloud keys, the scan revealed a wide array of exposed credentials, including keys for messaging systems (e.g., Slack webhooks, which can be leveraged for internal phishing attacks) and, alarmingly, 3.8% for version control platform keys, meaning access tokens to private repositories were found in public code. Specific examples included 9.7% Google API Keys and 3% AWS credentials.
• Private Code Secret Density: When scanning private source code, GitGuardian found that an average company with 400 developers typically has around 13,000 secrets exposed, with approximately 1,000 unique secrets. This creates an unmanageable workload for appsec engineers, who would each face investigating over 3,250 incidents, making effective remediation virtually impossible without automation.
• Compiled Application Vulnerability: Research into Docker images on Docker Hub revealed that approximately 8.5% of all images contain at least one hardcoded plain text credential. A larger study by a German university, scanning 337,000 images, found 52,000 private keys, 2,900 cloud keys, and even 25 financial tool keys (e.g., Stripe keys). This debunks the myth that compilation renders secrets inaccessible.
• Rapid Exploitation: A live honey token demonstration proved that publicly exposed credentials are discovered and exploited by automated bots within minutes of being committed to GitHub. This immediate response from attackers underscores the critical need for proactive detection and prevention.

Technical Deep Dive

▶ Watch: Honey token exploited by multiple attackers within minutes (24:00)

The core of Jackson's talk lay in its detailed exploration of the technical mechanisms by which secrets are leaked and subsequently exploited across different code environments.

Public Source Code Exploitation

Attackers leverage GitHub's vast public dataset in several ways:

1. GitHub Dorking (Inefficient): The simplest but least effective method involves using GitHub's built-in search feature with specific keywords or patterns. For example, searching for filename:credentials "aws_access_key_id" might yield results, but this approach only scans the "top layer" of the code. It fails to account for the crucial aspect of Git history, where secrets persist even after being removed from the latest commit.
2. Monitoring the GitHub Events API (Highly Effective): The most potent method, used by GitGuardian and attackers alike, is to monitor the public and unauthenticated api.github.com/events endpoint. This API acts as a real-time public ledger of all activities on GitHub. Attackers focus on two key event types:

• PublicEvent: When a private repository is made public, its entire history, including any hardcoded secrets, becomes accessible.
• PushEvent: When new code is pushed to a public repository.

This API also exposes user information, including email addresses, allowing attackers to target specific organizations by filtering for corporate email domains. Bots continuously scrape this API, parse new commits, and immediately test any identified high-entropy strings for validity, as demonstrated by the honey token example.

Private Source Code Exploitation

Gaining access to private source code is often more lucrative for attackers due to the higher likelihood of finding sensitive, actively used secrets. Jackson outlined several vectors:

1. Buying Access: Groups like Lapsus$ famously acquired access to corporate networks by paying insiders or purchasing stolen credentials on the dark web, subsequently gaining access to internal Version Control Systems.
2. Phishing Developers: Targeted phishing campaigns remain highly effective. Jackson shared an anecdote from a pen tester who created getlhub.com, a typo-squatted mirror of GitHub. While initial phishing attempts failed, search engines inadvertently indexed the malicious site, leading to tens of thousands of GitHub credentials being harvested from users making typos.
3. Exploiting Misconfigurations (e.g., .git Directory Exposure): A common misconfiguration involves the accidental public exposure of the .git directory. When a developer initializes a Git repository with git init, this hidden folder is created, containing all the repository's metadata and full history. If this directory is deployed to a web server and made publicly accessible, attackers can reconstruct the entire repository, including all past commits and any secrets embedded within them. Cyber News found 2 million such exposed .git directories globally. The Secura Samurai group exploited an exposed .git directory at the United Nations to find hardcoded database credentials.

• Workflow for finding .git directories: Attackers use tools like SubFinder (or chaos) to enumerate subdomains for a target. Then, specialized scanners like git-scanner (or custom scripts with masscan) check each URL for accessible .git directories. If found, the repository history can be reconstructed and scanned using tools like GG Shield for secrets.

1. Supply Chain Attacks: Compromising a component in an organization's software supply chain can grant access to private source code. The Codecov breach, for example, saw attackers compromise a public Docker image used by the code coverage tool. This image contained hardcoded credentials that provided access to a bucket holding a malicious bash upload script. This script, when executed in victim CI/CD pipelines, exfiltrated environment variables (containing secrets) and GitHub authentication credentials, thereby compromising the private source code of 20,000 organizations.

Secrets in Git History

The persistence of secrets in Git history is a fundamental challenge. A common scenario involves a developer hardcoding a credential for quick testing on a feature branch. Even if they later delete the secret and handle it correctly (e.g., using environment variables), and even if the code passes code review (which typically only looks at the latest diff), the secret remains permanently in the Git history. Neither simple deletion nor using git squash (which merely condenses commits, potentially moving the secret to the Git "garbage bin") fully removes the secret. A destructive operation like git rebase or git filter-repo is required, which is often complex and disruptive in large teams.

Common places secrets end up: autogenerated files, log files (especially debug logs), missing .gitignore entries, secrets.txt files committed accidentally, secrets embedded in templates, and simply developers hardcoding secrets believing their private repository will remain secure.

Secrets in Compiled Applications (Docker Images)

Jackson debunked the misconception that compiled applications like Docker images are secure "black boxes." While non-human readable, they are not inherently secure. Docker images are built in layers, and often, source code files (e.g., Python scripts) are copied into these layers. Tools like Dive allow attackers to inspect these layers and extract the original source code. If secrets were present in the source code, they will be extractable from the compiled image. Even attempts to delete secrets later in the Dockerfile (e.g., copying a .netrc file then attempting to remove it) are futile, as previous layers retain the data. Research shows a high percentage of Docker Hub images contain extractable secrets, posing a significant risk, as exemplified by the Codecov breach.

Demo / Proof of Concept

▶ Watch: Codecov supply chain attack explained (34:00)

Mackenzie Jackson reinforced his technical explanations with two compelling demonstrations, showcasing the speed and ease with which secrets are discovered and exploited:

Honey Token Demonstration

Jackson set up a honey token – a specially crafted AWS credential that was real in format but lacked any actual permissions. Its sole purpose was to log any attempts to use it, capturing the attacker's IP address and the commands they executed. He then publicly committed this honey token to a GitHub repository from a user account named "leaky McGee."

Within minutes of this commit, the demonstration revealed active exploitation. The honey token's logs showed attempts to use the credential from three different IP addresses across the globe (India, Netherlands, United States). These attackers used various tools, including trufflehog (an open-source secret scanning tool) and custom Powershell commands, to test the validity of the key. Common commands observed were describe-regions and get-caller-identity, indicating attempts to enumerate AWS services accessible with the leaked credential. This live demonstration powerfully illustrated that public GitHub repositories are under constant, automated surveillance by malicious actors, and any exposed secret will be discovered and probed almost instantly.

.git Directory Scanning Workflow

To illustrate how attackers find and exploit exposed .git directories, Jackson walked through a simulated workflow using commonly available open-source tools. He emphasized that this process is straightforward and widely practiced:

1. Subdomain Enumeration: The first step involves identifying as many subdomains as possible for a target organization. Jackson demonstrated using SubFinder (noting that chaos from Project Discovery is another popular alternative). This tool rapidly scrapes public records and various sources to generate a comprehensive list of potential target URLs.
2. .git Directory Scanning: Once a list of subdomains is obtained, the next step is to scan these URLs for the presence and accessibility of the .git directory. Jackson used a conceptual git-scanner (or similar custom scripts that might leverage tools like masscan) to check for these exposed folders. The output would typically categorize targets as "not vulnerable" (no .git directory), "may be vulnerable" (a .git directory exists but requires further investigation to bypass permissions), or "vulnerable" (a fully accessible .git directory).
3. Repository Reconstruction and Scanning: While a full live reconstruction was not performed due to time, Jackson explained that once a "vulnerable" .git directory is found, attackers can easily download its contents and use standard Git commands to recreate the entire repository locally, including all its historical commits. This reconstructed repository can then be scanned using tools like GG Shield (GitGuardian's open-source scanner) or other secret detection tools to uncover any hardcoded secrets within the full commit history.

These demonstrations provided tangible proof of the vulnerabilities discussed, transitioning the theoretical risks into observable, real-world threats.

Defensive Implications

▶ Watch: Q&A: Secrets in developers' personal dotfiles (41:00)

Addressing the pervasive issue of exposed secrets requires a multi-layered defensive strategy. Organizations must adopt proactive measures across the entire Software Development Life Cycle (SDLC):

1. Never Hardcode Secrets: This is the foundational principle. Developers must be educated and incentivized to avoid hardcoding any digital authentication credentials, even for temporary testing. This includes avoiding secrets in configuration files, scripts, or any part of the codebase.
2. Implement Robust Secret Managers:

• Dedicated Secret Servers (e.g., HashiCorp Vault): These offer the highest level of security and control but are "heavy," requiring a dedicated team for management and developer education. They are best suited for larger organizations with mature security postures.
• SaaS Secret Managers (e.g., OnePassword, Doppler, Keyless): These provide user-friendly interfaces, strong developer features, and managed infrastructure, making them accessible for a wider range of organizations.
• Cloud-Native Secret Managers: While useful, cloud provider-specific secret managers often lack some of the advanced features found in dedicated products.
• Avoid Encrypting in Git: Encrypting secrets and storing them in Git, often alongside the master decryption key, creates a single point of failure and is highly discouraged.

1. Automated Secret Detection:

• Integrate Scanning Tools: Deploy automated secret detection tools (e.g., GitGuardian, GG Shield, trufflehog) across the SDLC. This includes scanning pre-commit (using Git hooks), pre-receive, in CI/CD pipelines, and post-commit on repositories.
• Pre-Commit Hooks: Git hooks are particularly effective as they can block commits containing secrets before they ever enter the repository history.
• Continuous Monitoring: For public repositories, continuous monitoring solutions can alert organizations if their developers inadvertently leak corporate credentials on personal or open-source projects.

1. Rotate Keys Regularly: Establish and enforce a policy of rotating keys regularly. This practice serves two critical purposes:

• Reduced Impact of Leaks: If a secret is leaked, its limited lifespan ensures it will soon become inactive, minimizing the window of opportunity for attackers.
• Operational Readiness: Regular rotation ensures that the organization has a well-practiced process for revoking and redeploying secrets, which is crucial during a security incident.

1. Limit Privileges for Secrets (Least Privilege & Zero Trust):

• Granular Permissions: Apply the principle of least privilege by ensuring each secret only has the minimum necessary permissions to perform its intended function. Avoid issuing admin credentials for tasks that require only read-only access.
• Whitelisting: Restrict access to secrets by whitelisting specific IP addresses, services, or users that are authorized to use them.
• Zero Trust Architecture: Move towards a Zero Trust security model, where access is never implicitly granted but explicitly verified for every request, regardless of origin.

1. Developer Education and Awareness: Developers are on the front lines. They need comprehensive training on secure coding practices, the risks of hardcoding, proper use of secret managers, understanding Git history, and the importance of .gitignore files. Fostering a security-aware culture is paramount.

Key Takeaways

• Secrets are Ubiquitous and Critical: Modern applications rely heavily on digital authentication credentials, making them prime targets for attackers seeking initial access, privilege escalation, and lateral movement.
• Git History is a Persistent Threat: Secrets committed to version control systems, even if quickly removed, persist in Git history and can be easily extracted by attackers, making proactive prevention essential.
• Public and Private Code are Actively Scanned: Attackers utilize automated bots to continuously monitor the GitHub Events API for new secrets and exploit misconfigurations like exposed .git directories to gain access to private source code.
• Compiled Applications Are Not Secure: The misconception that Docker images or other compiled code are secure "black boxes" is dangerous; secrets embedded in source code are extractable from these artifacts.
• Proactive Prevention is Key: Implementing robust secret managers, integrating automated secret detection tools (including Git hooks), and enforcing regular key rotation are non-negotiable defensive measures.
• Least Privilege and Developer Education are Paramount: Adopting the principle of least privilege for all secrets and fostering a strong security culture through continuous developer education are crucial to mitigating secret sprawl.

About the Speaker(s)

Mackenzie Jackson is a passionate Developer Advocate at GitGuardian, a company at the forefront of tackling secret sprawl and code security. Originally from New Zealand, he now resides in the Netherlands and works for a French company, reflecting his diverse international experience. Mackenzie is also the host of the "Security Repo Podcast," which he humorously notes is his mom's favorite, offering listeners valuable insights into software security. His work focuses on educating developers and security professionals about the dangers of exposed credentials and best practices for securing source code. You can find him on social media as @advocatemac.