Breaking the Cloud: A Tale of 3 Breaches!
Ashish Rajan (Podcaster and YouTuber · Cloud Security Podcast)
Security BSides London 2023 · Day 1 · Track 3
Overview
In "Breaking the Cloud: A Tale of 3 Breaches!", Ashish Rajan, a seasoned cloud security expert and host of the Cloud Security Podcast, dissects a real-world incident that exposed profound vulnerabilities stemming from often-overlooked business logic flaws in cloud environments. The talk, born from a revelation during a cloud security training session, highlights how even organizations with established security postures and tools like Cloud Security Posture Management (CSPM) can fall victim to sophisticated attack chains when foundational security principles are neglected or cloud platform changes introduce new risks. Rajan’s narrative serves as a stark reminder that robust security extends beyond mere configuration checks, demanding a deep understanding of how various cloud services interact and how an attacker might exploit these relationships.

Key moments
- 0:00 Introduction: Accidental breach during training
- 3:30 Terraform state file exposed AWS credentials
- 7:30 Critical flaw: CloudTrail logging was disabled
- 10:40 Privilege escalation via EC2 role to S3 access
- 13:00 Stealthy S3 'dry run' and volume snapshotting
- 15:20 Remediation challenges: deleting GitHub history
- 19:40 Business logic flaws and Terraform Cloud solution
- 34:30 Q&A: AWS IAM Role vs. IAM User best practices
Breaking the Cloud: A Tale of 3 Breaches!
Speakers: Ashish Rajan, Podcaster and YouTuber, Cloud Security Podcast
Conference: BSides London
YouTube: https://www.youtube.com/watch?v=Vch8EycT6rk
Overview
In "Breaking the Cloud: A Tale of 3 Breaches!", Ashish Rajan, a seasoned cloud security expert and host of the Cloud Security Podcast, dissects a real-world incident that exposed profound vulnerabilities stemming from often-overlooked business logic flaws in cloud environments. The talk, born from a revelation during a cloud security training session, highlights how even organizations with established security postures and tools like Cloud Security Posture Management (CSPM) can fall victim to sophisticated attack chains when foundational security principles are neglected or cloud platform changes introduce new risks. Rajan’s narrative serves as a stark reminder that robust security extends beyond mere configuration checks, demanding a deep understanding of how various cloud services interact and how an attacker might exploit these relationships.
The core of the presentation revolves around a "breach" initiated by the accidental exposure of an AWS Identity and Access Management (IAM) user's access key within a Terraform state file on a public GitHub repository. This seemingly common misstep quickly escalated into a comprehensive compromise demonstration, revealing the absence of crucial logging (AWS CloudTrail), the presence of an over-permissive IAM role attached to an EC2 instance, and the ability to perform stealthy data reconnaissance and exfiltration techniques. Rajan meticulously walks through the attacker's perspective, illustrating how a series of seemingly minor misconfigurations, when chained together, can lead to complete data compromise and evade detection by conventional security tools.
This talk is particularly relevant for cloud architects, security engineers, DevOps professionals, and CISOs grappling with the complexities of securing dynamic cloud environments. It underscores the critical importance of a multi-layered defense strategy that not only addresses technical vulnerabilities but also accounts for human error, evolving cloud features, and the often-underestimated impact of business logic flaws. By sharing this candid account of his own team's security lapse, Rajan advocates for greater transparency in the security community and emphasizes continuous learning and adaptation in the face of an ever-changing threat landscape.
Background
▶ Watch: Introduction: Accidental breach during training (0:00)
The genesis of this talk lies in a practical, real-world scenario encountered during a cloud security training program run by Ashish Rajan’s team. During the training, a critical security oversight occurred: a Terraform state file containing sensitive AWS IAM user credentials (an access key and secret key) was inadvertently committed and pushed to a public GitHub repository. Terraform, an open-source Infrastructure as Code (IaC) tool by HashiCorp, is widely used to provision and manage cloud resources. Its state file maintains a mapping between the real-world infrastructure and the configuration defined in Terraform, often containing sensitive information necessary for interacting with cloud APIs.
A sharp-eyed security researcher participating in the training discovered this exposed credential. Instead of directly exploiting it maliciously, the researcher responsibly disclosed the finding to Rajan's team by providing a video demonstration of the potential breach. This video forms the basis of the "breach" discussed in the talk, serving as a powerful, albeit simulated, case study. The incident highlighted a crucial gap: despite having established cloud security practices, including the use of CSPM solutions, this fundamental error and its cascading implications went undetected.
Rajan emphasizes that the problem isn't just about a single exposed secret; it's about the subsequent chain of events enabled by that initial access, particularly how it exploits "business logic flaws" rather than traditional software vulnerabilities. These flaws represent weaknesses in the design or implementation of system interactions, often missed by automated vulnerability scanners that focus on known CVEs or common misconfigurations. The talk also draws parallels to high-profile cloud breaches, such as those affecting Uber, DataDog, Capital One, and FGX, where exposed credentials, over-permissive roles, and S3 misconfigurations were central to the compromise. These examples underscore that the scenario presented is not an isolated incident but a common pattern in cloud attacks, often exacerbated by the rapid evolution of cloud services and the challenges of maintaining comprehensive visibility and control.
Key Findings
▶ Watch: Critical flaw: CloudTrail logging was disabled (7:30)
The simulated breach meticulously demonstrated several critical security findings, highlighting common pitfalls in cloud security:
- Exposed Terraform IAM User Access Key: The initial point of compromise was an AWS IAM user access key inadvertently committed to a public GitHub repository within a Terraform state file. This key, intended for programmatic interaction with AWS, granted an attacker initial access to the cloud environment. The speaker noted the difficulty of completely removing such sensitive information from GitHub history once committed, necessitating the deletion and recreation of the entire repository.
- Disabled AWS CloudTrail: A major discovery was that AWS CloudTrail, the auditing service that records all API calls and account activity, was not enabled in the compromised account. This absence meant that any malicious actions performed by the attacker would leave no forensic trail, severely crippling incident response and detection capabilities. CloudTrail is considered a cornerstone of cloud security, and its deactivation or lack of configuration is a critical security lapse.
- Over-Permissive EC2 Instance IAM Role: Despite initial assumptions of "read-only" permissions, the attacker identified a running EC2 instance with an attached IAM role that possessed overly broad privileges. Specifically, this role, in addition to a standard
SSMManagementInstanceRole, had an additional policy grantingS3:Allow, effectively providing full access to all S3 buckets within the account. This demonstrated a critical privilege escalation path, allowing the attacker to move from initial access to sensitive data stores. - Stealthy S3 Reconnaissance (
--dryrun): The attacker successfully performed a "silent test" for data exfiltration using theaws s3 cp --dryruncommand. This command simulates a copy operation without actually moving data and, crucially, was observed not to be logged by the organization's SIEM solution or CloudTrail as a data exfiltration attempt. This bypasses typical alerts for mass data transfers, allowing an attacker to enumerate and validate exfiltration capabilities without raising alarms. - EBS Volume Snapshotting for Data Exfiltration: A sophisticated technique demonstrated was the ability to create snapshots of Elastic Block Store (EBS) volumes attached to EC2 instances. This allows an attacker to bypass endpoint security agents on live instances by creating a copy of the volume, attaching it to a newly provisioned, unmonitored EC2 instance (potentially in a different account or region), and then extracting data offline. This method is particularly effective for evading detection by traditional endpoint security tools.
- Business Logic Flaws Overlooked by CSPM: Rajan emphasized that the entire attack chain represented a series of "business logic flaws" rather than technical vulnerabilities that would be flagged by standard security tools. CSPMs and other automated scanners typically look for open S3 buckets or unencrypted resources, but they often fail to identify how a chain of legitimate, albeit misconfigured, permissions and service interactions can lead to a breach. This highlights a gap in many security tools' ability to understand contextual risk.
Technical Deep Dive
▶ Watch: Stealthy S3 'dry run' and volume snapshotting (13:00)
The attack vector detailed in the talk represents a classic kill chain in cloud environments, beginning with initial access and culminating in data exfiltration, all while attempting to evade detection.
The initial foothold was gained through the discovery of a Terraform state file in a public GitHub repository. This file contained the access key and secret key for an AWS IAM user. Terraform state files, especially when stored locally or in insecure version control, are a goldmine for attackers, as they often contain sensitive credentials or configuration details. In this case, the compromised IAM user was password123, an embarrassingly weak credential, underscoring the importance of strong, unique passwords and avoiding hardcoding secrets.
Upon gaining programmatic access via the AWS CLI with the exposed credentials, the attacker's first critical discovery was the state of AWS CloudTrail. CloudTrail is AWS's primary auditing and logging service, recording every API call made in an account. The demonstration clearly showed that CloudTrail was not enabled, a critical misconfiguration that effectively blinded the security team to any subsequent malicious activity. Without CloudTrail logs, forensic analysis, incident detection, and attribution become nearly impossible, making it a prime target for attackers to disable or ensure is inactive.
Navigating the AWS console with the compromised IAM user, which initially appeared to have "read-only" permissions, the attacker began to enumerate resources. A key discovery was a running EC2 instance in the North Virginia region. Crucially, this EC2 instance had an attached IAM role. While roles like SSMManagementInstanceRole are common for instance management, further inspection revealed that this specific role had an additional, highly permissive policy: S3:Allow. This policy granted full access to all S3 buckets within the AWS account, effectively bypassing the seemingly restrictive "read-only" nature of the initial IAM user. This represents a significant privilege escalation path, where an attacker leverages a less privileged entry point to assume a more powerful role associated with another resource.
The attacker then utilized AWS Systems Manager (SSM), a service that allows direct access and management of EC2 instances from the AWS console, to interact with the compromised EC2 instance. From within the instance, the attacker could execute AWS CLI commands using the instance's assumed IAM role. The first step was to list S3 buckets using aws s3 ls. This confirmed the broad S3 access granted by the EC2 instance's role.
Next, the demonstration focused on data exfiltration. A specific experimental S3 bucket was targeted. To test the ability to download files without triggering alarms, the attacker used aws s3 cp s3://experimental-bucket/file.txt . --dryrun. The --dryrun flag simulates the command's execution without actually performing the action. Critically, Rajan noted that this specific command was not logged by their existing SIEM solution as a data transfer event. This provides a stealthy method for attackers to assess the viability of data exfiltration targets and pathways without generating high-fidelity security alerts, exploiting a "business logic flaw" in how monitoring tools interpret such commands.
Finally, the attacker explored methods to exfiltrate data from EBS volumes. Recognizing that live EC2 instances might have endpoint security agents, the attacker demonstrated the ability to create snapshots of EBS volumes. An EBS snapshot is a point-in-time copy of the volume. This snapshot could then be shared with another AWS account or region, and a new EC2 instance could be launched from it. This new instance, unmonitored by the victim's security agents, would contain a complete copy of the original volume's data, allowing for offline analysis and exfiltration without interacting with the compromised, potentially monitored, live environment. This technique leverages the flexibility of AWS resource sharing for covert data extraction.
The entire sequence illustrates a sophisticated multi-stage attack that combines initial credential compromise, privilege escalation via misconfigured IAM roles, stealthy reconnaissance, and advanced data exfiltration techniques, all designed to bypass traditional security controls and remain undetected.
Demo / Proof of Concept
▶ Watch: Remediation challenges: deleting GitHub history (15:20)
The talk effectively utilized a pre-recorded video demonstration to illustrate the entire simulated breach, providing a tangible and step-by-step walkthrough from the attacker's perspective. Ashish Rajan played the video, pausing at critical junctures to provide context and elaborate on the implications of each action.
The video began by showcasing the exposed Terraform state file. The file, displayed in a text editor, clearly revealed an AWS IAM user's username and a laughably simple password123 credential, alongside an endpoint URL. This visual immediately established the initial compromise vector.
The demonstration then transitioned to the AWS console. The attacker, presumably using the exposed credentials, navigated through the IAM service. Initially, the IAM user's permissions seemed limited, but the video highlighted the existence of an access key associated with this user, which was the actual programmatic credential used for the AWS CLI and API calls. An attempt was made to launch a new EC2 instance, a common cryptojacking tactic, but this action was blocked due to insufficient permissions, confirming some level of initial control.
The critical moment arrived when the video showed the AWS CloudTrail dashboard. It conspicuously displayed that CloudTrail was not enabled in the account. Rajan paused to underscore the gravity of this finding, explaining that this absence meant zero auditability for any subsequent malicious activities.
The video then focused on discovering existing resources. An active EC2 instance in the North Virginia region was identified. The attacker then drilled down into the IAM role attached to this EC2 instance. While the SSMManagementInstanceRole was present, the crucial detail was the additional, broad S3:Allow permission. This immediately signaled a privilege escalation opportunity.
Using AWS Systems Manager (SSM), the attacker gained console-based access to the EC2 instance. From the instance's terminal, the aws s3 ls command was executed, successfully listing all S3 buckets in the account, confirming the S3 access derived from the instance's IAM role. The demo then proceeded to show the use of an "experimental bucket." The attacker executed aws s3 cp s3://experimental-bucket/file.txt . --dryrun. Rajan specifically pointed out that this command, designed to test download capabilities without actual data transfer, did not trigger any alarms in their monitoring systems, showcasing a stealthy reconnaissance technique.
Finally, the demonstration concluded with the exploration of EBS volume snapshotting. The video showed the process of identifying an EBS volume and initiating the creation of a snapshot. Rajan explained that this technique allows an attacker to bypass endpoint security agents on live instances by creating a copy of the volume and accessing its data offline, potentially from a newly provisioned, unmonitored EC2 instance.
While the video did not show actual data exfiltration to an external IP (as the target EC2 instance lacked a public IP, indicating some network egress controls were in place), it effectively illustrated the entire chain of actions an attacker could take, from initial access to preparing for covert data extraction, all within a seemingly "read-only" compromised account. The emphasis was on the "silent" nature of some of these actions, designed to evade detection.
Defensive Implications
▶ Watch: Q&A: AWS IAM Role vs. IAM User best practices (34:30)
The simulated breach provides a wealth of actionable insights for cloud defenders, emphasizing a shift from reactive security to proactive, context-aware strategies.
- Secure Infrastructure as Code (IaC) Practices:
- Eliminate Secrets from Source Control: The foundational lesson is to never commit sensitive credentials (AWS access keys, API tokens, passwords) directly into GitHub, GitLab, Bitbucket, or any other version control system. This requires strict adherence to secret management best practices.
- Leverage Remote State Management: For Terraform, migrate from local state files to secure, remote state management solutions like Terraform Cloud (which offers a free tier), AWS S3 with DynamoDB locking, or Azure Storage Accounts. This offloads the risk of state file exposure from developer machines and version control.
- Implement Pull Request (PR) Workflows for IaC: Even for senior engineers, enforce PR approval workflows for all IaC changes. This mandates peer review, allowing another set of eyes to catch misconfigurations, hardcoded secrets, or overly permissive policies before they are deployed. This extends standard software development lifecycle (SDLC) security to infrastructure.
- Robust Identity and Access Management (IAM):
- Principle of Least Privilege: Continuously review and enforce the principle of least privilege for all IAM users, roles, and policies. Regularly audit permissions attached to EC2 instances, Lambda functions, and other compute resources to ensure they only have the absolute minimum permissions required. Automated tools can help identify over-permissive roles.
- Credential Rotation and Lifecycle Management: Implement a robust secret rotation and key lifecycle management process. Use services like AWS Secrets Manager or Azure Key Vault to store, manage, and automatically rotate credentials rather than relying on static files or hardcoded values.
- Prioritize IAM Roles and Temporary Credentials: Whenever possible, use IAM roles with external IDs and temporary credentials for applications and third-party integrations instead of long-lived IAM user access keys. While some third-party vendors might default to requesting IAM users, advocate for role-based access.
- Adopt Identity Federation (SSO): Migrate away from individual IAM users towards AWS Identity Center (SSO) or other identity federation solutions. This centralizes user management, enforces strong authentication (like MFA), and simplifies access across multiple accounts.
- Periodic User Reviews: Conduct regular, periodic reviews of all IAM users and roles to identify and remove dormant, unused, or unknown accounts that could serve as backdoors.
- Comprehensive Logging and Monitoring:
- Enable and Secure CloudTrail: AWS CloudTrail must be enabled in all regions and accounts. Crucially, its logs should be configured to be delivered to a centralized, secured S3 bucket in a separate, dedicated logging account to prevent tampering or deletion by an attacker in a compromised account. Enable CloudTrail integrity validation.
- Enhanced SIEM/CSPM Detection: Augment existing SIEM and CSPM solutions with custom detection rules for "business logic flaws" and stealthy attack techniques. Specifically, monitor for commands like
s3 cp --dryrunor patterns of unusual resource access, even if they don't immediately appear as direct data exfiltration. CSPMs may not catch these by default; custom rules are essential. - Data Loss Prevention (DLP): While challenging in multi-cloud environments, consider implementing DLP solutions or native cloud controls for detecting and preventing sensitive data exfiltration from S3 buckets, databases, and other storage services. Acknowledge that native cloud DLP might have limitations for multi-cloud or specific attack scenarios.
- Asset Inventory and Security Benchmarks:
- Maintain Accurate Asset Inventory: Develop and maintain a comprehensive, up-to-date asset inventory of all cloud resources across all accounts and regions. This helps identify unapproved, rogue, or forgotten resources that could be exploited.
- Utilize Security Benchmarks: Start with established security benchmarks like CIS Benchmarks for initial hardening. For multi-cloud environments, explore frameworks from organizations like the Cloud Security Alliance (CSA) that offer unified guidance.
- Security Education and Awareness:
- Employee Training: Provide continuous security education and awareness training for all employees, especially those involved in creating and managing cloud resources. This training should cover secure coding practices, secret management, IAM best practices, and the risks associated with misconfigurations.
- Threat Detection Evolution: Recognize that the cloud threat landscape is constantly evolving. Continuously update threat detection scenarios based on new services (e.g., AWS Bedrock for AI), emerging attack techniques, and real-world incidents shared by the community.
By diligently implementing these defensive measures, organizations can significantly reduce their attack surface, enhance their detection capabilities, and build a more resilient cloud security posture against both common misconfigurations and sophisticated business logic flaws.
Key Takeaways
- Business logic flaws pose a significant, often overlooked, risk in cloud environments, bypassing traditional security tools. These flaws exploit the interactions between legitimate cloud services rather than direct vulnerabilities.
- Exposure of Infrastructure as Code (IaC) secrets (e.g., Terraform state files in GitHub) is a critical initial access vector. Robust secret management and secure coding practices are paramount.
- AWS CloudTrail is fundamental for detection and incident response; its absence or misconfiguration blinds security teams to malicious activity. Ensure it's enabled, secured, and logs are exported to a centralized, tamper-proof location.
- Over-permissive IAM roles on compute resources (like EC2 instances) can lead to severe privilege escalation and unauthorized access to sensitive data. Strict adherence to the principle of least privilege and regular IAM audits are essential.
- Attackers employ stealthy techniques, such as
s3 cp --dryrunor EBS volume snapshots, to reconnoiter and exfiltrate data without triggering standard security alarms. Custom detection rules and enhanced monitoring are necessary to catch these subtle indicators. - Cloud security demands continuous adaptation and a multi-layered approach. This includes secure development practices, robust IAM, comprehensive logging, custom threat detection, and ongoing employee education to counter the rapid evolution of cloud platforms and attack methods.
About the Speaker(s)
Ashish Rajan is a highly experienced cybersecurity professional with over 13 years in the industry, dedicating the last seven years specifically to cloud security. Having previously served as a CISO, Ashish has transitioned into a full-time role as a podcaster and YouTuber, where he runs the popular Cloud Security Podcast. Originally from Australia, he moved to the UK seven months prior to giving this talk. Beyond his professional endeavors, Ashish has a keen interest in men's fashion and coffee. His practical experience and dedication to sharing knowledge make him a respected voice in the cloud security community.
Reviews
Dr. Zero (Offensive Security Researcher) — WEAK
Ashish Rajan takes standard AWS IAM misconfigurations that tools like Pacu have automated for years and tries to rebrand them as 'business logic flaws.' It's a basic cloud security 101 primer, but framing an exposed Terraform state file and missing CloudTrail logs as a 'sophisticated attack chain' is pure podcast hype.
Heather Calloway (CISO) — SOLID
This is a practical walkthrough of how a single developer mistake cascades into a full cloud compromise. It demonstrates exactly why cloud security posture management (CSPM) tools provide false comfort if they ignore identity pivot points. It covers a standard cloud kill chain, making it a solid baseline check for infrastructure and detection teams.