WhizBangLambdaFix: where AWS Misconfigurations...

WhizBangLambdaFix: where AWS Misconfigurations...

Speakers: Lily Chau, Lakshmanan Murthy

Conference: BSidesSF 2024

YouTube: https://www.youtube.com/watch?v=wru_AyiZdlY

Overview

This talk, "WhizBangLambdaFix: where AWS Misconfigurations...", presented by Lily Chau and Lakshmanan Murthy from Roku, introduces a novel framework for automated remediation of AWS misconfigurations. The core problem addressed is the overwhelming volume of security findings and the inefficiency of traditional remediation processes, which rely heavily on manual intervention through Jira tickets and Slack alerts. Despite increased visibility into cloud environments, security teams face challenges such as resource constraints, a lack of security expertise among developers, and the sheer complexity of AWS services and their potential misconfigurations.

The speakers argue that current approaches often stop at merely identifying vulnerabilities or notifying owners, leading to a backlog of unaddressed issues and prolonged remediation times. Their "WhizBangLambdaFix" framework aims to empower security professionals to move beyond these limitations by providing a robust, scalable, and cost-effective solution to safely and effectively mitigate AWS misconfigurations at scale. This framework focuses on proactive, automated fixes, reducing the reliance on manual processes and significantly improving an organization's security posture by minimizing attack surface, optimizing costs, and reducing blast radius.

The talk delves into the architecture, practical playbooks, and real-world metrics from Roku's implementation, demonstrating how a Lambda-centric approach can overcome the shortcomings of traditional security tools and processes. It highlights the importance of a dual strategy: establishing strong secure defaults through Infrastructure as Code (IaC) and implementing automated remediation for "invariants" – those resources that deviate from the desired secure state, often due to manual actions or edge cases not covered by guardrails.

Background

▶ Watch: Problem: Too many findings, slow remediation, lack of security expertise on d... (0:40)

The journey to securing AWS environments typically involves a dual approach: establishing secure defaults and guardrails, and then addressing non-compliant or invariant services. In an ideal scenario, organizations adopt a "golden standard" architecture, such as Istio for service mesh or golden images for deployments. This involves setting up AWS Organizations, GuardDuty, CloudTrail, least privilege IAM roles, and Service Control Policies (SCPs), complemented by thousands of secure-by-default IaC templates. In this perfect world, any security issues would be embedded in code, allowing for scanning, blocking, and automated correction.

However, the reality often diverges from this ideal. Many users, particularly those new to AWS or more familiar with other technologies like Docker and Kubernetes, frequently bypass approved IaC workflows. They manually spin up EC2 instances, deploy applications via Lambdas or Platform-as-a-Service (PaaS) like Heroku, or simply perform "click operations" in the console. While technically possible to deploy securely in these non-approved environments, such manual practices commonly introduce security vulnerabilities that are difficult to track and remediate.

Furthermore, existing security mechanisms like SCPs, while powerful, have significant limitations. They are not well-suited for environments with numerous edge cases, and crucially, they often provide cryptic error messages (e.g., "You are not authorized to perform this operation") that offer no actionable guidance to developers. For instance, an admin might be blocked from launching an EC2 instance using the vulnerable IMDSv1 setting, but the error message fails to convey this specific reason. This lack of clarity contributes to developer frustration and a continued reliance on manual, potentially insecure, workarounds.

The speakers also critically evaluate existing AWS services and third-party tools for configuration management and remediation. AWS Config, for example, is acknowledged for tracking resource metadata and offering predefined rules with SSM document remediation. However, it comes with significant pain points: high cost (Roku experienced $50,000/month), potential for data loss due to excessive API calls impacting business-critical requests, and susceptibility to cost spikes during restart loops or frequent resource creation/deletion. The speakers argue that if an organization uses IaC, it should ideally have no need for Config, as EventBridge can provide configuration tracking much more cheaply. Similarly, tools like Cloud Custodian and Twistlock's Sockless operate on similar principles, but the key differentiator, as highlighted by the speakers, lies in the specific, impactful remediation playbooks developed within their framework. The overarching problem is not a lack of findings or tools, but a lack of effective prioritization and systemic, scalable remediation strategies that go beyond merely identifying issues.

Key Findings

▶ Watch: Generic Auto-remediation Workflow: Scan, risk appetite check, approval/except... (6:00)

The "WhizBangLambdaFix" framework represents a significant shift from reactive, alert-driven security to proactive, automated remediation. The key findings and contributions of this approach are:

• Automated Remediation at Scale: The framework's primary finding is the feasibility and effectiveness of automating the fix for AWS misconfigurations, moving beyond traditional Jira tickets and Slack alerts. This drastically reduces remediation time and human effort.
• Lambda-Centric Architecture: The entire workflow is powered by AWS Lambdas, with each remediation playbook implemented as a separate Lambda function. This design ensures scalability, modularity, and cost-efficiency, costing Roku approximately $1 per month compared to $50,000 per month for AWS Config.
• Dual Remediation Approach:
• Scheduled Remediation: For less critical misconfigurations that can wait up to 24 hours (e.g., unused Elastic IPs, EBS volumes, IMDSv1 updates). This approach is cost-optimized by avoiding additional services like EventBridge, CloudWatch, SNS, or SQS.
• Real-time Remediation: For critical or mutating actions, triggered instantly via CloudTrail and EventBridge (e.g., manual EC2 creation, IAM user creation, EBS deletion).
• Minimal Alert Fatigue: A core goal is "no Jira" and "no Slack" (or minimal Slack). Slack messages are used sparingly, primarily during initial tuning to address false positives or to request owner approval for production changes.
• Strategic IAM Role Design: The framework relies on three distinct IAM roles: a read-only "Security Auditor" role, a "Tagging" role for resource tracking, and a "Remediation" role with specific permissions to fix misconfigurations.
• Tagging as an "Unsung Hero": Strict tagging policies (e.g., RokuSecurity prefix) are crucial for tracking resources, identifying ownership via CloudTrail logs, and estimating cost savings from remediation efforts.
• Effective Manual Action Detection: The framework evolved from analyzing user agents to identifying events initiated by employees (e.g., role_session_name ending with @roku.com) and focusing on mutating actions (create, update, delete) by checking the readOnly flag in CloudTrail events.
• Prioritized Remediation Playbooks: The talk identifies specific, high-impact areas for remediation, categorized by:
• Cost Optimization: Deleting unused Elastic IPs, EBS volumes, idle RDS/Redshift instances (33% of Roku's databases were idle, saving significant costs), and stopping/deleting underutilized EC2s.
• Minimizing Attack Surface: Enforcing IMDSv2 for EC2s, Auto Scaling Groups, AMIs, and EKS (the single best thing to reduce SSRF attacks), mitigating DNS Zone Takeover by deleting Route 53 NS records without associated A/CNAME/TXT records, and performing residual cleanup after S3/ELB deletions.
• Reducing Blast Radius: Detecting and remediating lateral movement by chaining IAM mappings and stripping unnecessary admin/impersonate permissions.
• Quantifiable Impact: Roku achieved significant cost savings (e.g., 33% of databases were idle, saving up to $2,000/month per database), and gained insights into deployment methods (e.g., IMDSv1 fixes via CLI accounted for 43% of issues, CloudFormation 50%, Terraform 2%).
• Lessons Learned: The importance of company buy-in, the need for thorough testing (6 months to a year per playbook) to build trust, the value of monitoring "surges" in remediation to identify underlying issues, and the continuous challenge of edge cases even for universal standards like encryption.

Technical Deep Dive

▶ Watch: Deep dive: EC2 quarantine for manual creation (real-time playbook). (14:00)

The "WhizBangLambdaFix" framework is built upon a serverless architecture primarily leveraging AWS Lambda functions, designed for both cost-efficiency and scalability. The core principle is that each specific misconfiguration remediation workflow is encapsulated within its own Lambda function.

Core IAM Roles:

To operate effectively across multiple AWS accounts, the framework requires two foundational IAM roles in every target account:

1. Security Auditor IAM Role: This role is configured with security_audit permissions, granting read-only access to all AWS configurations. It is designed to be assumed exclusively by the central AWS Security account, ensuring that scanning operations are performed with minimal privilege.
2. Tagging IAM Role: This role allows the security team to apply tags to misconfigured resources. Roku enforces a strict tagging policy, using a unique prefix (e.g., RokuSecurity) for security-related tags. This prevents conflicts with user-defined tags and clearly identifies who applied the tags. Tagging is also used for ownership identification by parsing CloudTrail logs for resource creation events and extracting owner email addresses.

A third, crucial IAM role is dynamically assumed by the remediation Lambdas:

1. Remediation IAM Role: When a misconfiguration is detected, the relevant Lambda assumes this role in the targeted AWS account. This role is granted only the specific permissions necessary to remediate the identified misconfiguration, adhering to the principle of least privilege.

Remediation Workflows:

The framework supports two primary types of remediation:

1. Scheduled Remediation:

• Purpose: Addresses less critical misconfigurations where a 24-hour delay is acceptable.
• Architecture: A Lambda function iterates through a predefined list of active AWS accounts and regions. For each account/region, it scans for targeted misconfigured resources. Upon detection, the Lambda assumes the "Remediation IAM Role" to fix the issue.
• Cost Optimization: This architecture is intentionally minimalist, avoiding additional AWS components like EventBridge, CloudWatch, SNS, or SQS. This significantly reduces operational costs, as the misconfigurations addressed here are not time-sensitive.
• Examples: Deleting unused Elastic IPs, deleting unused EBS volumes, updating EC2 instances to use IMDSv2 over IMDSv1.

1. Real-time Remediation:

• Purpose: Addresses critical or mutating actions that require immediate intervention.
• Architecture: This workflow leverages a centralized CloudTrail logging account, where all member accounts send their logs. AWS EventBridge is configured to ingest these logs. Multiple EventBridge filters are then created to match specific CloudTrail events (e.g., RunInstances, CreateUser, DeleteVolume). When a filter matches, it invokes the appropriate remediation Lambda.
• Examples: Manual creation of EC2 instances in production accounts, manual creation of IAM users, manual deletion of EBS volumes. This allows the system to react instantly to actions that bypass IaC or introduce immediate risks.

Deep Dive: Unused EBS Volume Playbook (Scheduled Example):

This playbook targets EBS volumes not associated with an EC2 instance for 45 days.

1. Scan: The Lambda scans for EBS volumes across all active AWS accounts.
2. Tag Check: It checks for an existing last_seen tag. If absent, it's the first time the playbook is observing this volume. An invariant tag (identifying the playbook) and a last_seen timestamp (current time) are added.
3. Exception Check: It checks for an security_EBS_allowlist=true tag. If present, the resource owner has acknowledged the volume's use, and the workflow ends.
4. CloudTrail Analysis: To account for recent usage not visible in a single point-in-time scan, the playbook parses CloudTrail logs for AttachVolume or DetachVolume events within the last 24 hours. If such events are found, the last_seen tag is updated.
5. Deletion Logic: If the last_seen tag is older than 45 days, the EBS volume is deleted.
6. Notification: Logs are stored, and an email notification is sent to administrators.

Manual Action Detection:

Initially, manual actions were inferred from the userAgent field in CloudTrail logs. The improved method focuses on:

• Employee-Initiated Events: Identifying events where the role_session_name ends with @roku.com, indicating a responsible user.
• Mutating Actions: Filtering CloudTrail events where the readOnly flag is not set, signifying a create, update, or delete operation, which are more actionable than Describe type events.

Lateral Movement Detection and Remediation:

This advanced playbook aims to reduce the blast radius of compromised credentials:

1. IAM Mapping: Create mappings between principals (IAM users, roles, groups) and resources (other IAM users, EC2, EKS, Lambda).
2. Permission Identification: Identify principals with "admin IAM permissions" (e.g., CreateUser, CreatePolicy, wildcard IAM permissions) or "impersonate permissions" over resources.
3. Chain Analysis: Chain these mappings to detect potential lateral movement paths (e.g., an EC2 instance associated with an IAM role that can assume another role, which can then assume a third role with admin IAM permissions). An EC2 instance, for example, has no legitimate reason to create an IAM user.
4. Remediation: For identified resources with excessive or chained admin/impersonate permissions, the playbook checks IAM Access Analyzer or CloudTrail for legitimate usage within the last year. If no legitimate use is found, a new, stripped-down IAM policy is created and attached, and the old, overly permissive policy is detached. This can be applied for any IAM permission and across multiple "hops" in the chain.

Orchestration:

For managing a large number of Lambda playbooks, the speakers suggest wrapping them within orchestration tools like Argo Workflows. This allows for ingesting misconfiguration findings and triggering a sequence of remediation steps.

Comparison with AWS Config:

The speakers highlight several pain points with AWS Config:

• Data Loss: Config's extensive API calls can impact business-critical API requests.
• Deployment Complexity: Remediation playbooks need to be deployed to all regions for all accounts to avoid missing issues.
• High Cost: Config was significantly more expensive for Roku ($50,000/month) compared to their Lambda-based solution ($1/month).
• Cost Spikes: Prone to spikes during restart loops or frequent resource creation/deletion.

They argue that IaC should ideally eliminate the need for Config, and EventBridge offers a cheaper alternative for configuration change logging.

Demo / Proof of Concept

▶ Watch: Deep dive: S3 subdomain takeover residual remediation (complex example). (16:00)

While the talk did not feature a live coding demonstration, Lily Chau and Lakshmanan Murthy provided detailed conceptual walkthroughs of several practical remediation playbooks, effectively serving as proofs of concept for the "WhizBangLambdaFix" framework. These examples illustrate the framework's versatility in addressing various types of AWS misconfigurations.

1. Quarantining a Manually Created EC2 Instance:

This playbook targets instances spun up outside of approved IaC workflows, particularly in production environments.

• Detection: An EventBridge filter is configured to capture the RunInstances event from CloudTrail. This filter also includes logic to identify the user identity responsible for the action.
• Thresholding: To prevent false positives from legitimate bulk scripting (e.g., a Python script using SSO credentials to spin up multiple EC2s for brief data processing), a threshold is applied: if more than five EC2s are launched by the same user within a 15-minute window, the workflow is terminated. This ensures that the system only acts on genuinely anomalous manual creations.
• Remediation: If the threshold is not exceeded, the ec2_quarantine Lambda Playbook is invoked. This playbook performs several actions:
• Tags the EC2 instance as invariant.
• Restricts Ingress traffic by modifying its associated security groups.
• Attaches an IAM policy to the instance to deny access to other AWS resources, effectively isolating it.
• Deletion: After one month, if the quarantined instance still exists, it is automatically deleted. This provides a grace period for owners to address the issue or migrate the workload.

2. Residual Remediation After Manual S3 Bucket Deletion (Subdomain Takeover Prevention):

This playbook addresses the "bad cleanup" scenario where an S3 bucket is deleted, but associated DNS records or CloudFront distributions are left behind, creating a subdomain takeover vulnerability.

• Detection: An EventBridge filter monitors for the DeleteBucket event in CloudTrail.
• Remediation: Upon detection, the S3_subdomain_takeover Lambda Playbook is invoked.
• Domain Formulation: The playbook first formulates all possible domain names that the deleted S3 bucket could have had (e.g., foo.s3.amazon.com, foo.s3.region.amazonaws.com, foo.s3-website-region.amazonaws.com).
• Lookup and Deletion: It then checks both Route 53 CNAME records and the list of CloudFront distributions to see if any match the formulated domain names. If matches are found, they are deleted.
• IaC Integration: If the S3 bucket's domain was centrally managed by IaC, the remediation extends to the code repository, requiring a Terraform fix to ensure consistency and prevent the vulnerability from reappearing. This highlights the blurred line between secure defaults and manual remediation.

3. Residual Remediation After ELB Deletion:

Similar to S3, this playbook prevents subdomain takeover vulnerabilities arising from the deletion of Elastic Load Balancers (ELBs).

• Detection & Remediation: The process mirrors the S3 playbook, but focuses on ELB-specific domain name combinations (e.g., foo.elb.region.amazonaws.com, dualstack.foo.elb.region.amazonaws.com). It checks Route 53 and CloudFront for matching records and applies Terraform or CLI fixes as needed.
• Broader Application: The speakers emphasize that this subdomain takeover remediation playbook should be applied to other resources as well, including Elastic IPs, CloudFront distributions, API Gateways, and Elastic Beanstalk environments.

These detailed playbook descriptions serve as concrete examples of how the "WhizBangLambdaFix" framework translates the theoretical concept of automated remediation into practical, impactful security actions. Each playbook is designed to address a specific, common misconfiguration, demonstrating the framework's ability to systematically reduce attack surface, prevent critical vulnerabilities, and improve overall cloud hygiene.

Defensive Implications

▶ Watch: Metrics: Cost savings & deployment method targeting (real-world impact, data-... (26:00)

The "WhizBangLambdaFix" framework offers profound defensive implications for organizations operating in AWS, shifting the paradigm from reactive incident response to proactive, automated security posture management.

1. Proactive Security Posture: The most significant implication is the ability to move beyond merely identifying vulnerabilities to automatically fixing them. This transforms security teams from auditors and ticket-creators into active remediators, significantly reducing the window of exposure for misconfigurations.
2. Reduced Attack Surface:

• IMDSv2 Enforcement: Automating the update of EC2 instances, Auto Scaling Groups, AMIs, and EKS clusters to use IMDSv2 is highlighted as the "single best thing" to mitigate Server-Side Request Forgery (SSRF) attacks, a critical vulnerability.
• DNS Zone Takeover Prevention: Automatically deleting Route 53 NS records that lack corresponding A, CNAME, or TXT records eliminates a common vector for DNS zone takeover, preventing attackers from hijacking subdomains.
• Residual Cleanup: The framework ensures that when resources like S3 buckets or ELBs are deleted, associated DNS records and CloudFront distributions are also cleaned up, preventing subdomain takeover vulnerabilities.
• Quarantining Anomalous Resources: Manually created EC2 instances, often lacking proper security controls, are automatically quarantined by restricting network access and denying AWS resource access, effectively isolating potential threats.

1. Cost Optimization: The framework directly contributes to significant cost savings by identifying and remediating idle or unused resources. Examples include deleting unused Elastic IPs, EBS volumes, idle RDS and Redshift instances (Roku saved 33% on idle databases), and stopping/deleting underutilized EC2s. This not only improves security but also provides a tangible ROI for security investments.
2. Reduced Blast Radius:

• Lateral Movement Prevention: By mapping IAM permissions and chaining potential lateral movement paths, the framework can identify and remediate overly permissive IAM roles or users. Stripping unnecessary admin or impersonate permissions significantly limits an attacker's ability to move laterally within an AWS environment if initial credentials are compromised.
• Resource Isolation: Quarantining mechanisms for manually created EC2s prevent them from interacting with other AWS services, containing potential breaches.

1. Improved Operational Efficiency and Reduced Alert Fatigue: By automating fixes, the framework drastically reduces the number of Jira tickets and Slack alerts that security teams and developers need to manage. This frees up valuable security resources to focus on more complex, strategic issues rather than repetitive remediation tasks.
2. Enhanced Visibility and Control: The emphasis on strict tagging policies (e.g., RokuSecurity prefix, ownership tagging) provides better visibility into resource ownership, purpose, and compliance status. This aids in incident response and cost attribution.
3. Reinforcement of Secure Defaults: While the framework focuses on remediating "invariants," it implicitly reinforces the importance of establishing strong secure defaults through Infrastructure as Code (IaC). The remediation efforts act as a safety net for deviations from these defaults.
4. Strategic Prioritization: The framework provides a clear methodology for prioritizing remediation efforts based on their impact on cost optimization, attack surface reduction, and blast radius minimization, ensuring that security teams focus on what truly matters.
5. Building Trust and Confidence: The speakers highlight the importance of a phased rollout, starting with non-production environments (Dev), providing ample notification, and allowing for exception processes. This careful approach builds trust with development teams, making them more receptive to automated remediation and less likely to fear service disruptions. Each playbook requires extensive testing (6 months to a year) to ensure reliability.

In essence, the "WhizBangLambdaFix" framework empowers defenders to implement a robust, automated, and data-driven security program that not only identifies but actively fixes misconfigurations, leading to a more secure, compliant, and cost-efficient cloud environment.

Key Takeaways

• Automate Remediation Beyond Alerts: Do not stop at filing Jira tickets or sending Slack alerts. Implement automated remediation workflows to proactively fix AWS misconfigurations at scale, significantly reducing time to resolution and human effort.
• Prioritize Impactful Fixes: Focus remediation efforts on areas that deliver the most significant impact: cost optimization (e.g., idle databases, unused resources), attack surface reduction (e.g., IMDSv2 enforcement, DNS zone takeover prevention), and blast radius minimization (e.g., lateral movement detection).
• Leverage Serverless for Cost-Efficiency: AWS Lambdas provide a highly cost-effective foundation for automated remediation, costing as little as $1 per month for the framework itself, a stark contrast to the $50,000 per month Roku experienced with AWS Config.
• Implement Dual Remediation Strategies: Utilize both scheduled remediation for less critical, cost-saving issues (e.g., daily cleanup of unused resources) and real-time, event-driven remediation for critical, mutating actions (e.g., manual EC2 creation) via CloudTrail and EventBridge.
• Tagging is Crucial: Treat tagging as an "unsung hero" for security. Implement strict tagging policies for resource tracking, ownership identification (via CloudTrail), and to accurately measure the financial and security impact of remediation efforts.
• Build Trust Through Phased Implementation: Introduce automated fixes cautiously, starting in non-production environments (Dev), providing clear communication, and allowing for exception processes. Each playbook requires extensive testing (6 months to a year) to ensure reliability and gain company buy-in.

About the Speaker(s)

Lily Chau and Lakshmanan Murthy are security professionals from Roku. They are the architects and implementers of the "WhizBangLambdaFix" framework, a system designed to automate the remediation of AWS misconfigurations. Their work at Roku focuses on addressing the challenges of managing cloud security at scale, empowering security teams to move beyond traditional alert-and-ticket-based approaches, and actively fix issues to improve the organization's security posture. Lily Chau presented the overall problem, the framework, practical playbooks, and the resulting metrics, while Lakshmanan Murthy provided a deep dive into the generic workflow, IAM roles, and specific scheduled remediation playbooks.