Stop Training Engineers to Ignore You: Cloud Security Alerting at Scale

Stop Training Engineers to Ignore You: Cloud Security Alerting at Scale

Speakers: Paul Benoit, Cloud Security Engineer, Block

Conference: fwd:cloudsec

YouTube: https://www.youtube.com/watch?v=eocgfj-SXVo

Overview

In the dynamic and ever-expanding landscape of cloud infrastructure, effective security alerting at scale presents a formidable challenge for even the most sophisticated organizations. This talk, "Stop Training Engineers to Ignore You: Cloud Security Alerting at Scale," delivered by Paul Benoit, a Cloud Security Engineer at Block, delves into the intricate problems and hard-won solutions associated with operationalizing security alerts across thousands of cloud accounts and engineers. Benoit highlights that while smaller organizations might navigate these issues with manual effort, the sheer volume and complexity at enterprise scale can quickly overwhelm security teams and, crucially, desensitize the very engineers who are meant to resolve these issues.

The core premise of Benoit's presentation is that cloud security alerts are fundamentally ineffective without adequate context. A raw security finding, irrespective of its technical severity, becomes actionable only when enriched with business context, risk assessment, and clear ownership. Without this enrichment, alerts are perceived as noise, leading to frustration, ignored notifications, and ultimately, an erosion of trust between security teams and development engineers. Block’s multi-year effort to refine its alerting mechanisms across AWS and GCP serves as a practical case study, demonstrating how a strategic focus on context, automation, and empathy can transform a "minefield" of bad alerts into a streamlined, effective security detection and response program.

Benoit, part of Block's cloud security team, focuses on both preventive measures like guardrails and secure paved roads, as well as the detective side, which involves defining security baselines, measuring deviations, and issuing targeted alerts. The talk provides invaluable insights into Block's journey from an initial state of sending "a lot of bad alerts" to a mature system capable of delivering "good alerts" that foster collaboration and prompt remediation, offering a blueprint for other organizations grappling with similar challenges in their cloud security operations.

Background

▶ Watch: Cloud security alerting at scale is a minefield (0:50)

The journey to effective cloud security alerting at scale is fraught with operational complexities that often go unnoticed in smaller environments. As organizations grow to encompass thousands of cloud accounts and an equally large engineering workforce, the problems associated with security alerts amplify dramatically. Paul Benoit articulates that initial attempts at alerting often involve simply forwarding raw findings from native cloud security services and third-party tools directly to engineers. This approach, while seemingly straightforward, quickly leads to a deluge of alerts that lack crucial context, making them unactionable or, worse, misleading.

One of the most persistent and challenging problems identified is attribution. In an early-stage startup, identifying the owner of a problematic resource might be as simple as a quick Slack message. However, in a large enterprise like Block, with its diverse business units, thousands of cloud accounts, and frequent organizational changes, pinpointing the responsible team or individual for a specific cloud resource becomes a significant hurdle. Common solutions, such as tagging resources with owner information, are often insufficient. Benoit points out several limitations: tag propagation delays (up to seven minutes in GCP, five in AWS) can mean scanners miss ownership data, tags don't apply to all resources (e.g., GCP IAM bindings), and a single resource might have multiple owners (e.g., one team for an EC2 image, another for application data, and a third for infrastructure configuration). Sending alerts without clear, accurate attribution is a primary driver of engineer frustration and distrust.

Beyond attribution, the sheer volume of alerts, coupled with a high rate of false positives, further exacerbates the problem. Security tools, while powerful, often generate findings that are technically correct but not practically exploitable or relevant in a specific context. A CVE on a sandbox workload with no internet access and a short lifecycle, for instance, carries a vastly different risk profile than the same CVE on a production "crown jewel" asset handling sensitive data. Without the ability to differentiate these scenarios, security teams risk "crying wolf" too often, leading engineers to ignore even legitimate threats. Block’s initial experience mirrored this, where a high percentage of alerts were either irrelevant, unactionable, or poorly targeted, leading to a negative impact on engineering productivity and security posture.

Key Findings

▶ Watch: Core theme: Cloud security alerts need context (2:54)

Paul Benoit's talk distills Block's multi-year experience into several critical findings that underpin a successful cloud security alerting strategy at scale:

• Context is King: The overarching theme is that raw security findings are inherently insufficient. Alerts must be enriched with business context, risk context, and operational context to be meaningful. This includes understanding the environment (dev vs. prod), data sensitivity, internet exposure, and asset criticality. Without this, security issues appear generic and less urgent to engineers.
• Robust Attribution is Non-Negotiable but Complex: Simple tagging is a starting point but fails at scale due to technical limitations (propagation delays, untaggable resources) and organizational complexities (multiple owners per resource, reorgs). A multi-tiered ownership chain with fallbacks is essential, moving from strong signals (PagerDuty on-call) to weaker ones (Git owners files, account owners, or even hardcoded assignments as a last resort). Transparency in how ownership is determined is crucial for building trust.
• Issue Grouping Prevents Alert Fatigue: Sending one alert per misconfiguration can lead to thousands of tickets for a single underlying problem (e.g., a misconfigured autoscaling group). The key finding is to group issues by "one fix" rather than "one finding." This reduces noise and allows engineers to address the root cause efficiently.
• Aggressive False Positive Management is Critical: A high rate of false positives rapidly erodes trust and wastes engineering time. Specific examples like IAM Access Analyzer flagging decommissioned accounts as external, or DSPM (Data Security Posture Management) tools generating alerts for benign or ephemeral data, highlight the need for custom logic, circuit breakers, and careful exception handling beyond what vendors might offer. Block experienced a 70% false positive rate with DSPM at one point, underscoring the severity of this problem.
• Centralized Exception Management is Essential for Nuance: Not every security finding warrants immediate, mandatory remediation. A dedicated exception service allows for nuanced decisions, tiered approvals (manager, security), and predefined reasons (false positive, decommissioning, vendor issue). This acknowledges the reality of complex cloud environments and provides a structured way to manage acceptable risks.
• Effective Issue State Management Prevents "Flapping": Alerts that repeatedly open and close ("flapping issues") are frustrating. Often, the root cause is poor issue grouping or alerting on ephemeral resources. Implementing a "pending resolution" status in the ticketing system pauses SLAs and provides a clear workflow for engineers to indicate a fix, allowing security to re-scan without penalizing the engineer.
• AI Auto-Remediation is an Emerging Game Changer: While in early stages, leveraging Large Language Models (LLMs) and agents to automatically generate pull requests (PRs) for fixes represents a significant leap forward. It's more effective for deterministic fixes (e.g., upgrading a library version) but shows promise even for more complex misconfigurations, offering a path to dramatically reduce manual remediation effort.

Technical Deep Dive

▶ Watch: Problems with tagging resources for ownership attribution (3:30)

Block's approach to cloud security alerting at scale is built upon a sophisticated architecture that integrates native cloud services, third-party tools, and custom internal systems, all centered around the principle of context enrichment. The overall flow begins with raw findings generated by native cloud security services (like AWS Security Hub, GCP Security Command Center) and various third-party security vendors. These findings are then fed into a crucial alert enrichment pipeline. This pipeline is responsible for adding layers of context: attribution (who owns it?), risk adjustment (how critical is this asset?), and business context (what environment is it in, what data does it handle?). The enriched alerts are then routed to Block's internal ticketing system, which is part of their custom-built Security Command Center UI. This system handles SLAs, exceptions, and metrics, providing a centralized platform for managing security issues.

A cornerstone of their enrichment strategy is the ownership chain with fallbacks for attribution. Recognizing the limitations of simple resource tagging (e.g., propagation delays, untaggable resources like GCP IAM bindings, multiple owners per resource), Block developed a robust hierarchy:

1. Strongest Signal: An application associated with a PagerDuty schedule. This directly links an issue to an on-call engineer, ensuring high confidence in ownership.
2. Weaker Signal: An owning team member identified through internal directories. However, this often led to issues where "owners groups" were used as "users groups," resulting in engineers receiving alerts for applications they no longer managed. To mitigate this, assignments are distributed, typically one person per week from a team.
3. Further Fallbacks: Owners files in Git repositories, the owner of the AWS account itself, or, as a last resort, hardcoded mappings for known problematic routes. Critically, Block now includes the method of attribution (e.g., "you were assigned this because you are on the PagerDuty schedule") in the alert, empowering engineers to update outdated ownership information.

Managing false positives is another area of significant technical investment. For IAM Access Analyzer, which often flags decommissioned or suspended AWS accounts as "external" in trust policies, causing floods of alerts, Block implemented a custom check. Before routing an Access Analyzer alert, they verify if the flagged account ID belongs to an account previously owned and being decommissioned by Block. If so, the alert is dropped, recognizing it as a hygiene issue rather than an immediate security risk, as AWS confirms account IDs are not reused.

To combat alert floods, particularly when a new vendor role is deployed across many accounts, Block implemented a circuit breaker (credited to Stephanie Shai). The flow is: Access Analyzer -> EventBridge -> SQS Queue. If the SQS queue size grows abnormally, indicating a potential flood, the connection to the ticketing system is halted, and alerts are redirected to a dead-letter queue. This prevents thousands of tickets from being created, retains the alerts for later review, and minimizes embarrassment for the security team. Archival rules for Access Analyzer are also managed via a Python script to handle known exceptions.

For DSPM (Data Security Posture Management), which initially suffered from a 70% false positive rate, Block had to develop custom solutions. They identified patterns of benign data that were being flagged as sensitive and, lacking sufficient expressiveness in the vendor's exception logic, built their own. This highlights the need for organizations to be prepared to extend or even bypass vendor-provided exception mechanisms for highly specific use cases. Another DSPM challenge was ephemeral data: if DSPM scans run every three weeks, but data rotates every 48 hours, an alert received days later is unactionable. Block now identifies and suppresses alerts for known ephemeral data sources (e.g., certain Snowflake tables).

The dedicated exception service acts as a centralized repository for managing all security exceptions. It provides a consistent interface across disparate detectors and offers tiered approvals (e.g., requiring manager or security approval for higher-risk exceptions). Predefined reasons (false positive, decommissioning, vendor issue with feature request in progress) streamline the process and provide valuable metrics on common exception types.

Finally, AI auto-remediation represents Block's cutting-edge efforts. Using LLMs and agents, they automatically generate Pull Requests (PRs) to fix security issues. This is highly effective for deterministic fixes, such as upgrading a vulnerable library version in an application vulnerability. While more challenging for misconfigurations (where multiple remediation paths or an exception might be valid), Block is experimenting with LLMs for these scenarios, indicating a future where routine security fixes are increasingly automated.

Demo / Proof of Concept

▶ Watch: Issue grouping strategy: One fix, not finding, per ticket (6:04)

While the talk does not describe a live demonstration of a specific tool or exploit, Paul Benoit provides a comprehensive overview of Block's operationalized security alerting system, which itself serves as a proof of concept for effective cloud security at scale. The entire technical deep dive effectively functions as a detailed explanation of their implemented solutions.

The architecture described—from the initial ingestion of raw findings from diverse sources (native cloud services, third-party tools) to the custom-built alert enrichment pipeline and the internal Security Command Center—represents a robust, real-world implementation. The attribution system, with its ownership chain and fallbacks, is not a theoretical model but a production-grade solution. Similarly, the circuit breaker for alert floods (Access Analyzer -> EventBridge -> SQS -> Dead Letter Queue) and the custom logic for DSPM false positives are live mechanisms that Block employs daily to manage millions of potential security events. The mention of AI auto-remediation agents that open PRs against internal repos further solidifies the practical application of advanced techniques, demonstrating how Block is actively experimenting with and deploying intelligent automation to address security issues. The absence of a traditional "demo" does not detract from the technical depth, but rather highlights the maturity of Block's operational security engineering efforts, where the solutions are integrated into their daily workflows.

Defensive Implications

▶ Watch: Dealing with false positives and IAM Access Analyzer limitations (6:47)

The insights shared by Paul Benoit offer crucial defensive implications for security teams operating in cloud environments, particularly at scale:

• Prioritize Context Enrichment: Defenders must move beyond simply forwarding raw security findings. Investing in robust systems to enrich alerts with business context (e.g., application criticality, data sensitivity), risk context (e.g., internet exposure, environment type), and operational context (e.g., ephemeral vs. persistent resources) is paramount. This ensures that alerts are not just accurate, but also relevant and actionable for engineers.
• Develop Sophisticated Attribution Mechanisms: Relying solely on simple resource tagging is insufficient. Security teams should architect a multi-tiered ownership chain with fallbacks, incorporating data from PagerDuty schedules, Git repositories, account ownership, and internal directories. Furthermore, being transparent about how ownership was determined in the alert message fosters trust and empowers engineers to correct outdated information.
• Aggressively Manage False Positives: High false positive rates are detrimental to security posture by eroding trust and inducing alert fatigue. Implement custom logic to filter out known benign findings (e.g., IAM Access Analyzer flags for decommissioned accounts, DSPM alerts for ephemeral or benign data). Deploy circuit breakers (like the SQS-based solution at Block) to prevent alert floods during large-scale deployments or tool misconfigurations, ensuring critical systems aren't overwhelmed.
• Establish a Centralized Exception Management System: Not all security findings are black and white. A dedicated, centralized exception service with tiered approvals and predefined reasons (false positive, planned decommissioning, vendor limitation) allows for a nuanced approach to risk management. This helps security teams say "yes" more often in a controlled manner, fostering better relationships with engineering.
• Focus on Issue Grouping by "Fix": To prevent alert fatigue, configure alerting systems to group related issues into a single ticket that represents one remediation action, rather than one alert per individual finding. This is particularly important for resources managed by autoscaling groups or similar patterns.
• Embrace AI for Auto-Remediation (Strategically): Start experimenting with AI-powered agents to automatically generate PRs for highly deterministic fixes, such as library version upgrades for application vulnerabilities. While more complex for misconfigurations, the potential for significant reduction in manual remediation effort is immense. Defenders should identify low-hanging fruit for automation.
• Foster Empathy and Collaboration: Fundamentally, effective cloud security at scale requires empathy for the engineers receiving alerts. Security teams should strive to make alerts as clear, contextual, and actionable as possible. Proactive collaboration with platform and infrastructure teams is also vital to address systemic issues and prevent patterns of recurring exceptions.
• Implement "Pending Resolution" Status: To improve workflow and reduce "flapping" issues, incorporate a "pending resolution" status in ticketing systems. This allows engineers to signal a fix, pause their SLA, and await re-scanning, streamlining the remediation cycle.

Key Takeaways

• Context is paramount for effective security alerts. Raw findings are noise; enrichment with business, risk, and operational context transforms them into actionable intelligence.
• Robust and transparent attribution is critical. Simple tagging is insufficient at scale; complex ownership chains with fallbacks and clear communication on how owners are identified build trust.
• Aggressive false positive management is essential. Implement custom logic, circuit breakers, and tailored exception handling for tools like IAM Access Analyzer and DSPM to prevent alert floods and maintain engineer confidence.
• Group issues by "one fix" rather than "one finding." This reduces alert fatigue and allows engineers to address the root cause of problems more efficiently.
• A centralized exception service with tiered approvals provides a crucial mechanism for managing nuanced security risks and fostering collaboration with engineering teams.
• AI auto-remediation is a powerful, emerging capability. Start with deterministic fixes (e.g., library upgrades) to automate remediation and free up security and engineering resources.
• Empathy for engineers is non-negotiable. The ultimate success of cloud security alerting hinges on making alerts easy to understand, act upon, and trust.

About the Speaker(s)

Paul Benoit is a Cloud Security Engineer at Block, a global technology company known for its diverse portfolio including Square, Cash App, Afterpay, and Tidal. In his role, Paul works on both the preventive and detective aspects of cloud security. On the preventive side, he is instrumental in developing and implementing guardrails and secure "paved roads" to guide engineers towards secure practices by default. His detective responsibilities include defining what constitutes a secure cloud environment, continuously measuring gaps against those definitions, and, as highlighted in this talk, building and refining the alerting systems that flag deviations. His expertise lies in navigating the complex challenges of cloud security at an enterprise scale, particularly in operationalizing security intelligence to be effective and trustworthy for thousands of engineers.