Challenges implementing egress controls in a large AWS environment

Challenges Implementing Egress Controls in a Large AWS Environment

Speakers: Greg Aumann

Conference: fwd:cloudsec North America 2025

YouTube: https://www.youtube.com/watch?v=W5yH-2mf0o4

Overview

Greg Aumann, a member of Block's cloud security team (previously on the product security engineering team managing the Afterpay AWS environment), presented a detailed and operationally honest account of deploying network egress controls at scale. The environment processes approximately 25 terabytes of egress traffic per day across nearly 200 VPCs in six AWS regions. Aumann covered the architecture, the staggering log volumes, the cost dynamics that can actually make egress controls save money, the practical challenges of building and maintaining allow lists, and several methods for bypassing network firewalls — along with the missing AWS features that would make the entire project easier.

Background

▶ Watch: Scale: 25 TB/day egress across 200 VPCs in six regions (1:30)

Following Block's acquisition of Afterpay three and a half years prior, Aumann was responsible for securing the Afterpay AWS environment, which has a distinct networking architecture from the broader Block environment. The primary threats that egress controls are designed to mitigate are exfiltration of sensitive customer data and command and control (C2) traffic. The mechanisms for these threats include standard network traffic, DNS exfiltration, and abuse of VPC endpoints.

Prior to the egress control project, the environment used a typical architecture: ingress to load balancers, workloads in EKS and EC2, and egress through NAT gateways per VPC. The pre-existing transit gateway network between VPCs became a critical enabler for the centralized egress architecture.

Key Findings

▶ Watch: Log volume shock: 280 GB/day alert logs, 30x improvement with Parquet (8:00)

AWS Network Firewall alert logs are vastly larger than flow logs. On peak traffic days, the system generates 280 GB/day of alert logs versus only 3 GB/day of flow logs — a ratio that was the opposite of what Aumann expected. This is because the monitoring rule set logs all traffic for allow-list development, and each alert contains rich protocol-level detail (SNI, TLS version, server certificates).

Network egress controls can pay for themselves. By gaining visibility into egress traffic patterns, Aumann identified cost-saving opportunities that, if fully implemented, would reduce the total cost of the network egress solution to less than half the cost of the original NAT-gateway-only architecture. The biggest savings come from redirecting vendor traffic (48% of Kubernetes VPC egress went to DataDog domains alone) through AWS PrivateLink connections, bypassing both the firewall and NAT gateway costs entirely.

Application owners do not know their egress destinations. Creating allow lists requires months of observation because application teams seldom have a complete understanding of where their outbound traffic goes. You cannot ask them for an allow list — you must build it from observed traffic.

Default rule ordering in AWS Network Firewall prevents logging allowed traffic. The default "grouped by action" rule ordering places pass rules before alert rules, making it impossible to log traffic that matches the allow list. Switching to strict rule ordering is required, but the Terraform provider is buggy for this migration and may require manual recreation.

Technical Deep Dive

▶ Watch: Allow-list creation requires months of observation (13:30)

The architecture replaces per-VPC NAT gateways with a centralized egress model. NAT gateways are deleted from workload VPCs, route tables are updated to send egress traffic into the transit gateway network, and six inspection VPCs (one per region) contain AWS Network Firewall instances plus NAT gateways. Logs flow to an S3 bucket in a separate logging account.

Critically, the firewall sits before the NAT gateways in the traffic path. This means the firewall sees private source IP addresses rather than NAT-translated addresses, enabling per-VPC identification of traffic sources and the possibility of per-VPC allow lists with different rule sets.

AWS Network Firewall is based on the open-source Suricata intrusion detection/prevention system and supports Suricata's full rule syntax alongside a simplified AWS-specific syntax. Both log types (flow logs and alert logs) include flow IDs that enable correlation. For HTTPS traffic, alert logs capture Server Name Indication (SNI), TLS version, and server certificate information.

The log volume challenge required converting logs from JSON to Parquet format, which improved Athena query performance by approximately 30x — from being able to query only 12 hours before timeout to querying an entire month. Aumann also annotated logs with VPC names during the Parquet conversion to simplify attribution queries.

Several bypass techniques for network egress controls were discussed: SNI forgery (Network Firewall cannot detect forged SNI); Encrypted Client Hello (ECH), promoted by Cloudflare for privacy, which encrypts the destination even from TLS-intercepting proxies (the only mitigation is blocking ECH traffic entirely, as the Great Firewalls of Russia and China do); DNS exfiltration via external resolvers; and VPC endpoint abuse for data exfiltration, though restricting S3 endpoint policies is complicated by AWS services' dependency on special Amazon-owned buckets that are poorly documented.

A notable operational pitfall: deploying network egress to a VPC containing AWS Glue caused all Glue jobs to fail because Glue detects S3 connectivity by checking for the existence of a NAT gateway or S3 VPC endpoint in the VPC — it does not actually test network connectivity.

Demo / Proof of Concept

▶ Watch: Bypass techniques: SNI forgery, Encrypted Client Hello, DNS exfiltration (15:30)

There was no live demo. The talk was a fast-paced operational walkthrough with architecture diagrams, cost breakdowns, and log volume statistics drawn from the production Afterpay environment.

Defensive Implications

▶ Watch: AWS Glue broken connectivity check causes silent job failures (18:00)

Organizations implementing egress controls should plan for a multi-service defense. Network Firewall alone is insufficient — combine it with DNS Firewall (deployed in source VPCs with matching allow lists), CloudTrail Network Activity Logs, and VPC endpoint policies for comprehensive coverage.

Block Encrypted Client Hello traffic if your threat model prioritizes exfiltration prevention over user privacy. Block DNS traffic to external resolvers and maintain DNS logs from the Route 53 Resolver.

Invest heavily in the Parquet conversion pipeline early. The raw JSON log volume from Network Firewall at scale is effectively unqueryable in Athena without format optimization. Plan for months of monitoring-only operation before attempting to enforce allow lists.

Consider the cost optimization angle as a way to fund the project. Publishing monthly charts of the most expensive egress traffic by destination creates organizational pressure to rearchitect inefficient traffic patterns, and PrivateLink migrations for major vendors can dramatically reduce both firewall processing costs and NAT gateway data transfer fees.

Key Takeaways

• Egress controls can be net cost-negative: PrivateLink migrations and traffic rearchitecting can reduce total costs below the pre-firewall baseline by 50% or more.
• Alert logs dwarf flow logs at scale: 280 GB/day of alert logs versus 3 GB/day of flow logs was unexpected and drives the need for Parquet conversion and Suricata threshold rules.
• Application owners cannot tell you their egress destinations: Allow-list creation requires months of traffic observation; you cannot ask teams and get complete answers.
• Default Network Firewall rule ordering blocks allow-list logging: Strict rule ordering is required but the Terraform provider migration path is buggy.
• Encrypted Client Hello is the hardest bypass to mitigate: The only option is blocking ECH entirely, which is a significant policy decision.
• AWS Glue has a broken S3 connectivity check: It tests for NAT gateway or VPC endpoint existence rather than actual connectivity, causing silent failures when migrating to centralized egress.

About the Speaker(s)

Greg Aumann is a member of Block's cloud security team with eight years of professional experience securing AWS environments. Prior to cloud security, he worked as an AWS infrastructure and platform engineer, and before that held several networking-focused roles. His networking background directly informed the architecture of the centralized egress solution for the Afterpay AWS environment.