Centralizing Egress Access Controls Across a Hybrid Environment

Centralizing Egress Access Controls Across a Hybrid Environment

Speaker: Ramesh Ramani

Conference: BSidesSF 2025 — April 26-27, 2025, San Francisco

YouTube: Watch on YouTube

Reading time: ~6 minutes

TL;DR

When applications running across Kubernetes clusters, multiple clouds, and on-premises data centers each manage their own egress rules, the result is a fragmented, unauditable mess. Ramesh Ramani, a security engineer at Block, presented a centralized egress access control system that unifies policy creation, enforces partner compliance, and deploys rules automatically to heterogeneous enforcement endpoints — all through a single UI backed by a structured group management repository and LLM-assisted validation.

Introduction

Network egress control — governing what your applications are allowed to talk to on the internet — sounds like a solved problem. Set your firewall rules, define your allowlists, and move on. But for an organization operating at the scale and complexity of Block, with applications spread across Kubernetes clusters, cloud-native services (Lambdas, EC2 instances), and physical data center equipment, the naïve approach collapses quickly. Each technology stack has its own policy language. Each business unit may manage its own enforcement endpoint. And when an incident occurs, determining which applications communicate with a newly-compromised partner requires parsing dozens of disconnected log sources.

Ramesh Ramani, a security engineer with fifteen years of experience and over five years at Block, opened his BSidesSF 2025 talk by framing the challenge as building a network airport: a system with consistent security checkpoints, clear routing, and comprehensive visibility over everything that tries to leave the environment.

Why Centralized Egress Controls Matter

▶ Watch: The problem at Block (04:00)

Without centralized egress controls, three categories of risk accumulate:

Data exposure and exfiltration. Applications that can reach arbitrary internet destinations can inadvertently — or maliciously — send sensitive data to unauthorized recipients. Egress controls act as the inspection layer, equivalent to a TSA checkpoint that examines what is leaving, not just what is arriving.

Regulatory compliance. For an organization subject to PCI and GDPR, understanding exactly where customer data flows, and maintaining audit trails for those decisions, is not optional. Distributed egress policies make this audit impossible; centralized ones make it tractable.

Incident response latency. When a partner is compromised, determining which internal applications communicate with that partner should be instantaneous. In a fragmented environment it requires correlating VPC flow logs, Kubernetes telemetry, proxy logs, and firewall logs across multiple systems — a process that delays response and increases exposure.

Block's environment includes Kubernetes network policies, Kubernetes DNS policies, Istio egress gateways, Suricata-backed firewalls, and proxy infrastructure, each speaking a different configuration language. Before this system, an application team needing to whitelist a new domain had to navigate multiple configuration systems, often with no automated validation that the domain or partner was approved.

System Architecture: Five Components

▶ Watch: Solution architecture overview (10:00)

Ramani's system consists of five integrated components:

1. Centralized UI — A single interface where application owners request egress access. Users search for a domain group (representing a partner or destination), then request membership for their application workload. The UI collects the application's name, the intended destination, the port and protocol, and critically the intended data security level — classifying the sensitivity of the data to be shared.

2. Group Management System (GMS) — A structured repository that maintains the canonical mapping of application-to-domain. Every application in Block's environment is identified by a SPIFFE ID, an X.509-based identifier that encodes rich contextual metadata: application name, business unit, environment (production/staging), and region. A SPIFFE ID for a payment processor in the Cash business unit's US West production environment might read: spiffe://block.com/app/cash/production/uswest/payment-processor. Every domain is treated as a first-class object — a group file in the repository — and SPIFFE IDs of authorized applications are written into the corresponding domain group file when access is granted.

▶ Watch: Governance policies and LLM validation (16:01)

3. Governance Policies — All access requests go through multi-party approval. A request to access an existing domain group (e.g., github.com) is validated by the system and then approved by an application-owning team member. A request to create a new domain group routes to the network security team for approval. Both paths use LLM-assisted validation: when a user submits a domain, port, protocol, and product ID, the LLM cross-references the Block Software List to verify that the domain legitimately belongs to the identified partner and that the partner is approved within Block's ecosystem.

4. Block Software List — A central catalog of all partners, describing their company name, product URL, product ID, approved data security levels, and information sharing policies. This catalog is the authoritative source of truth consulted during validation. It defines what type of data each partner is permitted to receive — enabling the system to automatically deny or grant access based on whether the intended data security level matches the partner's approved classification. If a partner's compliance status changes (losing a certification, for example), the system automatically identifies and updates all affected access mappings without manual intervention.

5. Automation Modules — Serverless components that read the GMS repository, identify each application's SPIFFE ID, determine the corresponding enforcement endpoint based on the business unit, and deploy the appropriate policy rule. A Kubernetes workload gets a Kubernetes network policy or DNS policy. A firewall-protected service gets a firewall rule. The translation and deployment are automatic — within seconds of an access request being approved, policies are live across all relevant enforcement endpoints and cloud environments.

End-to-End Workflow

▶ Watch: End-to-end access request workflow (22:02)

Walking through a concrete example: an application owner for kubenetsec needs to reach chatgpt.com for LLM transactions. They open the UI, search for a chatgpt.com domain group. If the group already exists, they request membership, specifying their workload and the intended data security level. The system validates the request against the Block Software List via the LLM — confirming chatgpt.com belongs to OpenAI and that OpenAI is an approved partner at the intended data sensitivity. An application-owning team member approves, the SPIFFE ID is written to the chatgpt.com group file in GMS, and the automation module deploys the corresponding rule to the appropriate enforcement endpoint.

If chatgpt.com does not yet exist as a domain group, the user creates a domain creation request, providing domain, port, protocol, and product ID. The LLM validates the association, and the request is routed to the network security team for approval before the domain is created and populated.

The result is a complete fingerprint of every application's external communication partners, maintained in a version-controlled repository, accessible in a single UI, and enforceable across every infrastructure technology in Block's stack.

Notable Quotes

"Because we maintain this mapping of application to domain in a central repository, we're now able to take intelligent actions on this — we have lightweight modules which actually pick up this mapping, understand where these egress policies have to be deployed, and deploy it seamlessly regardless of where the application exists." — ▶ 06:00

"If a particular partner is no longer approved to receive a particular type of data, the system automatically identifies this and updates it without any intervention." — ▶ 08:00

"We're actually creating paved paths. Users and product managers used to go to different systems to set this up. Now it's just one system. If anything, they'd be happy about it." — ▶ 28:02

Key Takeaways

• SPIFFE IDs enable cross-environment policy portability. Using a workload identity standard that encodes business unit, environment, and region means the same access request mechanism works uniformly across Kubernetes, cloud-native, and on-premises deployments.
• Treating domains as first-class objects enables intelligent automation. When every partner domain is a named group with SPIFFE IDs as members, automation modules can translate that single representation into whatever enforcement language a given infrastructure requires.
• LLM-assisted validation closes the approval gap. Integrating an LLM to verify domain-to-partner associations against a central catalog reduces human error and prevents unauthorized domains from being added through mistakes or social engineering.
• A central partner catalog transforms compliance from reactive to proactive. When partner compliance status changes, the system can automatically propagate the access policy change across all affected applications — no manual audit required.
• Centralized egress dramatically improves incident response. When a partner is compromised, finding every application that communicates with it is a single query against the GMS repository rather than a multi-system log correlation exercise.