Beyond the Perimeter: Retrofitting VPC-SC at Enterprise Scale

Beyond the Perimeter: Retrofitting VPC-SC at Enterprise Scale

Speakers: Priya Puranik (Senior Cloud Security Engineer, Wayfair), Akshay Mahajan (Production Security Team Lead, Wayfair)

Conference: fwd:cloudsec

YouTube: https://www.youtube.com/watch?v=0PCw7kb5y7c

Overview

In the modern cloud landscape, traditional network firewalls are increasingly insufficient to prevent data exfiltration. While Identity and Access Management (IAM) controls dictate who can access data, they often fall short in defining where that data is allowed to go and under what conditions. This critical gap is precisely what VPC Service Controls (VPCSC) aims to address, establishing a robust data perimeter around sensitive cloud resources. This talk by Priya Puranik and Akshay Mahajan from Wayfair delves into the intricate journey of retrofitting VPCSC at an enterprise scale, a monumental task involving over 5,000 GCP projects.

The speakers, both instrumental in Wayfair's production security, share their practical experience in designing, implementing, and managing VPCSC in a brownfield environment. Their insights are particularly valuable for organizations grappling with the complexities of securing vast, existing cloud footprints against sophisticated data exfiltration vectors, such as compromised service account keys. The presentation offers a candid exploration of the challenges encountered, the strategic decisions made, and the lessons learned in transforming a theoretical security concept into an operational, scalable defense mechanism within a large-scale enterprise.

This article dissects Wayfair's approach to implementing a data perimeter, highlighting their methodology for discovery, analysis, and enforcement. It underscores the importance of a phased rollout, meticulous planning, and leveraging automation to overcome the inherent difficulties of retrofitting advanced security controls. For any organization looking to enhance their Google Cloud security posture and protect sensitive data from exfiltration, Wayfair's journey provides a detailed blueprint and a wealth of actionable advice.

Background

▶ Watch: Introduction: The data exfiltration problem (1:00)

The evolution of cloud security has brought forth new challenges that traditional security paradigms struggle to address. A core problem highlighted by the speakers is the inherent limitation of network firewalls and even IAM in fully preventing data exfiltration. While IAM effectively controls who has access to data, it doesn't dictate where that data can be moved or under what circumstances. This leaves a significant vulnerability, particularly in scenarios involving compromised service account keys, where an attacker could potentially exfiltrate data to an untrusted external location even with valid credentials.

Google Cloud's VPC Service Controls (VPCSC) emerges as the solution to this problem, designed specifically to establish a data perimeter. The speakers clarify a common point of confusion: VPCSC is distinct from Virtual Private Cloud (VPC). While VPC provides private network connectivity for compute resources, VPCSC focuses on data exfiltration control, defining the boundaries within which sensitive data can operate.

The fundamental components of VPCSC, as explained, include:

• Service Perimeter: This is the core boundary, a logical "bold line" drawn around a set of trusted GCP projects. By default, communication between projects inside the perimeter is allowed, but communication between projects inside and outside the perimeter is blocked.
• Restricted Services: Within a service perimeter, specific GCP services (e.g., BigQuery, Cloud Storage, Pub/Sub) are designated as "restricted." This means VPCSC enforces policies on data movement for these services.
• Access Context Policies: These policies provide granular control over access based on various conditions. They allow defining rules for ingress (who and which destinations are allowed to call into trusted accounts) and egress (who is allowed to call out of the trusted perimeter). Conditions can include user identity, service account, source IP address, region, device management status (e.g., managed device), and more.

While implementing VPCSC in a greenfield environment (new projects) is relatively straightforward, the talk focuses on the much more complex scenario of a brownfield implementation. For an enterprise like Wayfair, with thousands of existing projects and established application patterns, retrofitting VPCSC involves navigating potential service disruptions, analyzing billions of logs, and meticulously identifying and remediating exceptions before enforcement. This distinction between greenfield and brownfield significantly impacts the level of effort required for discovery, remediation, and ultimate enforcement.

Key Findings

▶ Watch: Understanding VPCSC components: Perimeters, rules, policies (2:40)

Wayfair's journey in retrofitting VPCSC at enterprise scale yielded several critical findings and strategic approaches:

1. Design Model Selection: The choice between a monolithic and multi-perimeter design is crucial. A monolithic design offers simpler management and robust defense against external threats, making it suitable for scalable enterprise-wide protection. A multi-perimeter model, conversely, provides granular segmentation and strict isolation for highly sensitive data, often requiring cross-perimeter bridges for controlled interactions. Wayfair's discussion implies a lean towards the monolithic model for broader enterprise defense, while acknowledging the multi-perimeter's benefits for specific high-sensitivity use cases.

1. Strategic Resource and Service Inclusion:

• Resource Selection: Defining the scope of deployment (entire account, specific folders, or individual projects) is the initial step that dictates the complexity of subsequent phases.
• Service Inclusion: A counter-intuitive but highly effective strategy is to start with high-volume, high-complexity services like BigQuery and Cloud Storage. While riskier, success here validates the approach and ensures that the most critical data assets are protected from the outset. Starting with low-complexity services can lead to a false sense of security, only to encounter insurmountable difficulties when attempting to secure high-value targets later.

1. Hierarchy Alignment for Automation: Integrating VPCSC enrollment with existing platform automations is vital. This ensures that new resources are automatically included within the perimeter, preventing "bolt-on" security solutions that can lead to gaps and unmanaged resources falling outside the perimeter.

1. Dry Run Mode as a Cornerstone: For brownfield deployments, the dry run mode is indispensable. It allows VPCSC to log potential policy violations without actively blocking requests. This generates a massive volume of logs (tens of thousands to millions), which are crucial for identifying existing access patterns that would be blocked under enforcement.

1. Centralized Logging and Visual Insights: Managing the sheer volume of dry run logs necessitates centralized logging. Wayfair implemented a log sync to direct all VPCSC error logs from individual project GCL logging to a central BigQuery dataset or GCS bucket, significantly simplifying analysis. The Google VPCSC dashboard was also highlighted for its ability to aggregate similar findings, providing visual insights and reducing the analytical burden.

1. Pattern-Based Exception Handling: Wayfair categorized findings into four key patterns for managing exceptions:

• User Compliance: Handled using context-aware access policies, verifying factors like managed devices or trusted networks.
• Trusted Applications: Preferred service account impersonation or Workload Identity over static keys for in-house applications.
• External/Vendor Applications: Integrated the allow-listing process with their Third-Party Risk Management (TPRM) system, ensuring legitimate business needs and approved vendors. Exceptions were tied to contract end dates.
• Google Dependencies: Acknowledging that GCP services have inter-dependencies (e.g., Cloud Build storing artifacts in GCS), requiring specific allow-lists for Google-owned projects.

1. Scalable Exception Management with Google Groups: To address the challenge of managing exceptions across thousands of projects, Wayfair leveraged Google Groups. Identities requiring exceptions are added to a Google Group, which is then referenced in the VPCSC policy. This allows for dynamic updates to access without modifying the VPCSC policy directly, significantly enhancing scalability. An automated workflow was also developed for adding service accounts to these groups.

1. Understanding GCP Limitations: The team identified specific limitations, such as:

• Cloud Storage (GCS): VPCSC error logs for GCS public buckets only show the resource name (bucket name), not the project ID, making project-based allow-listing difficult.
• BigQuery: Error logs might not list all external resources queried, potentially leading to missed allow-lists. Log syncs to BigQuery itself can also generate errors if the BigQuery dataset is outside the perimeter.
• Google-owned Projects: Services like Cloud Build operate from Google-owned GCP projects. When Cloud Build interacts with a restricted service (e.g., storing artifacts in a GCS bucket within the perimeter), VPCSC will block it, requiring specific allow-lists for Google's internal project IDs.

These findings collectively underscore the complexity but also the feasibility of implementing VPCSC at an unprecedented scale, provided a methodical and automated approach is adopted.

Technical Deep Dive

▶ Watch: Road to enforcement: Design, discovery, remediation (4:40)

Implementing VPCSC at Wayfair's scale (over 5,000 GCP projects) required a deeply technical and programmatic approach. The core of their strategy revolved around leveraging Infrastructure as Code (IaC) and robust logging mechanisms.

VPCSC Core Components in Practice:

• Service Perimeters: Wayfair defined its service perimeter as the "bold line" encircling its trusted GCP projects. This perimeter inherently blocks any direct communication from projects or resources outside this boundary to restricted services inside it, and vice-versa, unless explicitly allowed by Access Context Policies.
• Restricted Services: For critical data stores, Wayfair primarily focused on BigQuery and Cloud Storage (GCS) as initial targets due to their high volume and complexity. The decision to prioritize these services first, despite their complexity, was a strategic one to ensure the most sensitive data was protected early.
• Access Context Policies (ACPs): These are the brain of VPCSC enforcement. Wayfair utilized ACPs to define granular ingress and egress rules.
• Ingress Rules: Specify who (e.g., specific users, service accounts, Google Groups) and from where (e.g., specific IP ranges, regions, managed devices) are allowed to access resources within the perimeter.
• Egress Rules: Define who is allowed to initiate calls out of the perimeter and to which external destinations.
• Context-Aware Access: For human identities, Wayfair implemented context-aware access, checking factors like whether the user is accessing from a managed device (verified via Google Workspace integration). This aligns with a zero-trust philosophy.
• Service Account Impersonation/Workload Identity: For trusted in-house applications, these mechanisms were preferred over static service account keys, enhancing security by eliminating long-lived credentials.

Deployment Strategy: Terraform for Enterprise Scale:

Wayfair explicitly chose Terraform for its VPCSC deployment. The reasons were compelling:

• Scalable Governance: Terraform facilitates managing VPCSC policies across thousands of projects by defining them in code, enabling version control, peer review, and automated deployment.
• Auditable IaC: All changes to the VPCSC perimeter are captured in Terraform configurations, providing an auditable history of policy modifications. This is a significant advantage over manual changes via the Google Cloud Console or G-Cloud CLI, which lack inherent auditability at scale.
• Native Feature Support: Terraform providers for GCP offer comprehensive support for VPCSC features, allowing Wayfair to implement complex perimeter structures and access policies programmatically.

Discovery and Analysis Phase (The "Dry Run" Engine):

The dry run mode was the cornerstone of Wayfair's brownfield discovery.

• Mechanism: When enabled, VPCSC runs in a monitoring-only mode. It logs any requests that would violate the perimeter policies if enforcement were active, but it does not block them. This is crucial for identifying existing operational patterns without causing outages.
• Log Volume: Depending on service usage, dry run mode can generate "tens of thousands to potentially millions" of error logs.
• Centralized Logging: To manage this immense log volume, Wayfair implemented a log sync. VPCSC error logs, which are initially scattered across individual project's GCL logging, were routed to a centralized BigQuery dataset or a GCS bucket. This provided a single, queryable source for analysis.
• Visual Insights: The Google VPCSC dashboard was utilized to aggregate similar findings, reducing noise and highlighting common patterns that required attention.
• Pattern Identification: Logs were analyzed to identify recurring patterns of access, categorized as user compliance, trusted applications, external/vendor applications, and Google dependencies. This structured approach helped prioritize and streamline remediation efforts.

Exception Management and Scalability:

The challenge of managing exceptions for 5,000+ projects was addressed through:

• Google Groups: Instead of directly adding individual identities to VPCSC policies, Wayfair created Google Groups. These groups were then allow-listed in the VPCSC access policies. When an identity (user or service account) needed an exception, it was added to the relevant Google Group. Google automatically manages the syncing of group members with the policy, although there's a slight delay (around 5-10 minutes).
• Automated Workflow: An automated script was developed to add service accounts to Google Groups, integrating with their SSO system. This streamlined the exception granting process.
• TPRM Integration: For vendor applications, the allow-listing process was integrated with Wayfair's Third-Party Risk Management (TPRM) system. This ensured that only approved vendors with legitimate business needs received exceptions, and these exceptions were tied to contract end dates for automated removal.

Limitations Encountered:

Wayfair also highlighted specific GCP limitations that required careful handling:

• GCS Public Buckets: VPCSC error logs for GCS public buckets provide only the bucket name, not the project ID. Since VPCSC allow-lists are often project-based, this required manual lookup to identify the hosting project for each bucket needing an exception.
• BigQuery Query Details: When a BigQuery query accesses multiple resources outside the perimeter, not all of them might be explicitly listed in the VPCSC error log. This could lead to incomplete allow-lists and subsequent errors upon enforcement.
• Google-Owned Projects (e.g., Cloud Build): Services like Cloud Build, which operate from Google's own GCP projects, can interact with customer-owned resources. If Cloud Build needs to store artifacts in a GCS bucket within the customer's VPCSC perimeter, the call will be blocked because it originates from a Google-owned project outside the perimeter. Specific allow-lists for these Google-owned project IDs are necessary.

This deep dive illustrates that successful VPCSC implementation at scale is not merely about enabling a feature, but about a comprehensive engineering effort involving strategic design, robust automation, meticulous analysis of operational patterns, and a thorough understanding of the underlying cloud platform's nuances.

Demo / Proof of Concept

▶ Watch: Architecting the perimeter: Resource selection and inclusion (6:20)

The talk focused on Wayfair's journey and implementation strategy for retrofitting VPCSC at enterprise scale, rather than demonstrating a specific live demo or proof of concept. The speakers described the practical processes of configuring dry run mode, analyzing logs, and implementing exception handling using Terraform and Google Groups. While no live code execution or interactive demonstration was part of the presentation, the detailed methodology and specific examples provided serve as a robust blueprint for other organizations undertaking similar initiatives.

Defensive Implications

▶ Watch: Recommendation: Start with high-complexity services first (7:10)

Wayfair's experience provides invaluable insights for defenders looking to strengthen their cloud security posture with VPCSC:

1. Proactive Adoption is Key: If you're early in your GCP journey (greenfield), enable VPCSC early. It is significantly easier to establish security patterns that align with VPCSC from the outset rather than retrofitting them into an existing, complex environment. This minimizes disruption and accelerates time to security.

1. Strategic Design Decisions: Carefully evaluate whether a monolithic or multi-perimeter design best suits your organization's needs. Consider factors like desired level of segmentation, operational complexity, and the sensitivity of various data types. A monolithic approach might offer simpler management for broad enterprise protection, while multi-perimeter is ideal for strict isolation of highly sensitive data.

1. Prioritize High-Value Assets First: Contrary to intuition, start VPCSC enforcement with your high-volume, high-complexity, and most sensitive services (e.g., BigQuery, Cloud Storage). Successfully securing these critical assets provides immediate value and validates the overall strategy, building momentum for broader adoption.

1. Embrace Infrastructure as Code (IaC): Utilize tools like Terraform for VPCSC deployment. IaC ensures scalable governance, auditable changes, and consistent policy application across thousands of projects, which is critical for enterprise-level deployments. Avoid manual configurations via the console for anything beyond initial testing.

1. Robust Logging and Monitoring are Non-Negotiable: Implement centralized logging for all VPCSC dry run and enforcement logs. Direct these logs to a dedicated BigQuery dataset or GCS bucket for efficient analysis. Leverage dashboards (like the Google VPCSC dashboard) to aggregate findings and identify patterns quickly. This visibility is crucial for understanding potential impacts and validating policy effectiveness.

1. Granular Access Control and Zero Trust: Leverage Access Context Policies to define precise ingress and egress rules. Incorporate context-aware access for human identities (e.g., requiring managed devices). For service accounts, prioritize service account impersonation or Workload Identity over static keys to minimize the attack surface.

1. Automate Exception Management: Implement automated workflows for granting exceptions, particularly by using Google Groups. This approach provides scalability, allows for dynamic updates to access, and reduces manual overhead. Integrate with existing systems like SSO and TPRM to ensure exceptions are legitimate, temporary, and tied to business requirements.

1. Understand GCP-Specific Nuances: Be aware of the limitations and unique interactions of VPCSC with various GCP services, especially those involving Google-owned projects (like Cloud Build) or specific resource logging behaviors (like GCS bucket names). Engage with Google Cloud representatives for clarification and to advocate for feature requests.

1. Communicate Extensively: Establish clear pre- and post-enforcement communication channels with application owners. Inform them about impending changes, potential impacts, and how to report issues. Proactive communication minimizes surprises and fosters collaboration between security and engineering teams.

1. Advocate for Product Improvements: If you encounter bugs or identify needed features, open feature requests with Google. Collective feedback helps improve the VPCSC product for everyone.

By internalizing these defensive implications, organizations can navigate the complexities of VPCSC implementation more effectively, transforming it from a daunting task into a powerful layer of defense against data exfiltration in the cloud.

Key Takeaways

• VPCSC is Essential for Data Exfiltration Control: It complements IAM by defining where data can go and under what conditions, crucial for protecting against compromised credentials.
• Brownfield Implementation Requires Extensive Discovery: For existing environments (like Wayfair's 5,000+ GCP projects), dry run mode and centralized logging are critical for identifying and analyzing millions of potential policy violations before enforcement.
• Terraform and Google Groups Enable Scalability: Using Infrastructure as Code (Terraform) for policy deployment and Google Groups for managing exceptions are fundamental to achieving auditable, maintainable, and scalable VPCSC at enterprise scale.
• Prioritize High-Value Services and Integrate Automation: Start with complex, high-volume services (e.g., BigQuery, GCS) to ensure core data protection, and integrate VPCSC into existing platform automations and Third-Party Risk Management (TPRM) systems for seamless governance.
• Understand GCP Nuances and Communicate Proactively: Be aware of service-specific limitations (e.g., Cloud Build, GCS logging) and engage in robust pre- and post-enforcement communication with application owners to ensure a smooth rollout and effective remediation.

About the Speaker(s)

Priya Puranik is a Senior Cloud Security Engineer at Wayfair. She plays a pivotal role in driving the execution of Wayfair's data perimeter roadmap. Her expertise lies in the practical implementation and analysis of complex cloud security controls, particularly in large-scale GCP environments.

Akshay Mahajan is a Production Security Team Lead at Wayfair. He leads the team responsible for designing and implementing security controls for Wayfair's production applications. His experience encompasses the strategic planning and oversight required to secure critical cloud infrastructure at an enterprise level. Together, Priya and Akshay bring a wealth of practical experience from their work at Wayfair, a major e-commerce company leveraging Google Cloud Platform extensively.