From One to Hundreds: Reflections on a Decade of Building the Trenches

From One to Hundreds: Reflections on a Decade of Building the Trenches

Speakers: Joel Thompson, Co-founder, fwd:cloudsec

Conference: fwd:cloudsec Europe 2025

YouTube: https://www.youtube.com/watch?v=RAo8lN28xGk

Overview

Joel Thompson, co-founder of fwd:cloudsec, delivered a deeply practical retrospective on scaling an AWS footprint from a single test account to hundreds of production accounts over the course of a decade. Drawing from his experience as a "secure builder" — someone who always prioritized security without having it in his official job title — Thompson shared hard-won lessons about declarative inventory management, infrastructure-as-code tooling, CI/CD patterns for account provisioning, and account-level security patterns including DNS delegation and VPC endpoint policies. The talk is a rare look at the operational reality of enterprise AWS governance from someone who both built the platform and consumed it.

Background

▶ Watch: Joel introduces the vision of treating AWS accounts as cattle, not pets (1:30)

When Thompson started working with AWS around fall 2013, the cloud security landscape was fundamentally different. AWS CloudTrail had just launched at re:Invent 2013. There were no CloudFormation stack sets, no resource control policies, and many of the security primitives organizations now take for granted simply did not exist. The company operated a federated technology model — centralized security policies and controls, but empowered technology teams that were expected to be as self-service as possible. This tension between centralized governance and developer autonomy became the defining challenge of the entire journey.

The core hypothesis emerged early: AWS accounts should be treated as cattle, not pets. Just as the industry had learned to treat servers as ephemeral, disposable units configured through tools like Chef or Puppet, the same philosophy needed to apply to accounts. This meant having a large number of accounts but a deliberately small number of account shapes — standardized templates that could be reasoned about, secured, and replicated at scale. The DRY principle (Don't Repeat Yourself) from software engineering would be applied to account governance, bundling low-level security properties into higher-level abstractions.

Key Findings

▶ Watch: Declarative inventory as the source of truth for your entire AWS estate (6:00)

Thompson distilled his decade of experience into several key observations about scaling AWS account governance:

Informed security decisions require a small number of account shapes. With only two or three account shapes (e.g., "secure" and "playground"), security teams gain reusable precedent for answering questions. When someone asks "Can I do X in a secure account?", the answer creates precedent that applies to every secure account. This eliminates the need to reason about hundreds of individual accounts with subtle differences, enables quicker recall about each shape, and allows deeper knowledge accumulation about trade-offs and implications.

A declarative, computer-readable inventory is the foundation of everything. The inventory declares what accounts exist and what shape they should be — not how to make them that shape. This inventory must become the single source of truth for the entire account footprint, driving all automation and serving as an interface for developers to understand and even propose changes to accounts via pull requests.

Console clicking is a slippery slope. One-off manual changes to accounts outside the declarative inventory erode trust in the inventory, require tribal knowledge to understand, and become intractable at scale. Thompson illustrated this with a progression from a simple inventory to one riddled with undocumented exceptions. The answer is not to reject unexpected requests but to maintain engineering flexibility to implement them in code within a reasonable timeframe (approximately one week).

Idempotent custom code is necessary but expensive. CloudFormation cannot do everything, and custom code must correctly handle transitions between any two valid states. This adds developer time, testing complexity, and potential bugs — a trade-off Thompson considers well worth it but important to acknowledge upfront.

Technical Deep Dive

▶ Watch: Why console clicking is a slippery slope that destroys inventory trust (11:30)

The tooling architecture centered on generating CloudFormation templates from the declarative inventory. Before stack sets existed, this meant deploying individual stacks to each account. As account shapes grew more complex, the team repeatedly hit CloudFormation limits — particularly the logical resources per stack limit and the JSON template size limit. Thompson admitted to minifying JSON templates, a hack he described with appropriate remorse.

A persistent pain point was CloudFormation's inability to perform non-destructive stack refactoring. Moving resources between stacks naively causes CloudFormation to delete and recreate them, which is unacceptable when the resource in question is a production network. The convergence speed of CloudFormation also imposed a constant time tax on development and testing cycles.

For the CI/CD pipeline, the team adopted an open-source-like philosophy — all developers in the company could see the codebase and submit pull requests (though not approve them). This fostered collaboration, helped developers understand the platform team's work, and occasionally produced free contributions.

Automated testing was critical but challenging. The ideal — spinning up fresh AWS accounts per test run and discarding them afterward — proved infeasible. Instead, the team created a dedicated AWS Organization for testing with a pool of pre-created accounts managed through a reservation system. Failed tests frequently left orphaned resources that caused subsequent failures, a constant drag on velocity and morale.

Two account-level security patterns received detailed treatment. First, DNS subdomain delegation: individual AWS accounts receive delegated Route 53 subdomains, enabling fully self-service DNS management within those subdomains. This eliminated excuses for not using TLS everywhere, eliminated certificate expiration incidents through ACM auto-renewal, and critically limited the blast radius of DNS changes — a compromised account could only affect its own subdomains.

Second, VPC endpoint policies: Thompson pushed back against using these for application-level security (e.g., restricting which applications access which S3 buckets) and instead advocated using them exclusively for data exfiltration and data perimeter concerns. The majority of accounts only needed VPC endpoints for same-account principal-to-resource communication. The recommended policy pattern uses aws:PrincipalAccount and aws:ResourceAccount conditions on a wildcard allow, cleanly implementing the semantic of "this endpoint is only for principals and resources in this account."

Demo / Proof of Concept

▶ Watch: Dedicated AWS Organization for testing with account reservation system (14:00)

This talk was a reflective presentation rather than a tool demonstration, so no live demo was shown. However, Thompson walked through concrete examples of inventory configurations, DNS delegation setups, and VPC endpoint policy patterns using slides.

Defensive Implications

▶ Watch: VPC endpoint policies for data exfiltration control using account conditions (18:00)

Organizations managing multi-account AWS estates should prioritize:

• Establishing a declarative inventory as the single source of truth for all accounts, their shapes, and their configurations. This is the foundation upon which all other automation and security reasoning depends.
• Resisting the temptation to make one-off console changes, even under pressure. Every click outside the inventory erodes its trustworthiness and creates hidden technical debt.
• Implementing DNS subdomain delegation to limit blast radius, enable TLS everywhere, and eliminate certificate management as a manual process.
• Using VPC endpoint policies for data perimeter enforcement rather than application-level access control, with conditions on principal and resource account IDs.
• Investing seriously in automated testing for account provisioning tooling, recognizing that test infrastructure for AWS Organizations is complex and requires its own engineering investment.
• Deploying changes to canary accounts first before rolling them out to production, particularly for new AWS features like resource control policies.

Key Takeaways

• Treat AWS accounts as cattle, not pets — use a small number of standardized account shapes that can be reasoned about and secured at scale.
• A complete, declarative inventory must be the single source of truth for your entire AWS footprint; never make out-of-band console changes.
• Understand the limits of your infrastructure-as-code tooling (CloudFormation, Terraform, etc.) before you hit them in production.
• DNS subdomain delegation enables self-service, eliminates certificate incidents, and contains blast radius.
• VPC endpoint policies are best used for data perimeter enforcement, not application-level access control.
• Maintain engineering flexibility to handle unexpected feature requests within reasonable timelines to prevent shadow AWS usage.

About the Speaker(s)

Joel Thompson is a co-founder of fwd:cloudsec and spent over a decade building and securing enterprise AWS infrastructure. He describes himself as a "secure builder" — someone focused on building solutions in a highly secure fashion while simultaneously delivering business-facing applications. His experience spans the entire evolution of AWS enterprise patterns from the pre-CloudTrail era to the present day, giving him a unique perspective on what works (and what doesn't) at scale.