This Wasn't in the Job Description: Building a production-ready AWS environment from scratch

This Wasn't in the Job Description: Building a Production-Ready AWS Environment from Scratch

Speakers: Mohit Gupta; Nick Jones

Conference: fwd:cloudsec North America 2025

YouTube: https://www.youtube.com/watch?v=rai0bTOamG0

Overview

Nick Jones and Mohit Gupta, both security consultants (penetration testers) at Reverse X (formerly WithSecure Consulting), delivered an unusually candid account of building an entire production AWS environment from scratch following their company's divestiture from WithSecure. With no dedicated cloud engineering team, no substantial budget, and only three IT staff without AWS experience, two offensive security consultants took on the challenge of standing up a fully functional, security-hardened AWS organization. The talk covers their OU structure, authentication strategy, networking architecture, CI/CD automation, and monitoring approach — all through the lens of practitioners who know exactly how attackers exploit the gaps they were trying to close.

Background

▶ Watch: Origin story: divestiture from WithSecure, pentesters become cloud engineers (1:30)

Reverse X was born when Finnish parent company WithSecure sold off its consulting arm to a Swedish private equity firm. As with any divestiture, the IT infrastructure had to be rebuilt from the ground up. The company is an offensive security consultancy of approximately 160 consultants across Europe, with small teams in the US and Singapore. They are not a software company, have no hard SLAs, and are decidedly not well-funded.

What makes their AWS requirements unusual is three fundamentally different workload types with opposing security postures: client delivery accounts where consultants need freedom to stand up C2 infrastructure, phishing redirectors, and port scanners on demand; research accounts that must be an extremely lax sandbox for R&D experimentation; and production accounts hosting sensitive client vulnerability data, reporting tools, and source code repositories that require enterprise-grade hardening. Building a single organization that accommodates all three is a genuine architectural challenge.

Key Findings

▶ Watch: OU structure and account segregation strategy explained (10:00)

The team arrived at several opinionated but well-defended architectural decisions:

Control Tower was rejected early because its opinionated defaults conflict with the extreme flexibility required by consulting workloads. They borrowed SCPs from it but built their own organization management.

KMS keys are largely unnecessary for their threat model. Drawing on extensive offensive experience, Nick argued that he has never encountered a breach where custom KMS key policies would have been the decisive defensive control. The operational overhead of managing per-resource KMS keys was deemed not worth the investment for their use case, aligning with what he called the "Chris Farris school of thought."

IAM users still have a place — specifically for management account break-glass access. By deliberately not federating the management account through Entra ID and instead using two IAM users (one per administrator) with aggressive alerting, they add significant friction to the attack path of an Entra compromise propagating to organizational control.

Resource Control Policies (RCPs) proved difficult to apply broadly because many standard RCP examples (like S3 bucket hardening) break legitimate cross-organization data transfer workflows that consultants rely on daily. Only trusted OIDC assumption RCPs were deployed.

Technical Deep Dive

▶ Watch: The KMS debate: why custom keys are not worth the overhead (20:30)

The organization structure uses five top-level OUs: Core (security, logging, identity, networking, DNS), Workloads (prod and dev), Research, Client (per-client accounts for engagement isolation), and a General Service OU for shared tooling like Burp Collaborator instances.

Account creation is fully Terraform-managed with modules that automatically provision: Entra ID groups for admin and read-only access, Identity Center mappings, custom IAM roles, OIDC provider configurations for GitLab, a Terraform state file S3 bucket, and CI/CD deployment roles with trust relationships scoped to specific repositories and branches.

Networking uses a shared VPC model with AWS RAM for core workloads. Each workload receives a numeric offset that Terraform uses to automatically calculate and create subnet CIDR blocks (public, private, internal) with bespoke routing tables. Tailscale serves as the VPN layer, with subnet routers in the shared VPC enabling access controlled by the combination of Tailscale ACLs and AWS security groups. Research and client accounts get isolated VPCs with no corporate connectivity — only a shared prefix list of public IPs is distributed via RAM for ingress restriction.

DNS management uses a central DNS account with delegated record management per account. IAM conditions (route53:ChangeResourceRecordSetsNormalizedRecordNames and record type restrictions) constrain delegated accounts to specific subdomains and record types (A, AAAA, CNAME only), preventing NS record hijacking. ACM validation records are matched using a pattern of an underscore followed by 32 hexadecimal characters.

Authentication separates access tiers: core and workload accounts require cloud admin accounts with stricter Conditional Access Policies, while research and testing accounts use regular employee credentials. The management account is deliberately disconnected from Entra federation entirely.

CI/CD runs through GitLab with a group structure mirroring the AWS OU hierarchy. OIDC trust relationships are automatically created during account provisioning. The management account has its own dedicated GitLab instance and runners, completely isolated from the rest of the CI/CD infrastructure. Notably, the team discovered that IAM can only validate a limited subset of JWT claims from OIDC tokens — primarily just the sub claim — which limits the granularity of trust relationship enforcement.

Demo / Proof of Concept

▶ Watch: Shared VPC networking with Tailscale and RAM subnet sharing (24:00)

There was no live demo. The talk was a comprehensive architectural walkthrough with detailed slides showing OU structures, Terraform configurations, networking diagrams, and IAM policy examples. Mohit joined remotely via Zoom for portions of the presentation.

Defensive Implications

▶ Watch: Backup strategy: AWS Backup with cross-region, cross-org copies (36:00)

This talk is a masterclass in pragmatic security architecture for resource-constrained organizations. Several decisions are directly applicable:

Defenders should consider per-client or per-workload account isolation as a primary blast-radius control — it is more effective and lower overhead than fine-grained IAM policies within shared accounts. The management account isolation pattern (no federation, IAM user break-glass only, SCP-denied root access) deserves serious consideration for any organization concerned about identity provider compromise.

The honest assessment of KMS key overhead versus actual threat mitigation value is worth internalizing. Organizations should evaluate whether their KMS investment is driven by genuine threat modeling or by compliance checkbox behavior.

For monitoring on a budget, the team's approach of delegating to a managed detection and response provider — vetted by their own offensive consultants who regularly test such providers — is a smart force multiplier for small security teams.

Key Takeaways

• Account isolation is the highest-value security control: Per-client and per-workload account separation provides hard blast-radius boundaries with minimal operational overhead.
• Control Tower is not always the right choice: For organizations with non-standard workload requirements, building custom organization management with borrowed SCPs can be more effective.
• Management account federation is an attack surface: Deliberately isolating the management account from your identity provider adds meaningful friction to organization-level compromise.
• Custom KMS keys may not be worth the overhead: For many threat models, the operational cost of per-resource KMS management exceeds the actual security value delivered.
• Pentesters building infrastructure brings unique perspective: Knowing exactly how attackers exploit gaps leads to pragmatic, threat-informed architecture rather than compliance-driven checkbox security.
• OIDC claim validation in IAM is more limited than expected: Only a small set of JWT claims can be validated in trust relationships, constraining CI/CD security options.

About the Speaker(s)

Nick Jones is a security consultant at Reverse X (formerly WithSecure Consulting) specializing in offensive security, penetration testing, and red team engagements. He previously presented at fwd:cloudsec 2020 while at F-Secure. He has a well-known aversion to IAM users. Mohit Gupta is also a security consultant at Reverse X who has maintained and developed consulting infrastructure across multiple organizational transitions. He joined the talk remotely from outside the US. Both speakers have extensive experience attacking AWS environments, which informed their defensive architecture decisions.