Console Hero to IAM Zero: Learn from Temporal's Just-In-Time Journey

Console Hero to IAM Zero: Learn from Temporal's Just-In-Time Journey

Speakers: Brandon Sherman, Temporal Technologies

Conference: fwd:cloudsec Europe 2025

YouTube: https://www.youtube.com/watch?v=B2BIvs1pUmQ

Overview

Brandon Sherman of Temporal Technologies delivered a practitioner-focused talk on eliminating static credentials across AWS and GCP by implementing just-in-time (JIT) access as a core security control. Drawing from his experience as a solo security engineer at a fast-growing startup, Sherman laid out a phased, human-centered strategy for moving an entire engineering organization from persistent administrative access to time-bounded, approval-gated credentials — without destroying developer productivity. The talk is equal parts technical playbook and organizational change-management guide, offering concrete lessons on earning user trust while systematically reducing the blast radius of compromised credentials.

Background

▶ Watch: The problem stated bluntly: over half of cloud breaches start with static cre... (1:30)

The motivation is stark and well-documented: over half of all cloud breaches begin with compromised static credentials, a finding that has been consistent across Datadog's State of Cloud Security reports for three consecutive years. Leaked API keys are exploited almost instantly — it is effectively a race between AWS's quarantine mechanism and attackers pillaging infrastructure. Sherman frames the solution bluntly: "Don't have static credentials."

But achieving that outcome in an organization where engineers legitimately need the "trifecta of privilege escalation" — IAM, STS, and CloudFormation-level permissions — is a multi-year journey. Temporal was a roughly 50-person startup when Sherman joined, with high security needs but also engineers who required deep infrastructure access as part of their daily workflow. Rather than chasing ever-smaller IAM policies (a task that yields diminishing returns when users genuinely need broad permissions), Sherman chose to constrain when access is granted rather than what access is granted.

Key Findings

▶ Watch: Introducing 'sudo for cloud' — the mental model for just-in-time access (6:00)

Sherman's phased rollout surfaced several important insights:

• Most engineer activity is read-only. When Sherman audited how often engineers actually needed administrative privileges, the answer was rarely. The majority of cloud console interactions were investigative — checking S3 bucket names, reviewing security group assignments. This observation justified a dual-tier model: always-on read-only access plus on-demand elevated privileges.

• Incremental friction is more effective than big-bang policy changes. Sherman introduced JIT requirements gradually — first optional, then requiring approvals, then requiring a reason, then shrinking the duration from 8 hours to 10 (a pragmatic compromise after user pushback), and finally requiring a ticket. Each step was small enough that users adapted without revolting.

• The "reason field" becomes a data goldmine. Collecting justification text from access requests enabled later automation. Repeated reasons flagged tasks ripe for scripting, and the data informed which permissions could be further scoped.

• New-hire-only enrollment creates compound growth. Rather than migrating every existing user at once, Sherman enrolled only new hires in JIT-only access. Over time, the proportion of JIT-only users grew organically, and grandfathered users faced increasing social and managerial pressure to transition.

• Machine identities are the harder problem. Most static credentials in the environment existed not for humans but for CI/CD pipelines, vendor integrations, and automated workflows. Eliminating those required OIDC federation, IAM Roles Anywhere, and vendor selection criteria that rejected static-key-only integrations.

Technical Deep Dive

▶ Watch: AWS Partner CRM Connector nightmare — a service that doesn't support IAM roles (16:00)

The JIT Model: Sudo for Cloud

Sherman's mental model is "sudo for cloud." Nobody runs as root on their laptop; similarly, nobody should have persistent admin in a cloud account. The system presents users with two tiers: a "carrot" of always-available read-only access and a "carrot cake" of short-lived admin access that requires approval, a justification, and a time limit.

The implementation leverages AWS IAM Identity Center (formerly AWS SSO). Permission sets are provisioned on demand and revoked when the access window closes. Sherman flagged several operational pitfalls:

• EKS and KMS role churn: IAM Identity Center provisions roles with random suffixes and removes them when no matching user/permission-set/account exists. When JIT access expires overnight, EKS clusters and KMS key policies lose track of the principal. Sherman's team solved EKS by introducing an intermediate role with a static name and an IAM condition that absorbs the randomness. For KMS, the cost-benefit analysis often favors simply letting orphaned keys age out at a dollar per month.

• On-call auto-approval: Trusted on-call engineers get auto-approved production access — but Sherman immediately had to guard against schedule-override abuse (a five-minute PagerDuty override to gain auto-approval without actually being on call). The longer-term plan is to gate access on active incidents rather than on-call status.

Eliminating Machine Credentials

An SCP (Service Control Policy) blocks creation of new IAM users organization-wide. Every alert from this policy triggers a proactive outreach: "Hi, I got an alert. How can I help?" This converts user friction into a relationship-building opportunity.

For CI/CD, Sherman mandates OIDC federation from GitHub Actions into AWS. Actions credentials are ephemeral — lasting only the duration of the workflow — and AWS Resource Control Policies restrict which GitHub organizations and repositories can assume roles. This control even caught well-intentioned engineers who had set up OIDC from an unauthorized GitHub org.

In GCP, the equivalent is Workload Identity Federation, though Sherman notes vendor adoption is much weaker. GCP service account keys default to a thousand-year expiry and lack an external-ID equivalent to prevent confused deputy attacks, making per-customer service accounts the only safe pattern today.

GitOps as an Audit Trail

All infrastructure changes flow through GitHub pull requests with branch protection, required approvals, and deploy-before-merge gates. This creates an immutable audit trail: every change has an author, an approver, a Terraform plan, and an apply log. Sherman highlighted that this satisfies compliance regimes and eliminates the risk of forgotten or indefinitely deferred deployments.

IAM Identity Center Quirks

Sherman surfaced several IAM Identity Center operational gotchas. Because delegated admin involves cross-account role mechanics in the background, permission sets behave differently depending on whether they were created in the org management account or the delegated admin account. If the delegated admin tries to deploy a permission set to the org root account, that permission set becomes unmodifiable — the console displays a cryptic message about the resource originating from another source. Additionally, GCP's equivalent AM system (Zanzibar) is eventually consistent, and "eventually" can mean five minutes or more of waiting after modifying a role assignment before the change takes effect — a painful delay when working with users in real time.

Demo / Proof of Concept

▶ Watch: EKS and KMS breakage caused by IAM Identity Center role churn (22:00)

Sherman did not perform a live demo. The talk was structured as a narrative case study, walking through Temporal's multi-year journey with concrete screenshots and policy examples, including the SCP that blocks IAM user creation and the Resource Control Policy restricting GitHub OIDC federation.

Defensive Implications

▶ Watch: The full JIT maturity pyramid: from identity consolidation to self-healing in... (36:00)

• Block IAM user creation organization-wide via SCP and alert on every denied attempt. Treat each alert as a user-engagement opportunity, not a punishment.
• Require OIDC/role-based integration for all vendors. Make it a procurement criterion. If a vendor doesn't support IAM role assumption, add the risk to a register and prioritize migration.
• Maintain a break-glass IAM user in the delegated admin account — MFA-only, stored in a shared secret vault, tested regularly but used less than once a year.
• Invest in user experience. If the JIT system is slow or confusing, users will circumvent it. Test the system yourself with the strictest settings ("eat your own dog food") so you lead from the front.
• Audit the reason field to identify automation candidates. When the same manual task appears repeatedly, it's time to replace a human with a GitHub Action.
• Scope down permissions after JIT is established. The pyramid of maturity is: consolidate identities → grant JIT access → reduce permissions → automate common tasks → self-healing infrastructure.

Key Takeaways

• Static credentials are the #1 cloud breach vector. Eliminating them — not just rotating them — should be the strategic goal.
• JIT access is "sudo for cloud": users get broad access only when needed, with approvals, justification, and short durations.
• Incremental rollout beats big-bang migration. Enroll new hires first, help existing users migrate, and let compound growth do the work.
• Machine credentials are the bigger challenge. OIDC federation, Resource Control Policies, and vendor selection standards are essential.
• User experience is a security control. Friction drives shadow IT; hospitality drives adoption.
• The journey never ends, but other priorities exist. Know when JIT maturity is "good enough for now" and shift focus to the next highest-risk area.

About the Speaker(s)

Brandon Sherman is a security engineer at Temporal Technologies, where he has been the driving force behind the company's cloud security posture since its early startup days. A veteran of the security conference circuit — he traces his community roots to DerbyCon in 2019 — Sherman combines deep AWS and GCP operational expertise with a pragmatic, user-first approach to security engineering.