The Hidden Access Paths to Smaug's Cavern

The Hidden Access Paths to Smaug's Cavern

Speaker: Ben Arent

Conference: BSidesSF 2025 — April 26-27, 2025, San Francisco

YouTube: Watch the full talk

Reading time: 7 minutes

TL;DR

Developers and SREs accumulate access the way Tolkien's Smaug accumulates treasure — and the hidden pathways they create to get their jobs done become the backdoors that attackers exploit. Ben Arent, drawing on over a decade of experience building SaaS products in San Francisco, examined three classes of access risk — admin panel impersonation, production secrets in local environments, and unbounded background job permissions — and proposed a framework of security invariants, behavioral analytics, and "desire path" design to close them.

Introduction

Ben Arent opened his BSidesSF 2025 talk with a reference to Tolkien: Smaug the dragon from The Hobbit, hoarding gold in a mountain cavern. The parallel to software teams is intentional — developers and SREs, once granted access to a system, rarely give it up. And some things are more precious than ordinary access: the Arkenstone equivalent in infrastructure is the private key, the KMS key, the admin credential that unlocks everything else.

But Arent's talk is less about external attackers than about the pathways that legitimate users create — and that those same pathways become the routes through which data leaks and money disappears. Arent, who works at Teleport (an infrastructure access platform) and is based in Oakland, has spent the better part of 13 years building and supporting SaaS companies in the Bay Area. His perspective is explicitly that of a developer who has historically circumvented whatever security controls were in place to get work done. "I come from a product/slightly hacky developer background," he noted, "and I've normally circumvented whatever controls most security people had in place."

The framework he applied throughout the talk is architectural: desire paths. In urban planning, a desire path is the route people actually take between two points, regardless of where the architect placed the walkways. In software, it's the ad-hoc access method a developer uses because the official process is too slow. Securing systems requires understanding those paths — and either blocking them or making the secure route the path of least resistance.

Act One: The Admin Panel and Production Secrets

▶ Watch: SaaS admin panels and the "login as user" problem (05:00)

Arent's early-career experience at companies handling exception tracking tools (Exceptional and its competitor Airbrake) illustrated a persistent problem: misconfigured exception trackers ingest application secrets. Mail API keys, Stripe keys, and AWS credentials regularly appeared in crash reports submitted by customers who had inadvertently included them in their exception data. This isn't a theoretical risk — it happened continuously.

The admin panel is a second well-known vulnerability surface. Every SaaS application builds one: an internal dashboard showing accounts, subscriptions, billing, activity. And almost universally, it includes some form of "login as user" — an impersonation capability used for debugging and support. Arent ran an informal poll among SaaS peers and found that 60% of companies still allow engineers to log into customer accounts, often with limited logging. The most secure approach, he argued, is the 22% option: don't provide direct access to customer accounts at all, so that neither a malicious insider nor an attacker who compromises internal systems can reach them.

Production secrets in local development environments are the mirror-image problem. Developers pull production credentials to their laptops to debug CI/CD failures, send emails from a production mail key, or perform break-glass fixes. Each instance creates a path from a developer's machine — a relatively uncontrolled endpoint — directly into production infrastructure. "We know that these credentials and secrets, when they're local, are ripe for hackers to get access to and become a backdoor into your system," Arent said.

Act Two: The Salami Attack Hidden in Plain Sight

▶ Watch: The background job attack case study (12:30)

The most striking case study in the talk came from a CISO conversation Arent had earlier in 2025. An employee at a tech company had broader system access than strictly necessary — common in smaller organizations where roles aren't tightly scoped. In the course of that access, the employee discovered something: they could edit background jobs, and specifically, they could modify the recipient field in scheduled reward payment jobs.

The result was a salami attack — small adjustments accumulated over time — that extracted approximately $250,000 from the organization. The security team had deployed nearly every available tooling category, and none of it caught the exfiltration. What finally surfaced the anomaly was finance: when reconciling their books, someone noticed that the organization was paying out against a 2023 campaign instead of 2024. An accounting discrepancy, not a security alert, initiated the investigation.

"This highlights a pathway that you can get access to something which can also potentially go wrong," Arent said. The reference to Office Space was deliberate — the film is now 25 years old, and the same class of attack is still occurring in production environments.

The broader lesson is one of access granularity. This employee needed access to background jobs to do their work, but they needed far less access than they had. Write permissions to recipient fields in financial jobs are not a standard operational requirement. The access existed because nobody had thought to limit it.

Behavioral Analytics: Who's Accessing What, When, and Why

▶ Watch: Monitoring access patterns over time (18:00)

Arent identified several dimensions of behavioral monitoring that security teams underutilize:

Time of day and day of week. Malicious insiders frequently operate after hours and on weekends. An employee editing background jobs at 2 a.m. on a Sunday is a different signal than the same action during business hours, even if both are technically authorized.

Query scope. Database queries that span entire tables or return volumes of PII are a detectable anomaly. Monitoring for scope — not just presence of access — catches a class of exfiltration that authorization checks miss entirely.

Frequency. If a CTO has root access to everything but never uses it, they're both a prime target (their credentials appear on the About page) and an example of over-provisioned access that should be removed. Access that isn't used is access that should be revoked.

Graph visualization. Infrastructure access chains are increasingly opaque. Arent described a pattern he encounters frequently: an engineer joins, is added to an Okta group, the group grants an IAM role, the IAM role has an instance profile, the instance profile touches a production database — and nobody has traced the full chain from Okta group membership to database access. Visualizing the graph of access and how it changes over time is foundational to understanding what's actually exposed.

One-off events. Break-glass procedures and tabletop exercises trigger unusual access patterns. Using these events deliberately to understand what a SIEM does and does not capture gives security teams a realistic picture of their detection capability.

Act Three: Remediation — Making the Secure Path the Desire Path

▶ Watch: Security invariants and remediation framework (23:15)

Arent's remediation framework started with identifying the Arkenstone — the most sensitive thing in your organization — and working backwards from there to determine what access is truly necessary. In the case of the background job attack, the Arkenstone was payment job configuration. Once identified, you build protection specific to it.

His non-negotiable controls:

• Hardware MFA universally. Requiring YubiKeys for all infrastructure team members effectively stops the class of phishing attacks that bypass software-based 2FA.
• Zero direct access to user accounts. Build tooling that allows support teams to perform necessary actions without literal account impersonation. The 2024 Okta breach, in which a third-party support provider had impersonation capability, illustrates the downstream risk when this isn't enforced.
• AWS Service Control Policies. SCPs add an enforcement layer on top of IAM policies, enabling network segregation and hard constraints that IAM alone cannot enforce. They also allow automated alerting when invariants are violated — for example, blocking direct IAM user creation and alerting when someone attempts it.
• Automate invariant enforcement. Instead of relying on humans to remember rules, encode them. If a rule says users should never be created directly in AWS, build a control that prevents it and alerts when it's attempted.

The closing principle distilled everything: "Make the secure path the desire path." No matter how well-designed the controls are, people will find workarounds if those controls add meaningful friction. Design systems where the path of least resistance is also the secure one, and the desire paths will route through your controls rather than around them.

Notable Quotes

"Once they get access to something, they don't give it up. But there's some things that are a little bit more precious than just the gold and the jewels."

— Ben Arent, on developer access hoarding and the concept of the Arkenstone

"The thing that was interesting about this is this organization had nearly every security tool you could buy, but it was finance that finally spotted it."

— Ben Arent, on the salami attack that evaded all tooling

"Make the secure path the desire path. No matter what you put in place, people will — if your security tooling is tricky — most engineers will normally find a way to circumvent it."

— Ben Arent

Key Takeaways

• Admin panel impersonation is a high-value target. "Login as user" features are near-universal in SaaS but rarely have the logging, access controls, or audit trails they warrant. Consider whether direct account access is necessary at all.
• Local production secrets create invisible attack surfaces. Every developer laptop holding a production credential is an endpoint vulnerability. Enforce credential isolation and short-lived access tokens.
• Minimal access prevents salami attacks. The $250,000 background job exfiltration relied on write access that was never operationally necessary. Scope permissions to actual job function, not organizational role.
• Behavioral baselines catch what authorization can't. Access patterns — timing, frequency, query scope — reveal intent in ways that binary permission checks cannot. Invest in visualizing and monitoring the graph of access over time.
• Security invariants eliminate entire attack classes. Hardware MFA, zero user-account impersonation, and SCP policies each prevent specific well-understood attack patterns. Deploying them doesn't require a threat model; it requires a decision.