Scalably Securing Third-party Dependencies in Large Codebases

Scalably Securing Third-Party Dependencies in Large Codebases

Speakers: Ziyad Edher, Chris Norman

Conference: BSidesSF 2025 — April 26-27, 2025, San Francisco

YouTube: Watch on YouTube

Reading time: ~8 minutes

TL;DR

Supply chain attacks are now the most effective way to compromise highly secured environments, because everything else has gotten harder. Ziyad Edher and Chris Norman from Anthropic's security team explain why standard supply chain approaches like SLSA fall short for heterogeneous, research-driven infrastructure — and describe Dependent, the admission control system they built to govern third-party packages across Anthropic's AI training and inference clusters, without bringing the research environment to a halt.

Introduction

Supply chain security has a fundamental asymmetry problem. If an attacker discovers an RCE in a popular web framework, they can reuse that exploit for months or years against a large, undifferentiated target surface. A supply chain attack requires targeted reconnaissance, social engineering specific maintainers, or bribing specific open-source contributors — it does not scale. But when it works, it is nearly unstoppable: the defense mechanisms themselves become the attack vector.

Ziyad Edher and Chris Norman, both security engineers at Anthropic, opened their BSidesSF 2025 talk with a simple analogy: the "Moneybags" attacker doesn't try to breach your castle directly once you've built walls. He buys the wall factory. Recent real-world incidents — LibXZ, PyTorch Nightly's dependency confusion attack, the Eventstream npm hijacking, CodeCV — demonstrate that the analogy isn't fanciful. It's a documented playbook.

Why Supply Chain Is Becoming the Dominant Attack Vector

▶ Watch: Supply chain attack mechanics and recent incidents (05:00)

Edher's framing of supply chain risk centers on a comparative leverage analysis. Against software vulnerabilities: high scalability, but exploitation requires development of a payload and depends on unpatched targets. Against social engineering: high scalability, moderate detection risk. Against supply chain: very high leverage against specific targets, but low scalability.

The low-scalability caveat is important — it means supply chain attacks are most attractive as a tool for targeting specific, highly valuable, and otherwise well-defended organizations. "Hot take: supply chain attacks are the most effective way to infiltrate specific, highly secured environments nowadays," Edher said. "I'm not saying supply chain is getting particularly worse. I think it's actually getting kind of better. But everything else is getting better so much more quickly."

The attack patterns he described include three categories. Force means compromising a build system and taking over the packages it produces — the PyTorch Nightly incident, where a dependency confusion attack against a core component gave attackers control over packages distributed to anyone pulling PyTorch nightly builds. Deception means tricking maintainers or reviewers into accepting malicious code — the LibXZ backdoor, where a contributor with a long track record across multiple repositories submitted a PR containing a subtle backdoor to SSH, which was reviewed, approved, and merged before a Postgres developer noticed that SSH was taking a few hundred extra milliseconds on authentication checks. Acquisition means simply buying an existing repository with an established user base — Eventstream's maintainer transferred the package to a new owner who promptly shipped malware to its 4 million weekly downloads.

The economics of the acquisition vector deserve attention. A 50-star GitHub repository that sits in a long dependency chain of a specific valuable target might be worth very little to its maintainer and worth a great deal to a well-resourced attacker. "It's turning out that it's a lot cheaper to buy up that random repo than it is to develop a novel RCE nowadays."

Why SLSA Isn't Enough for Anthropic

▶ Watch: SLSA limitations and Anthropic's threat model (18:00)

Supply Chain Levels for Software Artifacts (SLSA) is the industry's primary framework for reasoning about dependency provenance — where a package came from, what went into it, how it was built, who signed it. Edher described it as analogous to factory seals on dependencies: Moneybags shows up with your walls, but Salsa flags that they didn't come from the factory you expected and weren't signed by the expected key.

For a conventional SaaS company with well-defined development and production environments and standard branch protection workflows, SLSA plus a tool like GitHub Dependabot is a reasonable implementation. Chris Norman described that roadmap as the standard model: understand your dependencies, implement pull-request-based controls, use Dependabot for automated updates.

Anthropic's infrastructure doesn't fit that model. Anthropic operates what Norman described as more analogous to a shared university supercomputer than a traditional SaaS company: a large number of researchers working across multiple cloud providers and multiple accelerator chip architectures, constantly pulling in new dependencies, often custom patches or specific versions needed for individual experiments. The environment is intentionally heterogeneous and intentionally fast-moving. Slowing researchers down or forcing them to move to a homogeneous environment would break the research operation.

SLSA addresses provenance but not content. It can tell you that OpenSSL version X was built by the OpenSSL Foundation with dependencies A and B. It cannot tell you whether version X has known vulnerabilities, or whether the OpenSSL Foundation has been compromised. For an organization with Anthropic's threat model — "we have very powerful technology and lots of folks are after that technology" — provenance verification alone leaves too much surface area open.

Dependent: Admission Control for a Research-Driven Organization

▶ Watch: Dependent architecture and design (26:00)

The system Anthropic built to address this gap is called Dependent. It operates as an admission control layer between the internet and Anthropic's internal artifact registries. Researchers run a CLI command to request dependencies; the system runs admission control checks; approved packages are pulled into the internal registry; workloads then pull from that registry rather than directly from the internet.

The design principles reflect careful attention to where friction should and shouldn't be introduced:

Meet users where they are. Anthropic researchers spend their days in terminal environments. Dependent is a terminal-native tool. Requesting a new package doesn't require filing a ticket or reading documentation — it happens in the same environment where researchers already work. Once a package clears admission control, it installs using standard package managers (pip, npm) without additional tooling.

Policies as code. Admission control checks are written in Python, which means they are subject to the same software development lifecycle controls (testing, code review, mandatory review for policy changes) as any other production code. This avoids the configuration drift and manual inconsistency problems that plague policy systems implemented as static rules or forms.

Transitive dependency resolution. The system handles not just the directly requested package but the full graph of its dependencies. This is technically complex — replicating the resolution logic that package managers themselves implement — but necessary, since a clean top-level package with a compromised transitive dependency provides no meaningful protection.

An escape hatch. In a research environment, there are always situations where a researcher needs a dependency immediately to evaluate whether it will actually work for their experiment. Dependent provides a documented break-glass mechanism for exceptional circumstances, with the expectation that justification is provided inline.

The admission control checks themselves include: package maturity checks (time delay before new packages can be ingested), integration with open-source threat intelligence APIs like socket.dev, CVE/vulnerability scanning (critical CVEs block ingestion), and license compliance verification.

Rollout Challenges and the Cultural Shift

▶ Watch: Rollout challenges and adoption lessons (38:00)

Norman was candid about the implementation difficulties. The initial design required human review for all new packages, and that review was centralized — meaning the security team was the bottleneck for the entire package request pipeline. At the volume Anthropic researchers were requesting new packages, this didn't scale. The model shifted to distributed review with clearer criteria and automated enforcement for straightforward cases.

The tension with researchers was a recurring theme. Researchers often need to actually try a dependency before they can make an informed request for it — they want to evaluate it in an experiment, but the current workflow requires them to request it before they can use it. This creates a chicken-and-egg friction that the team is still working to resolve.

A subtler benefit emerged from the thoughtful-friction design: by adding a small amount of friction to bringing in new dependencies, Dependent began to shift the culture around dependency usage. Researchers who previously might have pulled in a new package reflexively started asking whether they actually needed it. "It contributes to a cultural shift within the company around trying to minimize third-party dependency usage," Norman observed.

The longer-term roadmap includes network-level controls to enforce that all package requests flow through Dependent (preventing any package manager from talking directly to PyPI or npm), betting on Nix for reproducible builds, and sandboxing and isolation of third-party dependencies so that even an approved package with a compromised component cannot affect the broader environment.

Notable Quotes

"Supply chain attacks are the most effective way to infiltrate specific, highly secured environments nowadays. Everything else is getting better so much more quickly — supply chain kind of stays the way that it is." — Ziyad Edher (15:00)

"It's turning out that it's a lot cheaper to buy up that random repo than it is to develop a novel RCE nowadays. That might actually be Moneybags going and trying to buy up real estate wherever he thinks might be useful in the future." — Ziyad Edher (17:00)

"The thoughtful friction contributes to a cultural shift around trying to minimize third-party dependency usage. We're trying to encourage people to ask: have you considered something that we already have in our ecosystem?" — Chris Norman (40:00)

Key Takeaways

• Supply chain attacks are a targeting problem, not a scalability problem. They are most attractive against specific, high-value, otherwise well-defended targets — which makes AI infrastructure and research environments a primary concern.
• SLSA solves provenance, not content. Knowing where a package came from doesn't tell you whether it is safe; high-threat-model organizations need additional admission control layers beyond provenance attestation.
• Heterogeneous environments require custom tooling. Standard dependency management workflows assume well-defined, stable environments. Research-driven infrastructure with multiple clouds, multiple accelerators, and constantly shifting experiment dependencies breaks those assumptions.
• Admission control friction, done thoughtfully, changes culture. Adding a small checkpoint before dependencies are approved shifts researchers' default behavior from "pull it in and see" to "do I actually need this?"
• Network-level enforcement is the endgame. Admission control is most effective when researchers cannot bypass it by talking directly to upstream registries — network controls that route all package traffic through the controlled registry are the enforcing mechanism that makes the whole system work.