Keeping Up with the Dependencies

Keeping Up with the Dependencies

Speakers: Dhivya Chandramouleeswaran, Security Engineer, Lyft

Conference: LocoMocoSec

YouTube: https://www.youtube.com/watch?v=qt17PMTL_Mg

Overview

In this insightful LocoMocoSec talk, Dhivya Chandramouleeswaran, a Security Engineer at Lyft, delves into the multifaceted challenges and strategic approaches required for effective dependency management in modern software development. The presentation tackles the pervasive reliance on open-source software (OSS) and its inherent security implications, emphasizing that while OSS accelerates development, it introduces a significant external attack surface that often goes unaddressed until a critical vulnerability emerges. Chandramouleeswaran leverages her dual experience as both a software developer and a security engineer to bridge the gap between developer productivity and security imperatives, offering a holistic perspective on building sustainable dependency management programs.

The core of the talk revolves around understanding the unique characteristics of dependency lifecycles across different programming languages—specifically Python, JavaScript, and Go—and integrating developer experience (DX) into the security process. Chandramouleeswaran argues that successful dependency management is not merely a technical problem but also a cultural one, requiring careful consideration of developer workflows, communication strategies, and shared ownership. She outlines both reactive and proactive measures, discusses common anti-patterns, and provides actionable advice for sustaining a robust program, including crucial metrics for measuring success and failure.

This topic is profoundly relevant in today's software landscape, where breaches stemming from vulnerable third-party components, such as the infamous Apache Struts vulnerability, underscore the critical need for proactive security hygiene. Chandramouleeswaran's talk serves as a practical guide for organizations grappling with the complexities of managing hundreds or even thousands of transitive dependencies, offering a roadmap to mitigate risks, streamline updates, and foster a stronger security posture across their development ecosystem.

Background

▶ Watch: Overview of talk: dependencies, languages, developer experience (0:00)

The proliferation of open-source software (OSS) has dramatically transformed the development landscape, empowering developers with readily available libraries and frameworks that expedite project delivery. As Chandramouleeswaran points out, a simple Google search can often yield a library that solves a specific problem, allowing developers to quickly integrate functionalities like web servers in Python with minimal effort. This ease of integration, however, often overshadows critical security considerations. Developers, in their pursuit of functionality, frequently prioritize whether a library "solves their problem" (an estimated 80% of the initial evaluation) over deeper security vetting. Questions such as "when was the last commit?", "how old is this library?", "does the repository have active issues?", or "who are the contributors?" are often secondary, leading to the adoption of potentially insecure or unmaintained components.

From a security perspective, dependencies are any additional code called by a program, effectively standing outside the application's trust boundary. This external code introduces a new dimension to the threat model, necessitating careful scrutiny. Dependencies manifest in two primary forms: direct dependencies, explicitly listed and used by the application, and transitive dependencies, which are pulled in by the direct dependencies themselves. The talk highlights that dealing with uncomplicated transitive dependencies is manageable, but dependency hell arises from convoluted or circular dependency graphs, where different direct dependencies require conflicting versions of the same transitive dependency (e.g., A depends on C v1.0, B depends on C v2.0).

Beyond application security, dependency management carries significant compliance and legal implications. It is a fundamental aspect of vulnerability management (VM) and external dependencies management (EDM), often mandated by frameworks like the NIST Cybersecurity Framework. Organizations must satisfy these baseline requirements, but true security hygiene demands a more integrated approach combining AppSec, compliance, and legal considerations.

Chandramouleeswaran specifically addresses the challenges posed by popular languages like Python, JavaScript, and Go, all of which exhibit high volumes of transitive dependencies.

• Python uses managers like pip, pipenv, and pip-tools. While it defaults to project-level loading and breadth-first loading (direct first, then transitive), a significant security concern is its support for loose top-level pinning (e.g., requests > 4.4). This flexibility, while convenient for developers, introduces uncertainty at runtime as the exact version pulled can vary, making security tracking difficult.
• JavaScript typically has the highest volume of transitive dependencies, leading to the common joke about "downloading the entire system" for a single library. Its loading behavior is not standardized, allowing for both depth-first and breadth-first.
• Go modules support lazy module loading, offering some control over pulling in transitive dependencies. Go also uniquely provides proxy setups for libraries, allowing internal mirroring of external dependencies, which can enhance control and developer experience by providing a consistent internal source.

The speaker's experience across different companies and roles (software developer, security researcher, security engineer) underscores the widespread and persistent nature of dependency management issues, irrespective of the organization or specific technology stack.

Key Findings

▶ Watch: Unique dependency challenges in Python, JavaScript, and Go (8:00)

Dhivya Chandramouleeswaran's talk distills several critical findings regarding the implementation and sustainability of dependency management programs:

1. Dependency Management is Language-Specific but Universally Challenging: While each language (Python, JavaScript, Go) has unique dependency managers and loading behaviors, they all share the fundamental challenge of managing a high volume of transitive dependencies and the potential for "dependency hell." Go, for instance, has fewer transitive dependencies than Python, which in turn has fewer than JavaScript, but none are immune to complexity.
2. Developer Experience (DX) is Central to Adoption: A dependency management program designed without considering developer workflows and preferences is destined to fail. Engaging developers through requirement surveys, understanding their pain points (e.g., alert fatigue, broken CI/CD), and designing solutions that minimize friction are essential for successful rollout and sustained usage.
3. 100% Automation is an Aspiration, Not a Starting Point: The speaker cautions against expecting full automation from the outset. Initial rollout often involves significant manual effort, especially when dealing with legacy codebases or complex version jumps. Automation is a gradual process that requires sorting out "kinks" and establishing strong foundational hygiene.
4. Expect Counterintuitive Initial Trends: Post-program rollout, instead of an immediate decrease in reported vulnerabilities, organizations might observe an increase in volume. This is often due to improved discovery mechanisms unearthing previously unknown issues, particularly in obsolete services. This initial "speed bump" can be discouraging but signifies progress in visibility.
5. Security Cannot Fully Offload Responsibility: While developers are crucial, the security team retains ownership of the supply chain aspect. This includes validating vulnerability accuracy, assessing compensatory controls, and ensuring the overall integrity of the dependency ecosystem. A 50/50 split in ownership is suggested.
6. Uncertainty Requires Proactive Planning: Critical vulnerabilities like the Apache Struts incident can emerge unexpectedly. Organizations must have predefined processes and written agreements between security and development teams to handle such emergencies swiftly, including clear communication channels and resolution timelines.
7. Accurate Asset Management and Reporting are Paramount: Knowing exactly what dependencies are in use (including local forks and internal libraries) and receiving highly accurate vulnerability reports are foundational. Frivolous or inaccurate alerts contribute to alert fatigue, undermining trust and slowing response times.
8. Severity Negotiation is Key: A vulnerability reported as "critical" might not be exploitable in a specific application's context if the vulnerable code path isn't used. Security teams must be prepared to negotiate severities with developers, considering compensatory controls and actual exploitability, especially when developer resources are constrained.
9. Alert Granularity Prevents Fatigue: Alerts must be highly granular, indicating not just the presence of a vulnerability but also its exploitability, the availability of a patch, and potential compensatory controls. Generic, high-volume alerts lead developers to question their relevance, reducing response efficiency.
10. Language Selection Influences Security Posture: The inherent design and community characteristics of a programming language (e.g., Go's built-in security controls, JavaScript's vast ecosystem) directly impact the complexity and risk associated with its dependencies. Proactive teams should consider these factors when adopting new languages.

Technical Deep Dive

▶ Watch: Reactive vs. proactive dependency management strategies (18:00)

Effective dependency management hinges on a deep understanding of the dependency life cycle and the technical characteristics of the languages involved. Chandramouleeswaran emphasizes treating each version of a program as an immutable source. This means that once a program is running with a specific set of dependencies, those dependencies should not be updated at runtime. Instead, any updates should result in a new, versioned immutable state. This principle is critical for security, as it allows for precise tracking and auditing of what is running. The speaker highlights the problem of loose top-level dependencies (e.g., package > 1.0), which, while offering flexibility, introduce ambiguity about the exact version used at runtime. To counter this, dependencies must be locked as close to runtime as possible, ensuring that the exact versions are known for accurate security assessment.

The choice of programming language significantly influences the complexity of dependency management and vulnerability exposure:

• Go is positioned as generally less risky due to its design as a secure language with built-in semantic security controls (e.g., memory overflow handling) and typically fewer transitive dependencies. It also explicitly supports lazy module loading and proxy setups for libraries, allowing organizations to internally mirror external dependencies for enhanced control and a consistent internal consumption experience.
• Python sits in the middle, with a moderate volume of transitive dependencies. Its default breadth-first loading and loose top-level pinning create challenges for precise version tracking.
• JavaScript is often the riskiest due to its notoriously high volume of transitive dependencies, which can quickly inflate a project's attack surface. The joke about "downloading the entire system" for a single JavaScript library underscores this reality.

Despite these differences, dependency hell – where conflicting versions of transitive dependencies create irreconcilable conflicts – is a possibility across all three languages. None of them offer built-in safety controls against this complex scenario.

To maintain good dependency management hygiene, several technical practices are crucial:

• Signing packages: This helps prevent supply chain attacks by verifying the integrity and origin of downloaded packages.
• Locking dependencies: Pinning to exact versions (e.g., package == 1.2.3) is essential for accurate security tracking and ensuring consistent builds.
• Accurate alert reporting: Systems must provide precise and actionable vulnerability information.
• Proxy loading: Leveraging internal proxies (like Go modules or open-source solutions for Python/JavaScript) allows organizations to vet and control the flow of external libraries into their ecosystem.

The talk also identifies several anti-patterns that undermine security:

• Forking and fixing locally: Developers might fork a dependency to apply a quick fix if the upstream maintainer is slow. While a temporary solution, this creates code conflicts during future upgrades, making it hard to track and integrate official patches.
• Abandoned dependencies in some branch: Old projects or experimental branches often accumulate unused dependencies, which remain in the codebase, increasing the potential attack surface without providing any value.
• Copy-paste corrected code: Instead of using an entire library, developers might copy-paste specific functions or corrected code snippets directly into their codebase. This makes tracking security issues impossible for automated tools and is not a scalable solution.

Chandramouleeswaran outlines two main strategic approaches: reactive and proactive.

The reactive approach focuses on detecting and remediating vulnerabilities after they are introduced:

1. Discovery: Requires a robust asset management program that scans and accurately reports vulnerabilities across both external and internal/local libraries. The speaker notes that vulnerability databases for newer languages like Go can be sparse, requiring extra due diligence.
2. Matching: Involves correlating identified vulnerabilities with the organization's specific development ecosystem, accounting for customized tools (e.g., a custom pip-tools version or Artifactory with npm). This matching must be exact, not approximate.
3. Mitigation: Involves applying available patches or, in their absence, having a contingency plan, such as finding alternatives or implementing temporary compensatory controls.

The proactive approach aims to prevent vulnerabilities by influencing developer choices and environment:

1. Understanding the Developer Ecosystem: Identifying commonly used languages and libraries (e.g., Python's requests library) helps security teams anticipate needs.
2. Monitoring Industry Trends: Staying abreast of new languages and frameworks allows security to pre-review and approve libraries, creating a list of "approved libraries" before mass adoption.
3. Centralized Repository/Queue: Maintaining a controlled repository of approved and banned libraries, with a review mechanism for new additions, places a burden on the security team but significantly reduces risk.

Implementation of any dependency management program typically involves three phases: Discovery, Triaging, and Mitigation. Automated solutions like WhiteSource, Snyk, Dependabot, and OWASP Dependency Check can assist, but customized tools are also an option. The crucial elements for success are asset management and accurate reporting, enabling matching vulnerabilities to specific code details and agreeing on severities with development teams based on actual exploitability.

Demo / Proof of Concept

▶ Watch: Addressing common challenges like alert fatigue and old code (30:00)

The talk by Dhivya Chandramouleeswaran focuses on strategic planning, process design, and the challenges of implementing dependency management programs within an organizational context. It does not include a live demonstration or a specific proof of concept of a tool or an exploit. Instead, the speaker shares practical experiences and insights gained from her work in the field.

Defensive Implications

▶ Watch: Q&A: How to handle slow updates using local proxying (37:00)

Organizations seeking to fortify their security posture against dependency-related risks should heed several key defensive implications from Chandramouleeswaran’s talk:

1. Embrace Immutable Infrastructure and Dependency Locking: Treat application versions and their associated dependencies as immutable. This means locking dependencies to exact versions (e.g., package==1.2.3) in configuration files and avoiding loose top-level pinning. Updates should trigger a new build with new locked versions, rather than dynamic updates at runtime, ensuring a predictable and auditable dependency graph.
2. Prioritize Comprehensive Asset Management: A foundational defense is knowing what you have. Implement robust asset management systems that scan and track all dependencies—direct, transitive, external, and internal/local forks. This provides the necessary visibility for accurate vulnerability assessment.
3. Implement Granular and Actionable Vulnerability Scanning: Utilize industry-standard tools like WhiteSource, Snyk, Dependabot, or OWASP Dependency Check for continuous scanning. Crucially, configure these tools to provide granular alerts that indicate not only the presence of a vulnerability but also its exploitability in context, the availability of a patch, and potential compensatory controls. This reduces alert fatigue and ensures developers focus on high-priority issues.
4. Tailor Security Processes to Language Maturity: Be aware that vulnerability databases for newer languages (like Go) may be less mature than those for established languages (Python, JavaScript). Adjust expectations and potentially increase manual due diligence for findings in less mature ecosystems.
5. Actively Combat Anti-Patterns: Educate developers about and discourage practices such as forking and fixing locally, leaving abandoned dependencies in branches, and copy-pasting corrected code. These anti-patterns create untrackable code, hinder updates, and complicate security assessments. Implement code review processes and automated checks to identify and prevent these issues.
6. Shift Towards a Proactive Security Stance: Instead of solely reacting to vulnerabilities, adopt a proactive approach. Understand the developer ecosystem, identify popular libraries, and pre-approve secure versions or alternatives. Consider maintaining a centralized repository of vetted libraries to guide developer choices and enforce security standards from the outset.
7. Foster Strong Developer-Security Collaboration: Dependency management is a shared responsibility. Engage developers early in process design through surveys to understand their workflows, pain points, and preferred communication channels for alerts (e.g., Slack vs. email). Establish clear escalation paths and Service Level Agreements (SLAs) for vulnerability remediation, distinguishing between critical (e.g., 2-day ETA) and lower-severity issues.
8. Plan for Critical Incidents and Uncertainties: Develop a formal incident response plan specifically for critical dependency vulnerabilities (e.g., a Struts-like event). This plan should outline roles, responsibilities, communication protocols, and expected resolution times, with clear written agreements between security and development teams.
9. Balance Automation with Human Oversight: While automation is vital for scale, acknowledge that 100% automation is rarely achievable initially. Start with a balanced approach, automating discovery and initial triaging, but retain manual review for complex cases, false positives, and contextual exploitability assessments. Aim to increase automation gradually as process hygiene improves.
10. Measure Success Beyond Raw Vulnerability Counts: Track metrics that reflect security maturity and developer adoption, such as the percentage of teams consistently updating libraries, the average time to remediation for different severity levels, and the ease of creating secure, immutable images. This provides a more holistic view of program effectiveness and highlights areas for improvement.

Key Takeaways

• Open-source dependencies are a critical security vector: While enabling rapid development, the widespread use of OSS introduces significant external code that often falls outside traditional security scrutiny, making robust dependency management non-negotiable.
• Dependency management is a holistic challenge: Effective programs must integrate application security with compliance and legal requirements, ensuring a comprehensive approach that addresses both technical risks and regulatory obligations.
• Language characteristics profoundly impact complexity: The inherent design, dependency volumes (e.g., JavaScript's high transitive count vs. Go's lower count), and available tooling for each programming language dictate unique challenges and require tailored management strategies.
• Developer experience is paramount for adoption: Designing dependency management processes with developer workflows in mind, addressing friction points, and fostering collaborative ownership are crucial for successful implementation and long-term sustainability.
• Expect initial setbacks and non-linear progress: Rolling out a dependency management program often leads to an initial increase in reported vulnerabilities and demands significant manual effort, requiring realistic expectations and a commitment to continuous improvement.
• Balance automation with strategic human intervention: While automation is essential for scaling discovery and basic triaging, human expertise is indispensable for contextualizing alerts, negotiating severities, and addressing complex "dependency hell" scenarios. Granular, actionable alerts are key to preventing alert fatigue.

About the Speaker(s)

Dhivya Chandramouleeswaran is a Security Engineer at Lyft, where she focuses on enhancing the security posture of the company's applications and infrastructure. Her professional journey includes previous roles as a Security Researcher at Adobe and a Software Developer at SAP. This diverse background provides her with a unique and invaluable perspective, allowing her to understand security challenges from both a developer's and a security engineer's viewpoint. Having been involved with dependency management in various capacities at each company she has worked for, Dhivya brings a wealth of practical experience and insights into the complexities of securing modern software supply chains.