Action Required: A Mixed-Methods Study of Security Practices in GitHub Actions

Action Required: A Mixed-Methods Study of Security Practices in GitHub Actions

Speakers: Yusuke Kubo

Conference: NDSS Symposium 2026

YouTube: https://www.youtube.com/watch?v=7XqHcNuD-po

Overview

GitHub Actions has become the mainstream CI/CD platform for modern software development, and its popularity makes it a highly attractive target for attackers. The 2025 compromise of TJ Actions and the SRT campaign hack of the NX package demonstrated the real-world consequences of insecure CI/CD pipelines. While GitHub provides security guidelines recommending 16 security practices, this talk presents the first large-scale study investigating how well developers actually implement them. Analyzing over 338,000 public repositories and surveying 102 developers, the researchers found that security practices are largely unimplemented -- security features and tools (P1, P3, P5) are used in only 0.6% to 10.7% of applicable repositories. The primary barriers are lack of awareness (21-72% of developers), maintenance overhead concerns, and developer misconceptions about what the practices do and who they apply to.

Background

▶ Watch: 2025 GitHub Actions incidents: TJ Actions compromise and SRT campaign (0:30)

GitHub Actions is a CI/CD platform integrated directly into GitHub, enabling developers to automate build, test, and deployment workflows through declarative YAML configurations. Its seamless integration with GitHub repositories has driven widespread adoption, but the same integration creates security risks: workflows can access repository secrets, execute arbitrary code, and interact with external services.

GitHub's official security documentation recommends 16 security practices covering security features, coding workflows, and automated tools. The researchers narrowed the study to five representative practices due to feasibility constraints: practices covering security features (like OpenSSF Scorecards for security auditing), coding practices (like pinning actions to commit SHAs rather than tags), and automated tools (like Dependabot for dependency management).

Recent supply chain attacks against CI/CD platforms -- including the TJ Actions compromise in 2025 and the SRT campaign targeting the NX package -- underscore the importance of these practices. These attacks exploited exactly the kinds of weaknesses that GitHub's security guidelines are designed to prevent.

Key Findings

▶ Watch: Results: security practices implemented in only 0.6-10.7% of repos (5:30)

Security practices are rarely implemented: None of the five studied practices achieved widespread adoption. Security features and tools (P1, P3, P5) were used in only 0.6% to 10.7% of applicable repositories. This is remarkably low given the known risks and the availability of GitHub's official guidance.

Organization ownership promotes security: Repositories owned by organizations are more likely to implement security practices (P1, P3, P5) than user-owned repositories, with odds ratios less than one for user-owned repos. This suggests a governance effect -- organizational oversight and policies promote security practice adoption in ways that individual developer initiative does not.

Repository popularity does not predict security: While metrics like star count and contributor count show statistically significant associations with some practices, the odds ratios are very close to one, indicating negligible practical effect. Popular, well-known repositories are not meaningfully more secure than obscure ones.

Three primary barriers to implementation:

1. Lack of awareness (21.3-71.6%): The most prominent barrier. For OpenSSF Scorecards (P3), 71.6% of developers were simply unaware the tool existed.
2. Maintenance overhead (25.3% for P4): Pinning actions to commit SHAs is maintenance-intensive because developers must manually look up and update hashes. 79.4% of developers cited low maintenance overhead as critical for adoption.
3. Misconceptions: Developers avoid Dependabot because their project's programming language is "unsupported" (overlooking that in GitHub Actions context, Dependabot manages action dependencies regardless of language). Others believe SHA pinning makes them vulnerable to outdated dependencies (when dependency tools can auto-update hashes).

Technical Deep Dive

▶ Watch: Organization ownership effect on security practice adoption (6:30)

The study combines quantitative measurement and qualitative user research.

Measurement Study: The researchers used the SGHS dataset (containing repository metadata) as a base, then used the GitHub API to identify which repositories use GitHub Actions, yielding over 338,000 public repositories. A custom detection framework statically analyzes each repository to determine: (1) whether each security practice is applicable to the repository, and (2) whether it is implemented. Implementation rates are calculated among applicable repositories only. Logistic regression analysis identifies which repository characteristics (star count, contributor count, owner type, etc.) significantly affect implementation.

User Study: The top 100 contributors from each repository were identified as recruitment candidates. After filtering and random sampling, 1,800 developers were contacted via email, yielding 102 valid responses (approximately 5.7% response rate). The survey covered implementation status, reasons for non-implementation, and key driving factors. All questions were multiple-choice with free-text "other" options. Responses were analyzed using descriptive statistics and qualitative coding for open-ended responses.

Two pilot studies informed the questionnaire design, and the iterative coding process revealed the three barrier categories (awareness, maintenance, misconceptions) through bottom-up analysis of developer responses.

Demo / Proof of Concept

▶ Watch: Developer misconceptions about Dependabot and SHA pinning (9:30)

The study's quantitative results provide a comprehensive snapshot of GitHub Actions security practice adoption at scale. The detection framework, data collection scripts, and survey materials are all publicly available on GitHub as research artifacts. The recommendations were shared with GitHub directly, and GitHub provided positive feedback stating the findings align with their future direction and will inform their platform decisions. The specific misconceptions identified (e.g., Dependabot language support, SHA pinning vulnerability beliefs) provide actionable targets for documentation improvements and developer education.

Defensive Implications

▶ Watch: Recommendations: notifications, platform support, and documentation (11:00)

For DevOps and security teams, the finding that awareness is the primary barrier suggests that simply surfacing security practice recommendations at the right moment could significantly improve adoption. GitHub could integrate security practice detection into the repository dashboard, notifying developers of unimplemented practices. However, notification design must be careful to avoid notification fatigue.

For platform providers (GitHub, GitLab, etc.), the maintenance overhead barrier points to platform-level solutions. If pinning to commit SHAs were automated with human-approval workflows, the maintenance burden would drop dramatically. IDE-level integration with real-time security guidance during development (a "shift left" approach) would address coding practice adoption.

For security governance teams, the organization ownership effect confirms the value of mandatory security policies for CI/CD pipelines. If your organization uses GitHub Actions, explicit policies requiring security practices like SHA pinning, Dependabot, and Scorecards would likely improve adoption more effectively than training alone.

For documentation teams, mapping each practice to specific risks and real-world incidents (like the TJ Actions compromise) would motivate implementation. Clearly explaining who should use each practice and under what circumstances would prevent the misconceptions identified in the study.

Key Takeaways

• GitHub Actions security practices are implemented in only 0.6-10.7% of applicable repositories
• Lack of awareness is the top barrier, with up to 71.6% of developers unaware of security tools like OpenSSF Scorecards
• Organization-owned repositories are more likely to implement security practices than individual-owned ones, suggesting governance drives adoption
• Repository popularity (stars, contributors) has negligible effect on security practice implementation
• Developer misconceptions about tools and practices actively prevent adoption
• Maintenance overhead, especially for SHA pinning, deters implementation; 79.4% of developers want low-overhead practices
• Findings shared with GitHub, who confirmed alignment with their future platform direction

About the Speaker(s)

Yusuke Kubo is a researcher at NTT Docomo Business and a PhD student at Waseda University in Japan. The research is a collaboration with colleagues at Waseda University, combining large-scale software repository analysis with developer survey research to understand the human factors behind CI/CD security practice adoption.