Hack to the Future: Owning AI-Powered Tools with Old School Vulns

Hack to the Future: Owning AI-Powered Tools with Old School Vulns

Speakers: Neal Samier, Security Researcher, Kudelski Security; Nathan Hammel, Senior Director of Research, Kudelski Security

Conference: Black Hat USA 2025 — August 6-7, 2025, Mandalay Bay, Las Vegas

YouTube: https://www.youtube.com/watch?v=oaU6a8nuyT8

Reading Time: ~11 minutes

Type: Briefing

TL;DR

Kudelski Security's research team audited over a dozen AI-powered developer tools — code review agents, data analytics assistants, and AI coding agents — and found vulnerabilities in every single one. The most severe: a RuboCop configuration injection in CodeRabbit (the most-installed AI code review app on GitHub) that yielded the app's private GitHub key, granting write access to over one million repositories, including private ones. Classic vulnerabilities — RCE, SQL injection, privilege escalation, hardcoded credentials — are reappearing at scale inside an entirely new attack surface that most organizations haven't begun to assess.

Introduction

The shift toward AI-assisted development has created a new category of high-privilege software running inside developer workflows: tools that automatically review pull requests, generate documentation, analyze databases, and write code. These tools are granted write access to repositories, database credentials, and API keys as a matter of course — often with permissions that aren't clearly explained to the users granting them.

Neal Samier and Nathan Hammel spent months auditing this category. Their findings are a systematic indictment of how AI tool developers treat security: most have no security staff, no structured vulnerability reporting channels, and no validation of security practices. When Kudelski disclosed bugs, some vendors replied quickly; others never replied at all; at least one had broken disclosure instructions on their website. Meanwhile, the underlying vulnerabilities are not novel — they're SQL injection, path traversal, RCE, privilege escalation, and hardcoded credentials. The same bugs the industry has been fighting since the early days of web applications, now embedded in tools that touch every developer's workflow.

The Attack Surface: What AI Developer Tools Actually Do

The talk examines two categories of AI developer tools in depth.

AI Code Review Tools integrate with Git providers like GitHub and GitLab. When a developer pushes code, the tool fetches the diff, runs static analyzers, sends the code and analysis output to an LLM, and posts the generated review as a comment. These tools typically require write access to repositories — ostensibly to post comments, but often with permissions that extend to editing releases and modifying workflow files.

AI Data Analytics Tools accept database credentials, read schema information, translate natural language questions into SQL queries via LLM, and execute those queries. The data flows from the user's production database through the vendor's infrastructure and back.

▶ Watch: Overview of AI Tool Categories and Attack Surface (02:00)

The fundamental problem, as Hammel frames it: "Gone are the days of knowing the difference between good security practices and bad security practices. When you deploy these tools, you increase your attack surface. Deploying these tools creates new vulnerabilities where there weren't any previously." These systems execute code whose behavior cannot be fully predicted — the developer who ships a product using an LLM-powered component doesn't necessarily know what code will run at runtime. That is new, and it matters.

CodeRabbit: From RuboCop Config to One Million Repositories

The headline finding of the research is the CodeRabbit compromise. CodeRabbit is the most-installed GitHub app in the AI-assisted code review category, reviewing over one million GitHub repositories at the time of disclosure.

▶ Watch: The CodeRabbit Discovery (14:01)

The attack vector was deceptively simple. CodeRabbit's documentation states that if a RuboCop configuration file exists anywhere in a repository, it will be passed to the RuboCop Ruby static analyzer. RuboCop configuration files can instruct the analyzer to load additional Ruby files — which execute arbitrary Ruby code.

The attack chain:

1. Create a pull request containing a dummy Ruby file (to trigger RuboCop)
2. Include a malicious RuboCop config file instructing it to load x.rb
3. x.rb contains arbitrary Ruby code — in this case, code that sends all environment variables to an attacker-controlled endpoint

The exfiltrated environment variables contained the CodeRabbit GitHub App private key. Because CodeRabbit had been granted write access to repositories by over a million users, possession of that private key meant the researchers could act as the CodeRabbit GitHub App — with write access to every one of those repositories, including private ones.

"There were so many secrets in those environment variables that it actually took me a whole minute to figure out what we had gotten access to here," Samier noted. CodeRabbit confirmed the fix after responsible disclosure.

CodeMerge (PR Agent), Dot, and a Pattern of Repeated Failures

The vulnerability work began with CodeMerge (formerly PR Agent), which Kudelski's developers were using internally. Two vulnerabilities were found:

1. On GitLab, a reporter-role user could trigger CodeMerge to post a comment containing a GitLab quick action that executed with maintainer privileges — a classic prompt injection leading to privilege escalation.
2. On GitHub, a PR comment could overwrite the GITHUB_API_BASE_URL setting via the Dynaconf library that CodeMerge uses for configuration management, redirecting API calls to an attacker-controlled endpoint and leaking access tokens.

▶ Watch: CodeMerge Repeated Bypasses (22:01)

When CodeMerge patched the second vulnerability by blocking the keyword dot_base_url, the researchers bypassed it using Dynaconf's special syntax to transform strings into JSON objects — effectively achieving the same effect with different characters. The developers patched again by blocking dot_user. The researchers bypassed again. The pattern repeated across multiple iterations, demonstrating a fundamental failure mode: patching individual exploit strings instead of fixing the underlying architecture.

The CodeMerge Pro SaaS variant added an administrator-access AWS secret key to the list of exfiltrated credentials — one with permissions to do everything on the vendor's AWS infrastructure, including accessing repositories of all CodeMerge Pro users.

Dot, an AI data analytics tool, had a command injection vulnerability in an API endpoint that yielded RCE on their production system, leaking Postgres database credentials that contained the database access credentials of all Dot customers.

Ask Your Database, another analytics tool, allowed arbitrary Python execution by design — and despite restricting pip, had full web access. With web access, any Python module can be downloaded directly, completely bypassing the pip restriction. The deployed tool became an anonymous proxy for attacking other systems.

Structural Problems: Why These Bugs Keep Appearing

Hammel identifies several systemic anti-patterns driving this vulnerability wave:

Blind execution of input. The canonical AI tool architecture takes user input, combines it with LLM output, and executes the result. This is, by definition, RCE as a service. "We are congratulating people building tools which are basically RCE as a service."

Least privilege replaced by most privilege. GitHub repositories were found running five different AI code review agents simultaneously, each with write access. AI tools are routinely granted administrative or near-administrative permissions by default, often without users understanding what those permissions actually enable.

▶ Watch: Why These Vulnerabilities Keep Appearing (08:00)

Security in the prompt doesn't work. Multiple vendors attempted to prevent SQL injection or unauthorized actions by including restrictions in LLM system prompts. "Never do security in the prompt. Prompts are more like suggestions than commands." The Amazon Q incident — where someone planted machine-wiping commands inside the coding agent's prompt — illustrates why prompt-based access control fails at scale.

Permissions obscured by GitHub documentation. The "write to code" permission granted during GitHub App installation also silently grants the ability to edit existing releases. The "write to workflows" permission enables modifying GitHub Actions — which can be used to leak repository secrets. These implications are not surfaced in the GitHub installation UI without consulting documentation.

Hard-coded credentials. One project treated TLS certificates as secrets (not publishing them) while publishing hard-coded database credentials in a public repository. The inversion of what constitutes a secret is becoming common.

Sorcery and the IDOR Pattern

Sorcery, an AI code review product with Sentry integration, had an IDOR (Insecure Direct Object Reference) vulnerability in an internal API endpoint that allowed reading data from any Sorcery account. Since Sorcery used itself for its own source code and used the Sentry integration, the researchers could access Sorcery's own production error messages — some of which contained GitHub access tokens belonging to Sorcery users.

▶ Watch: Sorcery IDOR and Permissions (30:02)

Sorcery had also requested the "write to workflows" permission for its GitHub App — meaning that if the app's private key had been leaked (as CodeRabbit's was), it would grant modification rights to GitHub Actions across all user repositories, with trivial path to leaking repository secrets.

How to Test AI Tools in Your Own Environment

The presenters close with practical testing guidance for security teams:

• Inventory before you audit. Many developers deploy AI tools without security team awareness. Audit what's actually running first, especially tools integrated into CI/CD pipelines.
• Test dynamically. Most AI tools are closed-source or black boxes. Static analysis of the source code won't reveal what the LLM does with inputs — dynamic testing is required.
• Assume a larger input scope. AI tools don't just accept chat inputs. They consume code diffs, configuration files, README files, database schemas, and pull request comments. All are attack surfaces.
• Apply the "refrain, restrict, trap" framework. Refrain from using AI in high-risk contexts. Restrict permissions to the minimum necessary. Trap inputs and outputs with monitoring and guardrails.
• Treat generative AI components as lazy, intoxicated interns, not as highly capable systems. This mental model leads to better security decisions around what AI components are allowed to do autonomously.

▶ Watch: Recommendations for Security Teams (34:02)

Notable Quotes

"We found vulnerabilities in all of them — it's kind of like pulling the lever on a slot machine and winning every single time." — Neal Samier ▶ 02:00

"AI security is just application and product security. Models do nothing on their own. The big problem is if you don't have a mature application security program in place, you're going to amplify the amount of vulnerabilities that go into production." — Nathan Hammel ▶ 36:02

"Changing the name of your company is not a valid patching strategy." — Neal Samier ▶ 28:02

"Think of prompts less like rigid, specific commands and more like suggestions — this has led to a lot of problems." — Nathan Hammel ▶ 36:02

Key Takeaways

• Classic vulnerabilities dominate AI tool security failures. RCE, SQL injection, IDOR, hardcoded credentials, and privilege escalation — not AI-specific attack classes — are responsible for the most severe findings. Existing AppSec practices apply directly.
• Write access to repositories is the nuclear permission. Any tool granted GitHub App write access to repositories creates catastrophic risk if its private key is compromised. The CodeRabbit finding (one million repos) demonstrates this at scale.
• Configuration files are an overlooked injection vector. RuboCop configs, Dynaconf settings, and similar configuration mechanisms can execute arbitrary code when processed by AI tools that blindly pass repository-resident files to third-party analyzers.
• Prompt security controls fail. Attempting to restrict AI agent behavior through system prompt instructions is not a security control. Architectural restrictions — sandboxing, least privilege, output validation — are required.
• AI tool vetting must become standard practice. Before deploying any AI developer tool, security teams should audit the permissions it requests, test it dynamically for injection vulnerabilities, and verify that a responsible disclosure process exists.

Slides

No slides PDF was listed for this talk, but the presenters noted all slides are available for download with attacks preserved (only secrets removed). A bonus section covering additional vulnerabilities and recommendations was included in the distributed slide deck.