AI Web App hacking

AI Web App hacking

Speakers: Tim Grogan, Cloud Security Engineer

Conference: CrikeyCon

YouTube: https://www.youtube.com/watch?v=d8QfnLkWixE

Overview

This talk, presented by Tim Grogan, a seasoned Cloud Security Engineer, delves into the burgeoning and often overlooked security challenges posed by the integration of Artificial Intelligence into web applications. Moving beyond the typical discussions of large language models (LLMs) and chatbots, Grogan focuses on bespoke, proprietary AI models developed for specific business functions within both government and private sectors. The core of his presentation revolves around two distinct penetration test case studies, highlighting contrasting outcomes: one where robust defenses and a closed AI model successfully thwarted attacks, and another where a generative, learning AI significantly expanded the attack surface, leading to multiple successful exploits.

Grogan emphasizes that while AI adoption is accelerating across all facets of technology, current risk management frameworks often provide high-level guidance without specifying the technical assurance measures, such as penetration testing, needed to secure these integrations. This gap, coupled with the proprietary nature of many custom AI models, prevents the wider industry from learning from common mistakes, making talks like this crucial for sharing practical insights. His work demonstrates that many traditional web application security risks, as outlined by frameworks like the OWASP Top 10, are directly transferable and amplified when AI components are introduced, necessitating a proactive and technically rigorous approach to security testing.

The talk serves as a critical wake-up call for organizations integrating AI into their web services. It underscores that even highly complex, purpose-built AI systems are not immune to fundamental security flaws, particularly concerning input validation, output handling, and the management of internal trust boundaries. Grogan's experiences offer invaluable lessons for defenders and penetration testers alike, illustrating how a blend of established web application hacking techniques and AI-specific threat modeling can uncover significant vulnerabilities in these emerging technological landscapes.

Background

▶ Watch: Introduction: AI in web apps, pentesting challenges (0:00)

The pervasive integration of Artificial Intelligence into daily life and enterprise operations—from consumer devices to critical infrastructure—has created an urgent need for robust security frameworks. As Tim Grogan notes, it's increasingly difficult to find a device or service that doesn't incorporate some form of AI. This rapid adoption has spurred the development of various AI risk management frameworks by governmental bodies and international organizations, including the Australian DTA, the EU AI Act, the US Department of State's enterprise strategy, and the widely adopted NIST AI Risk Management Framework. These frameworks consistently advocate for proactive mitigation of potential AI-related risks and vulnerabilities. However, a critical observation made by Grogan is that they often fall short on providing specific technical guidance, rarely explicitly recommending hands-on technical assurance activities like penetration testing or red team assessments.

While high-level policy is essential, the practical "how-to" for securing AI remains a challenge. Some technical frameworks are emerging, such as Google's advice on red-teaming AI infrastructure and Microsoft's Pirate, a Python-based red team framework designed to automate attacks against AI systems. OWASP, a long-standing authority in web application security, has also released recommendations specifically addressing risks within AI and Large Language Models (LLMs). Grogan's talk, however, highlights a particular niche: proprietary, custom-developed AI models, often trained on specific, closed datasets for unique business use cases, rather than general-purpose LLMs. These bespoke integrations present a unique set of challenges, as their black-box nature and lack of public scrutiny make it difficult for organizations to learn from industry-wide mistakes, thus necessitating diligent, focused security assessments.

The problem, as illuminated by Grogan's case studies, stems from the fact that these custom AI components, while powerful, dramatically expand the attack surface of traditional web applications. They introduce new vectors for well-known vulnerabilities like data poisoning, injection attacks, and sensitive information disclosure, often bypassing existing perimeter defenses designed for conventional web traffic. The talk's background sets the stage for understanding why even organizations with mature security postures can find themselves vulnerable when integrating AI without specific technical foresight and rigorous testing.

Key Findings

▶ Watch: Client 1: Visual vs. data similarity attack (unsuccessful) (9:00)

Tim Grogan's presentation detailed two distinct penetration test engagements, revealing contrasting security postures and outcomes related to AI integration in web applications.

Client 1: Federal Government Agency (Unsuccessful Attack)

The first client was a large federal government agency operating a public-facing web service for intellectual property rights, specifically for comparing submitted logos and imagery against existing proprietary data.

• AI Model Architecture: This service featured a closed AI model developed internally and trained exclusively on a proprietary, on-premise dataset. Crucially, the model had no generative capabilities and was not designed to accept new information for retraining post-deployment, operating as a static, closed circuit. The web service itself ran on an Azure app utilizing modern frameworks, protected by cloud infrastructure, edge security devices (including a WAF), and other perimeter controls.
• Threat Model & Attack Vectors: The pen test team identified potential risks including data/model poisoning through user submissions, fuzzing, and attempting to manipulate output handling to achieve favorable outcomes for unfavorable submissions. Specific attacks included:
• Iterative Image Modification: Submitting a baseline image and then iteratively modifying it (over a million iterations) by subtly altering bytes via a hex editor. The goal was to map the model's similarity bounds and trick it into misclassifying visually distinct images as highly similar (e.g., 99.99% data similarity for visually half-rendered images).
• Prompt Injection: Attempting to embed prompt injection payloads as metadata or directly within images to override the AI's core instructions, aiming for a "favorable outcome for everything" directive, similar to LLM jailbreaks.
• Standard Web Application Attacks: Fuzzing and other common web application attacks were also applied to validate overall application security.
• Results: The attacks were unsuccessful on all fronts. The robust defense-in-depth strategy, including modern web application frameworks, cloud-hosted infrastructure, WAFs, and other security devices, effectively identified and blocked fuzzing attacks and data manipulations. The closed nature of the AI model, which did not accept new training data or exhibit generative capabilities, proved highly resilient against poisoning attempts. Prompt injection attempts showed no evidence of being accepted or interpreted by the engine, largely due to the model's black-box nature and closed operational design.

Client 2: Private Software Developer (Successful Attack)

The second client was a software developer offering a web-based systems engineering platform. This platform utilized a generative AI to assist engineers in building and managing complex, model-based technical diagrams, with a key feature allowing users to export and share these systems.

• AI Model Architecture: Each customer of this platform received an isolated cloud tenancy, running the application on IaaS with local databasing and a dedicated copy of the generative AI engine. This AI engine was designed to be contextual, continuously learning from user projects and providing recommendations based on previously created steps and components. It received regular updates and algorithms from the central developer environment. A shared project library stored exported diagrams.
• Threat Model & Attack Vectors: Concerns included model poisoning, data exfiltration, sensitive information disclosure via the export/sharing function, and the integrity of AI recommendations. Attack vectors included:
• Data/Model Poisoning with Payloads: Creating multiple projects with deliberately malformed data and standard web application injection payloads (e.g., XSS script tags) embedded within diagram steps, component names, and custom icons (taglines, filenames). These were designed with conditional triggers to fire if recommended in future projects.
• Canaries for Data Exfiltration: Embedding "canaries" (unique identifiers or sensitive-looking data) across various fields (dependencies, data flows, content) within the diagrams to track what information was redacted, if any, during the export and sharing process.
• Output Handling Assessment: Testing whether AI recommendations, if poisoned, would undergo any due diligence, output encoding, or security validation before being presented to end-users.
• WAF Bypass: Observing if internal AI-generated payloads could bypass perimeter WAFs.
• External Reach: Checking if the AI model could initiate connections to untrusted external domains.
• Results: This engagement was highly successful in identifying significant vulnerabilities.
• Data Poisoning & XSS: The generative AI model successfully learned and reproduced malicious payloads from the poisoned training data. When these learned patterns were recommended in subsequent projects, they triggered cross-site scripting (XSS) alerts, prompts, and other malicious behaviors directly within the user interface.
• WAF Bypass: Payloads introduced through the AI, acting as an internal trusted source, bypassed the edge security devices (WAFs) that would typically block such input from an end-user, demonstrating a critical internal trust boundary failure.
• Lack of Output Encoding: The AI's recommendations were output directly without any form of encoding or sanitization, propagating malicious data to end-users.
• External Connectivity: The internal AI model was able to reach out to untrusted external domains, indicating a lack of internal network access controls or whitelisting.
• Data Exfiltration: When a "Trojan horse" project was exported and shared to a different client tenant, the canaries confirmed that sensitive information (relationships, dependencies, desired outcomes, component names) was still present. Redaction efforts only targeted a few specific "content" fields, leaving much other sensitive project data exposed.
• Extensive Metadata Leakage: Exported projects contained a significant amount of cloud metadata, including user identities (creators, modifiers), infrastructure details, and unique identifiers for cloud components and services. This provided a "nice JSON blob" of reconnaissance data, invaluable to an attacker.

The stark contrast between the two engagements highlights that the architecture, design, and operational characteristics (closed vs. generative/learning) of an AI model profoundly impact its security posture. While standard web application defenses are foundational, AI integrations demand additional layers of security specifically tailored to their unique behaviors, particularly concerning input trust, output sanitization, and internal communication.

Technical Deep Dive

▶ Watch: Client 2: Software developer, generative AI for diagrams (13:00)

The technical deep dive into both engagements reveals critical insights into securing AI-integrated web applications, emphasizing the differences between static, closed models and dynamic, generative ones.

Client 1: Federal Government Agency – A Resilient, Closed Model

The federal government client's application involved a public-facing web service for intellectual property rights comparison. Architecturally, user submissions from an uncontrolled endpoint (e.g., a personal PC) first passed through standard cloud infrastructure and edge security devices, including a Web Application Firewall (WAF) and other perimeter controls. The core web service was an Azure app built with modern frameworks incorporating standard web application security practices. This application then interfaced with a proprietary, closed AI comparison engine, which was hosted on-premise and trained on a specific, proprietary dataset. A key characteristic was that this AI model was static; it did not accept new information for training post-deployment and had no generative capabilities, functioning as a "closed circuit" algorithm.

The pen test team's attack methodology for data and model poisoning focused on iterative manipulation. They generated a baseline image and then, through approximately one million iterations, subtly altered its data and visuals. This involved using a hex editor to swap bytes within the image file. Grogan illustrated this with an example where two images, visually starkly different (one complete, one half-rendered with two dots vs. one dot), were determined to be 99.99% similar by the model from a data standpoint. The objective was to map the model's decision boundaries for similarity and potentially manipulate its outcome. Additionally, prompt injections were attempted by embedding specially crafted instructions within image metadata or directly into the image data itself, aiming to override the model's core instructions and force a "favorable outcome" regardless of the submission.

The failure of these attacks was attributed to several factors. The robust defense-in-depth strategy, encompassing modern web application frameworks, cloud-hosted infrastructure, WAFs, and other security devices, effectively intercepted and mitigated most fuzzing attempts and data manipulations at the perimeter. More significantly, the AI model's closed-source nature and its design as a black box (no new training data accepted, no generative capabilities) rendered data poisoning and prompt injection attempts ineffective. The model's pre-trained, static state meant it simply did not interpret or incorporate new, malicious instructions or data into its core logic. This highlights that a well-designed, strictly closed AI model, combined with strong traditional web application security, can be highly resilient.

Client 2: Private Software Developer – A Vulnerable, Generative Model

In contrast, the private software developer's web-based systems engineering platform, featuring a generative AI for diagram creation, presented a significantly expanded attack surface. The architecture involved isolated cloud tenancies for each client, with the application running on IaaS. Each tenant housed its own local database and a dedicated copy of the AI engine. This engine was designed to be contextual and continuously learning, evolving its recommendations based on the projects created by that client. It also received ongoing updates and algorithm improvements from the central developer. A shared project library facilitated the export and sharing of diagrams.

The attack methodology capitalized on the AI's learning and generative capabilities. The team created multiple "poisoned" projects, each containing hundreds of artifacts or objects. Within these objects (e.g., diagram steps, component names, custom icons), they embedded various web application injection payloads, including cross-site scripting (XSS) script tags. For instance, payloads were placed within image taglines, filenames, and the descriptive text of diagram elements. The intent was for the AI to learn these malicious patterns as "valid" inputs and then propagate them as recommendations in future projects. To track data exfiltration, "canaries" were strategically placed in fields like dependency names, data flows, and desired outcomes.

The success of the attacks stemmed from critical security omissions:

1. Data Poisoning and XSS Propagation: The generative AI successfully learned the malicious payloads embedded in the training projects. When the AI subsequently generated "recommendations" for new projects, it directly presented these payloads to the user. Grogan reported triggering alert boxes, prompts, and other XSS behaviors as a direct result of the AI's recommendations.
2. Bypass of Edge Security: A key technical finding was that while direct user input of script tags might be blocked by the perimeter WAF, payloads that were learned and then recommended by the AI (an internal, implicitly trusted source) successfully bypassed these edge security devices. This demonstrated a critical failure in the internal trust model, where AI-generated content was not treated with the same scrutiny as external user input.
3. Lack of Output Encoding: There was a complete absence of output encoding or sanitization for AI-generated recommendations. The malicious data, once learned, was rendered directly in the user's browser, leading to the XSS vulnerabilities.
4. External Reach and Network Controls: The AI model, operating within the isolated tenant, was able to reach out to untrusted external domains to pull in malicious, externally hosted data. This indicated a lack of internal network access controls, such as IP whitelisting or allow lists, for the AI component.
5. Sensitive Information Disclosure and Metadata Leakage: Exporting a "Trojan horse" project to a different client tenant revealed significant data exfiltration. The canaries confirmed that while some "content" fields were redacted, crucial sensitive information like relationships, dependencies, desired outcomes, and component names remained present. Furthermore, the exported project contained a substantial amount of cloud metadata in a JSON blob. This included details about users (original creators, modifiers), underlying infrastructure, and unique identifiers for cloud components and services within that tenancy—providing invaluable reconnaissance data for an attacker.

These findings underscore that generative, learning AI models introduce a far more complex attack surface. They can inadvertently become vectors for traditional web application vulnerabilities, especially when proper input validation, output encoding, and internal trust boundaries are not rigorously enforced. The AI's ability to learn and propagate malicious data, combined with a lack of internal security controls, amplifies the impact of otherwise standard injection attacks.

Demo / Proof of Concept

▶ Watch: Client 2: Data exfiltration and cloud metadata leakage (23:00)

While the talk did not feature a live, interactive demonstration during the presentation, Tim Grogan vividly described the methodologies and artifacts used to prove the existence of vulnerabilities during the actual penetration tests. These "proofs of concept" were instrumental in illustrating the successful exploitation of the generative AI application (Client 2) and the resilience of the closed AI model (Client 1).

For the federal government client (Client 1), the proof of concept involved generating a large dataset of images with subtle, byte-level modifications. Grogan showcased how an image that was visually starkly different—half-rendered with only one dot instead of two—could still be deemed "99.99% similar" to a baseline image by the AI model from a data perspective, due to specific byte swaps made via a hex editor. This demonstrated the team's ability to map the model's internal similarity metrics, even if they couldn't exploit it. The prompt injection attempts, while unsuccessful, would have been proven by evidence of the AI overriding its safety instructions, had they worked.

For the private software developer (Client 2), the proofs of concept were much more impactful and directly demonstrated active exploitation:

• Cross-Site Scripting (XSS) Alerts: The most direct proof of concept involved triggering JavaScript alert boxes and other XSS prompts within the end-user's browser. This occurred when the generative AI, having "learned" malicious script tags embedded in previously created project components, then recommended these same script tags as part of a new diagram element. The lack of output encoding meant these scripts executed directly in the browser, unequivocally proving the XSS vulnerability.
• Malicious Custom Icons: The demonstration involved creating custom icons for diagram elements where the image taglines, filenames, or embedded metadata contained injection payloads. These payloads, when rendered by the application through the AI's recommendations, would also execute, showing how even seemingly innocuous features like custom imagery could be weaponized.
• Data Exfiltration through Canaries: The team created "Trojan horse" projects containing strategically placed canaries (e.g., unique strings in dependency names, data flows, desired outcomes). When these projects were exported and then imported into a separate client tenant, the presence of these canaries in the exported JSON blob served as undeniable proof that sensitive, unredacted data was being transferred, confirming the data exfiltration vulnerability.
• Cloud Metadata Leakage: The exported project file itself served as a proof of concept for metadata leakage. Grogan described how this file contained a "nice JSON blob" at the beginning, detailing extensive cloud infrastructure information, unique identifiers for services, and user details (creators, modifiers). This readily available reconnaissance data was a clear demonstration of a significant information disclosure vulnerability.

These proofs of concept, though described rather than interactively shown, effectively illustrated the severity of the identified vulnerabilities and the direct impact they could have on the integrity, confidentiality, and availability of the AI-integrated application. They underscored that the AI, when improperly secured, could become an internal attack vector, circumventing traditional perimeter defenses and propagating malicious content.

Defensive Implications

▶ Watch: Conclusion: OWASP Top 10 and LLM risks overlap (27:00)

The findings from Tim Grogan's penetration tests provide critical defensive implications for organizations integrating AI into web applications. The contrasting outcomes between the two clients underscore that while standard web application security remains foundational, AI introduces unique challenges that demand specific considerations.

1. Re-evaluate Trust Boundaries for AI-Generated Content:

The most significant implication is that AI-generated content, especially from generative or learning models, cannot be implicitly trusted. It must be treated with the same, or even greater, skepticism as direct user input. The successful bypass of WAFs and perimeter defenses by AI-propagated payloads highlights a critical internal trust boundary failure. Defenders must ensure that any output from an AI model, whether a recommendation, a generated image, or a piece of text, undergoes rigorous security checks before being presented to the end-user or used elsewhere in the application.

2. Implement Robust Input Validation and Sanitization:

While common for web applications, input validation is crucial for AI models, even if the input is intended for training. For generative AI, strict validation and sanitization of all data fed into the model are paramount. This includes not just text fields but also metadata, filenames, and embedded data within files (e.g., images). The goal is to prevent malicious payloads from ever entering the AI's "knowledge base" or influencing its learning process.

3. Enforce Comprehensive Output Encoding:

A glaring vulnerability in the second case study was the complete lack of output encoding for AI recommendations. Defenders must implement robust output encoding for all AI-generated content before it is rendered in a web browser or used in other parts of the application. This mitigates XSS and other client-side injection attacks by ensuring that any potentially malicious characters are neutralized.

4. Secure the AI Model Itself:

• Closed Models: For static, closed AI models, ensure they truly are closed. Confirm that no new data is inadvertently accepted for training or model modification post-deployment. Regular audits of the model's operational environment and data inputs are essential.
• Generative/Learning Models: These require more sophisticated controls. Implement rule sets akin to a WAF directly within or around the AI model. This means training the model to identify and filter out malicious payloads, meta-characters, and untrusted data types from both its inputs and its outputs. Develop and maintain allow lists for acceptable data types, objects, and recommendation structures. Continuous retraining with known payloads can help build resilience.
• Black-Box Nature: Acknowledge the black-box nature of many AI models. Since internal workings might be opaque, focus security efforts on the inputs and outputs, ensuring both are strictly controlled and validated.

5. Implement Strong Internal Network Segmentation and Access Controls:

The ability of the generative AI to reach untrusted external domains was a significant vulnerability. AI components, like any other sensitive system, should operate within strict network segmentation. Implement IP whitelisting and other network access controls to restrict outbound connections from the AI model to only necessary, trusted endpoints.

6. Prioritize Data Redaction and Minimization:

For applications with export or sharing functionalities, a thorough review of data redaction mechanisms is critical. As demonstrated, simply redacting "content" fields is insufficient. Organizations must identify all sensitive data (including relationships, dependencies, component names, and especially cloud metadata) and ensure it is properly redacted, anonymized, or removed before sharing or exporting. Adopt a data minimization approach, only sharing what is absolutely necessary.

7. Conduct AI-Specific Threat Modeling and Penetration Testing:

The most crucial defensive implication is the need for proactive, technical security assurance specifically tailored for AI integrations. Generic risk frameworks are insufficient. Organizations must:

• Perform AI-specific threat modeling to identify potential attack vectors like data poisoning, model evasion, prompt injection, and output manipulation.
• Engage in red team assessments and penetration testing that target the AI components directly, using methodologies that combine traditional web application hacking techniques (OWASP Top 10) with AI-specific attack strategies (OWASP LLM Top 10). This hands-on testing is vital to uncover vulnerabilities that policy documents or automated scans might miss.

By adopting these defensive strategies, organizations can move beyond theoretical AI risk management to implement tangible, technical controls that secure their AI-integrated web applications against sophisticated and evolving threats. The balance between functionality and security, as highlighted by Grogan's work with the second client, is achievable through diligent secure development practices and continuous security testing.

Key Takeaways

• AI Expands Attack Surface: Integrating AI into web applications significantly broadens the attack surface, introducing new vectors for traditional vulnerabilities and amplifying their potential impact.
• Traditional Web App Security Remains Critical: Many core web application security principles, particularly those from the OWASP Top 10, are directly applicable and essential for securing AI-integrated apps. The crossover between OWASP's LLM/AI risks and traditional web app risks is almost one-to-one.
• Generative AI Poses Higher Risk: Generative and learning AI models, which continuously incorporate new data and provide recommendations, are inherently more vulnerable to data poisoning and output handling issues compared to closed, static models.
• Output Encoding and Input Validation are Paramount: Treat all AI-generated content as untrusted user input. Robust output encoding is crucial to prevent XSS and other injection attacks, while strict input validation and sanitization are necessary to prevent malicious data from entering the AI's training data or influencing its behavior.
• Beware of Internal Trust Bypasses: Malicious payloads propagated by an internal AI can bypass perimeter defenses like WAFs, highlighting the need for internal network segmentation, access controls, and security checks on AI-generated content before it reaches the end-user.
• Proactive Technical Assurance is Essential: High-level AI risk management frameworks are insufficient. Hands-on penetration testing and red teaming, specifically targeting AI components and combining traditional and AI-specific attack methodologies, are vital to identify and mitigate real-world vulnerabilities.

About the Speaker(s)

Tim Grogan is a Cloud Security Engineer with over a decade of experience in the security domain. His career path reflects a diverse background, starting predominantly as a penetration tester, a role he continues to embrace for web application assessments. He has also held positions as an infrastructure engineer, security consultant, and briefly in GRC (Governance, Risk, and Compliance), before transitioning back into engineering. Currently, he focuses on cloud security engineering, leveraging his extensive pentesting experience to secure modern cloud environments and applications. His practical insights into hacking AI-integrated web applications stem directly from his ongoing work in penetration testing for various clients.