Unveiling the Dual Nature of ChatGPT and Copilot in Secure Development

Kalyani Pawar (Application Security Engineer · Zipline)

AppSec Village @ DEF CON 31 · Day 3 · AppSec Village

Overview

In her compelling talk at AppSec Village @ DEF CON 31, Kalyani Pawar (Kaye) explored the duality of ChatGPT and GitHub Copilot in the realm of application security. Her presentation, "Unveiling the Dual Nature of ChatGPT and Copilot in Secure Development," delved into how these powerful AI models can be leveraged to address persistent challenges faced by AppSec engineers, while simultaneously highlighting the significant risks and limitations inherent in their current capabilities. The core message resonated with a crucial caveat: while AI offers immense potential for efficiency and automation, its outputs must never be blindly trusted.

Watch on YouTube

Visual summary for Unveiling the Dual Nature of ChatGPT and Copilot in Secure Development by Kalyani Pawar
Visual summary for Unveiling the Dual Nature of ChatGPT and Copilot in Secure Development by Kalyani Pawar

Key moments

  1. 0:00 Introduction and speaker background
  2. 4:15 Key challenges for AppSec engineers
  3. 7:55 How AI can assist AppSec engineers
  4. 14:00 ChatGPT's nuanced risk scoring for SQL injection
  5. 19:30 Generating security test cases with Co-pilot
  6. 24:30 ChatGPT fails at CodeQL query generation
  7. 26:00 Critical limitations of AI tools in AppSec
  8. 27:55 Main takeaway: do not blindly trust AI

Unveiling the Dual Nature of ChatGPT and Copilot in Secure Development

Speakers: Kalyani Pawar (Application Security Engineer, Zipline)

Conference: AppSec Village @ DEF CON 31

YouTube: https://www.youtube.com/watch?v=rYdmoKhaXHE

Overview

In her compelling talk at AppSec Village @ DEF CON 31, Kalyani Pawar (Kaye) explored the duality of ChatGPT and GitHub Copilot in the realm of application security. Her presentation, "Unveiling the Dual Nature of ChatGPT and Copilot in Secure Development," delved into how these powerful AI models can be leveraged to address persistent challenges faced by AppSec engineers, while simultaneously highlighting the significant risks and limitations inherent in their current capabilities. The core message resonated with a crucial caveat: while AI offers immense potential for efficiency and automation, its outputs must never be blindly trusted.

Kalyani Pawar, who leads application and vendor security at a drone company, Zipline, brought her nearly half-decade of security experience to dissect the practical applications and pitfalls of integrating AI into secure development workflows. She framed the discussion around the pressing issues in AppSec today, such as the scarcity of security talent, the rapid pace of development, and the ever-evolving threat landscape. AI, she argued, presents a compelling "possible solution" to these problems, acting as an instructional, scalable, and efficient assistant.

However, the talk meticulously demonstrated that this assistance comes with a critical trade-off. Through a series of practical examples, Pawar illustrated how AI can both accurately identify vulnerabilities and provide secure coding suggestions, yet also fail spectacularly by introducing new insecurities, generating erroneous code, or completely missing critical attack vectors. The presentation served as an essential guide for security professionals and developers looking to harness AI responsibly, emphasizing the indispensable role of human oversight, domain expertise, and rigorous verification in an AI-assisted future.

Background

▶ Watch: Introduction and speaker background (0:00)

The rapid adoption of AI language models like ChatGPT and GitHub Copilot has profoundly impacted various industries, and application security is no exception. Kalyani Pawar began by introducing these tools: ChatGPT, an AI language model developed by OpenAI, is based on the generative pre-trained Transformer architecture and is trained on a vast and diverse dataset from the internet, enabling it to generate human-like text and understand complex contexts. GitHub Copilot, an AI pair programmer developed by GitHub in collaboration with OpenAI, is trained on a diverse range of public code repositories. It understands coding styles and generates real-time code suggestions and comments directly within integrated development environments (IDEs) like VS Code. Both tools promise unprecedented efficiency and assistance, from translating languages to writing entire programs.

However, the AppSec community faces a litany of persistent challenges that these AI tools might help address. Pawar outlined several "unsung hero" problems for Application Security Engineers:

  1. Late Vulnerability Discovery: Code is often pushed to production "many sprints ago" before vulnerabilities are identified, leading to costly remediation cycles. The fast pace of DevOps and agile development often leaves security gaps due to insufficient coverage.
  2. Lack of Security Education: A staggering 48% of developers lack security education, according to DocuSign. They primarily focus on functionality, inadvertently pushing insecure code full of vulnerabilities from the outset.
  3. Resource Constraints: Security teams are typically small, often with a ratio as extreme as one AppSec engineer supporting 200 developers. This makes it challenging to review every line of code generated by an organization.
  4. Tooling Overload and False Positives: To compensate for resource gaps, organizations onboard multiple security tools (SAST, DAST, SCA). However, these tools often generate a "bunch of false positives," forcing AppSec engineers to sift through noise, prioritize risks, and clear backlogs, consuming significant time and effort.
  5. Ever-Emerging Threats: The threat landscape is constantly evolving, with new attack vectors emerging daily, making it difficult for security professionals to stay updated.
  6. Legal Regulations: Compliance with industry standards like SOC 2 or GDPR adds an additional layer of complexity and audit tasks, often overlooked or deprioritized.

Pawar suggested that leveraging AI for AppSec could be a "possible solution" to these challenges. AI's capabilities for speedy outcomes, conversational learning, handling large data volumes, seamless automation, and user-friendly interfaces make it an appealing prospect for tasks such as code vulnerability identification, secure coding suggestions, and explaining complex security concepts to developers. The potential to reduce context switching – a significant cause of burnout for AppSec engineers – by having AI assist with routine tasks, was highlighted as a key benefit, allowing security professionals to focus on higher-value activities.

Key Findings

▶ Watch: How AI can assist AppSec engineers (7:55)

The central finding of Kalyani Pawar's talk is the dual nature of AI in secure development: it offers unprecedented potential to augment AppSec capabilities but harbors significant risks if used without critical human oversight. AI tools like ChatGPT and GitHub Copilot can indeed be powerful assistants, capable of identifying vulnerabilities, suggesting secure code, and generating security test cases, thereby improving efficiency and addressing resource constraints. However, the talk emphatically demonstrated that these tools are not infallible and can lead to a false sense of security.

Key specific findings include:

  • Vulnerability Identification: AI can accurately identify common vulnerabilities like buffer overflows and SQL injection in provided code snippets.
  • Secure Coding Suggestions: Copilot can offer real-time, often effective, secure coding suggestions, replacing vulnerable functions with safer alternatives (e.g., strcpy with strncpy).
  • Test Case Generation: Copilot can generate useful unit tests, including those specifically designed to trigger and detect certain security flaws, such as integer overflow.
  • Limitations of AI Knowledge: Both ChatGPT and Copilot have a knowledge cutoff (September 2021), meaning they are unaware of vulnerabilities or packages that emerged afterward. This makes them unreliable for identifying zero-days or recent CVEs.
  • Dependence on Prompt Engineering: The quality of AI output is directly proportional to the clarity and context provided in the prompt. Vague prompts lead to poor or irrelevant responses.
  • Intellectual Property Concerns: Users are explicitly warned against inputting sensitive or business-critical information, PII, or secrets into these public AI models due to potential data leakage and intellectual property risks.
  • Bias and Insecure Consequences: AI models are prone to bias and can generate false positives or negatives. Critically, they can introduce unintended insecure consequences, such as suggesting overly complex or partially vulnerable fixes, or completely missing primary attack vectors (e.g., SQL injection in test case generation).
  • Lack of Domain-Specific Knowledge: AI often lacks the nuanced, contextual understanding that human AppSec engineers possess. It may suggest generic, verbose solutions when a simpler, domain-specific fix is more appropriate (e.g., os.path.basename for path traversal).
  • Poor Reliability for Complex Tasks: For highly specialized tasks, such as generating complex CodeQL queries, AI can fail spectacularly, producing erroneous and time-wasting results. This highlights that AI is not a replacement for specialized expertise.

In essence, the core finding is that while AI can be a valuable assistant for AppSec, it currently serves best as a sophisticated "spell checker" for security, requiring human verification, refinement, and contextual understanding to prevent the introduction of new security risks.

Technical Deep Dive

▶ Watch: Generating security test cases with Co-pilot (19:30)

Kalyani Pawar structured the "how-to" of leveraging AI for AppSec around several practical applications, demonstrating both successes and significant failures.

Vulnerability Identification and Remediation with ChatGPT

The speaker outlined a straightforward four-step process: input your code, engineer a prompt, assess the response, and then validate, test, and review.

  1. Success Example: Buffer Overflow in C
  • Vulnerable Code: A C snippet using strcpy to copy a longer string (string2) into a smaller buffer (string1).
  • Prompt: "Hey is this code vulnerable to what and please suggest what the remediation for this is."
  • ChatGPT's Response: Correctly identified two issues: an incorrect assignment (string2 should be an array/pointer) and the use of the vulnerable strcpy function. It suggested using strncpy as remediation.
  • Remediated Code: ChatGPT adjusted the size of string1 to accommodate string2 (15 characters), performed correct variable assignment, used strncpy (checking buffer size), and ensured null termination. This was presented as a clear win for AI-assisted remediation.
  1. Partial Success/Limitations Example: SQL Injection with Risk Scoring
  • Vulnerable Code: A Python snippet demonstrating SQL injection via string concatenation for fetching user information from a public-facing website.
  • Prompt: Included additional context for a custom severity risk score calculation (Ease of Exploit, Hostility, Impact, scored 0-10, averaged) and asked for the vulnerability and its severity.
  • ChatGPT's Response: Accurately identified SQL injection. However, when calculating the risk score, it assigned a "10" for Ease of Exploit, Hostility, and Impact.
  • Speaker's Critique: Pawar argued that these scores are too generic and depend heavily on the application's context. Factors like existing guardrails on the public-facing website or the actual sensitivity of the data in the database would significantly alter the Hostility and Impact scores. This highlighted that while AI can identify issues, its contextual risk assessment lacks human nuance. The takeaway: "the more clear you are with your questions, the more context you give ChatGPT, the better is the quality of the output... but again do not blindly trust it, trust but verify."

Secure Coding Suggestions with Copilot

This application involves writing code, receiving real-time suggestions, assessing the generated code, refining the prompt, and finally reviewing and testing.

  1. Success Example: Buffer Overflow in C (again)
  • Vulnerable Code: The same strcpy example.
  • Copilot's Suggestion: Identified buffer overflow and the vulnerable strcpy. It suggested replacing it with the safer strncpy and explained why.
  • Speaker's Critique: While good, Copilot could have been even more secure by explicitly adjusting the size of string1 to precisely accommodate string2 for maximum safety, though its suggestion was still a significant improvement.
  1. Partial Success/Limitations Example: Path Traversal
  • Vulnerable Code: A Python snippet using os.path.join and user input without proper validation, making it susceptible to path traversal.
  • Prompt: "Identify this vulnerability for me, rewrite the code to eliminate it."
  • Copilot's Suggestion: Correctly identified path traversal. However, it then suggested eight different functions (os.path.join, os.path.abspath, os.path.commonprefix, os.path.exists, os.path.isfile, os.path.realpath, os.path.normpath) and generated a verbose 25-line code block to fix it.
  • Speaker's Critique: A human with domain-specific knowledge would likely use a simpler, more direct approach like os.path.basename to extract only the filename, effectively neutralizing the directory component of the path. While Copilot's fix was "more secure" than the original, it was overly complex and could still be vulnerable to maliciously crafted regular expressions. This example underscored the value of human expertise in simplifying and optimizing security fixes.

Security Test Case Generation Using Copilot

Pawar explored using Copilot to generate security-focused unit tests, following steps of identifying vulnerability, launching Copilot, writing a prompt, generating test cases, executing, and reviewing.

  1. Success Example: Integer Overflow in Python
  • Vulnerable Code: A Python program designed to demonstrate integer overflow when using C-compatible data types, where adding a large number (2^31 - 1) to 1 results in a negative value due to wraparound.
  • Prompt: "Write unit tests for vulnerable addition function in vulnerable.py."
  • Copilot's Response: Generated three test cases: one for normal addition, and two specifically to raise an OverflowError for maximum and minimum 32-bit signed integer values.
  • Execution: The tests correctly identified failures, confirming that the vulnerable_addition function did not raise the expected OverflowError for the overflow and underflow cases. This demonstrated Copilot's utility in creating targeted security tests.
  1. Failure Example: SQL Injection Test Cases
  • Vulnerable Code: The same SQL injection example from earlier.
  • Prompt: "Generate security focused test cases for me for the previous function."
  • Copilot's Response: Generated test cases for a correct username/password combination (success), and a non-existent user (expected None or failure).
  • Speaker's Critique: Copilot completely ignored the SQL injection vulnerability, failing to suggest test cases with maliciously crafted inputs (e.g., ' OR 1=1--). This was a critical miss, highlighting AI's inability to consistently identify and test for the most severe attack vectors without explicit, highly detailed prompting.

CodeQL Query Generation (Significant Failure)

  • Task: Generate a CodeQL query to find "to-do" comments in a repository.
  • Human-Written Query: A concise and effective CodeQL query was shown (from Comment c where c.getText().matches("%todo%") select c).
  • ChatGPT's Attempt: When asked the same question, ChatGPT generated a lengthy, complex, and ultimately erroneous query that produced numerous errors and unusable results.
  • Speaker's Conclusion: "I would have rather invested all my time understanding how CodeQL works... instead, ChatGPT generated this query for me for 5 hours and then I was trying to debug it... it's not a good CodeQL query developer." This starkly illustrated that for highly specialized and complex security tasks, current AI capabilities are severely limited and can be counterproductive.

Overarching Limitations

Pawar summarized several critical limitations observed throughout her research:

  • Data Cutoff: The September 2021 knowledge cutoff for ChatGPT and Copilot means they are oblivious to newer vulnerabilities, attack vectors, and package updates.
  • Prompt Dependence: The quality of AI output is highly dependent on the specificity and context of the prompts.
  • Intellectual Property Concerns: Users must never input business-critical information, PII, or secrets into these models due to potential data leakage. Fake data (e.g., www.example.com) should be used for examples.
  • Bias and False Positives/Negatives: AI models can exhibit bias and produce unreliable results.
  • Unintended Insecure Consequences: AI can introduce new vulnerabilities or suggest fixes that are incomplete or overly complex, leading to a false sense of security.

Demo / Proof of Concept

▶ Watch: ChatGPT fails at CodeQL query generation (24:30)

While Kalyani Pawar's talk did not feature a live, interactive demonstration, she effectively presented a series of illustrative code snippets and screenshots of real-world interactions with ChatGPT and GitHub Copilot. These served as compelling proof-of-concept examples, showcasing the AI tools' capabilities and limitations in action. Each example clearly depicted the input code, the prompt given to the AI, and the resulting AI-generated output, allowing the audience to critically assess the responses and understand the speaker's analysis of their accuracy, completeness, and potential pitfalls. This method of showcasing specific AI outputs against security challenges provided a tangible and practical understanding of AI's current state in AppSec.

Defensive Implications

▶ Watch: Main takeaway: do not blindly trust AI (27:55)

The insights from Kalyani Pawar's talk offer crucial defensive implications for organizations and AppSec teams looking to integrate AI into their security practices. The overarching message is clear: AI is a powerful assistant, but not a replacement for human expertise and critical thinking.

  1. Do Not Blindly Trust AI: The most critical defensive posture is to avoid a false sense of security. Every AI-generated suggestion, code snippet, or vulnerability assessment must be assessed, reviewed, and validated by a human security engineer or developer with domain knowledge.
  2. Invest in Prompt Engineering Education: Organizations should train their developers and security teams in effective prompt engineering. Understanding how to provide precise context, constraints, and iterative feedback to AI models is paramount to generating higher quality, more relevant, and secure outputs. Pawar highlighted an iterative prompt engineering hack that can significantly improve results.
  3. Implement Robust Review Processes: AI-generated code or security fixes should be subjected to the same, if not more stringent, code review and security testing processes as human-written code. This includes static analysis (SAST), dynamic analysis (DAST), and manual peer review to catch errors or unintended insecure consequences.
  4. Combine AI with Traditional Security Tools: AI should be seen as an augmentation, not a replacement. Integrate AI suggestions into existing security workflows that include SAST, DAST, SCA, and penetration testing. AI can help prioritize findings or suggest initial fixes, but traditional tools and human testers are essential for comprehensive coverage.
  5. Be Aware of AI Limitations: Defenders must be acutely aware of AI's inherent limitations:
  • Knowledge Cutoff: AI models won't know about recent vulnerabilities or zero-days (e.g., post-September 2021). Supplement with up-to-date threat intelligence.
  • Intellectual Property Concerns: Never input sensitive, proprietary, or PII data into public AI models. Use sanitized or fake data for examples and testing.
  • Bias and Inaccuracy: Recognize that AI can generate false positives, false negatives, or overly complex/suboptimal solutions.
  1. Prioritize Human Domain Knowledge: The talk repeatedly emphasized that human domain expertise is irreplaceable. Security teams should leverage their knowledge to simplify AI-generated complex fixes, identify missed attack vectors (like SQL injection in test cases), and provide contextual risk assessments that AI cannot.
  2. Focus on Developer Security Education: While AI can explain vulnerabilities, it doesn't replace foundational security education. Continue to educate developers on secure coding principles, threat modeling, and common attack vectors to reduce the reliance on AI for basic security hygiene.
  3. Contextual Risk Assessment: AI's generic risk scoring (e.g., all 10s for SQL injection) is insufficient. Defenders need to apply their understanding of the application's specific environment, guardrails, and data sensitivity to conduct accurate, contextual risk assessments.

In summary, AI offers a promising avenue for combating AppSec challenges, but its adoption must be approached with a healthy dose of skepticism, rigorous validation, and a firm commitment to human oversight.

Key Takeaways

  • AI is an AppSec assistant, not a replacement: ChatGPT and Copilot can significantly boost efficiency in vulnerability identification, secure coding, and test case generation, but they are tools to augment human capabilities, not to fully automate security.
  • Blind trust leads to false security: Relying solely on AI outputs without human verification introduces a dangerous false sense of security, as AI can miss critical vulnerabilities or introduce new insecure consequences.
  • Human oversight and domain expertise are irreplaceable: Contextual understanding, simplified solutions, and identifying nuanced attack vectors (e.g., path traversal, SQL injection in test cases) still require an AppSec engineer's domain knowledge.
  • Prompt engineering is paramount: The quality of AI's output is directly proportional to the clarity, context, and iterative refinement of the prompts provided. Investing in this skill is crucial.
  • Be aware of significant AI limitations: Current AI models have knowledge cutoffs (e.g., September 2021), pose intellectual property risks, can exhibit bias, and may generate erroneous or overly complex solutions for specialized tasks like CodeQL.
  • Integrate AI cautiously and with verification: Use AI to hack the game and alleviate burnout, but always integrate it into existing robust security workflows that include human review, validation, and comprehensive testing.

About the Speaker(s)

Kalyani Pawar, who also goes by Kaye, is a prominent figure in the application security community. She currently leads the application security and vendor security functions at Zipline, a drone company. With nearly half a decade of experience in the security domain, Kaye is passionate about her work and actively contributes to the community. Beyond her professional role, she dedicates her time to reviewing talks submitted to prestigious conferences such as Grace Hopper and the AppSec Village at DEF CON, demonstrating her commitment to fostering knowledge sharing and innovation in the field. Her insights shared in this talk are her own opinions and do not represent her employer. She can be found on Twitter/X, where she plans to share prompts and learnings from her research.

Reviews

Dr. Zero (Offensive Security Researcher) — WEAK

This is an AppSec 101 awareness session masquerading as DEF CON research. Feeding strcpy and string-concatenation SQLi into ChatGPT to prove it hallucinates isn't a breakthrough—it's what every junior dev did the week OpenAI launched. If your core takeaway is 'trust but verify' and 'don't paste company secrets into the prompt,' you are in the wrong lane.

Heather Calloway (CISO) — SOLID

A grounded reality check on developer AI tools that cuts through current vendor hype. Pawar demonstrates exactly where Copilot accelerates AppSec workflows and where it confidently introduces new vulnerabilities, giving security leaders the practical data points needed to govern AI rollouts.

→ Top-rated talks at AppSec Village @ DEF CON 31

All talks from AppSec Village @ DEF CON 31