Uplevel your security program with AI

Uplevel Your Security Program with AI

Speaker: Aditi Bhatnagar

Conference: NorthSec 2025 — May 15–16, 2025, Marché Bonsecours, Montreal

Watch on YouTube: https://www.youtube.com/watch?v=M2UQkdYoYaU

Reading time: ~7 minutes

TL;DR

Security teams are chronically understaffed, buried in manual triage work, and struggling to communicate across organizational lines. Aditi Bhatnagar, a product security veteran who has led security programs at Microsoft, Atlassian, and Apollo, argues that large language models are not a replacement for security engineers but a force multiplier that can absorb the repetitive context-gathering, documentation, and communication overhead that consumes most of a security team's working hours. The talk surveys practical LLM integrations — many of them open-source — including dynamic risk assessment, automatic security ticket generation, threat model assistance, and cross-team communication improvement.

Introduction

The structural problem in enterprise product security is not a shortage of talented people who can find and reason about vulnerabilities. It is the ratio of security engineers to the surface area they are expected to cover. Bhatnagar spent several years leading platform and product security teams at Atlassian and Apollo, and in that time she interviewed more than forty security engineers across different organizations about how they actually spend their time. The picture that emerged was consistent: the majority of the working day is consumed by tasks that require contextual knowledge but not deep security expertise — reading product documentation to understand what a service does, writing up security tickets that developers will actually act on, assembling scorecard data for risk reporting, and routing review requests to the right team.

Her talk opens with a frank admission that when she transitioned from software engineering to product security, she was struck by how much less automation existed in security workflows compared to engineering workflows. Software engineers had CI/CD pipelines, linters, automated test suites, and deployment tooling. Security engineers had spreadsheets, manual processes, and best-effort context gathered through conversations. Scaling security under those conditions means hiring more people, which most organizations cannot do fast enough to keep pace with engineering growth.

LLMs change that equation — not by automating the work of finding vulnerabilities, which requires contextual judgment, but by automating the scaffolding work that surrounds it.

▶ Watch: The scaling problem in product security (0:00)

Understanding LLMs Through a Security Lens

Before presenting use cases, Bhatnagar applies a security engineer's lens to the standard definition of a large language model: a deep neural network trained on vast datasets to understand and generate text by predicting word sequences, adjusted through feedback. Each component of that definition carries a security implication.

"Deep neural network" means retraining is expensive — mitigating a flaw in the model's behavior by retraining is not a practical response in most organizations. "Vast datasets" introduces the possibility of poisoned training data and recursive pollution, where adversarially generated content on the internet influences model behavior in ways the user cannot detect or control. "Predict word sequences" surfaces the LLM's fundamental limitation: it is a statistical predictor, not a reasoner. Bhatnagar uses the analogy of someone with no subject expertise who has overheard five conversations and confidently tells the sixth person what they expect to hear.

The "feedback" component introduces risks of bias: cultural, geographic, and domain-specific biases baked into training can affect which security recommendations an LLM generates, which threat models it considers complete, and which code patterns it classifies as safe.

These are not arguments against using LLMs in security workflows. They are calibration points. The correct deployment model, which Bhatnagar returns to throughout the talk, keeps humans in the loop for decisions and uses LLMs for the draft, the summary, the first pass — not the final judgment.

▶ Watch: LLM limitations through a security lens (8:00)

Practical Use Case 1: Dynamic Risk Assessment and Security Review

One of the most time-consuming phases of a security review is the initial context-gathering phase. Before a security engineer can assess risk, they need to understand what a product does, what its threat model looks like, what similar products have experienced in the past, and what compliance requirements apply. For a security engineer covering dozens of products, this context has to be rebuilt from scratch for each review.

Bhatnagar describes using LLMs to assemble a dynamic risk profile for a product before the human review begins. The system pulls product documentation, past security findings, known architecture patterns, and regulatory context, and generates a structured summary that the security engineer reviews before the meeting. Instead of spending the first half of a review session establishing context, the engineer can focus the conversation on the specific risks that matter for this feature.

The same approach applies to security scorecards — the per-team security posture reports that product security programs use to track compliance and communicate risk to business stakeholders. Assembling scorecards manually requires interviewing each team, reading their documentation, and normalizing the results into a consistent format. An LLM-assisted pipeline can do much of the data assembly, leaving the engineer to review and validate rather than collect from scratch.

▶ Watch: Dynamic risk assessment and scorecard automation (4:00)

Practical Use Case 2: Cross-Team Communication and Ticket Generation

Security teams frequently encounter a communication problem that is orthogonal to technical skill: the people who find security issues are often not the best communicators of those issues to developers. A ticket that says "fix this bug, it's problematic" does not give a developer what they need to prioritize, understand, or remediate the finding. Effective security tickets require empathy, context-setting, clear remediation guidance, and an appropriate tone — a set of skills that is separate from the ability to identify vulnerabilities.

Bhatnagar describes building ticket templates at Atlassian that standardized the communication of security findings: what the vulnerability is, why it matters, how to fix it, and how to test the fix. LLMs can generate a first draft of these tickets given the raw finding, the product context, and the relevant engineering team, adjusting tone and technical depth based on the audience. A ticket aimed at a junior developer looks different from one aimed at a senior engineer, and the LLM can be prompted to produce each.

She also recounts the specific incident that first led her to explore automation: a routing change at Atlassian that required two hundred customized tickets to be sent to two hundred different microservice teams, each specifying the exact file and line of code that needed to change. She wrote a Python script with GitHub APIs to handle it. With modern LLMs, she notes, that same task would be faster and more flexible — and the technique generalises to any situation where security findings need to be propagated across a large engineering organization at scale.

▶ Watch: Communication gaps and automated ticket generation (4:00)

Practical Use Case 3: Threat Modeling and Log Analysis

Bhatnagar surveys several additional domains where LLMs are seeing adoption in security programs, presenting open-source implementations the audience can examine and deploy.

Threat modeling is a structured activity that is chronically underperformed in most engineering organizations because it requires domain knowledge that security engineers struggle to distribute at scale. LLM-assisted threat modeling tools can take an architecture diagram or a description of a system and generate a first-pass STRIDE analysis — identifying potential spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege vectors. The output is not a finished threat model; it is a structured starting point that a security engineer and an engineering team can review, challenge, and extend in a fraction of the time a blank-slate session would require.

Log analysis is a second high-value domain. Human analysts reviewing security logs are slower and less consistent than LLMs at pattern recognition across large datasets. The automation opportunity is real: LLMs can surface anomalies, cluster related events, and generate plain-language summaries of what a log sequence appears to represent. The human analyst's role shifts from reading raw logs to reviewing and acting on the LLM's hypothesis.

Throughout this section, Bhatnagar is careful to repeat the core qualifier: more successful implementations augment human teams rather than replacing them. The LLM generates; the human decides.

▶ Watch: Threat modeling assistance and log analysis at scale (12:00)

Notable Quotes

"As a software engineer I felt I had a good set of tools and automations in process to make my life easier. But when I switched into security engineering, most of the time it was best effort — we were trying to provide full security, but we only know so much."

"Thirty percent of security engineers were still skipping review requests — we literally wrote that our team is at capacity, we cannot review this."

"More successful implementations augment human teams — that is the narrative that the industry is largely going ahead with. Just don't plug in LLM and forget about it. Have human feedback in the loop."

"People who control data on the internet have a larger say on how the model would be. If you use Claude or Gemini or anything, you really have no control on what it was trained on. You just live with it."

Key Takeaways

• The security scaling problem is structural, not a hiring problem. The ratio of security engineers to engineering surface area is unlikely to be solved through headcount alone; automation of the surrounding work is the realistic path.
• LLMs are force multipliers for context-intensive, low-judgment tasks. Assembling product context, drafting security tickets, generating scorecard data, and producing first-pass threat models are high-volume tasks where LLMs reduce human time substantially.
• Keeping humans in the loop is not optional. LLMs are statistical predictors without accountability; consequential security decisions — vulnerability triage, risk acceptance, remediation priority — require human review of LLM-generated outputs.
• Communication quality is a security program output. A finding that a developer cannot act on does not improve security posture; LLM-assisted ticket generation can close the gap between "identified a vulnerability" and "developer remediated it."
• Understanding LLM limitations in security terms enables better deployment. Poisoned training data, opaque feedback-driven biases, and the fundamental statistical nature of LLM outputs are operational risks that practitioners should model before deploying LLM-assisted security tooling.
• Most of the relevant tooling is open source. Practitioners can evaluate LLM integration in their own environments today without building from scratch.