Everyday AI: Leveraging LLMs for Simple Security Tasks

Everyday AI: Leveraging LLMs for Simple Security Tasks

Speakers: Matthew Sullivan, Dominic Zanardi

Conference: BSidesSF 2025 — April 26-27, 2025, San Francisco

YouTube: Watch the full talk

Reading time: ~6 minutes

TL;DR

Instacart's infrastructure security team built a suite of LLM-powered automations to tackle identity and access management problems that deterministic code could not solve — access request classification, role right-sizing, Terraform policy cleanup — achieving 95% auto-approval rates for access requests and dramatically reducing engineering burden. Matthew Sullivan and Dominic Zanardi share practical patterns, working code, and hard-learned guardrails for teams that want to automate the gray areas in security operations.

Introduction

Security teams face a structural problem: the surface area they are responsible for keeps expanding while headcount stays flat or shrinks. Automation has always been the answer in theory; in practice, automation breaks down exactly where security work is hardest — on the non-deterministic inputs, the ambiguous justification statements, the edge cases that require judgment rather than rules.

This is where Sullivan and Zanardi situate LLMs. Not as a replacement for security engineers, and not as a magic multiplier that makes everyone quadruple their output. Rather, as a tool that unlocks automation in precisely the gray areas where previous approaches failed. The talk is grounded firmly in production experience at Instacart, covering the team's deployment methodology, real automation examples with shared code, and clear-eyed guidance on what LLMs can and cannot be trusted to do.

▶ Watch: Introduction and the case for LLMs in security (00:00)

The Deployment Architecture: Python and Tines

Before the use cases, the speakers establish two deployment patterns Instacart uses to put LLMs into production:

Pure Python running on a Lambda function or Docker container, using the OpenAI Python library or a plain requests library. This is the classic path for teams comfortable writing code.

Workflow builder as middleware — specifically, Tines, Instacart's SOAR platform. Tines offers a community free tier, and the speakers have published their example workflows for import. Workflow builders can be invoked via webhook and return JSON responses, meaning teams can construct LLM-powered REST APIs in minutes without writing server code. Sullivan demonstrated this with a toy app that uses OpenAI's then-new internet search capability to query current weather conditions for any city — a workflow built from a URL on screen.

The core flexibility claim: simple automations can be built with a few sentences of prompt. When Python is needed, it's typically a few lines of code. And the automation can QA itself by adding additional LLM calls to validate outputs.

▶ Watch: Python and Tines deployment patterns (04:00)

The Problem Space: Human Identity at Scale

The unifying theme across all of Instacart's LLM use cases is the identity problem. The numbers in play: tens of roles, hundreds of daily access requests, thousands of employees, millions of permissioned objects. No human can perform identity engineering at this scale through manual review. Traditional automation can do pattern matching, but cannot handle natural language justifications or ambiguous resource references.

Sullivan's illustration: when a manager goes on vacation — let's say, "gallivants around South America trying to find their truest self" — access requests assigned to them sit unapproved for days. Instacart's solution: run an LLM evaluation against the approver's Slack status message to assess the likelihood they are available to take an approval action. This works for obvious cases ("I'm on vacation") and, Sullivan reports, also handles edge cases like an approver whose entire status is the skiing emoji. LLMs can also assess duration semantically — "out of surgery" is likely to extend longer than "out for lunch."

Sullivan's summary of this particular automation: "I thought of this idea at about ten in the morning, and it was in production by lunch. That is the true power of using no-code tools plus LLMs."

▶ Watch: Human identity at scale and the Slack status use case (08:00)

Access Request Classification and Role Right-Sizing

Zanardi walks through the more complex examples, starting with a notorious class of access request: the bad justification statement. Real example from Instacart's ticket queue: a request for an S3 bucket that simply says, "the one with the data." This request is completely non-actionable under traditional automation.

The LLM-powered workflow handles it in a multi-step sequence:

1. Resource identification — translate the natural language description ("the orders table") into an actual AWS resource, resolving DynamoDB vs. RDS, confirming the resource exists, catching spelling errors, and identifying the target environment
2. Role eligibility assessment — given the identified resource and the requester's job function, determine what roles the person is eligible for
3. Policy-as-code proposal — suggest the Terraform changes needed to grant appropriate access
4. Human review — surface the LLM's recommendation to a human approver who makes the final merge decision

The result: 95% of access requests made through Instacart's identity governance tool are auto-approved with no human intervention. The remaining 5% are routed to the LLM-assisted workflow described above. This reduces approval latency from days to seconds for the vast majority of requests, without removing human oversight from edge cases.

▶ Watch: Access request classification and role recommendation demo (12:00)

Automated Terraform Permission Cleanup

The third use case addresses permission creep in Terraform-defined IAM roles. Instacart built an automation to remove unused permissions from roles defined in Terraform, scheduled to run quarterly. The pipeline analyzes CloudTrail data to identify permissions that have not been used, maps unused permissions back to the Terraform source, generates a pull request with the proposed changes, and surfaces it for human review before merging.

Zanardi notes that while several security vendors offer unused-privilege detection, very few support doing so by modifying the Terraform at the source on a sustainable schedule. Instacart's implementation creates a continuous right-sizing process rather than a point-in-time remediation event.

▶ Watch: Automated Terraform permission cleanup (14:02)

Security and Guardrails: What LLMs Don't Know

Zanardi covers the input side of the problem with the same care given to outputs. The key point: LLMs know the cloud — they have been trained on AWS, GCP, and Azure IAM structures and can reason about them fluently. But they don't know your organization. Feeding them too much context to compensate creates a different problem: data leakage.

The practical guidance:

• Do not send PII to the LLM. If IAM policies contain user email addresses or account identifiers, tokenize those values with nonces before sending them to the model and reverse the mapping on the return.
• Use existing platform sanitization rather than building your own. If users provide input through a mature platform like Jira or ServiceNow, that platform has already invested in input sanitization. Use it as middleware rather than allowing open-ended input directly into an LLM API call.
• Calibrate sanity checking to stakes. For the Slack status use case, a wrong answer means an approval sits a bit longer — low stakes, minimal validation needed. For Terraform changes, higher stakes require more rigorous validation before any action is taken.

On hallucinations: the risk is decreasing with each model generation, but it remains real. The team's stance is that LLM recommendations should improve context and reduce the decision burden on humans, but LLMs should not be making final decisions on high-stakes actions. The explicit line Instacart has drawn: the LLM can generate code, create a branch, and open a pull request — but merging that PR is a full stop for human review.

▶ Watch: Input sanitization, PII handling, and human-in-the-loop decisions (14:02)

Notable Quotes

"Security team surface area keeps widening, but budgets and headcount are not. We must automate to keep up, and LLMs have helped the team at Instacart do just that." — Dominic Zanardi at 02:00

"I thought of this idea at about ten in the morning, and it was in production by lunch. That is the true power of using no-code tools plus LLMs." — Matthew Sullivan at 10:00

"We haven't arrived at the point of handing over critical decision-making to an LLM. Our recommendation is to let the LLM provide better context and recommendation, and then you or your deterministic code base need to be making the final decision." — Matthew Sullivan at 16:02

Key Takeaways

• LLMs solve the non-deterministic input problem that blocked earlier automation. Access request triage, resource identification from natural language descriptions, and permission analysis all require judgment that deterministic code cannot provide — LLMs can now handle these at production scale.
• 95% auto-approval is achievable with LLM-assisted access control. Instacart routes the majority of identity governance requests through LLM classification with no human review, reserving human attention for the ambiguous 5%.
• No-code workflow builders accelerate LLM deployment dramatically. Tines and similar SOAR platforms allow teams without full-time engineers to build and deploy LLM automations in hours rather than sprints. The code is published and importable.
• Sanitize inputs before sending to the model. Tokenize PII, use mature platforms for user input rather than open-ended LLM prompts, and calibrate validation rigor to the stakes of the decision being made.
• Keep humans in the loop for final decisions. The productive boundary Instacart has established: generate code, open PRs, surface recommendations — but require human approval before merging changes to production systems.