AI: Best Janitor or Worst Superhero?

AI: Best Janitor or Worst Superhero?

Speakers: Adrian Sanabria

Conference: BSidesSF 2024

YouTube: https://www.youtube.com/watch?v=S34uHyuy6yc

Overview

Adrian Sanabria's talk, "AI: Best Janitor or Worst Superhero?", delivered at BSidesSF 2024, critically examines the current state of generative AI, particularly large language models (LLMs), within the cybersecurity industry. Sanabria argues that while AI is undoubtedly a useful and accessible technology, the pervasive tendency to over-pitch its capabilities as a "superhero" solution risks damaging its reputation and hindering its adoption for genuinely valuable applications. He posits that AI's true strength lies not in performing miracles, but in its role as a "janitor"—automating mundane, low-stakes tasks that save time and increase productivity.

The presentation delves into the historical cycles of technological hype in security, drawing parallels between past over-promises and the current AI fervor. Sanabria emphasizes that the accessibility of modern LLMs, coupled with a robust open-source ecosystem, makes this wave of AI different, ensuring its longevity despite the hype. However, he cautions against deploying AI in high-stakes scenarios where its inherent inconsistencies and propensity for "hallucination" can lead to significant failures and ultimately, organizational disillusionment. The talk serves as a guide for security professionals to become "good stewards of the technology," advocating for a pragmatic approach that leverages AI for what it does best: augmenting human effort in repetitive, text-based, or code-generation tasks.

Background

▶ Watch: Over-pitching AI can do damage to the technology. (00:40)

The cybersecurity industry has a well-documented history of buzzword-driven hype cycles, where new technologies are often over-promised by vendors and venture capitalists (VCs) seeking "moonshot" returns. Sanabria references a talk by Jay Jacobs, Wade Baker, and Alex Pinto at RSA, which compiled and analyzed buzzword trends in conference titles over time, illustrating this recurring phenomenon. This cycle typically involves an initial vision, followed by numerous companies iterating on the idea, aggressive marketing with bold promises, and then, ideally, a "managed disillusionment" phase where the technology's true utility is discovered.

However, Sanabria highlights a critical problem within the security market: this disillusionment phase is often circumvented by early acquisitions. Companies, sometimes even before emerging from stealth, are acquired based on hype rather than proven utility. This frequently leads to useful technologies being "shunned" or disappearing entirely if the acquiring company fails to integrate or develop the product beyond its initial, often unrealistic, promises. Examples cited include application control, which promised the end of malware but proved unmanageable in dynamic environments; micro-segmentation, which struggled with scalability; and network access control (NAC), which found a niche in guest networks but failed to deliver on enterprise-wide promises. The common pitfalls for these technologies were excessive labor demands on the buyer, scalability issues, or inherent complexity.

Sanabria points out that AI itself has experienced this cycle before, not once but twice, leading to "AI winters" in the 1980s and 1990s where mentioning AI could lead to grant money being denied. He contrasts this with the current environment where, just last year, his company was "dinked" for not mentioning AI in an innovation sandbox. IBM's Watson is presented as a prime example of over-promising and under-delivering, particularly in healthcare, a pattern they are now attempting to rehash with LLMs. The core issue, according to Sanabria, is the marketing and sales strategy where companies feel compelled to "sell 1,000 to be able to sell something that can do 100" just to capture attention in a saturated market.

Key Findings

▶ Watch: What's different here... generative AI is different because of that. There's ... (10:00)

Sanabria's talk presents several key findings regarding the current state and future trajectory of AI in security:

• AI's Accessibility is a Game Changer: Unlike previous iterations of AI and machine learning, modern generative AI is highly accessible to consumers and developers. The availability of free tools like ChatGPT, coupled with a vast ecosystem of open-source LLMs (e.g., Meta's Llama) that can run on consumer hardware, allows for widespread experimentation and validation. This "freemium" model and open-source availability mean that the technology's utility can be "sussed out" early and broadly, fostering genuine adoption based on merit rather than just marketing.
• Rapid Improvement and Mitigation of Weaknesses: The pace of development in generative AI is unprecedented. Issues that were widely mocked 12 months ago, such as hallucination and poor performance in deterministic tasks, are rapidly being addressed through techniques like fine-tuning data, grounding data, system prompts, and Retrieval Augmented Generation (RAG). This continuous evolution suggests that AI, in some form, is "here to stay."
• Strengths Lie in Language and Code, Weaknesses in Deterministic Facts: LLMs excel at tasks involving language understanding, generation, summarization, and translation. They are also remarkably proficient at generating code, treating it as "just another language." However, they struggle with deterministic tasks requiring precise math or factual recall, often needing to externalize these functions (e.g., writing a Python script for calculations). Visual interpretation also remains a significant challenge.
• Inconsistency and the "Human in the Loop": Even with the same prompt, LLMs can produce different answers minutes apart, or even claim inability to answer before successfully doing so later. This inherent inconsistency necessitates a "human in the loop" for quality assurance, especially for any output used in critical contexts. Sanabria humorously notes that some users even resort to "stroking the AI's ego" in prompts (e.g., "you're very smart and pretty") to elicit better responses, highlighting the unusual nature of interacting with this technology.
• The "Janitor" Role is AI's Best Fit (for now): The most effective and least risky application of AI in security is for "low-stakes" tasks that humans find boring or time-consuming. These "janitorial" tasks, such as filling out security questionnaires, summarizing documents, or translating between different query languages, increase human productivity without risking critical failures. Deploying AI in high-stakes scenarios where it fails can lead to organizational distrust and the technology being "burned."
• Enterprise Adoption is Cautious: Enterprises are currently in a "piloting" phase with tools like Microsoft Co-pilot, demonstrating a cautious approach rather than widespread, heavy adoption. Misunderstandings about data privacy (e.g., fear of prompt data leaking into training models) and the potential for software bugs are significant concerns.

Technical Deep Dive

▶ Watch: Math and facts it's not great at... but we've got solutions for that... it'll... (14:00)

The core technical distinction of the current AI wave, as highlighted by Sanabria, is its unprecedented accessibility. Unlike previous machine learning models that required specialized knowledge and interfaces, generative AI has been "thrust in front of consumers for free" by companies like OpenAI. This, combined with a burgeoning open-source ecosystem (e.g., Meta's Llama models runnable on discrete GPU laptops), allows individuals and organizations to readily experiment and integrate the technology. This widespread, hands-on testing is crucial for validating utility and fostering genuine adoption, a stark contrast to the expensive, late-stage proof-of-concepts typical of traditional security product evaluations.

Sanabria delves into the strengths and weaknesses of LLMs. Their primary strength lies in their mastery of language. They excel at tasks such as summarizing text, generating creative content, and translating between different languages—including programming languages. For an LLM, a Python script is simply another form of language, which explains its proficiency in code generation. This capability is leveraged to overcome inherent weaknesses; for instance, while LLMs are "bad at math," they can be prompted to write and execute a Python script to perform calculations, effectively externalizing the deterministic task to a tool they can generate. This is a key insight into how LLMs are being engineered to handle complex problems.

However, LLMs struggle with deterministic facts and visual interpretation. They are prone to hallucination, generating plausible but incorrect information. Sanabria notes that the technology is rapidly improving, with solutions like fine-tuning data, grounding data, system prompts, and Retrieval Augmented Generation (RAG) being developed to mitigate these issues. RAG, for example, allows LLMs to retrieve information from external, authoritative knowledge bases before generating a response, significantly reducing factual errors.

The talk also touches on the practical implications of LLM implementation. Sanabria observes that different implementations of the same underlying model (e.g., ChatGPT vs. Microsoft Co-pilot using OpenAI libraries) can yield vastly different results due to varying system prompts, fine-tuning data, and connections to enterprise data sources like Microsoft Graph. This underscores the importance of understanding the specific configuration and context of an LLM deployment.

Furthermore, the discussion highlights the emerging trend of specialized LLMs. Sanabria mentions "purpose-built LLMs" for tasks like finding vulnerabilities or reversing binaries into functional source code. He also anticipates the ability to "pump a peap into it," suggesting future multimodal LLMs capable of processing network traffic captures for analysis. The development of offline, air-gapped versions of Co-pilot (a partnership between Microsoft and Intel) indicates a move towards addressing data privacy and security concerns by enabling local model execution.

Despite these advancements, a critical technical challenge remains: consistency. Sanabria recounts instances where the same prompt given minutes apart yielded different answers, or where an LLM initially claimed ignorance before successfully answering the same question later. This unpredictable behavior necessitates a "human in the loop" for quality assurance, especially when the output is used for anything beyond low-stakes, internal tasks. The peculiar observation that "stroking the AI's ego" in prompts can lead to better results further illustrates the non-deterministic and sometimes counter-intuitive nature of interacting with these advanced models.

Demo / Proof of Concept

▶ Watch: Point it at something high stakes and it fails that's a good way to get it bu... (24:00)

Adrian Sanabria illustrated the capabilities and limitations of generative AI through several practical examples, some of which served as direct demonstrations during the talk:

1. AI-Generated Slides and Images: The presentation itself served as an initial proof of concept. Sanabria revealed that not only were the slides' ideas often AI-generated, but the images within them were also created by AI. He noted the characteristic "pop out as AI generated" look of the images and humorously pointed out that despite prompting for "worst superhero or best janitor," the AI struggled to depict the "worst" aspect, instead showing a superhero performing janitorial duties. This highlighted the AI's difficulty in accurately interpreting nuanced or abstract concepts, even with explicit prompts.

1. Pizza Math Problem: Sanabria presented a classic problem: "Are two 12-inch pizzas more or less than one 18-inch pizza?" He explained that while this is difficult for humans to calculate mentally, and LLMs are "bad at math," they overcome this by writing a Python script. The LLM is internally prompted to use Python's math library to perform the calculation, treating the script as "just another language." The answer, confirmed by the AI, is that two 12-inch pizzas offer more area than one 18-inch pizza. This demonstrated the AI's ability to leverage its code generation strength to compensate for its mathematical weakness.

1. Summarization and Categorization: The speaker showed an example where an LLM was asked to summarize the evolution of the SIEM market. He noted that the AI broke it down into three eras and described each in a way that matched the consensus of experienced security professionals. This highlighted the AI's proficiency in understanding, summarizing, and structuring complex textual information.

1. Code Generation for Security Operations: A practical example involved asking the AI to "write me python code that isolates an ec2 instance." Sanabria stated that the AI did a "great job" at this, demonstrating its utility in generating scripts for Sore workflows or other automation tasks. He also mentioned seeing AI create config files for early-stage SIEM startups, even with custom query languages, implying that its training data had consumed enough documentation to perform such specific tasks.

1. Job Description Generation: Challenging the notion that AI couldn't handle specific, nuanced tasks, Sanabria demonstrated asking an LLM to "write me a job description for a detection engineer." He recounted a real-time interaction where he pasted the AI's output into a Zoom chat, surprising a colleague with its quality and specificity. This underscored the AI's ability to generate professional, contextually relevant text for specialized roles.

1. Personal Bio Hallucination: To illustrate the LLM's tendency to hallucinate and its difficulty with factual accuracy, Sanabria showed an AI-generated bio for himself. While one version claimed he worked for the NSA (which he stated never happened), even the "best version" still got details wrong, such as his past employment at Tenable Security, because web pages hadn't been updated. This highlighted the need for human verification, especially when dealing with personal or time-sensitive factual data, as LLMs often lack real-time access to authoritative sources like LinkedIn.

1. Visual Landmark Challenge: Sanabria presented a "Geo challenge" where he fed a picture of a unique sculpture (giant bowling pins and balls) to a multimodal LLM and asked, "Where was this picture taken?" Initially, the AI responded that it was "not possible to determine the exact location just from the image" due to a lack of specific landmarks. When Sanabria replied, "It's a photo of a landmark, you just said you needed a landmark," the AI immediately corrected itself, providing the location, sculptor, and all requested information. This peculiar interaction demonstrated the AI's inconsistency, its literal interpretation, and the strange phenomenon where "stroking the AI's ego" or rephrasing a prompt can unlock correct answers.

These demonstrations collectively reinforced Sanabria's central thesis: AI is a powerful tool for specific tasks, but its limitations, inconsistencies, and propensity for hallucination mean it's best suited for "janitorial" roles where human oversight can mitigate risks.

Defensive Implications

▶ Watch: You can have the LLM write that query language for you and now you don't have... (32:00)

Adrian Sanabria's talk provides crucial guidance for cybersecurity defenders on how to pragmatically approach and integrate generative AI into their operations. The overarching defensive implication is to shift focus from viewing AI as a "superhero" capable of miraculous, high-stakes problem-solving to recognizing its value as a "janitor" that automates mundane, low-stakes tasks, thereby enhancing human productivity.

1. Prioritize Low-Stakes, High-Volume Tasks: Defenders should identify and target tasks that are repetitive, boring, and time-consuming but do not carry significant risk if the AI makes an error. Examples include:

• Security Questionnaires: Automating responses to third-party risk management or vendor questionnaires. While the AI might occasionally get something wrong, the human in the loop can easily spot-check and correct, saving significant time.
• Documentation and Summarization: Using AI to summarize lengthy reports, threat intelligence feeds, or policy documents.
• Query Language Translation: A significant pain point for security analysts is learning and maintaining proficiency in multiple query languages (e.g., for SIEM systems, EDR tools, osquery, Velociraptor, Elastic). AI can act as a universal translator, allowing analysts to describe what they want in natural language and have the AI generate the correct query for the specific tool. This eliminates the need for "cheat sheets" and reduces the learning curve for new tools.
• Job Description Generation: As demonstrated, AI can quickly draft detailed job descriptions for specialized security roles like detection engineers, freeing up HR and hiring managers.
• Basic Code Generation: Generating scripts for automation, Sore workflows, or simple utility functions (e.g., isolating an EC2 instance).

1. Maintain a "Human in the Loop" for Quality Assurance: Due to the inherent inconsistencies and potential for hallucination in LLMs, human oversight is non-negotiable. Defenders must implement robust quality assurance processes for any AI-generated output, especially for tasks that feed into decision-making or external communications. This involves spot-checking, reviewing, and validating AI responses.

1. Understand AI's Strengths and Weaknesses: Defenders need to educate themselves on what generative AI is genuinely good at (language processing, code generation, summarization) and where it struggles (deterministic math, factual accuracy without grounding, visual interpretation, consistency). This understanding prevents misapplication and subsequent disillusionment.

1. Leverage Accessibility for Experimentation: The widespread availability of open-source LLMs and freemium commercial offerings means defenders can experiment with AI locally and safely. Sanabria encourages having AI tools "pinned in your browser" or "on your phone" to constantly ask, "Can AI do this thing I hate doing?" This hands-on approach builds practical knowledge and identifies viable use cases.

1. Be Wary of High-Stakes Deployments: Deploying AI in mission-critical, high-stakes scenarios where failure has severe consequences is risky. A single, public failure can "burn" the technology within an organization, leading to its complete rejection. The Air Canada example, where an AI-generated refund policy led to a legal obligation, serves as a stark warning against trusting AI with policy-making or customer-facing commitments without stringent human review.

1. Address Data Privacy Concerns: Enterprises are concerned about data leakage from prompts. While Sanabria suggests software bugs are a more likely vector than training model ingestion, defenders must understand how their chosen AI services handle data, whether prompts are used for training, and explore options like isolated, on-premise, or air-gapped LLMs (e.g., Microsoft/Intel's Co-pilot initiative) for sensitive data.

1. Prepare for Evolving Capabilities: The rapid pace of AI development means that current limitations may be solved quickly. Defenders should stay informed about advancements like RAG for factual accuracy, multimodal LLMs for diverse data inputs (e.g., peap analysis), and specialized LLMs for security-specific tasks (e.g., vulnerability finding, binary analysis).

In essence, the defensive implication is to adopt a pragmatic, iterative, and human-centric approach to AI. Use it to augment, not replace, human intelligence in areas where it can genuinely save time and reduce drudgery, always with a critical eye and a robust quality assurance process.

Key Takeaways

• AI is a useful "janitor," not a "superhero": Its true value lies in automating mundane, low-stakes tasks that save time and increase human productivity, rather than performing miracles or solving complex, high-stakes problems autonomously.
• Accessibility and open-source are game-changers: Unlike previous AI waves, the widespread availability of free and open-source LLMs allows for broad experimentation and validation, ensuring the technology's longevity despite market hype.
• Understand AI's strengths and weaknesses: LLMs excel at language-based tasks (summarization, code generation, translation) but struggle with deterministic facts, precise math (often externalizing to Python scripts), and visual interpretation.
• Human oversight is crucial due to inconsistency: LLMs can produce varying results for the same prompt and are prone to hallucination. A "human in the loop" for quality assurance is essential, especially for any output used in critical contexts.
• Avoid high-stakes deployments to prevent "burning" the technology: Using AI for critical functions where failure has significant consequences can lead to organizational distrust and rejection of the technology. Focus on tasks like filling out security questionnaires or translating query languages.
• Experiment continuously and pragmatically: Defenders should actively explore how AI can automate their most disliked, repetitive tasks, leveraging its capabilities to augment their work rather than expecting it to replace them.

About the Speaker(s)

Adrian Sanabria is an experienced professional in the cybersecurity field. He is known for hosting a podcast and is the organizer of BSides Knoxville, which celebrated its 10th year in 2024. Additionally, he serves as faculty over at Dian. His background includes working for companies like Veilance Security, where he observed the industry's shifting attitudes towards AI.