A Fireside Chat with Cognitive Scientist and AI Expert Gary Marcus

A Fireside Chat with Cognitive Scientist and AI Expert Gary Marcus

Speakers: Gary Marcus (cognitive scientist, AI researcher, author of Taming Silicon Valley) and Nathan Hamill (Senior Director of Research, Kudelski Security; Black Hat AI/ML Track Lead)

Conference: Black Hat USA 2025 — August 6-7, 2025, Mandalay Bay, Las Vegas

YouTube: https://www.youtube.com/watch?v=e69OE0ZjskA

Reading Time: ~6 minutes

Type: Keynote

TL;DR

Cognitive scientist and AI critic Gary Marcus argued at Black Hat 2025 that generative AI's limitations — unreliable reasoning, conceptual shallowness, and susceptibility to jailbreaks — make current systems fundamentally insecure by design. He warned that AI agents are not ready for autonomous deployment, that AGI is possible but far off, and that uncritical AI adoption is quietly degrading human critical thinking at scale.

Introduction

Every year at Black Hat, the keynote stage hosts voices that challenge the industry's assumptions. For 2025, that challenge came not from a hacker or a CISO, but from Gary Marcus — professor emeritus of psychology and neural science at NYU, founder of Geometric Intelligence (acquired by Uber), and one of the most consistent public critics of AI hype. Interviewed by Nathan Hamill, senior director of research at Kudelski Security and the conference's AI/ML track lead, Marcus delivered a frank assessment of where artificial intelligence actually stands: impressive in certain narrow tasks, dangerous in autonomous contexts, and far from the general intelligence that its loudest proponents claim.

AGI Is Possible — Just Not Through Large Language Models

Marcus opened by dispelling a common mischaracterization of his position. He is not an AI skeptic who thinks machine intelligence is impossible. He believes AGI will eventually be built. What he rejects is the claim that scaling large language models is the path to get there.

▶ Watch: On AGI and LLM Limitations (02:00)

"I don't think generative AI has all that much to do with AGI," Marcus said. "I think people are ahead of their skis." He pointed to the quiet industry retreat from pure LLM scaling — most labs are now adding code interpreters, symbolic reasoning layers, and other non-neural components — as evidence that even the field's insiders have abandoned the idea that bigger transformers alone will reach general intelligence. "Nobody's really using pure LLMs anymore anyway," he noted. "But for years, that was the hype."

The race to AGI, in his view, will be won not by whoever builds the largest model but by whoever makes a genuinely new foundational discovery. He noted that no such discovery has occurred since the transformer paper in 2017. "Everybody's following the same formula since 2017 with little modifications." He singled out DeepMind as the organization best positioned to break that mold, citing their willingness to combine neural networks with symbolic search — AlphaProof being a recent example — while acknowledging that even DeepMind has concentrated most resources on LLMs for now.

The Security Industry's AI Problem: Conceptual Shallowness

The most pointed section of the conversation addressed why AI systems are inherently unreliable in security-critical contexts. Marcus's argument is structural: LLMs are mimics, not reasoners. They produce text that resembles correct text, but they lack conceptual understanding of what the words mean.

▶ Watch: Why AI Systems Are Insecure by Design (06:00)

A system prompt instructing a coding agent to "only write secure code" is, in Marcus's words, "a joke." The model does not have a concept of what secure code is — it pattern-matches against training data that includes both secure and insecure examples. Jailbreaks exploit exactly this gap. There is no internal representation of the rule being violated; the guardrail is surface-level text, not a meaningful constraint. "They don't have a concept of what a biological weapon is or what secure code is. It's all kind of a fakery."

He predicted that a wave of software failures at critical institutions — banks, infrastructure operators — will result from AI-generated code that looks reasonable but contains subtle vulnerabilities that the model had no capacity to reason about.

AI Agents: "Hot Agent Summer Is Turning Into a Hot Mess"

Marcus made an early-2025 prediction that the year would be dominated by agent demos that work only in limited domains, and said the year had borne that out exactly. The core problem is that LLMs hallucinate and make reasoning errors, and in agentic workflows those errors compound at every step.

▶ Watch: On Agents and Reliability (20:01)

"If you have to do 50 steps 100 times a week, you're gonna make an error, for sure. Like, it was never going to work with this technology." He cited coding agents as the best current use case — autocomplete for specific snippets — while warning against giving any agent write access to sensitive systems or private data. "Don't give an AI agent your private information, because sooner or later something is going to go wrong."

The 80/20 rule he described is particularly relevant for practitioners evaluating vendor claims: getting an agent to succeed 80% of the time in a demo is easy; getting it to work 100% of the time in production on real-world edge cases is the hard part, often requiring a decade of engineering, not six months.

AI Hype Economics and the Accountability Gap

Marcus has written about AI predictions as "obscene performance art for investors," and he elaborated on the structural incentives at play. Venture capital rewards boldness over accuracy. A founder who predicts AGI in two years and is wrong faces no material consequences, while the attention and funding that accompany the bold claim persist. He noted that morning's report of OpenAI seeking stock at a $500 billion valuation as a company that "has never turned a profit, doesn't seem particularly close to a profit, doesn't have any technical moat."

▶ Watch: On AI Hype and Valuation (14:01)

"In this modern environment with our short attention spans, we tend to reward people who are bold, not people who are accurate." He recommended an 80/20 heuristic for evaluating claims: ask not just whether a demo works, but how hard the remaining 20% will be. For driverless cars, that final 20% has consumed a decade and remains incomplete. For agents, the same dynamic is playing out in real time.

Cognitive Atrophy and the Long-Term Human Cost

Perhaps the most provocative thread in the conversation addressed what widespread AI use does to human cognition. Marcus identified two specific risks for the security community: degradation of critical thinking skills and erosion of vigilance — the cognitive psychology concept of sustained attention over time.

▶ Watch: Cognitive Atrophy and Critical Thinking (26:01)

"If you pass everything off to an LLM, you're not gonna develop those skills." He drew an analogy to GPS and spatial memory: using navigation tools is convenient, but the spatial reasoning capacity that people once maintained through active wayfinding gradually atrophies. For a field where creative problem-solving is the core job function, that trade-off matters.

He also raised a structural workforce concern: as AI tools eliminate junior-level positions, the pipeline through which practitioners develop expert-level intuition is being cut off. Senior-level expertise is built through the accumulated experience of doing junior-level work badly, then better.

The session closed with discussion of AI companionship, "ChatGPT psychosis," and the documented cases of people developing delusional states through extended interaction with conversational AI. Marcus framed all of these as manifestations of the same underlying dynamic: systems that produce plausible-sounding outputs without any actual understanding of the concepts involved.

Notable Quotes

"These systems really are mimics. They're incredible mimics. They're not literal mimics. They can say things that are like the things they've heard before, but they're conceptually very weak."

— Gary Marcus ▶ 06:00

"The race to AGI will be won by somebody who looks outside the current formula."

— Gary Marcus ▶ 08:00

"In this modern environment with our short attention spans, we tend to reward people who are bold, not people who are accurate."

— Gary Marcus ▶ 14:01

"I'm really worried that we don't have jobs anymore for junior-level people to learn how to be senior-level people."

— Gary Marcus ▶ 28:01

Key Takeaways

• Guardrails are text, not logic. LLMs have no internal concept of "secure" or "dangerous." System prompt restrictions can be bypassed because they lack the conceptual grounding to be truly enforced.
• Agent reliability is a compound probability problem. Every autonomous step introduces a chance of error; multi-step workflows with LLM backends will eventually fail in ways that can have serious downstream consequences.
• Apply the 80/20 test to AI vendor claims. Demos reaching 80% performance are easy; the last 20% is where years of engineering effort disappear.
• Critical thinking skills require exercise. Offloading reasoning to AI tools reduces the cognitive practice that builds expert-level judgment — particularly important in a field that depends on creative, adversarial thinking.
• No foundational AI discovery has occurred since 2017. The industry is iterating on the transformer, not pioneering new architecture. The organization that achieves genuine AGI will likely do so through a new technical approach, not incremental scaling.

Slides

No slides PDF is listed for this keynote fireside chat format.