When the well is dry, we know the worth of water: A firsthand account of a DEFCON Franklin volunteer

Tim Pappa (Walmart Global Tech)

Critical Effect DC 2025 · Day 1 · Track II: Strategic Effect

Overview

In "When the well is dry, we know the worth of water," Tim Pappa delivers a compelling and uniquely human perspective on the challenges of securing critical local infrastructure through volunteer initiatives. Drawing on his extensive background in behavioral analysis and cyber deception, Pappa offers a candid ethnographic account of his first six months volunteering with DEFCON Franklin, a program aimed at bolstering the cybersecurity posture of small, often under-resourced, public utilities. The talk transcends typical technical vulnerability assessments, instead focusing on the intricate dynamics of communication, trust, and human behavior that ultimately dictate the success or failure of cybersecurity interventions at the local level.

Watch on YouTube

Visual summary for When the well is dry, we know the worth of water: A firsthand account of a DEFCON Franklin volunteer by Tim Pappa
Visual summary for When the well is dry, we know the worth of water: A firsthand account of a DEFCON Franklin volunteer by Tim Pappa

Key moments

  1. 0:00 Introduction: FBI profiler's journey to cyber deception
  2. 1:04 Cyber deception as a defense for local infrastructure
  3. 4:10 The stark reality of a small town utility's IT
  4. 7:50 Firsthand view of critical infrastructure OT controls
  5. 10:50 A targeted BEC attempt reveals deep threat actor research
  6. 12:35 Why cyber deception remains untouched due to risks
  7. 16:15 Future workshop plans: bringing in a cybercrime felon

When the well is dry, we know the worth of water: A firsthand account of a DEFCON Franklin volunteer

Speakers: Tim Pappa (Walmart Global Tech)

Conference: Critical Effect DC 2025

YouTube: https://www.youtube.com/watch?v=1N5SpSAZMXw

Overview

In "When the well is dry, we know the worth of water," Tim Pappa delivers a compelling and uniquely human perspective on the challenges of securing critical local infrastructure through volunteer initiatives. Drawing on his extensive background in behavioral analysis and cyber deception, Pappa offers a candid ethnographic account of his first six months volunteering with DEFCON Franklin, a program aimed at bolstering the cybersecurity posture of small, often under-resourced, public utilities. The talk transcends typical technical vulnerability assessments, instead focusing on the intricate dynamics of communication, trust, and human behavior that ultimately dictate the success or failure of cybersecurity interventions at the local level.

Pappa, a former FBI profiler specializing in online influence and cyber deception, approaches his volunteer experience as an "ethnography of communication," meticulously observing how threats are discussed, how decisions are made, and how relationships shape security outcomes. His insights reveal a stark reality: while technical solutions exist, the primary impediments to strengthening local infrastructure often lie in deeply entrenched organizational norms, resource constraints, and a critical lack of awareness regarding actual threat landscapes. This talk is crucial for anyone involved in cybersecurity outreach, volunteer programs, or policy-making for critical infrastructure, offering invaluable lessons on the practical, social, and psychological hurdles that must be overcome to achieve meaningful security improvements.

Background

▶ Watch: Introduction: FBI profiler's journey to cyber deception (0:00)

Tim Pappa's journey into the realm of local infrastructure cybersecurity is rooted in a distinguished career blending law enforcement, behavioral science, and cyber operations. For 16 years, he served as an FBI agent, with the last six years as a profiler in the FBI's Behavioral Analysis Unit (BAU). In this role, Pappa focused on understanding the motivations and methods of hackers, particularly in the context of online influence, strategic communication, and cyber deception. This unique lens led him to ponder how these principles could be applied to protect vulnerable local and state infrastructure.

The impetus for his current research and volunteer work crystallized in 2020, following a significant ransomware attack on Louisiana school districts. At the time, Pappa, still with the BAU, observed the widespread impact and began to question why such networks were being compromised. His initial query to the FBI and CISA—whether it was possible to create the impression that networks were stronger than they actually were, leveraging cyber deception—was met with skepticism, based on the premise that "access brokers have ground truth." However, Pappa countered this by citing research indicating that even publicly acknowledging the presence of deception can effectively influence attacker decision-making, slow them down, and even impact them physiologically. He saw an opportunity to apply these principles to the often-neglected domain of state and local government infrastructure.

Currently, Pappa works with Walmart's cyber deception operations team, a role that shares conceptual parallels with his FBI experience. His personal research continues to explore contributions to infrastructure resilience. His chosen methodology for understanding the DEFCON Franklin volunteer experience is an ethnography of communication, a qualitative research approach pioneered by scholars like Duff over 20 years ago. This method focuses on how people communicate about threats, their actions, and their perceptions, aiming to unravel the complex, naturally occurring interactions that shape outcomes. Pappa's goal is to observe and understand the communication dynamics between himself, the water utility representative, and the DEFCON Franklin organizers, recognizing that human relationships and nuanced interactions often complicate, rather than simplify, cybersecurity challenges. This approach acknowledges that perceived realities can be as powerful as actual realities, especially when considering the decision-making processes of both defenders and attackers.

Key Findings

▶ Watch: The stark reality of a small town utility's IT (4:10)

Pappa's ethnographic study of his six months volunteering with DEFCON Franklin yielded several critical insights into the real-world challenges of securing small, local critical infrastructure:

  1. Profound Vulnerability of Small Utilities: The water utility Pappa was paired with served a small ski community of just 500 people and had only one individual responsible for managing its entire network. This extreme resource constraint is typical for many small municipalities and highlights an inherent, pervasive vulnerability that national-level studies have consistently identified: local governments often do not manage cybersecurity effectively.
  1. Dangerous Misconceptions of Threat: The network defender expressed a common sentiment: "We're a pretty small town... we don't really matter to anyone in Russia or Iran." Pappa unequivocally refutes this, noting the prevalence of attacks on water utilities and, strikingly, the fact that a famous Russian defector actually lives in the town. This specific detail underscores that even seemingly insignificant targets can become strategically relevant, challenging the notion that small size equates to security through obscurity.
  1. Mismatched Expectations in Volunteer Programs: Despite Pappa explicitly stating his focus on cyber deception and behavior, not technical implementation, he was selected for the program. The network defender later revealed that Pappa's government background (FBI) was a key appeal, believing he "knew something if someone took over our system." This created an immediate misalignment, as Pappa felt uncomfortable providing direct technical guidance, recognizing his own limitations and the defender's misperceptions of his role.
  1. Stuck at Tier 1: The Inertia of Basic Security: After six months, the utility had not progressed beyond Tier 1 of DEFCON Franklin's suggested vulnerability remediation framework. This tier includes fundamental actions like changing default passwords and enabling multi-factor authentication (MFA). Pappa attributed this delay not to negligence, but to the inherent difficulty, organizational norms, and existing relationships that make even basic changes challenging. He found himself in "uncomfortable conversations" navigating these entrenched practices.
  1. Direct Exposure of Operational Technology (OT): In a particularly eye-opening moment, the network defender signed into his Operational Technology (OT) system and turned his screen around during a Zoom call, allowing Pappa to observe real-time manipulation of chlorine levels. This starkly illustrated the direct, unfiltered access to critical industrial controls that often exists in these environments, confirming the grave concerns surrounding OT security.
  1. Lack of Volunteer Program Transparency: Pappa expressed significant frustration at having "no idea what's going on with other volunteers." He lacked visibility into their experiences, challenges, or successes, limiting opportunities for shared learning, best practices, and a collective understanding of the program's overall impact or direction. He questioned his own "real and imagined role" within the broader DEFCON Franklin initiative.
  1. Impact of Personalities and Trust: Pappa observed that communication became "a little more comfortable" and effective when DEFCON Franklin representatives were not on the call, fostering direct, uninhibited dialogue between himself and the network defender. He noted that the defender "already knows what he actually needs, but he's looking for help on what he needs," highlighting the critical role of trust and a supportive, non-judgmental relationship.
  1. Targeted Business Email Compromise (BEC) Attempt: A business email compromise (BEC) attempt occurred a year prior, involving a spear-phishing email spoofing a planning commission member. Pappa noted the sophisticated reconnaissance implied by this attack, especially given that only one ethnic Chinese person lived in the town and the spoofed email mimicked that individual's role. This incident was not initially reported to the Internet Crimes Complaint Center (IC3), demonstrating a significant gap in incident response and reporting. Pappa eventually submitted it on the utility's behalf, underscoring the need for external guidance even on basic security hygiene.
  1. Incrementalism as a Path Forward: Pappa concluded that changes were occurring incrementally, often requiring gentle nudges rather than forceful directives. This slow, relationship-dependent progress, while perhaps frustrating, appeared to be the only practical way to effect change within the existing organizational culture and resource constraints of the utility.

Technical Deep Dive

▶ Watch: Firsthand view of critical infrastructure OT controls (7:50)

While Tim Pappa's talk focuses heavily on the human and organizational aspects of cybersecurity, it implicitly reveals critical technical vulnerabilities and challenges prevalent in small, under-resourced critical infrastructure environments. The "technical deep dive" here is less about exploiting specific CVEs or analyzing malware, and more about understanding the types of technical weaknesses exposed and the systemic difficulties in addressing them.

The most fundamental technical issues highlighted revolve around basic cyber hygiene, which Pappa refers to as "Tier 1" of DEFCON Franklin's remediation framework. This includes:

  • Default Passwords: The utility had not changed critical passwords for many years, and these did not meet modern complexity requirements. This is a perennial vulnerability that attackers frequently exploit for initial access.
  • Lack of Multi-Factor Authentication (MFA): The absence of MFA means that a stolen or guessed password provides full access, significantly increasing the risk of compromise. Implementing MFA is a cornerstone of modern identity and access management.

Beyond these foundational issues, the talk uncovered more sophisticated vectors and profound architectural risks:

  • Operational Technology (OT) Exposure: The incident where the network defender openly logged into and manipulated chlorine levels on the OT system via Zoom was a stark revelation. This demonstrated a critical lack of segmentation and potentially direct internet exposure for industrial control systems (ICS). In many small utilities, IT and OT networks are not adequately separated, or OT systems are managed with IT tools and practices that are insufficient for their unique security requirements. Direct manipulation of physical processes (like chlorine levels) underscores the severe physical and public safety implications of such compromises.
  • Supply Chain Risk via Remote Access: Pappa identified a significant vulnerability stemming from the supplier of the OT technology having remote access capabilities for updates and maintenance. While necessary for support, this creates a potential backdoor into critical systems. The challenge of pentesting this specific aspect indicates a lack of clear security agreements or understanding of the attack surface introduced by third-party access. Such remote access, if not properly secured with strong authentication, least privilege, and continuous monitoring, can be a major entry point for sophisticated adversaries.
  • Targeted Business Email Compromise (BEC) and Spear-Phishing: The BEC attempt targeting a planning commission member revealed a level of reconnaissance that contradicts the utility's perception of irrelevance. Attackers performed open-source intelligence (OSINT) to identify key personnel and even demographic details (the "one ethnic Chinese person living in this town") to craft a highly believable spear-phishing email. This demonstrates that even if network perimeter defenses are strong, human factors remain a critical vulnerability, requiring sophisticated social engineering defenses and user awareness training. The failure to report this incident to the Internet Crimes Complaint Center (IC3) also highlights a systemic technical and procedural gap in incident response, preventing law enforcement from aggregating intelligence and preventing future attacks.

The "technical deep dive" in this context also extends to the process of implementing technical solutions. Pappa's experience demonstrates that even with a volunteer offering assistance, the actual implementation of technical fixes is hindered by:

  • Resource Constraints: One IT person cannot dedicate sufficient time to proactive security.
  • Organizational Inertia: Existing "norms" and relationships within the utility (e.g., between the network defender and his boss regarding password changes) can override technical best practices.
  • Lack of Expertise/Guidance: The network defender, while knowing he "needs help," lacks the specialized cybersecurity expertise to identify and prioritize technical solutions, or even to navigate the process of engaging external testers for complex issues like supplier remote access.

In essence, the talk illustrates that the technical vulnerabilities are often well-known and basic, but the socio-technical environment—the interplay of human behavior, organizational structure, and resource limitations—creates an impenetrable barrier to their remediation.

Demo / Proof of Concept

▶ Watch: Why cyber deception remains untouched due to risks (12:35)

While Tim Pappa's talk did not feature a traditional, controlled "demo" of a security exploit or a proof-of-concept tool, it contained a vivid and unsettling account that served as an unplanned, real-time "demonstration" of critical infrastructure vulnerability.

During one of his monthly calls with the water utility's network defender, Pappa described an "eye-opening" and "uncomfortable" moment: the defender logged into his Operational Technology (OT) system and, without prompting, turned his screen around to show Pappa the interface. Pappa watched as the defender manipulated controls, specifically demonstrating the ability to change chlorine levels in the water supply.

This impromptu display was a powerful, albeit unintentional, proof of concept for the ease of access and control over critical infrastructure systems. For Pappa, who had previously worked on the theoretical aspects of cyber deception and the vulnerabilities of local infrastructure, this was a "first experience." It brought into sharp focus the very real, physical consequences of compromised OT systems that cybersecurity professionals frequently warn about. The ease with which the defender could showcase this manipulation underscored the potential impact if a malicious actor gained similar access, whether through basic password compromise, exploited remote access channels, or sophisticated spear-phishing. It served as a stark, visceral reminder of the "what everyone's so worried about" scenario in critical infrastructure security.

Defensive Implications

▶ Watch: Future workshop plans: bringing in a cybercrime felon (16:15)

The insights gleaned from Tim Pappa's experience with DEFCON Franklin offer crucial defensive implications for small critical infrastructure operators, volunteer cybersecurity programs, and policymakers alike:

  1. Prioritize the Fundamentals ("Tier 1"): The fact that a utility was still struggling with basic measures like changing default passwords and enabling Multi-Factor Authentication (MFA) after six months highlights a systemic problem. Defenders, especially in under-resourced environments, must ruthlessly prioritize these foundational security controls. External programs should recognize the significant effort required for even "basic" changes and tailor their support accordingly.
  1. Challenge the "We Don't Matter" Mindset: The perception that small towns or utilities are too insignificant to be targeted is a dangerous fallacy. Attackers, whether opportunistic or state-sponsored, conduct reconnaissance and will exploit any vulnerability. The presence of a "famous Russian defector" in the town, or the targeted nature of the Business Email Compromise (BEC) attempt, demonstrates that perceived irrelevance offers no true security. Defenders must operate with the assumption that they are targets.
  1. Proactive Incident Reporting is Critical: The failure to report the BEC attempt to the Internet Crimes Complaint Center (IC3) is a significant gap. Small organizations often lack the awareness or procedures for incident reporting. Defenders must establish clear, actionable plans for reporting any suspicious activity or confirmed breaches to relevant authorities. This not only aids law enforcement but also contributes to a broader understanding of the threat landscape.
  1. Scrutinize and Secure OT/IT Interconnections and Remote Access: The remote access capabilities of OT suppliers, combined with the direct demonstration of OT system control, underscores a major vulnerability. Defenders need to rigorously assess all supply chain risks, particularly those involving remote access to Operational Technology (OT). This includes implementing robust access controls, strong authentication, network segmentation, and independent security audits (e.g., penetration testing) of third-party connections.
  1. Build Trust and Embrace Incrementalism: Volunteer programs and external consultants must adopt an empathetic, trust-building approach. Pappa's experience shows that direct, authoritative guidance can be counterproductive. Instead, fostering comfortable relationships, understanding existing organizational norms, and advocating for incremental changes are more effective strategies for long-term security improvement. Change is often slow, but consistent, gentle pressure can yield results.
  1. Foster Community and Shared Learning: Pappa's lack of visibility into other volunteers' experiences within DEFCON Franklin points to a missed opportunity. Volunteer programs should facilitate platforms for shared learning, peer support, and aggregated intelligence among volunteers and participating utilities. The idea of a one-day workshop for other water utilities, potentially featuring a cyber crime felon to provide attacker perspectives, is an excellent model for community-driven defense.
  1. Address the Human Element and Training: The success of even basic security measures hinges on the human factor. Regular, context-specific cybersecurity awareness training for all staff, from IT managers to administrative personnel, is crucial to combat threats like spear-phishing. Training should not just focus on what to do, but why it matters, using real-world examples relevant to their specific operations.
  1. Define Boundaries and Roles Clearly: For both volunteers and the organizations they assist, clearly defined roles, responsibilities, and boundaries of engagement are essential to manage expectations and ensure sustainable partnerships. Pappa's discomfort in giving direct guidance highlights the need for volunteer programs to provide guidelines and support for their volunteers in navigating these complex relationships.

Key Takeaways

  • Small, under-resourced critical infrastructure entities, like the water utility in this talk, are profoundly vulnerable to both basic and sophisticated cyberattacks, often due to a single IT manager and limited resources.
  • The belief that small towns "don't matter" to foreign adversaries is a dangerous misconception; sophisticated threat actors conduct reconnaissance and target entities based on strategic value or opportunistic vulnerabilities, regardless of size.
  • Implementing fundamental cybersecurity measures (e.g., changing default passwords, enabling MFA) is significantly more challenging and slower in practice than anticipated, often due to deeply entrenched organizational norms, existing relationships, and resource constraints.
  • Operational Technology (OT) systems in critical infrastructure may have dangerously direct access and exposure, as demonstrated by the real-time manipulation of chlorine levels, posing severe physical and public safety risks.
  • Effective volunteer-based cybersecurity assistance requires an ethnographic approach focused on building trust, understanding local communication dynamics, and advocating for incremental changes rather than imposing radical overhauls.
  • Proactive incident reporting (e.g., to the Internet Crimes Complaint Center) is a critical, often overlooked, defensive measure that small organizations must integrate into their security posture.
  • Volunteer programs like DEFCON Franklin could benefit significantly from improved internal communication and shared learning among volunteers to foster collective intelligence and best practices.

About the Speaker(s)

Tim Pappa is a cybersecurity professional with a unique and extensive background spanning law enforcement, behavioral science, and cyber operations. He served for 16 years as an FBI agent, with the last six years dedicated to the FBI's Behavioral Analysis Unit (BAU). In the BAU, Pappa specialized in profiling hackers and focusing on online influence, strategic communication, and cyber deception. His work at the FBI involved understanding the psychological and behavioral aspects of cyber threats. Currently, Pappa is part of the cyber deception operations team at Walmart Global Tech, where he continues to apply his expertise in influencing adversary behavior and strengthening digital defenses. His research interests remain centered on finding innovative ways to contribute to the resilience of critical infrastructure, particularly at the local and state levels.

Reviews

Dr. Zero (Offensive Security Researcher) — SOLID

Tim Pappa's post-mortem on volunteering for DEFCON Franklin is a brutal, necessary dose of reality about securing local critical infrastructure. It proves that dropping an ex-FBI profiler into a local water utility doesn't matter if the solo admin still won't change default passwords. It's a harsh wake-up call for the hacker volunteer crowd.

Heather Calloway (CISO) — SOLID

Pappa delivers a necessary, unsentimental look at why local critical infrastructure remains indefensible despite available technical solutions. By treating cybersecurity as an organizational and behavioral problem, he exposes the structural realities at the bottom of our supply chains. It’s required viewing for anyone involved in infrastructure policy, vendor risk management, or national cyber strategy.

→ Top-rated talks at Critical Effect DC 2025

All talks from Critical Effect DC 2025