AI-in-the-Loop: Privacy Preserving Real-Time Scam Detection and Conversational Scambaiting by Leveraging LLMs and Federated Learning

Ismail Hossain (Paper author · University of Texas at El Paso), Sai Puppala (Paper author · Southern Illinois University Carbondale), Md Jahangir Alam (Paper author · University of Texas at El Paso), Sajedul Talukder (Paper author · University of Texas at El Paso)

Privacy Enhancing Technologies Symposium (PETS) 2026 · Day 2 · Moose

Overview

In an increasingly interconnected digital world, the pervasive threat of online scams continues to victimize individuals across social media and messaging platforms. The "AI-in-the-Loop" talk, presented by Ismail Hossain at PETS, introduced a groundbreaking approach to combat this challenge: a privacy-preserving real-time scam detection and conversational scambaiting system that intelligently leverages Large Language Models (LLMs) and Federated Learning. This research proposes a proactive digital guardian designed not only to identify potential scams in live conversations but also to actively engage and disrupt scammers, all while meticulously safeguarding user privacy.

Watch on YouTube · Slides

Visual summary for AI-in-the-Loop: Privacy Preserving Real-Time Scam Detection and Conversational Scambaiting by Leveraging LLMs and Federated Learning by Ismail Hossain, Sai Puppala, Md Jahangir Alam, Sajedul Talukder
Visual summary for AI-in-the-Loop: Privacy Preserving Real-Time Scam Detection and Conversational Scambaiting by Leveraging LLMs and Federated Learning by Ismail Hossain, Sai Puppala, Md Jahangir Alam, Sajedul Talukder

Key moments

  1. 0:00 Introduction: The scam problem and research questions
  2. 2:50 AI-in-the-Loop system: Detection, warning, and engagement
  3. 4:20 Federated learning for privacy-preserving, distributed model updates
  4. 5:25 Why LLMs are chosen for multi-task scam detection
  5. 6:30 Human evaluation confirms AI's effective and engaging responses
  6. 6:50 Ensuring model privacy with differential privacy in federated learning
  7. 8:15 Key takeaways: proactive defense, privacy, and future work

AI-in-the-Loop: Privacy Preserving Real-Time Scam Detection and Conversational Scambaiting by Leveraging LLMs and Federated Learning

Speakers: Ismail Hossain (Paper author, University of Texas at El Paso), Sai Puppala (Paper author, Southern Illinois University Carbondale), Md Jahangir Alam (Paper author, University of Texas at El Paso), Sajedul Talukder (Paper author, University of Texas at El Paso)

Conference: PETS

YouTube: https://www.youtube.com/watch?v=gV9qp2ZxKUY

Overview

In an increasingly interconnected digital world, the pervasive threat of online scams continues to victimize individuals across social media and messaging platforms. The "AI-in-the-Loop" talk, presented by Ismail Hossain at PETS, introduced a groundbreaking approach to combat this challenge: a privacy-preserving real-time scam detection and conversational scambaiting system that intelligently leverages Large Language Models (LLMs) and Federated Learning. This research proposes a proactive digital guardian designed not only to identify potential scams in live conversations but also to actively engage and disrupt scammers, all while meticulously safeguarding user privacy.

The core innovation lies in its ability to operate autonomously and learn continuously without centralizing sensitive user data, addressing critical limitations of traditional reactive security measures. By integrating advanced AI for detection and response generation with a distributed learning paradigm, the system aims to empower users with an unparalleled defense mechanism. The talk delved into the system's architecture, its fine-tuned LLMs, the role of federated learning in continuous improvement, and the rigorous evaluation demonstrating its efficacy, realism, and privacy guarantees.

This work is particularly significant for its dual focus on proactive prevention and privacy. It moves beyond mere detection to active disruption through AI-driven scambaiting, transforming a passive defense into an active countermeasure. Furthermore, its commitment to federated learning and differential privacy sets a new standard for how AI can be deployed in sensitive security contexts, ensuring that collective intelligence can be harnessed to fight evolving threats without compromising individual user data.

Background

▶ Watch: Introduction: The scam problem and research questions (0:00)

The digital landscape, particularly social media and messaging applications, has become a fertile ground for scammers. Individuals, often unknowingly, leave extensive digital footprints through their posts, comments, and interactions, making them susceptible targets for social engineering attacks. Scammers exploit these vulnerabilities, reaching out through messaging apps with deceptive offers—like the apartment rental scam scenario described by the speaker—leading to significant financial and emotional distress. The problem is global and ubiquitous, necessitating a robust and intelligent defense.

Existing solutions often fall short in several key areas, creating significant research gaps that this work aims to address. Firstly, there's a notable absence of smart, proactive systems capable of simultaneously detecting and preventing scams during live, ongoing conversations. Most current defenses are reactive, flagging scams after the damage is done, or rely on static blacklists that quickly become outdated. Secondly, while AI has shown promise in detection, engaging scammers in real-time conversation presents complex challenges related to minimizing risk to the user and, crucially, preserving user privacy. An AI system must be able to generate convincing, engaging responses without inadvertently revealing sensitive user information. Finally, the dynamic nature of scams means that any effective defense mechanism must be capable of continuous learning and adaptation to new scam scenarios and evolving conversational patterns. Traditional centralized AI models often struggle with this, requiring constant retraining on potentially sensitive aggregated data.

To tackle these challenges, the research posed three fundamental questions:

  1. Can a system detect and prevent a scam simultaneously during a live conversation?
  2. Can AI engage a scammer in real-time while minimizing risk and preserving privacy?
  3. Can AI be updated by learning new scam scenarios or scammer conversational patterns?

The overarching objective of the "AI-in-the-Loop" system is therefore to build a proactive, privacy-preserving AI system that detects, disrupts, and learns from scammer interactions in real-time, crucially without centralizing any sensitive user data. This objective directly addresses the identified research gaps by integrating real-time detection, AI-driven conversational scambaiting, and a distributed learning framework.

Key Findings

▶ Watch: Federated learning for privacy-preserving, distributed model updates (4:20)

The "AI-in-the-Loop" research yielded several significant findings that collectively advance the state-of-the-art in proactive scam defense and privacy-preserving AI.

Firstly, the proposed system effectively demonstrates the feasibility of a proactive defense system that integrates real-time scam detection with active, conversational scambaiting. By fine-tuning Large Language Models (LLMs) with conversational scam data, the system can not only identify potential scams but also generate engaging and disruptive responses, moving beyond mere alerts to active prevention.

Secondly, the study confirmed the multi-tasking capabilities of LLMs, particularly open-source safeguard models like LlamaGuard variants, for this complex domain. These models were successfully adapted to perform several critical functions concurrently: accurate scam detection, calculation of a Personal Identifiable Information (PII) score to assess privacy risk, and an engagement score to evaluate the effectiveness of AI-generated responses. This holistic approach ensures that AI responses are both safe and effective.

Thirdly, human evaluation played a crucial role in validating the system's performance. Assessments of 100 random conversations confirmed that the AI-generated responses exhibited high levels of realism, engagement, and effectiveness, indicating that the scambaiting aspect of the system is practical and convincing enough to disrupt scammer operations.

Finally, the research emphatically demonstrated the critical role and effectiveness of federated learning in enabling continuous model updates while rigorously preserving user privacy. By allowing local models to train on device-specific data and only sharing aggregated model weights (not raw data) with a global model, the system can adapt to new scam patterns globally. The integration of differential privacy (DP) further strengthens this by protecting the shared model weights from inference attacks, ensuring that individual contributions cannot be reverse-engineered. This distributed, privacy-centric learning mechanism is vital for maintaining the system's relevance against evolving threats.

Technical Deep Dive

▶ Watch: Why LLMs are chosen for multi-task scam detection (5:25)

The "AI-in-the-Loop" system is architecturally designed as a sophisticated, multi-component framework, seamlessly integrating real-time detection, intelligent response generation, and a continuous learning mechanism. Its core strength lies in leveraging advanced AI techniques—specifically Large Language Models (LLMs) and Federated Learning—to achieve its proactive and privacy-preserving objectives.

At the heart of the system is the scam detection module. This module utilizes fine-tuned AI models, primarily LLMs, which have been trained on extensive conversational scam datasets. When a user engages in a conversation via a messaging app, the system continuously monitors the dialogue. For each pair of messages, it calculates a scam risk score. This score is derived by evaluating the terms and context of the messages, and a weighted average is then computed. If this average score surpasses a predefined threshold value, the system flags the conversation as a potential scam and issues a warning to the user. This initial warning system is designed to be non-intrusive, merely alerting the user to a potential threat.

Should the user choose to enable AI assistance, the system transitions into AI engagement mode, initiating conversational scambaiting. In this phase, the AI takes over the interaction with the scammer. The generation of AI responses is governed by a set of sophisticated utility functions. These functions evaluate potential responses based on two primary criteria:

  1. Harmfulness: Assessing the risk that a response might inadvertently reveal sensitive user information or escalate the situation negatively.
  2. Engagement: Measuring how effectively a response can prolong the conversation, gather information about the scammer, or simply waste the scammer's time, thereby disrupting their operation.

The system continuously evaluates generated responses against a privacy threshold. If an AI-generated response is deemed to cross this threshold—for instance, by risking the disclosure of Personal Identifiable Information (PII)—the system is designed to prevent that response from being sent and may even stop the AI from continuing the conversation, ensuring user safety remains paramount.

For the underlying AI models, the researchers experimented with various transformer models for scam detection, noting their strong performance. However, for the multi-tasking requirements of detection, PII risk assessment, engagement scoring, and response generation, LLMs were specifically chosen. The talk highlighted the use of open-source LLMs available on platforms like Hugging Face, specifically mentioning safeguard models such as two variants of LlamaGuard, including "MD gels." These safeguard models are crucial for ensuring that the AI itself is designed with privacy and safety in mind, preventing exploitation.

The system's ability to adapt to new scam techniques is facilitated by Federated Learning (FL). FL provides a distributed machine learning framework where raw user data never leaves the user's device. Instead, each local device trains its AI model using its unique conversational data. After local training, only the model weights (parameters representing the learned patterns) are sent to a global model residing on a central server. This global model then aggregates the weights received from numerous local models, synthesizing a more robust and comprehensive understanding of scam patterns worldwide. The updated global weights are then sent back to the local devices, effectively updating their models with collective intelligence from across the network. This mechanism ensures that if one user encounters a new scam, all participating models can learn from it, significantly enhancing the collective defense without compromising individual privacy.

To further bolster privacy during the federated learning process, the system incorporates Differential Privacy (DP). DP is a technique that adds statistical noise to the model weights before they are sent to the global server. This noise makes it mathematically difficult for an attacker to infer specific information about any individual's data from the aggregated model weights, even if the attacker has access to the global model. The researchers evaluated their system with different DP settings: no DP, 0.1 DP, and 0.2 DP, demonstrating its effectiveness in preserving model weight privacy against potential inference attacks.

The quality of AI-generated responses was rigorously evaluated using several metrics:

  • Perplexity: A measure of how well a probability model predicts a sample. Lower perplexity generally indicates better language modeling.
  • Distinct-1 and Distinct-2: These metrics quantify the diversity of the dialogue by counting the number of unique unigrams and bigrams, respectively. Higher values indicate more diverse and less repetitive responses.
  • Dialogue Ranking Pretend Transformer: This metric assesses the engagement quality of the generated dialogue, ensuring the AI can maintain a convincing and interactive conversation with the scammer.

These technical components—fine-tuned LLMs for multi-tasking, utility-driven response generation with privacy thresholds, and a federated learning framework enhanced by differential privacy—form a cohesive and powerful system for privacy-preserving, real-time scam detection and conversational scambaiting.

Demo / Proof of Concept

▶ Watch: Ensuring model privacy with differential privacy in federated learning (6:50)

While the talk provided a comprehensive architectural overview and detailed evaluation metrics, it did not feature a live, interactive demonstration of the AI-in-the-Loop system actively engaging a scammer in real-time. Instead, the authors focused on presenting the robust evaluation methodology and the empirical results that validate the system's capabilities.

The proof of concept was primarily established through human evaluation of AI-generated responses. The researchers randomly selected and evaluated 100 conversations, assessing the AI's performance across critical dimensions such as realism, engagement, and effectiveness. This human-centric validation, using a 5-point scale, confirmed that the AI's conversational abilities were convincing and appropriate for scambaiting purposes. This approach, while not a live demo, served as a strong empirical proof of concept, demonstrating that the AI could generate high-quality, effective, and safe responses in simulated scam scenarios.

Defensive Implications

▶ Watch: Key takeaways: proactive defense, privacy, and future work (8:15)

The "AI-in-the-Loop" system presents profound defensive implications for individuals, security professionals, and the broader digital ecosystem, offering a significant leap forward in combating online fraud.

For individual users, this technology represents the advent of a truly proactive digital guard. Instead of merely reacting to scams, users are equipped with a real-time warning system that operates directly within their messaging applications. This immediate feedback loop empowers users to make informed decisions about potentially malicious interactions. Furthermore, the option to delegate the conversation to an AI-driven scambaiting agent provides an unprecedented layer of defense, allowing users to disengage from dangerous dialogues while the AI actively disrupts the scammer's efforts. This system significantly reduces the emotional and cognitive burden on individuals, minimizing their direct exposure to manipulative tactics.

For security professionals and developers, the research highlights the immense potential of integrating Large Language Models (LLMs) with federated learning for advanced threat detection and response. It provides a blueprint for building intelligent, adaptive security tools that can operate at the edge, protecting individual privacy while benefiting from collective intelligence. The emphasis on privacy-preserving AI architectures, particularly through differential privacy in federated learning, offers a model for deploying AI in sensitive security contexts where data centralization is a non-starter. This approach can be extended beyond scam detection to other forms of online abuse, harassment, or disinformation campaigns.

The system also has implications for threat intelligence sharing. By allowing models to learn from diverse scam scenarios globally without exchanging raw data, it fosters a more dynamic and responsive threat landscape. Security vendors and platform providers could leverage similar federated learning frameworks to rapidly update their defense mechanisms against emerging threats, making it harder for attackers to exploit new vulnerabilities or social engineering tactics.

However, the deployment of such systems also introduces new considerations. The need for robust PII scoring and engagement metrics within the AI is critical to ensure both user safety and system effectiveness. Future work, as highlighted by the speakers, includes expanding the system to handle phone call scam conversations and multi-modalities (e.g., images, audio), as well as scaling it for real-life deployment. These advancements will require continuous research into ethical AI, bias mitigation, and the delicate balance between privacy preservation and detection accuracy. Ultimately, "AI-in-the-Loop" offers a compelling vision for a future where AI actively protects users from digital threats, moving beyond passive detection to intelligent, privacy-aware defense.

Key Takeaways

  • Proactive, Privacy-Preserving Defense: The "AI-in-the-Loop" system introduces a novel, proactive defense mechanism for real-time scam detection and conversational scambaiting, designed to protect users on social media and messaging platforms.
  • LLMs for Multi-Tasking: The system effectively leverages fine-tuned Large Language Models (LLMs), including safeguard models like LlamaGuard, for simultaneous scam detection, Personal Identifiable Information (PII) risk assessment, engagement scoring, and intelligent response generation.
  • Federated Learning for Continuous Adaptation: Federated learning enables the system to continuously update its models with new scam patterns from diverse users globally, without centralizing sensitive user data, ensuring the AI remains effective against evolving threats.
  • Differential Privacy for Enhanced Security: The integration of differential privacy safeguards the shared model weights during federated learning, protecting against inference attacks and ensuring that individual user data cannot be reconstructed from the aggregated model updates.
  • Validated Effectiveness and Realism: Human evaluation of 100 random conversations confirmed that the AI-generated scambaiting responses demonstrated high levels of realism, engagement, and effectiveness, validating the system's practical utility.
  • Smart Scoring System: The AI employs sophisticated utility functions to evaluate potential responses based on "harmfulness" and "engagement," ensuring that generated dialogues are both safe (preserving privacy) and effective in disrupting scammers.

About the Speaker(s)

The "AI-in-the-Loop" talk was presented by Ismail Hossain, who is also listed as a paper author from the University of Texas at El Paso. He was joined by co-authors Sai Puppala from Southern Illinois University Carbondale, and Md Jahangir Alam and Sajedul Talukder, both also from the University of Texas at El Paso. This collaborative team brings together expertise from different academic institutions to address the critical challenge of online scam detection and prevention through advanced AI and privacy-preserving technologies.

Reviews

Dr. Zero (Offensive Security Researcher) — SOLID

They glued LlamaGuard to a federated learning framework to auto-troll scammers, which is a great premise for burning adversary cycles. But doing it purely in an academic sandbox with zero live demos and a static chat-log evaluation means it's strictly theoretical right now.

Heather Calloway (CISO) — WEAK

This research demonstrates how to use local LLMs and federated learning to detect text-based scams without centralizing sensitive message data. The underlying privacy architecture is highly relevant for securing executive communications and BYOD fleets where we cannot legally monitor traffic. However, the proposed automated 'scambaiting' feature carries unacceptable corporate liability.

→ Top-rated talks at Privacy Enhancing Technologies Symposium (PETS) 2026

All talks from Privacy Enhancing Technologies Symposium (PETS) 2026