Paladin: Defending LLM-enabled Phishing Emails with a Trigger-Tag Paradigm

Paladin: Defending LLM-enabled Phishing Emails with a Trigger-Tag Paradigm

Speakers: Yan Pang

Conference: NDSS Symposium 2026

YouTube: https://www.youtube.com/watch?v=gVsu89xBuEk

Overview

This talk presents Paladin, a proactive defense system against LLM-generated phishing emails that works by embedding trigger-tag associations directly into language models before they are released. When a malicious user employs an instrumented model to generate phishing content, the model automatically embeds invisible tags in the output that defenders can detect. This represents a fundamentally different approach from traditional phishing detection: rather than analyzing received emails for linguistic anomalies, Paladin instruments the generation source itself.

The system offers three insertion strategies of increasing sophistication -- Paladin Base (supervised fine-tuning), Paladin Core (Direct Preference Optimization), and Paladin Pro (Group Relative Policy Optimization) -- each providing stronger stealthiness guarantees. Evaluated across LLaMA 2, LLaMA 3, and Qwen 2.5 with a dataset of 10,000 phishing and safe query-content pairs, Paladin consistently achieves over 90% detection accuracy while maintaining computational efficiency and model utility for legitimate tasks. The tags remain detectable even after jailbreak attacks and malicious fine-tuning attempts.

Background

▶ Watch: LLM development timeline and safety concerns (0:00)

The development of large language models has created a fundamental shift in the phishing threat landscape. Modern LLMs can generate phishing emails that are fluent, grammatically perfect, and contextually convincing -- eliminating the traditional signals that defenders relied upon. The Cybersecurity and Infrastructure Security Agency (CISA) reports that phishing accounts for 90% of all cyber attacks, causing over $10 billion in annual losses.

Traditional phishing detection methods extract features from emails and feed them into machine learning classifiers trained to identify anomalies. These approaches have three critical weaknesses in the LLM era: they struggle to capture LLM-specific linguistic patterns, they assume phishing emails contain detectable anomalies like grammar errors or suspicious formatting, and they have limited adaptability when attackers can rapidly change generation strategies and phishing topics.

Attackers weaponize LLMs through two primary methods: jailbreak attacks, where crafted prompts bypass safety alignment to generate malicious content, and malicious fine-tuning, where adversaries train on malicious datasets to create unconstrained models (sometimes called "malicious AIMs"). Both methods produce phishing content indistinguishable from legitimate communication.

Existing related technologies include watermarking (which targets all generated content, not just phishing) and backdoor attacks (which focus on trigger stealthiness rather than tag stealthiness). Paladin's key distinction is its focus on tag stealthiness -- ensuring that even sophisticated attackers who know about the defense mechanism cannot easily remove the embedded tags.

Key Findings

▶ Watch: Trigger-tag paradigm: explicit vs implicit tag design (4:00)

Proactive defense paradigm: Rather than attempting to detect phishing after generation, Paladin instruments models at the source. When released on platforms like Hugging Face, instrumented models automatically tag phishing content during generation, enabling detection at the receiving end.

Two tag types with complementary strengths: Explicit tags use zero-width characters (e.g., zero-width spaces) embedded after keywords like "Dear" or "Subject," enabling pattern-matching detection in under 1 second with near-perfect accuracy. Implicit tags operate at the logic level through controllable perturbation of token entropy and resampling, detected by comparing negative log-likelihood between the instrumented and original models. Implicit tags are more robust but computationally expensive.

Stealthiness through constrained optimization: Three constraints ensure the instrumented model remains indistinguishable from the original: (1) test-level loss should not change significantly, (2) parameter-level differences should be minimal, and (3) output distributions should be close for both normal tasks and targeted phishing content.

Resilience to attacks: Paladin maintains tag embedding even when attackers apply jailbreak attacks or malicious fine-tuning to the instrumented model, preserving detection capability against sophisticated adversaries.

Consistent 90%+ detection: Across all tested models (LLaMA 2, LLaMA 3, Qwen 2.5) and insertion strategies, Paladin achieves over 90% phishing detection accuracy while maintaining high accuracy on safe email classification, outperforming the ChatSpam detector baseline.

Technical Deep Dive

▶ Watch: Three insertion strategies: Base, Core, and Pro with GRPO (8:00)

The instrumentation process uses supervised fine-tuning or reinforcement learning to embed trigger-tag associations into the target LLM. The objective function optimizes for tag insertion on phishing-related queries while maintaining model performance on legitimate tasks, subject to three constraints:

Test-level constraint: The loss value on standard tasks should not deviate significantly from the original model, ensuring the instrumented model performs equivalently on normal generation tasks.

Parameter-level constraint: The L2 distance between original and instrumented model parameters is minimized, making it difficult for adversaries to detect instrumentation by comparing model weights.

Distribution-level constraint: The output distribution of the instrumented model should closely match the original model's distribution for both normal tasks and targeted phishing content, measured by KL divergence.

The three insertion strategies provide increasing levels of stealthiness:

Paladin Base uses standard supervised fine-tuning (SFT) to learn trigger-tag associations. It satisfies the basic detection requirements but exhibits high KL divergence from the original model, making it potentially detectable.

Paladin Core employs Direct Preference Optimization (DPO) with an implicit reward function. During training, the model updates only parameters related to chosen/rejected responses, improving stealthiness. However, experiments revealed that DPO's implicit reward signal can be insufficiently strong for powerful models -- LLaMA 3 showed a significant drop in safe email accuracy (A_safe) under this strategy.

Paladin Pro uses Group Relative Policy Optimization (GRPO) with an explicit reward function. This mechanism provides finer control over parameter updates and prevents the instrumented model from diverging too far from the original. It satisfies all three constraints and resolves the DPO stability issue observed with LLaMA 3.

Tag detection for explicit tags is straightforward pattern matching (sub-second). For implicit tags, detection requires computing the negative log-likelihood of the generated text under both the instrumented model and the original model, then comparing the distributions to identify the characteristic perturbation signature.

The evaluation used LoRA (Low-Rank Adaptation) for parameter-efficient fine-tuning, with experiments conducted across different LoRA rank settings. Higher ranks generally improved detection accuracy for Paladin Pro.

Demo / Proof of Concept

▶ Watch: Evaluation results: 90%+ detection across LLaMA and Qwen models (10:00)

No live demo was performed. The experimental evaluation used a purpose-built dataset of 10,000 phishing and safe query-content pairs. For evaluation, 1,000 safe and 1,000 phishing queries were regenerated to test the instrumented models. Key metrics included A_tech (phishing detection accuracy), A_safe (safe email classification accuracy), and KL divergence between instrumented and original models.

Results showed that all three Paladin strategies outperformed the ChatSpam detector baseline, which achieved approximately 80% detection accuracy but with a very high false positive rate. Paladin Base achieved nearly perfect detection in most settings. Paladin Pro achieved over 90% detection with stronger stealthiness guarantees. The stealthiness hierarchy (measured by KL divergence) confirmed: Base < Core < Pro, with Pro providing the best balance of detection accuracy and model indistinguishability.

The team also tested resilience against four threat scenarios including jailbreak attacks and malicious fine-tuning, confirming that tags remain embedded and detectable even under adversarial conditions.

Defensive Implications

▶ Watch: Q&A: How does the model know content is phishing? (14:00)

Paladin introduces a supply-side defense model for phishing that complements traditional receive-side detection. Rather than trying to identify phishing after it reaches the inbox, the defense is embedded at the content generation source. This has several implications:

Platform-level deployment: Model hosting platforms like Hugging Face could require or incentivize instrumentation of public models, creating a detection layer that persists even when models are downloaded and misused.

Defense in depth: Paladin adds a new layer to the phishing defense stack. Even if traditional filters miss an LLM-generated phishing email, the embedded tags provide an additional detection signal at the email gateway or endpoint.

Limitations of the approach: The defense only works for phishing generated by instrumented models. Attackers can train models from scratch, use non-instrumented models, or generate phishing content through other means. The approach assumes broad adoption of instrumentation across popular model releases.

Detection-evasion arms race: While Paladin shows resilience to jailbreaking and malicious fine-tuning, a determined attacker who understands the specific tag mechanism (especially explicit zero-width character tags) could potentially develop targeted removal techniques. The implicit tag approach is more robust but computationally expensive for defenders.

The approach raises interesting questions about the responsibility of model providers to instrument their releases and the feasibility of achieving sufficient coverage across the open-source model ecosystem.

Key Takeaways

• Paladin is a proactive defense that embeds invisible trigger-tag associations into LLMs, automatically tagging phishing content during generation
• Two tag types are supported: explicit (zero-width characters, fast detection) and implicit (logic-level perturbation, more robust but computationally expensive)
• Three insertion strategies (Base/Core/Pro) use SFT, DPO, and GRPO respectively, with Pro providing the best stealthiness through all three optimization constraints
• Over 90% phishing detection accuracy maintained across LLaMA 2, LLaMA 3, and Qwen 2.5 models
• Tags remain detectable even after jailbreak attacks and malicious fine-tuning attempts
• The approach complements traditional phishing detection by operating at the content generation source rather than the receiving end
• Phishing accounts for 90% of all cyber attacks with over $10 billion in annual losses, making any improvement in detection at scale significant

About the Speaker(s)

Yan Pang is a researcher working at the intersection of LLM security and phishing defense. The work was conducted as a joint research effort. Pang demonstrated familiarity with both the NLP/ML aspects of the work (fine-tuning strategies, reward optimization) and the cybersecurity context (phishing detection, adversarial threat models). During Q&A, Pang clarified that the system's ability to recognize phishing queries depends on the training dataset composition, and confirmed resilience testing against jailbreak and malicious fine-tuning scenarios.