ViGText: Deepfake Image Detection with Vision-Language Model Explanations and Graph Neural Networks

ViGText: Deepfake Image Detection with Vision-Language Model Explanations and Graph Neural Networks

Speakers: Ahmad ALBarqawi

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=BlTBCHcgSjI

Overview

This talk presents ViGText, a deepfake image detection system that combines vision-language model (VLM) explanations with graph neural networks (GNNs) to achieve state-of-the-art generalization and robustness against both fine-tuned model variants and adversarial foundation model-based attacks. Unlike previous approaches that use simple concatenation of image and text features, ViGText builds integrated graphs that cross-reference forensic explanations with specific image regions, enabling context-aware relational inference.

The system maintains near-perfect accuracy on unseen deepfake variants -- including 16 fine-tuned variants of Stable Diffusion 1.5 and eight additional Stable Diffusion 3.5 variants -- while remaining robust against foundation model-based adversarial attacks that cause substantial performance degradation in competing detectors like DeFake, DCD, and JUNIP.

Background

▶ Watch: The attack landscape: fine-tuned model evasion and foundation model attacks (0:00)

AI-generated images have become increasingly indistinguishable from real content, enabling misinformation campaigns and threatening intellectual property and creative rights. Current deepfake detectors face two critical challenges that empower attackers:

Fine-tuned model evasion: Variants of generative models (particularly LoRA fine-tuned versions) produce images that evade detectors trained on the base model's outputs. Even minor fine-tuning creates sufficient distributional shift to fool existing detection methods.

Foundation model-based attacks: A new class of attacks uses foundation models (EfficientNet, ViT, CLIP-ResNet) as target detectors, enabling adversaries to generate images that subtly modify source content to specifically evade detection models.

Previous approaches like DeFake combined textual captions with images but used general-purpose captions and simple embedding concatenation (averaging both modalities). This approach fails to capture the forensic details needed for robust detection. The key insight of ViGText is that modern vision-language models can generate region-aligned forensic explanations identifying specific artifacts (lighting inconsistencies, texture distortions, anatomical errors), and that these explanations need context-aware integration with image features rather than simple concatenation.

Key Findings

▶ Watch: ViGText pipeline: integrated graph approach for cross-referencing (4:00)

Near-perfect generalization to unseen fine-tuned variants: When trained on Stable Diffusion 1.4 deepfakes, ViGText maintains near-perfect accuracy on 16 fine-tuned variants of Stable Diffusion 1.5 (both LoRA and fully fine-tuned) and 8 variants of Stable Diffusion 3.5. All baseline methods show substantial performance degradation on these unseen variants.

Robust against foundation model-based attacks: When trained on StyleGAN 2 fakes, ViGText remains robust against adversarial variants generated using three foundation model-based attacks (EfficientNet, ViT, CLIP-ResNet). Competing approaches show severe degradation.

Context-aware integration outperforms concatenation: Simply giving DeFake the same forensic explanations ViGText uses does not achieve comparable performance, demonstrating that the graph-based cross-referencing approach is essential -- better explanations alone are not sufficient without proper integration methodology.

Multi-domain feature encoding matters: Combining spatial features, frequency domain features (DCT), and textual features from explanations provides complementary signals that improve robustness against image transformations like JPEG compression, blurring, and brightness adjustments.

Technical Deep Dive

▶ Watch: VLM-generated forensic explanations aligned to image regions (7:30)

ViGText operates through a multi-step pipeline:

Step 1 -- Image Graph Construction: The input image is divided into 16 uniform patches (4x4 grid). An empty graph is initialized where each node represents a patch region. Adjacent patches are connected to preserve spatial relationships. Each node is encoded with two feature types:

• Spatial embeddings from a vision embedding model applied to the raw patch
• Frequency embeddings from the Discrete Cosine Transform (DCT) of the patch

This dual encoding captures artifacts visible in both spatial and frequency domains.

Step 2 -- Explanation Graph Construction: A grid overlay highlights specific regions of the image. The image with grid is passed to a vision-language model using visual prompting, requesting forensic explanations about whether the image is real or fake. The VLM generates region-aligned explanations (e.g., "the window lighting in regions B3-B4 shows inconsistent shadows" or "the oven handle in regions D1-D2 appears distorted").

Each explanation becomes a graph: nodes represent words, edges encode syntactic relationships, and a text embedding model encodes each word into its respective node.

Step 3 -- Graph Integration: The image graph and explanation graphs are integrated by connecting each explanation to the image patches it references. Because the VLM provides region-aligned explanations, this alignment is direct. The resulting integrated graph enables cross-referencing: the GNN can verify whether the forensic description in an explanation is consistent with the actual features of the referenced image region.

Step 4 -- GNN Classification: The integrated graphs are processed by a graph neural network for training and inference. The GNN learns to identify inconsistencies between explanations and image features that indicate deepfakes.

Demo / Proof of Concept

▶ Watch: Generalization results: near-perfect accuracy on unseen fine-tuned variants (8:00)

The evaluation used two primary datasets:

• Stable Diffusion dataset: Real and fake images from Stable Diffusion 1.4, with 16 fine-tuned variants of SD 1.5 (8 fully fine-tuned, 8 LoRA) and 8 additional SD 3.5 variants for generalization testing
• Style dataset: Fake images from StyleGAN 2 with three adversarial test sets using foundation model-based attacks

In-distribution performance: All methods achieve high metrics. ViGText achieves the highest but the differences are small.

Generalization performance: Substantial degradation for all baselines on fine-tuned variants. ViGText maintains near-perfect accuracy across all 24+ unseen variants.

Robustness performance: Severe degradation for DeFake, DCD, and JUNIP against foundation model-based attacks. ViGText remains robust.

Image transformation resilience: Testing against JPEG compression, blurring, and brightness adjustments shows some degradation but ViGText outperforms baselines, attributed to the multi-feature (spatial + frequency + textual) approach.

Defensive Implications

▶ Watch: Q&A: deployment costs, real-world data, and video extension challenges (12:00)

ViGText addresses the critical challenge of deepfake detection generalization -- the ability to detect deepfakes from models and techniques not seen during training. This is the primary failure mode of current detection approaches and the key challenge for practical deployment.

For content moderation teams: The ability to detect deepfakes from fine-tuned model variants without retraining is essential for platforms where new generative models appear constantly. ViGText's graph-based approach provides more robust generalization than feature-concatenation methods.

For security operations: The foundation model-based attack resilience is particularly important. Adversaries who use detection models as targets for adversarial training can evade most current detectors. ViGText's multi-modal cross-referencing provides additional robustness layers.

Cost considerations: The Q&A revealed that generating high-quality VLM explanations using closed-source models is the primary cost factor. Scaling to large image volumes requires considering both API costs and the memory requirements of large image embedding models. Organizations evaluating deployment should plan for these computational requirements.

Current limitations: Fixed 4x4 patching may miss fine-grained details in high-resolution images. The approach has not been tested against replay attacks (photographing a screen showing a deepfake). Video extension is an active research area but faces significant scaling challenges with frame-by-frame graph construction.

Key Takeaways

• ViGText achieves near-perfect generalization to unseen fine-tuned deepfake variants by cross-referencing VLM forensic explanations with image features via graph neural networks
• Context-aware graph integration dramatically outperforms simple embedding concatenation, even when using the same forensic explanations
• Dual spatial + frequency domain encoding captures artifacts that appear in different feature spaces
• The system remains robust against foundation model-based adversarial attacks that cause severe degradation in competing detectors
• Main deployment costs are VLM explanation generation and large embedding model inference
• Future work includes adaptive patching, video/audio extension, and learning-based frequency domain analysis

About the Speaker(s)

Ahmad ALBarqawi is a researcher at the New Jersey Institute of Technology (NJIT). The research addresses the intersection of deepfake detection, vision-language models, and graph neural networks, with practical applications in content moderation and digital forensics. The team is currently working on extending the approach to video deepfake detection.