Autonomous Video Hunter AI Agents for Real Time OSINT

Autonomous Video Hunter AI Agents for Real Time OSINT

Speakers: Kevin Dela Rosa, Founder, Cloud Glue

Conference: Recon Village

YouTube: https://www.youtube.com/watch?v=oHjQSpcP664

Overview

In an era where video content dominates digital landscapes, a vast amount of intelligence remains "trapped" within these visual and auditory streams, largely inaccessible to traditional analytical methods. Kevin Dela Rosa, founder of Cloud Glue, presented a groundbreaking talk at Recon Village, introducing the Autonomous Video Hunter, an AI agent designed to revolutionize Real-Time OSINT (Open-Source Intelligence) extraction from video. This innovative system leverages the latest advancements in multimodal AI to transform overwhelming video data into actionable insights, addressing a critical challenge for intelligence professionals, security researchers, and even general users seeking to monitor specific events or individuals.

The core premise of Dela Rosa’s work is to empower AI agents with the ability to "see" and "understand" video, orchestrating a suite of lower-level vision and audio analysis tools to autonomously identify targets, extract context, and synthesize comprehensive reports. This talk is particularly significant because it demonstrates a practical, proof-of-concept system that bridges the gap between raw, unstructured video footage—from sources like social media (TikTok, Reddit) to public cameras and government proceedings—and structured, queryable intelligence. By showcasing live demonstrations ranging from celebrity tracking to brand monitoring and real-time screen analysis, Dela Rosa underscores the profound implications of this technology for enhancing situational awareness and uncovering hidden information at scale.

Background

▶ Watch: Introduction to Autonomous Video Hunter AI (1:15)

The proliferation of video content across platforms like TikTok, Reddit, YouTube, and even public surveillance feeds has created an unprecedented volume of data. While rich in information, this data remains largely unstructured and difficult to process manually for intelligence purposes. Traditional OSINT methodologies, often reliant on text-based searches or manual video review, are inadequate for the sheer scale and complexity of video analysis. The challenge lies in extracting specific, relevant details—such as faces, logos, spoken words, on-screen text, or environmental context—from hours of footage efficiently and accurately.

Prior to the recent advancements in multimodal AI, the capability to automatically process video for complex queries was limited. Early attempts involved isolated computer vision tasks like object detection or facial recognition, but these lacked the overarching intelligence to contextualize findings or autonomously formulate and execute investigative plans. The emergence of powerful Large Language Models (LLMs) and sophisticated Vision Language Models (VLMs) has changed this landscape, providing AI systems with enhanced reasoning capabilities and the ability to understand and generate human-like language based on diverse inputs, including visual and auditory data. This technological leap enables the development of agentic systems—AI programs that can perceive their environment, form plans, use tools, and execute actions to achieve a goal. The Autonomous Video Hunter stands as a testament to this evolution, aiming to "vibe code" an intelligence report from video, much like advanced AI agents might "vibe code" a website.

Key Findings

▶ Watch: Broad Architecture of the Video Hunter System (3:30)

The central contribution of Kevin Dela Rosa's talk is the demonstration and architectural outline of the Autonomous Video Hunter, a proof-of-concept AI agent system designed for real-time OSINT from video. The key findings revolve around its ability to:

1. Orchestrate Multimodal Analysis: The system effectively combines various lower-level computer vision and audio analysis tools under the control of a central AI agent. This allows for a holistic understanding of video content, integrating visual descriptions, automatic speech recognition (ASR), and on-screen text extraction.
2. Autonomous Plan Generation and Execution: Unlike traditional systems that require explicit scripting for each analytical task, the Autonomous Video Hunter leverages an LLM to dynamically generate a "plan of attack" based on a user's query. It then intelligently executes the necessary tools in the correct sequence to fulfill that plan.
3. Synthesize Actionable Intelligence: Beyond mere detection, the agent synthesizes raw findings into comprehensive, human-readable reports. These reports provide not just what was found (e.g., a face, a logo) but also where (video, timestamp), what context (what they were doing, emotional state), and why it matters (aggregate summaries).
4. Scalability for Video Corpora: The architecture is designed to process large corpuses of videos, filtering out noise and extracting targeted intelligence. This moves beyond single-video analysis to enabling large-scale monitoring and investigation.
5. Flexibility and Configurability: The system is built with modularity in mind, allowing users to swap out underlying models (e.g., for face detection) with more state-of-the-art or resource-intensive options depending on accuracy requirements and available hardware (e.g., CPU vs. GPU). The demo largely utilized CPU-biased models for accessibility.

These findings collectively highlight a significant leap in automated OSINT capabilities, transforming overwhelming video data into structured, queryable, and actionable intelligence with unprecedented efficiency.

Technical Deep Dive

▶ Watch: Demo Scenario: Tracking Celebrities in Zurich (4:25)

The Autonomous Video Hunter's architecture is a sophisticated orchestration of several key components, designed to process, analyze, and report on video content autonomously. Dela Rosa detailed a pipeline comprising a video listener, a video processor, a searchable context store, and the central video analysis agent.

Core Architecture

1. Video Listener: This is the most variable component, responsible for acquiring videos of interest. For the demonstrations, Dela Rosa showcased a social media intelligence video listener, specifically a TikTok scraper built using Gumloop, a no-code platform. This allowed for ingesting videos based on criteria like city hashtags (e.g., #Zurich) and automatically notifying users via email.
2. Video Processor: Once videos are acquired, this component processes them to extract general context. It performs two primary functions:

• Mechanical Frame Handling: Stores video frames locally on the system, which is crucial for subsequent analysis.
• Vision Language Model (VLM) Application: A VLM is run over the video frames to generate a dense, play-by-play description across different modalities for each scene. This includes:
• Visual descriptions of what is happening.
• Automatic Speech Transcription (AST) of what was said.
• Extraction of any on-screen text.

This rich, multimodal context is invaluable for ad-hoc search and initial filtering.

• Targeted Extraction: The processor also performs initial, specific extractions to quickly identify the presence of key elements. This includes determining if a video contains a logo, a face, any speech, is set outdoors versus indoors, or contains any text. This helps the agent "short circuit" its orchestration by filtering out irrelevant videos early.

1. Searchable Context Store: The extracted general and targeted context is then stored in a format that allows for efficient querying and retrieval. While not explicitly detailed in a demo, this component is critical for the agent to access and reason about the video's content during its analysis phase.
2. Video Analysis Agent: This is the "meat" of the system, acting as the intelligent orchestrator. It uses a framework called Langraph to pipe different tools and information together. The agent's operation is based on four core components:

• Planning: An LLM generates a detailed plan of action to answer the user's query.
• Sub-agents/Tools: It orchestrates calls to various lower-level machine learning models and utilities.
• File System/Memory: It interacts with a storage mechanism to write and fetch information as it processes.
• System Prompt: Instructions guide the AI on the specific task it needs to accomplish.

Underlying Machine Learning Models and Tools

Dela Rosa detailed several specific models and libraries powering the targeted extraction and analysis capabilities:

• Face Detection and Matching:
• Leverages the Deepface library.
• RetinaFace is used for detecting where faces occur within a video frame.
• VGGFace is employed to find key points on detected faces and compare them to determine if two faces belong to the same person.
• Notably, these models were chosen for their ability to run efficiently on a CPU, making the proof-of-concept accessible on standard laptops.
• General Image Matching (Logos, Facades):
• Powered by OpenCV.
• Utilizes classic Scale-Invariant Feature Transform (SIFT) visual descriptors, which convert image frames into 128-dimensional vectors.
• The Random Sample Consensus (RANSAC) algorithm is then used to robustly match these descriptors and determine if two images are a good match. This is crucial for identifying specific logos or building facades.
• Zero-Shot Object Detection:
• Employs the open-source Hugging Face library.
• Specifically uses the Owl-ViT v2 model. This model allows the agent to find arbitrary concepts within an image (e.g., "flag," "rocket," "logo") without prior training on those specific objects, providing bounding box detections.

Agentic Interaction

The main video analysis agent handles general video context, search execution, and instance finding. It delegates specific tasks to three specialized sub-agents:

1. Facial Analysis Agent: Focuses on detecting, recognizing, and analyzing faces, including emotional state.
2. Visual Analysis Agent: Handles object detection, image matching (logos, facades), and general visual understanding.
3. Content Analysis Agent: Determines sentiment, extracts temporal information, and performs broader contextual analysis.

All these agents write their findings to an investigation markdown file, ensuring a structured and readable output report. This modular design allows for clear separation of concerns and facilitates the integration of diverse analytical capabilities.

Demo / Proof of Concept

▶ Watch: Demo: Video Analysis Agent Identifies Target Face (7:50)

Kevin Dela Rosa presented three distinct demonstrations showcasing the versatility and power of the Autonomous Video Hunter.

Demo 1: Celebrity Tracking (Will Smith in Zurich)

The first demo illustrated how the system could track a specific individual across social media video streams. The scenario involved an imaginary city official in Zurich interested in celebrity movements.

1. Video Ingestion: The process began with the Gumloop-based social media intelligence video listener, scraping TikTok videos tagged with #Zurich. The input was simply the city and an email for notifications.
2. Target Definition: A reference image of Will Smith was provided to the agent, with the query: "Find the videos that are about Zurich, tell me where this man appears, broadly what are they doing, and the context of why they're in Zurich."
3. Agent Execution: The agent, using its LLM-powered planning, orchestrated calls to the underlying computer vision tools (Deepface, RetinaFace, VGGFace) to perform face detection and recognition across a database of approximately 50 Zurich TikTok videos.
4. Output Report: The system generated a detailed markdown report, including:

• Key Findings: Identification of the specific video and timestamp where Will Smith appeared.
• Narrative: A summary of his activities (e.g., "taking a walk due to an unexpected travel day in Zurich").
• Emotional State: A breakdown of his detected emotional state.
• Visual Evidence: A reference frame from the video and a cropped image of the matching face.

This demo successfully demonstrated the agent's ability to identify a target face, contextualize its appearance, and synthesize a comprehensive narrative.

Demo 2: Logo Monitoring (Starbucks)

The second demo extended the concept to arbitrary image targets, specifically a corporate logo, and aggregated contextual information around its appearances.

1. Target Definition: A Starbucks logo image was provided as the target. The query asked: "Where does it appear? Give me some aggregate summary about the context in which it appears, and show me the actual appearances."
2. Agent Execution: The agent formulated a plan and utilized OpenCV with SIFT descriptors and RANSAC to match the Starbucks logo across a set of 15 videos.
3. Output Report: The system found the logo in four of the 15 videos and produced a summary detailing:

• Appearance Locations: On building facades, on cups, and within a "work context" (e.g., employees wearing the logo).
• Aggregate Context: The videos were categorized as customer testimonials, employees demonstrating activities, or promotional material.

This demonstration highlighted the system's capability for brand monitoring, competitive intelligence, or even supply chain verification by tracking product presence.

Demo 3: Real-Time Screen Summarization with Claude Desktop

The final, more advanced demo showcased integrating the Autonomous Video Hunter with a real-time screen recorder and an external AI chat application for interactive analysis and visualization.

1. Real-Time Ingestion: Dela Rosa connected a custom-rolled screen recorder to the video processor. This setup continuously sent 10-second segments of the screen to be indexed by the agent.
2. Integration with Claude Desktop: The indexed video content and the agent's capabilities were exposed to Anthropic's Claude Desktop via the Model Context Protocol (MCP). MCP is a protocol enabling AI systems to expose tools and resources to AI chat applications.
3. Interactive Analysis: Users could then ask Claude questions about the live screen content, such as "What are people talking about in this Discord channel?" The agent processed the screen (attending to both text and visuals) and provided summaries of conversations, topics, and concerns.
4. Advanced Visualization: Claude Desktop, leveraging its own capabilities, generated ad-hoc visualizations based on the extracted insights. These included:

• Mentions of specific items.
• Types of implementations discussed.
• Concerns raised by users.
• General topics and their frequency.
• A knowledge graph visualization, allowing users to explore relationships between discussed concepts.

This demo illustrated the potential for real-time monitoring of digital environments, incident response, or even competitive analysis of online discussions, transforming passive screen activity into dynamic, queryable intelligence.

Defensive Implications

▶ Watch: Agent's Generated OSINT Report and Key Findings (8:15)

While the Autonomous Video Hunter is presented as a tool for OSINT, its capabilities have significant defensive implications. Understanding how such agents can rapidly and effectively extract intelligence from video is crucial for individuals, organizations, and governments looking to protect their privacy, assets, and operations.

1. Privacy Awareness and Best Practices: The ability to identify individuals (faces), track movements, and infer activities from seemingly innocuous social media posts (e.g., Will Smith in Zurich) underscores the need for heightened privacy awareness. Individuals should be mindful of what they share online, especially videos that might reveal their location, personal items, or daily routines. Organizations should educate employees on responsible social media use to prevent inadvertent disclosure of sensitive information.
2. Physical Security and Surveillance Countermeasures: The system's capacity to identify specific objects (logos, building facades) and track individuals has direct implications for physical security. Defenders can use this insight to understand how adversaries might gather intelligence on their facilities, personnel, or event locations. This could inform strategies for reducing visual identifiable markers, implementing anti-surveillance measures, or deploying countermeasures against video-based reconnaissance.
3. Brand Protection and Reputation Management: The logo monitoring demo highlights how brand assets are exposed in public videos. While useful for marketing, this also means sensitive or negative associations could be automatically detected and aggregated. Defenders in brand management should be aware of this capability for rapid detection of misuse, negative sentiment, or unauthorized appearances of their brand.
4. Insider Threat Detection and Data Leakage: The real-time screen summarization demo illustrates a powerful capability for monitoring digital environments. While presented in a benign context (Discord channel analysis), this technology could, in theory, be adapted to monitor internal communications, screen recordings, or video conferences for sensitive information leakage, unauthorized activities, or indicators of insider threats. Organizations need robust data loss prevention (DLP) strategies that account for video and screen-based exfiltration.
5. Understanding Adversary Capabilities: For cybersecurity professionals and intelligence analysts, understanding the Autonomous Video Hunter's capabilities provides insight into the evolving toolset of adversaries. State-sponsored actors, sophisticated criminal groups, or even individual threat actors could deploy similar AI agents to gather intelligence for phishing campaigns, social engineering, or targeted attacks. Knowing these capabilities allows defenders to anticipate threats and develop more resilient security postures.
6. Content Moderation and Misinformation Detection: On a broader societal level, the ability to rapidly analyze video content for specific objects, text, or speech can be invaluable for platforms engaged in content moderation. Detecting harmful symbols, extremist propaganda, or misinformation embedded within video streams at scale can help mitigate their spread and impact.

In essence, the Autonomous Video Hunter serves as a potent reminder that "what you see is what AI can process." Defenders must internalize this paradigm shift and adapt their strategies to account for the pervasive and increasingly intelligent analysis of visual and auditory data.

Key Takeaways

• Multimodal AI Revolutionizes OSINT: The Autonomous Video Hunter demonstrates how advanced multimodal AI, combining vision, language, and audio processing, can extract actionable intelligence from unstructured video data at unprecedented scale and efficiency.
• Agentic Systems for Autonomous Analysis: The core innovation lies in the AI agent's ability to autonomously plan, orchestrate, and execute a series of specialized tools (e.g., Deepface, OpenCV, Owl-ViT v2) to answer complex user queries, moving beyond simple detection to contextualized reporting.
• Broad Application Across Industries: The demos highlighted diverse applications, including celebrity tracking, brand monitoring, real-time digital environment analysis, and potential use cases in protest monitoring, driver identification, and government intelligence.
• Accessibility Through CPU-Biased Models: For educational and practical accessibility, the proof-of-concept leverages models like RetinaFace, VGGFace, and SIFT that can run on standard CPUs, though it allows for swapping in more powerful, GPU-dependent models for higher accuracy needs.
• Critical Defensive Awareness: The capabilities of such video analysis agents underscore the importance of privacy awareness for individuals and the need for organizations to reassess physical security, brand protection, and insider threat mitigation strategies in light of advanced video OSINT.
• Integration with Emerging AI Protocols: The demonstration of integrating with Claude Desktop via the Model Context Protocol (MCP) showcases how these specialized video analysis agents can become powerful tools within broader AI ecosystems, enabling interactive analysis and rich visualizations.

About the Speaker(s)

Kevin Dela Rosa is the founder of Cloud Glue, a startup dedicated to building agentic systems that possess the ability to "see" and "understand" video. He brings an impressive 14 years of experience in the field of multimodal AI, specializing in transforming audio-visual content into structured data for a wide array of use cases. His extensive work in vision and speech technologies has led to his contributions being featured at prestigious technical conferences, including CVPR (Computer Vision and Pattern Recognition), KubeCon, and AWS re:Invent. Prior to founding Cloud Glue, Kevin held significant roles at prominent technology companies such as Amazon and Snapchat, where he honed his expertise in cutting-edge AI research and development. His insights, shared at Recon Village, reflect a deep understanding of how to leverage the latest AI advancements to tackle complex real-world challenges in intelligence gathering and data analysis.