SkyScan — Autonomously Filming Aircraft

SkyScan — Autonomously Filming Aircraft

Speakers: Luke Berndt, Mike Chadwick

Conference: ShmooCon

YouTube: https://www.youtube.com/watch?v=-_jUZBMeU5w

Overview

In their ShmooCon presentation, Luke Berndt and Mike Chadwick of iqt's Rapid Prototyping Group unveiled SkyScan, an innovative, open-source project designed to autonomously track and film aircraft broadcasting their position via ADS-B (Automatic Dependent Surveillance-Broadcast). This system offers a low-cost, portable solution for anyone interested in observing and documenting aerial activity, from commercial airliners to military cargo planes, helicopters, and even weather balloons equipped with transponders.

The core premise of SkyScan is to leverage publicly available ADS-B data, which provides real-time position, speed, heading, and climb rate information, to precisely steer a pan-tilt-zoom (PTZ) camera. By automating the tracking process, SkyScan transforms a hobbyist's curiosity into a powerful data collection tool. The entire setup can be built for approximately $500, with the camera being the most significant expense, making it accessible to a wide audience of makers, researchers, and security enthusiasts.

SkyScan's significance extends beyond mere observation; it enables users to establish a baseline of "known knowns" regarding normal air traffic patterns. This capability is crucial for identifying anomalous or unusual aerial phenomena, fostering community awareness, and potentially aiding in the investigation of unexplained airborne events. The project is fully open source, dockerized for easy deployment, and designed for portability, demonstrated by its successful deployment in various locations across the country.

Background

▶ Watch: Speakers introduce SkyScan project (10:00)

The foundation of SkyScan relies heavily on ADS-B (Automatic Dependent Surveillance-Broadcast), a critical component of modern air traffic control systems globally. ADS-B is a transponder-based technology where aircraft continuously broadcast their precise position, speed, heading, and climb rate approximately twice per second. A key characteristic of ADS-B is that this data is broadcast "in the clear," meaning it is unencrypted and can be received by inexpensive, off-the-shelf equipment, primarily Software Defined Radios (SDRs).

ADS-B is mandated for most civilian aircraft operations, though some federal and Department of Defense (DOD) aircraft may have exemptions. Even so, many military cargo flights and air tankers do broadcast their positions. There are two primary frequencies for ADS-B: 1090 MHz, which is the global standard, and 978 MHz, used in the US for aircraft operating below 18,000 feet. To achieve comprehensive coverage in the United States, two SDRs are typically required to monitor both frequencies simultaneously.

The reception of ADS-B data has spawned a vibrant crowdsourcing ecosystem. Platforms like FlightAware and FlightRadar24 are largely fed by a global network of individuals running local SDR receivers and contributing data to central APIs. This "maker community" approach allows anyone with a $40 RTL SDR and a small computer like a Raspberry Pi to participate. Software like dump1090, a long-standing and widely used tool for decoding ADS-B signals, forms the backbone of many such systems, often deployed in containerized environments for ease of use and featuring web-based user interfaces. This local reception provides a "raw feed" of data, bypassing any masking programs (like those offered by the FAA) that might obscure aircraft IDs on commercial flight tracking websites, with ADS-B Exchange being a notable exception that also provides raw data.

The rich data stream from ADS-B has numerous applications beyond simply tracking flights. Researchers map areas of GPS jamming by analyzing the reported quality of GPS information from aircraft. Open-source intelligence (OSINT) communities monitor military aircraft movements, particularly from other countries, to infer potential activities or geopolitical shifts. Bots, such as those pioneered by "Lemon odor," identify interesting flight patterns – like helicopters circling for surveillance, aircraft performing test flights, or surveying operations – and used to publicly tweet these observations. The data also fuels celebrity jet tracking, often leading to humorous or controversial outcomes regarding privacy.

For identifying aircraft beyond their basic ICAO 24-bit code (a unique global identifier broadcast by each transponder), SkyScan integrates with external databases. For US-registered aircraft, the FAA data is a highly regulated, current, and detailed source providing make, manufacturer, and owner information. For international or other non-US aircraft, Open Sky Network offers crowdsourced data, though it requires more data normalization due to varying input quality. By employing an ensemble approach—querying FAA data first, then falling back to Open Sky—SkyScan achieves roughly 90-95% aircraft identification coverage, even in areas with "weird aircraft" like the DC metropolitan region. The speakers also highlighted methods of aircraft ownership obfuscation, such as temporary ICAO codes for wealthy individuals or the use of LLCs (e.g., "Meatball Martini Airline LLC") and trustees for tax, liability, or non-US citizen registration purposes.

Key Findings

▶ Watch: SkyScan: Autonomous aircraft filming (TLDR) (10:48)

The SkyScan project demonstrates several key findings and contributions to the intersection of open-source hardware, software, and aerial surveillance:

Firstly, it proves the feasibility of autonomous aircraft filming using readily available, low-cost components. The ability to automatically track and record any aircraft broadcasting its position via ADS-B, without constant manual intervention, significantly lowers the barrier to entry for detailed aerial observation. This capability extends to a wide array of aerial vehicles, including planes, helicopters, drones, ships, and even weather balloons, provided they carry a transponder.

Secondly, SkyScan establishes a compelling argument for the power of open-source, low-cost solutions. The entire system can be built for approximately $500, with the PTZ camera being the primary expense. This affordability, coupled with the project's open-source nature and Dockerized deployment for both ARM and AMD architectures, makes sophisticated aerial tracking accessible to hobbyists, researchers, and communities without requiring significant financial investment or specialized expertise.

Thirdly, the project highlights the immense value of local ADS-B data reception. By running a local receiver, users gain access to raw, unmasked ADS-B feeds, circumventing any obfuscation programs that might be applied to data shared via commercial APIs. This raw data is crucial for truly independent and comprehensive tracking, enabling users to identify aircraft that might otherwise be hidden from public view on popular flight tracking platforms.

A significant finding is SkyScan's utility in establishing "known knowns" in the aerial environment. By systematically documenting and identifying regular air traffic, the system creates a baseline that allows users to more easily detect and focus on anomalous or "weird" flight patterns and unidentified objects. This capability is vital for community awareness and can assist in quickly clearing false alarms, as exemplified by discussions around recent "drone situations" where distinguishing between normal and unusual activity is paramount.

Finally, SkyScan's methodical data collection process, which links imagery directly to unique ICAO IDs and rich metadata from FAA and Open Sky Network databases, provides an invaluable resource for machine learning (ML) model training. The ability to collect vast quantities of labeled images (e.g., aircraft type, owner, flight path) offers a fast and efficient way to develop localizers for detecting aircraft in general camera feeds or classifiers for identifying specific aircraft models. This opens avenues for future research, including the potential to identify non-transponding aircraft based on visual signatures alone.

Technical Deep Dive

▶ Watch: Understanding ADSB: how aircraft broadcast position (11:39)

The technical architecture of SkyScan is a testament to modularity and leveraging existing open-source tools, integrated with custom logic for precise camera control.

ADS-B Reception and Processing

At the core of SkyScan's data acquisition is the reception of ADS-B signals. This is achieved using an inexpensive RTL SDR (Software Defined Radio), typically costing around $40. The SDR, connected to an appropriate antenna (which can even be custom-built from PVC pipe for optimal reception), receives raw radio signals on the ADS-B frequencies: 1090 MHz (global standard) and 978 MHz (US-specific for lower altitudes). For comprehensive US coverage, two SDRs are recommended.

The raw signals are then processed by dump1090, a widely adopted open-source software package. dump1090 decodes the ADS-B messages, extracting crucial aircraft information such as the ICAO 24-bit code (a unique identifier for each aircraft), longitude, latitude, altitude, speed, heading, and climb rate. SkyScan leverages piaware, a system maintained by FlightAware, which runs dump1090 underneath and aggregates this data, outputting it in a structured JSON format. This JSON output serves as the primary data feed for the rest of the SkyScan system.

Aircraft Identification and Metadata Enrichment

The received ICAO 24-bit code is a unique identifier but provides no contextual information about the aircraft. To enrich this data, SkyScan employs an ensemble approach to query external databases:

1. FAA Data: For US-registered aircraft, SkyScan first queries the FAA's highly regulated and current database. This source provides detailed information including the aircraft's make, manufacturer, and owner. This data is considered "chef's kiss" due to its accuracy and completeness.
2. Open Sky Network: If an aircraft is not found in the FAA database (e.g., military, government, or international aircraft), SkyScan falls back to the Open Sky Network. This crowdsourced database offers broader coverage but requires data normalization to handle inconsistencies in user-entered data (e.g., "Boeing" vs. "Boeing Company").

This two-tiered approach typically yields 90-95% identification coverage. The system also accounts for complexities like obfuscation techniques, such as temporary ICAO codes issued by the FAA to wealthy individuals, or the use of LLCs and trustees for ownership registration, which can obscure the true operator for tax, liability, or international ownership reasons.

Hardware Components

The physical build of SkyScan prioritizes cost-effectiveness and robustness:

• PTZ Security Camera: The most critical and expensive component, typically costing $350-$400 for last year's models found on platforms like eBay. Key requirements include:
• Positioning Accuracy: Essential for precise tracking, with recommended Axis cameras achieving accuracy within a tenth to a hundredth of a degree. These cameras often feature well-documented APIs, MQTT messaging, and even mini web servers, though the latter can pose security concerns if not regularly updated.
• Optical Zoom: High optical zoom (e.g., 32x or 40x) is necessary to image aircraft at distances up to 40-50 kilometers, especially at cruise altitudes.
• Resolution: Higher resolution is beneficial for detailed imagery once zoomed in, though it adds to the cost.
• Power over Ethernet (PoE): Highly recommended for simplified wiring, allowing a single Ethernet cable to provide both power and data.
• Compute Unit: A low-power single-board computer like a Raspberry Pi (models 3, 4, or 5) is sufficient. A laptop can also serve this purpose if on the same network as the camera.
• PoE Switch: Centralizes power and data distribution, especially useful when multiple PoE devices are used.
• Mounting Solutions: The system is designed for versatility, from simple "2x4 special" or "cinder block special" roof mounts to ruggedized enclosures and portable tripods. Weatherproofing is a surprising bonus, with off-the-shelf cameras often surviving harsh conditions without special modification.

Software Architecture and Tracking Algorithm

SkyScan's software follows a modular, containerized architecture using Docker, ensuring easy deployment and portability. The main modules are:

1. piaware (with dump1090): As mentioned, this module receives and decodes ADS-B data, outputting it as JSON.
2. Tracker Module: This module consumes the JSON data from piaware. It's designed to be object-agnostic, capable of tracking any object with latitude, longitude, altitude, velocity, and heading data, making it adaptable for tracking weather balloons, ships, or other transponding entities.
3. Camera Pointer Module: This is the "special sauce" that handles the complex calculations and camera control.

• Azimuth and Elevation Calculation: Given the camera's fixed longitude, latitude, and altitude, and the aircraft's real-time longitude, latitude, and altitude (from ADS-B), the module performs a significant amount of mathematical computation to determine the precise azimuth (pan) and elevation (tilt) angles required to point the camera directly at the aircraft.
• Smooth Tracking via Interpolation: A key challenge is the infrequent nature of ADS-B updates (best-case 2 seconds for position, 5 seconds for altitude). To achieve "movie-like" smooth tracking, the Camera Pointer module interpolates the aircraft's position between updates, utilizing its reported horizontal and vertical velocities. This predictive capability ensures the camera stays locked onto the moving target.
• Prioritization Logic: In areas with high air traffic, the system needs to prioritize which aircraft to track. This can be based on factors like proximity, "interestingness" (e.g., specific flight patterns), or an altitude threshold (e.g., ignoring aircraft above 8,000 feet during cloudy weather to focus on visible targets).

1. Camera API: This module translates the calculated azimuth and elevation commands from the Camera Pointer into specific instructions for the PTZ camera's API, physically steering the camera and initiating imagery capture.

Initial setup involves precise leveling and orientation of the camera (within 1-2 degrees), which can be fine-tuned via software controls. The entire system's code is available on GitHub for public access and contribution.

Demo / Proof of Concept

▶ Watch: DIY ADSB receiver with Raspberry Pi and SDR (12:59)

While the ShmooCon presentation itself did not feature a live, real-time demonstration of SkyScan in action, the speakers provided ample evidence of its capabilities through video footage and numerous photographs captured by the system. They highlighted the "movie-like" smooth tracking achieved through their interpolation algorithms, showcasing the camera's ability to maintain a lock on fast-moving aircraft despite intermittent ADS-B data updates.

The flexibility of the system was illustrated by various field deployments. These included highly portable setups using tripod mounts with small battery banks for day-long power, as well as more permanent installations ranging from improvised "2x4 special" or "cinder block special" mounts on roofs to ruggedized enclosures. The speakers mentioned frequently setting up near Hains Point in Washington D.C., directly across from Ronald Reagan Washington National Airport (DCA), to track and capture imagery of incoming and outgoing flights. They also humorously offered to share their extensive collection of aircraft photos with attendees, underscoring the system's prolific image capture. The robust nature of the chosen cameras was also emphasized, with units surviving over a year and a half of unattended operation on roofs through various weather conditions, including snowstorms and rain, without maintenance or calibration.

Defensive Implications

▶ Watch: Flight tracking ecosystem and GPS jamming detection (13:58)

SkyScan offers significant defensive implications by empowering individuals and communities to gain a clearer understanding of their local airspace. The ability to autonomously track and document aerial activity allows for the establishment of "known knowns" – a comprehensive baseline of regular and expected air traffic. This baseline is invaluable for rapidly identifying anything that deviates from the norm, effectively distinguishing between ordinary aircraft and anomalous or "weird" objects or flight patterns.

For instance, in situations like the "New Jersey drone situation" alluded to by the speakers, where unexplained aerial phenomena create public concern, a SkyScan-like system could quickly clear off false alarms by identifying known aircraft or flight paths. This rapid disambiguation helps investigators and authorities focus their resources on genuinely unusual events, separating "the wheat from the chaff."

Furthermore, by making this technology accessible and open-source, SkyScan fosters community awareness and education. A more informed public, equipped with tools to observe and analyze local airspace, can contribute to a distributed network of aerial monitoring. This collective observation can enhance situational awareness, potentially identifying threats or unusual activities that might otherwise go unnoticed.

The rich, labeled dataset generated by SkyScan also has profound implications for the development of machine learning (ML) models. The unique ICAO IDs linked to images allow for the creation of vast datasets with detailed metadata (e.g., aircraft type, owner, flight path). This data can be used to train ML models for:

• Aircraft Localizers: Models capable of identifying and bounding aircraft within general camera pictures, even when the SkyScan system isn't actively tracking.
• Aircraft Classifiers: Models that can identify specific aircraft types (e.g., Boeing 737, Airbus A320, 787) in real-time imagery.

Crucially, as the speakers noted, if well-trained ML models can be developed, they could theoretically be used to detect and identify non-transponding aircraft in the sky based purely on their visual characteristics. This capability would move beyond the current reliance on ADS-B signals, offering a more comprehensive aerial surveillance tool for defensive purposes and closing a significant gap in current observation methods.

Key Takeaways

• Low-Cost, Open-Source Accessibility: SkyScan provides an affordable ($500) and fully open-source solution for autonomous aircraft tracking and filming, making advanced aerial observation accessible to a broad audience.
• Leveraging Public ADS-B Data: The system effectively utilizes publicly broadcast ADS-B signals to precisely steer PTZ cameras, demonstrating the power of open radio frequencies.
• Robust, Modular Architecture: SkyScan's Dockerized, modular design (SDR, Tracker, Camera Pointer, Camera API) ensures portability, ease of deployment, and adaptability to various tracking needs.
• Enhanced Situational Awareness: By establishing a baseline of "known knowns" through automated documentation, SkyScan helps users quickly identify anomalous or "weird" aerial activity, aiding in threat detection and false alarm reduction.
• Powerful ML Data Collection: The automated collection of richly labeled images (linked to ICAO IDs and external databases) creates a valuable dataset for training machine learning models for aircraft detection and classification, with potential for identifying non-transponding aircraft.
• Community Empowerment: SkyScan encourages active community participation in monitoring and understanding local airspace, fostering an educated and aware public regarding aerial phenomena.

About the Speaker(s)

Luke Berndt and Mike Chadwick are members of the Rapid Prototyping Group at iqt, a non-profit strategic investor that accelerates the introduction of leading-edge technologies to the US national security community.

Luke Berndt, a local resident of the DC area, expressed his honor to speak at ShmooCon, highlighting his long-standing connection to the community where his children learned lockpicking. Mike Chadwick joined him in presenting SkyScan, a project they completed approximately a year prior. Their work at iqt focuses on developing innovative solutions like SkyScan, which bridge the gap between emerging technologies and practical applications, particularly within security and intelligence contexts.