Democratizing IoT Security Research and Training Through Advanced Instrumentation

Democratizing IoT Security Research and Training Through Advanced Instrumentation

Speakers: Kamel Ghali (COO, Kage)

Conference: DefCamp

YouTube: https://www.youtube.com/watch?v=lhLyulphkWM

Overview

In this insightful DefCamp presentation, Kamel Ghali, COO of Kage and a prominent figure in the automotive cybersecurity community, introduced "Project Shadow" – a groundbreaking initiative aimed at democratizing access to cyber-physical systems for security research and training. The talk highlighted the unique and substantial barriers preventing aspiring and established security professionals from engaging with real-world Internet of Things (IoT) and automotive hardware, contrasting this with the relative ease of access in traditional IT security. Ghali emphasized that while the digital realm offers abundant free resources, the physical world demands significant monetary and logistical investment, creating a critical talent gap in a sector where security vulnerabilities can have life-threatening consequences.

Ghali's presentation meticulously outlined the motivations behind Project Shadow, stemming from both personal experience in delivering hands-on automotive security workshops and a broader understanding of industry challenges. He detailed how the high costs of specialized hardware, the logistical complexities of shipping devices or flying experts, and the inherent risks of intellectual property leakage or device damage hinder effective security testing and talent development. Project Shadow, or the Secure Hardware Attack and Defense Operational Warehouse, emerges as a sophisticated solution, providing remote, browser-based access to instrumented physical hardware, enabling global participation in critical security research, penetration testing, bug bounties, and educational initiatives.

The significance of this endeavor cannot be overstated. As societies increasingly rely on connected vehicles, smart infrastructure, and medical devices, the security of these cyber-physical systems becomes paramount. By lowering the entry barrier for hands-on experience, Project Shadow promises to cultivate a larger, more diverse pool of skilled security professionals, accelerate vulnerability discovery, and ultimately contribute to a safer, more resilient digital-physical future. This initiative directly addresses the critical supply-demand imbalance for hardware security expertise, fostering innovation and collaboration across the cybersecurity ecosystem.

Background

▶ Watch: Introduction and Car Hacking Village mission (0:00)

The journey into cyber-physical system security, encompassing domains like automotive, industrial control systems (ICS), and IoT devices, presents a stark contrast to learning traditional IT security. Kamel Ghali, drawing from his extensive experience, including five years volunteering with the Car Hacking Village and teaching numerous workshops, highlighted the prohibitive challenges. While platforms like Hack The Box, PortSwigger Academy, and various online CTFs offer accessible, low-investment pathways for IT security learning—requiring little more than a laptop and internet connection—the landscape for cyber-physical systems is entirely different.

Ghali recalled his early days teaching CANbus hacking, which involved lugging "5 kg of speaker wire," numerous CAN dongles, OBD2 adapters, and a toolkit for on-the-fly repairs. Even with advancements like Toyota's open-source Ramen RN platform, which integrates many necessary components, equipping a class of 15 students still means transporting 15 physical devices and their accessories. This physical dependency translates into considerable space and monetary investment for researchers. Acquiring and storing multiple ECUs (Electronic Control Units), specialized interfaces for CANbus, LINbus, or wireless technologies like Bluetooth and Zigbee, quickly becomes unscalable for individuals or smaller organizations without dedicated lab space.

The motivation for overcoming these hurdles is profound. Unlike a hacked computer leading to financial loss or data compromise, a compromised car, pacemaker, or industrial system can result in "fatal" consequences, causing physical damage to people or property. Recognizing these severe risks, international bodies have introduced regulations such as UNR155 (for vehicle type approval in the EU, mandating a cybersecurity management system) and the forthcoming EU Cyber Resilience Act. These regulations incentivize manufacturers to invest in security by design, creating a surging demand for skilled professionals capable of performing security testing and management for hardware.

However, existing approaches to addressing this demand are fraught with issues. Manufacturers might ship hardware to remote experts, risking loss, damage, export control violations, or intellectual property (IP) leakage if the device falls into the wrong hands or if the researcher (intentionally or unintentionally) bricks the device. Alternatively, companies "import hackers" to their facilities, a costly and time-consuming endeavor involving travel, accommodation, and premium pay, with no guarantee of findings in a short timeframe. This approach also carries the inherent risk of unsupervised access to sensitive IP.

Public bug bounty programs, exemplified by the Pwn2Own competition run by the Zero Day Initiative (ZDI), offer significant cash prizes for discovering vulnerabilities in consumer electronics, including a recent automotive-specific event in Tokyo. Yet, participation remains exclusive. Ghali cited a Kenwood automotive target from an upcoming Pwn2Own event, costing $1,400. This initial hardware investment, coupled with travel expenses to the competition location (e.g., Tokyo), accommodation, and the immense time commitment for vulnerability research and exploit development, can make a potential $20,000 payout a financially risky proposition for many researchers. The Car Hacking Village previously mitigated this by purchasing Pwn2Own targets and making them available at events, sparking interest but still limited by physical presence.

The genesis of Project Shadow can be traced back to a COVID-era challenge. Tasked with delivering a wireless network attack training for the Japanese Automotive ISAC (Information Sharing and Analysis Center) executives, Ghali faced strict in-person gathering restrictions. His "janky" but functional solution involved participants connecting remotely via Zoom to a physical laptop in his lab, which then provided SSH access to a Raspberry Pi equipped with car hacking tools. This experience, despite its latency and corporate IT policy hurdles, demonstrated the potential for remote access to physical hardware, laying the groundwork for a more robust and scalable solution.

Key Findings

▶ Watch: Why securing cyber-physical systems is critical (12:00)

The central "key finding" or contribution of Kamel Ghali's talk is the conceptualization and initial implementation of Project Shadow, officially known as the Secure Hardware Attack and Defense Operational Warehouse. This platform represents a paradigm shift in how security research, penetration testing, bug bounties, and training are conducted for cyber-physical systems. By providing remote, browser-based access to real, instrumented hardware, Project Shadow effectively addresses the multifaceted challenges of cost, physical proximity, and specialized expertise that have historically limited participation in this critical field.

The core innovation lies in its ability to abstract away the physical barriers, making advanced hardware security accessible to a global audience. This democratizes opportunities for individuals who lack the financial resources, physical space, or geographical proximity to acquire and maintain complex IoT and automotive testbeds. The platform ensures that regardless of location, a researcher or student can interact with actual devices, perform genuine vulnerability validation, and gain hands-on experience that is otherwise unattainable.

Key findings and contributions of Project Shadow include:

• Global Accessibility: Eliminates geographical and logistical constraints, allowing security professionals and students from anywhere in the world to access specialized hardware labs.
• Cost Reduction: Significantly lowers the personal investment required for hardware acquisition, maintenance, and setup, removing a major barrier to entry for aspiring hardware hackers.
• Enhanced Training and Education: Enables the delivery of immersive, hands-on training for IoT and automotive security without the need for physical hardware distribution, supporting both instructor-led and self-paced learning models.
• Streamlined Bug Bounty and Penetration Testing: Provides a controlled, secure environment for manufacturers to host private or public bug bounty programs and penetration tests, mitigating risks associated with IP leakage, device damage, and export controls.
• Real-World Validation: Offers interaction with actual physical devices, providing a more authentic and impactful experience than purely virtual or simulated environments, crucial for validating vulnerabilities and developing exploits that work consistently.
• Scalability for Research: Facilitates concurrent or scheduled access for multiple users to a variety of hardware targets, expanding the capacity for diverse research efforts.

In essence, Project Shadow's key finding is that with advanced instrumentation and a well-designed remote access platform, the traditional limitations of cyber-physical security research can be largely overcome, fostering a more inclusive and productive security community for these vital technologies.

Technical Deep Dive

▶ Watch: The 'janky' origin of remote hardware access (26:00)

Project Shadow's technical architecture is built upon the principle of comprehensive instrumentation and secure remote access. The platform's objective is to transform any IoT or automotive component into a remotely controllable and observable testbed. This involves a meticulous setup process for each physical device in Kage's Tokyo-based lab.

Upon receiving an IoT or automotive component, Kage engineers equip it with a full array of interfaces necessary for security testing. This includes:

• CANbus: The ubiquitous Controller Area Network for in-vehicle communication.
• Bluetooth: For short-range wireless communication.
• USB: For direct host-to-device communication, often used for debugging or flashing.
• JTAG/SWD: Low-level debugging interfaces crucial for firmware extraction, memory analysis, and exploit development.
• SPI/I2C/UART: Serial communication protocols used for interacting with microcontrollers, flash memory, and other internal components.
• Other in-vehicle networks: Such as LINbus, FlexRay, and automotive Ethernet, depending on the target device.

Each instrumented device is then connected to a control system that allows for remote manipulation and data capture. Users gain access to this environment via a secure VPN connection, which then presents an in-browser terminal interface. This terminal acts as the primary interaction point, providing a Linux-based environment pre-loaded with a comprehensive suite of car hacking and IoT security tools.

Beyond the terminal, Project Shadow offers several critical features to enhance the remote testing experience:

• Customized Scripts: Automated scripts are provided to streamline common testing and validation tasks, aiding in compliance testing and rapid assessment.
• Documentation: Comprehensive documentation for the target hardware, including schematics, datasheets, and known vulnerabilities, is made available.
• Firmware Access: For many devices, the platform facilitates downloading the firmware directly, saving researchers the effort of extracting it themselves. For devices where multiple firmware versions are available, the platform can even reflash the device to a clean, default state for each new user session, ensuring a consistent testing environment. This capability, however, often requires collaboration with the manufacturer for proprietary or state-of-the-art hardware.
• Camera Feed: For devices with a visual component (e.g., infotainment systems with touchscreens, medical devices with displays), a live camera feed is provided. Ghali specifically mentioned overcoming the challenge of remotely interfacing with touchscreens, which is crucial for many bug bounty programs.
• Flexible Environments: The platform supports more complex setups than just single devices. It can host multiple ECUs connected in a network, simulating a more realistic vehicle or industrial system architecture. This includes supporting advanced protocols like CAFD (CAN Flexible Data Rate) for modern automotive systems.

The speaker demonstrated a prototype showing a terminal reading and injecting CANbus data, with an instrument cluster on the right visually responding to changes (e.g., modifying vehicle speed). While acknowledging minor Python errors in the prototype, the demonstration highlighted the platform's core functionality. A live demo attempt, though hampered by internet speed, showcased the login portal ([email protected]) leading to various "programs," which can be either public bug bounty initiatives or private penetration tests. This interface allows manufacturers to exert full control over who accesses their hardware, define terms of use, and integrate findings into their risk response policies, including vulnerability rating based on TARA (Threat Analysis and Risk Assessment) results or other risk assessment methodologies.

To address challenges like low-level attacks (e.g., chip whispering, voltage injection) that are difficult to automate remotely, Ghali stated the team is "going to do our damn best" to integrate such capabilities. Managing concurrent access to limited hardware resources is handled through a time scheduling and reservation system. Furthermore, to mitigate latency issues inherent in global remote access, Kage plans to establish additional labs in strategic locations such as the United States, Europe, and the Middle East, ensuring proximity to a wider user base. The platform leverages existing open-source hardware like the Ramen RN to deliver accessible, immersive training experiences.

Demo / Proof of Concept

▶ Watch: Project Shadow: Remote platform for real hardware hacking (27:00)

Kamel Ghali presented a compelling demonstration of Project Shadow's capabilities, illustrating how remote access to physical hardware can be achieved for security research and training. He initially showed a pre-recorded video of a prototype environment, which provided a clear visual representation of the platform's core functionality.

In this prototype video, the screen was divided, with a terminal on the left and a live camera feed of a vehicle's instrument cluster on the right. The demonstration showcased real-time interaction with the vehicle's CANbus. Ghali was able to remotely read data from the CANbus and, more notably, inject malicious messages to modify the displayed speed on the instrument cluster. He acknowledged some "Python errors" in the prototype, indicating ongoing development, but the fundamental proof of concept—remote manipulation of a physical automotive component—was clearly established. This particular instance was run from Kyoto, connecting to a lab based in Tokyo, highlighting the geographical independence.

Following the video, Ghali attempted a live demonstration of the Project Shadow platform, logging in with [email protected]. Although internet connectivity issues at the conference caused some slowdown, the interface revealed several "programs" that represent either bug bounty opportunities or private penetration tests. This aspect of the demo underscored the platform's flexibility for different types of security engagements. Ghali explained that manufacturers would have full control over these programs, including who gains access to their hardware, the terms of use, and how vulnerabilities are rated and integrated into their risk response policies.

The speaker also referenced the use of Ramen RN devices within the lab environment. These miniature automotive network platforms, which he typically distributes physically for in-person training, can now be accessed remotely through Project Shadow. This allows students to perform hands-on car hacking exercises on real hardware over the internet, providing a far more immersive and realistic experience than purely virtual environments. The emphasis throughout the demo was on interacting with "real hardware" to enable "real validation of vulnerabilities," a critical distinction from simulations.

While the live demo faced minor technical hitches, the combination of the pre-recorded prototype and the explanation of the live platform's features effectively conveyed Project Shadow's potential to bridge the gap between aspiring hackers and the physical hardware they need to practice on.

Defensive Implications

▶ Watch: Q&A: Manufacturer resistance and quantum impact (35:30)

Project Shadow offers significant defensive implications for manufacturers, security teams, and the broader cyber-physical systems ecosystem. By democratizing access to hardware and streamlining the vulnerability discovery process, it directly contributes to more robust and secure products.

1. Proactive Vulnerability Discovery: The platform enables manufacturers to proactively engage a global community of ethical hackers through structured bug bounty programs and penetration tests. This expands the reach beyond internal teams or expensive contracted experts, increasing the likelihood of identifying vulnerabilities before products reach the market. The controlled environment of Project Shadow mitigates risks like IP leakage, which often deter manufacturers from engaging with external researchers.

1. Enhanced Security by Design: As Ghali noted, regulations like UNR155 and the EU Cyber Resilience Act are forcing manufacturers to prioritize cybersecurity. Project Shadow provides a practical tool for compliance, allowing companies to demonstrate a commitment to security testing. By making it easier for researchers to test real hardware, it fosters a "security by design" mindset, encouraging manufacturers to build more resilient products from the outset.

1. Secure Firmware Practices: During the Q&A, Ghali provided direct defensive advice regarding car software security. He stressed the importance of cryptographically signed firmware images and secure boot mechanisms to prevent unauthorized or malicious firmware modifications. Additionally, he mentioned message authentication codes (MACs) and encryption for network traffic as crucial measures. Project Shadow, by allowing remote flashing and analysis of firmware, facilitates the testing and validation of these cryptographic controls in a real-world setting.

1. Talent Development and Workforce Readiness: A critical defensive implication is the cultivation of a larger, more skilled cybersecurity workforce for cyber-physical systems. By lowering the barrier to entry for hands-on training, Project Shadow helps address the supply-demand imbalance for hardware security experts. A well-trained workforce is the first line of defense against emerging threats.

1. Controlled Research Environment: The platform provides manufacturers with granular control over access to their hardware. They can define terms of use, set up specific testing environments, and monitor researcher activity. This contrasts sharply with sending physical hardware or hosting researchers on-site, where control and oversight are often limited. The ability to reflash devices to a clean state after each session also ensures data integrity and prevents persistent tampering.

1. Better Incident Response and Remediation: By integrating vulnerability findings directly into a manufacturer's risk response procedure, Project Shadow helps companies remediate issues more efficiently. The ability to test patches and verify fixes in a remote, controlled environment can significantly shorten the vulnerability lifecycle, reducing exposure time.

In summary, Project Shadow empowers defenders by transforming the previously exclusive and high-friction world of hardware security research into an accessible, scalable, and secure collaborative environment. It moves beyond legislative mandates by providing the practical means for continuous, expert-driven security validation.

Key Takeaways

• High Barriers to Entry for Cyber-Physical Security: Unlike IT security, learning and researching IoT and automotive security demands significant financial investment, physical space, and logistical effort for acquiring and managing hardware.
• Project Shadow Democratizes Access: The Secure Hardware Attack and Defense Operational Warehouse (Project Shadow) provides remote, browser-based access to real, instrumented cyber-physical systems, effectively removing geographical and financial barriers for researchers and students globally.
• Comprehensive Instrumentation is Key: The platform's technical core involves meticulously instrumenting target devices with interfaces like CANbus, Bluetooth, USB, JTAG, and SPI, enabling deep-level interaction and analysis remotely.
• Benefits for Industry and Talent Development: Project Shadow facilitates proactive vulnerability discovery for manufacturers through controlled bug bounties and penetration tests, while simultaneously fostering a larger, more skilled workforce for cyber-physical systems cybersecurity.
• Challenges and Future Vision: While addressing major hurdles, challenges remain, such as accommodating highly specialized low-level attacks, managing concurrent user access (via scheduling), and mitigating latency (through globally distributed labs).
• Collaboration and Regulations Drive Progress: Manufacturer collaboration, often spurred by regulations like UNR155 and the EU Cyber Resilience Act, is crucial for providing access to proprietary hardware and integrating security findings for a safer connected society.

About the Speaker(s)

Kamel Ghali is a dynamic and passionate figure in the global cybersecurity community, with a particular focus on automotive and IoT security. He currently serves as the Chief Operating Officer (COO) and co-founder of Kage Corporation, a Japanese company established to advance cybersecurity development both domestically and internationally.

Originally from the United States, Ghali has been living in Tokyo, Japan, for approximately six years, where he is deeply involved with the local cybersecurity and hacking communities. His commitment to fostering talent extends to his role as the Automotive Cyber Security Director for Akatsky, a non-profit organization dedicated to supporting the Japanese government's efforts to cultivate skilled cybersecurity professionals.

Beyond his corporate and non-profit roles, Ghali is the Vice President of International Affairs for the Car Hacking Village, a renowned organization that raises awareness for vehicle cybersecurity and provides educational materials and workshops globally, including at major events like Defcon. He also organizes local community events such as Besides Tokyo and the Tangukon conference. Ghali humorously describes himself as an "amateur pizza chef," a devoted owner of "two cats," and a "professional yapper," reflecting his engaging and personable presentation style. His involvement with the Car Hacking Village spans five years, starting as a university student, and he has since become a prominent educator, developing and delivering numerous car hacking security workshops.