A Cloud Native Workflow for Hardware-in-the-Loop Software Development - Miguel Angel Ajo, Red Hat

A Cloud Native Workflow for Hardware-in-the-Loop Software Development

Speakers: Miguel Angel Ajo; Red Hat

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=40OmDwTgl1A

Overview

In a KubeCon EU presentation, Miguel Angel Ajo from Red Hat unveiled the JumpStarter project, an innovative open-source framework designed to bring Hardware-in-the-Loop (HIL) testing into the cloud-native era. This talk addressed a critical gap in modern software development: the often-manual, unscalable, and fragile process of testing software that interacts directly with physical hardware. By leveraging the power and scalability of Kubernetes, JumpStarter aims to transform HIL testing from an arcane art into a streamlined, automated, and integral part of the CI/CD pipeline.

The core of JumpStarter's contribution lies in its ability to treat physical hardware devices and their diverse interfaces as cluster resources. This abstraction allows developers and automated systems to remotely provision, control, and test embedded systems, much like they would virtual machines or containers. The project promises to accelerate development cycles for embedded devices, enhance software quality, and prevent costly or even dangerous failures that can arise from inadequate hardware testing in fields ranging from automotive to IoT.

This shift is particularly significant for industries reliant on embedded systems, where the consequences of software bugs can extend beyond mere inconvenience, potentially leading to physical damage, safety hazards, or system-wide failures. JumpStarter offers a robust solution for integrating the physical world into the digital development workflow, ensuring that software destined for hardware is rigorously validated at every stage, from individual developer workstations to large-scale, automated testing labs.

Background

▶ Watch: Introduction to Hardware-in-the-Loop and challenges (0:00)

The landscape of software development has evolved rapidly, embracing robust methodologies, extensive automation, and sophisticated testing frameworks. From unit tests to end-to-end integration, the tools and practices for ensuring software quality are well-established. However, this maturity often grinds to a halt when the software needs to interact with the physical world, specifically with embedded devices and dedicated hardware. This is the domain of Hardware-in-the-Loop (HIL) testing, a technique where a subset of the actual hardware is integrated into the testing environment, simulating the real-world conditions.

Miguel Angel Ajo highlights the stark contrast: while virtual machines and containers provide easily manageable, scalable, and reproducible environments for pure software, physical hardware presents numerous challenges. Embedded devices often lack out-of-band management capabilities, making remote control difficult. They feature a bewildering array of interfaces—CAN buses, video inputs/outputs, serial ports, human interfaces—each requiring specialized handling. The result is that HIL testing frequently devolves into a manual, time-consuming, and error-prone process. Companies often find themselves manually testing different hardware variants before a release, a practice that simply doesn't scale.

The consequences of inadequate HIL testing are severe. Beyond the typical software bugs like regressions or difficult-to-diagnose issues, hardware-interacting software failures can manifest as physical damage—devices might "explode or crash." This not only leads to missed deadlines but also incurs significant financial and safety risks. The need for a scalable, automated, and reproducible approach to HIL testing is therefore paramount.

The JumpStarter project originated from Red Hat Emerging Technologies, recognizing this critical need. It has since grown to include an unnamed automotive OEM, the Red Hat automotive team, and a growing community of contributors. While the project initially saw significant traction in the automotive sector, its design is deliberately generic, aiming to serve any domain involving embedded devices and hardware-in-the-loop validation. The overarching goal is to enable open-source HIL testing, providing a seamless experience from a developer's local desk to a fully automated CI/CD integrated lab environment.

Key Findings

▶ Watch: Introducing the Yams Arthur open-source HIL project (3:48)

The JumpStarter project introduces a paradigm shift in how embedded systems and physical hardware are integrated into modern development and testing workflows. Its core contribution is an open-source framework that effectively bridges the gap between the cloud-native ecosystem and the tangible world of hardware.

The primary discovery and contribution of JumpStarter is its Kubernetes-native architecture for HIL testing. It conceptualizes physical hardware devices and their interfaces as first-class resources within a Kubernetes cluster. This abstraction allows for the dynamic allocation, management, and remote interaction with hardware, mirroring how compute resources are handled in a cloud environment.

Key findings and contributions include:

• Scalable and Reproducible HIL Testing: JumpStarter enables the automation of tests that require actual hardware, moving beyond manual processes. By managing hardware as a shared resource, it allows multiple developers or CI/CD pipelines to access and utilize devices concurrently or sequentially, ensuring consistent test environments.
• Modular, Driver-Based Architecture: The framework is built around a flexible driver model. Each hardware interface (e.g., serial, power, video, debug probe, storage) is controlled by a pair of drivers: a server-side component residing near the physical hardware and a client-side Python library. This modularity allows for easy extension to new hardware types and proprietary interfaces without requiring core framework changes.
• Seamless CI/CD Integration: A significant achievement is the ability to directly integrate HIL tests into existing CI/CD pipelines. Tools like Tekton, GitLab runners, and Jenkins can interact with the JumpStarter controller to request hardware leases, execute tests, and release hardware resources, making HIL testing an automated gate in the software delivery process.
• Developer Experience Enhancement: JumpStarter provides a consistent experience from local development to remote lab testing. Developers can write configurations once and use them across different environments. The Python client offers both a powerful command-line interface (gmp shell, J command) and a programmatic API, facilitating test script development.
• Resource Management and Visibility: By using Kubernetes Custom Resource Definitions (CRDs) for exporters, clients, and leases, JumpStarter provides cluster operators with clear visibility into hardware availability, usage, and access patterns. This data can inform decisions on hardware scaling and resource allocation.
• Open-Source and Extensible: The project emphasizes its open-source nature, encouraging community contributions to drivers and the core framework. This fosters a collaborative environment for addressing the diverse needs of embedded systems development.

In essence, JumpStarter's key finding is that the principles of cloud-native computing—resource abstraction, automation, scalability, and API-driven management—can be successfully applied to the traditionally challenging domain of physical hardware testing, fundamentally transforming the development lifecycle for embedded systems.

Technical Deep Dive

▶ Watch: Local developer workflow with Yams Arthur framework (5:58)

JumpStarter's architecture is a sophisticated blend of Kubernetes-native components and specialized hardware-interfacing elements, designed to provide a cohesive and scalable HIL testing environment. The framework is divided into three primary logical components: the Controller, Exporters, and Drivers, all orchestrated to manage and interact with physical hardware.

At the heart of the system is the JumpStarter Controller, a Go-based application deployed within a Kubernetes cluster. This controller acts as the central brain, managing the state of all registered hardware and handling requests from clients. It leverages Kubernetes Custom Resource Definitions (CRDs) to define and manage key entities:

• exporters: Representing physical Linux devices that host the hardware under test.
• clients: Representing authenticated users or CI/CD systems requesting hardware access.
• leases: Tracking current allocations of hardware resources to clients, including duration and associated permissions.

This Kubernetes-native approach allows JumpStarter to benefit from Kubernetes' inherent capabilities for scaling, resilience, and Role-Based Access Control (RBAC), enabling fine-grained control over who can access which hardware and for how long. The speaker noted that while CRDs are currently used for leases, a future shift to a dedicated database might be necessary for long-term usage tracking and cleanup due to the volume of data.

Connecting the Kubernetes cluster to the physical world are the JumpStarter Exporters. These are Linux devices (e.g., a Raspberry Pi) physically located "side-by-side" with the hardware under test. An exporter runs a service that hosts the server-side components of the hardware drivers. Its configuration, typically defined in YAML, specifies its endpoint, authentication token, and, crucially, the various physical interfaces it provides access to. For example, an exporter configuration might define:

• A storage device that can be controlled.
• A power interface connected to an SNMP PDU (Power Distribution Unit), allowing remote power cycling.
• A serial connection, identified by a specific ID, to the embedded device.
• A video input (e.g., a webcam) pointing at the device for visual verification.

The exporter acts as a localized gateway, translating remote commands from the controller into physical actions on the hardware.

The modularity of JumpStarter is largely thanks to its Drivers. Each driver is composed of two parts:

1. Server-side Driver: This component runs on the exporter and directly interacts with the physical hardware interfaces using low-level system calls or specialized libraries. For instance, a serial driver might use udev to identify and manage serial ports, while a power driver might send commands to an SNMP PDU.
2. Client-side Driver: This is a Python library that developers or CI systems use to interact with the hardware. It communicates with the JumpStarter controller, which then relays commands to the appropriate server-side driver on the exporter. The Python client provides a rich API for common hardware operations like flashing images, powering devices on/off, capturing video, and interacting with serial consoles.

The choice of Python for the client-side framework is strategic, offering ease of use and broad adoption in scripting and testing. The client provides a dynamic command-line interface (gmp shell followed by the J command) where available commands adapt based on the drivers configured for the leased hardware. For example, a JPro info command might query a debug probe (like the probe-rs tool) for connected device information.

For test development, JumpStarter offers helpers for popular Python testing frameworks like Pytest. This enables developers to write automated tests that abstract away the complexities of hardware access. Tests can be chained and stateful, allowing for quicker iterations where an entire hardware reset isn't required for every single test case, although this can reduce strict determinism. Examples include tests for booting Linux, verifying GPU driver functionality, or checking LED blinking patterns.

Integration with broader cloud-native development environments is key. The demo showcased integration with Eclipse Dev Spaces (a cloud-based IDE), where client credentials (similar to Kubernetes kubeconfig files) are securely mounted as secrets. For CI/CD, JumpStarter seamlessly integrates with pipeline orchestrators like Tekton. A Tekton PipelineRun can define tasks to request a hardware lease, execute tests (e.g., building a Rust binary and running Pytest scripts), and then release the lease, making the hardware available for the next job. This complete workflow ensures that every code change can be validated against real hardware in an automated fashion.

Demo / Proof of Concept

▶ Watch: Scaling HIL testing to a remote lab setup (6:38)

Miguel Angel Ajo's demonstration vividly illustrated JumpStarter's capabilities, showcasing a complete cloud-native development and testing workflow for embedded hardware. The setup involved a Kubernetes cluster in the cloud integrated with several development and CI/CD tools, connected to physical hardware located remotely in his home office in Madrid.

The Kubernetes cluster hosted the JumpStarter controller, along with Eclipse Dev Spaces operator for cloud-based IDEs, Tekton for CI/CD pipelines, Dex for authentication, and GitLab for source code repositories. The physical hardware was connected via two JumpStarter Exporters, which were essentially Linux devices running near the target hardware.

Exporter 1 was configured with:

• A Raspberry Pi Pico as the target embedded device.
• A debug probe (using the probe-rs tool) for flashing and debugging the Pico.
• A camera pointed at the Pico to capture video feedback.

Exporter 2 contained another debug probe and a logic analyzer (though the driver for the logic analyzer was not yet complete).

The demo began with a developer using a cloud-based Eclipse Dev Space. Within this environment, JumpStarter client credentials were automatically mounted as Kubernetes secrets, granting access to the remote hardware. The developer could then:

1. List available hardware: Using gmp get exporters, the developer saw the registered exporters on the cluster.
2. Request a hardware lease: gmp shell --selector webcam was used to request access to hardware matching the "webcam" selector, establishing a connection that would expire after 30 minutes (configurable).
3. Interact with hardware via CLI: The dynamic J command allowed interaction with configured drivers. Examples included JPro info to query the debug probe, JSerial start console to open a serial console to the Raspberry Pi Pico, and JVR reset to reset the device. The speaker demonstrated remotely interacting with the Pico's console.
4. Run Python scripts: A Python script was shown to stream video from the webcam, demonstrating real-time visual feedback from the physical device.
5. Execute automated Pytest suite: A comprehensive set of Pytest tests was run. These tests performed various actions:

• Verifying the debug probe connection and identifying the ARM device.
• Downloading and flashing a Rust-compiled binary onto the Raspberry Pi Pico.
• Interacting with the serial console, resetting the device, and asserting specific output (e.g., "hello cubecon").
• Checking the functionality of regular and DMA writes.
• Interacting with the device to control LEDs.
• Crucially, testing that LEDs were blinking in opposite directions, a specific functional requirement.

The CI/CD integration using Tekton was then demonstrated. The speaker introduced a deliberate "breaking change" in the Rust code, modifying the LED blinking pattern to be non-alternating. Upon committing and pushing this change to GitLab, a Tekton pipeline was automatically triggered. The pipeline's tasks included:

• git-clone: Fetching the repository.
• build: Compiling the Rust binary.
• request-jumpstarter-lease: Acquiring a lease for the required hardware.
• run-commands: Executing the Pytest suite on the leased hardware.
• release-jumpstarter-lease: Releasing the hardware back to the cluster.

As expected, the Tekton pipeline failed, specifically during the LED blinking test (test_leds_opposite_directions), confirming that JumpStarter successfully detected the regression introduced by the code change. This demonstrated the framework's ability to integrate HIL testing directly into an automated CI/CD workflow, providing immediate feedback on hardware-related regressions.

Finally, the speaker briefly showed the Kubernetes CRDs in action, listing exporters, jumpstarterclients, and leases on the cluster, providing visibility into the managed hardware resources and their usage.

Defensive Implications

▶ Watch: CI/CD integration and practical Python test examples (8:20)

While JumpStarter is primarily a development and testing tool, its capabilities have significant indirect defensive implications for the security posture of embedded systems and IoT devices. By enabling robust, automated Hardware-in-the-Loop testing, JumpStarter contributes to security in several critical ways:

1. Enhanced Software Quality and Reduced Attack Surface: The most direct defensive benefit is the ability to develop higher-quality, more reliable embedded software. Rigorous, automated testing helps catch bugs, regressions, and unintended behaviors before deployment. Many security vulnerabilities stem from functional flaws or unexpected states. By improving overall software quality, JumpStarter indirectly reduces the potential attack surface that could be exploited by adversaries.

1. Reproducible Vulnerability Testing: JumpStarter provides a controlled, reproducible environment for security testing. Security researchers and penetration testers can leverage the framework to automate the process of flashing firmware, interacting with hardware interfaces, and observing system behavior under various inputs. This is invaluable for:

• Fuzzing: Repeatedly sending malformed inputs to hardware interfaces (e.g., serial, network, CAN bus) to uncover crashes or unexpected states.
• Exploit Development/Verification: Testing and verifying exploits against actual hardware in a safe, isolated, and repeatable manner.
• Regression Testing for Security Patches: Ensuring that security patches effectively mitigate vulnerabilities without introducing new ones or breaking functionality.

1. Supply Chain Security for Firmware/Hardware: As embedded systems increasingly rely on third-party components and firmware, ensuring their integrity is paramount. JumpStarter can be integrated into supply chain security workflows to automatically test new firmware versions or hardware revisions. This allows organizations to verify that updates perform as expected on physical hardware and do not introduce new, exploitable behaviors or backdoors.

1. Incident Response and Forensics: In the event of a security incident involving an embedded device, the ability to precisely reproduce the hardware state and execute specific sequences of operations is crucial for forensic analysis and root cause identification. JumpStarter's capacity to manage and interact with hardware in a controlled manner facilitates this, allowing security teams to recreate attack scenarios and test remediation strategies without risking production systems.

1. Secure Remote Management: While JumpStarter enables remote access to physical hardware, it also provides the necessary framework for securing that access. The use of Kubernetes RBAC ensures that only authorized users or CI/CD pipelines can lease and interact with specific hardware. Proper configuration of Kubernetes Secrets for credentials and secure network segmentation for the Exporters are vital to prevent unauthorized access to the physical test infrastructure, which could otherwise be a target for attackers seeking to manipulate or damage devices.

1. Driver Security: The custom drivers are the direct interface to the hardware. It is critical that these drivers themselves are securely developed and audited. Vulnerabilities in a JumpStarter driver could potentially expose the underlying physical hardware to compromise. Organizations adopting JumpStarter should ensure that their custom drivers follow secure coding practices and undergo security reviews.

By fostering a culture of comprehensive, automated testing that includes the physical layer, JumpStarter helps organizations build more resilient and secure embedded systems, mitigating risks throughout the product lifecycle.

Key Takeaways

• Cloud-Native HIL Testing: JumpStarter introduces an open-source framework that brings Hardware-in-the-Loop (HIL) testing into the cloud-native ecosystem, leveraging Kubernetes to manage physical hardware as cluster resources.
• Scalable Automation: It enables scalable, automated testing of embedded devices, transitioning HIL from manual, ad-hoc processes to fully integrated, reproducible steps within CI/CD pipelines.
• Modular Architecture: The framework's modular, driver-based design allows for seamless integration with diverse hardware interfaces (e.g., serial, power, video, debug probes) and supports custom driver development without modifying the core system.
• Developer & CI/CD Integration: JumpStarter provides a consistent developer experience, from local development to remote lab testing, and integrates effortlessly with cloud development environments (like Eclipse Dev Spaces) and CI/CD tools (Tekton, GitLab).
• Enhanced Quality & Security: By facilitating rigorous, automated testing of hardware-interacting software, JumpStarter significantly improves product quality, reduces the risk of physical device failures, and indirectly strengthens the security posture of embedded systems.
• Open Community Project: JumpStarter is an open-source project from Red Hat, actively seeking community contributions for drivers and framework enhancements, fostering collaboration in the embedded testing domain.

About the Speaker(s)

Miguel Angel Ajo is a professional from Red Hat who works within their Emerging Technologies division. He is the driving force behind the JumpStarter project, which aims to provide an open-source, cloud-native workflow for Hardware-in-the-Loop software development. His work focuses on bridging the gap between traditional embedded systems development and modern cloud-native practices, enabling more efficient and scalable testing of physical hardware.