OROCHIFY - An Assist to Product Developers in Manufacturing Industry to Find Vulnerabilities

OROCHIFY - An Assist to Product Developers in Manufacturing Industry to Find Vulnerabilities

Speakers: Yuki Osawa (Product Security Center, Panasonic Holdings Corporation); Chih-Hsiang Chang (Panasonic Cyber Security Lab)

Conference: FIRSTCON

YouTube: https://www.youtube.com/watch?v=KBLbyHP8t7Y

Overview

This talk introduces Orchestrate (referred to as Orca 5 and Autify in the transcript, standardized to Orchestrate for clarity), an in-house vulnerability assessment tool developed by Panasonic Holdings Corporation. Presented by Yuki Osawa and Chih-Hsiang Chang, the session addresses the critical need for shift-left security in the manufacturing industry, particularly for IoT products. The core motivation stems from the high cost and business risks associated with detecting vulnerabilities late in the product life cycle, specifically just before shipment. Orchestrate is designed to empower product developers, who may lack specialized security expertise, to conduct early-stage security testing, thereby reducing the workload for expert security teams and mitigating risks more effectively.

The tool’s development journey highlights a strategic pivot: initially aimed at supporting security experts with automation, it evolved to focus on simplicity and ease of use for non-security professionals. This shift recognizes that while comprehensive expert assessment remains crucial, a significant portion of basic vulnerability detection can and should be performed earlier by development teams themselves. Orchestrate facilitates this by providing a consistent, automated, and user-friendly platform for pre-testing, aligning with Panasonic’s move towards a business unit-driven approach to product security management.

Ultimately, Orchestrate serves as a practical solution to a pervasive challenge in large manufacturing organizations: scaling security efforts without prohibitive costs or compromising quality. It not only aids in internal vulnerability management but also supports compliance with evolving security labeling schemes, such as Japan's J-Star Level 1, by enabling product teams to perform required self-assessments. This initiative underscores a proactive stance on product security, embedding it into the development process rather than treating it as a final gate.

Background

▶ Watch: Introduction and motivation for early vulnerability detection (0:00)

The traditional approach to product security often places a heavy emphasis on pre-shipment assessment, conducted by security experts as the final gate before a product goes to market. While essential for catching critical issues, this late-stage detection carries significant drawbacks. When vulnerabilities are found just before shipment, the product is often nearly complete, meaning even minor issues can lead to substantial rework, project delays, and considerable business risks. This reactive model is inefficient and costly, prompting a global industry push towards shift-left security principles. The core idea is simple: finding and fixing vulnerabilities earlier in the product life cycle is far less expensive and disruptive than addressing them later.

Panasonic’s internal organizational changes in 2022 further necessitated this shift. The company transitioned from a headquarters-centric security management model to an operating company structure, where individual business units gained greater autonomy and responsibility for product security. This decentralization meant that business units, often lacking dedicated security specialists, needed effective tools and guidance to manage their security posture. The challenges were clear: assigning security specialists to every development team is expensive, and many teams lack the experience to perform consistent, high-quality security tests. This could lead to varying test quality across different business units.

To support this new, distributed security paradigm, Panasonic initiated several activities: providing product security training for developers, preparing practical design guidebooks, and developing internal tools like Orchestrate. The development of Orchestrate itself reflects an iterative learning process. From 2019 to 2021, the initial goal was to automate routine tasks for security experts during pre-shipment assessments, standardizing environment setup, scan execution, and logging. However, this approach proved insufficient. Security experts require flexibility, iterative testing, and the ability to adapt their approach based on results, often involving manual checks and direct interaction with IoT devices, which automation alone could not fully replicate. This realization led to a pivot. From 2022 to 2024, the focus shifted to non-security experts (developers in business units), prioritizing simplicity, automated log analysis, and consistent results, even if it meant initially limiting test cases to basic scenarios. This evolution gave rise to Orchestrate, designed specifically for pre-tests – easy-to-run, automatic checks by developers, complementing the comprehensive, expert-driven pre-shipment assessments.

Key Findings

▶ Watch: Orchestrate's pivot: focusing on non-security experts (8:00)

The development and deployment of Orchestrate have yielded several significant insights into effective vulnerability management within a large manufacturing enterprise, particularly concerning IoT products and the implementation of shift-left security.

Firstly, the experience revealed that automation alone is insufficient for expert-level vulnerability assessment. While initially aiming to automate pre-shipment assessments for security specialists, Panasonic found that experts require an iterative, flexible testing approach. Their work often involves dynamic decision-making, adapting test strategies based on preliminary results, and conducting manual interactions with IoT devices that cannot be fully automated. This underscores that expert assessment is a complex, adaptive process that tools can support but not entirely replace.

Secondly, the talk strongly advocates for the power of pre-tests performed by non-security experts. By shifting the focus of the automation tool from experts to developers, Panasonic discovered that even basic, automated vulnerability checks performed early in the development cycle can significantly reduce the overall security burden. These pre-tests, while limited in coverage compared to comprehensive expert assessments, are crucial for early detection of common vulnerabilities, leading to substantial cost reductions and improved product quality by addressing issues when they are easiest and cheapest to fix.

Thirdly, simplicity and consistency are paramount for tools aimed at non-security professionals. Orchestrate's success among business units is attributed to its user-friendly design, automated log analysis, and consistent results, making it accessible to developers without specialized security skills. This allows for a standardized "risk language" across the organization, facilitating communication between developers and security teams.

Finally, the presentation highlights Orchestrate's role in enabling self-conformance for industry standards. The tool specifically aids business units in meeting the vulnerability assessment requirements of J-Star Level 1, Japan's IoT product labeling scheme. This demonstrates that by empowering developers with simplified security tools, companies can not only enhance internal security posture but also streamline compliance processes, particularly for baseline security requirements that product teams are already somewhat familiar with.

Technical Deep Dive

▶ Watch: Live demo of Orchestrate web GUI (18:00)

Orchestrate is an in-house developed, basic toolset for vulnerability assessment, specifically tailored for Panasonic's manufacturing environment. Its primary target is IoT devices, though it can scan servers (with a caveat against production environments due to potential instability) but does not support application testing. The name "Orchestrate" is an acronym for "orchestration oriented cyber hive for you," drawing inspiration from the Japanese mythology of Yamata no Orochi, a multi-headed snake, symbolizing the tool's ability to orchestrate various components like scanners, tools, containers, and test cases smoothly.

The operational flow for developers using Orchestrate is straightforward:

1. Prepare a PC with Ubuntu OS: Orchestrate is designed to run on a Linux environment.
2. Install Orchestrate: The tool is installed on the Ubuntu PC.
3. Network Connection: The target IoT device must be connected to the same network as the Orchestrate PC.
4. Parameter Setup: The tester inputs a few essential parameters, such as the target device's IP address and web URL.
5. Execution: The test is initiated.
6. Result Review: Testers can check the results directly on the PC.

Orchestrate's design is underpinned by four crucial principles:

1. Docker Containerization: Orchestrate leverages Docker to ensure that each scan environment is independent and isolated. This provides several key advantages:

• Consistency: Every scan runs in the same runtime environment, guaranteeing reproducible results.
• Extensibility: Adding new test cases or integrating new OS scanners is simplified. It merely requires creating a new Docker image, making test case and environment management efficient.
• Isolation: Each scanner operates within its own container, preventing tools from interfering with each other's dependencies or configurations.
• Host-Free Operation: Developers are spared from installing a multitude of testing tools directly on their host machines, avoiding potential dependency conflicts or system clutter.

1. Test Queue System: Recognizing the limited computational power and potential instability of IoT devices, Orchestrate implements a queue system. Tests are executed one by one, preventing simultaneous scans that could overload the device, lead to instability, or produce erroneous results. This is particularly vital for non-security expert users, who might struggle to diagnose scan failures caused by device overload.

1. Dual User Interfaces (UI): Orchestrate offers flexibility through two distinct interfaces:

• Web GUI: Designed for non-security experts, this interface provides a simple, browser-based experience for running scans and viewing results with minimal setup.
• CLI (Command Line Interface): Catering to advanced users, the CLI offers more detailed settings, granular control, and comprehensive logs. Both interfaces utilize the same underlying engine, ensuring consistent results regardless of the chosen interaction method.

1. Automated Judgment: A critical feature for internal consistency, Orchestrate automatically judges every test result against Panasonic's internal security testing standards. While open-source scanners often have their own risk levels, Orchestrate translates these into the company's established guidelines. This creates a unified "risk language" that is consistent across developers and testers, streamlining discussions and remediation efforts for identified vulnerabilities.

This architecture allows Orchestrate to provide a robust, scalable, and user-friendly platform for early-stage vulnerability assessment, effectively bridging the gap between security experts and product developers.

Demo / Proof of Concept

▶ Watch: Practical use cases for Orchestrate in development (22:00)

The talk included a practical demonstration of Orchestrate using its web GUI version, showcasing its user-friendliness for product developers. The demonstration used a Wi-Fi router as the test target.

The steps involved in performing a vulnerability scan were as follows:

1. Network Setup: First, the test target (the Wi-Fi router) and the Orchestrate PC (running Ubuntu) were connected to the same network. This ensures direct communication between the tool and the device under test.

1. Start Orchestrate Daemon: The user initiated Orchestrate by opening a Linux terminal on the Orchestrate PC and entering a simple command: orchify-ctl start. This command starts the Orchestrate daemon service, which then provides a URL for accessing the web GUI.

1. Access Web GUI: The user copied the provided URL and pasted it into a web browser to access the Orchestrate web GUI.

1. Create a Project: The first action within the GUI was to create a new project. Users clicked the "New Project" button and were prompted to fill in two fields:

• Test Set: Orchestrate offers various predefined test sets for different scenarios. For the demo, the "basic test set" was chosen.
• Project Name: A descriptive name was entered for the project.

1. Set Up Target Connection Information: After project creation, the user proceeded to configure the connection details for the test target. This involved clicking a button to set up the IP address and URL of the target device. The web GUI intentionally keeps these options simple, with more detailed settings available via the CLI for advanced users.

1. Add Test Cases: The next step was to select the specific test cases to be executed. A dialogue box displayed available test cases corresponding to the chosen test set, along with their descriptions. In the demo, "all" test cases within the basic set were selected and submitted.

1. Initiate and Monitor Scan: Upon submission, the scan automatically started. The web GUI provided a "scan information" button, allowing users to monitor the progress of each test case in the queue. An "I" button next to individual test cases offered detailed logs for real-time monitoring. The option to stop a scan mid-progress was also available.

1. Review Results: Once the test finished, the results were presented with severity labels. Clicking on a severity label revealed detailed information about the detected vulnerability, including a description, the specific log file where the vulnerability was identified, and its assigned severity level.

This straightforward demonstration underscored Orchestrate's design philosophy of simplicity and ease of use, enabling developers to perform basic security pre-tests without requiring extensive security expertise. The CLI version was also mentioned as providing more custom settings and detailed logs for advanced users, maintaining consistency by using the same underlying engine as the GUI.

Defensive Implications

▶ Watch: Audience Q&A on tool availability and updates (30:00)

Orchestrate offers significant defensive implications for organizations, particularly in the manufacturing sector dealing with IoT products, by integrating security earlier and more broadly into the development lifecycle.

Firstly, the tool is a direct enabler of early vulnerability detection during the implementation stage. By making it easy for developers to conduct pre-tests, Orchestrate helps identify and address vulnerabilities when they are least costly to fix. This shift-left approach not only reduces remediation expenses but also frees up budget and resources that can then be allocated to more comprehensive, expert-led assessments, ultimately leading to higher quality products with lower security risks.

Secondly, Orchestrate drastically improves the efficiency of retesting after vulnerability remediation. In traditional workflows, developers might send a device for testing, receive a report, fix issues, and then send it back for re-verification by security experts. This iterative process can be time-consuming, especially if fixes aren't perfect on the first attempt. With Orchestrate, security testers can create Proof-of-Concept (POC) test cases and share them with developers. Developers can then use Orchestrate to self-check their fixes before sending the device back to the security team, significantly reducing the back-and-forth and accelerating the overall remediation cycle for both sides.

Thirdly, Orchestrate facilitates security review as an integral part of Quality Assurance (QA). It achieves this in several ways:

• Shared Load and Standardized Rules: Security experts can define the necessary test cases, and developers can easily run them using Orchestrate. This distributes the testing workload and ensures that security checks are consistently applied across different business units, even without dedicated security personnel in each team.
• Easy Comparison with Previous Results: Because Orchestrate generates results in a consistent log format, it simplifies the comparison of current product versions with previous models. This helps verify that all previously identified issues have been genuinely fixed and that no new regressions or old flaws have been reintroduced.
• Streamlined Result Management: Orchestrate saves test results locally, allowing for easy export, import, and historical tracking. This ensures that test history is readily accessible and manageable, aiding in long-term security posture analysis and improvement.

Finally, and crucially, Orchestrate plays a vital role in conformance with security labeling schemes. The talk highlights its utility for J-Star Level 1, Japan’s labeling scheme for IoT products. J-Star Level 1 requirements, which align with international standards like the UK PSTI Act and Singapore Cybersecurity Labeling Scheme, include a vulnerability assessment component. While other requirements (e.g., firmware updates, disclosure policies) are familiar to product teams, vulnerability scanning typically demands security expertise. Orchestrate empowers product development teams to perform these required self-checks for J-Star Level 1, ensuring compliance without overburdening central security teams or requiring external consultants. This capability is poised to become even more critical with the emergence of regulations like the EU's Cyber Resilience Act.

Key Takeaways

• Prioritize Simplicity for Early Detection: Starting with simple, automated solutions for non-security experts is highly effective for finding vulnerabilities earlier in the product life cycle, even if full automation for expert-level assessment isn't feasible.
• Pre-tests Complement Expert Assessment: While pre-tests by developers are invaluable for early detection and cost reduction, comprehensive pre-shipment assessments by security experts remain essential for ensuring high-quality, robust security.
• Empower Developers with Consistent Tools: Tools like Orchestrate, designed for ease of use, automated log analysis, and consistent results, enable non-security experts to perform reliable security checks, democratizing basic security testing.
• Shift Left Reduces Cost and Risk: Implementing a shift-left approach, supported by tools like Orchestrate, significantly reduces the cost of vulnerability remediation and mitigates business risks associated with late-stage discoveries.
• Facilitate Regulatory Compliance: Orchestrate helps product teams achieve self-conformance for baseline security requirements in labeling schemes like Japan's J-Star Level 1, streamlining the path to market for secure IoT products.
• Internal Development Ensures Perfect Fit: Developing tools in-house, as Panasonic did with Orchestrate, allows for perfect alignment with specific organizational needs, product types (e.g., IoT devices), and internal security standards.

About the Speaker(s)

Yuki Osawa is from the Product Security Center at Panasonic Holdings Corporation in Japan. She leads several R&D projects, including Orchestrate, the vulnerability assessment tool discussed in this talk, and Astera, an IoT threat intelligence platform that incorporates a honeypot system. Yuki is actively involved in strengthening product security within the Panasonic group.

Chih-Hsiang Chang, also known as Jimmy, is a Technical Lead at Panasonic Cyber Security Lab. He is responsible for security testing within the lab and expresses a keen interest in discovering vulnerabilities and learning attack techniques. Jimmy also functions as a developer, having been one of the core developers behind the Orchestrate tool. This presentation marked his first time at FIRSTCON.