Into the Shopfloor: Moving Manufacturing Execution Syste... Manuel Peuster & Andrei Traian Cucuruzac

Into the Shopfloor: Moving Manufacturing Execution Systems to Kubernetes

Speakers: Manuel Peuster, Senior DevOps Engineer, Bosch Connected Industry; Andrei Traian Cucuruzac, Junior DevOps Engineer, Bosch Connected Industry

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=GgxRHpQIEfg

Overview

This talk, "Into the Shopfloor: Moving Manufacturing Execution Systems to Kubernetes," delves into the ambitious journey undertaken by Bosch Connected Industry to modernize its complex Manufacturing Execution Systems (MES) by migrating them to a cloud-native Kubernetes architecture. Presented by Manuel Peuster and Andrei Traian Cucuruzac, the session highlights the significant challenges inherent in transforming mission-critical, highly individualized industrial software, and the innovative solutions developed to overcome them. Bosch, a global manufacturing giant with over 400 plants worldwide, faces immense pressure to optimize production, reduce downtime, and embrace the demands of Industry 4.0.

The core of the presentation focuses on how Bosch tackled the complexities of standardizing, deploying, and operating MES across diverse factory environments. The speakers meticulously detail their phased approach, from initial containerization to the sophisticated implementation of library Helm charts, advanced schema validation, and custom Kubernetes operators. This talk is particularly relevant for organizations struggling with legacy industrial systems, those embarking on cloud-native transformations in operational technology (OT) environments, and anyone interested in practical strategies for managing large-scale, multi-team Kubernetes deployments.

The speakers articulate not just the technical hurdles but also the crucial "human factor" – the challenge of upskilling development teams with varying levels of Kubernetes expertise. Their solutions, such as the adoption of a robust library Helm chart strategy and dedicated DevOps exchanges, offer valuable blueprints for fostering collaboration and ensuring consistent quality across a distributed development landscape. Ultimately, the talk showcases how a strategic shift to Kubernetes and cloud-native practices can dramatically improve deployment times, enhance system resilience, and lay the groundwork for future innovation in manufacturing.

Background

▶ Watch: Introduction to Bosch and manufacturing complexity (0:00)

Manufacturing Execution Systems (MES) are the digital backbone of modern factories, serving as the critical bridge between enterprise-level planning systems and the real-time operations on the shop floor. As described by the speakers, an MES monitors, manages, and optimizes production processes in real-time, ensuring efficiency, quality, and traceability. To illustrate its comprehensive nature, the presenters use an apt analogy: a factory as a human body. The MES's compute and storage power act as the brain, processing decisions. External systems (like ERP) are the circulatory system, distributing vital information such as production orders and inventory data. Message brokers function as nerves, ensuring communication across the factory. Industry protocols and sensors are the senses, collecting real-time input like temperature, speed, and pressure from machinery. Finally, the shop floor equipment – from assembly lines to automated guided vehicles (AGVs) – represents the muscles, executing production tasks based on MES instructions. Bosch's proprietary MES is called Nixed MES.

Historically, these legacy MES implementations, while successfully powering manufacturing operations for years, presented significant challenges. Each plant, and often each production line, required highly individual configurations, making standardization, replication, scaling, and upgrading extremely difficult. Deployments were typically performed manually on individually ordered Windows virtual machines, a process that was both slow and resource-intensive. Manual configuration existed at two levels: operations (setting up databases, network policies, user permissions) and business logic (creating product recipes, routing rules). This manual, sequential deployment process often took a long time, increasing the risk of misconfigurations and leading to system failures. The consequences of MES downtime are severe, translating directly to production stops, idle workers, lost materials, missed delivery deadlines, and "millions in revenue slipping away."

With the advent of Industry 4.0, the demand for faster, more flexible, and more scalable approaches became imperative. Bosch recognized the need to move beyond traditional "next, next, finish" installations. Their initial step was a "lift and shift" approach, building on Nixed MES to develop Nix IAS, a "cloud-capable design software." This first phase involved containerizing existing MES modules; 30 different modules transformed into over 200 microservices. These containerized microservices were then deployed onto Linux machines, with minimal refactoring focused primarily on portability rather than re-architecting. Deployment strategies included Docker Compose with Ansible for on-premise environments and Ansible rendering Kubernetes manifests for early cloud deployments. However, even with containerization, deployments remained sequential and slow, with updates often taking 6 to 10 hours depending on the installed modules.

Over time, Bosch evolved its deployment strategy. The current state involves simultaneous pipeline deployment of microservices in multi-cluster nodes for cloud scenarios and single-node scenarios for on-premise. Looking to the future, Bosch is developing the Bosch Manufacturing and Logistic Platform (BMLP), envisioned as an "App Store for factories." In this model, each factory will have its microservices running in a multi-node cluster, with changes applied automatically using Argo CD, centralizing management and further automating deployments. This continuous evolution underscores the critical need for robust, scalable, and manageable deployment mechanisms to meet the dynamic demands of modern manufacturing.

Key Findings

▶ Watch: MES components explained using a human body analogy (4:15)

Bosch's journey to modernize its MES revealed three critical dimensions of complexity that shaped their architectural and operational solutions:

1. Manufacturing Context Specificity: Unlike typical microservice deployments that often involve replicating identical instances, manufacturing plants and individual production lines demand highly individualized configurations. This makes standardization challenging, as bespoke input values are often non-negotiable for specific plant operations.
2. Kubernetes Adoption: While some initial Kubernetes knowledge existed within the Bosch teams, the platform itself proved relatively straightforward to adopt for packaging and deploying containerized components. The technical hurdles of Kubernetes were manageable.
3. The Human Factor: The most significant challenge emerged from the organizational structure. With over 30 different MES modules, each developed by an individual team, there was a highly unequal distribution of Kubernetes knowledge. Many teams had no prior Kubernetes experience, posing a significant burden on the central DevOps platform team.

To address these complexities, particularly the human factor and the need for standardization, Bosch opted for a sophisticated library Helm chart strategy. This approach became the cornerstone of their deployment methodology, offering several key advantages:

• Standardization and Quality Enforcement: Instead of allowing each development team to create their own Helm charts (which would lead to inconsistency and quality issues due to varying Kubernetes expertise), or centralizing Helm chart creation (which would create a massive bottleneck), Bosch mandated the use of a central library Helm chart. This chart acts as a funnel, controlling what developers can render to the clusters, ensuring consistent quality and adherence to internal standards.
• Accelerated Development and Release Cycles: By providing a standardized, well-maintained library chart, development teams can focus on their application logic rather than Kubernetes intricacies. Moreover, by allowing development teams to contribute via pull requests to the library chart, new features can be integrated more quickly, avoiding a bottleneck in a central DevOps team.
• Full Software Development Lifecycle (SDLC) for Infrastructure: The library Helm chart is treated as a "first-class citizen," not just a deployment script. It undergoes a full SDLC, including requirements management, backlog maintenance, and a clear roadmap. This ensures its stability, reliability, and continuous improvement.
• Opinionated Data Structure and Abstraction: Module teams are required to place all their configuration directly within the values.yaml file of their module's Helm chart, following a predefined, "opinionated data structure." This abstracts away many Kubernetes details (e.g., ingress definitions are built-in by default), simplifying the developer experience while still allowing for overrides when necessary.
• Two-Level Schema Validation: To ensure configuration correctness and prevent errors, Bosch implemented schema validation at two levels:
• Module-level schema: Maintained by development teams for input parameter validation (e.g., using regex for specific values).
• Global schema: Maintained by the library chart team for structural checking, ensuring the overall data structure aligns with the current library chart version. This provides early feedback during templating, catching errors before deployment.
• Separation of Concerns in Data Models: For external connections (databases, message brokers), the data model is split. A global level, filled by operators, contains environment-specific connection details (e.g., server, port). A module level, defined by development teams, expresses their requirements (e.g., "my module needs an MSSQL database" and specific database roles). Helm templating then merges these to create the final, correctly configured deployment.
• Cross-Module Configuration Sharing: While Helm doesn't natively support sharing configuration between subcharts, Bosch implemented a clever trick. By importing data from a subchart into the global dictionary of the parent chart, that data immediately becomes available to all other subcharts within the deployment, enabling seamless information exchange between modules.
• Controlled Templating in Values: To allow module teams to use Go templating language for dynamic environment variables within their values.yaml files, the library chart explicitly uses the TPL function wherever such templating is allowed. This gives the library team precise control over where and when templating can occur, maintaining quality.
• Custom Kubernetes Operators: Beyond Helm, Bosch developed two classes of custom operators:
• Infrastructure operators: Manage common infrastructure components like databases and schema creation.
• Application-specific operators: Integrate with Bosch's custom identity and access management (IAM) solution, automating module registration, permission assignments, and secret exchange, eliminating manual intervention.
• Significant Operational Improvements: The migration yielded dramatic results. Deployment times for a full MES system dropped from "hours or days to minutes." The system now comprises approximately 1,500 Kubernetes resources per instance. Performance improvements in Helm 3.14 significantly reduced templating time from over 100 seconds to under 10 seconds, further accelerating deployments.

Technical Deep Dive

▶ Watch: Challenges and burdens of legacy MES deployments (6:00)

The core of Bosch's technical solution revolves around a sophisticated library Helm chart architecture designed to manage the complexity of hundreds of microservices across numerous, highly individualized manufacturing environments. This approach addresses the dual challenge of standardization and developer enablement.

At its foundation, each MES module, comprising multiple microservices, is packaged as an individual Helm chart. Crucially, these module charts are minimalistic. They contain only a Chart.yaml file with basic metadata (name, version) and a single template file. This template's sole purpose is to invoke a rendering function provided by the central library Helm chart. For instance, a typical module chart's template might contain just one line: {{ include "my-library-chart.renderModule" . }}. This establishes a single, controlled entry point for all module deployments, ensuring that every resource ultimately rendered into a Kubernetes cluster adheres to the standards defined by the library chart.

All module-specific configurations, rather than being scattered across numerous template files, are consolidated within the values.yaml file of the module's Helm chart. This might seem unconventional, but it enables the library chart team to define an "opinionated data structure" that module teams must follow. This structure abstracts away many Kubernetes-specific details that most developers wouldn't need to interact with directly. For example, common requirements like Ingress definitions are handled by default by the library chart, unless explicitly overridden in the values.yaml. This greatly simplifies the configuration burden for module developers, allowing them to focus on application-level parameters.

To maintain the integrity and correctness of these extensive values.yaml files, Bosch implemented a robust two-level schema validation system:

1. Module-level schema: Each module team is required to include a values.schema.json file within their own Helm chart. This schema is used for input parameter validation, allowing teams to define specific data types, ranges, and even regular expressions (regex) for individual parameters. This ensures that the data provided by the module team is internally consistent and valid.
2. Global schema: The library chart team maintains a comprehensive global schema. This schema performs structural checking, validating the overall data structure of a module's values.yaml against the expectations of the current library chart version. This global validation catches structural inconsistencies early in the development lifecycle, preventing deployment failures and ensuring compatibility across the entire MES system.

The management of external system connections, such as databases and message brokers, also employs a separation of concerns within the data model. Operators (typically the central DevOps team) define global-level information in a dedicated values context, including sensitive connection details like database server addresses, ports, and credentials. Concurrently, module teams declare their requirements at the module level, specifying, for example, that their module needs an MSSQL or Oracle database and outlining necessary database roles. The Helm templating engine then intelligently mixes and matches these global and module-specific values, rendering a final deployment that correctly configures all connections. This approach ensures that sensitive infrastructure details are managed centrally, while application-specific requirements remain with the development teams.

For scenarios requiring shared configuration between different modules (subcharts), Bosch devised a clever workaround to Helm's native limitations. By leveraging the global dictionary within Helm's values context, data can be effectively shared. A trick involves importing specific data from one subchart (e.g., subchartA) into the parent chart's global dictionary. Once in the global dictionary, this information becomes accessible to all other subcharts within the deployment. While powerful for sharing parameters like common service endpoints or feature flags, the speakers caution that this operates within a single global namespace and should be used with care to avoid unintended side effects.

Furthermore, to provide developers with flexibility for dynamic configurations, the library chart utilizes the TPL function (Go templating language function). This allows module teams to embed Go templating logic directly within string values in their values.yaml files, particularly useful for constructing dynamic environment variables. The library chart then explicitly calls the TPL function at the point of rendering these environment variables, giving the library team precise control over where and when such templating is permitted, thereby maintaining quality and security standards.

Beyond Helm, custom Kubernetes operators play a vital role in automating application-specific lifecycle management. Bosch developed two main classes:

1. Infrastructure Operators: These are responsible for common infrastructure tasks, such as provisioning databases, creating schemas, and managing other foundational services. They ensure that the underlying infrastructure required by the MES modules is consistently set up and maintained.
2. Application-Specific Operators: A unique operator integrates with Bosch's custom-built identity and access management (IAM) solution. This operator automates critical processes like registering new modules, assigning appropriate permissions for inter-module communication, and securely exchanging secrets. This eliminates manual intervention, significantly streamlining deployments and enhancing the overall security posture.

The scale of this migration is considerable: 30 original MES modules translated into over 200 microservices, collectively managing approximately 1,500 Kubernetes resources for a single MES instance. The emphasis on unit testing, particularly using the helm unittest plugin, became paramount for the library chart due to its central role and broad impact. Managing Helm state also presented challenges, as the large number of resources could quickly hit the 1MB boundary for secrets, necessitating considerations for switching Helm backends. Finally, the speakers highlighted a significant performance improvement observed with Helm 3.14, which drastically reduced templating times from over 100 seconds to under 10 seconds, underscoring the benefits of leveraging open-source community advancements.

Demo / Proof of Concept

▶ Watch: Serious consequences and costs of MES downtime (6:50)

The talk primarily focused on the architectural and operational strategies employed by Bosch to migrate and manage their Manufacturing Execution Systems (MES) on Kubernetes, rather than featuring a live demonstration or proof of concept of the MES application itself. The speakers presented detailed diagrams and explanations of their Helm chart structure, schema validation processes, and operator functionalities, effectively serving as an architectural blueprint and a "proof of concept" of their methodology. While no direct system demo was shown, the deep dive into their implementation details, including code snippets and architectural patterns, provided ample evidence of their successful cloud-native transformation.

Defensive Implications

▶ Watch: Embracing cloud-native: Initial lift and shift approach (8:00)

The migration of Bosch's Manufacturing Execution Systems (MES) to a Kubernetes-native architecture, driven by the principles outlined in this talk, carries significant defensive implications that enhance the overall security posture and operational resilience of their industrial systems.

Firstly, the adoption of library Helm charts and a mandated "opinionated data structure" is a powerful defensive mechanism against misconfiguration. By standardizing how applications are deployed and configured, the risk of human error – a common source of security vulnerabilities – is drastically reduced. The library chart enforces best practices, ensuring consistent application of network policies, resource limits, and security contexts across all modules. This standardization is particularly crucial given the highly individualized nature of manufacturing plant configurations and the varying levels of Kubernetes expertise among development teams.

Secondly, the two-level schema validation system (module-level and global-level) acts as an early warning system for potential security flaws. By validating input parameters and overall data structures at templating time, configuration mistakes that could lead to insecure defaults, exposed ports, or incorrect access controls are caught before deployment. This proactive approach prevents erroneous configurations from ever reaching production, significantly hardening the system.

The development of custom Kubernetes operators, especially the one integrating with Bosch's custom IAM solution, centralizes and automates critical security functions. This operator ensures that modules are correctly registered, assigned appropriate permissions, and secrets are exchanged securely. Automating these processes removes manual touchpoints, which are often points of vulnerability, and ensures that access control policies are consistently applied and enforced throughout the MES environment.

Furthermore, the shift to cloud-native principles inherently improves the system's resilience and security. Ephemeral containers and immutable infrastructure mean that compromised instances can be quickly replaced, limiting the blast radius of an attack. The ability to perform simultaneous pipeline deployments and drastically reduce deployment times (from hours/days to minutes) means that security patches and vulnerability remediations can be applied much faster. This agility is a critical defensive capability in an era of rapidly evolving cyber threats.

The speakers also touched upon the "human problem" – the unequal Kubernetes knowledge across development teams. Addressing this through DevOps exchanges and continuous training is a vital defensive strategy. A knowledgeable workforce is less likely to introduce vulnerabilities, better equipped to identify suspicious activity, and more capable of implementing secure coding and deployment practices. This investment in human capital directly translates to a stronger security posture.

Finally, the talk acknowledges the persistent challenge of integrating with legacy systems, some of which may still run on Windows VMs. While Bosch's primary MES operates closer to the edge on Kubernetes, the discussion around projects like Istio on Windows highlights a future defensive implication: extending modern service mesh capabilities to legacy components could provide enhanced network security, traffic encryption, and policy enforcement even for parts of the system that cannot be fully migrated. For the public cloud deployments, ensuring that only "non-latency sensitive modules" are hosted there and that these are primarily for external customers (not controlling Bosch's own factory lines) is a critical defensive segregation, limiting the attack surface and potential impact of a public cloud compromise on core operational technology.

Key Takeaways

• Library Helm charts are essential for standardizing complex, multi-team Kubernetes deployments: They enforce quality, abstract away Kubernetes complexities for developers, and accelerate release cycles by preventing central team bottlenecks.
• Robust two-level schema validation for Helm values.yaml is critical: Implementing both module-level and global-level schema checks catches configuration errors early, improving reliability and preventing misconfigurations before deployment.
• Strategic separation of concerns in configuration data models simplifies management: Distinguishing between operator-managed global infrastructure details and module team-defined application requirements makes complex deployments more manageable and secure.
• Custom Kubernetes operators automate application-specific infrastructure and IAM: These operators are crucial for streamlining database provisioning, schema creation, module registration, permission management, and secret exchange, significantly reducing manual intervention and enhancing security.
• Addressing the "human factor" through knowledge sharing is paramount for cloud-native adoption: Regular DevOps exchanges and continuous support for development teams with varying Kubernetes expertise are as vital as technical solutions for successful and secure transformations.
• Migration to Kubernetes and Helm dramatically improves operational efficiency: Bosch's experience demonstrates a significant reduction in deployment times (from hours/days to minutes), showcasing the tangible benefits of adopting cloud-native practices for mission-critical industrial systems.

About the Speaker(s)

Manuel Peuster is a Senior DevOps Engineer at Bosch Connected Industry. He plays a key role in the company's efforts to move manufacturing execution systems to cloud-native architectures, specifically leveraging Kubernetes. His work focuses on developing and optimizing the deployment strategies for Bosch's Industry 4.0 solutions, ensuring scalability, efficiency, and reliability for critical industrial software.

Andrei Traian Cucuruzac is a Junior DevOps Engineer, also with Bosch Connected Industry. Working alongside Manuel, Andrei contributes to the implementation and maintenance of the cloud-native MES platform. He is involved in the technical details of containerization, microservice deployment, and the evolving strategies for managing complex software in manufacturing environments.

Both speakers are part of Bosch, a vast German multinational engineering and technology company with over 400,000 employees and more than 400 plants worldwide. Their expertise lies in Bosch Connected Industry, a division focused on providing Industry 4.0 solutions, including their proprietary Manufacturing Execution System, Nixed MES, and its cloud-capable evolution, Nix IAS. Their insights come from direct experience in transforming mission-critical industrial applications for the demands of modern manufacturing.