SIG Docs and You: Modernizing API Reference Generation - Kat Cosgrove & Xander Grzywinski

SIG Docs and You: Modernizing API Reference Generation

Speakers: Kat Cosgrove, SIG Docs Technical Lead, Wayland Utani; Xander Grzywinski, SIG Docs Technical Lead, Shinra

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=RdT6P5x_fDM

Overview

This talk, presented by SIG Docs Technical Leads Kat Cosgrove and Xander Grzywinski at KubeCon EU, delves into the critical but often overlooked process of generating API reference documentation for Kubernetes and kubectl. Far from a triumphant showcase of innovation, the presentation served as a candid and urgent "cry for help," exposing a deeply problematic and unsustainable system that has become a significant source of technical debt for the Kubernetes project. The speakers meticulously outlined the current, highly manual, and error-prone workflow for creating these essential reference materials, highlighting its numerous points of failure, lack of documentation, and reliance on a select few individuals with esoteric knowledge.

The core of the problem lies in an antiquated, custom-written toolchain described as a "rat king" of bash, Python, and Go scripts. This convoluted system not only obstructs the involvement of new contributors but also prevents the Kubernetes release team from performing their designated task of generating these docs, effectively creating a single point of failure within a critical project component. The talk underscores the paradox of SIG Docs, a special interest group dedicated to documentation, struggling with its own undocumented and unmaintainable internal processes.

The importance of this discussion extends beyond mere technical curiosity; it touches upon the health, sustainability, and community engagement of one of the world's most vital open-source projects. Accurate and accessible API reference documentation is fundamental for developers, operators, and anyone interacting with Kubernetes. A broken generation process directly impacts the quality and timeliness of these resources, ultimately hindering adoption, troubleshooting, and further development within the ecosystem. This presentation aimed to galvanize the community to address this foundational issue, presenting a vision for a modernized, maintainable, and community-driven approach to API reference generation.

Background

▶ Watch: Introduction to SIG Docs API reference generation problem (0:10)

The Kubernetes project, like any complex software system, relies heavily on comprehensive and accurate documentation. Among the most crucial components are the API reference docs for Kubernetes itself and its command-line interface, kubectl. These documents serve as the authoritative source for understanding the structure, parameters, and behavior of the Kubernetes API, essential for both automated systems and human operators. However, the process by which these vital references are generated has evolved into a significant bottleneck and source of technical debt.

Historically, the generation of these API reference documents has been managed by a custom-written toolchain, born out of necessity in the project's earlier days. Over time, as Kubernetes grew in complexity and scale, this toolchain became increasingly fragile and difficult to maintain. The speakers detailed that the original intent was for the release team to handle this generation as part of each Kubernetes release cycle. However, the current state of the process makes this practically impossible, pushing the responsibility onto a dwindling number of highly specialized maintainers.

The problem isn't just about outdated code; it's about a complete breakdown of documentation and maintainability around the tool itself. The process, as it currently stands, is largely undocumented, relying on tribal knowledge possessed by only one or two individuals. This creates an extreme single point of failure for a critical project output. New contributors, who often gravitate towards SIG Docs as an entry point into the Kubernetes community, are completely blocked from assisting with this fundamental task due to the system's opacity and unreliability. The irony, as highlighted by the SIG Docs leads, is profound: a special interest group dedicated to documentation finds itself unable to document or easily use its own core tooling. This foundational issue compromises the project's long-term sustainability and its ability to onboard and empower new community members effectively.

Key Findings

▶ Watch: Summary of the tool's severe undocumented failure modes (4:00)

The talk meticulously dissected the current, deeply flawed process for generating Kubernetes and kubectl API reference documentation, revealing a series of critical findings that underscore the urgent need for modernization. The existing system is characterized by its fragility, manual overhead, and complete lack of maintainability, presenting a significant impediment to the Kubernetes project.

Firstly, the entire process must be executed twice: once for Kubernetes and once for kubectl, doubling the manual effort and potential for errors. The initial setup itself is cumbersome, requiring a specific local workspace, correct Go paths, and local clones of several repositories, including the K website repo and the KK repo (which mysteriously needs renaming). This early stage is often complicated by "confusing build variables" that repeatedly cause issues.

A major discovery highlighted by the speakers is the convoluted architecture of the generation tool: a bash script that calls a Python script, which in turn calls "several Go scripts." This layered and interdependent scripting environment is described as a "rat king," making debugging and understanding the flow exceptionally difficult.

The process consistently encounters specific failure points:

• make copy API command: This step, intended to copy API definitions and generate two crucial files, "tends to fail so hard... every single time." Despite its apparent simplicity, it frequently "barfs," requiring undocumented manual intervention.
• Manual Markdown Modification: For every single release, and for both Kubernetes and kubectl, maintainers must manually open a markdown file and change the "exact same thing every time." This repetitive, non-automatable task is a glaring example of inefficiency and a prime candidate for automation or flag-based configuration.
• Unreliable Local Testing: Attempting to locally test the generated API reference "will fail every single time." What once might have been a functional testing mechanism has become "irreversibly" broken due to increasing complexity.

Beyond these specific command failures, the overarching problem is the profound lack of documentation. The failure modes are entirely undocumented, and the tool itself provides "not helpful or coherent error messages." Maintainers are forced to "read a stack trace" to even locate where a failure occurred, with no guidance on why it failed or how to fix it. This frequently necessitates "manual fixes to generated files," a practice universally discouraged in software development.

The culmination of these issues is that the entire API reference generation process is "essentially only doable by one or two people"—the original authors and maintainers. Even the SIG Docs technical leads, Kat Cosgrove and Xander Grzywinski, admitted they cannot reliably execute the process, despite their deep involvement in the SIG. This creates an extreme single point of failure for a critical component of Kubernetes documentation, preventing the release team from fulfilling their responsibilities and completely blocking new contributors from engaging with this essential area. The code itself is "almost entirely undocumented," directly contradicting the ideals and purpose of SIG Docs. The analytical data from the Kubernetes website confirms that the reference docs are a "hugely utilized part of the documentation," making the fragility of their generation process a critical concern for the entire project.

Technical Deep Dive

▶ Watch: Goals: Making API reference generation accessible for contributors (5:20)

The technical intricacies of the existing Kubernetes API reference generation process reveal a system deeply entrenched in legacy practices and undocumented complexity. At its core, the system's architecture is a multi-layered script execution chain, described as a "bash script that calls a Python script that calls several Go scripts." This polyglot scripting pipeline significantly complicates maintainability, debugging, and understanding the overall flow, as developers must navigate different language contexts and execution environments.

The process begins with a series of foundational setup steps. A maintainer must first establish a local workspace and correctly configure their Go paths. Subsequently, several critical repositories need to be cloned locally. These include the primary Kubernetes repository (KK repo), the Kubernetes website repository (K website repo), and other related documentation sources. A peculiar, undocumented requirement is the renaming of the KK repo for reasons unknown to the current maintainers, signaling an early point of friction and a lack of clarity in the system's design. The configuration is further complicated by "confusing build variables" that are prone to misconfiguration and frequently lead to errors.

Once the environment is set up, the process proceeds to fetch the OpenAPI specification for the Kubernetes API. The OpenAPI spec (formerly Swagger) is a language-agnostic, human-readable description format for REST APIs, designed to enable automated tooling. Ironically, despite the Kubernetes API being defined via OpenAPI, the existing generation tool is largely a custom solution that doesn't fully leverage the broader OpenAPI ecosystem.

A consistent point of failure arises during the make copy API command. This operation is designed to copy API definitions and generate specific files necessary for the reference documentation. However, it reliably "barfs" or fails, requiring manual intervention from the few experts who understand its hidden quirks. The specific nature of these failures—whether due to environment mismatches, unexpected API changes, or internal script logic errors—remains largely undocumented and uncommunicated through coherent error messages, forcing maintainers to resort to deciphering raw stack traces.

Another critical, and technically avoidable, step is the manual modification of markdown files. For every new Kubernetes release, and for both the Kubernetes and kubectl documentation, specific changes must be manually applied to generated markdown files. This indicates a hardcoded assumption or a limitation in the generation logic that prevents dynamic adjustment, forcing a repetitive, error-prone human step into an otherwise automated pipeline. This manual intervention is a prime example of technical debt, as it could likely be handled by a conditional flag or a more sophisticated templating mechanism within the generation scripts.

Finally, the ability to locally test the generated API reference documentation is completely broken. While such a feature is standard for any documentation generation pipeline, the current system "starts to fall apart pretty irreversibly" when attempting to update, build, and then test the output. This means that errors in the generated documentation might only be discovered after deployment, or worse, not at all, compromising the quality and accuracy of the public-facing resources.

The fundamental technical issue is that this entire custom-written tool is effectively a black box. Its code is "almost entirely undocumented," and its operational quirks are passed down through oral tradition rather than formal documentation. This lack of transparency and maintainability stands in stark contrast to modern software engineering practices, especially for a project as critical and community-driven as Kubernetes. The speakers explicitly noted that the existence of a robust OpenAPI ecosystem offers a clear path forward for a future rewrite, leveraging existing, well-maintained tools instead of continuing with a bespoke, fragile solution.

Demo / Proof of Concept

▶ Watch: Call for help: Improving the existing generation code (6:30)

This talk did not feature a demonstration or proof of concept for a new, functional system. Instead, it served as a candid exposition of the severe challenges faced by the SIG Docs team in their existing API reference generation process. The presentation itself effectively demonstrated the brokenness of the current system through a detailed, step-by-step walkthrough of its numerous failure points and manual interventions. The "demo" was, in essence, a live accounting of the technical debt and operational hurdles that plague the maintainers, emphasizing the urgent need for community intervention rather than showcasing a solution. The speakers' narrative highlighted how attempts to perform the documented process invariably lead to failure, underscoring the lack of a reliable, repeatable workflow.

Defensive Implications

▶ Watch: Call for help: Full rewrite using OpenAPI tooling (8:00)

While this KubeCon talk is not about security vulnerabilities, the "defensive implications" within the context of open-source project health and sustainability are profound. For the Kubernetes project and its maintainers, "defending" against the current state of API reference generation means addressing critical points of fragility, improving maintainability, and ensuring the longevity of essential project components.

1. Mitigating Single Points of Failure: The most immediate implication is the extreme single point of failure represented by the one or two individuals who possess the "deep magic" to operate the current generation tool. Should these individuals become unavailable, the project risks being unable to generate up-to-date API reference documentation, a catastrophic outcome for users and developers. Defensively, the project must work to distribute this knowledge and capability across a broader base of contributors, making the process resilient to individual departures.

1. Reducing Technical Debt: The "rat king" of bash, Python, and Go scripts, coupled with undocumented code and processes, constitutes significant technical debt. This debt slows down development, introduces unpredictable failures, and consumes valuable maintainer time that could be spent on more impactful work. A defensive strategy involves actively prioritizing the reduction of this technical debt through refactoring, documentation, and ultimately, a rewrite.

1. Enhancing Project Inclusivity and Onboarding: The inability of new contributors to engage with this fundamental aspect of the documentation process is a defensive failure against community growth. A healthy open-source project thrives on new blood and diverse contributions. Modernizing the API reference generation process, making it accessible and understandable, is crucial for fostering a more welcoming environment for new contributors and strengthening the long-term vitality of SIG Docs and the Kubernetes project as a whole.

1. Ensuring Documentation Quality and Timeliness: The current error-prone and manual process inherently risks the accuracy and timeliness of the API reference documentation. Outdated or incorrect documentation can lead to misconfigurations, errors in development, and frustration for users. Defensively, the project needs a robust, automated, and reliable generation pipeline to ensure that users always have access to the most precise and current API references, directly supporting the usability and adoption of Kubernetes.

1. Leveraging Existing Ecosystems: The fact that the Kubernetes API is defined by an OpenAPI spec but the generation tool is entirely custom-written represents a missed opportunity. Defensively, the project should migrate towards leveraging the mature OpenAPI ecosystem tooling. This would provide a more standardized, well-documented, and community-supported foundation, offloading maintenance burden and benefiting from ongoing improvements by a wider community of developers.

To address these defensive implications, the speakers outlined clear calls to action:

• Immediate Improvements: Contributors can help by adding comments to the existing Go, Python, and bash scripts, and by cleaning up the existing code to reduce the complexity of the script calls.
• Strategic Rewrite: For a longer-term solution, the project aims for a full rewrite, with contributors encouraged to join SIG Docs meetings to help design the new architecture, specifically focusing on how to integrate and utilize existing OpenAPI tooling.

Ultimately, "defending" the Kubernetes API reference generation means transforming it from a fragile, expert-dependent bottleneck into a robust, automated, and community-maintainable system, thereby securing a vital component of the project's success and ensuring its continued growth.

Key Takeaways

• The current process for generating Kubernetes and kubectl API reference documentation is critically broken, highly manual, error-prone, and unsustainable.
• The generation tool is a complex, undocumented "rat king" of bash, Python, and Go scripts, making it inaccessible to most contributors.
• Specific failure points include recurring errors during API copy operations (make copy API), mandatory manual markdown edits for every release, and completely unreliable local testing.
• The entire process is currently only executable by one or two highly specialized individuals, creating an extreme single point of failure and preventing the release team and new contributors from participating.
• SIG Docs is actively seeking community help to both improve the existing system (e.g., adding code comments, simplifying script calls) and, ideally, to undertake a full rewrite leveraging the robust OpenAPI ecosystem tooling.
• Modernizing this critical documentation generation process is essential for the long-term health, maintainability, and community engagement of the Kubernetes project.

About the Speaker(s)

Kat Cosgrove is a SIG Docs Technical Lead, working for Wayland Utani. Her involvement in the Kubernetes SIG Docs team highlights her commitment to improving the documentation and maintainability of critical open-source projects.

Xander Grzywinski is also a SIG Docs Technical Lead, currently employed by Shinra. As a co-lead of SIG Docs, Xander shares the responsibility for guiding the documentation efforts within the Kubernetes project and advocating for necessary technical improvements.