Popping podman/dockers bonnet: Unraveling the image pull process while developing for Forgejo

Erik

Gulaschprogrammiernacht 24 (GPN24) · Day 4 · ZKM Medientheater

Overview

In this insightful talk from GPN24, Erik delves into the often-obscured mechanics of how container runtimes like Docker and Podman pull images from registries. Driven by the need to develop a pull-through cache for Forgejo, a self-hostable software forge with a package registry, Erik embarked on a journey to reverse-engineer the image pull process. His motivation stemmed from a surprising lack of clear, technical documentation, often encountering generic guides or AI-generated content instead of the detailed specifications required for building a robust caching solution.

Watch on YouTube

Visual summary for Popping podman/dockers bonnet: Unraveling the image pull process while developing for Forgejo by Erik
Visual summary for Popping podman/dockers bonnet: Unraveling the image pull process while developing for Forgejo by Erik

Key moments

  1. 0:00 Introduction and motivation for understanding image pull
  2. 3:40 Why a pull-through cache is needed
  3. 6:00 Deconstructing the image pull URI and manifests
  4. 8:20 Authentication process for image registries
  5. 11:30 Full sequence: retrieving manifests and blobs
  6. 13:00 Summary and future of the pull-through cache
  7. 15:30 Q&A: Caching logic and refresh strategy

Popping podman/dockers bonnet: Unraveling the image pull process while developing for Forgejo

Speakers: Erik

Conference: GPN24

YouTube: https://www.youtube.com/watch?v=cAEzeMn7ncg

Overview

In this insightful talk from GPN24, Erik delves into the often-obscured mechanics of how container runtimes like Docker and Podman pull images from registries. Driven by the need to develop a pull-through cache for Forgejo, a self-hostable software forge with a package registry, Erik embarked on a journey to reverse-engineer the image pull process. His motivation stemmed from a surprising lack of clear, technical documentation, often encountering generic guides or AI-generated content instead of the detailed specifications required for building a robust caching solution.

The core objective of Erik's work is to enhance the resilience and efficiency of software development workflows, particularly for continuous integration (CI) systems. A pull-through cache for container images addresses critical pain points such as rate limiting, network instability, the need for offline capabilities, and navigating restrictive corporate firewalls. Beyond practical benefits, this initiative also champions the decentralization of software development, reducing reliance on large, centralized platforms—a growing concern in the open-source community.

Background

▶ Watch: Introduction and motivation for understanding image pull (0:00)

The genesis of this project lies in a common frustration faced by developers and system administrators: the difficulty in finding comprehensive, low-level documentation for seemingly fundamental operations. While tools like Docker and Podman are ubiquitous, the intricate details of their underlying protocols for fetching container images are not readily apparent. Erik highlighted a significant gap in accessible information, noting that search results often yield superficial "how-to" articles or AI-generated summaries rather than the deep technical specifications needed for custom development.

This knowledge void prompted Erik to investigate the Open Container Initiative (OCI) Distribution Specification, the de facto standard governing container registries. His goal was to develop a pull-through cache for Forgejo's existing package registry. The necessity for such a cache is multi-faceted:

  • CI System Relief: Continuous Integration pipelines frequently pull the same images, quickly hitting rate limits imposed by public registries (e.g., Docker Hub). A local cache drastically reduces external requests.
  • Network Resilience: Caching images locally provides immunity against transient network issues and allows builds to proceed even if external connectivity is lost.
  • Offline Operation: For development environments or air-gapped systems, a cache ensures that necessary images are available without an internet connection.
  • Corporate Network Evasion: Organizations with strict firewall policies often require tedious approval processes for external connections. A local cache minimizes these requirements by proxying requests.
  • Decentralization: A key philosophical driver, particularly within the Codeberg.org community where Forgejo is hosted, is to foster less dependence on large, centralized platforms, promoting a more distributed and resilient software ecosystem. This cache empowers users to control their image sources.

Key Findings

▶ Watch: Deconstructing the image pull URI and manifests (6:00)

Erik's deep dive into the container image pull process yielded several key discoveries, illuminating the complex, multi-step interaction between a client, a local cache, and a remote registry. He successfully reverse-engineered the precise sequence of HTTP requests, authentication flows, and data structures involved, which are crucial for building a functional pull-through cache.

One significant finding was the hierarchical nature of container image metadata, which Erik likened to a Merkle tree structure. While acknowledging that the analogy isn't perfectly one-to-one (as nodes hold multiple hashes rather than a single one), the core concept of data integrity verifiable through cryptographic hashes (digests) propagating up the structure holds true. This structure dictates the pull behavior, where higher-level manifests reference lower-level components, each secured by its unique SHA-256 digest.

The process begins with a series of authentication steps, followed by metadata retrieval, and finally, the fetching of actual image data. Erik uncovered that the client (Docker or Podman daemon) plays a critical role in selecting the appropriate image manifest based on its architecture and platform, rather than the registry making this decision. This understanding was vital for correctly implementing the caching logic within Forgejo, ensuring that the cache correctly serves the manifest index and allows the client to make its selection. The overall sequence, while not overtly complex once understood, involved navigating several distinct HTTP methods and parsing specific headers and JSON payloads, a process that was far from intuitively documented.

Technical Deep Dive

▶ Watch: Authentication process for image registries (8:20)

The journey to understand the container image pull process starts with a seemingly simple docker pull command, which internally translates into a sophisticated series of HTTP requests. Erik's investigation began with the basic assumption of a curl request, quickly leading to the discovery of the /v2/ endpoint as the standard entry point for container registry interactions.

The URI structure for image pulls is critical:

REGISTRY_ADDRESS/v2/NAMESPACE/ENDPOINT/REFERENCE

  • Registry Address: The hostname of the container registry (e.g., registry.example.com).
  • /v2/: A mandatory endpoint indicating compliance with the OCI Distribution Specification.
  • Namespace: The organization or user namespace for the image (e.g., my-org).
  • Endpoint: Specifies the type of resource being requested, primarily manifests or blobs.
  • Reference: Can be either an image tag (e.g., latest, 1.2.3) or a cryptographic digest (e.g., sha256:abcdef...).

Authentication Flow: OAuth 2.0 Bearer Token

The first interaction with a registry typically results in an unauthorized (HTTP 401) error. This response is not an actual failure but a prompt for authentication. The response includes a WWW-Authenticate header, which provides crucial information for obtaining a bearer token. This header specifies the realm (the token service URL), service (the container registry's service name, which can also be a URL), and scope (the permissions required, e.g., repository:my-org/my-image:pull,push).

  1. The client (or Forgejo acting as a proxy) extracts this information.
  2. A GET request is then made to the realm URL, providing the service and scope parameters.
  3. The token service returns a JSON payload containing an OAuth 2.0 bearer token. This token must then be included in the Authorization header of all subsequent requests to the registry.

Manifest Structure: Index, Manifest, Config, and Layers

Container image metadata is organized hierarchically:

  • Manifest Index (or Image Index): This is a list of references to individual manifests, each corresponding to a specific platform (e.g., linux/amd64, linux/arm64). It contains mediaType, size, digest, and platform information (including architecture and os). When a client requests an image by tag, it often receives a manifest index first.
  • Manifest: An individual manifest describes a specific image for a particular platform. It's a JSON document containing two key types of references:
  • Config: A reference to a config blob (also identified by mediaType, size, and digest). This blob contains metadata about the image's configuration, such as environment variables, entry points, and build history.
  • Layers: A list of references to layer blobs. Each layer is an immutable filesystem change, also identified by mediaType, size, and digest. These layers are typically compressed (e.g., application/vnd.docker.image.rootfs.diff.tar.gzip).

The Image Pull Sequence (as implemented in Forgejo's cache):

  1. Initial Request & Authentication: The client (docker or podman daemon) initiates a GET /v2/ request to the Forgejo instance. Forgejo returns unauthorized with WWW-Authenticate. The client then requests a bearer token from Forgejo's token service using the provided realm, service, and scope (e.g., repository:my-org/my-image:pull). Forgejo authenticates, potentially against an external registry if acting as a proxy, and issues a token.
  2. Metadata Discovery (HEAD Request): The client performs a HEAD request to GET /v2/namespace/manifests/image_tag (e.g., latest) on Forgejo, including the bearer token. Forgejo, acting as a proxy, forwards this HEAD request to the remote registry. The remote registry responds with headers, including the digest (SHA-256 sum) of the manifest associated with the image_tag. Forgejo caches this digest and returns it to the client. This step allows the client to know the exact content hash of the image it's about to pull.
  3. Manifest Index Retrieval: The client then issues a GET request for the manifest, this time using the known digest: GET /v2/namespace/manifests/sha256:digest. Forgejo forwards this to the remote registry. The remote registry might return a manifest index if the tag refers to a multi-platform image. Crucially, Erik discovered that Forgejo should simply pass this manifest index directly to the client. The client itself is responsible for parsing the index and selecting the appropriate manifest for its specific architecture (e.g., linux/amd64).
  4. Specific Manifest Retrieval: Once the client has chosen its architecture, it makes another GET request to Forgejo for the specific manifest corresponding to that architecture's digest. Forgejo, again, forwards this to the remote registry. The remote registry returns the actual manifest JSON for the chosen platform.
  5. Blob Pre-fetching and Caching: Before serving the manifest to the client, Forgejo parses the manifest to identify all associated config blobs and layer blobs by their digests. Forgejo then proactively fetches all these blobs from the remote registry. This process leverages the Docker daemon's default timeout of 5 minutes, providing ample time to download even large images. As each blob is downloaded, it is cached locally within Forgejo.
  6. Serving Cached Data: Once all blobs are cached, Forgejo serves the specific manifest to the client. The client then proceeds to request each config and layer blob individually from Forgejo, which now serves them directly from its local cache.

This intricate sequence ensures that the cache is populated with all necessary components before the client even begins to download the layers, providing a seamless and efficient pull experience. The use of SHA-256 digests throughout this process is fundamental for ensuring the integrity and immutability of the cached content.

Demo / Proof of Concept

▶ Watch: Summary and future of the pull-through cache (13:00)

Erik confirmed the existence of a working Proof of Concept (POC) for the pull-through cache for container images, stating that it is available at a provided link (which would be shared with the slides). This POC demonstrates the successful implementation of the detailed pull sequence described in the technical deep dive, allowing Forgejo to intercept, proxy, and cache container image pulls. While functional for container images, the current iteration of the POC does not yet include a cache refresh mechanism, meaning it primarily checks for the existence of files and pulls from the upstream registry only if not found.

Looking ahead, Erik expressed plans to expand the scope of this pull-through cache beyond just container images. The vision is to support a wider array of package types, including Maven, NPM, and Cargo, further solidifying Forgejo's role as a comprehensive and decentralized software forge. The ongoing discussion on Codeberg.org regarding merging this feature into Forgejo highlights its potential impact and the community's interest in its development.

Defensive Implications

▶ Watch: Q&A: Caching logic and refresh strategy (15:30)

The implementation of a pull-through cache for container images, as demonstrated by Erik's work on Forgejo, carries significant defensive implications for organizations and individual developers alike:

  1. Enhanced Supply Chain Security: By caching images locally, organizations reduce their direct reliance on external, potentially untrusted, registries. This mitigates risks associated with upstream registry compromises, denial-of-service attacks against public registries, or even accidental deletions. A local cache provides a controlled environment, allowing for local scanning and auditing of images before they are ever used in build or deployment pipelines.
  2. Improved Reliability and Availability: The cache acts as a buffer against network outages and upstream registry issues. If an external registry experiences downtime or connectivity is lost, CI/CD pipelines can continue to function by pulling images from the local cache. This is particularly crucial for critical infrastructure or environments with intermittent internet access.
  3. Rate Limit Evasion: Public registries like Docker Hub impose rate limits on image pulls. A pull-through cache significantly reduces the number of requests sent to these external registries, effectively circumventing rate limits and ensuring uninterrupted operations for large-scale CI/CD systems or numerous developers.
  4. Performance Optimization: Serving images from a local network or even the same machine drastically reduces latency compared to pulling from a remote registry over the internet. This translates to faster build times, quicker deployments, and an overall more efficient development workflow.
  5. Network Cost Reduction: For organizations that pay for egress bandwidth, a local cache can reduce internet traffic by serving frequently accessed images locally, leading to potential cost savings.
  6. Decentralization and Sovereignty: Aligning with the ethos of Forgejo, the cache promotes decentralization. It gives users greater control over their software dependencies, lessening dependence on a few dominant platforms. This is a strategic advantage for open-source projects and organizations aiming for greater autonomy and resilience against platform-specific policies or failures.
  7. Simplified Firewall Management: In corporate environments, the cache can act as a single point of egress for container image pulls, simplifying firewall rules and reducing the need for numerous, granular exceptions for various external registries.

While the current POC lacks cache refresh, its core functionality provides a robust foundation for securing and optimizing container image workflows, offering a tangible step towards more resilient and self-reliant software development.

Key Takeaways

  • Complexity Under the Hood: The seemingly simple docker pull command orchestrates a multi-step process involving HTTP HEAD and GET requests, OAuth 2.0 bearer token authentication, and the parsing of intricate JSON data structures like manifest indexes and manifests.
  • Hierarchical Data Integrity: Container image metadata and content are structured hierarchically, similar to a Merkle tree, where cryptographic SHA-256 digests ensure the integrity of each component (config, layers) and their references.
  • Client-Driven Architecture Selection: The client (Docker/Podman daemon) plays a crucial role in interpreting the manifest index and selecting the appropriate image manifest based on its specific platform and architecture.
  • Benefits of Pull-Through Caches: Implementing a local pull-through cache for container images significantly enhances CI/CD system reliability, bypasses rate limits, enables offline operations, improves performance, and reduces reliance on external registries.
  • Decentralization as a Core Value: The development of such a cache for Forgejo underscores a broader movement towards decentralizing software development infrastructure and reducing dependency on large, centralized platforms.
  • Future Expansion: The current Proof of Concept for container images is envisioned to expand its caching capabilities to other package types like Maven, NPM, and Cargo, further solidifying Forgejo's utility as a comprehensive, self-hostable forge.

About the Speaker(s)

Erik, the speaker, presented this detailed and technical talk at GPN24. His presentation highlighted his practical experience and insights gained from developing a pull-through cache for Forgejo, demonstrating a deep understanding of container image distribution protocols.

Reviews

Dr. Zero (Offensive Security Researcher) — SOLID

Competent walkthrough of OCI Distribution Spec mechanics that would help someone building their first registry proxy. The Forgejo integration is practical, but this is fundamentally 'I read the spec so you don't have to' rather than novel research.

Heather Calloway (CISO) — SOLID

A competent technical deep-dive into container image pull mechanics, useful for anyone building registry infrastructure or understanding OCI distribution internals. Limited direct security or governance implications—this is developer tooling work, not threat research or defender operations.

→ Top-rated talks at Gulaschprogrammiernacht 24 (GPN24)

All talks from Gulaschprogrammiernacht 24 (GPN24)