Reproducible Builds

At Rugix, we are huge proponents of reproducible builds.

“A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.” – https://reproducible-builds.org/docs/definition/

We aim to enable reproducible builds of Rugix itself as well as of system images built with Rugix.

Note that this is a work-in-progress effort and we are not fully there yet.

Source Date

Rugix Bakery supports a --source-date option which allows you to pin the source date of the build to a specific date and time (formatted according to RFC 3339). If you do not specify this option, Rugix Bakery will use the current date and time. In any case, Rugix Bakery will clamp the modification time of all files to the source date. The source date is also used to set the SOURCE_DATE_EPOCH environment variable in the build environment. Bakery also passes this value to filesystem construction tools so that creation timestamps do not depend on the time of the build.

Reproducibility of Rugix

Given a commit hash of Rugix, everyone should be able to reproduce all Rugix-related binaries that may end up in images as well as the Docker image for Rugix Bakery. This includes binaries of third-party software, e.g., Grub and U-Boot.

Reproducible builds require a known build environment. To this end, we use Debian’s official snapshots as a basis. Note that most Debian packages are also fully reproducible, so one could in principle trace everything back to its source code. Building upon Debian snapshots also has the advantage that it reduces the surface area that must be checked for Rugix, provided that one trusts Debian.

To obtain a reproducible build environment, we follow a two stage process:

• Stage 0: We build a Debian Docker image containing mmdebstrap. This image will not be reproducible as it is based on the latest version of Debian Bookworm. We are only using it to bootstrap Stage 1.
• Stage 1: Using the previously built Docker image, we bootstrap a Debian Docker image based on an official snapshot. This image will be fully reproducible and everything we install will be pinned to the respective snapshot. Thereby, all images that we derive from it, e.g., to build Rugix, Grub, or U-Boot, are also fully reproducible.

Everything else we built will be based on the reproducible Stage 1 image.

The infrastructure for reproducible builds is implemented as part of Rugix’s xtask.

Reproducibility of Images

🚧 This is work-in-progress. 🚧

Reproducibility of Rugix is a prerequisite for reproducible system images.

Filesystem Images

Bakery represents a finished layer as a directory tree after all recipes have run. It constructs ext4 filesystems directly from this tree and passes the configured source date to mkfs.ext4. Reproducible targets must also specify a stable filesystem UUID, directory hash seed, and filesystem feature set in their ext4 options, as Rugix’s default system partitions do. Given those fixed inputs, the same tree, partition size, source date, and Bakery version produce a byte-for-byte identical ext4 filesystem image.

Bakery stores intermediate layer trees as PAX tar archives. These archives preserve deployment-relevant filesystem metadata, including ownership, permission and special mode bits, hardlinks, symbolic links, sparse files, device nodes, FIFOs, POSIX ACLs, and extended attributes such as file capabilities and SELinux labels. Bakery fails the build if it cannot restore this metadata. When importing ext filesystems, Bakery rejects sockets, project quota IDs, and inode flags such as immutable or append-only that the layer format cannot represent instead of silently dropping them.

Modification times are clamped to the configured source date. Other timestamps, source inode numbers, block placement, journal contents, and free-space history are not layer semantics; they are normalized when constructing the output filesystem.

Filesystem reproducibility does not by itself make the complete disk image reproducible. Partition-table identifiers, FAT filesystems, and bootloader artifacts must also be deterministic. This guarantee applies to each ext4 partition image, not the assembled disk image, unless the selected target documents a stronger guarantee.

Furthermore, reproducibility of images requires support by the underlying Linux distribution. In particular, it requires that a distribution provides immutable snapshots of their repositories (rebuilding packages is outside the scope of Rugix).

Among the distributions supported by Rugix, only Debian officially provides immutable snapshots of their repositories. Furthermore, Debian also participates in the Reproducible Builds project’s continuous reproducibility testing of their packages. We aim to eventually support building fully reproducible Debian images based on the official snapshots.

For other distributions and base layers built with third-party tools (e.g., Yocto Project and Buildroot), we aim to make images reproducible under the assumption that the external inputs (repositories, base layer, …) do not change.