Use a minimal initrd to switch to the full initrd stored in /usr

The growth of binaries over time and the inclusion of new features
filled the available boot partition space, so that the kernel+initrd
almost couldn't fit twice anymore as required for updates. We employed
workarounds such as wrapper scripts for ignition, afterburn and other
binaries so that they are loaded from /usr. However, this was still not
enough and we would have to do the same for (network) kernel modules and
firmware. To avoid making this ever more complex we can use a dedicated
initrd focused on loading the full initrd from /usr and then this full
initrd can use dracut as before and even drop all the workarounds we
accumulated.

Generate a minimal initrd to use instead of the full bootengine initrd.
The bootengine initrd gets stored as squashfs on /usr. The minimal
initrd still includes the early_cpio for amd64 microcode updates.
We have a fixed list of modules or module directories to include, only
focused on loading /usr and any emergency console interaction. This
requires also checking for module dependencies to copy over.
The busybox, veritysetup, and kmod binaries are needed and get their
required libraries resolved and copied over. They are not static and
use shared libraries which should be ok for now. The resulting vmlinuz
file is 27 MB for amd64, down from ~60 MB, so we have enough room to
include more kernel modules and so on for the next years while we also
grow the boot partition and wait for users to redeploy until we can rely
on a larger boot partition and eventually drop the minimal initrd again.

Pulls in flatcar/bootengine#110 for the
minimal initrd script and flatcar/seismograph#12
for making the device mapper discovery for the "rootdev" command more
reliable.

This also requied a backport of a kernel patch from 2017 that exposes
the PARTUUID in the /sys uevent file.

Co-authored-by: James Le Cuirot <[email protected]>
Signed-off-by: Kai Lueke <[email protected]>

Author: pothos

build_library/build_image_util.sh

@@ -582,6 +582,8 @@ finish_image() {
   local image_initrd_contents="${11}"
   local image_initrd_contents_wtd="${12}"
   local image_disk_space_usage="${13}"
+  local image_realinitrd_contents="${14}"
+  local image_realinitrd_contents_wtd="${15}"
 
   local install_grub=0
   local disk_img="${BUILD_DIR}/${image_name}"
@@ -877,6 +879,20 @@ EOF
       rm -rf "${BUILD_DIR}/tmp_initrd_contents"
   fi
 
+  if [[ -n ${image_realinitrd_contents} || -n ${image_realinitrd_contents_wtd} ]]; then
+        mkdir -p "${BUILD_DIR}/tmp_initrd_contents"
+        sudo mount "${root_fs_dir}/usr/lib/flatcar/bootengine.img" "${BUILD_DIR}/tmp_initrd_contents"
+        if [[ -n ${image_realinitrd_contents} ]]; then
+            write_contents "${BUILD_DIR}/tmp_initrd_contents" "${BUILD_DIR}/${image_realinitrd_contents}"
+        fi
+
+        if [[ -n ${image_realinitrd_contents_wtd} ]]; then
+            write_contents_with_technical_details "${BUILD_DIR}/tmp_initrd_contents" "${BUILD_DIR}/${image_realinitrd_contents_wtd}"
+        fi
+        sudo umount "${BUILD_DIR}/tmp_initrd_contents"
+        rm -rf "${BUILD_DIR}/tmp_initrd_contents"
+    fi
+
   if [[ -n "${image_disk_space_usage}" ]]; then
       write_disk_space_usage "${root_fs_dir}" "${BUILD_DIR}/${image_disk_space_usage}"
   fi

build_library/prod_image_util.sh

@@ -83,6 +83,8 @@ create_prod_image() {
   local image_initrd_contents="${image_name%.bin}_initrd_contents.txt"
   local image_initrd_contents_wtd="${image_name%.bin}_initrd_contents_wtd.txt"
   local image_disk_usage="${image_name%.bin}_disk_usage.txt"
+  local image_realinitrd_contents="${image_name%.bin}_realinitrd_contents.txt"
+  local image_realinitrd_contents_wtd="${image_name%.bin}_realinitrd_contents_wtd.txt"
   local image_sysext_base="${image_name%.bin}_sysext.squashfs"
 
   start_image "${image_name}" "${disk_layout}" "${root_fs_dir}" "${update_group}"
@@ -180,7 +182,9 @@ EOF
       "${image_kconfig}" \
       "${image_initrd_contents}" \
       "${image_initrd_contents_wtd}" \
-      "${image_disk_usage}"
+      "${image_disk_usage}" \
+      "${image_realinitrd_contents}" \
+      "${image_realinitrd_contents_wtd}"
 
   # Official builds will sign and upload these files later, so remove them to
   # prevent them from being uploaded now.

changelog/changes/2025-09-19-minimal-initrd.md

@@ -0,0 +1 @@
+- Reduced the kernel+initrd size on `/boot` by half. Flatcar now uses a minimal first stage initrd just to access the `/usr` partition and then switches to the full initrd that does the full system preparation as before. Since this means that the set of kernel modules available in the first initrd is reduced, please report any impact.

ci-automation/image_changes.sh

@@ -729,6 +729,15 @@ function print_image_reports() {
     echo "Note that vmlinuz-a also contains the kernel code, which might have changed too, so the reported difference does not accurately describe the change in initrd."
     echo
 
+    yell "Real/full init ramdisk (bootengine.img) differences compared to ${previous_version_description}"
+    underline "Real/full init ramdisk (bootengine.img) file changes, compared to ${previous_version_description}:"
+    env \
+        "${package_diff_env[@]}" FILE=flatcar_production_image_realinitrd_contents.txt FILESONLY=1 CUTKERNEL=1 \
+        "${flatcar_build_scripts_repo}/package-diff" "${package_diff_params[@]}" 2>&1 || true
+
+    underline "Real/full init ramdisk (bootengine.img) file size changes, compared to ${previous_version_description}:"
+    "${size_changes_invocation[@]}" "${size_change_report_params[@]/%/:realinitrd-wtd}" 2>&1 || true
+
     local base_sysext
     for base_sysext in "${base_sysexts[@]}"; do
         yell "Base sysext ${base_sysext} changes compared to ${previous_version_description}"

sdk_container/src/third_party/coreos-overlay/sys-apps/seismograph/seismograph-9999.ebuild

@@ -7,7 +7,7 @@ EGIT_REPO_URI="https://github.com/flatcar/seismograph.git"
 if [[ "${PV}" == 9999 ]]; then
 	KEYWORDS="~amd64 ~arm ~arm64 ~x86"
 else
-	EGIT_COMMIT="e32ac4d13ca44333dc77e5872dbf23f964b6f1e2" # main
+	EGIT_COMMIT="231f8b31c576133f75151d34cb90890bfaf15ebe" # main
 	KEYWORDS="amd64 arm arm64 x86"
 fi
 

sdk_container/src/third_party/coreos-overlay/sys-kernel/bootengine/bootengine-9999.ebuild

@@ -7,7 +7,8 @@ EGIT_REPO_URI="https://github.com/flatcar/bootengine.git"
 if [[ "${PV}" == 9999 ]]; then
 	KEYWORDS="~amd64 ~arm ~arm64 ~x86"
 else
-	EGIT_COMMIT="daf43bf9c1ca45bf1a43566c3a6f96ec0cb44a36" # flatcar-master
+	#EGIT_COMMIT="daf43bf9c1ca45bf1a43566c3a6f96ec0cb44a36" # flatcar-master
+	EGIT_BRANCH="kai/initrd-in-usr"
 	KEYWORDS="amd64 arm arm64 x86"
 fi
 
@@ -23,6 +24,7 @@ src_install() {
 	insinto /usr/lib/dracut/modules.d/
 	doins -r dracut/.
 	dosbin update-bootengine
+	dosbin minimal-init
 
 	# must be executable since dracut's install scripts just
 	# re-use existing filesystem permissions during initrd creation.

sdk_container/src/third_party/coreos-overlay/sys-kernel/coreos-kernel/coreos-kernel-6.12.51.ebuild

@@ -26,6 +26,7 @@ DEPEND="
 	coreos-base/coreos-init:=
 	sys-apps/azure-vm-utils[dracut]
 	sys-apps/baselayout
+	sys-apps/busybox
 	sys-apps/coreutils
 	sys-apps/findutils
 	sys-apps/grep
@@ -72,6 +73,17 @@ src_prepare() {
 	use amd64 && config_update "CONFIG_EXTRA_FIRMWARE_DIR=\"${fw_dir}\""
 }
 
+copy_in() {
+  # Simple setup, assume we only have /lib64 to care about
+  cp "${ESYSROOT}"/usr/"$1" ./"$1" || die
+  local LIBFILE
+  for LIBFILE in $(patchelf --print-needed ./"$1"); do
+    if [[ ! -e ./lib64/${LIBFILE} ]]; then
+      copy_in /lib64/"${LIBFILE}"
+    fi
+  done
+}
+
 src_compile() {
 	local BE_ARGS=()
 
@@ -89,6 +101,59 @@ src_compile() {
 
 	tc-export PKG_CONFIG
 	"${ESYSROOT}"/usr/bin/update-bootengine -k "${KV_FULL}" -o "${S}"/build/bootengine.cpio "${BE_ARGS[@]}" || die
+	# Copy full initrd over to /usr as filesystem image
+	mkdir "${S}"/build/bootengine || die
+	pushd "${S}"/build/bootengine || die
+	sudo lsinitrd --kver SILENCEERROR --unpack "${S}"/build/bootengine.cpio || die
+	sudo mksquashfs . "${S}"/build/bootengine.img -noappend -xattrs-exclude ^btrfs. || die
+	popd || die
+	# Create minimal initrd
+	if use amd64; then
+		mkdir "${S}"/build/early-cpio || die
+		pushd "${S}"/build/early-cpio || die
+		sudo lsinitrd --kver SILENCEERROR --unpackearly "${S}"/build/bootengine.cpio || die
+		# Recreate to only contain the early cpio for microcode
+		find . -print0 | cpio --null --create --verbose --format=newc > "${S}"/build/bootengine.cpio || die
+		# Debug: List contents after recreation
+		cpio -t < "${S}"/build/bootengine.cpio
+		popd || die
+	else
+		# No early cpio, drop full initrd
+		> "${S}"/build/bootengine.cpio
+	fi
+	mkdir "${S}"/build/minimal || die
+	pushd "${S}"/build/minimal || die
+	mkdir -p {etc,dev,proc,sys,dev,usr/bin,usr/lib64,realinit,sysusr/usr} || die
+	ln -s usr/bin bin || die
+	ln -s usr/bin sbin || die
+	ln -s bin usr/sbin || die
+	ln -s usr/lib64 lib || die
+	ln -s usr/lib64 lib64 || die
+	ln -s lib64 usr/lib || die
+	mkdir -p lib/modules/"${KV_FULL}"/ || die
+	# Instead from ESYSROOT we can also copy kernel modules from the dracut pre-selection
+	cp "${S}"/build/bootengine/usr/lib/modules/"${KV_FULL}"/modules.* lib/modules/"${KV_FULL}"/ || die
+	mkdir -p lib/modprobe.d/ || die
+	cp "${S}"/build/bootengine/lib/modprobe.d/* lib/modprobe.d/ || die
+	# Only include modules related to mounting /usr and for interacting with the emergency console
+	pushd "${S}/build/bootengine/usr/lib/modules/${KV_FULL}" || die
+	find kernel/drivers/{ata,block,hid,hv,input/serio,mmc,nvme,pci,scsi,usb} kernel/fs/{btrfs,overlayfs,squashfs} kernel/security/keys -name "*.ko.*" -printf "%f\0" | xargs --null sudo DRACUT_NO_XATTR=1 "${BROOT}"/usr/lib/dracut/dracut-install --destrootdir "${S}"/build/minimal --kerneldir . --sysrootdir "${S}"/build/bootengine/ --firmwaredirs "${S}"/build/bootengine/usr/lib/firmware --module dm-verity dm-mod virtio_console || die
+	popd || die
+	echo '$MODALIAS=.*	0:0 660 @/sbin/modprobe "$MODALIAS"' > ./etc/mdev.conf || die
+	# We can't use busybox's modprobe because it doesn't support the globs in module.alias, breaking module loading
+	sudo DRACUT_NO_XATTR=1 "${BROOT}"/usr/lib/dracut/dracut-install --destrootdir . --sysrootdir "${ESYSROOT}" --ldd /bin/veritysetup /bin/dmsetup /bin/busybox /sbin/modprobe || die
+	cp -a "${ESYSROOT}"/usr/bin/minimal-init ./init || die
+	# Make it easier to debug by not relying too much on the first commands
+	ln -s busybox ./bin/sh || die
+	mknod ./dev/console c 5 1 || die
+	mknod ./dev/null c 1 3 || die
+	mknod ./dev/tty c 5 0 || die
+	mknod ./dev/urandom c 1 9 || die
+	mknod ./dev/random c 1 8 || die
+	mknod ./dev/zero c 1 5 || die
+	# No compression because CONFIG_INITRAMFS_COMPRESSION_XZ should take care of it
+	find . -print0 | cpio --null --create --verbose --format=newc >> "${S}"/build/bootengine.cpio || die
+	popd || die
 	kmake "$(kernel_target)"
 
 	# sanity check :)
@@ -111,4 +176,7 @@ src_install() {
 	# For easy access to vdso debug symbols in gdb:
 	#   set debug-file-directory /usr/lib/debug/usr/lib/modules/${KV_FULL}/vdso/
 	kmake INSTALL_MOD_PATH="${ED}/usr/lib/debug/usr" vdso_install
+
+	insinto "/usr/lib/flatcar"
+	doins build/bootengine.img
 }

sdk_container/src/third_party/coreos-overlay/sys-kernel/coreos-sources/coreos-sources-6.12.51.ebuild

@@ -43,4 +43,5 @@ UNIPATCH_LIST="
 	${PATCH_DIR}/z0006-mtd-disable-slram-and-phram-when-locked-down.patch \
 	${PATCH_DIR}/z0007-arm64-add-kernel-config-option-to-lock-down-when.patch \
 	${PATCH_DIR}/z0008-tools-hv-fix-cross-compilation-for-ARM64.patch \
+	${PATCH_DIR}/z0009-block-add-partition-uuid-into-uevent.patch \
 "

sdk_container/src/third_party/coreos-overlay/sys-kernel/coreos-sources/files/6.12/z0009-block-add-partition-uuid-into-uevent.patch

@@ -0,0 +1,36 @@
+From 758737d86f8a2d74c0fa9f8b2523fa7fd1e0d0aa Mon Sep 17 00:00:00 2001
+From: Konstantin Khlebnikov <[email protected]>
+Date: Fri, 4 Oct 2024 17:13:43 -0700
+Subject: [PATCH] block: add partition uuid into uevent as "PARTUUID"
+
+Both most common formats have uuid in addition to partition name:
+GPT: standard uuid xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+DOS: 4 byte disk signature and 1 byte partition xxxxxxxx-xx
+
+Tools from util-linux use the same notation for them.
+
+Signed-off-by: Konstantin Khlebnikov <[email protected]>
+Reviewed-by: Kyle Fortin <[email protected]>
+[dianders: rebased to modern kernels]
+Signed-off-by: Douglas Anderson <[email protected]>
+Signed-off-by: Douglas Anderson <[email protected]>
+Reviewed-by: Christoph Hellwig <[email protected]>
+Link: https://lore.kernel.org/r/20241004171340.v2.1.I938c91d10e454e841fdf5d64499a8ae8514dc004@changeid
+Signed-off-by: Jens Axboe <[email protected]>
+---
+ block/partitions/core.c | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/block/partitions/core.c b/block/partitions/core.c
+index cdad05f9764768..815ed33caa1b86 100644
+--- a/block/partitions/core.c
++++ b/block/partitions/core.c
+@@ -256,6 +256,8 @@ static int part_uevent(const struct device *dev, struct kobj_uevent_env *env)
+ 	add_uevent_var(env, "PARTN=%u", bdev_partno(part));
+ 	if (part->bd_meta_info && part->bd_meta_info->volname[0])
+ 		add_uevent_var(env, "PARTNAME=%s", part->bd_meta_info->volname);
++	if (part->bd_meta_info && part->bd_meta_info->uuid[0])
++		add_uevent_var(env, "PARTUUID=%s", part->bd_meta_info->uuid);
+ 	return 0;
+ }
+