Mitigation Strategy for Completing Adopting usrmerge filesystem layout for /lib

What will happen the day of the transition

The transition is planned to start during normal business hours (U.S) on June 23, 2025, with the following action plan: 
 

• Affected packages will be updated to start installing binaries in /usr/lib, instead of /lib.

• Our base filesystem layout will be updated to symlink /lib and /lib64 to /usr/lib and /usr/lib/64 to /usr/lib

Once the transition is complete, and all packages have been updated, we expect our images to be rebuilt as scheduled on a nightly basis. New images should be available overnight. Please ensure any custom (e.g. jfrog or Artifactory) Docker registry or package mirrors are updated accordingly.

Please note that this plan may be subject to change should there be any unforeseen circumstances. Should this happen, we will communicate with you promptly.

How do I know if I may be impacted?

You are likely to be impacted in one of the following situations:

Installing packages and replacing your root folder

If you are installing packages into a root folder to later copy/replace the root directory of your runtime image with it.

• Why will this impact me? Because the usrmerge transition changes the base filesystem layout, newer usrmerged packages may be incompatible with the old filesystem layout (or vice versa).

   

• How will this impact me? You may experience one of the following two situations:

  • Images fail to build altogether.

  • Or images may build successfully, but may have degraded operation (e.g. won’t be able to find binaries or libraries in expected directories).

   

• How do I mitigate or resolve this issue? Please review the Appendix - Installing packages and replacing your root folder.

Installing post-usrmerge packages in pre-usrmerge image

If you are pinning a package version and attempting to force the installation of a post usrsbin-merge or lib-merge package in an image that’s created before the transition.

• Why will this impact me? Because post-usrmerge packages are not compatible with the base image filesystem layout.

• How will this impact me? Images may normally fail to build.

• How do I mitigate or resolve this? Do not pin dependency versions, specially on packages with a revision of 50 or higher (e.g. apk add package=<version>-r50)

Installing pre-usrmerge packages in post-usrmerge image

If you are pinning a package version and  attempting to force the installation of a pre usrsbin-merge or lib-merge package in an image that’s created after the transition.

• Why will this impact me? Because pre-usrmerge packages may overwrite symlinks of post-usrmerge images or install binaries in locations that are no longer supported.

• How will this impact me? You may experience one of the following two situations:

  • The image may be successfully built, but operation may be degraded, as a pre-usrmerge package may install binaries/libraries in unexpected locations and cause applications to fail to execute correctly.

• How do I mitigate or resolve this? Do not pin dependency versions with a revision lower than 50 (e.g. apk add package=<version>-r49)

Installing libraries in /lib

If you are installing libraries directly into /lib and attempt to create /lib as a directory.

• Why will this impact me? Because /lib will be a symlink to /usr/lib and cannot be created as a directory (which your installs may attempt to do).

• How will this impact me?

  • Images may fail to build.

  • Images may NOT fail to build but will experience degraded operations (e.g. they may not be able to find binaries/libraries in the expected locations and fail to execute applications).

• How do I mitigate or resolve this? Install libraries in /usr/lib.

Using apk –initdb in a post-usrmerge image

If you use apk’s –initdb flag, the version (>= v2.14.10-3) of apk-tools in Wolfi has been updated to support /lib as a symlink and will correctly create the apk DB in /usr/lib. If however, apk-tools is older than v2.14.10-3 or you are using the new version on a pre-usrmerge image, it would create /lib as a directory instead.

• Why will this impact me? If /lib is a directory, the new wolfi-baselayout will fail to install and cause all other package installs to fail.

• How will this impact me?

  • Images may fail to build.

• How do I mitigate or resolve this?

  • Start from new images and don’t pin the apk-tools package.

Using apko and melange to build images or packages

If you are using Chainguard’s tools for building packages (melange) or images (apko or bazel rules_apko), you will need to upgrade these tools to the latest versions to support the post usrmerge lib world. Specifically, versions at or greater than:

• apko v0.27.6

• melange v0.23.15

• rules_apko for bazel v1.5.3

   

• Why will this impact me? These tools contain important changes to support the transition to a usrmerge /lib world. Apk’s DB is now located in /usr/lib/apk instead of /lib/apk. These tools will work in all filesystem layout scenarios (old or new). However, if you are building packages or images based on a post-usrmerge /lib world, you need to use the new versions.
   

• How will this impact me?

  • Image or package builds may fail with errors that look like the following or similar:

ERRO failed to build package: unable to build guest: unable to generate image: installing apk packages: installing packages: installing wolfi-baselayout (ver:20230201-r21 arch:x86_64): unable to install files for pkg wolfi-baselayout: unable to install symlink from lib -> usr/lib: symlink usr/lib /tmp/melange-guest-1421142584/lib: file exists

• How do I mitigate or resolve this? Update your tools to supported versions in your workflow or pipeline.

Appendix

Installing packages and replacing your root folder

If your Dockerfile looks like this, you will be impacted by the transition.

FROM cgr.dev/chainguard-private/jdk:openjdk-21.0.6-r1-dev AS builder (0)
[...]
ARG PACKAGES="apk-tools shadow curl busybox" (1)
RUN mkdir -p /newroot/etc && \
   cp -a /etc/apk /newroot/etc && \
   apk update && \
   apk add --no-cache --initdb --root /newroot $PACKAGES && \ (2)
   [...]



FROM cgr.dev/chainguard-private/jdk:openjdk-21.0.6-r1 (3)
USER root
COPY --chown=65532:65532 --from=builder /newroot / (4)
[...]

Why may this impact me?

Because the usrmerge transition changes the base filesystem layout, newer usrmerged  packages may be incompatible with the old filesystem layout (or vice versa).

Whenever you install packages inside a “root” folder (e.g. /newroot) (2), there’s no assurance that they are compatible with those packages within the image or the “base” layout itself. The problem, however, may not be noticeable until you attempt to copy (or replace) the contents of the root of an image (e.g. / ) (4). In this case, the above config will try to install lib-merged packages, while the base filesystem layout of the image remains the same.
 

Additionally, if the version of apk-tools is older than v2.14.10-3, the –initdb option will conflict with the symlinked /lib.
 

NOTE: It is assumed that (0) and (3) can be any pre-usrmerge (usrsbin-merge, lib-merge) image, regardless of the tag/epoch.

Mitigation Strategies

Mitigation Strategy 1 - Pinning pre-usrmerge (lib) packages

While the transition is in progress, you can prevent apk from installing older, pre-usrmerge (lib-merge) packages while still using pre-usrmerge base images (0) and (3). You can update (1) with any of:

• For the transition of /lib: wolfi-baselayout !merged-lib

For example, take a snippet from the config above, this would mean customers would need to add the following:

[...]

ARG PACKAGES="curl foo bar wolfi-baselayout !merged-lib" (1)

[...]

After the transition is complete

After the transition is complete, it is recommended that you:

• Update (0) and (3) to use the post-usrmerge images.

• AND update (1) to remove the added dependencies: wolfi-baselayout !merged-lib.

Mitigation Strategy 2 - Installing packages via the chroot method

You can change the way you build your images. Instead of installing packages into a “newroot” folder and copying/replacing them into the runtime image, you can leverage the use of a chroot.
 

The following is a basic example showing the use of a chroot, which ensures that post-usrmerge packages are not installed in pre-usrmerge images.

FROM cgr.dev/chainguard-private/jdk:openjdk-21.0.6-r1 AS base (1)

FROM cgr.dev/chainguard-private/jdk:openjdk-21.0.6-r1-dev AS build (2)
USER 0
COPY --from=base / /base-chroot (3)
ARG PACKAGES="apk-tools shadow curl busybox"
RUN apk add --root /base-chroot $PACKAGES (4)

FROM scratch AS custom-base
COPY --from=build /base-chroot /

NOTE: This may not work for all use cases, and more details can be provided upon request.

Mitigation Strategy 3 - Custom Assembly

If you use Chainguard Custom Assembly you will not be affected by these changes as it's a supported mechanism for adding packages to images. Custom Assembly allows you to create images that have different packages than those that are available in the image by default. Custom Assembly ensures that image builds are not impacted by package interdependence issues, like those experienced with the usrmerge transition or other issues such as those experienced with curl and libcurl.

How to reach out for support

Please contact your Technical Success Manager should you require more information. If you need immediate help, reach out through our support portal at support.chainguard.dev or via email at support@chainguard.dev.