Mitigation Strategy for usrmerge – Chainguard

Last Updated: Mar 18, 2025

What will happen the day of the transition

The transition is planned to start during normal business hours (U.S) on March 31st, with the following action plan:

Affected packages will be updated to start installing binaries in /usr/bin, instead of /usr/sbin.
Our base filesystem layout will be updated to symlink /usr/sbin to /usr/bin.
Affected packages will be updated to install libraries in /usr/lib, instead of /lib.
Our base filesystem layout will be updated to symlink /lib to /usr/lib.

Once the transition is complete, and all packages have been updated, we expect our images to be rebuilt as scheduled on a nightly basis. New images should be available overnight. Please ensure any custom (e.g. jfrog or Artifactory) Docker registry or package mirrors are updated accordingly.

Please note that this plan may be subject to change should there be any unforeseen circumstances. Should this happen, we will communicate with you promptly.

How do I know if I may be impacted?

You are likely to be impacted in one of the following situations:

Installing packages and replacing your root folder

If you are installing packages into a root folder to later copy/replace the root directory of your runtime image with it.

Why will this impact me? Because the usrmerge transition changes the base filesystem layout, newer usrmerged packages may be incompatible with the old filesystem layout (or vice versa).

How will this impact me? You may experience one of the following two situations:
- Images fail to build altogether.
- Or images may build successfully, but may have degraded operation (e.g. won’t be able to find binaries or libraries in expected directories).

How do I mitigate or resolve this issue? Please review the Appendix - Installing packages and replacing your root folder.

Installing post-usrmerge packages in pre-usrmerge image

If you are pinning a package version and attempting to force the installation of a post usrsbin-merge or lib-merge package in an image that’s created before the transition.

Why will this impact me? Because post-usrmerge packages are not compatible with the base image filesystem layout.

How will this impact me? Images may normally fail to build.

How do I mitigate or resolve this? Do not pin dependency versions, specially on packages with a revision of 40 or higher (e.g. apk add package=<version>-r40)

Installing pre-usrmerge packages in post-usrmerge image

If you are pinning a package version and attempting to force the installation of a pre usrsbin-merge or lib-merge package in an image that’s created after the transition.

Why will this impact me? Because pre-usrmerge packages may overwrite symlinks of post-usrmerge images or install binaries in locations that are no longer supported.

How will this impact me? You may experience one of the following two situations:
- The image may be successfully built, but operation may be degraded, as a pre-usrmerge package may install binaries/libraries in unexpected locations and cause applications to fail to execute correctly.

How do I mitigate or resolve this? Do not pin dependency versions with a revision lower than 20 (e.g. apk add package=<version>-r19)

Installing binaries or libraries in /usr/sbin or libraries respectively

If you are installing binaries or libraries directly into /usr/sbin or /lib,

Why will this impact me? Because those locations will be symlinks to /usr/bin
How will this impact me?
- Images may fail to build.
- Images may NOT fail to build but will experience degraded operations (e.g. they may not be able to find binaries/libraries in the expected locations and fail to execute applications).

How do I mitigate or resolve this? Install binaries or libraries in /usr/bin or /usr/lib respectively.

Appendix

Installing packages and replacing your root folder

If your Dockerfile looks like this, you will be impacted by the transition.

FROM cgr.dev/chainguard-private/jdk:openjdk-21.0.6-r1-dev AS builder (0)

[...]
ARG PACKAGES="apk-tools shadow curl busybox" (1)

RUN mkdir -p /newroot/etc && \ 

   cp -a /etc/apk /newroot/etc && \

   apk update && \

   apk add --no-cache --initdb --root /newroot $PACKAGES && \ (2)

   [...]

FROM cgr.dev/chainguard-private/jdk:openjdk-21.0.6-r1 (3)

USER root

COPY --chown=65532:65532 --from=builder /newroot / (4)

[...]

Why may this impact me?

Because the usrmerge transition changes the base filesystem layout, newer usrmerged packages may be incompatible with the old filesystem layout (or vice versa).

Whenever you install packages inside a “root” folder (e.g. /newroot) (2), there’s no assurance that they are compatible with those packages within the image or the “base” layout itself. The problem, however, may not be noticeable until you attempt to copy (or replace) the contents of the root of an image (e.g. / ) (4). In this case, the above config will try to install usrsbin-merged or lib-merged packages, while the base filesystem layout of the image remains the same.

NOTE: It is assumed that (0) and (3) can be any pre-usrmerge (usrsbin-merge, lib-merge) image, regardless of the tag/epoch.

Mitigation Strategies

Mitigation Strategy 1 - Pinning pre-usrmerge (usr/bin, lib) packages

While the transition is in progress, you can prevent apk from installing older, pre-usrmerge (usrsbin-merge, lib-merge) packages while still using pre-usrmerge base images (0) and (3). You can update (1) with any of:

For the transition of usr/sbin: wolfi-baselayout !merged-usrsbin
For the transition of /lib: wolfi-baselayout !merged-lib
Or for both: wolfi-baselayout !merged-usrsbin !merged-lib

For example, take a snippet from the config above, this would mean customers would need to add the following:

[...]
ARG PACKAGES="curl foo bar wolfi-baselayout !merged-usrsbin !merged-lib" (1)

[...]

After the transition is complete

After the transition is complete, it is recommended that you:

Update (0) and (3) to use the post-usrmerge images.
AND update (1) to remove the added dependencies: wolfi-baselayout !merged-usrsbin !merged-lib.

Mitigation Strategy 2 - Installing packages via the chroot method

You can change the way you build your images. Instead of installing packages into a “newroot” folder and copying/replacing them into the runtime image, you can leverage the use of a chroot.

The following is a basic example showing the use of a chroot, which ensures that post-usrmerge packages are not installed in pre-usrmerge images.

FROM cgr.dev/chainguard-private/jdk:openjdk-21.0.6-r1 AS base (1)

FROM cgr.dev/chainguard-private/jdk:openjdk-21.0.6-r1-dev AS build (2)

USER 0

COPY --from=base / /base-chroot (3)

ARG PACKAGES="apk-tools shadow curl busybox"

RUN apk add --root /base-chroot $PACKAGES (4)

FROM scratch AS custom-base

COPY --from=build /base-chroot /

NOTE: This may not work for all use cases, and more details can be provided upon request.

Mitigation Strategy 3 - Custom Assembly

If you use Chainguard Custom Assembly you will not be affected by these changes as it's a supported mechanism for adding packages to images. Custom Assembly allows you to create images that have different packages than those that are available in the image by default. Custom Assembly ensures that image builds are not impacted by package interdependence issues, like those experienced with the usrmerge transition or other issues such as those experienced with curl and libcurl.

How to reach out for support

Please contact your Technical Success Manager should you require more information. If you need immediate help, reach out through our support portal at support.chainguard.dev or via support@chainguardhelp.zendesk.com.

What will happen the day of the transition

How do I know if I may be impacted?

Installing packages and replacing your root folder

Installing post-usrmerge packages in pre-usrmerge image

Installing pre-usrmerge packages in post-usrmerge image

Installing binaries or libraries in /usr/sbin or libraries respectively

Appendix

Installing packages and replacing your root folder

Related articles