systemd system extensions for Fedora image based systems

NOTE: This is currently an experimental project. Make sure to read the known limitations section. Use at your own risk.

This project makes sysexts available for Fedora CoreOS, Fedora Atomic Desktops, Fedora IoT, and other Fedora Bootable Container (bootc) systems (and classic ostree/rpm-ostree systems).

What are sysexts?

systemd system extensions (sysexts) are filesystem images that bundle content (binaries, libraries and configuration files) that can be overlayed on demand on image based systems. The content of the sysexts are “merged” with the base content of the OS using an overlayfs mount.

In the context of this project, the sysexts are EROFS disk images that include content to overlay on top of /usr.

If you want to know more about sysexts, you can watch the Waiter, an OS please, with some sysext sprinkled on top (video) talk at All systems go! Berlin 2024.

For a more general description of sysexts, see the Extension Images specification from the UAPI group.

Can sysexts replace package layering or container builds?

Sysexts can only modify the content in /usr and they are enabled relatively “late” during the boot process, thus they come with some limitations. They can not be used to:

  • install another kernel
  • install kernel modules
  • make changes to the initrd
  • make changes to /etc
  • add udev rules

If you need to do one of those, you should use package layering or do a container build. Otherwise, sysexts might be a good fit.

Available sysexts

You can find all the available sysexts in the list on the side of this page. The sysexts are built for the current stable releases of Fedora, for x86_64 and aarch64, if the software is available for it. Some images only target a specific variant (CoreOS, Silverblue, Kinoite, etc.). When this is the case, it is generally explicit from the name (wireshark-silverblue) or noted in the help text on the sysext page. The sysexts built only for Fedora CoreOS are also only built for the current Fedora releases used by each Fedora CoreOS streams (stable, testing and next).

Integration with bootc / Bootable Containers

The planned user experience for using those sysexts is that they are built as container layers, pushed to a registry as distinct tags, downloaded, managed and updated in sync with the OS by bootc. See: bootc#7 and README.containers.md.

This integration is currently still a work in progress. The first step will likely be a standalone sysext manager / updater application, which will then be integrated into bootc.

Installing and updating using systemd-sysupdate

You can currently install and update those sysexts using systemd-sysupdate. See the individual pages for instructions.

To update all sysexts on a system:

for c in $(/usr/lib/systemd/systemd-sysupdate components --json=short | jq --raw-output '.components[]'); do
    sudo /usr/lib/systemd/systemd-sysupdate update --component "${c}"
done

Installing directly via Ignition

On Fedora CoreOS, you can also directly download the sysexts images as files via Ignition. Here is an example Butane config:

variant: fcos
version: 1.5.0
storage:
  files:
      # Make sure to name your sysext as <sysext-name>.raw, without the version here
    - path: "/var/lib/extensions/kubernetes-cri-o-1.32.raw"
      contents:
        # Use the full URL with the version to download the sysext
        source: "https://extensions.fcos.fr/extensions/kubernetes-cri-o-1.32/kubernetes-cri-o-1.32-1.32.3-1.fc42-42-x86-64.raw"
systemd:
  units:
    # Enable systemd-sysext.service to merge the sysexts on boot
    - name: systemd-sysext.service
      enabled: true
    # We will use CRI-O
    - name: docker.socket
      enabled: false
      mask: true

See travier/fedora-coreos-kubernetes for a more complete example deploying Kubernetes on Fedora CoreOS using sysexts.

Know issues

Use systemctl restart systemd-sysext.service to refresh sysexts on Fedora 41

On Fedora 41 based images, make sure to use systemctl restart systemd-sysext.service instead of systemd-sysext merge. systemd-sysext unmerge is safe to use.

This issue is fixed in systemd v257 which landed in Fedora 42. See:

Current limitation of systemd-sysupdate

While installing and updating via systemd-sysupdate works, this also has a few limitations:

Common errors

Failed to read metadata for image …: No medium found

If you encounter this error, it likely means that you have set your sysext image with the full name in the /var/lib/extensions directory. systemd-sysext expects the sysexts images to be named only with their own name, without any version (thus why we use the /var/lib/extensions.d directory in the general case).

Building, contributing and license

See the project README.