erofs

EROFS Snapshotter

The EROFS snapshotter is a native containerd snapshotter to enable the EROFS filesystem, specifically to keep EROFS‑formatted blobs for each committed snapshot and to prepare an OverlayFS mount for each active snapshot.

In order to convert OCI container images directly into EROFS-formatted blobs, the EROFS differ must be specified with the EROFS snapshotter. Otherwise, if the walking differ is used, the EROFS snapshotter will behave much like the existing OverlayFS snapshotter: the applier of the walking differ will unpack the current layer into the active EROFS snapshot of the mounted OverlayFS, and the EROFS snapshotter will commit it into an EROFS-formatted blob, which is slower than using the EROFS differ. Also see the Configuration section.

Although the EROFS snapshotter sounds somewhat similar to an enhanced OverlayFS snapshotter, several kernel features are highly tied to the EROFS internals, so it would be better to leave it as an independent snapshotter. This way, existing OverlayFS users will not be impacted by the new EROFS‑specific behaviors, and interested users will also have a chance to use the EROFS filesystem and even develop the related ecosystems (such as ComposeFS, confidential containers, gVisor, Kata, nerdbox and more) together.

Use Cases

The EROFS snapshotter can benefit to several use cases:

For runC containers, instead of unpacking individual files into a directory on the backing filesystem, it applies OCI layers into EROFS blobs, therefore:

  • Improved image unpacking performance by live-converting tar archives into EROFS-formatted layers during unpacking, compared with directly unpacking to the host filesystem: By converting to EROFS-formatted layers, there is no extra filesystem metadata journal traffic during handling individual files and no need to remove a large number of files when GCing unused snapshots;

    Here are the unpacking benchmark results against the OverlayFS snapshotter using containerd 2.2.1 (pulling from a local registry, without parallel unpacking):

    Top25 container images Large AI container images

  • Parallel unpacking is now supported natively, similar to the OverlayFS snapshotter. This capability is difficult to implement in disk‑snapshot‑style snapshotters such as blockfile, devmapper and ZFS snapshotters. It also uses an efficient method to persist layer data (via fsync) compared to the OverlayFS snapshotter, which can only use syncfs;

  • Better data persistence guarantee: compared to directly unpacking to the host filesystem, it provides better semantics by fsyncing the individual EROFS-formatted layer blobs instead of syncfsing the whole disk each time.

  • Full data protection for each snapshot using the FS_IMMUTABLE_FL file attribute and fsverity. EROFS uses FS_IMMUTABLE_FL and fsverity to protect each EROFS layer blob, ensuring the mounted tree remains immutable. However, since FS_IMMUTABLE_FL and fsverity protect individual files rather than a sub-filesystem tree, other snapshotter implementations like the overlayfs snapshotter are not quite applicable due to less efficiency at least;

  • Support given‑size block devices as the upper layer for OverlayFS to limit the disk quota for writable layers (usually ephemeral storage);

  • A dedicated EROFS default mount handler enables EROFS file‑backed mounts to avoid loop devices on runC. Note that specific runtime shims can handle EROFS mounts without this built‑in handler; for more details, see containerd Mounts and Mount Management ;

  • Native EROFS layers can be pulled from registries without conversion.

For VM containers, the EROFS snapshotter can efficiently pass through and share image layers, offering several advantages (e.g. better performance and smaller memory footprints) over virtiofs or 9p . Besides, the popular application kernel gVisor also supports EROFS for efficient image pass-through.

Why consider EROFS over other kernel filesystems?

EROFS is specifically designed as an immutable filesystem with the following highlights:

  • Lightweight, flexible on-disk format: Designed for archival use to avoid any serious filesystem consistency issues and minimize attack vectors. There is no need to estimate filesystem size or total inode counts in advance, unlike generic filesystems like EXT4;

  • Multi-device support: Enables native layering or content-addressable storage;

  • Bdev- and file-backed mounts: File-backed mounts supported since Linux 6.12, eliminating the need for loopback devices. This covers the latest mainstream distributions such as RHEL 10, Fedora 40, Debian 13, Ubuntu 26.04 LTS (or 24.04 LTS with HWE kernels), and more;

  • Memory sharing: Supports FSDAX using virtio-pmem and per-inode page cache sharing.

Usage

Ensure that EROFS filesystem is available

containerd builds EROFS images itself, so erofs-utils is not required. It is needed only when the differ is configured to use mkfs.erofs (see Using mkfs.erofs ). On newer Ubuntu/Debian systems it can be installed with apt, and on Fedora with dnf:

# Debian/Ubuntu
$ apt install erofs-utils
# Fedora
$ dnf install erofs-utils

Make sure that erofs-utils version is 1.7 or higher.

When using the EROFS snapshotter, before starting containerd, make sure the EROFS kernel module is loaded (Linux 5.4 or later is required): it can be loaded with modprobe erofs.

Checking if the EROFS snapshotter and differ are available

To check if the EROFS snapshotter is available, run the following command:

$ ctr plugins ls | grep erofs

The following message will be shown like below:

io.containerd.snapshotter.v1           erofs                    linux/amd64    ok
io.containerd.differ.v1                erofs                    linux/amd64    ok

Configuration

The following configuration can be used in your containerd config.toml. Don’t forget to restart containerd after changing the configuration.

  [plugins."io.containerd.snapshotter.v1.erofs"]
      # Enable fsverity support for EROFS layers, default is false
      enable_fsverity = true

      # Enable fsyncing a snapshot's directory before the snapshot is committed,
      # default is false
      fsync_on_commit = true

      # Optional: Additional mount options for overlayfs
      ovl_mount_options = []

  [plugins."io.containerd.service.v1.diff-service"]
    default = ["erofs","walking"]

Conversion is reproducible: the same layer always produces the same EROFS image.

Using mkfs.erofs

The differ can build layers with mkfs.erofs from erofs-utils instead of the built-in converter, for example to use mkfs.erofs options such as per-file compression. Set use_mkfs, or set mkfs_options to pass extra options to mkfs.erofs, which also selects it. The differ is disabled when mkfs.erofs is missing or does not support tar mode.

If erofs-utils is version 1.8 or higher, add -T0 --mkfs-time to mkfs_options to enable reproducible builds:

  [plugins."io.containerd.differ.v1.erofs"]
    mkfs_options = ["-T0", "--mkfs-time"]

If erofs-utils is 1.8.2 or higher, it’s preferred to append --sort=none to avoid unnecessary tar data reordering for improved performance:

  [plugins."io.containerd.differ.v1.erofs"]
    mkfs_options = ["-T0", "--mkfs-time", "--sort=none"]

With mkfs.erofs selected and enable_tar_index, mkfs.erofs writes a single layer file with the tar index followed by the original tar, rather than a separate data device.

Running a container

To run a container using the EROFS snapshotter, it needs to be explicitly specified:

$ # ensure that the image we are using exists; it is a regular OCI image
$ ctr image pull docker.io/library/busybox:latest
$ # run the container with the provides snapshotter
$ ctr run -rm -t --snapshotter erofs docker.io/library/busybox:latest hello sh

Quota Support

EROFS supports block mode to generate fixed‑size virtual blocks as the upper layers for overlayfs with a given filesystem formatted in order enable the disk quota. The default_size option can be used in the containerd configuration:

  [plugins."io.containerd.snapshotter.v1.erofs"]
    default_size = "20GiB"

Data Integrity

The EROFS snapshotter provides three methods to consolidate data integrity:

Data Integrity with Immutable File Attribute

By setting set_immutable = true, the EROFS snapshotter marks each layer blob with IMMUTABLE_FL. This ensures that dirty data is flushed immediately and the EROFS layer blob cannot be deleted, renamed, or modified.

The immutable file attribute is mainly used to ensure data persistence and prevent artificial data loss, but it cannot detect data corruption caused by hardware failures. Since it can flush in-memory dirty data, it may significantly increase the unpacking time it takes to launch a container: for example, the unpacking time for tensorflow:2.19.0 increases by 108.86% (from 10.090s to 21.074s) on EXT4. However, it has no impact on runtime performance.

Data Integrity with fs-verity

By setting enable_fsverity = true, the EROFS snapshotter will:

  • Enable fs-verity on EROFS layers during commit;

  • Verify the fs-verity status before mounting layers;

  • Skip fs-verity if the filesystem or kernel does not support it.

The fs-verity method guarantees that EROFS blob layers never change, but it introduces additional runtime overhead since all container image reads from the container will be slower because it needs to verify the Merkle hash tree first.

Data Integrity with dm-verity

The EROFS snapshotter supports device-mapper verity to provide block-level integrity verification for each EROFS layer. This method creates a dm-verity device for each layer and mounts it read-only. The dm-verity implementation uses the go-dmverity Go library, eliminating the need for external veritysetup command-line tools. This requires a Linux kernel with dm-verity support (CONFIG_DM_VERITY) and the device-mapper kernel module loaded.

The differ must be configured to generate dm-verity metadata:

[plugins."io.containerd.differ.v1.erofs"]
  enable_dmverity = true

When dm-verity is enabled, the EROFS differ formats each layer with dm-verity by appending a Merkle hash tree to the EROFS blob and generating a root hash. The hash tree is stored inline within the layer blob itself. The root hash and hash offset are saved in a .dmverity metadata file alongside the layer blob in JSON format. All other dm-verity parameters (block sizes, salt, etc.) are stored in a superblock within the layer blob and are auto-detected when mounting. Regular mode uses 4096-byte blocks (standard page size), while tar-index mode uses 512-byte blocks (dm-verity logical_block_size constraint).

The snapshotter can be configured to control dm-verity behavior using dmverity_mode:

[plugins."io.containerd.snapshotter.v1.erofs"]
  dmverity_mode = "auto"  # Options: "auto" (default), "on", "off"

The available modes are:

  • "auto" (default): Uses dm-verity if .dmverity metadata exists for a layer, otherwise mounts as regular EROFS. This allows mixing dm-verity and non-dm-verity layers in the same system.

  • "on": Requires dm-verity for all layers. If a layer lacks .dmverity metadata, mounting will fail with an error. Use this mode when you want to enforce integrity verification for all layers.

    Important: If you enable dmverity_mode = "on" after layers have already been unpacked without dm-verity enabled in the differ, those existing layers will not have .dmverity metadata files. In this case, you must clean up the existing snapshots and re-pull the images with both enable_dmverity = true in the differ and dmverity_mode = "on" in the snapshotter configured. Alternatively, use dmverity_mode = "auto" to allow mixing dm-verity and non-dm-verity layers.

  • "off": Disables dm-verity completely, even if .dmverity metadata exists. Layers are mounted as regular EROFS without integrity verification. Use this for compatibility or when dm-verity overhead is unacceptable.

When mounting a layer with dm-verity enabled, the snapshotter reads the metadata from the .dmverity file and creates a dm-verity device. The dm-verity library automatically reads all parameters from the superblock, ensuring that any corruption or tampering will be detected at read time. The dm-verity device is then mounted as the backing layer in the OverlayFS stack

How It Works

For each layer, the EROFS snapshotter prepares a directory containing the following items:

  .erofslayer
  fs
  work

.erofslayer file is used to indicate that the layer is prepared by the EROFS snapshotter.

If the EROFS differ is also enabled, the differ will check for the existence of .erofslayer and convert the image content blob (e.g., an OCI layer) into an EROFS layer blob.

In this case, the snapshot layer directory will look like this:

  .erofslayer
  fs
  layer.erofs
  work

If dm-verity is enabled, a .dmverity metadata file will also be present:

  .erofslayer
  fs
  layer.erofs
  layer.erofs.dmverity
  work

In tar index mode, the layer also has a device-0.blob data device (see the Tar Index Mode section).

Then the EROFS snapshotter will check for the existence of layer.erofs: it will mount the EROFS layer blob to fs/ and return a valid overlayfs mount with all parent layers. If dm-verity is enabled and the .dmverity file exists, the snapshotter will create a dm-verity device and mount that instead.

If other differs (not the EROFS differ) are used, the EROFS snapshotter will convert the flat directory into an EROFS layer blob on Commit instead.

In other words, the EROFS differ can only be used with the EROFS snapshotter; otherwise, it will skip to the next differ. The EROFS snapshotter can work with or without the EROFS differ.

Tar Index Mode

The EROFS differ also supports a “tar index” mode. Instead of building one EROFS image that holds both the filesystem metadata and the file contents, the tar index mode writes two files into the snapshot directory:

  layer.erofs      EROFS metadata image: superblock, inodes, and a chunk
                   index for each regular file
  device-0.blob    data device: the file contents, each starting on a
                   512-byte boundary

layer.erofs is a complete EROFS image that uses 512-byte blocks and declares one extra device. The snapshotter mounts it with the data device attached as device=<path>/device-0.blob. A data device is a raw byte source, not an EROFS image. Devices are numbered from 0 in the order they are attached.

When dm-verity is enabled, the differ formats layer.erofs and each data device separately, and each gets its own .dmverity metadata file (device-0.blob.dmverity). The mount handler opens a dm-verity device for each of them. With dmverity_mode = "on", a layer is only mounted if the image and every data device have dm-verity metadata.

A layer with data devices has to be mounted through the mount manager’s EROFS mount handler. It uses a file-backed mount where the kernel supports one (Linux 6.12 and later) and otherwise attaches loop devices for both the image and its data devices. The kernel does not accept a regular file as a data device of a loop-backed mount.

Configuration

For the EROFS differ:

[plugins."io.containerd.differ.v1.erofs"]
  enable_tar_index = true

Layer Content Cache

When pulling a regular OCI image, every layer is converted on the node into either a native EROFS blob or an indexed tar (see the Tar Index Mode section). That is an EROFS image build per layer, on every node that pulls the image.

The conversion can be done upfront and handed to containerd as a pre-warmed cache. layer_content_caches lists directories of pre-built EROFS blobs keyed by layer diffID. When a layer is unpacked, the snapshotter checks the cache first. On a hit it records the path of the cached blob in the snapshot and mounts that blob without converting anything. A miss is unpacked and converted as usual, so a partially populated cache works. A hit also skips the layer download, which means a fully cached image does not fetch layer blobs.

  [plugins."io.containerd.snapshotter.v1.erofs"]
    layer_content_caches = ["/var/lib/erofs-cache/base-images", "/mnt/shared/erofs-cache"]

Directories are searched in order and the first hit wins. A directory that isn’t there counts as a miss, so an empty or not-yet-populated cache won’t break pulls.

Listing more than one is handy when they have different lifetimes or owners, e.g. a small local cache of base images baked into the machine image, plus a larger one on shared storage that’s refreshed on its own schedule.

Populating and maintaining a cache is the operator’s responsibility: containerd never writes to these directories, so they can be mounted read-only.

[!IMPORTANT] A hit mounts the blob from its location in the cache. An entry must stay in place for as long as any snapshot references it. Removing an entry that is still in use breaks those layers.

[!NOTE] For the same reason enable_fsverity and set_immutable can’t be used together with a cache, since both would have to modify a blob that every snapshot using that layer shares. Use dm-verity instead.

To build a new cache, fetch the image content and convert its layers:

$ ctr content fetch $IMG
$ ctr images build-erofs-cache \
    --compressors lz4 \
    --dmverity \
    $IMG ./output

Blobs land at <algo>/<xx>/<digest>.erofs (<xx> shards on the first two digest characters), and --dmverity writes a .dmverity sidecar alongside each one. Since the key is the diffID, one cache serves every image sharing those layers.