ZFS vs Btrfs snapshots: cost, rollback, send/receive

This test compares what snapshots cost and how rollback and replication behave, using ZFS from Ubuntu’s own packages and Btrfs, on the same two virtual disks. It does not say which filesystem is faster. Every disk here is a file on a shared host behind a virtual controller, so the throughput numbers cannot be transferred to real drives; the relative costs and the behaviors can be.

Setup

• Disk A (12 GiB, virtual) holds the tree under test; disk B (12 GiB, virtual) is the replication target, a second pool for ZFS and a second Btrfs filesystem for Btrfs. The VM had 4 vCPU and 3 GB of RAM.
• The tree: 2,048 files of 2 MiB of seeded random bytes (4 GiB), plus a 1 GiB file used for the write load. No compression option was set on either.
• The change set: 205 files (10%) overwritten in place, 50 files deleted, 50 new files of the same size.
• Each filesystem was rebuilt from scratch for each of 3 repetitions.

ZFS used one dataset in a one-disk pool (zpool create, then zfs create zA/data). Btrfs used a subvolume (btrfs subvolume create /mnt/a/data) and read-only snapshots under /mnt/a/snaps.

Time per operation

Snapshot creation under load ran while fio did 4k random writes (2 jobs, queue depth 16) into the 1 GiB file; three snapshots were taken 5 s apart, starting 8 s in. Both were sub-second at the median, and the load made each slower than idle (44 ms to 325 ms for ZFS, 18 ms to 173 ms for Btrfs). The size of that gap is a property of this virtual disk and the load. Page caches were dropped (echo 3 > /proc/sys/vm/drop_caches) once before the idle snapshots and once before each rollback; this was not checked to empty the ZFS ARC (ZFS keeps its own cache), so the two filesystems may not have started equally cold. Reading the table as “Btrfs snapshots faster” would be a claim this test cannot support, because the setup does not isolate the filesystem from the virtual disk and caching.

The Btrfs docs say “Subvolume creation needs to flush dirty data that belong to the subvolume and this step may take some time” and that, once that is done, “the snapshot is instantaneous”. That is consistent with the load rows being slower, but the page speaks of subvolume creation, not snapshot creation, and the test did not measure a flush separately.

Space held by a snapshot

The row to read is the pair of “after the change set” lines. Without a snapshot, applying the change set changed allocated space by under 2 MB on either filesystem, because the old blocks were freed. With the base snapshot held, the same change set left 537.14 MB allocated on ZFS and 536.18 MB on Btrfs. That is about the old blocks of the 255 files (205 overwritten plus 50 deleted) that the snapshot still points at. ZFS reported the same figure directly as usedbysnapshots (537.75 MB).

The incremental streams were 538.01 MB (ZFS) and 535.36 MB (Btrfs), close to the 255 files that were overwritten or created.

Rollback and cleanup behave differently

• ZFS has a command: zfs rollback -r zA/data@S1. Per the man page, by default it refuses to roll back to any snapshot but the most recent one; -r destroys the snapshots and bookmarks more recent than the target. In this test -r removed the intermediate snapshot (S2) that had just been replicated. It took a median of 194 ms.
• Btrfs has no rollback command in its subvolume documentation. The step timed here was btrfs subvolume delete /mnt/a/data followed by btrfs subvolume snapshot /mnt/a/snaps/S1 /mnt/a/data, a median of 89 ms. The deleted subvolume is cleaned up in the background; waiting for that with btrfs subvolume sync took a median of 11,011 ms (range 5,011 to 12,014) before the space was back.
• After each rollback, a SHA-256 of every file matched the base snapshot in all 3 runs on both filesystems.

Replication to a second pool

The full send moved 5 GiB of snapshot; the incremental after the change set moved only the changed files. ZFS used zfs send into zfs receive -F (-F was added after the first attempt; see below), Btrfs used btrfs send and, for the incremental, btrfs send -p S1 S2 into btrfs receive. In all 3 runs the received trees matched the sources file by file.

What went wrong

• First attempt: the ZFS incremental receive exited with an error (the error text was not captured). Adding -F to zfs receive made it succeed. The likely cause is that the target dataset had been read and checksummed after the base receive, but that is an inference.
• First attempt: the Btrfs half did not run because the ZFS pools still held the disks. The script now destroys them first, and the whole test was rerun, so every number above is from one run.

ZFS memory use

ZFS’s ARC grew to 1,567,346,728 bytes against a cap (c_max) of 1,558,142,976 bytes on the 3 GB VM (raw/zfs-1-final.txt). On a 3 GB VM that is a large share of RAM. This test did not isolate the effect of ARC pressure on any timing.

Commands used

#!/usr/bin/env bash
# Runs in the VM. Two raw virtual disks: /dev/disk/by-id/virtio-snapA (source) and virtio-snapB (replication target).
set -u
sudo mkdir -p /srv/snapraw; sudo chown lab:lab /srv/snapraw; RAW=/srv/snapraw; cd /srv/work
A=/dev/vdb; B=/dev/vdc
now() { date +%s%3N; }
tm() { # tm <label> <cmd...>: wall ms of the command, rc recorded
  local l=$1; shift; local s=$(now); "$@" >> $RAW/$FS.cmd.log 2>&1; local rc=$?; local e=$(now)
  echo "$FS $REP $l $((e-s)) $rc" >> $RAW/timings.txt; return $rc; }
man() { (cd "$1" && find . -type f -print0 | LC_ALL=C sort -z | xargs -0 sha256sum | sha256sum | cut -d' ' -f1); }
dropc() { sync; echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null; }
rec() { echo "$FS $REP $1 $2" >> $RAW/values.txt; }   # rec <key> <value>
FIO='fio --name=w --filename=DATA/fio.dat --size=1G --rw=randwrite --bs=4k --ioengine=libaio --iodepth=16 --numjobs=2 --time_based --runtime=45 --group_reporting --output-format=json'

# ---------- per-filesystem primitives
zfs_reset() { sudo zpool destroy -f zA 2>/dev/null; sudo zpool destroy -f zB 2>/dev/null; sudo wipefs -aq $A $B
  sudo zpool create -f -o ashift=12 -m /zA zA $A && sudo zpool create -f -o ashift=12 -m /zB zB $B && sudo zfs create zA/data && sudo chown lab:lab /zA/data
  DATA=/zA/data; BDATA=/zB/data; }
btrfs_reset() { sudo zpool destroy -f zA 2>/dev/null; sudo zpool destroy -f zB 2>/dev/null; sudo umount /mnt/a /mnt/b 2>/dev/null; sudo wipefs -aq $A $B; sudo mkdir -p /mnt/a /mnt/b
  sudo mkfs.btrfs -q -f $A && sudo mkfs.btrfs -q -f $B && sudo mount $A /mnt/a && sudo mount $B /mnt/b
  sudo btrfs subvolume create /mnt/a/data >/dev/null && sudo mkdir /mnt/a/snaps /mnt/b/snaps && sudo chown lab:lab /mnt/a/data
  DATA=/mnt/a/data; BDATA=/mnt/b/snaps; }
used() { sync; if [ $FS = zfs ]; then sudo zpool sync zA; sleep 5; zpool list -Hp -o allocated zA; else sudo btrfs filesystem sync /mnt/a >/dev/null; sleep 5; sudo btrfs filesystem usage -b /mnt/a | awk '/^ *Used:/{print $2; exit}'; fi; }
snap() { if [ $FS = zfs ]; then sudo zfs snapshot zA/data@$1; else sudo btrfs subvolume snapshot -r /mnt/a/data /mnt/a/snaps/$1; fi; }
snapdel() { if [ $FS = zfs ]; then sudo zfs destroy zA/data@$1; else sudo btrfs subvolume delete /mnt/a/snaps/$1; fi; }

for FS in zfs btrfs; do
 for REP in 1 2 3; do
  echo "=== $FS rep $REP $(date +%T)" >> $RAW/progress.txt
  ${FS}_reset >> $RAW/$FS.cmd.log 2>&1 || { echo "reset failed $FS $REP" >> $RAW/progress.txt; continue; }
  python3 fsgen.py gen $DATA 100 2048; dd if=/dev/urandom of=$DATA/fio.dat bs=1M count=1024 status=none
  sync; dropc
  # 1. snapshot creation, idle
  for i in 1 2 3; do tm snap-idle-$i snap i$i; sleep 2; done
  # 2. snapshot creation under 4k random-write load (fio, 2 jobs, qd16, 45 s)
  ${FIO//DATA/$DATA} > $RAW/$FS-$REP-fio.json 2>&1 &
  fp=$!; sleep 8
  for i in 1 2 3; do tm snap-load-$i snap u$i; sleep 5; done
  wait $fp
  for s in i1 i2 i3 u1 u2 u3; do tm snapdel-$s snapdel $s; done
  [ $FS = btrfs ] && sudo btrfs subvolume sync /mnt/a >/dev/null 2>&1
  # 3. control: change set with no snapshot held
  sync; U0=$(used); python3 fsgen.py change $DATA 11; U1=$(used); rec space_control_delta $((U1-U0))
  # 4. base snapshot, full replication to the second pool
  tm snap-S1 snap S1; M1=$(man $DATA); rec manifest_S1 $M1
  if [ $FS = zfs ]; then
    s=$(now); sudo zfs send zA/data@S1 | tee >(wc -c > $RAW/$FS-$REP-full.bytes) | sudo zfs receive -F zB/data 2>>$RAW/$FS.cmd.log; rc=$?; e=$(now)
  else
    s=$(now); sudo btrfs send /mnt/a/snaps/S1 2>>$RAW/$FS.cmd.log | tee >(wc -c > $RAW/$FS-$REP-full.bytes) | sudo btrfs receive /mnt/b/snaps >>$RAW/$FS.cmd.log 2>&1; rc=$?; e=$(now)
  fi
  sleep 1; echo "$FS $REP send-full $((e-s)) $rc" >> $RAW/timings.txt; rec full_bytes $(cat $RAW/$FS-$REP-full.bytes)
  if [ $FS = zfs ]; then rec manifest_B_S1 $(man /zB/data); else rec manifest_B_S1 $(man /mnt/b/snaps/S1); fi
  # 5. change set (10% of files overwritten, 50 deleted, 50 created) while S1 is held: space
  U0=$(used); python3 fsgen.py change $DATA 12; U1=$(used); rec space_snapshot_held_delta $((U1-U0))
  [ $FS = zfs ] && { rec zfs_usedbysnapshots $(zfs list -Hp -o usedbysnapshots zA/data); rec zfs_written_S1 $(zfs get -Hp -o value written@S1 zA/data); }
  [ $FS = btrfs ] && sudo btrfs filesystem usage -b /mnt/a > $RAW/$FS-$REP-usage.txt 2>&1
  tm snap-S2 snap S2; M2=$(man $DATA); rec manifest_S2 $M2
  # 6. incremental replication S1 -> S2
  if [ $FS = zfs ]; then
    s=$(now); sudo zfs send -i zA/data@S1 zA/data@S2 | tee >(wc -c > $RAW/$FS-$REP-inc.bytes) | sudo zfs receive -F zB/data 2>>$RAW/$FS.cmd.log; rc=$?; e=$(now)
  else
    s=$(now); sudo btrfs send -p /mnt/a/snaps/S1 /mnt/a/snaps/S2 2>>$RAW/$FS.cmd.log | tee >(wc -c > $RAW/$FS-$REP-inc.bytes) | sudo btrfs receive /mnt/b/snaps >>$RAW/$FS.cmd.log 2>&1; rc=$?; e=$(now)
  fi
  sleep 1; echo "$FS $REP send-incremental $((e-s)) $rc" >> $RAW/timings.txt; rec inc_bytes $(cat $RAW/$FS-$REP-inc.bytes)
  if [ $FS = zfs ]; then rec manifest_B_S2 $(man /zB/data); else rec manifest_B_S2 $(man /mnt/b/snaps/S2); fi
  # 7. rollback to S1 (live tree currently equals S2 plus nothing)
  dropc
  if [ $FS = zfs ]; then tm rollback sudo zfs rollback -r zA/data@S1
  else tm rollback bash -c 'sudo btrfs subvolume delete /mnt/a/data && sudo btrfs subvolume snapshot /mnt/a/snaps/S1 /mnt/a/data'; tm rollback-cleanup sudo btrfs subvolume sync /mnt/a; fi
  rec manifest_after_rollback $(man $DATA)
  if [ $FS = zfs ]; then { echo "--- arcstats"; grep -E '^(size|c_max|c_min) ' /proc/spl/kstat/zfs/arcstats; zpool status zA zB; zfs list -t all; } > $RAW/zfs-$REP-final.txt 2>&1
  else { btrfs filesystem usage /mnt/a; btrfs subvolume list /mnt/a; } > $RAW/btrfs-$REP-final.txt 2>&1; fi
 done
done
echo finished >> $RAW/progress.txt

The harness is run.sh next to it. In short:

# ZFS
zpool create -f -o ashift=12 -m /zA zA /dev/vdb && zfs create zA/data
zfs snapshot zA/data@S1
zfs send zA/data@S1 | zfs receive -F zB/data
zfs send -i zA/data@S1 zA/data@S2 | zfs receive -F zB/data
zfs rollback -r zA/data@S1

# Btrfs
mkfs.btrfs -f /dev/vdb && mount /dev/vdb /mnt/a && btrfs subvolume create /mnt/a/data
btrfs subvolume snapshot -r /mnt/a/data /mnt/a/snaps/S1
btrfs send /mnt/a/snaps/S1 | btrfs receive /mnt/b/snaps
btrfs send -p /mnt/a/snaps/S1 /mnt/a/snaps/S2 | btrfs receive /mnt/b/snaps
btrfs subvolume delete /mnt/a/data && btrfs subvolume snapshot /mnt/a/snaps/S1 /mnt/a/data

What this test did not cover

• Real drives. All disks were files behind a virtual controller; throughput, latency under load and write amplification do not carry over.
• Which filesystem is faster. The rows are one VM, one load, one disk type, 3 repetitions.
• Multi-disk layouts (RAID levels, mirrors), compression, deduplication, encryption, quotas, and ZFS properties other than the defaults. The mirror behavior has its own test in the ZFS mirror guide.
• Many snapshots. Only two or three existed at a time; how listing, deletion and space accounting behave with hundreds was not tested.
• Long-term behavior: fragmentation after many snapshot cycles, scrub, balance, and performance after the pool fills.
• Btrfs send/receive over a network, and tools that automate either filesystem’s snapshots (sanoid, snapper, btrbk).