Vaultwarden restore drill in 29.1 s, and live SQLite copies

A backup you have never restored is a guess. This drill backs up a real Vaultwarden vault, destroys the machine, and rebuilds it from nothing but a second virtual disk and a compose file, then proves the vault is back by logging in through the API and decrypting every item.

The drill, clocked

What the vault held: one account and 2,000 login items created through Vaultwarden’s own API, encrypted client-side with the account’s master key (the test script implements the Bitwarden key derivation and encryption in Python). The Docker volume was 59,591,243 bytes on disk. The backup target was a second virtual disk mounted at /mnt/backup; no cloud account was involved.

After the restore, vwclient.py verify logged in with the master password, called /api/sync, and for each of the 2,000 items checked the MAC and compared the decrypted username, password, URL and notes against the values the seed produced. The result: 2,000 of 2,000 matched, 0 errors. The raw output is in raw/api-verify-restored.txt.

The recovery time covers a boot, a restic download, an image pull, a restore, the integrity check and the API read-back. It starts with a VM that already has Docker and the compose plugin (the lab base image) and a backup disk attached at boot, so installing Docker and moving a disk are not in it. In this run most of it is fixed overhead rather than data: the restore itself took 1.6 s. This is one run; an earlier run of the same script, whose raw output was replaced by this one, took 36.7 s for the same steps without the integrity check (about 37.1 s with it), so expect a spread of several seconds. A vault that is 60 MB restores quickly everywhere; a volume with large attachments will not.

What the drill does not make faster

Steps 07 and 08 (booting a machine, installing the tool) dominate. Keeping the restic binary on the backup disk, or in your base image, removes the download. That change was not measured here.

The written checklist

This is the checklist behind the drill; steps 2 to 9 are what the script did, in this order. Copy it and write down your own timings the first time you run it.

1. Before anything breaks: confirm the backup disk is mounted, the restic repository password is stored somewhere that is not the machine being backed up, and restic check returns no errors. restic check verifies the repository structure; add --read-data to read and verify every pack, which the drill did not do.
2. Provision a replacement host with Docker and the compose plugin.
3. Attach the backup disk and mount it.
4. Install the same restic version (0.19.1 here).
5. Pull the image and create the empty volume: docker compose -p drill up --no-start.
6. Restore into the volume path: restic restore latest --target /. The snapshot stores absolute paths, so the target is /. As root this writes straight into the live filesystem, so on a host that holds anything else, add --include /var/lib/docker/volumes/drill_vw_data/_data (not used in the drill, where the new VM held nothing else) so that only the volume path is written.
7. Run PRAGMA integrity_check; on db.sqlite3 before starting the container.
8. Start the stack with the same compose file and wait for /alive.
9. Log in as a real user and read the data back. Do not stop at “the container is healthy”.
10. Write down the elapsed time and what slowed you down.

The compose file used (signups were enabled only so the test could create its account through the API; the stack in the Vaultwarden guide has them off):

# Compose used by the restore-drill test. SIGNUPS_ALLOWED is true only so the test can create its account through the API.
services:
  vaultwarden:
    image: vaultwarden/server:1.37.4
    restart: unless-stopped
    environment:
      SIGNUPS_ALLOWED: "true"
    ports:
      - "127.0.0.1:8080:80"
    volumes:
      - vw_data:/data
volumes:
  vw_data:

The backup command in step 05 was, in order: docker compose -p drill stop, restic init, restic backup /var/lib/docker/volumes/drill_vw_data/_data, docker compose -p drill start, restic check. With the container stopped, nothing writes during the copy. The Vaultwarden wiki recommends sqlite3 .backup for a database that may be in use, and says that if you copy db.sqlite3 directly you must restore it together with its -wal file.

What a backup of a live SQLite file did

The drill above used a stopped container. The more common mistake is to point a backup tool at the data directory of a running Vaultwarden. This part of the test did that on purpose, under load.

Setup: the database was filled to 30,000 items by script (32,000 with the drill’s 2,000), the container was stopped cleanly and the data directory copied aside as a pristine copy. Before every trial that pristine copy was restored over the volume and the container started, so each of the 40 trials began with the same 32,000-item database. Then six threads posted new login items for 20 seconds (as fast as the server accepted them), and three seconds in the data was copied four ways, ten trials each. The method order was shuffled within every round of four (seed 7, raw/order.txt). Vaultwarden’s database was in WAL mode (PRAGMA journal_mode returned wal in raw/db-info.txt). The pristine copy holds a 45 MB db.sqlite3 next to a 869 MB db.sqlite3-wal file, because the WAL file kept its size after the bulk load; a real vault that was never bulk-loaded would have a much smaller one. Each copy was opened with the SQLite CLI and checked with PRAGMA integrity_check. The restic copy was a backup into a fresh repository, timed without the restore that followed it.

• Copying the live directory with restic produced a database that passed the check in 10 of 10 trials, in a copy that took 3.4 to 4.8 s while about 4,000 writes were acknowledged during it. The copy contained the -wal file every time.
• A throttled rsync of the live directory (20 MB/s) also passed 10 of 10; it took 6.2 to 9.0 s.
• Copying db.sqlite3 alone, without its -wal file, failed in all 10 trials, with errors such as “database disk image is malformed” and b-tree page errors. The copy took about 40 ms; the likely reason it fails is that recent changes still live in the WAL file, which this copy leaves behind (not examined further).
• sqlite3 .backup passed all 10 and took 17.6 s every time, against about 4 s for the restic copy. The SQLite backup docs say the backup usually restarts when another process writes to the source; the 17.6 s is consistent with it waiting out the rest of the write burst, but restarts were not counted.

Per-trial detail is in sqlite-integrity-detail.json and raw/. The row count in each copy never fell below the rows present at trial start plus writes acknowledged when the copy began, except that one db-file-only trial has no acknowledged-write count (the counter file was read while being rewritten) and is left out of that comparison. Whether the table data itself was damaged in the failing copies was not examined beyond integrity_check.

The earlier run said otherwise

The first version of this test did not reset the database between trials, so it grew from 32,000 items to about 710,000 over the 40 trials, and the four methods ran one after another, each on a larger database than the last. That run reported restic failing the check in 7 of 10 trials and rsync in 2 of 10, with “row N missing from index” errors. Every one of those failures came after the database passed about 80,000 items; the three clean restic trials were the first three, at 32,000 to 65,000 items. When size and order were taken out, the restic and rsync failures did not reproduce in 10 trials each. The raw output of that run was replaced and is not kept, and the test did not repeat these trials at 80,000 items or more with a reset, so it cannot say whether size, the larger WAL or something else caused those failures. Do not read either result as “a live directory copy is safe”.

Treat this honestly in both directions. Ten clean restic and rsync trials at one database size, on a small VM, do not mean a live copy is usually fine: the earlier run found failures on a larger database, a real vault takes far fewer writes than this load, and a clean run proves nothing about the next one. The configurations that never failed in either run were the stopped container (one drill run) and .backup (the method the Vaultwarden wiki recommends). The only method that failed every time in both runs was copying db.sqlite3 without its -wal file.

What went wrong along the way

• POST /identity/accounts/register returned 404 on 1.37.4. Registration goes through /identity/accounts/register/send-verification-email and then /identity/accounts/register/finish; creating items also required encryptedFor set to the user id. Both were found in the Vaultwarden source.
• Earlier attempts of the load trials had harness bugs (a permission error made some copies look empty). Those runs were discarded and the test was rerun from the start. After the run described above showed a confound (a database growing between trials, see “The earlier run said otherwise”), the trials were rerun a second time with a reset before every trial; the numbers above come from that final run only.
• In the final run the harness read the acknowledged-writes counter while it was being rewritten once and got an empty value, so one trial has no acknowledged-write count at its start.

What this test did not cover

• A single drill run. The drill destroys its own VM, so the 29.1 s total is one measurement, not a median; the earlier run’s 36.7 s (see above) shows the spread.
• A hard failure. The source VM was powered off cleanly and its disk deleted; the backup disk was unmounted before that. A backup disk that was never unmounted, or corrupted media, was not tested.
• Attachments, Sends, organizations, 2FA, and the admin page’s config.json. Only the database and rsa_key.pem were in the volume.
• Remote or cloud repositories. Their restore speed will depend on bandwidth.
• The effect of a live copy on other tools (borg, kopia, plain cp -a of the whole directory) beyond the four methods listed, the effect at lower write rates, and at database sizes above 32,000 items with a reset between trials.
• Whether Vaultwarden clients behave correctly after restoring a backup older than their last sync; only the API read-back was checked.