Updating Docker containers safely without Watchtower

Automatic updaters answer “when” but not “what if it breaks”. This guide tests the parts that decide whether an update can be undone: a status check on Watchtower, a real app upgraded and rolled back with data, a database major-version bump, and a notifier that tells you without acting.

Watchtower is archived

The GitHub API for containrrr/watchtower, fetched at 19:13 UTC on the test day:

The repository is read-only, and its last commit is “Add maintenance notice to README”. Its README says “This project is no longer maintained” and “We do not recommend using Watchtower in a commercial or production environment” (repository). The test did not run Watchtower, so nothing here says whether it still works; only that upstream no longer accepts changes.

Upgrade and rollback of a real app

Uptime Kuma was started on a named volume with three HTTP monitors, left for about a minute, stopped, and its volume archived with tar. Then only the image tag in the compose file was changed and docker compose pull && docker compose up -d was run. The data was read back through Kuma’s Socket.IO API each time. Two scenarios, one run each:

Both upgrades kept all three monitors and their heartbeats. Heartbeat counts grow between reads because the monitors keep running, so a higher count after the upgrade is expected.

Two results need a caveat. First, changing only the tag back (rollback without restoring) worked in both scenarios: 2.5.8 and even 1.23.17 started on a data directory that 2.5.9 had already migrated, and showed 3 monitors. That held for a few minutes of heartbeats on a throwaway instance; the Kuma wiki says backing up the data directory is the only supported backup method and that an interrupted migration must be restored from backup. We therefore treat the working tag-only rollback as luck on a small dataset, not as a procedure. Second, in the patch scenario the first Socket.IO read after the upgrade, after the tag-only rollback and after the restore timed out after 60 s (raw/*.attempt1; the read before the upgrade worked first time), and a retry 10 s later worked every time. The cause was not investigated; it did not occur in the major scenario.

The rollback that is known to be right is the restore row: stop the stack, recreate the volume, extract the pre-upgrade archive, start the old tag. It took 3.5 s (patch scenario) and 0.5 s (major scenario) on these small volumes.

The compose file for the patch test (raw/kuma-patch-compose-old.yml); the test set UPTIME_KUMA_DB_TYPE for the 2.x line, a variable listed in the Kuma wiki:

services:
  kuma:
    image: louislam/uptime-kuma:2.5.8
    restart: unless-stopped
    environment:
      UPTIME_KUMA_DB_TYPE: sqlite
    ports:
      - "127.0.0.1:3001:3001"
    volumes:
      - kuma_data:/app/data
volumes:
  kuma_data:

A Postgres major bump on an old data directory

A PostgreSQL 16.15 container was seeded with a 500,000-row table (database app, 132,842,850 bytes of data directory). Then the same volume was mounted into postgres:17 and postgres:18:

postgres:17 stopped with FATAL: database files are incompatible with server, naming version 16 and 17.11, exit code 1. postgres:18 refused for a different reason: its image expects data under a versioned subdirectory, so a mount at /var/lib/postgresql/data with old data in it is rejected with an Error: message that recommends mounting /var/lib/postgresql and using pg_upgrade (full text in raw/pg-bump-18.log). The PostgreSQL docs say the same about majors: the storage format can change and the traditional method is dump and restore, while pg_upgrade is the faster one (docs).

The failed attempts did not harm the old volume: the checksum query returned the same row count, sum and md5 afterwards. The working path was pg_dump -Fc from the old container (median 1.0 s, 11,028,457 bytes), then pg_restore into a fresh 17 volume (median 0.4 s, exit 0 in 3 of 3 runs), with identical checksums in all three copies. pg_upgrade itself was not tested.

The test waited until the new container accepted connections (psql -c 'select 1' succeeded) before the restore; pg_restore run immediately after docker run will fail while the database is still initializing. The tag postgres:17 was a moving tag in the test (it resolved to 17.11); pin the exact version you use. The password is the throwaway value from the lab, so replace it.

docker exec pg16 pg_dump -U postgres -Fc app > app.dump
docker run -d --name pg17 -e POSTGRES_PASSWORD=lab-only-not-a-secret -e POSTGRES_DB=app -v pgdata17:/var/lib/postgresql/data postgres:17
until docker exec pg17 psql -U postgres -tAc 'select 1' >/dev/null 2>&1; do sleep 1; done
docker exec -i pg17 pg_restore -U postgres -d app --no-owner --exit-on-error < app.dump

The wait loop is a small addition to the tested commands (the script used its own pgwait helper). postgres:18 was also a moving tag and its exact version was not recorded.

Diun: tell me, don’t touch it

Diun v4.33.1 ran against a local registry:2 so the test needed no registry account. One container (app) used the digest check, another (repoapp) used diun.watch_repo=true with tags matching ^[0-9]+$. After the first notifications, app:1 was pushed again with different content and repoapp:2 was pushed.

The schedule was one check per minute (*/1), so the 50 s is the wait for the next scheduled check, not a detection latency; it could be anywhere from 0 to 60 s. At the first check, both images were reported as new. About 50 s after the push, the next check reported update for the changed app:1 digest and new for repoapp:2. The running container was not changed; docker inspect still showed localhost:5000/app:1. The configuration, using the script notifier from the Diun docs and insecureTLS from the registry options for a plain-HTTP local registry:

db:
  path: /data/diun.db
watch:
  workers: 2
  schedule: "*/1 * * * *"
  runOnStartup: true
  firstCheckNotif: true
regopts:
  - name: local
    insecureTLS: true
providers:
  docker:
    watchByDefault: false
notif:
  script:
    cmd: /bin/sh
    args:
      - -c
      - 'echo "$(date +%s) status=$DIUN_ENTRY_STATUS image=$DIUN_ENTRY_IMAGE digest=$DIUN_ENTRY_DIGEST" >> /data/notify.log'

services:
  diun:
    image: crazymax/diun:4.33.1
    command: serve --config /diun.yml
    network_mode: host
    volumes:
      - ./diun.yml:/diun.yml:ro
      - ./data:/data
      - /var/run/docker.sock:/var/run/docker.sock:ro

On a real host you would drop the regopts block and network_mode: host, and replace the script with a notifier you use.

What went wrong

• The first Kuma run used a monitor payload with a field that 1.23 does not have (conditions), and the Socket.IO setup call failed with a websocket error on a fresh 2.5.8 start until UPTIME_KUMA_DB_TYPE was set, so both Kuma scenarios were rerun with those fixed (the failed run’s output was overwritten).
• The first Postgres run restored a pg_dumpall file into a fresh container, which fails on role "postgres" already exists; the script now dumps one database with pg_dump -Fc. A shell loop variable clash also skipped runs 2 and 3 of that attempt.
• In the first Diun run the script retagged the local app:1 to the new image after the push, which made Docker report the running container by image ID, and Diun listed it as docker.io/library/alpine@sha256:... as new instead of reporting update for app:1 (raw/attempt1-diun-local-retag/). The final script puts the original tag back on the running container’s image after the push. A tag moved locally under a running container confuses this report; whether a real registry update behaves differently was not tested.

Reproduce

#!/usr/bin/env bash
# update-rollback: Watchtower upstream status (GitHub API), Uptime Kuma upgrade/rollback, Postgres major bump, Diun v4.33.1.
set -uo pipefail
source "$(dirname "$0")/../_lib.sh"
T=update-rollback; OUT="$ROOT/evidence/tests/$T"; RAW="$OUT/raw"; rm -rf "$OUT"; mkdir -p "$RAW"
# Watchtower upstream status, unauthenticated GitHub API, from the host
curl -s https://api.github.com/repos/containrrr/watchtower > "$RAW/watchtower-repo.json"
curl -s https://api.github.com/repos/containrrr/watchtower/releases/latest > "$RAW/watchtower-latest-release.json"
curl -s "https://api.github.com/repos/containrrr/watchtower/commits?per_page=1" > "$RAW/watchtower-last-commit.json"
date -u +%FT%TZ > "$RAW/watchtower-fetched-at.txt"
start_vm a-$T
vm scp-to "$ROOT/lab/tests/$T/guest.sh" guest.sh
vm scp-to "$ROOT/lab/tests/$T/kc.js" kc.js
gx 'bash guest.sh' > "$RAW/guest-stdout.txt" 2>&1
vm scp-from out "$RAW/"; mv "$RAW/out"/* "$RAW/"; rmdir "$RAW/out"
collect_env "$OUT"
python3 "$ROOT/lab/tests/$T/summarize.py" "$RAW" "$OUT"

What this test did not cover