Docker restart policies after a reboot, tested

Pick the policy by what should happen to a container you stopped on purpose. That is the only case where the policies differ for a service that exits cleanly, and this test checked it with real reboots rather than by reading the table in the docs.

Matrix

Each policy was started twice: once left running, once stopped with docker stop. Two kinds of process were used, because the exit code decides what on-failure does: a script that exits 0 on SIGTERM, and sleep as PID 1, which ignores SIGTERM and is killed after the stop timeout (exit 137). The state was recorded after systemctl restart docker and after each of three VM reboots. Before every reboot the script started the “running” containers and stopped the “stopped by hand” ones again, so each reboot is an independent test from the same starting layout (raw/2-before-reboot-N.tsv). An earlier version of the script reset the layout only once, which made reboots 2 and 3 repeat the state left by reboot 1 for some cells; it was rerun.

What the table says:

• no: never returned, in any case.
• always: returned every time, including the container that was stopped by hand. This matches the docs: a manually stopped always container restarts when the daemon restarts.
• unless-stopped: returned when it was running, stayed stopped when it was stopped by hand, in both process types.
• on-failure: the surprise. A process that exited 0 on SIGTERM stayed down after the daemon restart and all three reboots, running or stopped. The sleep container, which Docker had to kill, came back every time, even the one that had been stopped by hand. The docs say on-failure “doesn’t restart the container if the daemon restarts”; in this test a container that had exited with a non-zero code did come back. The likely reason is the exit code recorded when the daemon stopped it, but the test did not isolate that.

A one-shot container that exits 0 on its own under on-failure stayed exited (oneshot-ok-on-failure, in raw/).

How soon after boot

The test sampled every 3 s once SSH answered, and in every reboot the containers were already running at the first sample, so the table gives an upper bound on boot-to-running time of about 10 s of VM uptime, not a precise figure. It also shows on-failure and no never reaching running with the graceful container.

Crash loops and back-off

Three identical containers per scenario ran for 270 s. Each start printed the VM uptime, so the gaps below come from the containers’ own timestamps.

• A container that exits 1 immediately under always was restarted 13 times in 270 s. The gaps grow from under a second to roughly doubling (0.48, 0.58, 0.75, 1.31, 2.13, 3.46, 6.66, 13.07, 25.81, 51.44 s; the first few do not double cleanly) and settle at about 60 s (60.24, 60.27 s in replica 1). The Docker page cited here does not document the back-off, so the 60 s cap is only what this run measured. A few tenths of a second of start-up overhead in each gap is our guess; it was not isolated.
• on-failure:3 made 4 starts (the original plus 3 retries), then stayed exited with exit code 1.
• A container that ran for 12 s before failing was restarted about every 12.3 s, with no growth over 22 restarts. The docs say a policy takes effect only after a container has run successfully for at least 10 s, and the 12 s runs here did not accumulate delay.

Steps

The test script creates the containers like this (policy substituted for each of the four):

docker run -d --name g-run-unless-stopped --restart unless-stopped -v $HOME/s:/s:ro alpine:3.22 sh /s/graceful.sh
docker run -d --name k-run-unless-stopped --restart unless-stopped alpine:3.22 sleep 100000
docker stop k-stopped-unless-stopped   # for the "stopped by hand" case
sudo systemctl reboot

graceful.sh is three lines: a trap 'exit 0' TERM INT, then sleep 100000 & and wait. In Compose, the same setting is restart: unless-stopped on the service.

#!/usr/bin/env bash
# restart-policies: which containers return after a daemon restart and after 3 VM reboots, and crash-loop back-off timing.
set -uo pipefail
source "$(dirname "$0")/../_lib.sh"
T=restart-policies; OUT="$ROOT/evidence/tests/$T"; RAW="$OUT/raw"; rm -rf "$OUT"; mkdir -p "$RAW"
start_vm a-$T
POLICIES=(no always on-failure unless-stopped)
state() { gx "docker ps -a --format '{{.Names}}\t{{.State}}\t{{.Status}}' | sort"; }
gx 'docker pull -q alpine:3.22 >/dev/null; mkdir -p ~/s'
gx 'cat > ~/s/graceful.sh; chmod +x ~/s/graceful.sh' <<'EOS'
#!/bin/sh
# exits 0 on SIGTERM (a well-behaved service)
trap 'exit 0' TERM INT
sleep 100000 &
wait
EOS
gx 'cat > ~/s/crash.sh' <<'EOS'
#!/bin/sh
cut -d' ' -f1 /proc/uptime
exit 1
EOS
gx 'cat > ~/s/run12.sh' <<'EOS'
#!/bin/sh
cut -d' ' -f1 /proc/uptime
sleep 12
exit 1
EOS
# g-* = process that exits 0 on SIGTERM; k-* = `sleep` as PID 1, which ignores SIGTERM and is killed (exit 137) after the stop timeout
for p in "${POLICIES[@]}"; do
  gx "docker run -d --name g-run-$p --restart $p -v \$HOME/s:/s:ro alpine:3.22 sh /s/graceful.sh >/dev/null; docker run -d --name g-stopped-$p --restart $p -v \$HOME/s:/s:ro alpine:3.22 sh /s/graceful.sh >/dev/null; docker stop g-stopped-$p >/dev/null
docker run -d --name k-run-$p --restart $p alpine:3.22 sleep 100000 >/dev/null; docker run -d --name k-stopped-$p --restart $p alpine:3.22 sleep 100000 >/dev/null; docker stop k-stopped-$p >/dev/null"
done
gx 'docker run -d --name oneshot-ok-on-failure --restart on-failure alpine:3.22 sh -c "exit 0" >/dev/null; true'
sleep 8
state > "$RAW/0-before-anything.tsv"
# Phase 1: daemon restart only (no reboot)
gx 'sudo systemctl restart docker'; sleep 15
state > "$RAW/1-after-daemon-restart.tsv"
# Phase 2: three reboots. After each, sample every 3 s for 30 s from first ssh success.
# Before EVERY reboot the A/B layout is reset (run containers started, stopped containers stopped by hand), so each reboot starts from the same state.
for n in 1 2 3; do
  for p in "${POLICIES[@]}"; do gx "docker start g-run-$p k-run-$p >/dev/null 2>&1; docker stop g-stopped-$p k-stopped-$p >/dev/null 2>&1"; done
  sleep 3; state > "$RAW/2-before-reboot-$n.tsv"
  gx 'sudo systemctl reboot' </dev/null >/dev/null 2>&1
  sleep 15
  for _ in $(seq 1 80); do gx true 2>/dev/null && break; sleep 2; done
  { for s in 1 2 3 4 5 6 7 8 9 10; do
      echo "## uptime_s=$(gx 'cut -d" " -f1 /proc/uptime') docker_active_since=$(gx 'systemctl show docker -p ActiveEnterTimestampMonotonic --value')"
      state; sleep 3
    done; } > "$RAW/3-reboot-$n.txt" 2>&1
done
# Phase 3: crash loops. 3 identical containers per scenario so timings have replicates.
gx 'for i in 1 2 3; do
  docker run -d --name crash-always-$i --restart always -v $HOME/s:/s:ro alpine:3.22 sh /s/crash.sh >/dev/null
  docker run -d --name crash-onfail3-$i --restart on-failure:3 -v $HOME/s:/s:ro alpine:3.22 sh /s/crash.sh >/dev/null
  docker run -d --name run12-always-$i --restart always -v $HOME/s:/s:ro alpine:3.22 sh /s/run12.sh >/dev/null
done; true'
sleep 270
for c in crash-always-1 crash-always-2 crash-always-3 crash-onfail3-1 crash-onfail3-2 crash-onfail3-3 run12-always-1 run12-always-2 run12-always-3; do
  gx "docker logs $c 2>&1 | tr '\n' ' '; echo; docker inspect -f 'RestartCount={{.RestartCount}} State={{.State.Status}} ExitCode={{.State.ExitCode}}' $c" > "$RAW/4-crash-$c.txt" 2>&1
done
gx 'docker ps -a --format "{{.Names}}\t{{.Status}}" | sort' > "$RAW/4-crash-final-ps.tsv"
collect_env "$OUT"
python3 "$ROOT/lab/tests/$T/summarize.py" "$RAW" "$OUT"

What went wrong

The first run of this test used only sleep as the workload. It showed on-failure returning after every reboot, which contradicted the docs, so the script was changed to also use a process that exits 0 on SIGTERM; that run is the one reported. Its crash-loop timestamps had also come from busybox date +%s.%N, which printed no fractional part, so the script now reads /proc/uptime instead.

What this test did not cover