Docker logs grow forever by default: measured caps

Set max-size and max-file on every long-running container, or set them once in /etc/docker/daemon.json and then recreate the containers that already exist. The Docker docs list max-size as unlimited (-1) and max-file as 1 for json-file (JSON File driver docs), and docker info on the test box reported json-file as the logging driver; the measurements below show what that costs and what the caps do.

How fast an unlimited log grows

The generator was an Alpine 3.22 container that prints one 100-byte-ish request line in a loop with sleep 0.001. It reached about 594 lines per second (median of three 40 s runs; an earlier run of the same script, since replaced, gave 632 lines per second and 106.0 KiB/s, so the rate varies between runs and machines), and the container’s log file (*-json.log) grew like this:

The first rows are the unlimited case. 172 bytes per line on disk is the line plus the JSON wrapper that json-file adds. A straight-line extrapolation of 99.5 KiB/s is 8.80 GB per day; that figure is arithmetic on the measured rate, not a 24 h measurement. A loop with no sleep is much faster: in one 10 s run, yes wrote 648.21 MiB (819.88 MiB in the earlier run).

What max-size and max-file do

With --log-opt max-size=1m --log-opt max-file=3 the log stayed bounded: the largest of 12 samples over 120 s was 2.43 MiB, and the end state was three files:

json.log     557,537 B
json.log.1 1,000,000 B
json.log.2 1,000,169 B

1m rotated at about 1,000,000 bytes, not 1,048,576, in this test. The cost is history: of the 72,768 lines the container had emitted in 120 s, docker logs could return 14,794 (20.3%), because older files are discarded; the container was killed before both numbers were read, and the share falls the longer the container runs and varies with where the rotation is. Size the cap for how far back you need to look, or ship logs elsewhere.

The same options in a compose file produced the same result (2.24 MiB after 40 s, and docker inspect reported "max-file":"3","max-size":"1m"):

services:
  noisy:
    image: alpine:3.22
    logging:
      driver: json-file
      options:
        max-size: "1m"
        max-file: "3"

The daemon.json trap

The Docker docs say existing containers do not use new logging configuration automatically. The test confirmed it and shows how far that goes. After writing this file and restarting the daemon:

{"log-driver":"json-file","log-opts":{"max-size":"1m","max-file":"2"}}

The container created before the change (restart: always, so it came back by itself) still had an empty LogConfig and reached 10.23 MiB after a docker restart. A container created afterward carried max-size=1m, max-file=2 and sat at 1.63 MiB; one that passed its own --log-opt max-size=10m kept its own value. Removing and recreating the old container applied the new defaults. The test did not check whether docker compose up -d recreates a container when only daemon.json changed.

The local driver

The local driver stores logs in its own format and, per the docs, defaults to 20 MB per file, 5 files and compression enabled, about 100 MB per container. On the same fixed workload of 200,000 access-log-style lines:

local used 0.69 of the space of json-file for identical content, and docker logs returned the same 200,000 lines from both. Under the noisy loop with no options, the local container held 1.54 MiB, 3.13 MiB and 4.69 MiB at 20, 40 and 60 s, so it grows steadily too; its bound comes from the default rotation settings the docs describe, which this 60 s run did not reach. Choosing local is a way to get bounded logs without writing options.

What went wrong

The first run of the script discarded its own data: it named a container y, which Docker rejects (names need at least two characters), and its fixed-workload parser read an extra empty field. Both were fixed and the whole test was rerun; the numbers above are from the rerun.

Reproduce

#!/usr/bin/env bash
# docker-logs: json-file log growth with defaults, max-size/max-file, daemon.json scope, and the local driver.
set -uo pipefail
source "$(dirname "$0")/../_lib.sh"
T=docker-logs; OUT="$ROOT/evidence/tests/$T"; RAW="$OUT/raw"; rm -rf "$OUT"; mkdir -p "$RAW"
start_vm a-$T
vm scp-to "$ROOT/lab/tests/$T/guest.sh" guest.sh
gx 'bash guest.sh' > "$RAW/guest-stdout.txt" 2>&1
vm scp-from out "$RAW/"; mv "$RAW/out"/* "$RAW/"; rmdir "$RAW/out"
gx 'cat /etc/docker/daemon.json 2>&1; docker info -f "{{.LoggingDriver}}"' > "$RAW/daemon-after-cleanup.txt" 2>&1
collect_env "$OUT"
python3 "$ROOT/lab/tests/$T/summarize.py" "$RAW" "$OUT"

What this test did not cover