supervision as supervisord does it: Postfix stopped, give-ups FATAL, reload's exit read

C-P42. postfix-script's start-fg runs master as its child and master
calls setsid() (Postfix 3.7.11, conf/postfix-script; src/master/master.c);
supervisord signals the pid alone (supervisor 4.2.5, process.py,
Subprocess.kill()). A stop of program postfix left master running - a
restart then found "already running" - and a container stop cut Postfix
off with SIGKILL. The wrapper now stays the parent of "postfix start-fg"
and on TERM/INT runs "postfix stop", asking again while start-fg is still
in its checks (master -t). stopwaitsecs=15.

C-V6. The wrapper's give-up on the milter sockets came after the default
startsecs, so it was EXITED and restarted for ever. startsecs is now
MILTER_WAIT_TIMEOUT + 10, exported by the entrypoint as POSTFIX_STARTSECS
and expanded by supervisord (options.py, processes_from_section(); a
non-number MILTER_WAIT_TIMEOUT stops the start): a give-up is "Exited too
quickly", BACKOFF, FATAL after startretries, crashexit. For those seconds
after each start the program is STARTING (the healthcheck's 90 s start
period covers it; the sender check counts no downtime then).

C-V5. logrotate had startsecs=0, so a refused config restarted it every
second for ever. startsecs=1: FATAL after startretries; crashexit reads
the event's processname and leaves logrotate's FATAL on Server > Health.

B-U2. "supervisorctl start" returns when a startsecs=0 program is RUNNING,
not when it ends, and mapped "already started" to success. The panel now
speaks supervisord's XML-RPC itself (internal/supervisor, RunOnce):
startProcess without waiting, getProcessInfo until it leaves
STARTING/RUNNING, exitstatus 0 or an error; a run in progress is waited
out and one of its own follows. A failed reload while program postfix is
not RUNNING is logged, not an error: the next master reads the files.
C-V10: postfix-reload rests EXITED after a run; the e2e process check
takes STOPPED or EXITED. C-P34: the dead "postfix set-permissions" branch
goes - postfix check's create-missing builds the queue tree.

Stand (2026-10-14, make e2e, exit 0; MILTER_WAIT_TIMEOUT=10 on the stand):
new supervision_stops_postfix_and_gives_up - supervisorctl restart postfix
writes postfix-script's "stopping the Postfix mail system" and leaves one
master; with the panel's sockets removed, start answers "postfix: ERROR
(spawn error)" (finish() sets spawnerr "Exited too quickly", onwait()
reads it first) and Postfix comes up once the panel is back; a
group-writable logrotate config ends FATAL with the container up; the
panel's Reload configuration answers only after mail.log has "refreshing
the Postfix mail system", postfix-reload EXITED. session_survives_restart
sees the stop line for the container's restart. Measured: master's own
"terminating on signal 15" never reaches mail.log (master TERMs its group,
postlogd included, before it logs). go vet and go test ./... pass on the
server (24 packages; new unit tests of RunOnce against a fake supervisord).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

build/crashexit.py

@@ -1,10 +1,12 @@
 #!/usr/bin/env python3
 """Supervisor event listener: bring the container down on unrecoverable failure.
 
-If any managed program exhausts its restart retries and enters FATAL, this
-listener signals supervisord (its parent) to terminate, so the whole container
-exits and Docker's `restart` policy can recreate it cleanly — rather than
-lingering "alive" with a dead Postfix/OpenDKIM/panel.
+If a managed program of the mail path exhausts its restart retries and
+enters FATAL, this listener signals supervisord (its parent) to terminate, so
+the whole container exits and Docker's `restart` policy starts it again —
+rather than lingering "alive" with a dead Postfix/OpenDKIM/panel. logrotate is
+not on the mail path: its FATAL (a config it would ignore) stays on
+Server > Health, and mail keeps flowing (assumptions C-V5).
 
 Communication with supervisord uses the event listener protocol over stdin and
 stdout, so stdout must carry only protocol tokens.
@@ -12,6 +14,9 @@ stdout, so stdout must carry only protocol tokens.
 supervisor 4.2.5, docs/events.rst, "Event Listener Notification Protocol": the
 listener writes READY\\n, reads one header line of key:value tokens whose len
 is the payload's length, reads the payload, and answers "RESULT 2\\nOK".
+A PROCESS_STATE_FATAL payload is a token set, "processname:cat groupname:cat
+from_state:BACKOFF" (docs/events.rst; supervisor/events.py,
+ProcessStateEvent.payload()).
 supervisord starts the listener as its own child, and on SIGTERM it shuts down
 (docs/running.rst, "Signals"); it is PID 1 here (build/entrypoint.sh, exec).
 """
@@ -19,6 +24,9 @@ import os
 import signal
 import sys
 
+# Programs whose FATAL does not take the container down (build/supervisord.conf).
+SPARED = {"logrotate"}
+
 
 def write_stdout(s):
     sys.stdout.write(s)
@@ -35,16 +43,27 @@ def main():
             return
         headers = dict(pair.split(":", 1) for pair in line.split())
         payload_len = int(headers.get("len", 0))
-        if payload_len:
-            sys.stdin.read(payload_len)
+        payload = sys.stdin.read(payload_len) if payload_len else ""
+        tokens = dict(
+            pair.split(":", 1) for pair in payload.split() if ":" in pair
+        )
+        name = tokens.get("processname", "")
 
         # We only subscribe to PROCESS_STATE_FATAL, so any event means a managed
-        # program can no longer be restarted. Take the container down.
-        sys.stderr.write(
-            "crashexit: a managed process entered FATAL; shutting down container\n"
-        )
-        sys.stderr.flush()
-        os.kill(os.getppid(), signal.SIGTERM)
+        # program can no longer be restarted.
+        if name in SPARED:
+            sys.stderr.write(
+                "crashexit: %s entered FATAL; it is not on the mail path, the "
+                "container stays up\n" % name
+            )
+            sys.stderr.flush()
+        else:
+            sys.stderr.write(
+                "crashexit: %s entered FATAL; shutting down container\n"
+                % (name or "a managed process")
+            )
+            sys.stderr.flush()
+            os.kill(os.getppid(), signal.SIGTERM)
 
         write_stdout("RESULT 2\nOK")
 

build/entrypoint.sh

@@ -177,11 +177,12 @@ rm -f /run/opendkim/*.sock /run/selfpost/*.sock /run/selfpost/*.alive
 # start the same way the /data normalisation above is.
 /usr/local/bin/postfix-config.sh
 
-# Initialise the persistent queue tree on first start or after restore. postfix
-# set-permissions reads queue_directory from main.cf (set by postfix-config.sh).
-if [ ! -d /data/postfix/queue/active ]; then
-	postfix set-permissions
-fi
+# The queue tree, on a first start or after a restore, is already built here:
+# postfix-config.sh ends with "postfix check", whose check-fatal runs
+# "post-install create-missing" (Postfix 3.7.11, conf/postfix-script;
+# conf/postfix-files, "$queue_directory/active:d:$mail_owner:-:700:ucr").
+# A "postfix set-permissions" for a missing active/ stood here and never ran
+# (assumptions C-P34).
 chown -R postfix:postfix /data/postfix/queue
 # The recursive chown above would leave public/ and maildrop/ in group postfix.
 # Postfix keeps those two in its setgid group: postqueue and postdrop are setgid
@@ -199,4 +200,17 @@ chgrp postdrop /data/postfix/queue/public /data/postfix/queue/maildrop
 # root".)
 chown root /data/postfix/queue /data/postfix/queue/pid
 
+# [program:postfix]'s startsecs (build/supervisord.conf): longer than the
+# wrapper's wait for the milter sockets, so its exit at the end of the wait is
+# one supervisord counts as too quick. A value that is not a number stops the
+# start here.
+wait_timeout="${MILTER_WAIT_TIMEOUT:-30}"
+case "$wait_timeout" in
+'' | *[!0-9]*)
+	echo "entrypoint: MILTER_WAIT_TIMEOUT must be a whole number of seconds, got '$wait_timeout'" >&2
+	exit 1
+	;;
+esac
+export POSTFIX_STARTSECS=$((wait_timeout + 10))
+
 exec /usr/bin/supervisord -c /etc/supervisor/supervisord.conf

build/postfix-wrapper.sh

@@ -4,10 +4,15 @@
 #
 # It blocks until the milter sockets — OpenDKIM, the panel's sender check and
 # journal-milter and, with the inbound relay on, its inbound journal milter —
-# are present,
-# then execs `postfix start-fg`. If they are not ready within the
-# timeout it exits non-zero WITHOUT starting Postfix, so supervisord/Docker see
-# an explicit startup failure instead of a relay running blind.
+# are present, then runs `postfix start-fg` and stays its parent. If they are
+# not ready within the timeout it exits non-zero WITHOUT starting Postfix.
+# supervisord counts that exit as one that came too quickly - the program's
+# startsecs is the timeout plus ten seconds (build/entrypoint.sh) - so after
+# startretries it is FATAL and crashexit takes the container down, instead of
+# a relay running blind (supervisor 4.2.5, supervisor/process.py,
+# Subprocess.finish(): an exit before startsecs is "Exited too quickly" and
+# goes to BACKOFF; transition(): BACKOFF past startretries is FATAL;
+# assumptions C-V6).
 #
 # This handles cold start only. Once Postfix runs, a milter that does not
 # answer costs what its default_action says (build/postfix-config.sh): mail is
@@ -42,5 +47,41 @@ echo "postfix-wrapper: milter sockets ready, starting postfix"
 # Postfix 3.7.11: postfix(1) execs conf/postfix-script (src/postfix/postfix.c),
 # whose "start-fg" execs master only as PID 1 and otherwise runs "master -s" as
 # its child; master then makes itself a session leader (src/master/master.c,
-# setsid()). The process supervisord watches is that shell, not master.
-exec postfix start-fg
+# setsid()). A signal supervisord sends this program never reaches master
+# (supervisor 4.2.5, supervisor/process.py, Subprocess.kill(): to the pid
+# alone without stopasgroup, and master is not in its group anyway): before,
+# a stop left master running - a restart of the program then found Postfix
+# "already running", and a stop of the container cut Postfix off with
+# SIGKILL (assumptions C-P42). So this shell stays, and on TERM or INT runs
+# "postfix stop" (conf/postfix-script, "stop": TERM to master, then waits up
+# to five seconds and kills its group; src/master/master_sig.c,
+# master_sigdeath(): "terminating on signal 15").
+#
+# "postfix start-fg" spends its first seconds in its checks before master
+# runs; a stop asked then would find nothing to stop, so it is asked again
+# until the child is gone. "master -t" exits 0, silently, while no master
+# holds the lock (src/master/master.c, test_lock).
+daemon_directory=$(postconf -h daemon_directory)
+child=
+stop_postfix() {
+	trap '' TERM INT
+	while [ -n "$child" ] && kill -0 "$child" 2>/dev/null; do
+		if "$daemon_directory/master" -t 2>/dev/null; then
+			sleep 1
+		else
+			postfix stop || true
+		fi
+	done
+	exit 0
+}
+trap stop_postfix TERM INT
+postfix start-fg &
+child=$!
+# A trapped signal makes wait return at once, the trap is taken right after
+# (POSIX, Shell Command Language, 2.11 Signals and Error Handling) and exits.
+# Otherwise wait returns when Postfix ended on its own: its status is this
+# program's, and supervisord restarts it.
+status=0
+wait "$child" || status=$?
+echo "postfix-wrapper: postfix exited with status $status" >&2
+exit "$status"

build/supervisord.conf

@@ -8,9 +8,15 @@
 ; supervisord.py, runforever(), transitions every group in that order in one
 ; pass, with no wait for a program to be ready.)
 ;
-; If any managed process exhausts its restart retries (FATAL), the crashexit
-; event listener brings the whole container down so Docker's restart policy can
-; recreate it cleanly, rather than lingering with a dead component.
+; If a managed process of the mail path exhausts its restart retries (FATAL),
+; the crashexit event listener brings the whole container down so Docker's
+; restart policy starts it again (the same container: docs.docker.com, "Start
+; containers automatically", restart policies), rather than lingering with a
+; dead component. A program reaches FATAL only through BACKOFF, by exits that
+; come before its startsecs (supervisor 4.2.5, supervisor/process.py,
+; Subprocess.finish() and transition()): an exit after it is EXITED and simply
+; restarted, for ever. logrotate's FATAL is left on Server > Health
+; (build/crashexit.py).
 
 [supervisord]
 nodaemon=true
@@ -66,12 +72,22 @@ stdout_logfile_maxbytes=0
 stderr_logfile=/dev/stderr
 stderr_logfile_maxbytes=0
 
+; startsecs is MILTER_WAIT_TIMEOUT plus ten seconds (build/entrypoint.sh): the
+; wrapper's exit when the milter sockets never come counts as too quick, and
+; so does a Postfix that will not start. The price: for that long after each
+; start the program is STARTING, not RUNNING (supervisor 4.2.5,
+; supervisor/options.py, processes_from_section(): startsecs is expanded like
+; any value, %(ENV_x)s from supervisord's environment; assumptions C-V6).
+; stopwaitsecs covers "postfix stop", which waits five seconds for master
+; before it kills (build/postfix-wrapper.sh).
 [program:postfix]
 command=/usr/local/bin/postfix-wrapper.sh
 priority=300
 autostart=true
 autorestart=true
+startsecs=%(ENV_POSTFIX_STARTSECS)s
 startretries=3
+stopwaitsecs=15
 stdout_logfile=/dev/stdout
 stdout_logfile_maxbytes=0
 stderr_logfile=/dev/stderr
@@ -88,7 +104,9 @@ stderr_logfile_maxbytes=0
 ; crashexit listener. (start-fg: see build/postfix-wrapper.sh. supervisor
 ; 4.2.5, supervisor/process.py, Subprocess.finish(): with startsecs=0 no exit
 ; counts as "too quickly", so even a failed run ends EXITED - never BACKOFF,
-; never FATAL.)
+; never FATAL - with its exit status kept, which the panel reads
+; (internal/supervisor, RunOnce). Its resting state is STOPPED until the first
+; run, EXITED after it; assumptions C-V10.)
 [program:postfix-reload]
 command=/usr/sbin/postfix reload
 autostart=false
@@ -117,16 +135,21 @@ stdout_logfile_maxbytes=0
 stderr_logfile=/dev/stderr
 stderr_logfile_maxbytes=0
 
-; Periodic logrotate for /data/log/mail.log (daily, 7-14 files kept in the
-; image). Runs as root so logrotate can read/rotate the log. Exits non-zero when
-; the config is group/other-writable (logrotate would ignore it silently);
-; autorestart surfaces BACKOFF on the Status page.
+; Periodic logrotate for /data/log/mail.log (daily, 14 files kept). Runs as
+; root so logrotate can read/rotate the log. Exits non-zero at once when the
+; config is group/other-writable (logrotate would ignore it silently): with
+; startsecs=1 that exit is too quick; once startretries are spent the program
+; is FATAL, and Server > Health shows it. With startsecs=0 it was restarted
+; every second for ever, RUNNING and EXITED in turn (supervisor 4.2.5,
+; supervisor/process.py, Subprocess.finish(); assumptions C-V5). crashexit
+; leaves this FATAL alone: logrotate is not on the mail path.
 [program:logrotate]
 command=/usr/local/bin/logrotate-loop.sh
 priority=400
 autostart=true
 autorestart=true
-startsecs=0
+startsecs=1
+startretries=3
 stdout_logfile=/dev/stdout
 stdout_logfile_maxbytes=0
 stderr_logfile=/dev/stderr