supervision as supervisord does it: Postfix stopped, give-ups FATAL, reload's exit read
test / test (push) Canceled after 0s

C-P42. postfix-script's start-fg runs master as its child and master
calls setsid() (Postfix 3.7.11, conf/postfix-script; src/master/master.c);
supervisord signals the pid alone (supervisor 4.2.5, process.py,
Subprocess.kill()). A stop of program postfix left master running - a
restart then found "already running" - and a container stop cut Postfix
off with SIGKILL. The wrapper now stays the parent of "postfix start-fg"
and on TERM/INT runs "postfix stop", asking again while start-fg is still
in its checks (master -t). stopwaitsecs=15.

C-V6. The wrapper's give-up on the milter sockets came after the default
startsecs, so it was EXITED and restarted for ever. startsecs is now
MILTER_WAIT_TIMEOUT + 10, exported by the entrypoint as POSTFIX_STARTSECS
and expanded by supervisord (options.py, processes_from_section(); a
non-number MILTER_WAIT_TIMEOUT stops the start): a give-up is "Exited too
quickly", BACKOFF, FATAL after startretries, crashexit. For those seconds
after each start the program is STARTING (the healthcheck's 90 s start
period covers it; the sender check counts no downtime then).

C-V5. logrotate had startsecs=0, so a refused config restarted it every
second for ever. startsecs=1: FATAL after startretries; crashexit reads
the event's processname and leaves logrotate's FATAL on Server > Health.

B-U2. "supervisorctl start" returns when a startsecs=0 program is RUNNING,
not when it ends, and mapped "already started" to success. The panel now
speaks supervisord's XML-RPC itself (internal/supervisor, RunOnce):
startProcess without waiting, getProcessInfo until it leaves
STARTING/RUNNING, exitstatus 0 or an error; a run in progress is waited
out and one of its own follows. A failed reload while program postfix is
not RUNNING is logged, not an error: the next master reads the files.
C-V10: postfix-reload rests EXITED after a run; the e2e process check
takes STOPPED or EXITED. C-P34: the dead "postfix set-permissions" branch
goes - postfix check's create-missing builds the queue tree.

Stand (2026-10-14, make e2e, exit 0; MILTER_WAIT_TIMEOUT=10 on the stand):
new supervision_stops_postfix_and_gives_up - supervisorctl restart postfix
writes postfix-script's "stopping the Postfix mail system" and leaves one
master; with the panel's sockets removed, start answers "postfix: ERROR
(spawn error)" (finish() sets spawnerr "Exited too quickly", onwait()
reads it first) and Postfix comes up once the panel is back; a
group-writable logrotate config ends FATAL with the container up; the
panel's Reload configuration answers only after mail.log has "refreshing
the Postfix mail system", postfix-reload EXITED. session_survives_restart
sees the stop line for the container's restart. Measured: master's own
"terminating on signal 15" never reaches mail.log (master TERMs its group,
postlogd included, before it logs). go vet and go test ./... pass on the
server (24 packages; new unit tests of RunOnce against a fake supervisord).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
mix
2026-10-14 23:49:25 +03:00
co-authored by Claude Opus 5.5
parent ee37d16dc7
commit f2ad699473
15 changed files with 834 additions and 59 deletions
+31 -12
View File
@@ -1,10 +1,12 @@
#!/usr/bin/env python3
"""Supervisor event listener: bring the container down on unrecoverable failure.
If any managed program exhausts its restart retries and enters FATAL, this
listener signals supervisord (its parent) to terminate, so the whole container
exits and Docker's `restart` policy can recreate it cleanly — rather than
lingering "alive" with a dead Postfix/OpenDKIM/panel.
If a managed program of the mail path exhausts its restart retries and
enters FATAL, this listener signals supervisord (its parent) to terminate, so
the whole container exits and Docker's `restart` policy starts it again —
rather than lingering "alive" with a dead Postfix/OpenDKIM/panel. logrotate is
not on the mail path: its FATAL (a config it would ignore) stays on
Server > Health, and mail keeps flowing (assumptions C-V5).
Communication with supervisord uses the event listener protocol over stdin and
stdout, so stdout must carry only protocol tokens.
@@ -12,6 +14,9 @@ stdout, so stdout must carry only protocol tokens.
supervisor 4.2.5, docs/events.rst, "Event Listener Notification Protocol": the
listener writes READY\\n, reads one header line of key:value tokens whose len
is the payload's length, reads the payload, and answers "RESULT 2\\nOK".
A PROCESS_STATE_FATAL payload is a token set, "processname:cat groupname:cat
from_state:BACKOFF" (docs/events.rst; supervisor/events.py,
ProcessStateEvent.payload()).
supervisord starts the listener as its own child, and on SIGTERM it shuts down
(docs/running.rst, "Signals"); it is PID 1 here (build/entrypoint.sh, exec).
"""
@@ -19,6 +24,9 @@ import os
import signal
import sys
# Programs whose FATAL does not take the container down (build/supervisord.conf).
SPARED = {"logrotate"}
def write_stdout(s):
sys.stdout.write(s)
@@ -35,16 +43,27 @@ def main():
return
headers = dict(pair.split(":", 1) for pair in line.split())
payload_len = int(headers.get("len", 0))
if payload_len:
sys.stdin.read(payload_len)
payload = sys.stdin.read(payload_len) if payload_len else ""
tokens = dict(
pair.split(":", 1) for pair in payload.split() if ":" in pair
)
name = tokens.get("processname", "")
# We only subscribe to PROCESS_STATE_FATAL, so any event means a managed
# program can no longer be restarted. Take the container down.
sys.stderr.write(
"crashexit: a managed process entered FATAL; shutting down container\n"
)
sys.stderr.flush()
os.kill(os.getppid(), signal.SIGTERM)
# program can no longer be restarted.
if name in SPARED:
sys.stderr.write(
"crashexit: %s entered FATAL; it is not on the mail path, the "
"container stays up\n" % name
)
sys.stderr.flush()
else:
sys.stderr.write(
"crashexit: %s entered FATAL; shutting down container\n"
% (name or "a managed process")
)
sys.stderr.flush()
os.kill(os.getppid(), signal.SIGTERM)
write_stdout("RESULT 2\nOK")