supervision as supervisord does it: Postfix stopped, give-ups FATAL, reload's exit read
test / test (push) Canceled after 0s
test / test (push) Canceled after 0s
C-P42. postfix-script's start-fg runs master as its child and master calls setsid() (Postfix 3.7.11, conf/postfix-script; src/master/master.c); supervisord signals the pid alone (supervisor 4.2.5, process.py, Subprocess.kill()). A stop of program postfix left master running - a restart then found "already running" - and a container stop cut Postfix off with SIGKILL. The wrapper now stays the parent of "postfix start-fg" and on TERM/INT runs "postfix stop", asking again while start-fg is still in its checks (master -t). stopwaitsecs=15. C-V6. The wrapper's give-up on the milter sockets came after the default startsecs, so it was EXITED and restarted for ever. startsecs is now MILTER_WAIT_TIMEOUT + 10, exported by the entrypoint as POSTFIX_STARTSECS and expanded by supervisord (options.py, processes_from_section(); a non-number MILTER_WAIT_TIMEOUT stops the start): a give-up is "Exited too quickly", BACKOFF, FATAL after startretries, crashexit. For those seconds after each start the program is STARTING (the healthcheck's 90 s start period covers it; the sender check counts no downtime then). C-V5. logrotate had startsecs=0, so a refused config restarted it every second for ever. startsecs=1: FATAL after startretries; crashexit reads the event's processname and leaves logrotate's FATAL on Server > Health. B-U2. "supervisorctl start" returns when a startsecs=0 program is RUNNING, not when it ends, and mapped "already started" to success. The panel now speaks supervisord's XML-RPC itself (internal/supervisor, RunOnce): startProcess without waiting, getProcessInfo until it leaves STARTING/RUNNING, exitstatus 0 or an error; a run in progress is waited out and one of its own follows. A failed reload while program postfix is not RUNNING is logged, not an error: the next master reads the files. C-V10: postfix-reload rests EXITED after a run; the e2e process check takes STOPPED or EXITED. C-P34: the dead "postfix set-permissions" branch goes - postfix check's create-missing builds the queue tree. Stand (2026-10-14, make e2e, exit 0; MILTER_WAIT_TIMEOUT=10 on the stand): new supervision_stops_postfix_and_gives_up - supervisorctl restart postfix writes postfix-script's "stopping the Postfix mail system" and leaves one master; with the panel's sockets removed, start answers "postfix: ERROR (spawn error)" (finish() sets spawnerr "Exited too quickly", onwait() reads it first) and Postfix comes up once the panel is back; a group-writable logrotate config ends FATAL with the container up; the panel's Reload configuration answers only after mail.log has "refreshing the Postfix mail system", postfix-reload EXITED. session_survives_restart sees the stop line for the container's restart. Measured: master's own "terminating on signal 15" never reaches mail.log (master TERMs its group, postlogd included, before it logs). go vet and go test ./... pass on the server (24 packages; new unit tests of RunOnce against a fake supervisord). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+31
-12
@@ -1,10 +1,12 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Supervisor event listener: bring the container down on unrecoverable failure.
|
||||
|
||||
If any managed program exhausts its restart retries and enters FATAL, this
|
||||
listener signals supervisord (its parent) to terminate, so the whole container
|
||||
exits and Docker's `restart` policy can recreate it cleanly — rather than
|
||||
lingering "alive" with a dead Postfix/OpenDKIM/panel.
|
||||
If a managed program of the mail path exhausts its restart retries and
|
||||
enters FATAL, this listener signals supervisord (its parent) to terminate, so
|
||||
the whole container exits and Docker's `restart` policy starts it again —
|
||||
rather than lingering "alive" with a dead Postfix/OpenDKIM/panel. logrotate is
|
||||
not on the mail path: its FATAL (a config it would ignore) stays on
|
||||
Server > Health, and mail keeps flowing (assumptions C-V5).
|
||||
|
||||
Communication with supervisord uses the event listener protocol over stdin and
|
||||
stdout, so stdout must carry only protocol tokens.
|
||||
@@ -12,6 +14,9 @@ stdout, so stdout must carry only protocol tokens.
|
||||
supervisor 4.2.5, docs/events.rst, "Event Listener Notification Protocol": the
|
||||
listener writes READY\\n, reads one header line of key:value tokens whose len
|
||||
is the payload's length, reads the payload, and answers "RESULT 2\\nOK".
|
||||
A PROCESS_STATE_FATAL payload is a token set, "processname:cat groupname:cat
|
||||
from_state:BACKOFF" (docs/events.rst; supervisor/events.py,
|
||||
ProcessStateEvent.payload()).
|
||||
supervisord starts the listener as its own child, and on SIGTERM it shuts down
|
||||
(docs/running.rst, "Signals"); it is PID 1 here (build/entrypoint.sh, exec).
|
||||
"""
|
||||
@@ -19,6 +24,9 @@ import os
|
||||
import signal
|
||||
import sys
|
||||
|
||||
# Programs whose FATAL does not take the container down (build/supervisord.conf).
|
||||
SPARED = {"logrotate"}
|
||||
|
||||
|
||||
def write_stdout(s):
|
||||
sys.stdout.write(s)
|
||||
@@ -35,16 +43,27 @@ def main():
|
||||
return
|
||||
headers = dict(pair.split(":", 1) for pair in line.split())
|
||||
payload_len = int(headers.get("len", 0))
|
||||
if payload_len:
|
||||
sys.stdin.read(payload_len)
|
||||
payload = sys.stdin.read(payload_len) if payload_len else ""
|
||||
tokens = dict(
|
||||
pair.split(":", 1) for pair in payload.split() if ":" in pair
|
||||
)
|
||||
name = tokens.get("processname", "")
|
||||
|
||||
# We only subscribe to PROCESS_STATE_FATAL, so any event means a managed
|
||||
# program can no longer be restarted. Take the container down.
|
||||
sys.stderr.write(
|
||||
"crashexit: a managed process entered FATAL; shutting down container\n"
|
||||
)
|
||||
sys.stderr.flush()
|
||||
os.kill(os.getppid(), signal.SIGTERM)
|
||||
# program can no longer be restarted.
|
||||
if name in SPARED:
|
||||
sys.stderr.write(
|
||||
"crashexit: %s entered FATAL; it is not on the mail path, the "
|
||||
"container stays up\n" % name
|
||||
)
|
||||
sys.stderr.flush()
|
||||
else:
|
||||
sys.stderr.write(
|
||||
"crashexit: %s entered FATAL; shutting down container\n"
|
||||
% (name or "a managed process")
|
||||
)
|
||||
sys.stderr.flush()
|
||||
os.kill(os.getppid(), signal.SIGTERM)
|
||||
|
||||
write_stdout("RESULT 2\nOK")
|
||||
|
||||
|
||||
Reference in New Issue
Block a user