health: sockets dialled, the certificate Postfix serves, "deferred" only for the queue

Audit 2026-10-12, stage 2c, "Health says what is served"; the pages of
ea8b6bf's mockup.

Sockets (A-38). A unix socket's file outlives a killed OpenDKIM or panel
and refuses every connect() (unix(7)); Health said "Listening" from a
stat. CheckSocket and CheckMilter's unix case now dial: refused is
"Nothing listens on the ... socket: the program that made it has
stopped", EACCES is unknown (the panel may not connect), anything else
"does not answer". The probe closes before a byte, which no milter logs:
libmilter takes the EOF as SMFIC_EOF without a line (sendmail
8.17.1.9-2+deb12u2, the image's libmilter1.0.1, libmilter/comm.c
mi_rd_cmd(), engine.c mi_engine() - its source is now in dev/upstream),
OpenDKIM's mlfi_close() has no connection to free, go-milter v0.4.1
returns quietly on io.EOF (session.go, HandleMilterCommands()). The page
polls every 5 s while in view, so this mattered.

Certificate (A-65). Postfix serves build/postfix-config.sh's copy,
refreshed by cert-reload every TLS_RELOAD_INTERVAL_SECONDS. The panel now
reads smtpd_tls_cert_file from postconf -h at its start (as the retry
policy) and TLS_RELOAD_INTERVAL_SECONDS; CheckServedCertificate checks the
served copy and, when the mounted file differs byte for byte (cert-reload's
own cmp -s), names it as Renewal: its expiry and mtime, two facts on the
card. Waiting is fine for the interval: a nearing expiry of the served one
is then no warning (an expiry still an error); past interval + 1 h it is a
warning pointing at cert-reload's lines. The notifier sends no expiry
notice while a renewal waits on time.

Wording (D-P1, D-P3, and the same for the spam filter): Sender check
note, help topics sender-check, upstream, filter, certificate - word for
word as in the mockup; the filter row's detail too.

Tests: stale socket file (Linux) refused for CheckSocket and CheckMilter;
CheckServedCertificate (same, waiting, late, expired + waiting, no served
path); intervalText; the card's facts; the notifier quiet while a renewal
waits, speaking once it is late. Stand (2026-10-15, make e2e, exit 0):
new health_shows_what_is_served - the three sockets dialled ok by the
panel's user; a certificate mounted without its key is refused by
cert-reload ("do not belong together") and the card shows the served
expiry, "Renewal expires" with the mounted one's, "Renewal mounted", and
"Postfix still serves the certificate above."; a matching pair is then
served again. go vet and go test ./... pass on the server (24 packages).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

internal/health/cert.go

@@ -1,10 +1,13 @@
 package health
 
 import (
+	"bytes"
 	"crypto/x509"
 	"encoding/pem"
 	"fmt"
 	"os"
+	"os/exec"
+	"strings"
 	"time"
 )
 
@@ -25,6 +28,8 @@ const certWarnDays = 14
 // TLS_RELOAD_INTERVAL_SECONDS later, and each smtpd loads it when it starts
 // (Postfix 3.7.11, src/smtpd/smtpd.c, pre_jail_init(), TLS_SERVER_INIT) - so
 // the mounted file can be newer than what is served (assumptions A-65).
+// CheckServedCertificate reads the copy, and names the mounted file as a
+// renewal waiting while the two differ.
 type Certificate struct {
 	Path     string
 	Subject  string
@@ -32,6 +37,98 @@ type Certificate struct {
 	DaysLeft int
 	Status   Status
 	Detail   string
+	// Renewal is the mounted certificate while Postfix serves an older one.
+	Renewal *Renewal
+}
+
+// Renewal is a renewed certificate in the mounted file that Postfix has not
+// taken up yet.
+type Renewal struct {
+	NotAfter time.Time
+	Mounted  time.Time // the mounted file's modification time
+	// Late is true once it has waited longer than cert-reload's interval
+	// and an hour: the check that should have taken it up has passed.
+	Late bool
+}
+
+// ServedCertPath is the certificate file Postfix serves, smtpd_tls_cert_file
+// as postconf prints it (Postfix 3.7.11, postconf(1): -h, the value without
+// its name): build/postfix-config.sh's copy when the mounted files were there
+// at the start, the mounted file itself otherwise. "" when postconf fails.
+// It is read once, at the panel's start, as the retry policy is: main.cf is
+// written before supervisord starts the panel and not after.
+func ServedCertPath() string {
+	out, err := exec.Command("postconf", "-h", "smtpd_tls_cert_file").Output()
+	if err != nil {
+		return ""
+	}
+	return strings.TrimSpace(string(out))
+}
+
+// CheckServedCertificate checks the certificate Postfix serves (served) and,
+// when the mounted file differs from it byte for byte - which is how
+// build/postfix-cert-reload.sh decides to copy it (cmp -s) - reports the
+// mounted one as a renewal waiting. interval is TLS_RELOAD_INTERVAL_SECONDS:
+// cert-reload looks for a renewal that often. A renewal that waits is the
+// normal state for up to that long; the served certificate's nearing expiry
+// is then no warning, its expiry still an error (clients see what is served).
+// One waiting longer is a warning: cert-reload refused it (the key in the
+// mount does not belong to it) or Postfix's reload failed, and cert-reload's
+// lines in the container log say which. With served empty the mounted file is
+// checked, as before.
+func CheckServedCertificate(served, mounted string, interval time.Duration, now time.Time) Certificate {
+	if served == "" {
+		return CheckCertificate(mounted)
+	}
+	c := CheckCertificate(served)
+	if mounted == "" || mounted == served {
+		return c
+	}
+	have, err1 := os.ReadFile(served)
+	want, err2 := os.ReadFile(mounted)
+	fi, err3 := os.Stat(mounted)
+	if err1 != nil || err2 != nil || err3 != nil || bytes.Equal(have, want) {
+		return c
+	}
+	leaf, err := parseLeaf(want)
+	if err != nil {
+		return c
+	}
+	r := &Renewal{NotAfter: leaf.NotAfter, Mounted: fi.ModTime(), Late: now.Sub(fi.ModTime()) > interval+time.Hour}
+	c.Renewal = r
+	switch {
+	case r.Late:
+		c.Status = Worst(c.Status, StatusWarn)
+		c.Detail = fmt.Sprintf("A renewed certificate has been in the mounted file since %s UTC, longer than Postfix's check for a renewal allows, and Postfix still serves the one above. cert-reload's lines in the container log say why.",
+			r.Mounted.UTC().Format("2006-01-02 15:04"))
+	case c.Status == StatusError:
+		c.Detail += " A renewed one is waiting in the mounted file; Postfix takes it up at its next check for a renewal, at most " + intervalText(interval) + " after it was mounted."
+	default:
+		c.Status = StatusOK
+		c.Detail = "Postfix still serves the certificate above. A renewed one is waiting in the mounted file; Postfix takes it up at its next check for a renewal, at most " + intervalText(interval) + " after it was mounted."
+	}
+	return c
+}
+
+// intervalText says an interval the way the help says the default: "a day".
+func intervalText(d time.Duration) string {
+	unit, size := "second", time.Second
+	switch {
+	case d >= 24*time.Hour && d%(24*time.Hour) == 0:
+		unit, size = "day", 24*time.Hour
+	case d >= time.Hour && d%time.Hour == 0:
+		unit, size = "hour", time.Hour
+	case d >= time.Minute && d%time.Minute == 0:
+		unit, size = "minute", time.Minute
+	}
+	switch n := int64(d / size); {
+	case n == 1 && unit == "hour":
+		return "an hour"
+	case n == 1:
+		return "a " + unit
+	default:
+		return fmt.Sprintf("%d %ss", n, unit)
+	}
 }
 
 // CheckCertificate parses the leaf certificate at path and reports how much

internal/health/health_test.go

@@ -10,6 +10,7 @@ import (
 	"net"
 	"os"
 	"path/filepath"
+	"strings"
 	"testing"
 	"time"
 )
@@ -213,3 +214,77 @@ func TestUptime(t *testing.T) {
 		}
 	}
 }
+
+// The certificate shown is the one Postfix serves; a different mounted file is
+// a renewal waiting, a warning once it waited past cert-reload's interval
+// (assumptions A-65).
+func TestCheckServedCertificate(t *testing.T) {
+	dir := t.TempDir()
+	served := filepath.Join(dir, "served.pem")
+	mounted := filepath.Join(dir, "mounted.pem")
+	day := 24 * time.Hour
+
+	// The same file: nothing waits.
+	writeCert(t, served, "mail.example.com", 60*day)
+	b, _ := os.ReadFile(served)
+	if err := os.WriteFile(mounted, b, 0o600); err != nil {
+		t.Fatal(err)
+	}
+	if got := CheckServedCertificate(served, mounted, day, time.Now()); got.Status != StatusOK || got.Renewal != nil || got.Path != served {
+		t.Errorf("same certificate: %+v", got)
+	}
+
+	// Postfix's copy nears its end, a renewal was mounted an hour ago: no
+	// warning, the renewal named.
+	writeCert(t, served, "mail.example.com", 5*day)
+	writeCert(t, mounted, "mail.example.com", 90*day)
+	now := time.Now()
+	if err := os.Chtimes(mounted, now.Add(-time.Hour), now.Add(-time.Hour)); err != nil {
+		t.Fatal(err)
+	}
+	got := CheckServedCertificate(served, mounted, day, now)
+	if got.Status != StatusOK || got.Renewal == nil || got.Renewal.Late || got.DaysLeft > 5 {
+		t.Fatalf("renewal waiting: %+v", got)
+	}
+	if !strings.Contains(got.Detail, "at most a day after it was mounted") {
+		t.Errorf("renewal waiting, detail: %q", got.Detail)
+	}
+	if d := got.Renewal.NotAfter.Sub(now); d < 89*day {
+		t.Errorf("the renewal's expiry is the mounted certificate's: %v", d)
+	}
+
+	// Waiting past the interval and an hour: a warning, cert-reload named.
+	if err := os.Chtimes(mounted, now.Add(-26*time.Hour), now.Add(-26*time.Hour)); err != nil {
+		t.Fatal(err)
+	}
+	got = CheckServedCertificate(served, mounted, day, now)
+	if got.Status != StatusWarn || got.Renewal == nil || !got.Renewal.Late || !strings.Contains(got.Detail, "cert-reload") {
+		t.Errorf("renewal late: %+v", got)
+	}
+
+	// The served one expired: still an error, the renewal said to be coming.
+	writeCert(t, served, "mail.example.com", -day)
+	if err := os.Chtimes(mounted, now, now); err != nil {
+		t.Fatal(err)
+	}
+	got = CheckServedCertificate(served, mounted, day, now)
+	if got.Status != StatusError || got.Renewal == nil || !strings.Contains(got.Detail, "A renewed one is waiting") {
+		t.Errorf("expired, renewal waiting: %+v", got)
+	}
+
+	// No served path known: the mounted file, as before.
+	if got := CheckServedCertificate("", mounted, day, now); got.Path != mounted || got.Renewal != nil {
+		t.Errorf("served path unknown: %+v", got)
+	}
+}
+
+func TestIntervalText(t *testing.T) {
+	for d, want := range map[time.Duration]string{
+		24 * time.Hour: "a day", 48 * time.Hour: "2 days", time.Hour: "an hour",
+		6 * time.Hour: "6 hours", 2 * time.Second: "2 seconds", 90 * time.Minute: "90 minutes",
+	} {
+		if got := intervalText(d); got != want {
+			t.Errorf("intervalText(%v) = %q, want %q", d, got, want)
+		}
+	}
+}

internal/health/milter.go

@@ -54,6 +54,19 @@ func checkMilter(name, addr string, required bool, timeout time.Duration) Socket
 		case fi.Mode()&os.ModeSocket == 0:
 			return fail("The path exists but is not a socket.")
 		}
+		// Dialled as an inet address is: a socket file outlives the
+		// filter that made it (CheckSocket; assumptions A-38).
+		switch err := dialUnix(path); {
+		case err == nil:
+		case errors.Is(err, syscall.EACCES):
+			s.Status = StatusUnknown
+			s.Detail = "The panel may not connect to the socket, so whether the filter listens is not known."
+			return s
+		case errors.Is(err, syscall.ECONNREFUSED):
+			return fail("Nothing listens on the socket: the filter that made it has stopped.")
+		default:
+			return fail("The socket does not answer.")
+		}
 	case strings.HasPrefix(addr, "inet:"):
 		host, port, ok := splitInet(strings.TrimPrefix(addr, "inet:"))
 		if !ok {

internal/health/milter_test.go

@@ -5,6 +5,7 @@ import (
 	"net"
 	"os"
 	"path/filepath"
+	"runtime"
 	"strconv"
 	"strings"
 	"testing"
@@ -144,3 +145,33 @@ func TestCheckMilterInet6(t *testing.T) {
 		t.Errorf("IPv6 listener: %+v", got)
 	}
 }
+
+// A socket file whose listener is gone - a killed OpenDKIM or panel leaves
+// one (unix(7)) - is not a listener, though the file is there (A-38).
+func TestStaleSocketFileIsNotListening(t *testing.T) {
+	if runtime.GOOS == "windows" {
+		t.Skip("a stale AF_UNIX file is a Linux matter here")
+	}
+	sock := filepath.Join(t.TempDir(), "stale.sock")
+	l, err := net.Listen("unix", sock)
+	if err != nil {
+		t.Skipf("unix sockets unavailable here: %v", err)
+	}
+	l.(*net.UnixListener).SetUnlinkOnClose(false)
+	l.Close()
+	if _, err := os.Stat(sock); err != nil {
+		t.Fatalf("the socket file should outlive its listener: %v", err)
+	}
+
+	got := CheckSocket("OpenDKIM", sock, true)
+	if got.Present || got.Status != StatusError || !strings.HasPrefix(got.Detail, "Nothing listens on the OpenDKIM socket") {
+		t.Errorf("stale required socket: %+v", got)
+	}
+	if got := CheckSocket("send-log", sock, false); got.Status != StatusWarn {
+		t.Errorf("stale optional socket: status %q", got.Status)
+	}
+	got = CheckMilter("Spam filter", "unix:"+sock, true)
+	if got.Present || got.Status != StatusError || got.Detail != "Nothing listens on the socket: the filter that made it has stopped." {
+		t.Errorf("stale filter socket: %+v", got)
+	}
+}

internal/health/socket.go

@@ -1,8 +1,12 @@
 package health
 
 import (
+	"errors"
 	"fmt"
+	"net"
 	"os"
+	"syscall"
+	"time"
 )
 
 // Socket is the state of one milter socket Postfix connects to.
@@ -17,7 +21,7 @@ type Socket struct {
 	Detail  string
 }
 
-// CheckSocket stats a milter socket. required distinguishes the two milters:
+// CheckSocket checks a milter socket. required distinguishes the two milters:
 // Postfix attaches OpenDKIM with default_action=tempfail, so a missing socket
 // stops mail leaving the server, while the journal-milter is attached with
 // default_action=accept and fails open — mail still goes out, only the send
@@ -25,10 +29,17 @@ type Socket struct {
 // 3.7.11, src/milter/milter8.c, milter8_comm_error(): tempfail answers "451
 // 4.7.1 Service unavailable - try again later", accept answers nothing).
 //
-// A socket file that is there is reported as listening. It is not proof of
-// it: a unix socket's file outlives the process that bound it unless that
-// process unlinks it (unix(7)), so a killed OpenDKIM or panel leaves one that
-// refuses every connection - assumptions A-38.
+// The socket is dialled, as Postfix will dial it; a file is not a listener.
+// A unix socket's file outlives the process that bound it unless that process
+// unlinks it, and a connect() to it then fails with ECONNREFUSED (unix(7)), so
+// a killed OpenDKIM or panel leaves a file that refuses every connection -
+// assumptions A-38. The connection is closed before a byte is sent, which
+// none of the three milters logs: OpenDKIM's libmilter reads the EOF as
+// SMFIC_EOF and ends the session without a line (sendmail 8.17.1.9,
+// libmilter/comm.c, mi_rd_cmd(); engine.c, mi_engine()), and OpenDKIM's
+// mlfi_close() has no connection to clean up (opendkim 2.11.0~beta2,
+// opendkim/opendkim.c); go-milter v0.4.1 returns quietly on io.EOF
+// (session.go, HandleMilterCommands()).
 func CheckSocket(name, path string, required bool) Socket {
 	s := Socket{Name: name, Path: path}
 	if path == "" {
@@ -45,13 +56,38 @@ func CheckSocket(name, path string, required bool) Socket {
 		s.Status = missingStatus(required)
 		s.Detail = fmt.Sprintf("%s exists but is not a socket.", path)
 	default:
-		s.Present = true
-		s.Status = StatusOK
-		s.Detail = "Listening"
+		switch err := dialUnix(path); {
+		case err == nil:
+			s.Present = true
+			s.Status = StatusOK
+			s.Detail = "Listening"
+		case errors.Is(err, syscall.EACCES):
+			s.Status = StatusUnknown
+			s.Detail = fmt.Sprintf("The panel may not connect to %s, so whether anything listens is not known.", path)
+		case errors.Is(err, syscall.ECONNREFUSED):
+			s.Status = missingStatus(required)
+			s.Detail = fmt.Sprintf("Nothing listens on the %s socket: the program that made it has stopped. %s", name, missingCost(required))
+		default:
+			s.Status = missingStatus(required)
+			s.Detail = fmt.Sprintf("The %s socket does not answer. %s", name, missingCost(required))
+		}
 	}
 	return s
 }
 
+// unixDialTimeout bounds a dial of a local socket; a listener that is there
+// answers at once, from the kernel's backlog.
+const unixDialTimeout = 2 * time.Second
+
+// dialUnix connects to the unix socket at path and closes the connection.
+func dialUnix(path string) error {
+	c, err := net.DialTimeout("unix", path, unixDialTimeout)
+	if err != nil {
+		return err
+	}
+	return c.Close()
+}
+
 func missingStatus(required bool) Status {
 	if required {
 		return StatusError
@@ -60,8 +96,14 @@ func missingStatus(required bool) Status {
 }
 
 func missingDetail(name string, required bool) string {
+	return fmt.Sprintf("The %s socket is missing. %s", name, missingCost(required))
+}
+
+// missingCost is what a milter that cannot be reached costs, by its
+// default_action (see CheckSocket).
+func missingCost(required bool) string {
 	if required {
-		return fmt.Sprintf("The %s socket is missing. Postfix rejects mail with a temporary error until it is back.", name)
+		return "Postfix rejects mail with a temporary error until it is back."
 	}
-	return fmt.Sprintf("The %s socket is missing. Mail still goes out, but the send log is not being written.", name)
+	return "Mail still goes out, but the send log is not being written."
 }