logtail: count the message a hung sender check defers in cleanup (audit M13)
Seen on the e2e stand, the panel frozen with SIGSTOP once DATA was
answered: the client gets 451 4.7.1 after 30 s, and the warning that names
the socket is cleanup's, not smtpd's:
postfix/cleanup[1339]: warning: milter unix:/run/selfpost/sender.sock:
can't read SMFIC_HEADER reply packet header: Connection timed out
postfix/cleanup[1339]: 7BCAF3C75DA0: milter-reject: END-OF-MESSAGE ...
4.7.1 Service unavailable - try again later; ...
The counter read smtpd lines only, so Health's "messages deferred" missed
the one case that tells a hung panel from a stopped one. milterLineRe now
takes smtpd and cleanup; cleanup's "milter unix:<socket>:" warning counts a
message each time (one cleanup process serves message after message, and the
first warning ends the message); smtpd keeps once per connection. The
composed fixture is replaced by the seen lines, with two messages through one
cleanup process and OpenDKIM hung in cleanup (not counted).
e2e panel_hung_mid_message_is_counted: freezes the panel after DATA's 354,
wants a 4xx at the end of DATA, logs what mail.log says, and waits for
Health's today count to grow. kill through sh -c: the image has no kill(1).
CHANGELOG [Unreleased], audit plan line ticked, roadmap 13/61.
Evidence on the test server for this tree (with the next commit's docs):
go vet clean, go test ./... 23 packages ok, make e2e exit 0 -
renewed_certificate_picked_up 1.2 s, panel_hung_mid_message_is_counted
31.7 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -27,11 +27,21 @@ import (
|
||||
//
|
||||
// It is counted at the warning, which is the one line that names the socket.
|
||||
// The reject lines do not: the same words are written when OpenDKIM does not
|
||||
// answer and when the journal milter refuses under a level-2 limit, and for a
|
||||
// check that stops answering in the middle of a message they are written by
|
||||
// cleanup, another process, under a queue id. The warning for that case is
|
||||
// "milter unix:<socket>: can't read … reply packet header" (Postfix's
|
||||
// milter8.c; not seen on the stand).
|
||||
// answer and when the journal milter refuses under a level-2 limit.
|
||||
//
|
||||
// A check that is running and stops answering in the middle of a message -
|
||||
// its socket accepts, nothing replies - is met by cleanup, not smtpd: past the
|
||||
// DATA command it is cleanup that talks to the milters. Seen on the e2e stand
|
||||
// with the panel frozen (SIGSTOP) once DATA was answered (2026-10-14):
|
||||
//
|
||||
// postfix/cleanup[1339]: warning: milter unix:/run/selfpost/sender.sock: can't read SMFIC_HEADER reply packet header: Connection timed out
|
||||
// postfix/cleanup[1339]: 7BCAF3C75DA0: milter-reject: END-OF-MESSAGE from unknown[10.77.0.1]: 4.7.1 Service unavailable - try again later; from=<…> to=<…> proto=ESMTP helo=<…>
|
||||
//
|
||||
// One cleanup process serves message after message and the first warning
|
||||
// ends the message, so each such warning is a message, counted on its own.
|
||||
// Until 2.5.0 only smtpd's lines were read, and this case - the one that
|
||||
// tells a hung panel from a stopped one - was never counted (audit
|
||||
// 2026-10-12, M13).
|
||||
|
||||
// SenderCheckStore is where the count goes, and where the time of the last
|
||||
// counted line is kept. *store.Store satisfies it.
|
||||
@@ -47,7 +57,7 @@ type SenderCheck struct {
|
||||
Socket string
|
||||
}
|
||||
|
||||
var smtpdLineRe = regexp.MustCompile(`^\S+ postfix/(?:[a-z0-9-]+/)?smtpd\[(\d+)\]: (.*)$`)
|
||||
var milterLineRe = regexp.MustCompile(`^\S+ postfix/(?:[a-z0-9-]+/)?(smtpd|cleanup)\[(\d+)\]: (.*)$`)
|
||||
|
||||
const (
|
||||
// senderCheckPids bounds the processes remembered; smtpd processes are a
|
||||
@@ -89,11 +99,18 @@ func newDeferredCounter(sc *SenderCheck) *deferredCounter {
|
||||
// read to the end of the log for the first time.
|
||||
func (c *deferredCounter) handle(line string, now time.Time, catchingUp bool) {
|
||||
at, rest := lineTime(line, now)
|
||||
m := smtpdLineRe.FindStringSubmatch(rest)
|
||||
m := milterLineRe.FindStringSubmatch(rest)
|
||||
if m == nil {
|
||||
return
|
||||
}
|
||||
pid, text := m[1], m[2]
|
||||
program, pid, text := m[1], m[2], m[3]
|
||||
if program == "cleanup" {
|
||||
// A message cleanup could not finish because the check did not reply.
|
||||
if strings.HasPrefix(text, c.broken) {
|
||||
c.count(at, catchingUp)
|
||||
}
|
||||
return
|
||||
}
|
||||
switch {
|
||||
case strings.HasPrefix(text, "connect from "):
|
||||
delete(c.failed, pid)
|
||||
@@ -109,14 +126,19 @@ func (c *deferredCounter) handle(line string, now time.Time, catchingUp bool) {
|
||||
c.failed = make(map[string]bool)
|
||||
}
|
||||
c.failed[pid] = true
|
||||
if catchingUp && !at.After(c.last) {
|
||||
return
|
||||
}
|
||||
if at.After(c.last) {
|
||||
c.last = at
|
||||
}
|
||||
if err := c.st.AddSenderCheckDeferred(at, 1); err != nil {
|
||||
log.Printf("log-tailer: sender check statistics: %v", err)
|
||||
}
|
||||
c.count(at, catchingUp)
|
||||
}
|
||||
}
|
||||
|
||||
// count adds one deferred message at at, unless it is a line read again.
|
||||
func (c *deferredCounter) count(at time.Time, catchingUp bool) {
|
||||
if catchingUp && !at.After(c.last) {
|
||||
return
|
||||
}
|
||||
if at.After(c.last) {
|
||||
c.last = at
|
||||
}
|
||||
if err := c.st.AddSenderCheckDeferred(at, 1); err != nil {
|
||||
log.Printf("log-tailer: sender check statistics: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user