mixandClaude Opus 5.5
81df131217
Every mail.log line headed by a queue id is written to send_log_lines as the tailer reads it, dated by the line's own stamp. The id is taken from the head of the line only - the tail is the far end's text - and has to look like one: "NOQUEUE:" and "warning:" stand in the same place. The tailer does not check for a send-log row first: a message's first lines come before the journal milter writes it. Lines that never get a row are removed by the prune, which deletes lines older than an hour whose queue id no row carries. It runs at start, after each send-log sweep and hourly. While the tailer catches up at start a line is written only if it is not there already, as the inbound journal does: its place is saved every few seconds, so an unclean stop means reading some lines twice. Checked on Windows: go vet clean; go test ./internal/logtail passes except TestFollowTailsAndRotates, which fails there on renaming an open file and always did. New tests: which line heads are queue ids, a line dated by its stamp, no second copy after a rewound offset, the prune cadence and cutoff. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
83 lines
3.2 KiB
Go
83 lines
3.2 KiB
Go
package logtail
|
|
|
|
import (
|
|
"log"
|
|
"regexp"
|
|
"time"
|
|
|
|
"github.com/mixeme/selfpost/internal/store"
|
|
)
|
|
|
|
// A message's page shows what Postfix wrote about it. mail.log is rotated
|
|
// daily and fourteen files are kept, while a send-log row lives for the
|
|
// retention window, so the lines are kept in the database as the tailer reads
|
|
// them (architecture.md § Log tailer).
|
|
//
|
|
// queueLineRe finds the queue id at the head of a line, after the timestamp
|
|
// (already taken off by lineTime), the host name and "program[pid]:":
|
|
//
|
|
// mail postfix/smtp[26]: 41E862C00D9E: to=<a@example.net>, relay=…, status=sent (250 OK)
|
|
//
|
|
// It is matched from the start of the line and nowhere else: the tail of a
|
|
// line is the far end's words - a reply, a HELO name - and a pattern that
|
|
// could start anywhere would let them file a line under a queue id of their
|
|
// choosing.
|
|
//
|
|
// What stands in that place is not always a queue id: "NOQUEUE:" and
|
|
// "warning:" stand there too. A queue id is hexadecimal in capitals, or, with
|
|
// enable_long_queue_ids, digits and letters without vowels; neither word is
|
|
// either.
|
|
var queueLineRe = regexp.MustCompile(`^\S+ [^\s\[]+\[\d+\]: ([0-9A-F]{6,24}|[0-9B-DF-HJ-NP-TV-Zb-df-hj-np-tv-z]{10,24}): `)
|
|
|
|
// lineGrace is how old a line must be before the prune asks whether a send-log
|
|
// row carries its queue id. The first lines of a message are logged when it
|
|
// arrives and its row is written at end-of-message; an hour is a message body
|
|
// sent slowly.
|
|
const lineGrace = time.Hour
|
|
|
|
// linePruneInterval is how often lines without a row are removed between two
|
|
// send-log sweeps. With the inbound relay on, most of what Postfix logs under
|
|
// a queue id is mail that never gets a send-log row; it is not left to pile up
|
|
// for six hours. A var so tests can shorten it.
|
|
var linePruneInterval = time.Hour
|
|
|
|
// storeLine keeps line if it is headed by a queue id. It does not ask whether
|
|
// a send-log row exists for the id: for a message's first lines none does yet,
|
|
// and the lines of messages that never get one are removed by pruneLines.
|
|
//
|
|
// catchingUp is as in inboundJournal.handle: the line may have been read
|
|
// before the last stop, and is then written only if it is not there already.
|
|
// The line is dated by its own stamp - after the panel was down, that and now
|
|
// are hours apart, and a line is matched to its row by time.
|
|
func storeLine(st StatusStore, line string, now time.Time, catchingUp bool) {
|
|
at, rest := lineTime(line, now)
|
|
m := queueLineRe.FindStringSubmatch(rest)
|
|
if m == nil {
|
|
return
|
|
}
|
|
l := store.SendLogLine{QueueID: m[1], At: at, Line: line}
|
|
var err error
|
|
if catchingUp {
|
|
_, err = st.InsertSendLogLineOnce(l)
|
|
} else {
|
|
err = st.InsertSendLogLine(l)
|
|
}
|
|
if err != nil {
|
|
log.Printf("log-tailer: keep line of %s: %v", m[1], err)
|
|
}
|
|
}
|
|
|
|
// pruneLines removes the stored lines no send-log row carries the queue id of:
|
|
// those of rows the retention sweep deleted, and those of messages that never
|
|
// had a row.
|
|
func pruneLines(st StatusStore) {
|
|
n, err := st.PruneSendLogLines(time.Now().UTC().Add(-lineGrace))
|
|
if err != nil {
|
|
log.Printf("log-tailer: prune stored log lines: %v", err)
|
|
return
|
|
}
|
|
if n > 0 {
|
|
log.Printf("log-tailer: pruned %d stored log lines with no send-log row", n)
|
|
}
|
|
}
|