Technical review (Opus) and security review (Fable) of 9efc9c7..a59e3d4;
reports in the session's scratchpad. Fixed:
- B1 (verified bug): store.UpdateStatus read before writing in its
transaction; a commit by another connection in between (the dmarc-ingest
process opens the same file) failed it at once with "database is locked"
and the delivery line was lost. The first statement is now a write.
- R3: a delivery line updated every row of its queue id and recipient; it
now updates only the newest (Postfix reuses short ids). The sweep closes
by row id (QueuedDelivery.ID, CloseUnreported(id)).
- R4: a contradicted sweep keeps a mark, reason "reconcile-contradicted",
instead of losing which row it was.
- R1: contradicted cannot see a lost line (never read late). New counter
reconcile_missed: at close, the sweep looks for the row's delivery line
in mail.log and mail.log.1 (QueueLines). 0007 edited - it has run on no
database but throwaway stands.
- R2: queued_over_lifetime cannot measure "what removing the sweep would
leave" while the sweep runs; its text now says rows stuck despite the
sweep, and the rule uses closed - missed - contradicted.
- cert-reload (R5, R6, R7, security medium/low): both files copied to .new
and the copies checked as a pair before both are renamed; acts on the copy
only when postconf says Postfix reads it; a failed reload is retried next
pass; umask 077; says so when the mounted files cannot be read. The e2e
case asserts Postfix reads the copy, the copy equals the mount, the key is
600 root.
- Security low: a recipient with a quoted local part - the journal cut
"a b"@x at the space, deliveryRe cut "a>b"@x at the ">" - matched no
delivery line and was closed by the sweep. cleanAddress ends a bracketed
address at the first ">" outside quotes; deliveryRe takes the recipient
up to the ">, " before relay= (test fixtures given the relay= every real
delivery line has).
- Security low: postqueue -p's error carries at most 512 bytes of output.
- Nits: addDeliveryLogDay refuses a column not on its list; the truncation
test checks its message.
- To stage 2b (page wording, mockup-first): the history step of a bounced
row says the receiving server refused it, false for a row the sweep
closed. To the owner when stage 2b is reached: since M13 the "deferred"
figure counts whole messages too, which "Connections turned away" would
not name.
Found while verifying: the e2e's main positive path failed once with "553
not owned by user app1" - an smtpd idle from before the panel's reload took
the connection in the reload's second. Measured on the stand (one-off
probe, not committed): sending 0, 10, 40, 110 s after the save (two seconds
at the soonest) always met an smtpd started after the reload. The e2e now
waits for the sender map after each application it creates
(waitForSenderMap: AUTH, MAIL, RCPT, RSET - no message, no row), and the
level-1 case uses it instead of its own loop.
Docs: architecture § Queue reconcile, schema-migrations (rows 7 and 8,
chain head 8), CHANGELOG, audit plan (stage 1 note, the measured window,
a new stage 2b line: 62 lines), roadmap 15/62.
Evidence on the test server for this tree: go vet clean, go test ./... 23
packages ok, make e2e exit 0 (renewed certificate, hung panel, level 1 at
49/60).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Migration 0007 (DDL only): delivery_log_days, a row per UTC day, kept a
year, shown nowhere - read off a copy of the database after a month by the
decision rule written in the audit plan.
- reconcile_closed: the sweep adds the rows it actually closed.
- reconcile_contradicted: store.UpdateStatus, in one transaction, counts the
rows with reason "reconcile" a delivery line now speaks of, writes the
log's status and clears the reason - so a re-read line counts once.
- tailer_blind_seconds: follow() tracks stretches in which the file is gone
or reopening fails (logtail.blindness); stored when a stretch ends and
every minute while it lasts; under 3 s not counted (the rotation's tick
between rename and create loses nothing).
- queued_over_lifetime: every sweep overwrites the day's level with the rows
queued past LongestLifetime + MaximalBackoff (a message Postfix still
works on stays queued until its first retry after the lifetime).
- retentionLoop prunes days older than a year.
Tests: store (sums per day, the level overwritten, under a second nothing,
prune; a contradicted sweep counted once and its reason cleared; count of
queued rows); logtail (the sweep counts what it closed; countStuck records
the level; blindness - a one-tick gap not counted, 75 s stored as 60 + 15
from where the last part ended). Schema tests at user_version 7.
schema-migrations.md: chain head 7 (it still said 5 since 2.4.0), row 7.
CHANGELOG [Unreleased], audit plan line ticked, roadmap 10/61.
Evidence: go vet ./... clean; go test ./internal/store ok, ./internal/logtail
ok but for TestFollowTailsAndRotates (Windows); the server run of the stage
follows before the push.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
reconcile() closed a row through UpdateStatus (WHERE queue_id AND to_addr),
which could overwrite a "sent" the tailer wrote between the listing and the
close, and rewrite an old row of a reused queue id. store.CloseUnreported
sets status bounced and reason "reconcile" (store.ReasonReconcile; the
0006 column, no schema change) WHERE ... AND status = 'queued'; the sweep
counts rows actually closed.
Tests: store - a queued row is closed with the reason, a sent row is left
alone (0 rows); logtail - the sweep's close goes through CloseUnreported.
CHANGELOG [Unreleased], audit plan line ticked, roadmap 9/61.
Evidence: go vet ./... clean; go test ./internal/store ok, ./internal/logtail
ok but for TestFollowTailsAndRotates (Windows rename of an open file, fails
here always); the server run of the stage follows before the push.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Neither review (read-only agents, Opus and Fable, over ae06d8b..815236a) has
a Critical or High finding. What they found and what was done:
- A stranger on port 25 could grow the table: every queue-id line of inbound
mail was stored and pruned an hour or more later, his own message-id among
them. The tailer now leaves out the lines of a queue id that
postfix/inbound/smtpd opened, until qmgr removes the message or another
listener opens one under the id; 4096 such ids are remembered.
- A commit, so a disk sync, per stored line on the connection the journal
milter shares, and a backlog written line by line. Lines are now written in
one transaction per pass over the log, 500 at most, before the read offset
is saved (InsertSendLogLines; follow() calls the keeper's flush).
- The prune asked the send log once per line and ran hourly. It asks once per
queue id and runs at start and with the six-hourly sweep; with port-25 mail
left out there is little without a row to wait for.
- The 4096-byte cut could split a character; it keeps characters whole.
- security.md did not say that a backup now carries these lines, and the plan
said the time window excludes a reused queue id where it only limits the
chance to the row's queue lifetime. Both are corrected; the collision is
recorded as an accepted risk with its trigger.
- Tests that did not prove their comments: the prune is now tested with the
sweep, a rejected row next to the prune, batching and the change from
catching up to following, port-25 mail left out and its id reused.
Left as found and written down in the plan: lines the tailer never reads
(a file rotated away while the panel was down), and a message in the queue
across the upgrade, which shows only its later lines.
On the test server, for this tree: go vet clean, go test ./... 22 packages
ok, make e2e ok.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every mail.log line headed by a queue id is written to send_log_lines as the
tailer reads it, dated by the line's own stamp. The id is taken from the head
of the line only - the tail is the far end's text - and has to look like one:
"NOQUEUE:" and "warning:" stand in the same place.
The tailer does not check for a send-log row first: a message's first lines
come before the journal milter writes it. Lines that never get a row are
removed by the prune, which deletes lines older than an hour whose queue id
no row carries. It runs at start, after each send-log sweep and hourly.
While the tailer catches up at start a line is written only if it is not
there already, as the inbound journal does: its place is saved every few
seconds, so an unclean stop means reading some lines twice.
Checked on Windows: go vet clean; go test ./internal/logtail passes except
TestFollowTailsAndRotates, which fails there on renaming an open file and
always did. New tests: which line heads are queue ids, a line dated by its
stamp, no second copy after a rewound offset, the prune cadence and cutoff.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mixandClaude Opus 5.5