findings of the reviews of audit stage 1
test / test (push) Canceled after 0s

Technical review (Opus) and security review (Fable) of 9efc9c7..a59e3d4;
reports in the session's scratchpad. Fixed:

- B1 (verified bug): store.UpdateStatus read before writing in its
  transaction; a commit by another connection in between (the dmarc-ingest
  process opens the same file) failed it at once with "database is locked"
  and the delivery line was lost. The first statement is now a write.
- R3: a delivery line updated every row of its queue id and recipient; it
  now updates only the newest (Postfix reuses short ids). The sweep closes
  by row id (QueuedDelivery.ID, CloseUnreported(id)).
- R4: a contradicted sweep keeps a mark, reason "reconcile-contradicted",
  instead of losing which row it was.
- R1: contradicted cannot see a lost line (never read late). New counter
  reconcile_missed: at close, the sweep looks for the row's delivery line
  in mail.log and mail.log.1 (QueueLines). 0007 edited - it has run on no
  database but throwaway stands.
- R2: queued_over_lifetime cannot measure "what removing the sweep would
  leave" while the sweep runs; its text now says rows stuck despite the
  sweep, and the rule uses closed - missed - contradicted.
- cert-reload (R5, R6, R7, security medium/low): both files copied to .new
  and the copies checked as a pair before both are renamed; acts on the copy
  only when postconf says Postfix reads it; a failed reload is retried next
  pass; umask 077; says so when the mounted files cannot be read. The e2e
  case asserts Postfix reads the copy, the copy equals the mount, the key is
  600 root.
- Security low: a recipient with a quoted local part - the journal cut
  "a b"@x at the space, deliveryRe cut "a>b"@x at the ">" - matched no
  delivery line and was closed by the sweep. cleanAddress ends a bracketed
  address at the first ">" outside quotes; deliveryRe takes the recipient
  up to the ">, " before relay= (test fixtures given the relay= every real
  delivery line has).
- Security low: postqueue -p's error carries at most 512 bytes of output.
- Nits: addDeliveryLogDay refuses a column not on its list; the truncation
  test checks its message.
- To stage 2b (page wording, mockup-first): the history step of a bounced
  row says the receiving server refused it, false for a row the sweep
  closed. To the owner when stage 2b is reached: since M13 the "deferred"
  figure counts whole messages too, which "Connections turned away" would
  not name.

Found while verifying: the e2e's main positive path failed once with "553
not owned by user app1" - an smtpd idle from before the panel's reload took
the connection in the reload's second. Measured on the stand (one-off
probe, not committed): sending 0, 10, 40, 110 s after the save (two seconds
at the soonest) always met an smtpd started after the reload. The e2e now
waits for the sender map after each application it creates
(waitForSenderMap: AUTH, MAIL, RCPT, RSET - no message, no row), and the
level-1 case uses it instead of its own loop.

Docs: architecture § Queue reconcile, schema-migrations (rows 7 and 8,
chain head 8), CHANGELOG, audit plan (stage 1 note, the measured window,
a new stage 2b line: 62 lines), roadmap 15/62.

Evidence on the test server for this tree: go vet clean, go test ./... 23
packages ok, make e2e exit 0 (renewed certificate, hung panel, level 1 at
49/60).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
mix
2026-10-14 15:21:52 +03:00
co-authored by Claude Opus 5.5
parent f51f5db245
commit 60c7b85f84
26 changed files with 503 additions and 143 deletions
+29 -15
View File
@@ -417,27 +417,41 @@ guessing where it need not: it starts only after the tailer has read to
end-of-file once (on a restart the log itself holds the answer); the two-minute
grace covers messages merely in flight; a `postqueue` that cannot be read, or
whose listing is over the 4 MiB cap, leaves every row untouched rather than
closing them all or those past the cut; and the close is conditional on the
row still being `queued`, so a status the log wrote in the meantime stands.
closing them all or those past the cut; and the close, by the row's id, is
conditional on the row still being `queued`, so a status the log wrote in the
meantime stands. A delivery line updates only the newest row of its queue id
and recipient: Postfix reuses a short queue id once the message has left the
queue, and an old row must not be rewritten. `UpdateStatus`'s transaction
starts with a write, so a commit by the `dmarc-ingest` process in between
cannot fail it with `database is locked`.
Now that the log survives the container, reaching this path should mean the
lines are gone for good — rotated past fourteen files while the panel was down,
or deleted — or that the tailer was not reading.
Whether the sweep earns its place is measured, not argued (audit 2026-10-12,
H3). `delivery_log_days` (migration `0007`) keeps, per UTC day for a year:
`reconcile_closed`, the rows the sweep closed; `reconcile_contradicted`, those
of them a delivery line later spoke of - the log's status replaces the sweep's
and the reason is cleared, so a row counts once; `tailer_blind_seconds`, the
time `follow()` could not read `mail.log` (the file gone, reopening it failing;
stretches under three seconds - a rotation's tick between rename and create,
which loses nothing - are not counted); and `queued_over_lifetime`, the rows
still `queued` past the queue lifetime plus a maximal backoff, as the day's
last sweep saw them. No page shows the table: it is read off a copy of the
database. The rule, written before the data: `reconcile_closed = 0` for a month
→ remove the sweep; closed but never contradicted → keep it; contradicted →
each case explained by blind time means fix the tailer and keep the sweep,
otherwise mark such rows `lost` or remove the sweep. Decided after thirty days,
or sooner if `reconcile_contradicted` is ever non-zero.
`reconcile_closed`, the rows the sweep closed; `reconcile_missed`, those of
them whose delivery line was in `mail.log` or `mail.log.1` when the sweep
closed them - the tailer did not read a line that was there, which is what a
lost line looks like (it is never read late); `reconcile_contradicted`, those
a delivery line read later gave a status - the log's status replaces the
sweep's and the row's reason becomes `reconcile-contradicted`, so a row counts
once and can be found; `tailer_blind_seconds`, the time `follow()` could not
read `mail.log` (the file gone, reopening it failing; stretches under three
seconds - a rotation's tick between rename and create, which loses nothing -
are not counted); and `queued_over_lifetime`, the rows still `queued` past the
queue lifetime plus a maximal backoff, as the day's last sweep saw them - rows
stuck although the sweep runs (mail on hold, or a queue that could not be
listed). What removing the sweep would leave queued is closed − missed −
contradicted: the rows nothing in the log ever resolved. No page shows the
table: it is read off a copy of the database. The rule, written before the
data: `reconcile_closed = 0` for a month → remove the sweep; closed, but
neither missed nor contradicted → keep it; missed or contradicted → the tailer
loses or delays lines - fix the tailer (the two deferred steps: the rotation
window, pausing the sweep while blind) and keep the sweep; if what remains
after that is never resolved by the log, mark such rows `lost` or remove the
sweep. Decided after thirty days, or sooner if `reconcile_missed` or
`reconcile_contradicted` is ever non-zero.
**Stored lines** (`send_log_lines`, since 2.2.0;
[internal/logtail/lines.go](../internal/logtail/lines.go)). A send-log row