e166a2ebd60771e98ea9908611c48fad8456fd71
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  mixandClaude Opus 5.5
|
e166a2ebd6 |
refused messages: the page shows what Postfix wrote (stage 2b, 0009)
The Delivery log box of every refused row said "This message never reached
the queue, so Postfix wrote no delivery lines for it" - untrue: Postfix
wrote a line for each. The owner, 2026-10-13: the box is required on every
message page. Pictured for him first (dev/shots, the refused variant of
out-message: wrong From with its queue id lines, rate-limited with the
NOQUEUE line); the variant has no mockup screen of its own - the guard ties
screens to pages, and its outline is out-message's.
- Wrong From: cleanup gives the message a queue id before the headers are
judged; the sender check reads {i} at end-of-headers and writes it on the
refused row. Its lines - stored under the id like any other's - are on the
page, but only those from 10 minutes before to 1 minute after the refusal
(store.SendLogLines for a rejected row): Postfix frees the id at once, and
a later message with it lends nothing. No mail.log fallback for a refused
row, for the same reason.
- Rate-limited and wrong client: refused at MAIL FROM, no queue id. Migration
0009 adds send_log.refusal_line; logtail.refusalLines matches smtpd's
"NOQUEUE: milter-reject: MAIL from ...; from=<...>" (not port 25's, not
CONNECT/EHLO) and store.AttachRefusalLine puts it on the newest refused row
of that sender (and login, when the line names it) within two minutes; a
line whose row is not written yet waits up to 30 s (retried at each pass).
- The page: the queue id fact says "given, the message was not queued" or
"none - refused at MAIL FROM, before one is given"; the box's caption is
"The line from mail.log for this refusal" for the latter; a row from
before says the line was not kept. LogLineLevel colours a refusal by its
reply: 5xx red, 4xx amber (IPv6 clients included). The id in the fact is
not muted as in the picture: the kit has no muted inline.
Tests: store (attach by sender, login, time; not a queued row, not one an
hour off; a refused row's lines are its own minutes); logtail (attach, wait
and retry, port 25 and other lines ignored); milter (the refused row carries
{i}); view (facts, caption, line levels); handlers (the line on the page,
the sentence without it). e2e refusals_show_postfixs_line: a client-address
refusal made there, and the pages of a wrong From, a rate-limited and a
wrong client refusal each show Postfix's line.
schema-migrations row 9, CHANGELOG [Unreleased], audit plan line ticked,
roadmap 25/62.
Evidence on the test server for this tree: make e2e exit 0 (the new case
in 5.4 s); go test ./... ok but for two schema tests, fixed after the run
(the schema at 0009) - store tests ok locally; the full run follows before
the push.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
mixandClaude Opus 5.5
|
60c7b85f84 |
findings of the reviews of audit stage 1
test / test (push) Canceled after 0s
Technical review (Opus) and security review (Fable) of 9efc9c7..a59e3d4; reports in the session's scratchpad. Fixed: - B1 (verified bug): store.UpdateStatus read before writing in its transaction; a commit by another connection in between (the dmarc-ingest process opens the same file) failed it at once with "database is locked" and the delivery line was lost. The first statement is now a write. - R3: a delivery line updated every row of its queue id and recipient; it now updates only the newest (Postfix reuses short ids). The sweep closes by row id (QueuedDelivery.ID, CloseUnreported(id)). - R4: a contradicted sweep keeps a mark, reason "reconcile-contradicted", instead of losing which row it was. - R1: contradicted cannot see a lost line (never read late). New counter reconcile_missed: at close, the sweep looks for the row's delivery line in mail.log and mail.log.1 (QueueLines). 0007 edited - it has run on no database but throwaway stands. - R2: queued_over_lifetime cannot measure "what removing the sweep would leave" while the sweep runs; its text now says rows stuck despite the sweep, and the rule uses closed - missed - contradicted. - cert-reload (R5, R6, R7, security medium/low): both files copied to .new and the copies checked as a pair before both are renamed; acts on the copy only when postconf says Postfix reads it; a failed reload is retried next pass; umask 077; says so when the mounted files cannot be read. The e2e case asserts Postfix reads the copy, the copy equals the mount, the key is 600 root. - Security low: a recipient with a quoted local part - the journal cut "a b"@x at the space, deliveryRe cut "a>b"@x at the ">" - matched no delivery line and was closed by the sweep. cleanAddress ends a bracketed address at the first ">" outside quotes; deliveryRe takes the recipient up to the ">, " before relay= (test fixtures given the relay= every real delivery line has). - Security low: postqueue -p's error carries at most 512 bytes of output. - Nits: addDeliveryLogDay refuses a column not on its list; the truncation test checks its message. - To stage 2b (page wording, mockup-first): the history step of a bounced row says the receiving server refused it, false for a row the sweep closed. To the owner when stage 2b is reached: since M13 the "deferred" figure counts whole messages too, which "Connections turned away" would not name. Found while verifying: the e2e's main positive path failed once with "553 not owned by user app1" - an smtpd idle from before the panel's reload took the connection in the reload's second. Measured on the stand (one-off probe, not committed): sending 0, 10, 40, 110 s after the save (two seconds at the soonest) always met an smtpd started after the reload. The e2e now waits for the sender map after each application it creates (waitForSenderMap: AUTH, MAIL, RCPT, RSET - no message, no row), and the level-1 case uses it instead of its own loop. Docs: architecture § Queue reconcile, schema-migrations (rows 7 and 8, chain head 8), CHANGELOG, audit plan (stage 1 note, the measured window, a new stage 2b line: 62 lines), roadmap 15/62. Evidence on the test server for this tree: go vet clean, go test ./... 23 packages ok, make e2e exit 0 (renewed certificate, hung panel, level 1 at 49/60). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
ffefa9d6df |
delivery log: measure the sweep before deciding its fate (audit H3, 0007)
Migration 0007 (DDL only): delivery_log_days, a row per UTC day, kept a year, shown nowhere - read off a copy of the database after a month by the decision rule written in the audit plan. - reconcile_closed: the sweep adds the rows it actually closed. - reconcile_contradicted: store.UpdateStatus, in one transaction, counts the rows with reason "reconcile" a delivery line now speaks of, writes the log's status and clears the reason - so a re-read line counts once. - tailer_blind_seconds: follow() tracks stretches in which the file is gone or reopening fails (logtail.blindness); stored when a stretch ends and every minute while it lasts; under 3 s not counted (the rotation's tick between rename and create loses nothing). - queued_over_lifetime: every sweep overwrites the day's level with the rows queued past LongestLifetime + MaximalBackoff (a message Postfix still works on stays queued until its first retry after the lifetime). - retentionLoop prunes days older than a year. Tests: store (sums per day, the level overwritten, under a second nothing, prune; a contradicted sweep counted once and its reason cleared; count of queued rows); logtail (the sweep counts what it closed; countStuck records the level; blindness - a one-tick gap not counted, 75 s stored as 60 + 15 from where the last part ended). Schema tests at user_version 7. schema-migrations.md: chain head 7 (it still said 5 since 2.4.0), row 7. CHANGELOG [Unreleased], audit plan line ticked, roadmap 10/61. Evidence: go vet ./... clean; go test ./internal/store ok, ./internal/logtail ok but for TestFollowTailsAndRotates (Windows); the server run of the stage follows before the push. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
9adb44845c |
logtail: the sweep closes only queued rows, with reason "reconcile" (audit H3)
reconcile() closed a row through UpdateStatus (WHERE queue_id AND to_addr), which could overwrite a "sent" the tailer wrote between the listing and the close, and rewrite an old row of a reused queue id. store.CloseUnreported sets status bounced and reason "reconcile" (store.ReasonReconcile; the 0006 column, no schema change) WHERE ... AND status = 'queued'; the sweep counts rows actually closed. Tests: store - a queued row is closed with the reason, a sent row is left alone (0 rows); logtail - the sweep's close goes through CloseUnreported. CHANGELOG [Unreleased], audit plan line ticked, roadmap 9/61. Evidence: go vet ./... clean; go test ./internal/store ok, ./internal/logtail ok but for TestFollowTailsAndRotates (Windows rename of an open file, fails here always); the server run of the stage follows before the push. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
a990e7cc23 |
delivery-log-storage: findings of the technical and security reviews
test / test (push) Canceled after 0s
Neither review (read-only agents, Opus and Fable, over ae06d8b..815236a) has a Critical or High finding. What they found and what was done: - A stranger on port 25 could grow the table: every queue-id line of inbound mail was stored and pruned an hour or more later, his own message-id among them. The tailer now leaves out the lines of a queue id that postfix/inbound/smtpd opened, until qmgr removes the message or another listener opens one under the id; 4096 such ids are remembered. - A commit, so a disk sync, per stored line on the connection the journal milter shares, and a backlog written line by line. Lines are now written in one transaction per pass over the log, 500 at most, before the read offset is saved (InsertSendLogLines; follow() calls the keeper's flush). - The prune asked the send log once per line and ran hourly. It asks once per queue id and runs at start and with the six-hourly sweep; with port-25 mail left out there is little without a row to wait for. - The 4096-byte cut could split a character; it keeps characters whole. - security.md did not say that a backup now carries these lines, and the plan said the time window excludes a reused queue id where it only limits the chance to the row's queue lifetime. Both are corrected; the collision is recorded as an accepted risk with its trigger. - Tests that did not prove their comments: the prune is now tested with the sweep, a rejected row next to the prune, batching and the change from catching up to following, port-25 mail left out and its id reused. Left as found and written down in the plan: lines the tailer never reads (a file rotated away while the panel was down), and a message in the queue across the upgrade, which shows only its later lines. On the test server, for this tree: go vet clean, go test ./... 22 packages ok, make e2e ok. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
81df131217 |
delivery-log-storage: the tailer keeps the lines of queued messages
Every mail.log line headed by a queue id is written to send_log_lines as the tailer reads it, dated by the line's own stamp. The id is taken from the head of the line only - the tail is the far end's text - and has to look like one: "NOQUEUE:" and "warning:" stand in the same place. The tailer does not check for a send-log row first: a message's first lines come before the journal milter writes it. Lines that never get a row are removed by the prune, which deletes lines older than an hour whose queue id no row carries. It runs at start, after each send-log sweep and hourly. While the tailer catches up at start a line is written only if it is not there already, as the inbound journal does: its place is saved every few seconds, so an unclean stop means reading some lines twice. Checked on Windows: go vet clean; go test ./internal/logtail passes except TestFollowTailsAndRotates, which fails there on renaming an open file and always did. New tests: which line heads are queue ids, a line dated by its stamp, no second copy after a rewound offset, the prune cadence and cutoff. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mix
|
dca7d99c1b |
release: 1.5.0: Send-log retention configurable in panel Settings
Send-log retention is configurable from panel **Settings** (global administrator); `SEND_LOG_RETENTION_DAYS` seeds the initial default only. Added: - Panel **Settings** (global administrator): **Send log retention (days)** — Changed: - Deliveries list and delivery detail pages show the configured retention Full detail: CHANGELOG.md (1.5.0). |