mixandClaude Opus 5.5
a990e7cc23
Neither review (read-only agents, Opus and Fable, over ae06d8b..815236a) has a Critical or High finding. What they found and what was done: - A stranger on port 25 could grow the table: every queue-id line of inbound mail was stored and pruned an hour or more later, his own message-id among them. The tailer now leaves out the lines of a queue id that postfix/inbound/smtpd opened, until qmgr removes the message or another listener opens one under the id; 4096 such ids are remembered. - A commit, so a disk sync, per stored line on the connection the journal milter shares, and a backlog written line by line. Lines are now written in one transaction per pass over the log, 500 at most, before the read offset is saved (InsertSendLogLines; follow() calls the keeper's flush). - The prune asked the send log once per line and ran hourly. It asks once per queue id and runs at start and with the six-hourly sweep; with port-25 mail left out there is little without a row to wait for. - The 4096-byte cut could split a character; it keeps characters whole. - security.md did not say that a backup now carries these lines, and the plan said the time window excludes a reused queue id where it only limits the chance to the row's queue lifetime. Both are corrected; the collision is recorded as an accepted risk with its trigger. - Tests that did not prove their comments: the prune is now tested with the sweep, a rejected row next to the prune, batching and the change from catching up to following, port-25 mail left out and its id reused. Left as found and written down in the plan: lines the tailer never reads (a file rotated away while the panel was down), and a message in the queue across the upgrade, which shows only its later lines. On the test server, for this tree: go vet clean, go test ./... 22 packages ok, make e2e ok. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
9.3 KiB
Plan: delivery-log-storage
Status: agreed 2026-10-12 (owner: option 1 of the roadmap item, "store the lines")
Date: 2026-10-12
Version: 2.2.0 MINOR (one DDL migration; a 2.1.0 data directory boots unchanged).
Goal
A message's page (/outbound/log/{id}, mockup out-message) shows what
Postfix wrote about the message for as long as its row is in the send log.
Today the Delivery log box is filled by a search of the current mail.log
on every page view, so it is empty for anything sent before the last daily
rotation, while the row itself lives for ninety days.
Context (as-built)
- The log tailer (internal/logtail)
already reads every
mail.logline once, from a persisted offset, to advance delivery statuses. logtail.QueueLinessearches the last 4 MB of the current file for one queue id;Handlers.deliveryLogcalls it per page view.- A message with several recipients is several send-log rows under one queue id; the page of each shows every line of the message, as the search does today.
Decisions
What is stored
A mail.log line whose head is a queue id - after the timestamp, the host
name and program[pid]: - goes into a new table with that id, the line's own
time (lineTime, the same reading the inbound journal uses) and the line as
written, cut to 4096 bytes on a character boundary. The head is matched from
the start of the line: the tail is the far end's words. NOQUEUE and
warning are not queue ids - the pattern takes Postfix's short form
(hexadecimal) and its long form (no vowels) and nothing else.
The tailer does not ask whether the id belongs to a send-log row before it
writes. The first lines of a message (client=, message-id=) are logged
before the journal milter writes the row at end-of-message, so such a check
would drop exactly them.
Mail that came in on port 25 is left out (security review, 2026-10-12). A
queue id that postfix/inbound/smtpd opened is a stranger's message; it never
gets a send-log row, and what a stranger can make Postfix log - his own
message-id, up to the cut - is not stored to be pruned later. The tailer
remembers such ids (4096 at most) until qmgr's removed or until another
listener opens a message under the id. What is still stored without a row is
little: the bounces Postfix generates, and port-25 mail met in mid-flight
after a restart.
Lines are written in one transaction per pass over the log, 500 at most (technical review): a commit per line is a disk sync per line on the one connection the journal milter writes through.
Which lines are a row's
A line is shown on a row's page when the queue id is the same and the
line's time lies between one hour before the row was written and the queue
lifetime (the larger of maximal_queue_lifetime and bounce_queue_lifetime,
as postconf reports them) plus a day after it. Postfix reuses a short queue
id once the message has left the queue - the id is the microsecond of arrival
and the inode of the queue file - so a row that lives ninety days can meet a
later message under its id. The window does not rule that out, it limits it
to the days the row's own message could still be in the queue; this is the
bound the roadmap item asked for. The search of the current file that the
page did before had the same exposure for a day. enable_long_queue_ids
would end it and is a change to the mail path, not part of this item.
Retention
Lines live as long as a send-log row with their queue id does. One rule covers both ends: a line older than an hour whose queue id has no send-log row is deleted. That removes the lines of a row the retention sweep has just deleted, and the lines of messages that never had a row. It runs at start and after every send-log sweep, six-hourly. It asks the send log once per queue id; at a million lines that is still a scan of the table, of the order of a second, during which the one connection is taken.
There is no cap of its own: with port-25 mail left out, the table is bounded by what authenticated applications send, which level 1 and level 2 limit.
After an unclean stop
The tailer saves its place every few seconds, so after a crash it reads some lines twice. While it is catching up at start a line is written only if the same line is not there already - the rule the inbound journal follows.
What is not kept
- Lines the tailer never reads are not stored: the tail of a file that was rotated away while the panel was down, as with delivery statuses (architecture.md § Log tailer).
- A message in the queue across the upgrade to 2.2.0 has only its later lines stored, and the page then shows those alone: the search of the file runs only for a row with nothing stored. The only installation is the owner's test server; no merge of the two sources is built for one moment of it.
The page
The page reads the table: the latest 200 lines of the row, oldest first. When
the table has none, it still searches the current mail.log as before - that
is what shows a message accepted before 2.2.0 while its lines are in today's
file. Lines in rotated files are not imported: the only installation is the
owner's test server.
Nothing in the page's layout changes; the out-message mockup already draws
the box as a log_pane. Two texts change, mockup first: the Outbound log
help topic (it says an old message may show no lines) and the note shown when
there are none.
Size
Six to ten lines of about 200 bytes per message: 1-2 KB next to a row of a few
hundred bytes. The table becomes the larger part of what the send log costs in
/data and in a backup; the retention setting bounds both together.
Schema (migration 0003_send_log_lines.sql)
CREATE TABLE send_log_lines (
id INTEGER PRIMARY KEY AUTOINCREMENT,
queue_id TEXT NOT NULL,
logged_at TEXT NOT NULL,
line TEXT NOT NULL
);
CREATE INDEX idx_send_log_lines_queue_id ON send_log_lines (queue_id, logged_at);
No foreign key: a line is written before its row exists, and one queue id is several rows.
Scope
In: the table, the tailer writing and pruning it, the page reading it, the two texts, docs.
Out: the inbound journal (it decides for itself, roadmap § delivery-log-storage);
importing rotated files; enable_long_queue_ids; any change to mail.log
rotation or to the Server › Mail log page.
Security
- The lines carry the far end's text (replies, HELO names). They are shown today from the file through the same escaping; storing them changes where they are read from, not what is shown or to whom. A domain administrator sees the lines of messages of their own domains, as now.
- They now enter the database and so a full backup, which
log/does not: submitting clients' addresses and SASL logins, message ids, receivers' replies. security.md says so. - A line is matched from its head, so a reply cannot make itself a line of
another message; Postfix does not log raw line breaks, and
postlogdis the file's only writer. - A stranger on port 25 does not write to the table (above).
Done when
A message sent before the last rotation shows its lines on its page; the lines disappear with the row; the page's wording no longer blames rotation for an ordinary case.
Model routing (this plan)
| Role | Model |
|---|---|
| All implementation (code, tests, docs) | Opus |
| Technical review (reviewer ≠ author: a separate read-only agent) | Opus |
| Security review | Fable |
Implementation checklist
Target version cut: 2.2.0 (MINOR). One commit per step;
development.md § Plan checklists.
- Migration
0003_send_log_lines.sql, store methods (insert, insert once, lines of a row, prune) and their tests — Opus - Log tailer: store queue-id lines, write once while catching up, prune with the send-log sweep — Opus (hourly as first built; six-hourly after the reviews)
- The two texts, mockup first: the Outbound log help topic and the empty note (
design:commit) — Opus (the help topic is in the mockups; the empty note is not drawn there - theout-messagemockup shows a message with lines - and changes with the page) - Message page reads the table, falls back to the current
mail.log; help text inhelp.go— Opus - architecture.md, guide.md, schema-migrations.md — Opus (schema-migrations.md went in with the migration)
go vet,go test, e2e: a message's page still lists its lines after a forced rotation — Opus (on the test server: vet ok, 22 packages ok, e2e ok with the new stepdelivery_log_survives_rotation)- Technical review — Opus (2026-10-12; no Critical or High. Fixed: one transaction per pass instead of a commit per line, the prune asks the send log once per queue id and runs six-hourly, the cut keeps characters whole, tests for what the comments claimed. Left as found and written down under What is not kept: lines the tailer never read, and a message in the queue across the upgrade)
- Security review — Fable (2026-10-12; no Critical or High. Fixed: port-25 mail is not stored, so a stranger cannot grow the table; security.md states what a backup now carries. The window against queue-id reuse is described as the limit it is)