diff --git a/CHANGELOG.md b/CHANGELOG.md index d581777..b519e3d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,17 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); version ## [Unreleased] +### Fixed + +- **Queue ids no longer repeat**: Postfix makes long, non-repeating queue ids + (`enable_long_queue_ids=yes`). A short id - the microsecond of arrival and + the queue file's inode - came round once the inode did, and a later message + could then rewrite an older row of the Outbound log with the same id and + recipient, put its lines on that row's page, or reach the Inbound log's + row of an earlier message. New ids read like `3Pt2mN2VXxznjll`; mail + already in the queue keeps its short id; Outbound › Queue's Queue ID + column is wider. + ## [2.10.0] - 2026-10-16 Inbound › Log says what became of accepted mail: the upstream's answer, diff --git a/build/postfix-config.sh b/build/postfix-config.sh index 03104f8..def988c 100644 --- a/build/postfix-config.sh +++ b/build/postfix-config.sh @@ -440,6 +440,19 @@ postconf -e \ # panel's, were exposed to it as well. postconf -e "disable_mime_output_conversion=yes" +# Long, non-repeating queue ids (owner, 2026-10-16). The panel finds a +# message's rows, its stored lines and the inbound journal's upstream result +# by the queue id of mail.log; a short id is the microsecond of arrival and +# the queue file's inode in hex and repeats once the inode comes round, so a +# later message could meet an earlier one's rows. A long id adds the time in +# seconds and does not repeat. Postfix 3.10.13, src/global/mail_params.h:1010, +# DEF_LONG_QUEUE_IDS 0; src/global/mail_queue.h, MQID_SH_* and MQID_LG_*; +# proto/postconf.proto, enable_long_queue_ids: "Enable long, non-repeating, +# queue IDs", existing queue files keep their names, postqueue -p gets a +# wider Queue ID column. Every pattern of the panel takes both shapes +# (internal/logtail, internal/postfix/queue.go). +postconf -e "enable_long_queue_ids=yes" + # What Postfix tells a milter with MAIL FROM: its defaults plus the client's # address. The panel's milters judge every message by that address (an # application's client-address list) and record it (the inbound journal), and diff --git a/docs/architecture.md b/docs/architecture.md index 3273ec2..59f77dc 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -687,7 +687,11 @@ folding case). A delivery line names it in `orig_to=` when Postfix rewrote the address, in `to=` otherwise (`log_adhoc.c` writes `orig_to=` only then), and is matched without regard to case. It updates only the newest row of its queue id and recipient: Postfix reuses a short queue id once the message has -left the queue, and an old row must not be rewritten. `UpdateStatus`'s transaction +left the queue, and an old row must not be rewritten. Since `2.10.1` the +image makes long, non-repeating queue ids (`enable_long_queue_ids=yes`, +[build/postfix-config.sh](../build/postfix-config.sh)): the time in seconds, +then the microseconds, `z` and the inode; a queue file from before keeps its +short name, and every pattern of the panel takes both. `UpdateStatus`'s transaction starts with a write, so a commit by the `dmarc-ingest` process in between cannot fail it with `database is locked`. Now that the log survives the container, reaching this path should mean the diff --git a/docs/plans/delivery-log-storage.md b/docs/plans/delivery-log-storage.md index b49fbd2..1667d6f 100644 --- a/docs/plans/delivery-log-storage.md +++ b/docs/plans/delivery-log-storage.md @@ -68,6 +68,7 @@ to the days the row's own message could still be in the queue; this is the bound the roadmap item asked for. The search of the current file that the page did before had the same exposure for a day. `enable_long_queue_ids` would end it and is a change to the mail path, not part of this item. +*It was made in `2.10.1` (owner, 2026-10-16).* ### Retention diff --git a/docs/security.md b/docs/security.md index fc76ba8..43d639f 100644 --- a/docs/security.md +++ b/docs/security.md @@ -384,10 +384,11 @@ operator can act on it. quoted RCPT TO holding `>`, `,` or `status=` cannot end the address early and pass fields of its own for another recipient's row - and one holding an `@` is a routed address, which port 25 refuses as relaying - (`allow_untrusted_routing=no`). Short queue ids repeat (Postfix's default - `enable_long_queue_ids=no`): before writing a message's rows the milter - detaches older rows of its queue id, so a message that is gone cannot take - a new one's results. Accepted: a row whose insert waited longer than the + (`allow_untrusted_routing=no`). Queue ids do not repeat since `2.10.1` + (`enable_long_queue_ids=yes`, below); for a queue file from before, which + keeps its short name, the milter detaches older rows of a queue id before + writing a message's rows, so a message that is gone cannot take a new + one's results. Accepted: a row whose insert waited longer than the milter's 30 s timeout - the database held - can miss its first delivery line and stay `queued`. - **The inbound journal is written from what strangers send.** The inbound @@ -765,8 +766,13 @@ deferred item from the roadmap. `deploy/docker-compose.yml` sets no memory limit: what is left - many messages at once, each held to the bounds above - is the host's to size. -- **A message's page can show another message's log lines after a queue-id - collision.** Postfix's short queue ids repeat: the id is the microsecond of +- **A message's page could show another message's log lines after a + queue-id collision - until `2.10.1`.** Since then the image makes long, + non-repeating queue ids (`enable_long_queue_ids=yes`, + `build/postfix-config.sh`; owner, 2026-10-16), which ends it, and with it a + delivery line of a later message rewriting an older send-log row of the + same id and recipient. What follows is the exposure of rows from before, + whose ids are short. Postfix's short queue ids repeat: the id is the microsecond of arrival and the queue file's inode, and inodes come round within seconds. The page takes the stored lines of the row's queue id dated within the row's own queue lifetime (about six days), so a second message has to get @@ -774,9 +780,9 @@ deferred item from the roadmap. million per reuse of the inode, and not something a sender can aim. When it happens, a `domain` user could read the log lines of a message that is not theirs. The search of the current `mail.log` that the page did before - 2.2.0 had the same exposure over a day. Trigger to revisit: a report of - foreign lines on a page, or a reason to turn on `enable_long_queue_ids`, - which ends it. + 2.2.0 had the same exposure over a day. It lasts for a short-id row only + as long as the row's queue lifetime, so it is gone from the server some + six days after `2.10.1`. - **A `domain` user may point an inbound domain's upstream at any host and port.** The upstream form checks that the host is a well-formed name or IP and the port is 1–65535; loopback, private and link-local addresses are diff --git a/internal/store/inboundspam.go b/internal/store/inboundspam.go index 77eddd1..274f987 100644 --- a/internal/store/inboundspam.go +++ b/internal/store/inboundspam.go @@ -135,11 +135,12 @@ const ( // ForgetInboundQueueID detaches the journal rows of a queue id from it: those // still open lose their result too, as nothing will ever report it. The -// milter calls it before it writes a new message's rows - Postfix reuses -// queue ids (enable_long_queue_ids=no, the default: microseconds and the -// queue file's inode, src/global/mail_queue.h), and only one live queue file -// has an id, so older rows of it are of a message that is gone and must not -// take the new one's results. The tailer calls it when cleanup removes a +// milter calls it before it writes a new message's rows - a short queue id +// repeats (microseconds and the queue file's inode, src/global/mail_queue.h; +// the image has made long, non-repeating ones since 2.10.1, but a queue file +// from before keeps its short name), and only one live queue file has an id, +// so older rows of it are of a message that is gone and must not take the +// new one's results. The tailer calls it when cleanup removes a // queue file it did not queue ("removed (canceled)"): the milter had written // the rows, but the message was never queued. func (s *Store) ForgetInboundQueueID(queueID string) error { diff --git a/test/e2e/delivery_log_test.go b/test/e2e/delivery_log_test.go index 3d140f9..72d1e4f 100644 --- a/test/e2e/delivery_log_test.go +++ b/test/e2e/delivery_log_test.go @@ -126,6 +126,9 @@ print("" if r is None else r[0]+" "+r[1])`, token) } if f := strings.Fields(out); len(f) == 2 && f[1] == "deferred" { queueID = f[0] + if !longQueueID.MatchString(queueID) { + return false, fmt.Errorf("queue id %q is not a long one (enable_long_queue_ids)", queueID) + } return true, nil } return false, fmt.Errorf("row %q", out) diff --git a/test/e2e/inbound_upstream_test.go b/test/e2e/inbound_upstream_test.go index 7a937f6..84d4799 100644 --- a/test/e2e/inbound_upstream_test.go +++ b/test/e2e/inbound_upstream_test.go @@ -9,6 +9,12 @@ import ( "time" ) +// longQueueID is the shape of a long queue id: six or more characters of the +// time in seconds and four of the microseconds, base 52 without vowels, "z", +// the inode in base 51, which has no "z" (Postfix 3.10.13, +// src/global/mail_queue.h, MQID_LG_*). +var longQueueID = regexp.MustCompile(`^[0-9B-DF-HJ-NP-TV-Zb-df-hj-np-tv-z]{10,}z[0-9B-DF-HJ-NP-TV-Zb-df-hj-np-tv-y]+$`) + // testInboundUpstreamDown: an inbound domain whose upstream refuses // connections (the sink's address, a port nothing listens on). Port 25 takes // the message; Postfix cannot hand it on and logs "relay=none ... @@ -65,6 +71,11 @@ print("" if r is None else "|".join(r))`, token) return false, fmt.Errorf("deferred with host %q, reply %q", f[2], f[3]) } queueID = f[0] + // A long queue id (enable_long_queue_ids=yes, + // build/postfix-config.sh): the time, "z", the inode. + if !longQueueID.MatchString(queueID) { + return false, fmt.Errorf("queue id %q is not a long one", queueID) + } return true, nil } return false, fmt.Errorf("row %q\n%s", out, mailLogTail(h, "postfix/"))