postfix: long, non-repeating queue ids
The owner, 2026-10-16, of three ways to stop a reused queue id reaching an older row (found by the technical review of inbound-upstream-result, L6): "1" - enable_long_queue_ids=yes. A short id is the microsecond of arrival and the queue file's inode in hex (Postfix 3.10.13, src/global/mail_queue.h, MQID_SH_*) and repeats once the inode comes round; a later message with no send-log row of its own (port-25 mail, a bounce Postfix sends) could then rewrite an older Outbound log row of the same id and recipient, its page could show the later message's stored lines (security.md's known exposure, "enable_long_queue_ids ... ends it"), and the inbound journal's rows needed detaching. A long id adds the time in seconds and does not repeat (MQID_LG_*; proto/postconf.proto: existing queue files keep their names, postqueue -p gets a wider Queue ID column). Every pattern of the panel takes both shapes (queueLineRe's long branch, the [0-9A-Za-z]+ of the others, queueEntryRe) - checked one by one. Tests: e2e reads a long id on an outbound send-log row (testExpiredMessageIsClosed) and on an inbound journal row (testInboundUpstreamDown). go vet + go test (26 packages) and make e2e on the test server for this tree. Docs: security.md (the collision bullet closed for new mail, the upstream bullet), architecture § Log tailer, delivery-log-storage's note, the ForgetInboundQueueID comment; CHANGELOG [Unreleased] § Fixed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -5,6 +5,17 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); version
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **Queue ids no longer repeat**: Postfix makes long, non-repeating queue ids
|
||||||
|
(`enable_long_queue_ids=yes`). A short id - the microsecond of arrival and
|
||||||
|
the queue file's inode - came round once the inode did, and a later message
|
||||||
|
could then rewrite an older row of the Outbound log with the same id and
|
||||||
|
recipient, put its lines on that row's page, or reach the Inbound log's
|
||||||
|
row of an earlier message. New ids read like `3Pt2mN2VXxznjll`; mail
|
||||||
|
already in the queue keeps its short id; Outbound › Queue's Queue ID
|
||||||
|
column is wider.
|
||||||
|
|
||||||
## [2.10.0] - 2026-10-16
|
## [2.10.0] - 2026-10-16
|
||||||
|
|
||||||
Inbound › Log says what became of accepted mail: the upstream's answer,
|
Inbound › Log says what became of accepted mail: the upstream's answer,
|
||||||
|
|||||||
@@ -440,6 +440,19 @@ postconf -e \
|
|||||||
# panel's, were exposed to it as well.
|
# panel's, were exposed to it as well.
|
||||||
postconf -e "disable_mime_output_conversion=yes"
|
postconf -e "disable_mime_output_conversion=yes"
|
||||||
|
|
||||||
|
# Long, non-repeating queue ids (owner, 2026-10-16). The panel finds a
|
||||||
|
# message's rows, its stored lines and the inbound journal's upstream result
|
||||||
|
# by the queue id of mail.log; a short id is the microsecond of arrival and
|
||||||
|
# the queue file's inode in hex and repeats once the inode comes round, so a
|
||||||
|
# later message could meet an earlier one's rows. A long id adds the time in
|
||||||
|
# seconds and does not repeat. Postfix 3.10.13, src/global/mail_params.h:1010,
|
||||||
|
# DEF_LONG_QUEUE_IDS 0; src/global/mail_queue.h, MQID_SH_* and MQID_LG_*;
|
||||||
|
# proto/postconf.proto, enable_long_queue_ids: "Enable long, non-repeating,
|
||||||
|
# queue IDs", existing queue files keep their names, postqueue -p gets a
|
||||||
|
# wider Queue ID column. Every pattern of the panel takes both shapes
|
||||||
|
# (internal/logtail, internal/postfix/queue.go).
|
||||||
|
postconf -e "enable_long_queue_ids=yes"
|
||||||
|
|
||||||
# What Postfix tells a milter with MAIL FROM: its defaults plus the client's
|
# What Postfix tells a milter with MAIL FROM: its defaults plus the client's
|
||||||
# address. The panel's milters judge every message by that address (an
|
# address. The panel's milters judge every message by that address (an
|
||||||
# application's client-address list) and record it (the inbound journal), and
|
# application's client-address list) and record it (the inbound journal), and
|
||||||
|
|||||||
@@ -687,7 +687,11 @@ folding case). A delivery line names it in `orig_to=` when Postfix rewrote
|
|||||||
the address, in `to=` otherwise (`log_adhoc.c` writes `orig_to=` only then),
|
the address, in `to=` otherwise (`log_adhoc.c` writes `orig_to=` only then),
|
||||||
and is matched without regard to case. It updates only the newest row of its
|
and is matched without regard to case. It updates only the newest row of its
|
||||||
queue id and recipient: Postfix reuses a short queue id once the message has
|
queue id and recipient: Postfix reuses a short queue id once the message has
|
||||||
left the queue, and an old row must not be rewritten. `UpdateStatus`'s transaction
|
left the queue, and an old row must not be rewritten. Since `2.10.1` the
|
||||||
|
image makes long, non-repeating queue ids (`enable_long_queue_ids=yes`,
|
||||||
|
[build/postfix-config.sh](../build/postfix-config.sh)): the time in seconds,
|
||||||
|
then the microseconds, `z` and the inode; a queue file from before keeps its
|
||||||
|
short name, and every pattern of the panel takes both. `UpdateStatus`'s transaction
|
||||||
starts with a write, so a commit by the `dmarc-ingest` process in between
|
starts with a write, so a commit by the `dmarc-ingest` process in between
|
||||||
cannot fail it with `database is locked`.
|
cannot fail it with `database is locked`.
|
||||||
Now that the log survives the container, reaching this path should mean the
|
Now that the log survives the container, reaching this path should mean the
|
||||||
|
|||||||
@@ -68,6 +68,7 @@ to the days the row's own message could still be in the queue; this is the
|
|||||||
bound the roadmap item asked for. The search of the current file that the
|
bound the roadmap item asked for. The search of the current file that the
|
||||||
page did before had the same exposure for a day. `enable_long_queue_ids`
|
page did before had the same exposure for a day. `enable_long_queue_ids`
|
||||||
would end it and is a change to the mail path, not part of this item.
|
would end it and is a change to the mail path, not part of this item.
|
||||||
|
*It was made in `2.10.1` (owner, 2026-10-16).*
|
||||||
|
|
||||||
### Retention
|
### Retention
|
||||||
|
|
||||||
|
|||||||
+15
-9
@@ -384,10 +384,11 @@ operator can act on it.
|
|||||||
quoted RCPT TO holding `>`, `,` or `status=` cannot end the address early
|
quoted RCPT TO holding `>`, `,` or `status=` cannot end the address early
|
||||||
and pass fields of its own for another recipient's row - and one holding an
|
and pass fields of its own for another recipient's row - and one holding an
|
||||||
`@` is a routed address, which port 25 refuses as relaying
|
`@` is a routed address, which port 25 refuses as relaying
|
||||||
(`allow_untrusted_routing=no`). Short queue ids repeat (Postfix's default
|
(`allow_untrusted_routing=no`). Queue ids do not repeat since `2.10.1`
|
||||||
`enable_long_queue_ids=no`): before writing a message's rows the milter
|
(`enable_long_queue_ids=yes`, below); for a queue file from before, which
|
||||||
detaches older rows of its queue id, so a message that is gone cannot take
|
keeps its short name, the milter detaches older rows of a queue id before
|
||||||
a new one's results. Accepted: a row whose insert waited longer than the
|
writing a message's rows, so a message that is gone cannot take a new
|
||||||
|
one's results. Accepted: a row whose insert waited longer than the
|
||||||
milter's 30 s timeout - the database held - can miss its first delivery
|
milter's 30 s timeout - the database held - can miss its first delivery
|
||||||
line and stay `queued`.
|
line and stay `queued`.
|
||||||
- **The inbound journal is written from what strangers send.** The inbound
|
- **The inbound journal is written from what strangers send.** The inbound
|
||||||
@@ -765,8 +766,13 @@ deferred item from the roadmap.
|
|||||||
`deploy/docker-compose.yml` sets no memory limit: what is left - many
|
`deploy/docker-compose.yml` sets no memory limit: what is left - many
|
||||||
messages at once, each held to the bounds above - is the host's to
|
messages at once, each held to the bounds above - is the host's to
|
||||||
size.
|
size.
|
||||||
- **A message's page can show another message's log lines after a queue-id
|
- **A message's page could show another message's log lines after a
|
||||||
collision.** Postfix's short queue ids repeat: the id is the microsecond of
|
queue-id collision - until `2.10.1`.** Since then the image makes long,
|
||||||
|
non-repeating queue ids (`enable_long_queue_ids=yes`,
|
||||||
|
`build/postfix-config.sh`; owner, 2026-10-16), which ends it, and with it a
|
||||||
|
delivery line of a later message rewriting an older send-log row of the
|
||||||
|
same id and recipient. What follows is the exposure of rows from before,
|
||||||
|
whose ids are short. Postfix's short queue ids repeat: the id is the microsecond of
|
||||||
arrival and the queue file's inode, and inodes come round within seconds.
|
arrival and the queue file's inode, and inodes come round within seconds.
|
||||||
The page takes the stored lines of the row's queue id dated within the
|
The page takes the stored lines of the row's queue id dated within the
|
||||||
row's own queue lifetime (about six days), so a second message has to get
|
row's own queue lifetime (about six days), so a second message has to get
|
||||||
@@ -774,9 +780,9 @@ deferred item from the roadmap.
|
|||||||
million per reuse of the inode, and not something a sender can aim. When it
|
million per reuse of the inode, and not something a sender can aim. When it
|
||||||
happens, a `domain` user could read the log lines of a message that is not
|
happens, a `domain` user could read the log lines of a message that is not
|
||||||
theirs. The search of the current `mail.log` that the page did before
|
theirs. The search of the current `mail.log` that the page did before
|
||||||
2.2.0 had the same exposure over a day. Trigger to revisit: a report of
|
2.2.0 had the same exposure over a day. It lasts for a short-id row only
|
||||||
foreign lines on a page, or a reason to turn on `enable_long_queue_ids`,
|
as long as the row's queue lifetime, so it is gone from the server some
|
||||||
which ends it.
|
six days after `2.10.1`.
|
||||||
- **A `domain` user may point an inbound domain's upstream at any host and
|
- **A `domain` user may point an inbound domain's upstream at any host and
|
||||||
port.** The upstream form checks that the host is a well-formed name or IP
|
port.** The upstream form checks that the host is a well-formed name or IP
|
||||||
and the port is 1–65535; loopback, private and link-local addresses are
|
and the port is 1–65535; loopback, private and link-local addresses are
|
||||||
|
|||||||
@@ -135,11 +135,12 @@ const (
|
|||||||
|
|
||||||
// ForgetInboundQueueID detaches the journal rows of a queue id from it: those
|
// ForgetInboundQueueID detaches the journal rows of a queue id from it: those
|
||||||
// still open lose their result too, as nothing will ever report it. The
|
// still open lose their result too, as nothing will ever report it. The
|
||||||
// milter calls it before it writes a new message's rows - Postfix reuses
|
// milter calls it before it writes a new message's rows - a short queue id
|
||||||
// queue ids (enable_long_queue_ids=no, the default: microseconds and the
|
// repeats (microseconds and the queue file's inode, src/global/mail_queue.h;
|
||||||
// queue file's inode, src/global/mail_queue.h), and only one live queue file
|
// the image has made long, non-repeating ones since 2.10.1, but a queue file
|
||||||
// has an id, so older rows of it are of a message that is gone and must not
|
// from before keeps its short name), and only one live queue file has an id,
|
||||||
// take the new one's results. The tailer calls it when cleanup removes a
|
// so older rows of it are of a message that is gone and must not take the
|
||||||
|
// new one's results. The tailer calls it when cleanup removes a
|
||||||
// queue file it did not queue ("removed (canceled)"): the milter had written
|
// queue file it did not queue ("removed (canceled)"): the milter had written
|
||||||
// the rows, but the message was never queued.
|
// the rows, but the message was never queued.
|
||||||
func (s *Store) ForgetInboundQueueID(queueID string) error {
|
func (s *Store) ForgetInboundQueueID(queueID string) error {
|
||||||
|
|||||||
@@ -126,6 +126,9 @@ print("" if r is None else r[0]+" "+r[1])`, token)
|
|||||||
}
|
}
|
||||||
if f := strings.Fields(out); len(f) == 2 && f[1] == "deferred" {
|
if f := strings.Fields(out); len(f) == 2 && f[1] == "deferred" {
|
||||||
queueID = f[0]
|
queueID = f[0]
|
||||||
|
if !longQueueID.MatchString(queueID) {
|
||||||
|
return false, fmt.Errorf("queue id %q is not a long one (enable_long_queue_ids)", queueID)
|
||||||
|
}
|
||||||
return true, nil
|
return true, nil
|
||||||
}
|
}
|
||||||
return false, fmt.Errorf("row %q", out)
|
return false, fmt.Errorf("row %q", out)
|
||||||
|
|||||||
@@ -9,6 +9,12 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// longQueueID is the shape of a long queue id: six or more characters of the
|
||||||
|
// time in seconds and four of the microseconds, base 52 without vowels, "z",
|
||||||
|
// the inode in base 51, which has no "z" (Postfix 3.10.13,
|
||||||
|
// src/global/mail_queue.h, MQID_LG_*).
|
||||||
|
var longQueueID = regexp.MustCompile(`^[0-9B-DF-HJ-NP-TV-Zb-df-hj-np-tv-z]{10,}z[0-9B-DF-HJ-NP-TV-Zb-df-hj-np-tv-y]+$`)
|
||||||
|
|
||||||
// testInboundUpstreamDown: an inbound domain whose upstream refuses
|
// testInboundUpstreamDown: an inbound domain whose upstream refuses
|
||||||
// connections (the sink's address, a port nothing listens on). Port 25 takes
|
// connections (the sink's address, a port nothing listens on). Port 25 takes
|
||||||
// the message; Postfix cannot hand it on and logs "relay=none ...
|
// the message; Postfix cannot hand it on and logs "relay=none ...
|
||||||
@@ -65,6 +71,11 @@ print("" if r is None else "|".join(r))`, token)
|
|||||||
return false, fmt.Errorf("deferred with host %q, reply %q", f[2], f[3])
|
return false, fmt.Errorf("deferred with host %q, reply %q", f[2], f[3])
|
||||||
}
|
}
|
||||||
queueID = f[0]
|
queueID = f[0]
|
||||||
|
// A long queue id (enable_long_queue_ids=yes,
|
||||||
|
// build/postfix-config.sh): the time, "z", the inode.
|
||||||
|
if !longQueueID.MatchString(queueID) {
|
||||||
|
return false, fmt.Errorf("queue id %q is not a long one", queueID)
|
||||||
|
}
|
||||||
return true, nil
|
return true, nil
|
||||||
}
|
}
|
||||||
return false, fmt.Errorf("row %q\n%s", out, mailLogTail(h, "postfix/"))
|
return false, fmt.Errorf("row %q\n%s", out, mailLogTail(h, "postfix/"))
|
||||||
|
|||||||
Reference in New Issue
Block a user