mix/selfpost
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity

inbound-antispam-panel: findings of the technical and security reviews
test / test (push) Canceled after 0s
Details

Browse Source 
Steps 11 and 12 of docs/plans/inbound-antispam-panel.md: 12/12. Two
read-only reviews of d65d632..HEAD by different models (Opus,
technical; Fable, security), each given the code and questions, not
conclusions. Nothing Critical or High in the feature; the High the
technical review found outside it is the commit before this one.

Log tailer (internal/logtail/inbound.go):
  - the reply of a refusal ends at the first "; from=<", Postfix's own,
    and the HELO name after it is whatever is left. Taking the longest
    reply let a HELO name that spelt "; from=<..> to=<..> proto=.."
    supply the row's sender, recipient and reply - and so make a
    refusal read as the deny list's;
  - an address with a quote, a bracket or a space is not kept: a quoted
    sender could otherwise supply the recipient;
  - the journal is cut to its row cap every 500 rows written, not only
    every six hours;
  - a queue id is forgotten when the message is queued or the id is
    opened by another listener, and a stale place in the queue of
    arrivals no longer drops a live id;
  - milter-hold lines are not read: the held message reaches the
    journal milter, which writes its row - it was written twice.
Lists (internal/inbound/lists.go): a change of the table and the
rewrite of the files are one step under a mutex; a failed rewrite puts
a removed entry back and rewrites after taking a new one out.
Store: the row cap removes the oldest by time, as the page orders them.
Page: on a sheet a refusal's text is a Reply whatever it starts with.

Help and the guide say that an allow entry for a sender address trusts
an address anyone can claim (design commit before this one).

docs/security.md: the two reviews; the roles table gains Inbound > Log
and Filter lists; Mail path describes the milters, the tailer and the
lists; four accepted risks - the filter's verdict while it is down, an
allow entry by sender address, the list files' mode, log rows held by
domain name. The plan's stale lines (size and rate refusals, who reads
the journal) are corrected.

Left as found, told to the owner: the journal's rows after an unclean
stop may be written twice (the tailer saves its place every five
seconds); a "Deny sender" link is offered for an address the form then
refuses; 1.2.3.4 and 1.2.3.4/32 are two entries.

Checked on the test server, uncommitted tree: go vet ./... ok,
go test ./... ok (22 packages), make e2e ok.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
mix
2026-10-12 07:51:50 +03:00
co-authored by Claude Opus 5.5
parent 5f86457848
commit c731a98a0f
15 changed files with 388 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/logtail/inbound.go
+70 -16
View File
@@ -34,6 +34,9 @@ type Inbound struct {
// postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from mx.example.net[203.0.113.9]: 550 5.1.1 <nobody@in.example.com>: Recipient address rejected: User unknown in relay recipient table; from=<a@example.net> to=<nobody@in.example.com> proto=ESMTP helo=<mx.example.net>
// postfix/cleanup[52]: 4A1B2C3D: milter-reject: END-OF-MESSAGE from mx.example.net[203.0.113.9]: 5.7.1 Spam message rejected; from=<a@example.net> to=<b@in.example.com> proto=ESMTP helo=<mx.example.net>
//
// A message the filter holds (milter-hold) is not read here: it goes on to the
// inbound journal milter, which writes its row with the subject.
//
// The first is Postfix's own policy, written by the port-25 smtpd, which
// build/postfix-config.sh names postfix/inbound so that its lines are told
// from those of the submission ports. The second is the filter's verdict at
@@ -45,9 +48,20 @@ type Inbound struct {
// name: the tail of such a line is the sender's own words (HELO name,
// addresses), and a pattern that could start anywhere would let him write
// journal rows.
//
// For the same reason the reply is the shortest text before "; from=<", and
// the HELO name - the last thing on the line, which Postfix logs as the client
// wrote it, spaces and brackets included - is whatever is left. Taking the
// longest reply instead would let a HELO name that itself spells
// "; from=<…> to=<…> proto=…" supply the sender, the recipient and the reply
// of the row (security review, 2026-10-12).
var (
inboundRejectRe = regexp.MustCompile(`^\S+ postfix/(inbound/smtpd|cleanup)\[\d+\]: ([0-9A-Za-z]+): (reject|milter-reject|milter-hold|milter-discard): \S+ from [^\[\s]*\[([0-9A-Fa-f:.]+)\]: (.*); from=<([^>]*)> to=<([^>]*)> proto=\S+(?: helo=<[^>]*>)?$`)
inboundRejectRe = regexp.MustCompile(`^\S+ postfix/(inbound/smtpd|cleanup)\[\d+\]: ([0-9A-Za-z]+): (reject|milter-reject|milter-discard): \S+ from [^\[\s]*\[([0-9A-Fa-f:.]+)\]: (.*?); from=<([^>]*)> to=<([^>]*)> proto=\S+(?: helo=<.*>)?$`)
inboundClientRe = regexp.MustCompile(`^\S+ postfix/inbound/smtpd\[\d+\]: ([0-9A-Za-z]+): client=`)
// A queue id stops being port 25's when the message is queued (qmgr takes
// it: the filter let it through) or when another listener opens a message
// under the same id - Postfix reuses ids.
inboundForgetRe = regexp.MustCompile(`^\S+ postfix/(?:qmgr\[\d+\]: ([0-9A-Za-z]+): from=|(?:[a-z0-9-]+/)?smtpd\[\d+\]: ([0-9A-Za-z]+): client=)`)
)
// A row holds one line of reason; Postfix's reply text carries the far end's
@@ -55,21 +69,40 @@ var (
const inboundDetailMax = 255
// inboundQueueIDs is how many port-25 queue ids are remembered while waiting
// for cleanup's verdict line on them. A message is in that state for the
// seconds its DATA takes; the cap only bounds a flood.
// for cleanup's verdict line on them. An id is forgotten when the verdict
// comes, when the message is queued, or when another listener reuses it; the
// cap bounds what is left - messages the client abandoned - and a flood.
const inboundQueueIDs = 4096
// inboundPruneEvery is how many rows the tailer writes before it has the
// journal cut back to its row cap, without waiting for the six-hourly prune.
// Refusals are written at the rate strangers send RCPT TO, which no limit of
// Postfix's counts, so between two prunes the journal would otherwise grow as
// fast as someone cared to make it.
const inboundPruneEvery = 500
// inboundJournal is the per-run state of the inbound half.
type inboundJournal struct {
st InboundStore
// port25 holds the queue ids the port-25 smtpd opened and cleanup has not
// reported on yet; order is their arrival, for dropping the oldest.
port25 map[string]struct{}
order []string
// port25 holds the queue ids the port-25 smtpd opened and nobody has
// accounted for yet, each with the number of its arrival; order is those
// arrivals, for dropping the oldest. An id that was forgotten and opened
// again is in order twice: the number tells the stale entry from the live
// one when the stale one comes up for dropping.
port25 map[string]uint64
order []queued
seq uint64
// written counts the rows since the journal was last cut to its cap.
written int
}
type queued struct {
id string
seq uint64
}
func newInboundJournal(st InboundStore) *inboundJournal {
return &inboundJournal{st: st, port25: make(map[string]struct{})}
return &inboundJournal{st: st, port25: make(map[string]uint64)}
}
// handle looks at one mail.log line and writes a journal row when it is a
@@ -80,11 +113,23 @@ func (j *inboundJournal) handle(line string, now time.Time) {
j.remember(m[1])
return
}
if m := inboundForgetRe.FindStringSubmatch(rest); m != nil {
delete(j.port25, m[1]+m[2])
return
}
m := inboundRejectRe.FindStringSubmatch(rest)
if m == nil {
return
}
program, queueID, kind, clientIP, reason, from, to := m[1], m[2], m[3], m[4], m[5], m[6], m[7]
// The sender's address is his own text too, and with a quoted local part it
// may hold "> to=<…> proto=…" - which would end the sender early and supply
// the recipient. Such a line shows it: the sender is left with an unclosed
// quote. An address with a quote, a bracket or a space in it is not one the
// journal keeps.
if strings.ContainsAny(from, "\"<> ") || strings.ContainsAny(to, "\"<> ") {
return
}
if program == "cleanup" {
if _, ok := j.port25[queueID]; !ok {
return // a submission-port message, or one opened before this run
@@ -121,11 +166,8 @@ func (j *inboundJournal) handle(line string, now time.Time) {
Detail: printable(reason, inboundDetailMax),
At: at,
}
switch {
case kind == "reject":
if kind == "reject" {
e.Engine = store.SpamEnginePostfix
case kind == "milter-hold":
e.Decision = store.SpamDecisionQuarantine
}
// "450 4.7.1 …" from Postfix, "4.7.1 …" from a milter: either way a reply
// that starts with 4 asks the sender to come back.
@@ -134,6 +176,14 @@ func (j *inboundJournal) handle(line string, now time.Time) {
}
if err := j.st.InsertInboundSpam(e); err != nil {
log.Printf("log-tailer: inbound journal: record %s -> %s: %v", from, to, err)
return
}
if j.written++; j.written >= inboundPruneEvery {
j.written = 0
// The zero time removes nothing by age: this is the cap alone.
if _, err := j.st.PruneInboundSpam(time.Time{}, inboundSpamLogMaxRows); err != nil {
log.Printf("log-tailer: inbound journal: cut to %d rows: %v", inboundSpamLogMaxRows, err)
}
}
}
@@ -141,12 +191,16 @@ func (j *inboundJournal) remember(queueID string) {
if _, ok := j.port25[queueID]; ok {
return
}
if len(j.order) >= inboundQueueIDs {
delete(j.port25, j.order[0])
for len(j.order) >= inboundQueueIDs {
oldest := j.order[0]
j.order = j.order[1:]
if j.port25[oldest.id] == oldest.seq {
delete(j.port25, oldest.id)
}
}
j.port25[queueID] = struct{}{}
j.order = append(j.order, queueID)
j.seq++
j.port25[queueID] = j.seq
j.order = append(j.order, queued{queueID, j.seq})
}
// printable cuts s to max bytes and replaces everything outside printable
internal/logtail/inbound_test.go
+90 -2
View File
@@ -106,7 +106,6 @@ func TestInboundMilterVerdictNeedsAPort25QueueID(t *testing.T) {
func TestInboundMilterDecisions(t *testing.T) {
for kind, c := range map[string]struct{ reply, decision string }{
"milter-reject": {"4.7.1 Try again later", store.SpamDecisionTempfail},
"milter-hold": {"milter triggers HOLD action", store.SpamDecisionQuarantine},
"milter-discard": {"milter triggers DISCARD action", store.SpamDecisionReject},
} {
j, st := inboundFixture()
@@ -128,7 +127,7 @@ func TestInboundIgnores(t *testing.T) {
"a connect line": head + "postfix/inbound/smtpd[41]: connect from mx.example.net[203.0.113.9]",
// The HELO name is the sender's; a line cannot be started from inside one.
"a refusal spelled inside another line": head + "postfix/inbound/smtpd[41]: warning: hostname x does not resolve: mail postfix/inbound/smtpd[1]: NOQUEUE: reject: RCPT from x[203.0.113.9]: 550 no; from=<a@example.net> to=<b@in.example.com> proto=ESMTP helo=<x>",
"an address with a bracket in it": head + "postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from x[203.0.113.9]: 550 no; from=<a> to=<v@in.example.com> proto=X helo=<y>@example.net> to=<b@elsewhere.example> proto=ESMTP helo=<x>",
"an address with a bracket in it": head + "postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from x[203.0.113.9]: 554 5.7.1 <b@elsewhere.example>: Relay access denied; from=<\"a> to=<v@in.example.com> proto=X helo=<y\"@example.net> to=<b@elsewhere.example> proto=ESMTP helo=<x>",
} {
j, st := inboundFixture()
j.handle(line, tailNow)
@@ -159,6 +158,95 @@ func TestInboundLookupFailureLeavesNoRow(t *testing.T) {
}
}
// A message the filter holds goes on to the inbound journal milter, which
// writes its row; a second one from the log would be the same message twice.
func TestInboundHoldIsTheMiltersRow(t *testing.T) {
j, st := inboundFixture()
j.handle(head+"postfix/inbound/smtpd[41]: Q1: client=mx.example.net[203.0.113.9]", tailNow)
j.handle(head+"postfix/cleanup[52]: Q1: milter-hold: END-OF-MESSAGE from mx.example.net[203.0.113.9]: milter triggers HOLD action; from=<a@example.net> to=<b@in.example.com> proto=ESMTP helo=<mx.example.net>", tailNow)
if len(st.rows) != 0 {
t.Errorf("rows = %+v", st.rows)
}
}
// The HELO name is the last thing on the line and is logged as the client
// wrote it. It cannot supply the row: the reply ends at the first "; from=<",
// which is Postfix's own.
func TestInboundHeloCannotWriteTheRow(t *testing.T) {
j, st := inboundFixture()
j.handle(head+"postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from x[203.0.113.9]: 550 5.1.1 <nobody@in.example.com>: Recipient address rejected: User unknown in relay recipient table; from=<real@example.net> to=<nobody@in.example.com> proto=ESMTP helo=<x 5.7.1 Refused by this server's deny list; from=<victim@bank.example> to=<ceo@in.example.com> proto=ESMTP helo=<y>", tailNow)
if len(st.rows) != 1 {
t.Fatalf("rows = %+v", st.rows)
}
got := st.rows[0]
if got.From != "real@example.net" || got.To != "nobody@in.example.com" ||
got.Detail != "550 5.1.1 <nobody@in.example.com>: Recipient address rejected: User unknown in relay recipient table" {
t.Errorf("row = %+v", got)
}
// A HELO name with a closing bracket in it does not lose the line either.
j, st = inboundFixture()
j.handle(head+"postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from x[203.0.113.9]: 550 5.1.1 <nobody@in.example.com>: no; from=<real@example.net> to=<nobody@in.example.com> proto=ESMTP helo=<a>b>", tailNow)
if len(st.rows) != 1 || st.rows[0].From != "real@example.net" {
t.Errorf("rows = %+v", st.rows)
}
// The same on a verdict line: the filter's reply is not extended by the
// HELO name, so a stranger cannot make a refusal look like the deny list's.
j, st = inboundFixture()
j.handle(head+"postfix/inbound/smtpd[41]: Q1: client=x[203.0.113.9]", tailNow)
j.handle(head+"postfix/cleanup[52]: Q1: milter-reject: END-OF-MESSAGE from x[203.0.113.9]: 5.7.1 Spam message rejected; from=<real@example.net> to=<b@in.example.com> proto=ESMTP helo=<x this server's deny list; from=<spoofed@example.org> to=<b@in.example.com> proto=ESMTP helo=<y>", tailNow)
if len(st.rows) != 1 || st.rows[0].Detail != "5.7.1 Spam message rejected" || st.rows[0].From != "real@example.net" {
t.Errorf("rows = %+v", st.rows)
}
}
// A queue id is port 25's only until the message is queued or the id is
// opened again by another listener - Postfix reuses ids.
func TestInboundQueueIDIsForgotten(t *testing.T) {
const verdict = "postfix/cleanup[52]: Q1: milter-reject: END-OF-MESSAGE from app[203.0.113.4]: 4.7.1 Service unavailable - try again later; from=<a@example.net> to=<b@in.example.com> proto=ESMTP helo=<app>"
for name, forget := range map[string]string{
"queued": "postfix/qmgr[58]: Q1: from=<a@example.net>, size=600, nrcpt=1 (queue active)",
"reused by the submission port": "postfix/smtpd[20]: Q1: client=app.example.com[203.0.113.4], sasl_method=PLAIN, sasl_username=app",
} {
j, st := inboundFixture()
j.handle(head+"postfix/inbound/smtpd[41]: Q1: client=mx.example.net[203.0.113.9]", tailNow)
j.handle(head+forget, tailNow)
j.handle(head+verdict, tailNow)
if len(st.rows) != 0 || len(j.port25) != 0 {
t.Errorf("%s: rows %+v, remembered %v", name, st.rows, j.port25)
}
}
}
// An id that was forgotten and opened again is not dropped by its own stale
// place in the queue of arrivals.
func TestInboundStaleArrivalDoesNotDropALiveID(t *testing.T) {
j, _ := inboundFixture()
j.remember("Q1")
delete(j.port25, "Q1") // its verdict came
for i := 0; i < inboundQueueIDs-2; i++ {
j.remember("F" + time.Duration(i).String())
}
j.remember("Q1") // reused; order is now full, with the stale Q1 first
j.remember("LAST")
if _, ok := j.port25["Q1"]; !ok {
t.Error("the live Q1 was dropped by its stale arrival")
}
}
// Between the six-hourly prunes the tailer has the journal cut to its cap
// itself, every so many rows: refusals come at the rate strangers send them.
func TestInboundCutsTheJournalAsItWrites(t *testing.T) {
j, st := inboundFixture()
for i := 0; i < inboundPruneEvery*2+10; i++ {
j.handle(unknownRcp, tailNow)
}
if len(st.pruned) != 2 || st.pruned[0] != inboundSpamLogMaxRows {
t.Errorf("cuts = %v, want two, to %d rows", st.pruned, inboundSpamLogMaxRows)
}
}
// The queue ids waiting for a verdict are bounded.
func TestInboundQueueIDsAreBounded(t *testing.T) {
j, _ := inboundFixture()
internal/logtail/offset.go
+7 -4
View File
@@ -20,8 +20,10 @@ const (
// which os.SameFile cannot answer across a restart.
fingerprintSize = 512
// persistInterval throttles the offset write. Losing up to this much
// progress on a crash only means re-parsing a few lines (UpdateStatus is
// idempotent), which is much cheaper than a database write per poll tick.
// progress on a crash means re-parsing a few lines, which is much cheaper
// than a database write per poll tick. For a delivery status that changes
// nothing (UpdateStatus is idempotent); a refusal in those lines gets a
// second row in the inbound journal.
persistInterval = 5 * time.Second
)
@@ -46,8 +48,9 @@ type tracker struct {
// state whose fingerprint still matches means continue from it, parsing the
// tail written while the panel was down; anything else means the file is not
// the one the offset referred to (rotated, recreated or truncated in the
// meantime), so read it from the start. Re-parsing lines already seen is
// harmless: UpdateStatus writes the same status onto the same row.
// meantime), so read it from the start. Re-parsing lines already seen writes
// the same status onto the same send-log row, and a second inbound journal
// row for a refusal among them.
func (t *tracker) resume(f *os.File) int64 {
size, err := fileSize(f)
if err != nil {