mix/selfpost
1
0
Fork 0
Code Issues Pull Requests Actions 9 Packages Projects Releases Wiki Activity

findings of the reviews of audit stage 2b

Browse Source 
Technical review (Opus) and security review (Fable) of 5be109b..5154bba;
reports in the session's scratchpad. Nothing above low from security; one
bug and three risks from the technical review. Migration 0009 is not
deployed, so it was extended.

- Bug: two refusals of one second swapped their NOQUEUE lines (each line
  went to the newest lineless row). AttachRefusalLine now matches the
  client address too (send_log.client_ip, written by the journal milter -
  Postfix's line names the client and the sender, never the login: the
  sasl_username part of the pattern never matched and could be fed by HELO,
  security F1/F2), takes the nearest row in time and the oldest of equals,
  and a line already on a row goes on no other (security F6).
- Waiting lines and recoveries lived only in memory while the saved offset
  moved past them: a stop within 30 s lost them. The tracker now never saves
  an offset past the oldest waiting item in its file (tracker.hold,
  refusalLines.oldest) - for a recovery, the client= line that names its
  login - so they are read again; the "once" guards make that harmless.
- A recovered row whose login the tailer did not see was nobody's: its
  domain_id falls back to the envelope domain's.
- The fingerprint's file identity is the inode alone: the device number can
  change across a host reboot and make the same log look rotated.
- A refused row carries its queue id; UpdateStatus's "newest row" now skips
  refused rows, so a later message with that id does not flip it to sent.
- The alive mark: its age is read first and kept from then on, so asking
  supervisord for 30 s cannot make it stale; the entrypoint removes it with
  the sockets - a container restart is not the check's downtime, and a mark
  from before would have counted the whole outage as "≈".
- Security F3: the sender check prints no ";" of the client's in the From:
  text of its reply, and the tailer reads the envelope after the first ";"
  - Postfix's own text.
- A refused row with a queue id and no stored lines says so (it was told the
  mail.log text of 2.2.0); the fact's em dash is the house hyphen.
- Docs: architecture (client address, no login), guide § Health (warns on
  either count), help topics log and sender-check (design 1ca87f2).
- e2e: refusals_show_postfixs_line wants the refusal's own sender in the
  line; refusal_recovered_from_the_log waits past the tailer's 30 s and
  checks that a refusal the panel wrote got no recovered twin.

Not changed, written down: review 9 (mail.log.1 read to its end at that
moment; postlogd may still write to it until the reload), 10 (a log under
512 bytes has no fingerprint, so places fall back to time briefly after a
rotation), 6 (a 451 at MAIL FROM of another milter reads the same - the
client and sender must match too now), security F5/F12 (info). Review 12:
the refused and closed-by-the-panel variants of out-message were shown to
the owner as pictures of the real templates, not drawn in out-message.html:
the guard ties every drawn screen to a page of the engine, and a variant
has the page's outline.

Evidence on the test server for this tree: go vet clean, go test ./... 23
packages ok, make e2e exit 0 (refusal_recovered_from_the_log 70 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
mix
2026-10-14 18:38:22 +03:00
co-authored by Claude Opus 5.5
parent 1ca87f2d1a
commit af130e268a
22 changed files with 295 additions and 116 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/logtail/refusals.go
+60 -18
View File
@@ -2,6 +2,7 @@ package logtail
import (
"log"
"net"
"regexp"
"strings"
"time"
@@ -12,13 +13,14 @@ import (
// Postfix's line about a message the journal milter refused at MAIL FROM - a
// level-2 limit, a client address off the application's list - is put on the
// row the milter wrote, so the message's page shows it (audit 2026-10-12,
// stage 2b). The refusal has no queue id; the line is matched to its row by
// sender, login and time. As the stand writes it:
// stage 2b). The refusal has no queue id and the line no login; it is
// matched to its row by client address, sender and time. As the stand writes
// it:
//
// postfix/smtpd[1149]: NOQUEUE: milter-reject: MAIL from unknown[10.77.0.1]: 451 4.7.1 Service unavailable - try again later; from=<billing@sender.e2e.test> proto=ESMTP helo=<e2e-client.e2e.test>
//
// Port 25's smtpd (postfix/inbound/smtpd) is a stranger's and has no such row.
var refusalLineRe = regexp.MustCompile(`^\S+ postfix/(?:([a-z0-9-]+)/)?smtpd\[\d+\]: NOQUEUE: milter-reject: MAIL from \S+: [45]\d\d .*?; from=<([^>]*)>(?:.*? sasl_username=<?([^>\s]*)>?)?`)
var refusalLineRe = regexp.MustCompile(`^\S+ postfix/(?:([a-z0-9-]+)/)?smtpd\[\d+\]: NOQUEUE: milter-reject: MAIL from [^\[\s]*\[([^\]]+)\]: [45]\d\d .*?; from=<([^>]*)>`)
// refusalLineWait is how long a line waits for its row: the milter writes the
// row when it refuses, which can be read a moment after the line.
@@ -26,7 +28,7 @@ const refusalLineWait = 30 * time.Second
// RefusalStore is where the lines go. *store.Store satisfies it.
type RefusalStore interface {
AttachRefusalLine(login, from string, at time.Time, line string) (bool, error)
AttachRefusalLine(clientIP, from string, at time.Time, line string) (bool, error)
RecoverRefusal(e store.SendLogEntry, at time.Time) (bool, error)
}
@@ -36,8 +38,10 @@ type RefusalStore interface {
// postfix/cleanup[1339]: E543D3C75DA0: milter-reject: END-OF-MESSAGE from unknown[10.77.0.1]: 5.7.1 From: director@other.e2e.test - not an address this application sends as; from=<alerts@sender.e2e.test> to=<rcpt@sink.e2e.test> proto=ESMTP helo=<e2e-client.e2e.test>
//
// The From: line as the check printed it runs to the last " - " before the
// semicolon; the reason after it holds neither.
var fromRefusalRe = regexp.MustCompile(`^\S+ postfix/(?:[a-z0-9-]+/)?cleanup\[\d+\]: ([0-9A-Za-z]+): milter-reject: END-OF-MESSAGE from \S+: 5\.7\.1 From: (.*) - ([^;]*); from=<([^>]*)> to=<([^>]*)>`)
// first semicolon; the reason after it holds neither, and the check prints no
// ";" of the client's (sender.go printable), so that semicolon is Postfix's
// and what follows it - the envelope - is Postfix's own text.
var fromRefusalRe = regexp.MustCompile(`^\S+ postfix/(?:[a-z0-9-]+/)?cleanup\[\d+\]: ([0-9A-Za-z]+): milter-reject: END-OF-MESSAGE from \S+: 5\.7\.1 From: ([^;]*) - ([^;]*); from=<([^>]*)> to=<([^>]*)>`)
// clientLoginRe is smtpd's line naming the login a queue id was submitted
// under: "<id>: client=host[addr], sasl_method=PLAIN, sasl_username=<login>".
@@ -47,8 +51,9 @@ var clientLoginRe = regexp.MustCompile(`^\S+ postfix/(?:([a-z0-9-]+)/)?smtpd\[\d
const refusalLogins = 4096
type pendingRefusal struct {
login, from, line string
at, until time.Time
clientIP, from, line string
at, until time.Time
start Place // where the line starts in the log
}
// pendingRecovery is a From: refusal whose row is looked for once the panel
@@ -56,6 +61,14 @@ type pendingRefusal struct {
type pendingRecovery struct {
e store.SendLogEntry
at, until time.Time
start Place // where its client= line - or the refusal - starts
}
// loginLine is the login a queue id was submitted under and where smtpd's
// line saying so starts.
type loginLine struct {
login string
start Place
}
// refusalLines puts each refusal line on its row, and keeps for a while the
@@ -65,24 +78,49 @@ type refusalLines struct {
st RefusalStore
pending []pendingRefusal
recover []pendingRecovery
logins map[string]string // queue id -> login, from smtpd's client= line
logins map[string]loginLine // queue id -> login, from smtpd's client= line
loginKeys []string
}
// handle looks at one mail.log line.
func (r *refusalLines) handle(line string, now time.Time) {
// oldest is the earliest place in the file fp a waiting item starts at: the
// tracker saves no offset past it, so a stop before the wait is over reads the
// lines again - the client= line of a refusal to recover among them.
func (r *refusalLines) oldest(fp string) (Place, bool) {
var p Place
found := false
take := func(q Place) {
if q.FP == fp && (!found || q.End < p.End) {
p, found = q, true
}
}
for _, x := range r.pending {
take(x.start)
}
for _, x := range r.recover {
take(x.start)
}
return p, found
}
// handle looks at one mail.log line, which starts at start.
func (r *refusalLines) handle(line string, now time.Time, start Place) {
at, rest := lineTime(line, now)
if m := clientLoginRe.FindStringSubmatch(rest); m != nil && m[1] != "inbound" {
r.remember(m[2], m[3])
r.remember(m[2], loginLine{login: m[3], start: start})
return
}
if m := fromRefusalRe.FindStringSubmatch(rest); m != nil {
from := start
login, ok := r.logins[m[1]]
if ok && login.start.FP == start.FP && login.start.End < start.End {
from = login.start
}
r.recover = append(r.recover, pendingRecovery{
e: store.SendLogEntry{
QueueID: m[1], AppLogin: r.logins[m[1]], From: m[4], To: m[5], Domain: domainOf(m[4]),
QueueID: m[1], AppLogin: login.login, From: m[4], To: m[5], Domain: domainOf(m[4]),
Reason: store.ReasonFrom, HeaderFrom: m[2],
},
at: at, until: now.Add(refusalLineWait),
at: at, until: now.Add(refusalLineWait), start: from,
})
return
}
@@ -90,7 +128,11 @@ func (r *refusalLines) handle(line string, now time.Time) {
if m == nil || m[1] == "inbound" {
return
}
p := pendingRefusal{login: m[3], from: m[2], line: line, at: at, until: now.Add(refusalLineWait)}
ip := net.ParseIP(strings.TrimPrefix(m[2], "IPv6:"))
if ip == nil {
return
}
p := pendingRefusal{clientIP: ip.String(), from: m[3], line: line, at: at, until: now.Add(refusalLineWait), start: start}
if !r.attach(p) {
r.pending = append(r.pending, p)
}
@@ -127,9 +169,9 @@ func (r *refusalLines) retry(now time.Time) {
// remember keeps the login a queue id was submitted under, the oldest going
// first beyond refusalLogins.
func (r *refusalLines) remember(queueID, login string) {
func (r *refusalLines) remember(queueID string, login loginLine) {
if r.logins == nil {
r.logins = make(map[string]string)
r.logins = make(map[string]loginLine)
}
if _, ok := r.logins[queueID]; !ok {
r.loginKeys = append(r.loginKeys, queueID)
@@ -150,7 +192,7 @@ func domainOf(addr string) string {
}
func (r *refusalLines) attach(p pendingRefusal) bool {
ok, err := r.st.AttachRefusalLine(p.login, p.from, p.at, p.line)
ok, err := r.st.AttachRefusalLine(p.clientIP, p.from, p.at, p.line)
if err != nil {
log.Printf("log-tailer: refusal line: %v", err)
return true // not retried: the store said no for its own reason