findings of the reviews of audit stage 2b

Technical review (Opus) and security review (Fable) of 5be109b..5154bba;
reports in the session's scratchpad. Nothing above low from security; one
bug and three risks from the technical review. Migration 0009 is not
deployed, so it was extended.

- Bug: two refusals of one second swapped their NOQUEUE lines (each line
  went to the newest lineless row). AttachRefusalLine now matches the
  client address too (send_log.client_ip, written by the journal milter -
  Postfix's line names the client and the sender, never the login: the
  sasl_username part of the pattern never matched and could be fed by HELO,
  security F1/F2), takes the nearest row in time and the oldest of equals,
  and a line already on a row goes on no other (security F6).
- Waiting lines and recoveries lived only in memory while the saved offset
  moved past them: a stop within 30 s lost them. The tracker now never saves
  an offset past the oldest waiting item in its file (tracker.hold,
  refusalLines.oldest) - for a recovery, the client= line that names its
  login - so they are read again; the "once" guards make that harmless.
- A recovered row whose login the tailer did not see was nobody's: its
  domain_id falls back to the envelope domain's.
- The fingerprint's file identity is the inode alone: the device number can
  change across a host reboot and make the same log look rotated.
- A refused row carries its queue id; UpdateStatus's "newest row" now skips
  refused rows, so a later message with that id does not flip it to sent.
- The alive mark: its age is read first and kept from then on, so asking
  supervisord for 30 s cannot make it stale; the entrypoint removes it with
  the sockets - a container restart is not the check's downtime, and a mark
  from before would have counted the whole outage as "≈".
- Security F3: the sender check prints no ";" of the client's in the From:
  text of its reply, and the tailer reads the envelope after the first ";"
  - Postfix's own text.
- A refused row with a queue id and no stored lines says so (it was told the
  mail.log text of 2.2.0); the fact's em dash is the house hyphen.
- Docs: architecture (client address, no login), guide § Health (warns on
  either count), help topics log and sender-check (design 1ca87f2).
- e2e: refusals_show_postfixs_line wants the refusal's own sender in the
  line; refusal_recovered_from_the_log waits past the tailer's 30 s and
  checks that a refusal the panel wrote got no recovered twin.

Not changed, written down: review 9 (mail.log.1 read to its end at that
moment; postlogd may still write to it until the reload), 10 (a log under
512 bytes has no fingerprint, so places fall back to time briefly after a
rotation), 6 (a 451 at MAIL FROM of another milter reads the same - the
client and sender must match too now), security F5/F12 (info). Review 12:
the refused and closed-by-the-panel variants of out-message were shown to
the owner as pictures of the real templates, not drawn in out-message.html:
the guard ties every drawn screen to a page of the engine, and a variant
has the page's outline.

Evidence on the test server for this tree: go vet clean, go test ./... 23
packages ok, make e2e exit 0 (refusal_recovered_from_the_log 70 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
mix
2026-10-14 18:38:22 +03:00
co-authored by Claude Opus 5.5
parent 1ca87f2d1a
commit af130e268a
22 changed files with 295 additions and 116 deletions
+5 -2
View File
@@ -156,8 +156,11 @@ chmod 2750 /run/opendkim /run/selfpost
# A container that was killed leaves its sockets behind, and a file nobody # A container that was killed leaves its sockets behind, and a file nobody
# listens on would satisfy postfix-wrapper.sh's wait: Postfix would start # listens on would satisfy postfix-wrapper.sh's wait: Postfix would start
# before the sender check does and defer mail until it is there. The check's # before the sender check does and defer mail until it is there. The check's
# alive mark stays - it is how the downtime of a restart is told. # alive mark goes too: it tells the downtime of the panel restarting inside a
rm -f /run/opendkim/*.sock /run/selfpost/*.sock # running container, and a restart of the whole container is not the check's
# downtime (guide § Health) - a mark left from before would count the whole
# outage when supervisord cannot say since when Postfix runs.
rm -f /run/opendkim/*.sock /run/selfpost/*.sock /run/selfpost/*.alive
# Generate the outbound-relay Postfix configuration from the environment (spec # Generate the outbound-relay Postfix configuration from the environment (spec
# 5). Kept out of the image build so cert paths, rate limits, hostname and the # 5). Kept out of the image build so cert paths, rate limits, hostname and the
+15 -8
View File
@@ -33,13 +33,21 @@ func serveSenderCheck(ctx context.Context, cfg config, st *store.Store) error {
// Statistics are the database's business and supervisord's, and the check // Statistics are the database's business and supervisord's, and the check
// is neither: they are kept beside it, and nothing of them is waited for // is neither: they are kept beside it, and nothing of them is waited for
// before it serves. // before it serves.
go func() {
now := time.Now() now := time.Now()
mark := cfg.senderSocket + ".alive" mark := cfg.senderSocket + ".alive"
// The mark's age is read before the mark is kept again: asking supervisord
// may take half a minute, and the mark must not go stale meanwhile, or a
// restart in that half minute would count it as downtime.
var markAt time.Time
if fi, err := os.Stat(mark); err == nil {
markAt = fi.ModTime()
}
go keepAlive(ctx, mark, st)
go func() {
if err := st.StartSenderCheckStats(now); err != nil { if err := st.StartSenderCheckStats(now); err != nil {
log.Printf("sender-check: statistics: %v", err) log.Printf("sender-check: statistics: %v", err)
} }
if down, estimated := senderCheckDowntime(mark, now, postfixStarted); down > 0 { if down, estimated := senderCheckDowntime(markAt, now, postfixStarted); down > 0 {
if estimated { if estimated {
log.Printf("sender-check: was not answering for at most %s (supervisord did not say since when Postfix ran)", down.Round(time.Second)) log.Printf("sender-check: was not answering for at most %s (supervisord did not say since when Postfix ran)", down.Round(time.Second))
} else { } else {
@@ -51,7 +59,6 @@ func serveSenderCheck(ctx context.Context, cfg config, st *store.Store) error {
} }
} }
} }
keepAlive(ctx, mark, st)
}() }()
// The two tables the mail path itself goes by: whom Postfix lets send as // The two tables the mail path itself goes by: whom Postfix lets send as
@@ -69,19 +76,19 @@ type postfixState struct {
} }
// senderCheckDowntime is for how long the check was not serving before now // senderCheckDowntime is for how long the check was not serving before now
// while Postfix was: the time since the mark at path was last set, or since // while Postfix was: the time since the mark was last set (markAt; zero
// when there is none), or since
// Postfix started if that was later. No mark means the container is new, and // Postfix started if that was later. No mark means the container is new, and
// a Postfix that is not running means mail was stopped for another reason - // a Postfix that is not running means mail was stopped for another reason -
// a rollout, a reboot - and neither is the check's downtime. When supervisord // a rollout, a reboot - and neither is the check's downtime. When supervisord
// could not say, the time since the mark is taken and estimated is true: an // could not say, the time since the mark is taken and estimated is true: an
// upper bound, since Postfix may not have been running all of it (audit // upper bound, since Postfix may not have been running all of it (audit
// 2026-10-12, stage 2b - until then such a start counted nothing). // 2026-10-12, stage 2b - until then such a start counted nothing).
func senderCheckDowntime(path string, now time.Time, ask func() postfixState) (down time.Duration, estimated bool) { func senderCheckDowntime(markAt, now time.Time, ask func() postfixState) (down time.Duration, estimated bool) {
fi, err := os.Stat(path) if markAt.IsZero() {
if err != nil {
return 0, false return 0, false
} }
from := fi.ModTime() from := markAt
p := ask() p := ask()
switch { switch {
case !p.known: case !p.known:
+2 -12
View File
@@ -1,8 +1,6 @@
package main package main
import ( import (
"os"
"path/filepath"
"testing" "testing"
"time" "time"
) )
@@ -13,16 +11,8 @@ import (
// that time. // that time.
func TestSenderCheckDowntime(t *testing.T) { func TestSenderCheckDowntime(t *testing.T) {
now := time.Date(2026, 10, 12, 18, 0, 0, 0, time.UTC) now := time.Date(2026, 10, 12, 18, 0, 0, 0, time.UTC)
mark := filepath.Join(t.TempDir(), "sender.sock.alive") var mark time.Time // the alive mark's time; zero is no mark
set := func(at time.Time) { set := func(at time.Time) { mark = at }
t.Helper()
if err := os.WriteFile(mark, nil, 0o644); err != nil {
t.Fatal(err)
}
if err := os.Chtimes(mark, at, at); err != nil {
t.Fatal(err)
}
}
postfix := func(started time.Time, running bool) func() postfixState { postfix := func(started time.Time, running bool) func() postfixState {
return func() postfixState { return postfixState{started: started, running: running, known: true} } return func() postfixState { return postfixState{started: started, running: running, known: true} }
} }
+5 -3
View File
@@ -422,9 +422,11 @@ judged); the sender check writes it on the row, and the message's page shows
the lines stored under it - only those of the minutes around the refusal, since the lines stored under it - only those of the minutes around the refusal, since
Postfix frees the id at once. A message the journal milter refused at Postfix frees the id at once. A message the journal milter refused at
`MAIL FROM` has none: the tailer reads smtpd's `NOQUEUE: milter-reject: MAIL `MAIL FROM` has none: the tailer reads smtpd's `NOQUEUE: milter-reject: MAIL
from …; from=<…>` and puts it on the newest refused row of that sender (and from <client>[<address>]: …; from=<…>` - it names no login - and puts it on
login, when the line names it) within two minutes (`send_log.refusal_line`), the newest refused row of that client address and sender within two minutes
retrying for 30 s when the row is not written yet. The sender check writes its (`send_log.refusal_line`; the milter writes `send_log.client_ip`), retrying
for 30 s when the row is not written yet; a line already on a row goes on no
other. The sender check writes its
row through a buffer after answering 550; a panel stopped in between wrote row through a buffer after answering 550; a panel stopped in between wrote
none. The tailer remembers each queue id's login from smtpd's `client=` line, none. The tailer remembers each queue id's login from smtpd's `client=` line,
and 30 s after cleanup's `milter-reject: END-OF-MESSAGE … 5.7.1 From:` line it and 30 s after cleanup's `milter-reject: END-OF-MESSAGE … 5.7.1 From:` line it
+1 -1
View File
@@ -414,7 +414,7 @@ readings, in the order of what stops mail first:
is one the panel stopped answering for after it had begun. A time marked ≈ is one the panel stopped answering for after it had begun. A time marked ≈
is at most what it shows: at that start supervisord could not say since is at most what it shows: at that start supervisord could not say since
when Postfix ran. Days are UTC days. The card warns when when Postfix ran. Days are UTC days. The card warns when
a message was deferred in the last 24 hours. If these numbers do not stay a connection was turned away or a message deferred in the last 24 hours. If these numbers do not stay
at nothing, the panel is in the way of your mail more than it should be - at nothing, the panel is in the way of your mail more than it should be -
report it. report it.
- **Hostname and reverse DNS** — compares `SELFPOST_HOSTNAME` against the PTR - **Hostname and reverse DNS** — compares `SELFPOST_HOSTNAME` against the PTR
+7 -4
View File
@@ -8,9 +8,12 @@ import (
"syscall" "syscall"
) )
// fileID names the file itself - device and inode - which a rename keeps and // fileID names the file itself - its inode - which a rename keeps and a new
// a new file does not share. It is part of the log's fingerprint, so two log // file does not share. It is part of the log's fingerprint, so two log files
// files whose first bytes happen to match are still told apart. // whose first bytes happen to match are still told apart. The device number
// is left out: it can change across a host's reboot (LVM, btrfs, network
// file systems), and the same log would then pass for a rotated one (review
// of stage 2b).
func fileID(f *os.File) string { func fileID(f *os.File) string {
info, err := f.Stat() info, err := f.Stat()
if err != nil { if err != nil {
@@ -20,5 +23,5 @@ func fileID(f *os.File) string {
if !ok { if !ok {
return "" return ""
} }
return fmt.Sprintf("%d.%d", st.Dev, st.Ino) return fmt.Sprintf("%d", st.Ino)
} }
+3 -2
View File
@@ -168,7 +168,7 @@ func Run(ctx context.Context, path string, st StatusStore, retention RetentionDa
keeper.flush() keeper.flush()
refusals.retry(time.Now().UTC()) refusals.retry(time.Now().UTC())
} }
tr := &tracker{st: st, path: path} tr := &tracker{st: st, path: path, hold: refusals.oldest}
return follow(ctx, path, tr, caughtUp, flush, func(line string) { return follow(ctx, path, tr, caughtUp, flush, func(line string) {
// Until the first read reaches end-of-file the lines are those written // Until the first read reaches end-of-file the lines are those written
// while the panel was down - and, after an unclean stop, some it had // while the panel was down - and, after an unclean stop, some it had
@@ -187,7 +187,7 @@ func Run(ctx context.Context, path string, st StatusStore, retention RetentionDa
deferred.handle(line, now, catchingUp, tr.line) deferred.handle(line, now, catchingUp, tr.line)
} }
keeper.add(line, now, catchingUp) keeper.add(line, now, catchingUp)
refusals.handle(line, now) refusals.handle(line, now, Place{FP: tr.line.FP, End: tr.lineStart})
queueID, recipient, status, ok := parseDelivery(line) queueID, recipient, status, ok := parseDelivery(line)
if !ok { if !ok {
return return
@@ -647,6 +647,7 @@ func follow(ctx context.Context, path string, tr *tracker, caughtUp chan struct{
full := pending + line full := pending + line
pending = "" pending = ""
tr.line = Place{FP: tr.fp, End: consumed} tr.line = Place{FP: tr.fp, End: consumed}
tr.lineStart = consumed - int64(len(full))
handle(strings.TrimRight(full, "\r\n")) handle(strings.TrimRight(full, "\r\n"))
} }
} }
+15 -2
View File
@@ -47,9 +47,16 @@ type tracker struct {
behind bool behind bool
prev store.LogtailState prev store.LogtailState
// line is the place of the line being handled: set by follow and // line is the place of the line being handled, lineStart the offset it
// readRotated before each call of handle. // starts at: set by follow and readRotated before each call of handle.
line Place line Place
lineStart int64
// hold, when set, names the earliest place in a file that something read
// from the log still waits on (a refusal's line, a refusal to recover):
// no offset past it is saved, so a stop - clean or not - before the wait
// is over reads it again (review of stage 2b).
hold func(fp string) (Place, bool)
} }
// Place is where a line of the log ends: the fingerprint of its file and the // Place is where a line of the log ends: the fingerprint of its file and the
@@ -158,6 +165,7 @@ func (t *tracker) readRotated(handle func(string)) bool {
r := bufio.NewReader(f) r := bufio.NewReader(f)
for { for {
line, err := r.ReadString('\n') line, err := r.ReadString('\n')
t.lineStart = end
end += int64(len(line)) end += int64(len(line))
if line != "" { if line != "" {
t.line = Place{FP: fp, End: end} t.line = Place{FP: fp, End: end}
@@ -198,6 +206,11 @@ func (t *tracker) adopt(f *os.File) {
// record persists offset, at most once per persistInterval unless force is set // record persists offset, at most once per persistInterval unless force is set
// (rotation and shutdown, where the write must not be skipped). // (rotation and shutdown, where the write must not be skipped).
func (t *tracker) record(f *os.File, offset int64, force bool) { func (t *tracker) record(f *os.File, offset int64, force bool) {
if t.hold != nil {
if p, ok := t.hold(t.fp); ok && p.End < offset {
offset = p.End
}
}
if offset == t.saved { if offset == t.saved {
return return
} }
+59 -17
View File
@@ -2,6 +2,7 @@ package logtail
import ( import (
"log" "log"
"net"
"regexp" "regexp"
"strings" "strings"
"time" "time"
@@ -12,13 +13,14 @@ import (
// Postfix's line about a message the journal milter refused at MAIL FROM - a // Postfix's line about a message the journal milter refused at MAIL FROM - a
// level-2 limit, a client address off the application's list - is put on the // level-2 limit, a client address off the application's list - is put on the
// row the milter wrote, so the message's page shows it (audit 2026-10-12, // row the milter wrote, so the message's page shows it (audit 2026-10-12,
// stage 2b). The refusal has no queue id; the line is matched to its row by // stage 2b). The refusal has no queue id and the line no login; it is
// sender, login and time. As the stand writes it: // matched to its row by client address, sender and time. As the stand writes
// it:
// //
// postfix/smtpd[1149]: NOQUEUE: milter-reject: MAIL from unknown[10.77.0.1]: 451 4.7.1 Service unavailable - try again later; from=<billing@sender.e2e.test> proto=ESMTP helo=<e2e-client.e2e.test> // postfix/smtpd[1149]: NOQUEUE: milter-reject: MAIL from unknown[10.77.0.1]: 451 4.7.1 Service unavailable - try again later; from=<billing@sender.e2e.test> proto=ESMTP helo=<e2e-client.e2e.test>
// //
// Port 25's smtpd (postfix/inbound/smtpd) is a stranger's and has no such row. // Port 25's smtpd (postfix/inbound/smtpd) is a stranger's and has no such row.
var refusalLineRe = regexp.MustCompile(`^\S+ postfix/(?:([a-z0-9-]+)/)?smtpd\[\d+\]: NOQUEUE: milter-reject: MAIL from \S+: [45]\d\d .*?; from=<([^>]*)>(?:.*? sasl_username=<?([^>\s]*)>?)?`) var refusalLineRe = regexp.MustCompile(`^\S+ postfix/(?:([a-z0-9-]+)/)?smtpd\[\d+\]: NOQUEUE: milter-reject: MAIL from [^\[\s]*\[([^\]]+)\]: [45]\d\d .*?; from=<([^>]*)>`)
// refusalLineWait is how long a line waits for its row: the milter writes the // refusalLineWait is how long a line waits for its row: the milter writes the
// row when it refuses, which can be read a moment after the line. // row when it refuses, which can be read a moment after the line.
@@ -26,7 +28,7 @@ const refusalLineWait = 30 * time.Second
// RefusalStore is where the lines go. *store.Store satisfies it. // RefusalStore is where the lines go. *store.Store satisfies it.
type RefusalStore interface { type RefusalStore interface {
AttachRefusalLine(login, from string, at time.Time, line string) (bool, error) AttachRefusalLine(clientIP, from string, at time.Time, line string) (bool, error)
RecoverRefusal(e store.SendLogEntry, at time.Time) (bool, error) RecoverRefusal(e store.SendLogEntry, at time.Time) (bool, error)
} }
@@ -36,8 +38,10 @@ type RefusalStore interface {
// postfix/cleanup[1339]: E543D3C75DA0: milter-reject: END-OF-MESSAGE from unknown[10.77.0.1]: 5.7.1 From: director@other.e2e.test - not an address this application sends as; from=<alerts@sender.e2e.test> to=<rcpt@sink.e2e.test> proto=ESMTP helo=<e2e-client.e2e.test> // postfix/cleanup[1339]: E543D3C75DA0: milter-reject: END-OF-MESSAGE from unknown[10.77.0.1]: 5.7.1 From: director@other.e2e.test - not an address this application sends as; from=<alerts@sender.e2e.test> to=<rcpt@sink.e2e.test> proto=ESMTP helo=<e2e-client.e2e.test>
// //
// The From: line as the check printed it runs to the last " - " before the // The From: line as the check printed it runs to the last " - " before the
// semicolon; the reason after it holds neither. // first semicolon; the reason after it holds neither, and the check prints no
var fromRefusalRe = regexp.MustCompile(`^\S+ postfix/(?:[a-z0-9-]+/)?cleanup\[\d+\]: ([0-9A-Za-z]+): milter-reject: END-OF-MESSAGE from \S+: 5\.7\.1 From: (.*) - ([^;]*); from=<([^>]*)> to=<([^>]*)>`) // ";" of the client's (sender.go printable), so that semicolon is Postfix's
// and what follows it - the envelope - is Postfix's own text.
var fromRefusalRe = regexp.MustCompile(`^\S+ postfix/(?:[a-z0-9-]+/)?cleanup\[\d+\]: ([0-9A-Za-z]+): milter-reject: END-OF-MESSAGE from \S+: 5\.7\.1 From: ([^;]*) - ([^;]*); from=<([^>]*)> to=<([^>]*)>`)
// clientLoginRe is smtpd's line naming the login a queue id was submitted // clientLoginRe is smtpd's line naming the login a queue id was submitted
// under: "<id>: client=host[addr], sasl_method=PLAIN, sasl_username=<login>". // under: "<id>: client=host[addr], sasl_method=PLAIN, sasl_username=<login>".
@@ -47,8 +51,9 @@ var clientLoginRe = regexp.MustCompile(`^\S+ postfix/(?:([a-z0-9-]+)/)?smtpd\[\d
const refusalLogins = 4096 const refusalLogins = 4096
type pendingRefusal struct { type pendingRefusal struct {
login, from, line string clientIP, from, line string
at, until time.Time at, until time.Time
start Place // where the line starts in the log
} }
// pendingRecovery is a From: refusal whose row is looked for once the panel // pendingRecovery is a From: refusal whose row is looked for once the panel
@@ -56,6 +61,14 @@ type pendingRefusal struct {
type pendingRecovery struct { type pendingRecovery struct {
e store.SendLogEntry e store.SendLogEntry
at, until time.Time at, until time.Time
start Place // where its client= line - or the refusal - starts
}
// loginLine is the login a queue id was submitted under and where smtpd's
// line saying so starts.
type loginLine struct {
login string
start Place
} }
// refusalLines puts each refusal line on its row, and keeps for a while the // refusalLines puts each refusal line on its row, and keeps for a while the
@@ -65,24 +78,49 @@ type refusalLines struct {
st RefusalStore st RefusalStore
pending []pendingRefusal pending []pendingRefusal
recover []pendingRecovery recover []pendingRecovery
logins map[string]string // queue id -> login, from smtpd's client= line logins map[string]loginLine // queue id -> login, from smtpd's client= line
loginKeys []string loginKeys []string
} }
// handle looks at one mail.log line. // oldest is the earliest place in the file fp a waiting item starts at: the
func (r *refusalLines) handle(line string, now time.Time) { // tracker saves no offset past it, so a stop before the wait is over reads the
// lines again - the client= line of a refusal to recover among them.
func (r *refusalLines) oldest(fp string) (Place, bool) {
var p Place
found := false
take := func(q Place) {
if q.FP == fp && (!found || q.End < p.End) {
p, found = q, true
}
}
for _, x := range r.pending {
take(x.start)
}
for _, x := range r.recover {
take(x.start)
}
return p, found
}
// handle looks at one mail.log line, which starts at start.
func (r *refusalLines) handle(line string, now time.Time, start Place) {
at, rest := lineTime(line, now) at, rest := lineTime(line, now)
if m := clientLoginRe.FindStringSubmatch(rest); m != nil && m[1] != "inbound" { if m := clientLoginRe.FindStringSubmatch(rest); m != nil && m[1] != "inbound" {
r.remember(m[2], m[3]) r.remember(m[2], loginLine{login: m[3], start: start})
return return
} }
if m := fromRefusalRe.FindStringSubmatch(rest); m != nil { if m := fromRefusalRe.FindStringSubmatch(rest); m != nil {
from := start
login, ok := r.logins[m[1]]
if ok && login.start.FP == start.FP && login.start.End < start.End {
from = login.start
}
r.recover = append(r.recover, pendingRecovery{ r.recover = append(r.recover, pendingRecovery{
e: store.SendLogEntry{ e: store.SendLogEntry{
QueueID: m[1], AppLogin: r.logins[m[1]], From: m[4], To: m[5], Domain: domainOf(m[4]), QueueID: m[1], AppLogin: login.login, From: m[4], To: m[5], Domain: domainOf(m[4]),
Reason: store.ReasonFrom, HeaderFrom: m[2], Reason: store.ReasonFrom, HeaderFrom: m[2],
}, },
at: at, until: now.Add(refusalLineWait), at: at, until: now.Add(refusalLineWait), start: from,
}) })
return return
} }
@@ -90,7 +128,11 @@ func (r *refusalLines) handle(line string, now time.Time) {
if m == nil || m[1] == "inbound" { if m == nil || m[1] == "inbound" {
return return
} }
p := pendingRefusal{login: m[3], from: m[2], line: line, at: at, until: now.Add(refusalLineWait)} ip := net.ParseIP(strings.TrimPrefix(m[2], "IPv6:"))
if ip == nil {
return
}
p := pendingRefusal{clientIP: ip.String(), from: m[3], line: line, at: at, until: now.Add(refusalLineWait), start: start}
if !r.attach(p) { if !r.attach(p) {
r.pending = append(r.pending, p) r.pending = append(r.pending, p)
} }
@@ -127,9 +169,9 @@ func (r *refusalLines) retry(now time.Time) {
// remember keeps the login a queue id was submitted under, the oldest going // remember keeps the login a queue id was submitted under, the oldest going
// first beyond refusalLogins. // first beyond refusalLogins.
func (r *refusalLines) remember(queueID, login string) { func (r *refusalLines) remember(queueID string, login loginLine) {
if r.logins == nil { if r.logins == nil {
r.logins = make(map[string]string) r.logins = make(map[string]loginLine)
} }
if _, ok := r.logins[queueID]; !ok { if _, ok := r.logins[queueID]; !ok {
r.loginKeys = append(r.loginKeys, queueID) r.loginKeys = append(r.loginKeys, queueID)
@@ -150,7 +192,7 @@ func domainOf(addr string) string {
} }
func (r *refusalLines) attach(p pendingRefusal) bool { func (r *refusalLines) attach(p pendingRefusal) bool {
ok, err := r.st.AttachRefusalLine(p.login, p.from, p.at, p.line) ok, err := r.st.AttachRefusalLine(p.clientIP, p.from, p.at, p.line)
if err != nil { if err != nil {
log.Printf("log-tailer: refusal line: %v", err) log.Printf("log-tailer: refusal line: %v", err)
return true // not retried: the store said no for its own reason return true // not retried: the store said no for its own reason
+40 -8
View File
@@ -19,13 +19,13 @@ func (r *refusalProbe) RecoverRefusal(e store.SendLogEntry, _ time.Time) (bool,
return true, nil return true, nil
} }
func (r *refusalProbe) AttachRefusalLine(login, from string, _ time.Time, line string) (bool, error) { func (r *refusalProbe) AttachRefusalLine(clientIP, from string, _ time.Time, line string) (bool, error) {
if r.rows == 0 { if r.rows == 0 {
return false, nil return false, nil
} }
r.rows-- r.rows--
r.lines = append(r.lines, line) r.lines = append(r.lines, line)
r.who = append(r.who, login+"|"+from) r.who = append(r.who, clientIP+"|"+from)
return true, nil return true, nil
} }
@@ -42,23 +42,23 @@ func TestRefusalLines(t *testing.T) {
`2026-10-14T12:00:00+00:00 mail postfix/smtpd[11]: NOQUEUE: milter-reject: CONNECT from unknown[10.77.0.1]: 451 4.7.1 Service unavailable - try again later; proto=SMTP`, `2026-10-14T12:00:00+00:00 mail postfix/smtpd[11]: NOQUEUE: milter-reject: CONNECT from unknown[10.77.0.1]: 451 4.7.1 Service unavailable - try again later; proto=SMTP`,
`2026-10-14T12:00:00+00:00 mail postfix/smtpd[11]: NOQUEUE: reject: RCPT from unknown[10.77.0.1]: 553 5.7.1 <a@x>: Sender address rejected: not owned by user app1; from=<a@x> to=<b@y> proto=ESMTP helo=<x>`, `2026-10-14T12:00:00+00:00 mail postfix/smtpd[11]: NOQUEUE: reject: RCPT from unknown[10.77.0.1]: 553 5.7.1 <a@x>: Sender address rejected: not owned by user app1; from=<a@x> to=<b@y> proto=ESMTP helo=<x>`,
} { } {
r.handle(other, now) r.handle(other, now, Place{})
} }
if len(r.pending) != 0 { if len(r.pending) != 0 {
t.Fatalf("lines that are not an application's refusal at MAIL wait: %+v", r.pending) t.Fatalf("lines that are not an application's refusal at MAIL wait: %+v", r.pending)
} }
r.handle(limit, now) // its row is not there yet r.handle(limit, now, Place{}) // its row is not there yet
if len(r.pending) != 1 { if len(r.pending) != 1 {
t.Fatalf("pending %d, want the line waiting", len(r.pending)) t.Fatalf("pending %d, want the line waiting", len(r.pending))
} }
st.rows = 1 // the milter's row is written st.rows = 1 // the milter's row is written
r.retry(now.Add(time.Second)) r.retry(now.Add(time.Second))
if len(r.pending) != 0 || len(st.lines) != 1 || st.who[0] != "l2app|billing@sender.e2e.test" { if len(r.pending) != 0 || len(st.lines) != 1 || st.who[0] != "10.77.0.1|billing@sender.e2e.test" {
t.Fatalf("after the retry: pending %d, attached %v", len(r.pending), st.who) t.Fatalf("after the retry: pending %d, attached %v", len(r.pending), st.who)
} }
r.handle(limit, now) // no row ever comes r.handle(limit, now, Place{}) // no row ever comes
r.retry(now.Add(refusalLineWait + time.Second)) r.retry(now.Add(refusalLineWait + time.Second))
if len(r.pending) != 0 { if len(r.pending) != 0 {
t.Fatal("a line kept waiting past its time") t.Fatal("a line kept waiting past its time")
@@ -73,8 +73,8 @@ func TestFromRefusalIsRecoveredAfterTheWait(t *testing.T) {
now := time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC) now := time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
st := &refusalProbe{} st := &refusalProbe{}
r := &refusalLines{st: st} r := &refusalLines{st: st}
r.handle(`2026-10-14T12:00:00+00:00 mail postfix/smtpd[11]: E543D3C75DA0: client=unknown[10.77.0.1], sasl_method=PLAIN, sasl_username=app1`, now) r.handle(`2026-10-14T12:00:00+00:00 mail postfix/smtpd[11]: E543D3C75DA0: client=unknown[10.77.0.1], sasl_method=PLAIN, sasl_username=app1`, now, Place{})
r.handle(`2026-10-14T12:00:01+00:00 mail postfix/cleanup[13]: E543D3C75DA0: milter-reject: END-OF-MESSAGE from unknown[10.77.0.1]: 5.7.1 From: "The - Shop" <director@other.e2e.test> - it names another sender of this server; from=<alerts@sender.e2e.test> to=<rcpt@sink.e2e.test> proto=ESMTP helo=<x>`, now) r.handle(`2026-10-14T12:00:01+00:00 mail postfix/cleanup[13]: E543D3C75DA0: milter-reject: END-OF-MESSAGE from unknown[10.77.0.1]: 5.7.1 From: "The - Shop" <director@other.e2e.test> - it names another sender of this server; from=<alerts@sender.e2e.test> to=<rcpt@sink.e2e.test> proto=ESMTP helo=<x>`, now, Place{})
r.retry(now.Add(time.Second)) r.retry(now.Add(time.Second))
if len(st.recovered) != 0 { if len(st.recovered) != 0 {
t.Fatal("recovered before the panel had the time to write the row") t.Fatal("recovered before the panel had the time to write the row")
@@ -90,3 +90,35 @@ func TestFromRefusalIsRecoveredAfterTheWait(t *testing.T) {
t.Fatalf("recovered %+v, want %+v", e, want) t.Fatalf("recovered %+v, want %+v", e, want)
} }
} }
// While a line or a recovery waits, the offset saved for its file stays at the
// earliest line it needs - for a recovery the client= line that names its
// login - so a stop before the wait is over reads them again (review of stage
// 2b).
func TestWaitingLinesHoldTheSavedOffset(t *testing.T) {
now := time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
r := &refusalLines{st: &refusalProbe{}}
if _, held := r.oldest("fp"); held {
t.Fatal("nothing waits, yet the offset is held")
}
r.handle(`2026-10-14T12:00:00+00:00 mail postfix/smtpd[11]: Q1: client=unknown[10.77.0.1], sasl_method=PLAIN, sasl_username=app1`, now, Place{FP: "fp", End: 100})
r.handle(`2026-10-14T12:00:01+00:00 mail postfix/cleanup[13]: Q1: milter-reject: END-OF-MESSAGE from unknown[10.77.0.1]: 5.7.1 From: d@other.example - not an address this application sends as; from=<a@sender.example> to=<r@x> proto=ESMTP helo=<x>`, now, Place{FP: "fp", End: 300})
if p, held := r.oldest("fp"); !held || p.End != 100 {
t.Fatalf("held at %+v %v, want the client= line at 100", p, held)
}
if _, held := r.oldest("another"); held {
t.Fatal("a place in another file holds this one")
}
r.retry(now.Add(refusalLineWait + time.Second))
if _, held := r.oldest("fp"); held {
t.Fatal("still held after the wait")
}
// The tracker clamps what it saves to the hold.
cs := &captureStore{}
tr := &tracker{st: cs, path: "x", fp: "fp", hold: func(string) (Place, bool) { return Place{FP: "fp", End: 50}, true }}
tr.record(nil, 400, true)
if cs.state.Offset != 50 {
t.Fatalf("saved %d, want the hold at 50", cs.state.Offset)
}
}
+1
View File
@@ -103,6 +103,7 @@ func (s *session) recordRejected(reason string) {
AppLogin: s.login, AppLogin: s.login,
From: s.from, From: s.from,
Reason: reason, Reason: reason,
ClientIP: s.clientIP, // Postfix's line about the refusal names it
}) })
if err != nil { if err != nil {
log.Printf("journal-milter: record rejected %s: %v", s.from, err) log.Printf("journal-milter: record rejected %s: %v", s.from, err)
+5
View File
@@ -463,6 +463,11 @@ func printable(v string) string {
switch { switch {
case r == '%' || r < 0x20 || r > 0x7e: case r == '%' || r < 0x20 || r > 0x7e:
sb.WriteByte('?') sb.WriteByte('?')
case r == ';':
// Postfix logs the reply and then "; from=<...> to=<...>" of
// its own; a ";" of the client's would let its text pass for
// Postfix's where the log tailer reads that line.
sb.WriteByte(',')
default: default:
sb.WriteRune(r) sb.WriteRune(r)
} }
@@ -7,6 +7,12 @@
-- headers are judged), and its lines are found by it like any other's. -- headers are judged), and its lines are found by it like any other's.
-- DDL only; a row written earlier has '' and says so on its page. -- DDL only; a row written earlier has '' and says so on its page.
ALTER TABLE send_log ADD COLUMN refusal_line TEXT NOT NULL DEFAULT ''; ALTER TABLE send_log ADD COLUMN refusal_line TEXT NOT NULL DEFAULT '';
-- The client a refusal at MAIL FROM came from, as the milter saw it: Postfix's
-- line names the client and the envelope sender and nothing else - no login
-- - so the two together are what finds the line's row; the sender alone would
-- let one application's line land on another's row of the same sender
-- (security review of stage 2b).
ALTER TABLE send_log ADD COLUMN client_ip TEXT NOT NULL DEFAULT '';
-- A refusal row the panel did not get to write - the sender check answers 550 -- A refusal row the panel did not get to write - the sender check answers 550
-- and writes the row a moment later, and a panel stopped in between wrote -- and writes the row a moment later, and a panel stopped in between wrote
+56 -22
View File
@@ -1,20 +1,23 @@
package store package store
import ( import (
"fmt"
"testing" "testing"
"time" "time"
) )
// Postfix's line about a refusal at MAIL FROM goes on the row of that sender // Postfix's line about a refusal at MAIL FROM goes on the row of that client
// and login refused about then, the newest without a line; a queued row, a // and sender refused about then; a queued row, a row of another client or
// row of another sender and a row from long ago are not it (audit // sender, and a row from long ago are not it; a line already on a row goes on
// 2026-10-12, stage 2b). // no other; and two refusals of one second take one line each, in the order
// they were written (security and technical reviews of stage 2b).
func TestAttachRefusalLine(t *testing.T) { func TestAttachRefusalLine(t *testing.T) {
st := openTestStore(t) st := openTestStore(t)
now := time.Now().UTC() now := time.Now().UTC()
for _, e := range []SendLogEntry{ for _, e := range []SendLogEntry{
{AppLogin: "l2app", From: "billing@example.com", Reason: ReasonLimit}, {AppLogin: "l2app", From: "billing@example.com", Reason: ReasonLimit, ClientIP: "10.0.0.1"},
{AppLogin: "l2app", From: "other@example.com", Reason: ReasonLimit}, {AppLogin: "l2app", From: "billing@example.com", Reason: ReasonLimit, ClientIP: "10.0.0.1"},
{AppLogin: "other", From: "billing@example.com", Reason: ReasonClient, ClientIP: "10.0.0.2"},
} { } {
if err := st.InsertRejected(e); err != nil { if err := st.InsertRejected(e); err != nil {
t.Fatal(err) t.Fatal(err)
@@ -23,33 +26,37 @@ func TestAttachRefusalLine(t *testing.T) {
if err := st.InsertQueued(SendLogEntry{QueueID: "Q1", AppLogin: "l2app", From: "billing@example.com", To: "x@example.net"}); err != nil { if err := st.InsertQueued(SendLogEntry{QueueID: "Q1", AppLogin: "l2app", From: "billing@example.com", To: "x@example.net"}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
line := "Oct 14 12:00:00 mail postfix/smtpd[1]: NOQUEUE: milter-reject: MAIL from unknown[10.77.0.1]: 451 4.7.1 Service unavailable" first := "Oct 14 12:00:00 mail postfix/smtpd[1]: NOQUEUE: milter-reject: MAIL from unknown[10.0.0.1]: 451 4.7.1 first"
if ok, err := st.AttachRefusalLine("l2app", "Billing@example.com", now, line); err != nil || !ok { second := "Oct 14 12:00:00 mail postfix/smtpd[2]: NOQUEUE: milter-reject: MAIL from unknown[10.0.0.1]: 451 4.7.1 second"
t.Fatalf("attach = %v, %v", ok, err) for _, line := range []string{first, second} {
if ok, err := st.AttachRefusalLine("10.0.0.1", "Billing@example.com", now, line); err != nil || !ok {
t.Fatalf("attach %q = %v, %v", line, ok, err)
} }
// The row has its line now; a second line finds no row without one.
if ok, _ := st.AttachRefusalLine("l2app", "billing@example.com", now, line); ok {
t.Fatal("a second line took a row that has one")
} }
if ok, _ := st.AttachRefusalLine("l2app", "other@example.com", now.Add(-time.Hour), line); ok { // Read again after a restart: no row takes it a second time.
if ok, _ := st.AttachRefusalLine("10.0.0.1", "billing@example.com", now, first); ok {
t.Fatal("a line already on a row took another")
}
if ok, _ := st.AttachRefusalLine("10.0.0.3", "billing@example.com", now, "a third"); ok {
t.Fatal("a line of another client took a row")
}
if ok, _ := st.AttachRefusalLine("10.0.0.2", "billing@example.com", now.Add(-time.Hour), "an hour off"); ok {
t.Fatal("a line an hour off took a row") t.Fatal("a line an hour off took a row")
} }
if ok, _ := st.AttachRefusalLine("someone", "other@example.com", now, line); ok { lines := map[string]string{}
t.Fatal("a line of another login took a row")
}
rows, _ := st.QuerySendLog(SendLogFilter{AllDomains: true}, 10, 0) rows, _ := st.QuerySendLog(SendLogFilter{AllDomains: true}, 10, 0)
got := 0
for _, r := range rows { for _, r := range rows {
full, _ := st.GetSendLog(r.ID) full, _ := st.GetSendLog(r.ID)
if full.RefusalLine != "" { if full.RefusalLine != "" {
got++ lines[fmt.Sprint(full.ID)] = full.RefusalLine
if full.From != "billing@example.com" || full.Status != StatusRejected {
t.Errorf("the line went to %+v", full)
} }
} }
if len(lines) != 2 {
t.Fatalf("lines on rows: %v, want the first two rows", lines)
} }
if got != 1 { r1, _ := st.GetSendLog(rows[len(rows)-1].ID) // the oldest row
t.Fatalf("%d rows have a line, want 1", got) if r1.RefusalLine != first {
t.Fatalf("the first row has %q, want the first line", r1.RefusalLine)
} }
} }
@@ -113,3 +120,30 @@ func TestRecoverRefusal(t *testing.T) {
t.Fatalf("the recovered row: %+v", found) t.Fatalf("the recovered row: %+v", found)
} }
} }
// A recovered row whose login the tailer did not see is its envelope
// domain's, not nobody's (review of stage 2b).
func TestRecoverRefusalWithoutALogin(t *testing.T) {
st := openTestStore(t)
d := addTestDomain(t, st, "sender.example")
e := SendLogEntry{QueueID: "NOLOGIN", From: "a@sender.example", To: "r@x", Domain: "sender.example", Reason: ReasonFrom}
if wrote, err := st.RecoverRefusal(e, time.Now()); err != nil || !wrote {
t.Fatalf("recover = %v, %v", wrote, err)
}
rows, _ := st.QuerySendLog(SendLogFilter{DomainIDs: []int64{d.ID}}, 10, 0)
if len(rows) != 1 {
t.Fatalf("the domain's log has %d rows, want the recovered one", len(rows))
}
}
// A refused row carries the queue id it was given; a delivery line of a later
// message with that id does not touch it.
func TestUpdateStatusLeavesARefusalAlone(t *testing.T) {
st := openTestStore(t)
if err := st.InsertRejected(SendLogEntry{QueueID: "REUSED", To: "a@example.net", Reason: ReasonFrom}); err != nil {
t.Fatal(err)
}
if n, err := st.UpdateStatus("REUSED", "a@example.net", StatusSent); err != nil || n != 0 {
t.Fatalf("UpdateStatus = %d, %v; want 0 rows", n, err)
}
}
+1 -1
View File
@@ -47,7 +47,7 @@ func TestBaselineSchema(t *testing.T) {
"applications": "id domain_id login address_mode auth_ip_restrict auth_allowed_ips created_at", "applications": "id domain_id login address_mode auth_ip_restrict auth_allowed_ips created_at",
"application_addresses": "id application_id address", "application_addresses": "id application_id address",
"rate_limits": "id scope ref_id max_messages window_seconds mode auto_multiplier auto_updated_at", "rate_limits": "id scope ref_id max_messages window_seconds mode auto_multiplier auto_updated_at",
"send_log": "id queue_id domain app_login from_addr to_addr subject status created_at updated_at reason header_from domain_id refusal_line recovered", "send_log": "id queue_id domain app_login from_addr to_addr subject status created_at updated_at reason header_from domain_id refusal_line client_ip recovered",
"logtail_state": "path fingerprint read_offset updated_at", "logtail_state": "path fingerprint read_offset updated_at",
"inbound_domains": "id name recipient_mode created_at", "inbound_domains": "id name recipient_mode created_at",
"inbound_transports": "inbound_domain_id host port tls_mode", "inbound_transports": "inbound_domain_id host port tls_mode",
+33 -19
View File
@@ -38,6 +38,9 @@ type SendLogEntry struct {
// was refused (Reason*), and for ReasonFrom what its From: line said. // was refused (Reason*), and for ReasonFrom what its From: line said.
Reason string Reason string
HeaderFrom string HeaderFrom string
// ClientIP is the client a refusal at MAIL FROM came from (InsertRejected
// only), which Postfix's line about it names.
ClientIP string
} }
// loginDomainID is the domain_id a row is written with: the domain of the // loginDomainID is the domain_id a row is written with: the domain of the
@@ -73,9 +76,9 @@ func (s *Store) InsertRejected(e SendLogEntry) error {
now := time.Now().UTC().Format(time.RFC3339) now := time.Now().UTC().Format(time.RFC3339)
_, err := s.db.Exec( _, err := s.db.Exec(
`INSERT INTO send_log `INSERT INTO send_log
(queue_id, domain, app_login, from_addr, to_addr, subject, status, reason, header_from, created_at, updated_at, domain_id) (queue_id, domain, app_login, from_addr, to_addr, subject, status, reason, header_from, client_ip, created_at, updated_at, domain_id)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, `+loginDomainID+`)`, VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, `+loginDomainID+`)`,
e.QueueID, e.Domain, e.AppLogin, e.From, e.To, e.Subject, StatusRejected, e.Reason, e.HeaderFrom, now, now, e.AppLogin, e.QueueID, e.Domain, e.AppLogin, e.From, e.To, e.Subject, StatusRejected, e.Reason, e.HeaderFrom, e.ClientIP, now, now, e.AppLogin,
) )
if err != nil { if err != nil {
return fmt.Errorf("insert rejected send_log: %w", err) return fmt.Errorf("insert rejected send_log: %w", err)
@@ -113,7 +116,9 @@ func (s *Store) UpdateStatus(queueID, recipient, status string) (int64, error) {
return 0, fmt.Errorf("update send_log status: %w", err) return 0, fmt.Errorf("update send_log status: %w", err)
} }
defer tx.Rollback() defer tx.Rollback()
const newest = `id = (SELECT MAX(id) FROM send_log WHERE queue_id = ? AND to_addr = ? COLLATE NOCASE)` // A refused row carries the queue id it was given, and that id may be the
// next message's: a delivery line is never about a refusal.
const newest = `id = (SELECT MAX(id) FROM send_log WHERE queue_id = ? AND to_addr = ? COLLATE NOCASE AND status <> 'rejected')`
marked, err := tx.Exec( marked, err := tx.Exec(
`UPDATE send_log SET reason = ? WHERE `+newest+` AND reason = ?`, `UPDATE send_log SET reason = ? WHERE `+newest+` AND reason = ?`,
ReasonReconcileContradicted, queueID, recipient, ReasonReconcile, ReasonReconcileContradicted, queueID, recipient, ReasonReconcile,
@@ -235,22 +240,27 @@ type SendLogRow struct {
// AttachRefusalLine puts Postfix's line about a refusal at MAIL FROM on the // AttachRefusalLine puts Postfix's line about a refusal at MAIL FROM on the
// row the journal milter wrote for it (audit 2026-10-12, stage 2b). Such a // row the journal milter wrote for it (audit 2026-10-12, stage 2b). Such a
// refusal has no queue id, so the row is found by what the line says: a // refusal has no queue id, and the line names no login, so the row is found
// refused row of that sender - and login, when the line names one - without a // by what the line says: a refused row of that client and sender without a
// line yet, written within two minutes of the line's time; the newest when // line yet, written within two minutes of the line's time; the nearest in
// there are several. It reports whether a row took the line. // time, the oldest of equals, when there are several - lines are read in the
func (s *Store) AttachRefusalLine(login, from string, at time.Time, line string) (bool, error) { // order the rows were written, so two refusals of one second take one line
// each in turn (review of stage 2b). A line already on a row - read again
// after a restart - goes on no other. It reports whether a row took the line.
func (s *Store) AttachRefusalLine(clientIP, from string, at time.Time, line string) (bool, error) {
res, err := s.db.Exec( res, err := s.db.Exec(
`UPDATE send_log SET refusal_line = ? WHERE id = ( `UPDATE send_log SET refusal_line = ?1 WHERE id = (
SELECT id FROM send_log SELECT id FROM send_log
WHERE status = ? AND queue_id = '' AND refusal_line = '' WHERE status = ?2 AND queue_id = '' AND refusal_line = ''
AND reason IN ('', ?, ?) AND from_addr = ? COLLATE NOCASE AND reason IN ('', ?3, ?4) AND from_addr = ?5 COLLATE NOCASE AND client_ip = ?6
AND (? = '' OR app_login = ?) AND created_at BETWEEN ?7 AND ?8
AND created_at BETWEEN ? AND ? ORDER BY ABS(julianday(created_at) - julianday(?9)), id
ORDER BY id DESC LIMIT 1)`, LIMIT 1)
line, StatusRejected, ReasonLimit, ReasonClient, from, login, login, AND NOT EXISTS (SELECT 1 FROM send_log WHERE refusal_line = ?1)`,
line, StatusRejected, ReasonLimit, ReasonClient, from, clientIP,
at.Add(-refusalLineWindow).UTC().Format(time.RFC3339), at.Add(-refusalLineWindow).UTC().Format(time.RFC3339),
at.Add(refusalLineWindow).UTC().Format(time.RFC3339), at.Add(refusalLineWindow).UTC().Format(time.RFC3339),
at.UTC().Format(time.RFC3339),
) )
if err != nil { if err != nil {
return false, fmt.Errorf("attach refusal line: %w", err) return false, fmt.Errorf("attach refusal line: %w", err)
@@ -264,16 +274,20 @@ func (s *Store) AttachRefusalLine(login, from string, at time.Time, line string)
// wrote the row itself (audit 2026-10-12, stage 2b): the check answers 550 // wrote the row itself (audit 2026-10-12, stage 2b): the check answers 550
// and writes the row through a buffer a moment later. The message had a queue // and writes the row through a buffer a moment later. The message had a queue
// id; a refused row with that id from around at means the row was written, // id; a refused row with that id from around at means the row was written,
// and nothing is. It reports whether it wrote one. // and nothing is. The domain is the login's, or - when the tailer did not see the login, its
// client= line read before a restart - the envelope's, so the row is still
// its domain's administrator's (review of stage 2b). It reports whether it
// wrote one.
func (s *Store) RecoverRefusal(e SendLogEntry, at time.Time) (bool, error) { func (s *Store) RecoverRefusal(e SendLogEntry, at time.Time) (bool, error) {
stamp := at.UTC().Format(time.RFC3339) stamp := at.UTC().Format(time.RFC3339)
res, err := s.db.Exec( res, err := s.db.Exec(
`INSERT INTO send_log `INSERT INTO send_log
(queue_id, domain, app_login, from_addr, to_addr, subject, status, reason, header_from, recovered, created_at, updated_at, domain_id) (queue_id, domain, app_login, from_addr, to_addr, subject, status, reason, header_from, recovered, created_at, updated_at, domain_id)
SELECT ?, ?, ?, ?, ?, '', ?, ?, ?, 1, ?, ?, `+loginDomainID+` SELECT ?, ?, ?, ?, ?, '', ?, ?, ?, 1, ?, ?,
COALESCE(`+loginDomainID+`, (SELECT d.id FROM domains d WHERE d.name = ?))
WHERE NOT EXISTS (SELECT 1 FROM send_log WHERE NOT EXISTS (SELECT 1 FROM send_log
WHERE queue_id = ? AND status = ? AND created_at BETWEEN ? AND ?)`, WHERE queue_id = ? AND status = ? AND created_at BETWEEN ? AND ?)`,
e.QueueID, e.Domain, e.AppLogin, e.From, e.To, StatusRejected, e.Reason, e.HeaderFrom, stamp, stamp, e.AppLogin, e.QueueID, e.Domain, e.AppLogin, e.From, e.To, StatusRejected, e.Reason, e.HeaderFrom, stamp, stamp, e.AppLogin, e.Domain,
e.QueueID, StatusRejected, e.QueueID, StatusRejected,
at.Add(-refusalLineWindow).UTC().Format(time.RFC3339), at.Add(refusalLineWindow).UTC().Format(time.RFC3339), at.Add(-refusalLineWindow).UTC().Format(time.RFC3339), at.Add(refusalLineWindow).UTC().Format(time.RFC3339),
) )
@@ -273,6 +273,9 @@ func (h *Handlers) deliveryLog(row store.SendLogRow) ([]view.LogLine, string) {
return nil, "Could not read the mail log." return nil, "Could not read the mail log."
} }
} }
if len(lines) == 0 && row.Status == store.StatusRejected {
return nil, "Postfix's lines about this refusal are not stored: the message was refused before version 2.5.1, or the lines were never read."
}
if len(lines) == 0 { if len(lines) == 0 {
return nil, "Nothing is stored for this queue id and nothing is in the current mail log. Lines are kept with the row since version 2.2.0; a message accepted before that had them only in mail.log, which has since rotated." return nil, "Nothing is stored for this queue id and nothing is in the current mail log. Lines are kept with the row since version 2.2.0; a message accepted before that had them only in mail.log, which has since rotated."
} }
+2 -2
View File
@@ -162,12 +162,12 @@ func TestARefusedMessageShowsPostfixsLine(t *testing.T) {
d, _ := st.AddDomain("example.org", "mail") d, _ := st.AddDomain("example.org", "mail")
st.AddApplication(d.ID, "alerts", store.AddressModeWildcard, nil) st.AddApplication(d.ID, "alerts", store.AddressModeWildcard, nil)
for _, subject := range []string{"with its line", "from before"} { for _, subject := range []string{"with its line", "from before"} {
if err := st.InsertRejected(store.SendLogEntry{Domain: "example.org", AppLogin: "alerts", From: "alerts@example.org", Subject: subject, Reason: store.ReasonLimit}); err != nil { if err := st.InsertRejected(store.SendLogEntry{Domain: "example.org", AppLogin: "alerts", From: "alerts@example.org", Subject: subject, Reason: store.ReasonLimit, ClientIP: "198.51.100.7"}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if subject == "with its line" { if subject == "with its line" {
line := time.Now().UTC().Format("2006-01-02T15:04:05Z") + " mail postfix/smtpd[2131]: NOQUEUE: milter-reject: MAIL from unknown[198.51.100.7]: 451 4.7.1 Service unavailable - try again later; from=<alerts@example.org> proto=ESMTP helo=<app>" line := time.Now().UTC().Format("2006-01-02T15:04:05Z") + " mail postfix/smtpd[2131]: NOQUEUE: milter-reject: MAIL from unknown[198.51.100.7]: 451 4.7.1 Service unavailable - try again later; from=<alerts@example.org> proto=ESMTP helo=<app>"
if ok, err := st.AttachRefusalLine("alerts", "alerts@example.org", time.Now(), line); err != nil || !ok { if ok, err := st.AttachRefusalLine("198.51.100.7", "alerts@example.org", time.Now(), line); err != nil || !ok {
t.Fatalf("attach = %v, %v", ok, err) t.Fatalf("attach = %v, %v", ok, err)
} }
} }
+2 -2
View File
@@ -171,14 +171,14 @@ var outboundTopics = []HelpTopic{
Code("From:"), " - another of your domains, the server's own name, an address somewhere else, or none at all - is refused for good with ", Code("From:"), " - another of your domains, the server's own name, an address somewhere else, or none at all - is refused for good with ",
Code("550 5.7.1"), ". It is a rejected row of the Outbound log, and its page says which address was refused. This is what keeps one application from getting mail signed with another domain's key.", Code("550 5.7.1"), ". It is a rejected row of the Outbound log, and its page says which address was refused. This is what keeps one application from getting mail signed with another domain's key.",
Br(), Br(), Br(), Br(),
"The check is strict. While the panel does not answer, Postfix does not accept mail unchecked: it replies with a temporary error and the application sends again later. Server › Health counts what that costs: for how long the panel did not answer while Postfix itself was running; how many connections Postfix turned away because of it - it refuses from the connection on, before a message is offered, so a client that tries three times is three and a monitor that only connects is one; and how many messages were deferred in the middle, when the panel stopped answering after a message had begun. A time marked ≈ includes a start at which supervisord could not say since when Postfix was running: the time since the panel's last sign of life is counted, at most what Postfix was up. The card warns when a connection was turned away or a message deferred in the last 24 hours. If the numbers stay at nothing, the check is where it should be; if they do not, it belongs in a process of its own."), "The check is strict. While the panel does not answer, Postfix does not accept mail unchecked: it replies with a temporary error and the application sends again later. Server › Health counts what that costs: for how long the panel did not answer while Postfix itself was running; how many connections Postfix turned away because of it - it refuses from the connection on, before a message is offered, so a client that tries three times is three and a monitor that only connects is one; and how many messages were deferred in the middle, when the panel stopped answering after a message had begun. A time marked ≈ includes a start at which supervisord could not say since when Postfix was running: the time since the panel's last sign of life is counted, which may be more than Postfix was up. The card warns when a connection was turned away or a message deferred in the last 24 hours. If the numbers stay at nothing, the check is where it should be; if they do not, it belongs in a process of its own."),
}}, }},
{ID: HelpLimits, Title: "Rate limits", Body: []Text{ {ID: HelpLimits, Title: "Rate limits", Body: []Text{
Rich("Level 1 is a per-IP ceiling set in ", Code(".env"), ". Level 2 is optional, per domain and per application, entered by hand or computed from the last 30 days of sending, and can never exceed level 1. An application's own limit replaces its domain's."), Rich("Level 1 is a per-IP ceiling set in ", Code(".env"), ". Level 2 is optional, per domain and per application, entered by hand or computed from the last 30 days of sending, and can never exceed level 1. An application's own limit replaces its domain's."),
}}, }},
{ID: HelpLog, Title: "Outbound log", Body: []Text{ {ID: HelpLog, Title: "Outbound log", Body: []Text{
Rich("One row per message and recipient, with the status Postfix last reported: queued (accepted, nothing reported yet), sent (the receiving server took it; what the mailbox does next is not reported back), deferred (not taken yet, Postfix tries again), bounced (failed for good, or the queue lifetime ran out; or closed by the panel when Postfix no longer held the message and ", Code("mail.log"), " never said what became of it) and rejected (refused before queueing, so it never reached the queue: under a level-2 rate limit, from a client address the application does not allow, or for a ", Code("From:"), " line that is not the application's - the message's page says which). A message's page lists these steps and the lines Postfix wrote about it in ", Rich("One row per message and recipient, with the status Postfix last reported: queued (accepted, nothing reported yet), sent (the receiving server took it; what the mailbox does next is not reported back), deferred (not taken yet, Postfix tries again), bounced (failed for good, or the queue lifetime ran out; or closed by the panel when Postfix no longer held the message and ", Code("mail.log"), " never said what became of it) and rejected (refused before queueing, so it never reached the queue: under a level-2 rate limit, from a client address the application does not allow, or for a ", Code("From:"), " line that is not the application's - the message's page says which). A message's page lists these steps and the lines Postfix wrote about it in ",
Code("mail.log"), ", found by queue id. The lines are kept with the row: they stay on the page after that file has rotated, for as long as the row does. A message accepted before version 2.2.0 shows them only while they are still in the current file. Rows are deleted after the retention set in Server › Settings, 90 days by default, and their lines with them. A domain administrator sees only the rows of their own domains."), Code("mail.log"), ", found by queue id - and for a message refused before it had one, Postfix's one line about the refusal. The lines are kept with the row: they stay on the page after that file has rotated, for as long as the row does. A message accepted before version 2.2.0 shows them only while they are still in the current file. Rows are deleted after the retention set in Server › Settings, 90 days by default, and their lines with them. A domain administrator sees only the rows of their own domains."),
}}, }},
{ID: HelpQueue, Title: "Queue and retries", Body: []Text{ {ID: HelpQueue, Title: "Queue and retries", Body: []Text{
Rich("The page lists, as ", Code("postqueue -p"), " prints it, the messages Postfix has accepted and not yet delivered: mail being sent right now and mail deferred for a later attempt. Only the global role can open it, and it only shows; nothing here deletes or forces a message. A deferred message is tried again after the first retry delay, then at gaps that double up to a cap, until the receiving server takes it or the queue lifetime runs out and it bounces; there is no attempt counter, and the box on the page shows this server's values. A queue that keeps growing means a receiver is refusing or not answering: open the message from the Outbound log to read the reason Postfix logged."), Rich("The page lists, as ", Code("postqueue -p"), " prints it, the messages Postfix has accepted and not yet delivered: mail being sent right now and mail deferred for a later attempt. Only the global role can open it, and it only shows; nothing here deletes or forces a message. A deferred message is tried again after the first retry delay, then at gaps that double up to a cap, until the receiving server takes it or the queue lifetime runs out and it bounces; there is no attempt counter, and the box on the page shows this server's values. A queue that keeps growing means a receiver is refusing or not answering: open the message from the Outbound log to read the reason Postfix logged."),
+1 -1
View File
@@ -265,7 +265,7 @@ func NewOutMessage(m Meta, in MessageInput) *OutMessage {
facts = append(facts, Fact{Label: "Written", Value: Rich("from ", Code("mail.log"), " after a restart: the panel stopped before it wrote the refusal")}) facts = append(facts, Fact{Label: "Written", Value: Rich("from ", Code("mail.log"), " after a restart: the panel stopped before it wrote the refusal")})
} }
if in.QueueID != "" { if in.QueueID != "" {
facts = append(facts, Fact{Label: "Queue id", Value: Rich(Mono(in.QueueID), " — given, the message was not queued")}) facts = append(facts, Fact{Label: "Queue id", Value: Rich(Mono(in.QueueID), " - given, the message was not queued")})
} else { } else {
facts = append(facts, Fact{Label: "Queue id", Value: Rich("none - refused at ", Mono("MAIL FROM"), ", before one is given")}) facts = append(facts, Fact{Label: "Queue id", Value: Rich("none - refused at ", Mono("MAIL FROM"), ", before one is given")})
delivery.End = Rich("The line from ", Code("mail.log"), " for this refusal") delivery.End = Rich("The line from ", Code("mail.log"), " for this refusal")
+1 -1
View File
@@ -144,7 +144,7 @@ func TestOutMessageRefusedForItsFromLine(t *testing.T) {
out := renderSignedIn(t, "out-message", p) out := renderSignedIn(t, "out-message", p)
pageHas(t, "Message", out, `<span class="tag is-danger is-light">wrong From</span>`, pageHas(t, "Message", out, `<span class="tag is-danger is-light">wrong From</span>`,
`<dt>From: line</dt><dd class="sp-mono sp-small">director@example.org</dd>`, `<dt>Refused</dt>`, `<dt>From: line</dt><dd class="sp-mono sp-small">director@example.org</dd>`, `<dt>Refused</dt>`,
`<span class="sp-mono">7BCAF3C75DA0</span> — given, the message was not queued`, `href="/help#sender-check"`, `<span class="sp-mono">7BCAF3C75DA0</span> - given, the message was not queued`, `href="/help#sender-check"`,
`Lines from <code>mail.log</code> with this queue id`, `<span class="sp-e">postfix/cleanup[2120]: 7BCAF3C75DA0: milter-reject`) `Lines from <code>mail.log</code> with this queue id`, `<span class="sp-e">postfix/cleanup[2120]: 7BCAF3C75DA0: milter-reject`)
} }
+28 -5
View File
@@ -349,10 +349,18 @@ func testRefusalsShowPostfixsLine(t *testing.T, sc *scenario) {
rowRe := regexp.MustCompile(`(?s)<tr>.*?</tr>`) rowRe := regexp.MustCompile(`(?s)<tr>.*?</tr>`)
tagRe := regexp.MustCompile(`<span class="tag[^"]*">(wrong From|rate-limited|wrong client)</span>`) tagRe := regexp.MustCompile(`<span class="tag[^"]*">(wrong From|rate-limited|wrong client)</span>`)
linkRe := regexp.MustCompile(`href="(/outbound/log/\d+)[^"]*"`) linkRe := regexp.MustCompile(`href="(/outbound/log/\d+)[^"]*"`)
// Each with the refusal's own sender in Postfix's line: the line of one
// application's refusal must not land on another's row (review of
// stage 2b; the rate-limited refusals are l2app's, from billing@).
want := map[string]string{ want := map[string]string{
"wrong From": "milter-reject: END-OF-MESSAGE", "wrong From": "milter-reject: END-OF-MESSAGE",
"rate-limited": "NOQUEUE: milter-reject: MAIL from", "rate-limited": "NOQUEUE: milter-reject: MAIL from unknown[",
"wrong client": "NOQUEUE: milter-reject: MAIL from", "wrong client": "NOQUEUE: milter-reject: MAIL from unknown[",
}
sender := map[string]string{
"wrong From": "",
"rate-limited": "from=&lt;billing@" + senderDomain + "&gt;",
"wrong client": "from=&lt;alerts@" + senderDomain + "&gt;",
} }
for tag, line := range want { for tag, line := range want {
if err := waitFor("the page of a "+tag+" refusal to show Postfix's line", 30*time.Second, time.Second, func() (bool, error) { if err := waitFor("the page of a "+tag+" refusal to show Postfix's line", 30*time.Second, time.Second, func() (bool, error) {
@@ -370,7 +378,7 @@ func testRefusalsShowPostfixsLine(t *testing.T, sc *scenario) {
if err != nil { if err != nil {
return false, err return false, err
} }
if strings.Contains(page, line) { if strings.Contains(page, line) && strings.Contains(page, sender[tag]) {
return true, nil return true, nil
} }
} }
@@ -397,6 +405,14 @@ func asPanel(code string, args ...string) (string, error) {
// saved place moved back before the message's first line. Started again, the // saved place moved back before the message's first line. Started again, the
// panel writes the row from mail.log, marked, and once. // panel writes the row from mail.log, marked, and once.
func testRefusalRecoveredFromTheLog(t *testing.T, sc *scenario) { func testRefusalRecoveredFromTheLog(t *testing.T, sc *scenario) {
// A refusal whose row the panel writes as usual: it must not get a second
// row from the log once the wait is over.
written := uniqueToken("written")
if res := attemptSend(sendAttempt{authLogin: sc.appLogin, authPassword: sc.appPassword,
from: "alerts@" + senderDomain, headerFrom: "director@" + otherDomain, to: recipient,
subject: "e2e " + written, body: "refused, its row written"}); res.refusedAt("DATA", 550, "5.7.1 From:") != nil {
t.Fatalf("the written refusal: %v", res.firstErr())
}
token := uniqueToken("recover") token := uniqueToken("recover")
res := attemptSend(sendAttempt{authLogin: sc.appLogin, authPassword: sc.appPassword, res := attemptSend(sendAttempt{authLogin: sc.appLogin, authPassword: sc.appPassword,
from: "alerts@" + senderDomain, headerFrom: "director@" + otherDomain, to: recipient, from: "alerts@" + senderDomain, headerFrom: "director@" + otherDomain, to: recipient,
@@ -454,9 +470,16 @@ print(sqlite3.connect("/data/selfpost.db").execute("select count(*), coalesce(su
}); err != nil { }); err != nil {
t.Fatal(err) t.Fatal(err)
} }
// Once: the next passes over the log write it no more. // Once: past another wait of the tailer's (30 s), the passes over the log
time.Sleep(3 * time.Second) // write it no more, and the refusal the panel wrote itself is still one
// row, not recovered.
time.Sleep(35 * time.Second)
if out, _ := count(); out != "(1, 1)" { if out, _ := count(); out != "(1, 1)" {
t.Fatalf("after more passes the rows are %s, want (1, 1)", out) t.Fatalf("after more passes the rows are %s, want (1, 1)", out)
} }
out, err := asPanel(`import sqlite3,sys
print(sqlite3.connect("/data/selfpost.db").execute("select count(*), coalesce(sum(recovered),0) from send_log where subject like ?",("%"+sys.argv[1]+"%",)).fetchone())`, written)
if err != nil || out != "(1, 0)" {
t.Fatalf("the refusal the panel wrote: rows (count, recovered) = %s, %v; want (1, 0)", out, err)
}
} }