mix/selfpost
1
0
Fork 0
Code Issues Pull Requests Actions 21 Packages Projects Releases Wiki Activity

from-header-binding: the sender check, what it costs, its pages; guide, architecture, security
test / test (push) Canceled after 0s
Details

Browse Source 
A message's From: line is held against the application that sends it, under
the rule Postfix holds the envelope sender to. OpenDKIM chooses the key by
that line, so until now an application could have mail signed as another
domain of the server or as the server itself.

The check (internal/milter/sender.go)
- A role of the panel on a socket of its own (SENDER_CHECK_SOCKET), first in
  smtpd_milters and strict: default_action=tempfail. The journal milter keeps
  its socket and stays fail-open. postfix-wrapper waits for the new socket.
- It reads no database: what an application may send as comes from Postfix's
  own sender_login_maps, what the server signs for from OpenDKIM's
  SigningTable. Both are read again when they change; one that cannot be
  read defers mail.
- One From: field, one address, allowed by the envelope's lookup; then no
  other name of the SigningTable may appear anywhere in the raw line.
  Refused: 550 5.7.1 naming the line, reduced to printable ASCII.
- A refusal is a send-log row with reason "from" and the header address
  (migration 0006), written by another goroutine and not waited for. The
  journal's refusals record their reason too (limit, client), and the
  notifier takes only a limit's.

What it costs (docs/plans/from-header-binding.md, Statistics)
- The role marks itself alive every 2 s; at start the gap since the mark is
  downtime, counted from Postfix's start if later and not at all when
  supervisord says Postfix is not running.
- The log tailer counts the connections Postfix turned away: a warning that
  names the socket, then milter-reject "Service unavailable" from the same
  smtpd. Seen on the stand: the client is refused from CONNECT on and leaves
  at AUTH (454), so no MAIL line is ever written.
- sender_check_days, a row per UTC day, kept a year.

Pages
- Health: the card "Sender check", the socket's row, and the cards layout -
  boxes numbered by importance in two stacks read in rows; the left stack
  polls, the three readings of the right one are swapped in by id.
- Outbound log: a refused row says why in its tag (wrong From, wrong client,
  rate-limited); the message's page shows the From: line and what the
  application may send as.
- Help: topic "Sender check", word for word the mockup's.

Found by the stand, fixed here: the names for the second look came from the
sender bindings, and a domain with a key and no application yet was not
among them.

e2e: negative_journal_milter_fail_open is gone - it proved that mail passes
with the panel down, which is no longer so. In its place
sender_check_holds_the_from_line (another domain's From:, the hostname's, a
stranger's, an address off the list, another sender in the display name -
refused; own addresses - delivered) and panel_down_defers_mail (4xx with the
panel stopped, the container alive, mail again after, downtime and one
deferred message on Health, the card warning).

Test server, this tree before the last cosmetic edits (a fixture row, a
fact's size, docs): go vet ok, go test 23 packages ok, make e2e ok (337 s).
Locally after them: internal/web/view ok.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
mix
2026-10-12 19:35:43 +03:00
co-authored by Claude Opus 5.5
parent 8542e673a8
commit c171fb5bc7
54 changed files with 2237 additions and 336 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/logtail/logtail.go
+12 -2
View File
@@ -114,8 +114,10 @@ func parseDelivery(line string) (queueID, recipient, status string, ok bool) {
// for the message's page, except those of mail that came in on port 25
// (lines.go). With inbound set it also writes the
// inbound journal's rows for refused mail and prunes that journal
// (inbound.go). It returns nil on a clean shutdown.
func Run(ctx context.Context, path string, st StatusStore, retention RetentionDays, inbound *Inbound) error {
// (inbound.go), and with sender set it counts the mail Postfix deferred
// because the sender check did not answer (sendercheck.go). It returns nil on
// a clean shutdown.
func Run(ctx context.Context, path string, st StatusStore, retention RetentionDays, inbound *Inbound, sender *SenderCheck) error {
go retentionLoop(ctx, st, retention)
var journal *inboundJournal
if inbound != nil {
@@ -130,6 +132,11 @@ func Run(ctx context.Context, path string, st StatusStore, retention RetentionDa
caughtUp := make(chan struct{})
go reconcileLoop(ctx, st, caughtUp)
var deferred *deferredCounter
if sender != nil {
deferred = newDeferredCounter(sender)
}
keeper := newLineKeeper(st)
return follow(ctx, path, &tracker{st: st, path: path}, caughtUp, keeper.flush, func(line string) {
// Until the first read reaches end-of-file the lines are those written
@@ -145,6 +152,9 @@ func Run(ctx context.Context, path string, st StatusStore, retention RetentionDa
if journal != nil {
journal.handle(line, now, catchingUp)
}
if deferred != nil {
deferred.handle(line, now)
}
keeper.add(line, now, catchingUp)
queueID, recipient, status, ok := parseDelivery(line)
if !ok {
internal/logtail/logtail_test.go
+2 -2
View File
@@ -264,7 +264,7 @@ func TestFollowTailsAndRotates(t *testing.T) {
cs := &captureStore{}
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { done <- Run(ctx, path, cs, func() int { return 90 }, nil) }()
go func() { done <- Run(ctx, path, cs, func() int { return 90 }, nil, nil) }()
// Give follow() time to open at EOF (it seeks to end immediately on start,
// so the seed line above is ignored), then append a delivery line.
@@ -347,7 +347,7 @@ func startRun(t *testing.T, path string, cs *captureStore) func() {
t.Helper()
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { done <- Run(ctx, path, cs, func() int { return 90 }, nil) }()
go func() { done <- Run(ctx, path, cs, func() int { return 90 }, nil, nil) }()
// follow() opens and seeks on start; give it a moment before the caller
// appends, so the append is not raced by the initial open.
time.Sleep(50 * time.Millisecond)
internal/logtail/sendercheck.go
+90
View File
@@ -0,0 +1,90 @@
package logtail
import (
"log"
"regexp"
"strings"
"time"
)
// The sender check is attached to Postfix strictly: while the panel does not
// answer on its socket, mail is deferred (docs/plans/from-header-binding.md).
// How many submissions that happened to is read here, from what Postfix
// writes about it, and shown on Server > Health. It is the figure that also
// sees a panel which is running and not answering.
//
// Postfix says it in lines of one smtpd process - why, then what:
//
// postfix/smtpd[1183]: warning: connect to Milter service unix:/run/selfpost/sender.sock: No such file or directory
// postfix/smtpd[1183]: NOQUEUE: milter-reject: CONNECT from unknown[10.77.0.1]: 451 4.7.1 Service unavailable - try again later; proto=SMTP
// postfix/smtpd[1183]: NOQUEUE: milter-reject: EHLO from unknown[10.77.0.1]: 451 4.7.1 Service unavailable - try again later; proto=SMTP helo=<app>
//
// and the client, told "454 4.3.0 Try again later" at AUTH, goes away without
// a line of its own. So what is counted is the connection Postfix turned
// away, once: a client brings a message when it connects, and it will bring
// it again.
//
// The reject lines do not name the milter, and the same words are written
// when OpenDKIM does not answer or when the journal milter refuses under a
// level-2 limit. So a process is remembered from the warning that names this
// socket, and counted at its next reject line.
// SenderCheckStore is where the count goes. *store.Store satisfies it.
type SenderCheckStore interface {
AddSenderCheckDeferred(at time.Time, n int64) error
}
// SenderCheck turns the counting on: the store and the socket Postfix was
// given for the check (SENDER_CHECK_SOCKET), as a path.
type SenderCheck struct {
Store SenderCheckStore
Socket string
}
var smtpdLineRe = regexp.MustCompile(`^\S+ postfix/(?:[a-z0-9-]+/)?smtpd\[(\d+)\]: (.*)$`)
const (
milterUnavailable = "Service unavailable - try again later"
// senderCheckPids bounds the processes remembered; smtpd processes are a
// few dozen, and a log that names more is not kept track of.
senderCheckPids = 1024
)
// deferredCounter is the per-run state of the count.
type deferredCounter struct {
st SenderCheckStore
needle string // the socket as Postfix names it in a warning
failed map[string]bool // smtpd pid -> the check did not answer its current client
}
func newDeferredCounter(sc *SenderCheck) *deferredCounter {
return &deferredCounter{st: sc.Store, needle: "unix:" + sc.Socket, failed: make(map[string]bool)}
}
// handle looks at one mail.log line.
func (c *deferredCounter) handle(line string, now time.Time) {
at, rest := lineTime(line, now)
m := smtpdLineRe.FindStringSubmatch(rest)
if m == nil {
return
}
pid, text := m[1], m[2]
switch {
case strings.HasPrefix(text, "connect from "):
delete(c.failed, pid)
case strings.HasPrefix(text, "warning: ") && strings.Contains(text, c.needle) && strings.Contains(strings.ToLower(text), "milter"):
// "connect to Milter service <socket>: …" when nobody listens, and
// "milter <socket>: can't read … reply packet header: …" when the
// check stopped answering in the middle of a message.
if len(c.failed) >= senderCheckPids {
c.failed = make(map[string]bool)
}
c.failed[pid] = true
case c.failed[pid] && strings.Contains(text, "milter-reject: ") && strings.Contains(text, milterUnavailable):
// Once for the connection, however many commands the client tries.
delete(c.failed, pid)
if err := c.st.AddSenderCheckDeferred(at, 1); err != nil {
log.Printf("log-tailer: sender check statistics: %v", err)
}
}
}
internal/logtail/sendercheck_test.go
+63
View File
@@ -0,0 +1,63 @@
package logtail
import (
"testing"
"time"
)
type deferredStore struct {
days map[string]int64
}
func (d *deferredStore) AddSenderCheckDeferred(at time.Time, n int64) error {
if d.days == nil {
d.days = make(map[string]int64)
}
d.days[at.UTC().Format("2006-01-02")] += n
return nil
}
// The lines are Postfix 3.7's, as the e2e stand wrote them with the panel
// stopped (2026-10-12), with the lines of two other things that use the same
// words: a level-2 refusal of the journal milter and OpenDKIM not answering.
func TestDeferredCounter(t *testing.T) {
st := &deferredStore{}
c := newDeferredCounter(&SenderCheck{Store: st, Socket: "/run/selfpost/sender.sock"})
now := time.Date(2026, 10, 12, 18, 0, 0, 0, time.UTC)
for _, line := range []string{
// The journal milter's own temporary refusal: no warning, not counted.
`2026-10-12T16:22:13.100000+00:00 mail postfix/smtpd[1149]: connect from unknown[10.77.0.1]`,
`2026-10-12T16:22:13.200000+00:00 mail postfix/smtpd[1149]: NOQUEUE: milter-reject: MAIL from unknown[10.77.0.1]: 451 4.7.1 Service unavailable - try again later; from=<billing@sender.e2e.test> proto=ESMTP helo=<e2e-client.e2e.test> sasl_username=<l2app>`,
// The check does not answer: one connection, counted once.
`2026-10-12T16:22:57.000000+00:00 mail postfix/smtpd[1183]: connect from unknown[10.77.0.1]`,
`2026-10-12T16:22:57.100000+00:00 mail postfix/smtpd[1183]: warning: connect to Milter service unix:/run/selfpost/sender.sock: No such file or directory`,
`2026-10-12T16:22:57.200000+00:00 mail postfix/smtpd[1183]: NOQUEUE: milter-reject: CONNECT from unknown[10.77.0.1]: 451 4.7.1 Service unavailable - try again later; proto=SMTP`,
`2026-10-12T16:22:57.300000+00:00 mail postfix/smtpd[1183]: NOQUEUE: milter-reject: EHLO from unknown[10.77.0.1]: 451 4.7.1 Service unavailable - try again later; proto=SMTP helo=<e2e-client.e2e.test>`,
`2026-10-12T16:22:57.400000+00:00 mail postfix/smtpd[1183]: disconnect from unknown[10.77.0.1] ehlo=1 auth=0/1 quit=1 commands=2/3`,
// The same process takes another client, and the check answers again:
// a level-2 refusal there is not the check's.
`2026-10-12T16:23:30.000000+00:00 mail postfix/smtpd[1183]: connect from unknown[10.77.0.1]`,
`2026-10-12T16:23:30.100000+00:00 mail postfix/smtpd[1183]: NOQUEUE: milter-reject: MAIL from unknown[10.77.0.1]: 451 4.7.1 Service unavailable - try again later; from=<a@sender.e2e.test> proto=ESMTP helo=<x>`,
// OpenDKIM does not answer: another socket, not counted.
`2026-10-12T16:24:00.000000+00:00 mail postfix/smtpd[1201]: connect from unknown[10.77.0.1]`,
`2026-10-12T16:24:00.100000+00:00 mail postfix/smtpd[1201]: warning: connect to Milter service unix:/run/opendkim/opendkim.sock: Connection refused`,
`2026-10-12T16:24:00.200000+00:00 mail postfix/smtpd[1201]: NOQUEUE: milter-reject: CONNECT from unknown[10.77.0.1]: 451 4.7.1 Service unavailable - try again later; proto=SMTP`,
// The check stops answering in the middle of a message, the next day.
`2026-10-13T00:00:05.000000+00:00 mail postfix/smtpd[1300]: connect from unknown[10.77.0.1]`,
`2026-10-13T00:00:21.000000+00:00 mail postfix/smtpd[1300]: warning: milter unix:/run/selfpost/sender.sock: can't read SMFIC_EOH reply packet header: Connection timed out`,
`2026-10-13T00:00:21.100000+00:00 mail postfix/smtpd[1300]: 4XcB7k2Jm9z1: milter-reject: END-OF-HEADER from unknown[10.77.0.1]: 4.7.1 Service unavailable - try again later; from=<a@sender.e2e.test> to=<b@sink.e2e.test> proto=ESMTP helo=<x>`,
// The check's own verdict is a refusal, not a deferral, and it is
// cleanup that writes it.
`2026-10-13T00:01:00.000000+00:00 mail postfix/cleanup[1195]: B6B484831946: milter-reject: END-OF-MESSAGE from unknown[10.77.0.1]: 5.7.1 From: director@other.e2e.test - not an address this application sends as; from=<alerts@sender.e2e.test> to=<rcpt@sink.e2e.test> proto=ESMTP helo=<x>`,
} {
c.handle(line, now)
}
if len(st.days) != 2 || st.days["2026-10-12"] != 1 || st.days["2026-10-13"] != 1 {
t.Fatalf("deferred per day = %v, want one on each of the two days", st.days)
}
}