from-header-binding: the sender check, what it costs, its pages; guide, architecture, security

A message's From: line is held against the application that sends it, under
the rule Postfix holds the envelope sender to. OpenDKIM chooses the key by
that line, so until now an application could have mail signed as another
domain of the server or as the server itself.

The check (internal/milter/sender.go)
- A role of the panel on a socket of its own (SENDER_CHECK_SOCKET), first in
  smtpd_milters and strict: default_action=tempfail. The journal milter keeps
  its socket and stays fail-open. postfix-wrapper waits for the new socket.
- It reads no database: what an application may send as comes from Postfix's
  own sender_login_maps, what the server signs for from OpenDKIM's
  SigningTable. Both are read again when they change; one that cannot be
  read defers mail.
- One From: field, one address, allowed by the envelope's lookup; then no
  other name of the SigningTable may appear anywhere in the raw line.
  Refused: 550 5.7.1 naming the line, reduced to printable ASCII.
- A refusal is a send-log row with reason "from" and the header address
  (migration 0006), written by another goroutine and not waited for. The
  journal's refusals record their reason too (limit, client), and the
  notifier takes only a limit's.

What it costs (docs/plans/from-header-binding.md, Statistics)
- The role marks itself alive every 2 s; at start the gap since the mark is
  downtime, counted from Postfix's start if later and not at all when
  supervisord says Postfix is not running.
- The log tailer counts the connections Postfix turned away: a warning that
  names the socket, then milter-reject "Service unavailable" from the same
  smtpd. Seen on the stand: the client is refused from CONNECT on and leaves
  at AUTH (454), so no MAIL line is ever written.
- sender_check_days, a row per UTC day, kept a year.

Pages
- Health: the card "Sender check", the socket's row, and the cards layout -
  boxes numbered by importance in two stacks read in rows; the left stack
  polls, the three readings of the right one are swapped in by id.
- Outbound log: a refused row says why in its tag (wrong From, wrong client,
  rate-limited); the message's page shows the From: line and what the
  application may send as.
- Help: topic "Sender check", word for word the mockup's.

Found by the stand, fixed here: the names for the second look came from the
sender bindings, and a domain with a key and no application yet was not
among them.

e2e: negative_journal_milter_fail_open is gone - it proved that mail passes
with the panel down, which is no longer so. In its place
sender_check_holds_the_from_line (another domain's From:, the hostname's, a
stranger's, an address off the list, another sender in the display name -
refused; own addresses - delivered) and panel_down_defers_mail (4xx with the
panel stopped, the container alive, mail again after, downtime and one
deferred message on Health, the card warning).

Test server, this tree before the last cosmetic edits (a fixture row, a
fact's size, docs): go vet ok, go test 23 packages ok, make e2e ok (337 s).
Locally after them: internal/web/view ok.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

internal/milter/sender_test.go

@@ -0,0 +1,226 @@
+package milter
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/emersion/go-milter"
+
+	"github.com/mixeme/selfpost/internal/store"
+)
+
+// senderMaps is a sender_login_maps file as internal/postfix writes it: shop
+// and news send as any address of their domains, billing only as two listed
+// addresses of example.org, and two applications share shop.example.org.
+const senderMaps = `@news.example.net news
+@shop.example.org shop,shop2
+invoices@example.org billing
+billing@example.org billing
+`
+
+// signingTable is OpenDKIM's SigningTable for the same server: the three
+// sending domains, one more that has no application yet, and the server's own
+// name.
+const signingTable = `*@news.example.net selfpost._domainkey.news.example.net
+*@shop.example.org selfpost._domainkey.shop.example.org
+*@example.org selfpost._domainkey.example.org
+*@empty.example.com selfpost._domainkey.empty.example.com
+*@mail.example.org selfpost._domainkey.mail.example.org
+`
+
+func signers(t *testing.T) *Signers {
+	t.Helper()
+	return NewSigners(writeMaps(t, signingTable))
+}
+
+func writeMaps(t *testing.T, content string) string {
+	t.Helper()
+	p := filepath.Join(t.TempDir(), "table")
+	if err := os.WriteFile(p, []byte(content), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	return p
+}
+
+// offer plays one message to the check and returns the reply at the end of
+// the headers, with what was left for the log.
+func offer(t *testing.T, bind *Bindings, login, envelope string, headers ...[2]string) (milter.Response, []store.SendLogEntry) {
+	t.Helper()
+	refused := make(chan store.SendLogEntry, 8)
+	s := &senderSession{bind: bind, signers: signers(t), refused: refused}
+	if _, err := s.MailFrom(envelope, mods(map[string]string{"{auth_authen}": login})); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := s.RcptTo("someone@elsewhere.example", mods(nil)); err != nil {
+		t.Fatal(err)
+	}
+	for _, h := range headers {
+		if _, err := s.Header(h[0], h[1], mods(nil)); err != nil {
+			t.Fatal(err)
+		}
+	}
+	resp, err := s.Headers(nil, mods(nil))
+	if err != nil {
+		t.Fatal(err)
+	}
+	close(refused)
+	var rows []store.SendLogEntry
+	for e := range refused {
+		rows = append(rows, e)
+	}
+	return resp, rows
+}
+
+func replyText(r milter.Response) string {
+	m := r.Response()
+	return string(rune(m.Code)) + " " + strings.TrimRight(string(m.Data), "\x00")
+}
+
+func TestSenderCheck(t *testing.T) {
+	bind := NewBindings(writeMaps(t, senderMaps))
+	from := func(v string) [2]string { return [2]string{"From", v} }
+
+	cases := []struct {
+		name     string
+		login    string
+		headers  [][2]string
+		pass     bool
+		wantText string // a part of the refusal
+	}{
+		{"own address", "shop", [][2]string{from("shop@shop.example.org")}, true, ""},
+		{"own domain, any address, with a name", "shop", [][2]string{from(`"The Shop" <Orders@Shop.Example.Org>`)}, true, ""},
+		{"second application of the domain", "shop2", [][2]string{from("x@shop.example.org")}, true, ""},
+		{"a listed address", "billing", [][2]string{from("Invoices <invoices@example.org>")}, true, ""},
+		{"header name in another case", "shop", [][2]string{{"FROM", "a@shop.example.org"}}, true, ""},
+		{"its own name inside the display name", "shop", [][2]string{from(`"shop@shop.example.org" <a@shop.example.org>`)}, true, ""},
+		{"an encoded display name", "shop", [][2]string{from("=?utf-8?b?0JzQsNCz0LDQt9C40L0=?= <a@shop.example.org>")}, true, ""},
+
+		{"another registered domain", "shop", [][2]string{from("director@example.org")}, false, "not an address this application sends as"},
+		{"the server's own name", "shop", [][2]string{from("postmaster@mail.example.org")}, false, "not an address this application sends as"},
+		{"a stranger's domain", "shop", [][2]string{from("someone@gmail.example")}, false, "not an address this application sends as"},
+		{"an address off the list", "billing", [][2]string{from("director@example.org")}, false, "not an address this application sends as"},
+		{"a subdomain of its own", "shop", [][2]string{from("a@sub.shop.example.org")}, false, "not an address this application sends as"},
+		{"no From: line", "shop", [][2]string{{"Subject", "hello"}}, false, "no From: line"},
+		{"two From: lines", "shop", [][2]string{from("a@shop.example.org"), from("director@example.org")}, false, "more than one From: line"},
+		{"two addresses in one line", "shop", [][2]string{from("a@shop.example.org, director@example.org")}, false, "one address"},
+		{"not an address", "shop", [][2]string{from("the shop")}, false, "one address"},
+		{"a group", "shop", [][2]string{from("friends: a@shop.example.org, b@shop.example.org;")}, false, "one address"},
+		{"another sender in the display name", "shop", [][2]string{from(`"director@example.org" <a@shop.example.org>`)}, false, "names another sender of this server"},
+		{"the server's name in a comment", "shop", [][2]string{from("a@shop.example.org (postmaster@mail.example.org)")}, false, "names another sender of this server"},
+		{"a domain with a key and no application, in the display name", "shop", [][2]string{from(`"x@empty.example.com" <a@shop.example.org>`)}, false, "names another sender of this server"},
+		{"a stranger's address in the display name", "shop", [][2]string{from(`"someone@gmail.example" <a@shop.example.org>`)}, true, ""},
+		{"no login", "", [][2]string{from("a@shop.example.org")}, false, "no authenticated application"},
+		{"an unknown login", "ghost", [][2]string{from("a@shop.example.org")}, false, "not an address this application sends as"},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			resp, rows := offer(t, bind, c.login, "shop@shop.example.org", c.headers...)
+			if c.pass {
+				if resp != milter.RespContinue {
+					t.Fatalf("refused: %s", replyText(resp))
+				}
+				if len(rows) != 0 {
+					t.Fatalf("a passed message left rows: %+v", rows)
+				}
+				return
+			}
+			text := replyText(resp)
+			if !strings.HasPrefix(text, "y 550 5.7.1 From: ") || !strings.Contains(text, c.wantText) {
+				t.Fatalf("reply %q, want a 550 5.7.1 saying %q", text, c.wantText)
+			}
+			if len(rows) != 1 || rows[0].Reason != store.ReasonFrom || rows[0].To != "someone@elsewhere.example" || rows[0].AppLogin != c.login {
+				t.Fatalf("log rows: %+v", rows)
+			}
+		})
+	}
+}
+
+// What the client wrote is said back to it, logged and stored: it must not
+// be able to put a line break or a control character there.
+func TestSenderCheckPrintsNothingRaw(t *testing.T) {
+	bind := NewBindings(writeMaps(t, senderMaps))
+	evil := "x@gmail.example\r\n250 ok %s \x00\x1b[31m " + strings.Repeat("a", 400)
+	resp, rows := offer(t, bind, "shop", "shop@shop.example.org", [2]string{"From", evil})
+	text := replyText(resp)
+	for _, bad := range []string{"\r", "\n", "%", "\x1b"} {
+		if strings.Contains(text, bad) || strings.Contains(rows[0].HeaderFrom, bad) {
+			t.Fatalf("%q got through: %q / %q", bad, text, rows[0].HeaderFrom)
+		}
+	}
+	if len(rows[0].HeaderFrom) > 130 {
+		t.Fatalf("stored %d bytes of the header", len(rows[0].HeaderFrom))
+	}
+}
+
+// Without its file the check knows nothing, and what cannot be checked is
+// deferred - neither passed nor refused for good.
+func TestSenderCheckDefersWithoutBindings(t *testing.T) {
+	bind := NewBindings(filepath.Join(t.TempDir(), "absent"))
+	resp, rows := offer(t, bind, "shop", "shop@shop.example.org", [2]string{"From", "a@shop.example.org"})
+	if resp != milter.RespTempFail {
+		t.Fatalf("reply %s, want a temporary failure", replyText(resp))
+	}
+	if len(rows) != 0 {
+		t.Fatalf("a deferred message left rows: %+v", rows)
+	}
+}
+
+// The panel rewrites the file when an application is added or deleted; the
+// check has to follow without a restart.
+func TestBindingsFollowTheFile(t *testing.T) {
+	p := writeMaps(t, senderMaps)
+	bind := NewBindings(p)
+	if ok, err := bind.Allowed("shop", "a@shop.example.org"); err != nil || !ok {
+		t.Fatalf("before: %v %v", ok, err)
+	}
+	if err := os.WriteFile(p, []byte("@news.example.net news\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	// A rewrite within the clock's resolution that keeps the size would not be
+	// seen; the panel replaces the file by rename, which gives it a new time.
+	later := time.Now().Add(2 * time.Second)
+	if err := os.Chtimes(p, later, later); err != nil {
+		t.Fatal(err)
+	}
+	if ok, err := bind.Allowed("shop", "a@shop.example.org"); err != nil || ok {
+		t.Fatalf("after the application was deleted: %v %v", ok, err)
+	}
+}
+
+// Without the signing table the second look cannot be taken, and a line it
+// would have to judge is deferred; a line that names nobody else is not held
+// up by it.
+func TestSenderCheckDefersWithoutSigningTable(t *testing.T) {
+	bind := NewBindings(writeMaps(t, senderMaps))
+	play := func(v string) milter.Response {
+		s := &senderSession{bind: bind, signers: NewSigners(filepath.Join(t.TempDir(), "absent"))}
+		_, _ = s.MailFrom("shop@shop.example.org", mods(map[string]string{"{auth_authen}": "shop"}))
+		_, _ = s.Header("From", v, mods(nil))
+		resp, _ := s.Headers(nil, mods(nil))
+		return resp
+	}
+	if resp := play(`"x@example.org" <a@shop.example.org>`); resp != milter.RespTempFail {
+		t.Fatalf("reply %s, want a temporary failure", replyText(resp))
+	}
+	if resp := play(`a@shop.example.org`); resp != milter.RespContinue {
+		t.Fatalf("a plain own address: %s", replyText(resp))
+	}
+}
+
+// A second message on the connection is judged by its own headers.
+func TestSenderCheckResetsBetweenMessages(t *testing.T) {
+	bind := NewBindings(writeMaps(t, senderMaps))
+	s := &senderSession{bind: bind, signers: signers(t)}
+	auth := mods(map[string]string{"{auth_authen}": "shop"})
+	for i, v := range []string{"director@example.org", "a@shop.example.org", "director@example.org"} {
+		_, _ = s.MailFrom("shop@shop.example.org", auth)
+		_, _ = s.Header("From", v, mods(nil))
+		resp, _ := s.Headers(nil, mods(nil))
+		if passed := resp == milter.RespContinue; passed != (i == 1) {
+			t.Fatalf("message %d (From: %s): passed=%v", i, v, passed)
+		}
+	}
+}