mix/selfpost
1
0
Fork 0
Code Issues Pull Requests Actions 21 Packages Projects Releases Wiki Activity

findings of the reviews of audit stage 2b

Browse Source 
Technical review (Opus) and security review (Fable) of 5be109b..5154bba;
reports in the session's scratchpad. Nothing above low from security; one
bug and three risks from the technical review. Migration 0009 is not
deployed, so it was extended.

- Bug: two refusals of one second swapped their NOQUEUE lines (each line
  went to the newest lineless row). AttachRefusalLine now matches the
  client address too (send_log.client_ip, written by the journal milter -
  Postfix's line names the client and the sender, never the login: the
  sasl_username part of the pattern never matched and could be fed by HELO,
  security F1/F2), takes the nearest row in time and the oldest of equals,
  and a line already on a row goes on no other (security F6).
- Waiting lines and recoveries lived only in memory while the saved offset
  moved past them: a stop within 30 s lost them. The tracker now never saves
  an offset past the oldest waiting item in its file (tracker.hold,
  refusalLines.oldest) - for a recovery, the client= line that names its
  login - so they are read again; the "once" guards make that harmless.
- A recovered row whose login the tailer did not see was nobody's: its
  domain_id falls back to the envelope domain's.
- The fingerprint's file identity is the inode alone: the device number can
  change across a host reboot and make the same log look rotated.
- A refused row carries its queue id; UpdateStatus's "newest row" now skips
  refused rows, so a later message with that id does not flip it to sent.
- The alive mark: its age is read first and kept from then on, so asking
  supervisord for 30 s cannot make it stale; the entrypoint removes it with
  the sockets - a container restart is not the check's downtime, and a mark
  from before would have counted the whole outage as "≈".
- Security F3: the sender check prints no ";" of the client's in the From:
  text of its reply, and the tailer reads the envelope after the first ";"
  - Postfix's own text.
- A refused row with a queue id and no stored lines says so (it was told the
  mail.log text of 2.2.0); the fact's em dash is the house hyphen.
- Docs: architecture (client address, no login), guide § Health (warns on
  either count), help topics log and sender-check (design 1ca87f2).
- e2e: refusals_show_postfixs_line wants the refusal's own sender in the
  line; refusal_recovered_from_the_log waits past the tailer's 30 s and
  checks that a refusal the panel wrote got no recovered twin.

Not changed, written down: review 9 (mail.log.1 read to its end at that
moment; postlogd may still write to it until the reload), 10 (a log under
512 bytes has no fingerprint, so places fall back to time briefly after a
rotation), 6 (a 451 at MAIL FROM of another milter reads the same - the
client and sender must match too now), security F5/F12 (info). Review 12:
the refused and closed-by-the-panel variants of out-message were shown to
the owner as pictures of the real templates, not drawn in out-message.html:
the guard ties every drawn screen to a page of the engine, and a variant
has the page's outline.

Evidence on the test server for this tree: go vet clean, go test ./... 23
packages ok, make e2e exit 0 (refusal_recovered_from_the_log 70 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
mix
2026-10-14 18:38:22 +03:00
co-authored by Claude Opus 5.5
parent 1ca87f2d1a
commit af130e268a
22 changed files with 295 additions and 116 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/store/migrations/0009_refusal_line.sql
+6
View File
@@ -7,6 +7,12 @@
-- headers are judged), and its lines are found by it like any other's.
-- DDL only; a row written earlier has '' and says so on its page.
ALTER TABLE send_log ADD COLUMN refusal_line TEXT NOT NULL DEFAULT '';
-- The client a refusal at MAIL FROM came from, as the milter saw it: Postfix's
-- line names the client and the envelope sender and nothing else - no login
-- - so the two together are what finds the line's row; the sender alone would
-- let one application's line land on another's row of the same sender
-- (security review of stage 2b).
ALTER TABLE send_log ADD COLUMN client_ip TEXT NOT NULL DEFAULT '';
-- A refusal row the panel did not get to write - the sender check answers 550
-- and writes the row a moment later, and a panel stopped in between wrote
internal/store/refusal_test.go
+57 -23
View File
@@ -1,20 +1,23 @@
package store
import (
"fmt"
"testing"
"time"
)
// Postfix's line about a refusal at MAIL FROM goes on the row of that sender
// and login refused about then, the newest without a line; a queued row, a
// row of another sender and a row from long ago are not it (audit
// 2026-10-12, stage 2b).
// Postfix's line about a refusal at MAIL FROM goes on the row of that client
// and sender refused about then; a queued row, a row of another client or
// sender, and a row from long ago are not it; a line already on a row goes on
// no other; and two refusals of one second take one line each, in the order
// they were written (security and technical reviews of stage 2b).
func TestAttachRefusalLine(t *testing.T) {
st := openTestStore(t)
now := time.Now().UTC()
for _, e := range []SendLogEntry{
{AppLogin: "l2app", From: "billing@example.com", Reason: ReasonLimit},
{AppLogin: "l2app", From: "other@example.com", Reason: ReasonLimit},
{AppLogin: "l2app", From: "billing@example.com", Reason: ReasonLimit, ClientIP: "10.0.0.1"},
{AppLogin: "l2app", From: "billing@example.com", Reason: ReasonLimit, ClientIP: "10.0.0.1"},
{AppLogin: "other", From: "billing@example.com", Reason: ReasonClient, ClientIP: "10.0.0.2"},
} {
if err := st.InsertRejected(e); err != nil {
t.Fatal(err)
@@ -23,33 +26,37 @@ func TestAttachRefusalLine(t *testing.T) {
if err := st.InsertQueued(SendLogEntry{QueueID: "Q1", AppLogin: "l2app", From: "billing@example.com", To: "x@example.net"}); err != nil {
t.Fatal(err)
}
line := "Oct 14 12:00:00 mail postfix/smtpd[1]: NOQUEUE: milter-reject: MAIL from unknown[10.77.0.1]: 451 4.7.1 Service unavailable"
if ok, err := st.AttachRefusalLine("l2app", "Billing@example.com", now, line); err != nil || !ok {
t.Fatalf("attach = %v, %v", ok, err)
first := "Oct 14 12:00:00 mail postfix/smtpd[1]: NOQUEUE: milter-reject: MAIL from unknown[10.0.0.1]: 451 4.7.1 first"
second := "Oct 14 12:00:00 mail postfix/smtpd[2]: NOQUEUE: milter-reject: MAIL from unknown[10.0.0.1]: 451 4.7.1 second"
for _, line := range []string{first, second} {
if ok, err := st.AttachRefusalLine("10.0.0.1", "Billing@example.com", now, line); err != nil || !ok {
t.Fatalf("attach %q = %v, %v", line, ok, err)
}
}
// The row has its line now; a second line finds no row without one.
if ok, _ := st.AttachRefusalLine("l2app", "billing@example.com", now, line); ok {
t.Fatal("a second line took a row that has one")
// Read again after a restart: no row takes it a second time.
if ok, _ := st.AttachRefusalLine("10.0.0.1", "billing@example.com", now, first); ok {
t.Fatal("a line already on a row took another")
}
if ok, _ := st.AttachRefusalLine("l2app", "other@example.com", now.Add(-time.Hour), line); ok {
if ok, _ := st.AttachRefusalLine("10.0.0.3", "billing@example.com", now, "a third"); ok {
t.Fatal("a line of another client took a row")
}
if ok, _ := st.AttachRefusalLine("10.0.0.2", "billing@example.com", now.Add(-time.Hour), "an hour off"); ok {
t.Fatal("a line an hour off took a row")
}
if ok, _ := st.AttachRefusalLine("someone", "other@example.com", now, line); ok {
t.Fatal("a line of another login took a row")
}
lines := map[string]string{}
rows, _ := st.QuerySendLog(SendLogFilter{AllDomains: true}, 10, 0)
got := 0
for _, r := range rows {
full, _ := st.GetSendLog(r.ID)
if full.RefusalLine != "" {
got++
if full.From != "billing@example.com" || full.Status != StatusRejected {
t.Errorf("the line went to %+v", full)
}
lines[fmt.Sprint(full.ID)] = full.RefusalLine
}
}
if got != 1 {
t.Fatalf("%d rows have a line, want 1", got)
if len(lines) != 2 {
t.Fatalf("lines on rows: %v, want the first two rows", lines)
}
r1, _ := st.GetSendLog(rows[len(rows)-1].ID) // the oldest row
if r1.RefusalLine != first {
t.Fatalf("the first row has %q, want the first line", r1.RefusalLine)
}
}
@@ -113,3 +120,30 @@ func TestRecoverRefusal(t *testing.T) {
t.Fatalf("the recovered row: %+v", found)
}
}
// A recovered row whose login the tailer did not see is its envelope
// domain's, not nobody's (review of stage 2b).
func TestRecoverRefusalWithoutALogin(t *testing.T) {
st := openTestStore(t)
d := addTestDomain(t, st, "sender.example")
e := SendLogEntry{QueueID: "NOLOGIN", From: "a@sender.example", To: "r@x", Domain: "sender.example", Reason: ReasonFrom}
if wrote, err := st.RecoverRefusal(e, time.Now()); err != nil || !wrote {
t.Fatalf("recover = %v, %v", wrote, err)
}
rows, _ := st.QuerySendLog(SendLogFilter{DomainIDs: []int64{d.ID}}, 10, 0)
if len(rows) != 1 {
t.Fatalf("the domain's log has %d rows, want the recovered one", len(rows))
}
}
// A refused row carries the queue id it was given; a delivery line of a later
// message with that id does not touch it.
func TestUpdateStatusLeavesARefusalAlone(t *testing.T) {
st := openTestStore(t)
if err := st.InsertRejected(SendLogEntry{QueueID: "REUSED", To: "a@example.net", Reason: ReasonFrom}); err != nil {
t.Fatal(err)
}
if n, err := st.UpdateStatus("REUSED", "a@example.net", StatusSent); err != nil || n != 0 {
t.Fatalf("UpdateStatus = %d, %v; want 0 rows", n, err)
}
}
internal/store/schema_test.go
+1 -1
View File
@@ -47,7 +47,7 @@ func TestBaselineSchema(t *testing.T) {
"applications": "id domain_id login address_mode auth_ip_restrict auth_allowed_ips created_at",
"application_addresses": "id application_id address",
"rate_limits": "id scope ref_id max_messages window_seconds mode auto_multiplier auto_updated_at",
"send_log": "id queue_id domain app_login from_addr to_addr subject status created_at updated_at reason header_from domain_id refusal_line recovered",
"send_log": "id queue_id domain app_login from_addr to_addr subject status created_at updated_at reason header_from domain_id refusal_line client_ip recovered",
"logtail_state": "path fingerprint read_offset updated_at",
"inbound_domains": "id name recipient_mode created_at",
"inbound_transports": "inbound_domain_id host port tls_mode",
internal/store/sendlog.go
+33 -19
View File
@@ -38,6 +38,9 @@ type SendLogEntry struct {
// was refused (Reason*), and for ReasonFrom what its From: line said.
Reason string
HeaderFrom string
// ClientIP is the client a refusal at MAIL FROM came from (InsertRejected
// only), which Postfix's line about it names.
ClientIP string
}
// loginDomainID is the domain_id a row is written with: the domain of the
@@ -73,9 +76,9 @@ func (s *Store) InsertRejected(e SendLogEntry) error {
now := time.Now().UTC().Format(time.RFC3339)
_, err := s.db.Exec(
`INSERT INTO send_log
(queue_id, domain, app_login, from_addr, to_addr, subject, status, reason, header_from, created_at, updated_at, domain_id)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, `+loginDomainID+`)`,
e.QueueID, e.Domain, e.AppLogin, e.From, e.To, e.Subject, StatusRejected, e.Reason, e.HeaderFrom, now, now, e.AppLogin,
(queue_id, domain, app_login, from_addr, to_addr, subject, status, reason, header_from, client_ip, created_at, updated_at, domain_id)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, `+loginDomainID+`)`,
e.QueueID, e.Domain, e.AppLogin, e.From, e.To, e.Subject, StatusRejected, e.Reason, e.HeaderFrom, e.ClientIP, now, now, e.AppLogin,
)
if err != nil {
return fmt.Errorf("insert rejected send_log: %w", err)
@@ -113,7 +116,9 @@ func (s *Store) UpdateStatus(queueID, recipient, status string) (int64, error) {
return 0, fmt.Errorf("update send_log status: %w", err)
}
defer tx.Rollback()
const newest = `id = (SELECT MAX(id) FROM send_log WHERE queue_id = ? AND to_addr = ? COLLATE NOCASE)`
// A refused row carries the queue id it was given, and that id may be the
// next message's: a delivery line is never about a refusal.
const newest = `id = (SELECT MAX(id) FROM send_log WHERE queue_id = ? AND to_addr = ? COLLATE NOCASE AND status <> 'rejected')`
marked, err := tx.Exec(
`UPDATE send_log SET reason = ? WHERE `+newest+` AND reason = ?`,
ReasonReconcileContradicted, queueID, recipient, ReasonReconcile,
@@ -235,22 +240,27 @@ type SendLogRow struct {
// AttachRefusalLine puts Postfix's line about a refusal at MAIL FROM on the
// row the journal milter wrote for it (audit 2026-10-12, stage 2b). Such a
// refusal has no queue id, so the row is found by what the line says: a
// refused row of that sender - and login, when the line names one - without a
// line yet, written within two minutes of the line's time; the newest when
// there are several. It reports whether a row took the line.
func (s *Store) AttachRefusalLine(login, from string, at time.Time, line string) (bool, error) {
// refusal has no queue id, and the line names no login, so the row is found
// by what the line says: a refused row of that client and sender without a
// line yet, written within two minutes of the line's time; the nearest in
// time, the oldest of equals, when there are several - lines are read in the
// order the rows were written, so two refusals of one second take one line
// each in turn (review of stage 2b). A line already on a row - read again
// after a restart - goes on no other. It reports whether a row took the line.
func (s *Store) AttachRefusalLine(clientIP, from string, at time.Time, line string) (bool, error) {
res, err := s.db.Exec(
`UPDATE send_log SET refusal_line = ? WHERE id = (
`UPDATE send_log SET refusal_line = ?1 WHERE id = (
SELECT id FROM send_log
WHERE status = ? AND queue_id = '' AND refusal_line = ''
AND reason IN ('', ?, ?) AND from_addr = ? COLLATE NOCASE
AND (? = '' OR app_login = ?)
AND created_at BETWEEN ? AND ?
ORDER BY id DESC LIMIT 1)`,
line, StatusRejected, ReasonLimit, ReasonClient, from, login, login,
WHERE status = ?2 AND queue_id = '' AND refusal_line = ''
AND reason IN ('', ?3, ?4) AND from_addr = ?5 COLLATE NOCASE AND client_ip = ?6
AND created_at BETWEEN ?7 AND ?8
ORDER BY ABS(julianday(created_at) - julianday(?9)), id
LIMIT 1)
AND NOT EXISTS (SELECT 1 FROM send_log WHERE refusal_line = ?1)`,
line, StatusRejected, ReasonLimit, ReasonClient, from, clientIP,
at.Add(-refusalLineWindow).UTC().Format(time.RFC3339),
at.Add(refusalLineWindow).UTC().Format(time.RFC3339),
at.UTC().Format(time.RFC3339),
)
if err != nil {
return false, fmt.Errorf("attach refusal line: %w", err)
@@ -264,16 +274,20 @@ func (s *Store) AttachRefusalLine(login, from string, at time.Time, line string)
// wrote the row itself (audit 2026-10-12, stage 2b): the check answers 550
// and writes the row through a buffer a moment later. The message had a queue
// id; a refused row with that id from around at means the row was written,
// and nothing is. It reports whether it wrote one.
// and nothing is. The domain is the login's, or - when the tailer did not see the login, its
// client= line read before a restart - the envelope's, so the row is still
// its domain's administrator's (review of stage 2b). It reports whether it
// wrote one.
func (s *Store) RecoverRefusal(e SendLogEntry, at time.Time) (bool, error) {
stamp := at.UTC().Format(time.RFC3339)
res, err := s.db.Exec(
`INSERT INTO send_log
(queue_id, domain, app_login, from_addr, to_addr, subject, status, reason, header_from, recovered, created_at, updated_at, domain_id)
SELECT ?, ?, ?, ?, ?, '', ?, ?, ?, 1, ?, ?, `+loginDomainID+`
SELECT ?, ?, ?, ?, ?, '', ?, ?, ?, 1, ?, ?,
COALESCE(`+loginDomainID+`, (SELECT d.id FROM domains d WHERE d.name = ?))
WHERE NOT EXISTS (SELECT 1 FROM send_log
WHERE queue_id = ? AND status = ? AND created_at BETWEEN ? AND ?)`,
e.QueueID, e.Domain, e.AppLogin, e.From, e.To, StatusRejected, e.Reason, e.HeaderFrom, stamp, stamp, e.AppLogin,
e.QueueID, e.Domain, e.AppLogin, e.From, e.To, StatusRejected, e.Reason, e.HeaderFrom, stamp, stamp, e.AppLogin, e.Domain,
e.QueueID, StatusRejected,
at.Add(-refusalLineWindow).UTC().Format(time.RFC3339), at.Add(refusalLineWindow).UTC().Format(time.RFC3339),
)