findings of the reviews of audit stage 1

Technical review (Opus) and security review (Fable) of 9efc9c7..a59e3d4;
reports in the session's scratchpad. Fixed:

- B1 (verified bug): store.UpdateStatus read before writing in its
  transaction; a commit by another connection in between (the dmarc-ingest
  process opens the same file) failed it at once with "database is locked"
  and the delivery line was lost. The first statement is now a write.
- R3: a delivery line updated every row of its queue id and recipient; it
  now updates only the newest (Postfix reuses short ids). The sweep closes
  by row id (QueuedDelivery.ID, CloseUnreported(id)).
- R4: a contradicted sweep keeps a mark, reason "reconcile-contradicted",
  instead of losing which row it was.
- R1: contradicted cannot see a lost line (never read late). New counter
  reconcile_missed: at close, the sweep looks for the row's delivery line
  in mail.log and mail.log.1 (QueueLines). 0007 edited - it has run on no
  database but throwaway stands.
- R2: queued_over_lifetime cannot measure "what removing the sweep would
  leave" while the sweep runs; its text now says rows stuck despite the
  sweep, and the rule uses closed - missed - contradicted.
- cert-reload (R5, R6, R7, security medium/low): both files copied to .new
  and the copies checked as a pair before both are renamed; acts on the copy
  only when postconf says Postfix reads it; a failed reload is retried next
  pass; umask 077; says so when the mounted files cannot be read. The e2e
  case asserts Postfix reads the copy, the copy equals the mount, the key is
  600 root.
- Security low: a recipient with a quoted local part - the journal cut
  "a b"@x at the space, deliveryRe cut "a>b"@x at the ">" - matched no
  delivery line and was closed by the sweep. cleanAddress ends a bracketed
  address at the first ">" outside quotes; deliveryRe takes the recipient
  up to the ">, " before relay= (test fixtures given the relay= every real
  delivery line has).
- Security low: postqueue -p's error carries at most 512 bytes of output.
- Nits: addDeliveryLogDay refuses a column not on its list; the truncation
  test checks its message.
- To stage 2b (page wording, mockup-first): the history step of a bounced
  row says the receiving server refused it, false for a row the sweep
  closed. To the owner when stage 2b is reached: since M13 the "deferred"
  figure counts whole messages too, which "Connections turned away" would
  not name.

Found while verifying: the e2e's main positive path failed once with "553
not owned by user app1" - an smtpd idle from before the panel's reload took
the connection in the reload's second. Measured on the stand (one-off
probe, not committed): sending 0, 10, 40, 110 s after the save (two seconds
at the soonest) always met an smtpd started after the reload. The e2e now
waits for the sender map after each application it creates
(waitForSenderMap: AUTH, MAIL, RCPT, RSET - no message, no row), and the
level-1 case uses it instead of its own loop.

Docs: architecture § Queue reconcile, schema-migrations (rows 7 and 8,
chain head 8), CHANGELOG, audit plan (stage 1 note, the measured window,
a new stage 2b line: 62 lines), roadmap 15/62.

Evidence on the test server for this tree: go vet clean, go test ./... 23
packages ok, make e2e exit 0 (renewed certificate, hung panel, level 1 at
49/60).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

internal/logtail/lines_test.go

@@ -176,7 +176,7 @@ func TestRunKeepsQueueLinesOnce(t *testing.T) {
 
 	const (
 		client = "host postfix/submission/smtpd[1]: 41E862C00D9E: client=unknown[203.0.113.7]"
-		sent   = "host postfix/smtp[1]: 41E862C00D9E: to=<a@example.net>, dsn=2.0.0, status=sent (ok)"
+		sent   = "host postfix/smtp[1]: 41E862C00D9E: to=<a@example.net>, relay=x, dsn=2.0.0, status=sent (ok)"
 		later  = "host postfix/qmgr[1]: 41E862C00D9E: removed"
 	)
 

internal/logtail/logtail.go

@@ -31,9 +31,10 @@ import (
 type StatusStore interface {
 	UpdateStatus(queueID, recipient, status string) (int64, error)
 	ListQueuedOlderThan(cutoff time.Time) ([]store.QueuedDelivery, error)
-	CloseUnreported(queueID, recipient string) (int64, error)
+	CloseUnreported(id int64) (int64, error)
 	// The counters of delivery_log_days (store/deliverylog.go).
 	AddReconcileClosed(at time.Time, n int64) error
+	AddReconcileMissed(at time.Time, n int64) error
 	AddTailerBlind(from time.Time, d time.Duration) error
 	CountQueuedOlderThan(cutoff time.Time) (int64, error)
 	SetQueuedOverLifetime(at time.Time, n int64) error
@@ -88,6 +89,10 @@ type RetentionDays func() int
 // The "<queue-id>: to=<addr>, …, status=<word>" shape is specific to the
 // delivery agents; qmgr/smtpd/cleanup lines do not match.
 //
+// The recipient runs to the ">, " that comes before relay= (and orig_to=,
+// when there is one): a quoted local part may hold a ">", and a recipient cut
+// there would match no row and leave it for the sweep (review of stage 1).
+//
 // It is matched at the head of the line - the host name, "program[pid]:",
 // then the queue id - after SplitTimestamp has taken the stamp off, like
 // queueLineRe and the inbound patterns. Anywhere else the text is a
@@ -100,7 +105,7 @@ type RetentionDays func() int
 // the line — and that one can come from the reply text, which the far end
 // controls. A bounce whose reply quoted "status=sent" would then be filed as a
 // success. The real field is always the first one after to=<…>.
-var deliveryRe = regexp.MustCompile(`^\S+ [^\s\[]+\[\d+\]: ([0-9A-Za-z]+): to=<([^>]*)>,.*?\bstatus=(\w+)`)
+var deliveryRe = regexp.MustCompile(`^\S+ [^\s\[]+\[\d+\]: ([0-9A-Za-z]+): to=<(.*?)>, (?:orig_to=<.*?>, )?relay=.*?\bstatus=(\w+)`)
 
 // parseDelivery extracts (queue-id, recipient, status) from a mail.log line.
 // ok is false for lines that are not recognised delivery results.
@@ -149,7 +154,7 @@ func Run(ctx context.Context, path string, st StatusStore, retention RetentionDa
 	// the sweep would otherwise close. follow() closes this once it has read to
 	// end-of-file for the first time.
 	caughtUp := make(chan struct{})
-	go reconcileLoop(ctx, st, caughtUp)
+	go reconcileLoop(ctx, st, caughtUp, path)
 
 	var deferred *deferredCounter
 	if sender != nil {
@@ -190,7 +195,7 @@ func Run(ctx context.Context, path string, st StatusStore, retention RetentionDa
 // up with the log, and then leaves the first sweep a full interval away, so a
 // restart resolves rows from the log — the accurate source — before the sweep
 // gets to guess at whatever the log could not explain.
-func reconcileLoop(ctx context.Context, st StatusStore, caughtUp <-chan struct{}) {
+func reconcileLoop(ctx context.Context, st StatusStore, caughtUp <-chan struct{}, path string) {
 	select {
 	case <-ctx.Done():
 		return
@@ -210,7 +215,7 @@ func reconcileLoop(ctx context.Context, st StatusStore, caughtUp <-chan struct{}
 			return
 		case <-t.C:
 			now := time.Now().UTC()
-			reconcile(st, now.Add(-reconcileGrace))
+			reconcile(st, now.Add(-reconcileGrace), path)
 			countStuck(st, now, overLifetime)
 		}
 	}
@@ -242,7 +247,12 @@ func countStuck(st StatusStore, now time.Time, overLifetime time.Duration) {
 // success because a delivery the panel cannot evidence must not be shown as
 // one. Rows whose message is still in the queue, and every row when the queue
 // cannot be listed at all, are left exactly as they are.
-func reconcile(st StatusStore, cutoff time.Time) {
+//
+// A closed row whose delivery line is in mail.log or mail.log.1 after all is
+// counted as missed: the line was there and the tailer did not read it. That
+// is what tells a tailer that loses lines from a log that lost them, which
+// reconcile_contradicted cannot - a lost line is never read late.
+func reconcile(st StatusStore, cutoff time.Time, path string) {
 	rows, err := st.ListQueuedOlderThan(cutoff)
 	if err != nil {
 		log.Printf("log-tailer: reconcile: list queued rows: %v", err)
@@ -260,26 +270,54 @@ func reconcile(st StatusStore, cutoff time.Time) {
 		return
 	}
 
-	var closed int64
+	var closed, missed int64
 	for _, row := range rows {
 		if _, still := held[row.QueueID]; still {
 			continue
 		}
-		n, err := st.CloseUnreported(row.QueueID, row.To)
+		n, err := st.CloseUnreported(row.ID)
 		if err != nil {
 			log.Printf("log-tailer: reconcile: close %s/%s: %v", row.QueueID, row.To, err)
 			continue
 		}
 		closed += n
+		if n > 0 && deliveryInLog(path, row) {
+			log.Printf("log-tailer: reconcile: %s/%s had a delivery line in the log the tailer did not read", row.QueueID, row.To)
+			missed++
+		}
 	}
 	if closed > 0 {
 		log.Printf("log-tailer: reconcile: closed %d row(s) Postfix no longer holds and never reported", closed)
 		if err := st.AddReconcileClosed(time.Now(), closed); err != nil {
 			log.Printf("log-tailer: reconcile: count: %v", err)
 		}
+		if err := st.AddReconcileMissed(time.Now(), missed); err != nil {
+			log.Printf("log-tailer: reconcile: count: %v", err)
+		}
 	}
 }
 
+// deliveryInLog says whether the current log or the one rotated last holds a
+// delivery line for row. Rows reach the sweep minutes after they were
+// written, so the end of those two files is where such a line would be.
+func deliveryInLog(path string, row store.QueuedDelivery) bool {
+	if path == "" {
+		return false
+	}
+	for _, p := range []string{path, path + ".1"} {
+		lines, err := QueueLines(p, row.QueueID, 256)
+		if err != nil {
+			continue
+		}
+		for _, line := range lines {
+			if q, r, _, ok := parseDelivery(line); ok && q == row.QueueID && strings.EqualFold(r, row.To) {
+				return true
+			}
+		}
+	}
+	return false
+}
+
 // retentionLoop prunes expired send-log rows immediately and then periodically,
 // and after each sweep the stored log lines no row carries the queue id of:
 // those the deleted rows leave behind, and those that never had a row

internal/logtail/logtail_test.go

@@ -116,6 +116,16 @@ func TestParseDelivery(t *testing.T) {
 			line:   "host postfix/cleanup[12]: BEEF03: message-id=<x@example.com>",
 			wantOK: false,
 		},
+		{
+			// A quoted local part may hold ">": the recipient runs to the
+			// ">, " before relay=.
+			name:      "a recipient with > in a quoted local part",
+			line:      `host postfix/smtp[26]: C0FFEE: to=<"a>b"@example.net>, relay=mx[203.0.113.9]:25, dsn=2.0.0, status=sent (250 OK)`,
+			wantOK:    true,
+			queueID:   "C0FFEE",
+			recipient: `"a>b"@example.net`,
+			status:    store.StatusSent,
+		},
 		{
 			// The stamp Postfix 3.7's postlogd writes, as on the stand.
 			name:      "syslog stamp",
@@ -178,6 +188,7 @@ type captureStore struct {
 	// closed sums AddReconcileClosed; stuckCount is what CountQueuedOlderThan
 	// answers and stuck what SetQueuedOverLifetime was given.
 	closed     int64
+	missed     int64
 	stuckCount int64
 	stuck      []int64
 
@@ -241,12 +252,18 @@ func (c *captureStore) UpdateStatus(queueID, recipient, status string) (int64, e
 	return 1, nil
 }
 
-// CloseUnreported records the sweep's close as "queue-id|recipient|reconcile".
-func (c *captureStore) CloseUnreported(queueID, recipient string) (int64, error) {
+// CloseUnreported records the sweep's close as "queue-id|recipient|reconcile",
+// finding the row by its id among queued.
+func (c *captureStore) CloseUnreported(id int64) (int64, error) {
 	c.mu.Lock()
 	defer c.mu.Unlock()
-	c.calls = append(c.calls, queueID+"|"+recipient+"|"+store.ReasonReconcile)
-	return 1, nil
+	for _, q := range c.queued {
+		if q.ID == id {
+			c.calls = append(c.calls, q.QueueID+"|"+q.To+"|"+store.ReasonReconcile)
+			return 1, nil
+		}
+	}
+	return 0, nil
 }
 
 func (c *captureStore) DeleteSendLogBefore(time.Time) (int64, error) { return 0, nil }
@@ -259,6 +276,12 @@ func (c *captureStore) AddReconcileClosed(_ time.Time, n int64) error {
 	c.closed += n
 	return nil
 }
+func (c *captureStore) AddReconcileMissed(_ time.Time, n int64) error {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	c.missed += n
+	return nil
+}
 func (c *captureStore) AddTailerBlind(time.Time, time.Duration) error { return nil }
 func (c *captureStore) CountQueuedOlderThan(time.Time) (int64, error) {
 	c.mu.Lock()
@@ -332,7 +355,7 @@ func TestFollowTailsAndRotates(t *testing.T) {
 	// Give follow() time to open at EOF (it seeks to end immediately on start,
 	// so the seed line above is ignored), then append a delivery line.
 	time.Sleep(50 * time.Millisecond)
-	appendLine(t, path, "host postfix/smtp[1]: Q1: to=<a@example.net>, dsn=2.0.0, status=sent (ok)")
+	appendLine(t, path, "host postfix/smtp[1]: Q1: to=<a@example.net>, relay=x, dsn=2.0.0, status=sent (ok)")
 	waitFor(t, func() bool { return contains(cs.snapshot(), "Q1|a@example.net|sent") })
 
 	// Rotate: move the current file aside and create a fresh one (logrotate
@@ -343,7 +366,7 @@ func TestFollowTailsAndRotates(t *testing.T) {
 	if err := os.WriteFile(path, nil, 0o644); err != nil {
 		t.Fatalf("recreate: %v", err)
 	}
-	appendLine(t, path, "host postfix/smtp[1]: Q2: to=<b@example.net>, dsn=5.1.1, status=bounced (nope)")
+	appendLine(t, path, "host postfix/smtp[1]: Q2: to=<b@example.net>, relay=x, dsn=5.1.1, status=bounced (nope)")
 	waitFor(t, func() bool { return contains(cs.snapshot(), "Q2|b@example.net|bounced") })
 
 	cancel()
@@ -375,12 +398,12 @@ func TestFollowResumesAfterRestart(t *testing.T) {
 
 	cs := &captureStore{}
 	stop := startRun(t, path, cs)
-	appendLine(t, path, "host postfix/smtp[1]: Q1: to=<a@example.net>, dsn=2.0.0, status=sent (ok)")
+	appendLine(t, path, "host postfix/smtp[1]: Q1: to=<a@example.net>, relay=x, dsn=2.0.0, status=sent (ok)")
 	waitFor(t, func() bool { return contains(cs.snapshot(), "Q1|a@example.net|sent") })
 	stop() // persists the offset past Q1
 
 	// Down: Postfix keeps delivering.
-	appendLine(t, path, "host postfix/smtp[1]: Q2: to=<b@example.net>, dsn=2.0.0, status=sent (ok)")
+	appendLine(t, path, "host postfix/smtp[1]: Q2: to=<b@example.net>, relay=x, dsn=2.0.0, status=sent (ok)")
 
 	cs.reset()
 	stop = startRun(t, path, cs)
@@ -394,7 +417,7 @@ func TestFollowResumesAfterRestart(t *testing.T) {
 	// The stored offset belongs to a file that no longer exists, so the new one
 	// must be read from the start.
 	if err := os.WriteFile(path, []byte(strings.Repeat("host postfix/qmgr[1]: fresh log after rotation\n", 20)+
-		"host postfix/smtp[1]: Q3: to=<c@example.net>, dsn=5.1.1, status=bounced (nope)\n"), 0o644); err != nil {
+		"host postfix/smtp[1]: Q3: to=<c@example.net>, relay=x, dsn=5.1.1, status=bounced (nope)\n"), 0o644); err != nil {
 		t.Fatalf("recreate log: %v", err)
 	}
 
@@ -490,17 +513,39 @@ func TestReconcileClosesOnlyWhatPostfixNoLongerHolds(t *testing.T) {
 	stubQueue(t, nil, "STILLQ")
 
 	cs := &captureStore{queued: []store.QueuedDelivery{
-		{QueueID: "STILLQ", To: "inflight@example.net"},
-		{QueueID: "GONEQ", To: "lost@example.net"},
+		{ID: 1, QueueID: "STILLQ", To: "inflight@example.net"},
+		{ID: 2, QueueID: "GONEQ", To: "lost@example.net"},
 	}}
-	reconcile(cs, time.Now().UTC())
+	reconcile(cs, time.Now().UTC(), "")
 
 	got := cs.snapshot()
 	if len(got) != 1 || got[0] != "GONEQ|lost@example.net|"+store.ReasonReconcile {
 		t.Fatalf("got %v, want only the message Postfix dropped closed by the sweep", got)
 	}
-	if cs.closed != 1 {
-		t.Fatalf("the day counts %d closed row(s), want 1", cs.closed)
+	if cs.closed != 1 || cs.missed != 0 {
+		t.Fatalf("the day counts %d closed and %d missed, want 1 and 0", cs.closed, cs.missed)
+	}
+}
+
+// A closed row whose delivery line is in the log after all - here in the file
+// rotated last - is counted as missed: the tailer did not read it.
+func TestReconcileCountsALineTheTailerMissed(t *testing.T) {
+	stubQueue(t, nil)
+	dir := t.TempDir()
+	path := filepath.Join(dir, "mail.log")
+	if err := os.WriteFile(path, []byte("Oct 14 11:00:00 mail postfix/qmgr[10]: OTHERQ: removed\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(path+".1", []byte("Oct 14 10:59:58 mail postfix/smtp[26]: SEENQ: to=<Seen@example.net>, relay=mx[203.0.113.9]:25, dsn=2.0.0, status=sent (250 OK)\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	cs := &captureStore{queued: []store.QueuedDelivery{
+		{ID: 1, QueueID: "SEENQ", To: "seen@example.net"},
+		{ID: 2, QueueID: "GONEQ", To: "lost@example.net"},
+	}}
+	reconcile(cs, time.Now().UTC(), path)
+	if cs.closed != 2 || cs.missed != 1 {
+		t.Fatalf("closed %d, missed %d; want 2 and 1", cs.closed, cs.missed)
 	}
 }
 
@@ -560,9 +605,9 @@ func TestReconcileLeavesRowsAloneWhenTheQueueCannotBeRead(t *testing.T) {
 	stubQueue(t, errors.New("postqueue: Permission denied"))
 
 	cs := &captureStore{queued: []store.QueuedDelivery{
-		{QueueID: "GONEQ", To: "lost@example.net"},
+		{ID: 1, QueueID: "GONEQ", To: "lost@example.net"},
 	}}
-	reconcile(cs, time.Now().UTC())
+	reconcile(cs, time.Now().UTC(), "")
 
 	if got := cs.snapshot(); len(got) != 0 {
 		t.Fatalf("got %v, want no row touched", got)
@@ -574,7 +619,7 @@ func TestReconcileLeavesRowsAloneWhenTheQueueCannotBeRead(t *testing.T) {
 func TestReconcileSkipsTheQueueWhenNoRowIsStuck(t *testing.T) {
 	calls := stubQueue(t, nil)
 
-	reconcile(&captureStore{}, time.Now().UTC())
+	reconcile(&captureStore{}, time.Now().UTC(), "")
 
 	if *calls != 0 {
 		t.Fatalf("queue listed %d time(s), want none", *calls)

internal/logtail/retention_test.go

@@ -23,10 +23,11 @@ func (s *retentionProbeStore) ListQueuedOlderThan(time.Time) ([]store.QueuedDeli
 	return nil, nil
 }
 
-func (s *retentionProbeStore) CloseUnreported(string, string) (int64, error) {
+func (s *retentionProbeStore) CloseUnreported(int64) (int64, error) {
 	return 0, nil
 }
 
+func (s *retentionProbeStore) AddReconcileMissed(time.Time, int64) error     { return nil }
 func (s *retentionProbeStore) AddReconcileClosed(time.Time, int64) error     { return nil }
 func (s *retentionProbeStore) AddTailerBlind(time.Time, time.Duration) error { return nil }
 func (s *retentionProbeStore) CountQueuedOlderThan(time.Time) (int64, error) { return 0, nil }

internal/milter/address_test.go

@@ -0,0 +1,23 @@
+package milter
+
+import "testing"
+
+// The journal row's recipient has to be the address Postfix later logs in
+// to=<…>, or the delivery line matches no row and the sweep closes it as
+// bounced. A quoted local part may hold a space or a ">".
+func TestCleanAddress(t *testing.T) {
+	for _, c := range []struct{ in, want string }{
+		{"<a@example.net>", "a@example.net"},
+		{"<a@example.net> SIZE=100 BODY=8BITMIME", "a@example.net"},
+		{"a@example.net", "a@example.net"},
+		{"a@example.net SIZE=100", "a@example.net"},
+		{"<>", ""},
+		{`<"a b"@example.net>`, `"a b"@example.net`},
+		{`<"a>b"@example.net> NOTIFY=SUCCESS`, `"a>b"@example.net`},
+		{`<"a\"> b"@example.net>`, `"a\"> b"@example.net`},
+	} {
+		if got := cleanAddress(c.in); got != c.want {
+			t.Errorf("cleanAddress(%q) = %q, want %q", c.in, got, c.want)
+		}
+	}
+}

internal/milter/milter.go

@@ -203,9 +203,30 @@ func (s *session) record(queueID string) {
 }
 
 // cleanAddress strips the angle brackets and any ESMTP parameters Postfix may
-// pass with an address, leaving the bare mailbox.
+// pass with an address, leaving the bare mailbox. A bracketed address ends at
+// the first ">" outside a quoted local part: a quoted local part may hold a
+// space or a ">", and an address cut there matched no delivery line, so its
+// row was left for the reconcile sweep to close as bounced (review of audit
+// stage 1).
 func cleanAddress(a string) string {
 	a = strings.TrimSpace(a)
+	if strings.HasPrefix(a, "<") {
+		quoted := false
+		for i := 1; i < len(a); i++ {
+			switch a[i] {
+			case '\\':
+				if quoted {
+					i++ // the escaped character is part of the local part
+				}
+			case '"':
+				quoted = !quoted
+			case '>':
+				if !quoted {
+					return a[1:i]
+				}
+			}
+		}
+	}
 	if i := strings.IndexByte(a, ' '); i >= 0 { // drop "addr SIZE=… BODY=…" params
 		a = a[:i]
 	}

internal/postfix/queue.go

@@ -62,7 +62,13 @@ func listQueue() (text string, truncated bool, err error) {
 		return "", false, fmt.Errorf("postqueue -p timed out after %s", postqueueTimeout)
 	}
 	if err != nil {
-		return "", false, fmt.Errorf("postqueue -p: %w: %s", err, strings.TrimSpace(out.buf.String()))
+		// stdout and stderr share the buffer: the end of it says why, and up
+		// to 4 MiB of listing does not belong in an error.
+		why := strings.TrimSpace(out.buf.String())
+		if len(why) > 512 {
+			why = "..." + why[len(why)-512:]
+		}
+		return "", false, fmt.Errorf("postqueue -p: %w: %s", err, why)
 	}
 	return out.buf.String(), out.truncated, nil
 }

internal/postfix/queue_test.go

@@ -49,8 +49,8 @@ func fakePostqueue(t *testing.T, size int) {
 // The queue page still shows the cut listing, with the note.
 func TestQueueIDsRefusesATruncatedListing(t *testing.T) {
 	fakePostqueue(t, 5<<20)
-	if ids, err := QueueIDs(); err == nil {
-		t.Fatalf("QueueIDs on a 5 MiB listing = %d ids and no error, want an error", len(ids))
+	if ids, err := QueueIDs(); err == nil || !strings.Contains(err.Error(), "listing is larger than") {
+		t.Fatalf("QueueIDs on a 5 MiB listing = %d ids, %v; want the listing refused for its size", len(ids), err)
 	}
 	text, err := Queue()
 	if err != nil {

internal/store/deliverylog.go

@@ -12,9 +12,17 @@ import (
 // deliveryLogKeepDays is how long a day of the counters is kept.
 const deliveryLogKeepDays = 366
 
-// addDeliveryLogDay adds n to one counter of the day of at. column is one of
-// the fixed names below, never anything a request carries.
+// deliveryLogCounters are the columns addDeliveryLogDay may add to: the
+// name is put into the statement, so nothing else may reach it.
+var deliveryLogCounters = map[string]bool{
+	"reconcile_closed": true, "reconcile_missed": true, "tailer_blind_seconds": true,
+}
+
+// addDeliveryLogDay adds n to one counter of the day of at.
 func (s *Store) addDeliveryLogDay(at time.Time, column string, n int64) error {
+	if !deliveryLogCounters[column] {
+		return fmt.Errorf("delivery log day: no counter %q", column)
+	}
 	if n <= 0 {
 		return nil
 	}
@@ -34,6 +42,12 @@ func (s *Store) AddReconcileClosed(at time.Time, n int64) error {
 	return s.addDeliveryLogDay(at, "reconcile_closed", n)
 }
 
+// AddReconcileMissed counts n rows the sweep closed whose delivery line was
+// in mail.log all along: the tailer did not read it.
+func (s *Store) AddReconcileMissed(at time.Time, n int64) error {
+	return s.addDeliveryLogDay(at, "reconcile_missed", n)
+}
+
 // AddTailerBlind counts a stretch the log tailer could not read mail.log,
 // on the day it began. Under a second counts nothing.
 func (s *Store) AddTailerBlind(from time.Time, d time.Duration) error {
@@ -71,14 +85,14 @@ func (s *Store) CountQueuedOlderThan(cutoff time.Time) (int64, error) {
 
 // DeliveryLogDay is one day of the counters, for tests and for reading a copy.
 type DeliveryLogDay struct {
-	Day                                       string
-	Closed, Contradicted, BlindSeconds, Stuck int64
+	Day                                               string
+	Closed, Missed, Contradicted, BlindSeconds, Stuck int64
 }
 
 // DeliveryLogDays returns the counted days, oldest first.
 func (s *Store) DeliveryLogDays() ([]DeliveryLogDay, error) {
 	rows, err := s.db.Query(
-		`SELECT day, reconcile_closed, reconcile_contradicted, tailer_blind_seconds, queued_over_lifetime
+		`SELECT day, reconcile_closed, reconcile_missed, reconcile_contradicted, tailer_blind_seconds, queued_over_lifetime
 		 FROM delivery_log_days ORDER BY day`)
 	if err != nil {
 		return nil, fmt.Errorf("delivery log days: %w", err)
@@ -87,7 +101,7 @@ func (s *Store) DeliveryLogDays() ([]DeliveryLogDay, error) {
 	var out []DeliveryLogDay
 	for rows.Next() {
 		var d DeliveryLogDay
-		if err := rows.Scan(&d.Day, &d.Closed, &d.Contradicted, &d.BlindSeconds, &d.Stuck); err != nil {
+		if err := rows.Scan(&d.Day, &d.Closed, &d.Missed, &d.Contradicted, &d.BlindSeconds, &d.Stuck); err != nil {
 			return nil, fmt.Errorf("scan delivery log day: %w", err)
 		}
 		out = append(out, d)

internal/store/deliverylog_test.go

@@ -21,6 +21,7 @@ func TestDeliveryLogDays(t *testing.T) {
 	must(st.AddReconcileClosed(day1, 2))
 	must(st.AddReconcileClosed(day1, 3))
 	must(st.AddReconcileClosed(day1, 0))
+	must(st.AddReconcileMissed(day1, 2))
 	must(st.AddTailerBlind(day1, 1500*time.Millisecond))
 	must(st.AddTailerBlind(day1, 900*time.Millisecond)) // under a second: nothing
 	must(st.SetQueuedOverLifetime(day1, 4))
@@ -30,7 +31,7 @@ func TestDeliveryLogDays(t *testing.T) {
 	days, err := st.DeliveryLogDays()
 	must(err)
 	want := []DeliveryLogDay{
-		{Day: "2026-10-14", Closed: 5, BlindSeconds: 1, Stuck: 1},
+		{Day: "2026-10-14", Closed: 5, Missed: 2, BlindSeconds: 1, Stuck: 1},
 		{Day: "2026-10-15", BlindSeconds: 60},
 	}
 	if len(days) != len(want) {
@@ -50,6 +51,14 @@ func TestDeliveryLogDays(t *testing.T) {
 	}
 }
 
+// Only the counters may be added to by name.
+func TestAddDeliveryLogDayRefusesAnotherColumn(t *testing.T) {
+	st := openTestStore(t)
+	if err := st.addDeliveryLogDay(time.Now(), "day", 1); err == nil {
+		t.Fatal("addDeliveryLogDay took the column day")
+	}
+}
+
 func TestCountQueuedOlderThan(t *testing.T) {
 	st := openTestStore(t)
 	for _, e := range []SendLogEntry{{QueueID: "Q1", To: "a@example.net"}, {QueueID: "Q2", To: "b@example.net"}} {

internal/store/migrations/0007_delivery_log_days.sql

@@ -3,16 +3,24 @@
 -- numbers the reconcile sweep's fate is decided by; no page reads them, they
 -- are read off a copy of the database. DDL only; nothing existing changes.
 --   reconcile_closed        rows the sweep closed as bounced (reason 'reconcile')
---   reconcile_contradicted  rows closed by the sweep that a delivery line later
---                           rewrote - the log wins, and is counted here
+--   reconcile_missed        of those, rows whose delivery line was in mail.log
+--                           (or mail.log.1) when the sweep closed them: the
+--                           tailer had not read it
+--   reconcile_contradicted  rows closed by the sweep that a delivery line read
+--                           later gave a status (reason becomes
+--                           'reconcile-contradicted'). closed - missed -
+--                           contradicted is what the log never resolved: what
+--                           removing the sweep would leave queued
 --   tailer_blind_seconds    seconds the log tailer could not read mail.log
 --                           (the file gone, reopening it failing)
---   queued_over_lifetime    rows still queued past Postfix's queue lifetime,
---                           as measured by the day's last sweep (a level, not
+--   queued_over_lifetime    rows still queued past Postfix's queue lifetime
+--                           and a backoff, as the day's last sweep saw them -
+--                           rows stuck although the sweep runs (a level, not
 --                           a sum)
 CREATE TABLE delivery_log_days (
     day                    TEXT PRIMARY KEY,
     reconcile_closed       INTEGER NOT NULL DEFAULT 0,
+    reconcile_missed       INTEGER NOT NULL DEFAULT 0,
     reconcile_contradicted INTEGER NOT NULL DEFAULT 0,
     tailer_blind_seconds   INTEGER NOT NULL DEFAULT 0,
     queued_over_lifetime   INTEGER NOT NULL DEFAULT 0

internal/store/schema_test.go

@@ -60,7 +60,7 @@ func TestBaselineSchema(t *testing.T) {
 		"send_log_lines":        "id queue_id logged_at line",
 		"notify_state":          "key phase count mark since sent_at",
 		"sender_check_days":     "day down_seconds deferred",
-		"delivery_log_days":     "day reconcile_closed reconcile_contradicted tailer_blind_seconds queued_over_lifetime",
+		"delivery_log_days":     "day reconcile_closed reconcile_missed reconcile_contradicted tailer_blind_seconds queued_over_lifetime",
 	}
 	rows, err := st.db.Query("SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%' ORDER BY name")
 	if err != nil {

internal/store/sendercheck.go

@@ -13,9 +13,15 @@ const (
 	ReasonFrom   = "from"   // the From: line is not the application's
 )
 
-// ReasonReconcile marks a "bounced" row the reconcile sweep closed (send_log.reason):
-// its message left Postfix's queue with no delivery line ever read for it.
-const ReasonReconcile = "reconcile"
+// Who decided a "bounced" row the log did not (send_log.reason).
+const (
+	// ReasonReconcile: the reconcile sweep closed the row - its message left
+	// Postfix's queue with no delivery line ever read for it.
+	ReasonReconcile = "reconcile"
+	// ReasonReconcileContradicted: the sweep closed it and a delivery line
+	// read afterwards gave the status the row now has.
+	ReasonReconcileContradicted = "reconcile-contradicted"
+)
 
 // Settings keys of the sender check's statistics.
 const (

internal/store/sendlog.go

@@ -84,10 +84,22 @@ func (s *Store) InsertRejected(e SendLogEntry) error {
 // number of rows updated so the caller can tell whether the line matched a
 // journal entry.
 //
+// Only the newest row of the pair is updated. Postfix reuses a short queue id
+// once the message has left the queue, and a row outlives that by months; a
+// line about today's message must not rewrite an old one (audit 2026-10-12,
+// review of stage 1).
+//
 // A row the reconcile sweep closed (reason "reconcile") that a delivery line
-// now speaks of was closed on a guess the log could have settled: the log
-// wins, the reason is cleared, and the day's reconcile_contradicted counts it
-// (delivery_log_days) - once, since the row no longer carries the reason.
+// now speaks of was closed on a guess the log could have settled: the log's
+// status wins, the reason becomes "reconcile-contradicted" - which row, and
+// updated_at says when - and the day's reconcile_contradicted counts it
+// (delivery_log_days), once, since the row no longer reads "reconcile".
+//
+// The transaction's first statement is a write. A transaction that reads
+// first and then writes fails at once with "database is locked" when another
+// connection - the dmarc-ingest process opens the same file - commits in
+// between, and the line would be lost; a write first takes the write lock,
+// and the busy timeout waits for it.
 func (s *Store) UpdateStatus(queueID, recipient, status string) (int64, error) {
 	now := time.Now().UTC()
 	tx, err := s.db.Begin()
@@ -95,18 +107,18 @@ func (s *Store) UpdateStatus(queueID, recipient, status string) (int64, error) {
 		return 0, fmt.Errorf("update send_log status: %w", err)
 	}
 	defer tx.Rollback()
-	var contradicted int64
-	if err := tx.QueryRow(
-		`SELECT COUNT(*) FROM send_log WHERE queue_id = ? AND to_addr = ? COLLATE NOCASE AND reason = ?`,
-		queueID, recipient, ReasonReconcile,
-	).Scan(&contradicted); err != nil {
+	const newest = `id = (SELECT MAX(id) FROM send_log WHERE queue_id = ? AND to_addr = ? COLLATE NOCASE)`
+	marked, err := tx.Exec(
+		`UPDATE send_log SET reason = ? WHERE `+newest+` AND reason = ?`,
+		ReasonReconcileContradicted, queueID, recipient, ReasonReconcile,
+	)
+	if err != nil {
 		return 0, fmt.Errorf("update send_log status: %w", err)
 	}
+	contradicted, _ := marked.RowsAffected()
 	res, err := tx.Exec(
-		`UPDATE send_log SET status = ?, updated_at = ?,
-			reason = CASE WHEN reason = ? THEN '' ELSE reason END
-		 WHERE queue_id = ? AND to_addr = ? COLLATE NOCASE`,
-		status, now.Format(time.RFC3339), ReasonReconcile, queueID, recipient,
+		`UPDATE send_log SET status = ?, updated_at = ? WHERE `+newest,
+		status, now.Format(time.RFC3339), queueID, recipient,
 	)
 	if err != nil {
 		return 0, fmt.Errorf("update send_log status: %w", err)
@@ -127,17 +139,16 @@ func (s *Store) UpdateStatus(queueID, recipient, status string) (int64, error) {
 	return n, nil
 }
 
-// CloseUnreported is the reconcile sweep's close of one row: "bounced", marked
-// reason "reconcile" so it is known who decided it (architecture.md § Log
-// tailer). Only a row still "queued" is closed - a result the log wrote an
-// instant earlier stands, and a months-old row of a reused queue id is not
-// rewritten (audit 2026-10-12, H3). It returns the number of rows closed.
-func (s *Store) CloseUnreported(queueID, recipient string) (int64, error) {
+// CloseUnreported is the reconcile sweep's close of one row, by its id:
+// "bounced", marked reason "reconcile" so it is known who decided it
+// (architecture.md § Log tailer). Only a row still "queued" is closed - a
+// result the log wrote an instant earlier stands (audit 2026-10-12, H3). It
+// returns the number of rows closed, 0 or 1.
+func (s *Store) CloseUnreported(id int64) (int64, error) {
 	now := time.Now().UTC().Format(time.RFC3339)
 	res, err := s.db.Exec(
-		`UPDATE send_log SET status = ?, reason = ?, updated_at = ?
-		 WHERE queue_id = ? AND to_addr = ? COLLATE NOCASE AND status = ?`,
-		StatusBounced, ReasonReconcile, now, queueID, recipient, StatusQueued,
+		`UPDATE send_log SET status = ?, reason = ?, updated_at = ? WHERE id = ? AND status = ?`,
+		StatusBounced, ReasonReconcile, now, id, StatusQueued,
 	)
 	if err != nil {
 		return 0, fmt.Errorf("close unreported send_log row: %w", err)
@@ -150,6 +161,7 @@ func (s *Store) CloseUnreported(queueID, recipient string) (int64, error) {
 // reduced to what the log-tailer's reconcile sweep needs to look it up in the
 // Postfix queue and, failing that, to close it (architecture.md § Log tailer).
 type QueuedDelivery struct {
+	ID      int64
 	QueueID string
 	To      string
 }
@@ -164,7 +176,7 @@ type QueuedDelivery struct {
 // format is chronologically correct.
 func (s *Store) ListQueuedOlderThan(cutoff time.Time) ([]QueuedDelivery, error) {
 	rows, err := s.db.Query(
-		`SELECT queue_id, to_addr FROM send_log
+		`SELECT id, queue_id, to_addr FROM send_log
 		 WHERE status = ? AND queue_id <> '' AND created_at < ?
 		 ORDER BY id`,
 		StatusQueued, cutoff.UTC().Format(time.RFC3339),
@@ -177,7 +189,7 @@ func (s *Store) ListQueuedOlderThan(cutoff time.Time) ([]QueuedDelivery, error)
 	var out []QueuedDelivery
 	for rows.Next() {
 		var d QueuedDelivery
-		if err := rows.Scan(&d.QueueID, &d.To); err != nil {
+		if err := rows.Scan(&d.ID, &d.QueueID, &d.To); err != nil {
 			return nil, fmt.Errorf("scan queued send_log row: %w", err)
 		}
 		out = append(out, d)

internal/store/sendlog_test.go

@@ -97,6 +97,16 @@ func TestUpdateStatusRecipientCaseInsensitive(t *testing.T) {
 	}
 }
 
+// rowID is the id of the newest send-log row of a queue id.
+func rowID(t *testing.T, st *Store, queueID string) int64 {
+	t.Helper()
+	var id int64
+	if err := st.db.QueryRow(`SELECT MAX(id) FROM send_log WHERE queue_id = ?`, queueID).Scan(&id); err != nil {
+		t.Fatal(err)
+	}
+	return id
+}
+
 // The sweep closes only what is still queued, and says it was the sweep: a
 // status the log wrote stands, whatever the sweep believed when it listed the
 // row.
@@ -111,10 +121,10 @@ func TestCloseUnreportedClosesOnlyQueuedRows(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	if n, err := st.CloseUnreported("Q1", "lost@example.net"); err != nil || n != 1 {
+	if n, err := st.CloseUnreported(rowID(t, st, "Q1")); err != nil || n != 1 {
 		t.Fatalf("CloseUnreported of a queued row = %d, %v; want 1", n, err)
 	}
-	if n, err := st.CloseUnreported("Q2", "done@example.net"); err != nil || n != 0 {
+	if n, err := st.CloseUnreported(rowID(t, st, "Q2")); err != nil || n != 0 {
 		t.Fatalf("CloseUnreported of a sent row = %d, %v; want 0", n, err)
 	}
 
@@ -133,14 +143,15 @@ func TestCloseUnreportedClosesOnlyQueuedRows(t *testing.T) {
 }
 
 // A delivery line about a row the sweep closed proves the sweep guessed what
-// the log could have said: the log's status wins, the reason goes, and the day
-// counts one contradiction - once, however often the line is read again.
+// the log could have said: the log's status wins, the row is marked
+// reconcile-contradicted, and the day counts one contradiction - once,
+// however often the line is read again.
 func TestUpdateStatusCountsAContradictedSweep(t *testing.T) {
 	st := openTestStore(t)
 	if err := st.InsertQueued(SendLogEntry{QueueID: "Q1", To: "a@example.net"}); err != nil {
 		t.Fatal(err)
 	}
-	if _, err := st.CloseUnreported("Q1", "a@example.net"); err != nil {
+	if _, err := st.CloseUnreported(rowID(t, st, "Q1")); err != nil {
 		t.Fatal(err)
 	}
 	for i := 0; i < 2; i++ {
@@ -152,8 +163,8 @@ func TestUpdateStatusCountsAContradictedSweep(t *testing.T) {
 	if err := st.db.QueryRow(`SELECT status, reason FROM send_log WHERE queue_id = 'Q1'`).Scan(&status, &reason); err != nil {
 		t.Fatal(err)
 	}
-	if status != StatusSent || reason != "" {
-		t.Fatalf("status %q reason %q, want sent and no reason", status, reason)
+	if status != StatusSent || reason != ReasonReconcileContradicted {
+		t.Fatalf("status %q reason %q, want sent and %s", status, reason, ReasonReconcileContradicted)
 	}
 	days, err := st.DeliveryLogDays()
 	if err != nil {
@@ -164,6 +175,32 @@ func TestUpdateStatusCountsAContradictedSweep(t *testing.T) {
 	}
 }
 
+// Postfix reuses a short queue id once its message has left the queue; a
+// line about the message of today rewrites today's row, not the months-old
+// one of the same id and recipient.
+func TestUpdateStatusRewritesOnlyTheNewestRow(t *testing.T) {
+	st := openTestStore(t)
+	for i := 0; i < 2; i++ {
+		if err := st.InsertQueued(SendLogEntry{QueueID: "REUSED", To: "a@example.net"}); err != nil {
+			t.Fatal(err)
+		}
+	}
+	old := rowID(t, st, "REUSED") - 1
+	if _, err := st.db.Exec(`UPDATE send_log SET status = ? WHERE id = ?`, StatusSent, old); err != nil {
+		t.Fatal(err)
+	}
+	if n, err := st.UpdateStatus("REUSED", "a@example.net", StatusBounced); err != nil || n != 1 {
+		t.Fatalf("UpdateStatus = %d, %v; want 1 row", n, err)
+	}
+	var status string
+	if err := st.db.QueryRow(`SELECT status FROM send_log WHERE id = ?`, old).Scan(&status); err != nil {
+		t.Fatal(err)
+	}
+	if status != StatusSent {
+		t.Fatalf("the old row became %q, want it left sent", status)
+	}
+}
+
 // The reconcile sweep asks for the rows old enough that Postfix should have
 // reported on them by now. A message accepted moments ago is simply in flight,
 // and one the milter refused never reached the queue at all, so neither is the
@@ -199,7 +236,7 @@ func TestListQueuedOlderThan(t *testing.T) {
 	if err != nil {
 		t.Fatalf("ListQueuedOlderThan: %v", err)
 	}
-	if len(got) != 1 || got[0] != (QueuedDelivery{QueueID: "OLD1", To: "stale@example.net"}) {
+	if len(got) != 1 || got[0] != (QueuedDelivery{ID: got[0].ID, QueueID: "OLD1", To: "stale@example.net"}) {
 		t.Fatalf("got %+v, want only the stale queued row", got)
 	}
 }