feat: log-tailer offset persistence + in-flight L2 rate-limit accounting (code-review.md § Phase 3)

- logtail: persist the read position (offset + fingerprint of the log's
  first 512 bytes) in a new logtail_state table (migration 0003) and
  resume from it on start, so delivery lines written while the panel was
  down are parsed instead of skipped and their send-log rows no longer
  stay "queued" forever. Fingerprint mismatch (rotated/recreated while
  down) reads the file from the start — re-parsing is idempotent; a
  first-ever start with nothing stored still begins at end-of-file.
  Writes are throttled to one per 5s, forced on rotation and shutdown.

- milter: count messages that passed the level-2 check but have not
  reached the send log yet (internal/milter/inflight.go), so concurrent
  SMTP sessions cannot each spend the same last slot. A literal
  count+insert transaction, as the review suggested, is not possible:
  the count happens at MAIL FROM and the insert at end-of-message.
  Reservations are released after the insert, on ABORT, and after a
  10-minute TTL — a client that drops mid-transaction must not be able
  to hold a slot, since the limiter is fail-open by design.

Docs: architecture.md (log tailer, persistence, L2 counting),
security.md and roadmap.md (restart gap closed, container recreate
remains), CHANGELOG, progress.md, code-review.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-06 17:12:37 +03:00
parent 4b0f537508
commit ed0a786739
15 changed files with 683 additions and 40 deletions
+20 -8
View File
@@ -81,7 +81,11 @@ One process, three roles:
proxy only.
2. **journal-milter** — unix socket `JOURNAL_MILTER_SOCKET`; records From/To/
Subject/SASL user at DATA; enforces level-2 rate limits; **fail-open**
(`default_action=accept`) so milter failure does not stop mail.
(`default_action=accept`) so milter failure does not stop mail. The level-2
count is the stored send-log rows plus the messages this process has admitted
but not yet written (`internal/milter/inflight.go`), so concurrent sessions
cannot each spend the same last slot; a reservation is released at
end-of-message, on ABORT, or after a 10-minute TTL.
3. **log-tailer** — follows `MAIL_LOG`, updates send-log delivery status by
queue-id. Send-log `queued → sent` transitions depend on this goroutine alone
(`UpdateStatus` is only called from [internal/logtail](../internal/logtail/logtail.go)).
@@ -99,15 +103,23 @@ panel user cannot read). `follow()` drains the old inode once more before
switching descriptors; the panel treats a missing log file as an empty tail, not
an error.
**Known gaps (same class of loss, not fixed by rename rotation):**
**Read offset is persisted** (`logtail_state` table, migration `0003`): the
tailer stores its position plus a fingerprint of the log's first 512 bytes, and
on start resumes from it, parsing the tail written while the panel was down. If
the fingerprint no longer matches (rotated or recreated in the meantime) it reads
the current file from the start; re-parsing lines is harmless because
`UpdateStatus` writes the same status onto the same row. Only a first-ever start,
with nothing stored, begins at end-of-file, so installing the panel does not
replay a pre-existing log.
**Remaining gap:**
- **Panel restart** — `follow()` starts at end-of-file; lines written while the
panel was down are never parsed; in-flight send-log rows may stay `queued`.
- **Container recreate** — `/var/log` is ephemeral; the log is lost with the
container.
container, so the delivery lines for rows still `queued` are gone with it and
those rows stay `queued` forever.
Possible follow-ups if these become painful: persist read offset across restarts,
mount mail log under `/data`, or reconcile stuck rows via `postqueue`.
Possible follow-ups if this becomes painful: mount the mail log under `/data`, or
reconcile stuck rows via `postqueue`.
---
@@ -158,7 +170,7 @@ cookie and idle timeout has not expired.
| Path | Contents |
|---|---|
| `selfpost.db` | SQLite: domains, apps, admin, sessions, send log, L2 limits |
| `selfpost.db` | SQLite: domains, apps, admin, sessions, send log, L2 limits, log-tailer offset |
| `setup-token` | First-run setup token file |
| `opendkim/` | DKIM keys + tables |
| `sasl/sasldb2` | Application SASL credentials |