From a6845ae9482738b68b8984adc1716008f779353f Mon Sep 17 00:00:00 2001 From: Mix Date: Mon, 12 Oct 2026 10:15:47 +0300 Subject: [PATCH] delivery-log-storage: architecture and guide MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit architecture.md § Log tailer gains "Stored lines": what is kept and why the queue id is read from the head of a line, why the tailer does not check for a row before writing, the one prune rule that is the table's retention, how a row is given its lines when Postfix reuses queue ids, the write-once rule while catching up, and the size. QueueLines is described as what it is now, the fallback for rows without stored lines. guide.md: the message page keeps its lines after rotation; retention also bounds the kept lines, which are in a full backup. Docs only. Co-Authored-By: Claude Opus 5.5 --- docs/architecture.md | 53 ++++++++++++++++++++++++++---- docs/guide.md | 8 +++-- docs/plans/delivery-log-storage.md | 2 +- docs/roadmap.md | 2 +- 4 files changed, 54 insertions(+), 11 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 1bbb27b..696c36e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -184,6 +184,8 @@ One process, four roles: 3. **log-tailer** — follows `MAIL_LOG`, updates send-log delivery status by queue-id. Send-log `queued → sent` transitions depend on this goroutine alone (`UpdateStatus` is only called from [internal/logtail](../internal/logtail/logtail.go)). + It also keeps the lines Postfix writes under a queue id, for the message's + page (§ Log tailer, *Stored lines*). 4. **rate-limit recalc** — every six hours (and on demand from Domain settings and the application form), recomputes level-2 **Auto** rate limits from send-log statistics. @@ -234,15 +236,52 @@ leaves every row untouched rather than closing them all. Now that the log survives the container, reaching this path means the lines are gone for good — rotated past fourteen files while the panel was down, or deleted. +**Stored lines** (`send_log_lines`, since 2.2.0; +[internal/logtail/lines.go](../internal/logtail/lines.go)). A send-log row +lives for the retention window, ninety days by default, and `mail.log` for a +day before it is rotated; a message's page (`/outbound/log/{id}`) shows what +Postfix wrote about it, so the tailer keeps those lines in the database as it +reads them: + +- **What is kept:** every line headed by a queue id — after the timestamp, the + host name and `program[pid]:` — with that id, the line's own time and the + line as written, cut to 4096 bytes. The id is read from the head of the line + only: the tail is the far end's text. `NOQUEUE` and `warning` stand in the + same place and are not ids; the pattern accepts Postfix's short form + (capital hexadecimal) and its long form (no vowels). +- **No check for a row at write time.** A message's first lines (`client=`, + `message-id=`) are logged before the journal milter writes its row at + end-of-message, so the tailer cannot ask whether the id is in the send log. + Lines of messages that never get a row — inbound relay mail, bounces Postfix + generates — are kept too, for a while. +- **Retention is one rule:** a line older than an hour whose queue id no + send-log row carries is deleted (`PruneSendLogLines`), at start, after every + send-log sweep and hourly between. That removes the lines of rows the sweep + deleted and the lines that never had a row. +- **Which lines are a row's.** Postfix reuses a queue id once the message has + left the queue, and a row outlives that by months. `SendLogLines` gives a row + the lines of its queue id dated from one hour before the row was written to + the queue lifetime (the larger of `maximal_queue_lifetime` and + `bounce_queue_lifetime`) plus a day after it. A message with several + recipients is several rows under one id, and each row's page shows all of + the message's lines. +- **Read twice, stored once.** The read offset is saved every few seconds, so + after an unclean stop some lines are read again; while the tailer is + catching up at start it writes a line only if the same line is not there. +- **Size:** six to ten lines of about 200 bytes per message — more than the row + itself. The table is in the database and therefore in a full backup, which + `log/` is not. + **Two one-shot reads** sit beside the follow loop and are unrelated to it, both serving panel pages on request: `TailLines` (the last *n* lines, for -`/server/log`) and `QueueLines` (the lines carrying one queue-id, for -`/outbound/log/{id}`). `QueueLines` scans a bounded tail of the current file — +`/server/log`) and `QueueLines` (the lines carrying one queue-id). `QueueLines` +is what a message's page falls back to when no lines are stored for its row — +a message accepted before 2.2.0. It scans a bounded tail of the current file — finding a message's lines means reading rather than seeking — and matches the id anchored on the character before it, since queue ids are hexadecimal runs and a -shorter one is regularly the tail of a longer one. Send-log rows outlive the log -(retention 90 days, rotation 14 files), so an empty result is the expected end -state for an older message and the page reports it as such, not as a failure. +shorter one is regularly the tail of a longer one. For such a message an empty +result is the expected end state once the log has rotated, and the page reports +it as such, not as a failure. --- @@ -617,6 +656,7 @@ The tables that carry the panel's model: (send-log retention). Nothing that belongs to a user. - `applications` (with `auth_ip_restrict` / `auth_allowed_ips`), `application_addresses`, `rate_limits` (no `allowed_ips` column), `send_log`, + `send_log_lines` (what Postfix logged under a queue id — § Log tailer), `logtail_state`, `inbound_domains`, `inbound_transports`, `inbound_recipients`, `dmarc_reports`, `dmarc_report_records`. @@ -624,7 +664,8 @@ The panel is the only writer; SQLite runs in WAL mode on one connection (see [Panel binary](#panel-binary-cmdpanel)), which is why a full backup snapshots with `VACUUM INTO` instead of copying the file. -**Rotation:** send-log retention `SEND_LOG_RETENTION_DAYS` (default 90); +**Rotation:** send-log retention `SEND_LOG_RETENTION_DAYS` (default 90), which +the stored log lines follow; `mail.log` via logrotate (14 rotated files, check every 6h, rename + `postfix reload` in `postrotate` — see § Log tailer above). diff --git a/docs/guide.md b/docs/guide.md index 04bd601..a86ffa5 100644 --- a/docs/guide.md +++ b/docs/guide.md @@ -215,7 +215,7 @@ expected to set; defaults match the code exactly. | `INBOUND_MESSAGE_SIZE_LIMIT` | Maximum message size in bytes on inbound smtpd (`message_size_limit`). When both inbound relay and DMARC ingest are on, the shared listener uses the **greater** of this and `DMARC_MESSAGE_SIZE_LIMIT`. | `26214400` (25 MiB) | `.env` | | `RATE_LIMIT_MESSAGES_PER_IP` | Level-1 backstop: maximum messages one client IP may submit per window (Postfix `smtpd_client_message_rate_limit`). See [Rate limiting — level 1](#rate-limiting--level-1-ip-backstop). | `100` | `.env` | | `RATE_LIMIT_WINDOW_SECONDS` | Level-1 window length in seconds (Postfix `anvil_rate_time_unit`). | `3600` | `.env` | -| `SEND_LOG_RETENTION_DAYS` | Initial default for how many days of send-log history are kept before the background sweep deletes rows — the main driver of `/data` growth over time. After the first panel start, change retention under **Server › Settings** (global role); the env value is only used to seed SQLite when the setting has never been saved. | `90` | `.env` | +| `SEND_LOG_RETENTION_DAYS` | Initial default for how many days of send-log history are kept before the background sweep deletes rows, and with them the `mail.log` lines kept for each message (1–2 KB a message) — the main driver of `/data` growth over time and of the size of a full backup. After the first panel start, change retention under **Server › Settings** (global role); the env value is only used to seed SQLite when the setting has never been saved. | `90` | `.env` | | `PANEL_SESSION_IDLE_DAYS` | Sliding idle timeout for the panel login session, in days. There is no absolute cap: a user who keeps coming back stays signed in indefinitely. | `7` | `.env` | | `SELFPOST_DNS_RESOLVERS` | Comma-separated recursive resolvers the panel's PTR/SPF/DKIM/DMARC checks query directly (so they report what the internet sees, not what this host's stub resolver synthesises). | `1.1.1.1:53`, `8.8.8.8:53`, `9.9.9.9:53` when unset | `.env` | | `TRUSTED_PROXY_CIDR` | Comma-separated CIDRs (bare IPs allowed) of reverse proxies allowed to supply `X-Forwarded-For` for login, setup, and account-change rate-limiting. **Leave unset unless you know the exact address of your reverse proxy.** A wrong value lets a client spoof its rate-limit key by sending a forged `X-Forwarded-For` header — the panel trusts the last hop only when the TCP peer matches one of these CIDRs. Behind the default Apache host-network setup this is typically the Docker bridge gateway, e.g. `172.18.0.1`. | *(empty — XFF ignored)* | `.env` | @@ -1144,8 +1144,10 @@ backoff cap, queue lifetime), the same numbers the Outbound queue shows; users with the *domain* role see them here because they cannot open the queue. Under both sit the `mail.log` lines for its queue id: the connection to the receiving server, the server's reply, and the status that reply was filed -as. Rows outlive `mail.log`, so an older message's lines may have rotated -away; the page says so. Retention is set under **Server › Settings** (global +as. The lines are kept in the database with the row, so they stay on the +page after `mail.log` has rotated and are deleted together with the row; a +message accepted before `2.2.0` shows them only while they are still in the +current file, and the page says so when there are none. Retention is set under **Server › Settings** (global role); `SEND_LOG_RETENTION_DAYS` in `.env` is only the initial default until it is changed there. diff --git a/docs/plans/delivery-log-storage.md b/docs/plans/delivery-log-storage.md index ce0c690..1e63e91 100644 --- a/docs/plans/delivery-log-storage.md +++ b/docs/plans/delivery-log-storage.md @@ -146,7 +146,7 @@ Target version cut: **`2.2.0`** (MINOR). One commit per step; - [x] Log tailer: store queue-id lines, write once while catching up, prune with the send-log sweep and hourly — **Opus** - [x] The two texts, mockup first: the *Outbound log* help topic and the empty note (`design:` commit) — **Opus** (the help topic is in the mockups; the empty note is not drawn there - the `out-message` mockup shows a message with lines - and changes with the page) - [x] Message page reads the table, falls back to the current `mail.log`; help text in `help.go` — **Opus** -- [ ] [architecture.md](../architecture.md), [guide.md](../guide.md), [schema-migrations.md](../schema-migrations.md) — **Opus** +- [x] [architecture.md](../architecture.md), [guide.md](../guide.md), [schema-migrations.md](../schema-migrations.md) — **Opus** (schema-migrations.md went in with the migration) - [ ] `go vet`, `go test`, e2e: a message's page still lists its lines after a forced rotation — **Opus** - [ ] Technical review — **Opus** - [ ] Security review — **Fable** diff --git a/docs/roadmap.md b/docs/roadmap.md index 303d506..59ecd99 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -36,7 +36,7 @@ in `git log` and [CHANGELOG.md](../CHANGELOG.md). | structured-logging | `log/slog` with levels and fields | candidate | — | — | | review-2026-08-followups | Remaining findings of the 2026-08-19 code review | candidate | — | — | | panel-notifications | E-mail notifications about important events | candidate | — | — | -| delivery-log-storage | Keep a message's `mail.log` lines with its send-log row | in progress | 4/8 | [plans/delivery-log-storage.md](plans/delivery-log-storage.md) | +| delivery-log-storage | Keep a message's `mail.log` lines with its send-log row | in progress | 5/8 | [plans/delivery-log-storage.md](plans/delivery-log-storage.md) | | password-reset | Password reset by e-mail | candidate | — | — | | phone-desktop-mode | A phone browser in "desktop site" mode gets the phone navigation | candidate | — | — | | panel-redesign | Panel redesign (accepted mockups, design contract) | shipped `2.0.0` | 27/27 | [plans/panel-redesign.md](plans/panel-redesign.md) |