mix/selfpost
1
0
Fork 0
Code Issues Pull Requests Actions 21 Packages Projects Releases Wiki Activity

delivery-log-storage: architecture and guide

Browse Source 
architecture.md § Log tailer gains "Stored lines": what is kept and why the
queue id is read from the head of a line, why the tailer does not check for a
row before writing, the one prune rule that is the table's retention, how a
row is given its lines when Postfix reuses queue ids, the write-once rule
while catching up, and the size. QueueLines is described as what it is now,
the fallback for rows without stored lines. guide.md: the message page keeps
its lines after rotation; retention also bounds the kept lines, which are in
a full backup.

Docs only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
mix
2026-10-12 10:15:47 +03:00
co-authored by Claude Opus 5.5
parent 1e6836b3c1
commit a6845ae948
4 changed files with 54 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+47 -6
View File
@@ -184,6 +184,8 @@ One process, four roles:
3. **log-tailer** — follows `MAIL_LOG`, updates send-log delivery status by
queue-id. Send-log `queued → sent` transitions depend on this goroutine alone
(`UpdateStatus` is only called from [internal/logtail](../internal/logtail/logtail.go)).
It also keeps the lines Postfix writes under a queue id, for the message's
page (§ Log tailer, *Stored lines*).
4. **rate-limit recalc** — every six hours (and on demand from Domain settings and
the application form), recomputes level-2 **Auto** rate limits from send-log statistics.
@@ -234,15 +236,52 @@ leaves every row untouched rather than closing them all. Now that the log
survives the container, reaching this path means the lines are gone for good —
rotated past fourteen files while the panel was down, or deleted.
**Stored lines** (`send_log_lines`, since 2.2.0;
[internal/logtail/lines.go](../internal/logtail/lines.go)). A send-log row
lives for the retention window, ninety days by default, and `mail.log` for a
day before it is rotated; a message's page (`/outbound/log/{id}`) shows what
Postfix wrote about it, so the tailer keeps those lines in the database as it
reads them:
- **What is kept:** every line headed by a queue id — after the timestamp, the
host name and `program[pid]:` — with that id, the line's own time and the
line as written, cut to 4096 bytes. The id is read from the head of the line
only: the tail is the far end's text. `NOQUEUE` and `warning` stand in the
same place and are not ids; the pattern accepts Postfix's short form
(capital hexadecimal) and its long form (no vowels).
- **No check for a row at write time.** A message's first lines (`client=`,
`message-id=`) are logged before the journal milter writes its row at
end-of-message, so the tailer cannot ask whether the id is in the send log.
Lines of messages that never get a row — inbound relay mail, bounces Postfix
generates — are kept too, for a while.
- **Retention is one rule:** a line older than an hour whose queue id no
send-log row carries is deleted (`PruneSendLogLines`), at start, after every
send-log sweep and hourly between. That removes the lines of rows the sweep
deleted and the lines that never had a row.
- **Which lines are a row's.** Postfix reuses a queue id once the message has
left the queue, and a row outlives that by months. `SendLogLines` gives a row
the lines of its queue id dated from one hour before the row was written to
the queue lifetime (the larger of `maximal_queue_lifetime` and
`bounce_queue_lifetime`) plus a day after it. A message with several
recipients is several rows under one id, and each row's page shows all of
the message's lines.
- **Read twice, stored once.** The read offset is saved every few seconds, so
after an unclean stop some lines are read again; while the tailer is
catching up at start it writes a line only if the same line is not there.
- **Size:** six to ten lines of about 200 bytes per message — more than the row
itself. The table is in the database and therefore in a full backup, which
`log/` is not.
**Two one-shot reads** sit beside the follow loop and are unrelated to it, both
serving panel pages on request: `TailLines` (the last *n* lines, for
`/server/log`) and `QueueLines` (the lines carrying one queue-id, for
`/outbound/log/{id}`). `QueueLines` scans a bounded tail of the current file —
`/server/log`) and `QueueLines` (the lines carrying one queue-id). `QueueLines`
is what a message's page falls back to when no lines are stored for its row —
a message accepted before 2.2.0. It scans a bounded tail of the current file —
finding a message's lines means reading rather than seeking — and matches the id
anchored on the character before it, since queue ids are hexadecimal runs and a
shorter one is regularly the tail of a longer one. Send-log rows outlive the log
(retention 90 days, rotation 14 files), so an empty result is the expected end
state for an older message and the page reports it as such, not as a failure.
shorter one is regularly the tail of a longer one. For such a message an empty
result is the expected end state once the log has rotated, and the page reports
it as such, not as a failure.
---
@@ -617,6 +656,7 @@ The tables that carry the panel's model:
(send-log retention). Nothing that belongs to a user.
- `applications` (with `auth_ip_restrict` / `auth_allowed_ips`),
`application_addresses`, `rate_limits` (no `allowed_ips` column), `send_log`,
`send_log_lines` (what Postfix logged under a queue id — § Log tailer),
`logtail_state`, `inbound_domains`, `inbound_transports`,
`inbound_recipients`, `dmarc_reports`, `dmarc_report_records`.
@@ -624,7 +664,8 @@ The panel is the only writer; SQLite runs in WAL mode on one connection (see
[Panel binary](#panel-binary-cmdpanel)), which is why a full backup snapshots
with `VACUUM INTO` instead of copying the file.
**Rotation:** send-log retention `SEND_LOG_RETENTION_DAYS` (default 90);
**Rotation:** send-log retention `SEND_LOG_RETENTION_DAYS` (default 90), which
the stored log lines follow;
`mail.log` via logrotate (14 rotated files, check every 6h, rename +
`postfix reload` in `postrotate` — see § Log tailer above).
docs/guide.md
+5 -3
View File
@@ -215,7 +215,7 @@ expected to set; defaults match the code exactly.
| `INBOUND_MESSAGE_SIZE_LIMIT` | Maximum message size in bytes on inbound smtpd (`message_size_limit`). When both inbound relay and DMARC ingest are on, the shared listener uses the **greater** of this and `DMARC_MESSAGE_SIZE_LIMIT`. | `26214400` (25 MiB) | `.env` |
| `RATE_LIMIT_MESSAGES_PER_IP` | Level-1 backstop: maximum messages one client IP may submit per window (Postfix `smtpd_client_message_rate_limit`). See [Rate limiting — level 1](#rate-limiting--level-1-ip-backstop). | `100` | `.env` |
| `RATE_LIMIT_WINDOW_SECONDS` | Level-1 window length in seconds (Postfix `anvil_rate_time_unit`). | `3600` | `.env` |
| `SEND_LOG_RETENTION_DAYS` | Initial default for how many days of send-log history are kept before the background sweep deletes rows — the main driver of `/data` growth over time. After the first panel start, change retention under **Server › Settings** (global role); the env value is only used to seed SQLite when the setting has never been saved. | `90` | `.env` |
| `SEND_LOG_RETENTION_DAYS` | Initial default for how many days of send-log history are kept before the background sweep deletes rows, and with them the `mail.log` lines kept for each message (1–2 KB a message) — the main driver of `/data` growth over time and of the size of a full backup. After the first panel start, change retention under **Server › Settings** (global role); the env value is only used to seed SQLite when the setting has never been saved. | `90` | `.env` |
| `PANEL_SESSION_IDLE_DAYS` | Sliding idle timeout for the panel login session, in days. There is no absolute cap: a user who keeps coming back stays signed in indefinitely. | `7` | `.env` |
| `SELFPOST_DNS_RESOLVERS` | Comma-separated recursive resolvers the panel's PTR/SPF/DKIM/DMARC checks query directly (so they report what the internet sees, not what this host's stub resolver synthesises). | `1.1.1.1:53`, `8.8.8.8:53`, `9.9.9.9:53` when unset | `.env` |
| `TRUSTED_PROXY_CIDR` | Comma-separated CIDRs (bare IPs allowed) of reverse proxies allowed to supply `X-Forwarded-For` for login, setup, and account-change rate-limiting. **Leave unset unless you know the exact address of your reverse proxy.** A wrong value lets a client spoof its rate-limit key by sending a forged `X-Forwarded-For` header — the panel trusts the last hop only when the TCP peer matches one of these CIDRs. Behind the default Apache host-network setup this is typically the Docker bridge gateway, e.g. `172.18.0.1`. | *(empty — XFF ignored)* | `.env` |
@@ -1144,8 +1144,10 @@ backoff cap, queue lifetime), the same numbers the Outbound queue shows; users
with the *domain* role see them here because they cannot open the queue. Under
both sit the `mail.log` lines for its queue id: the connection to the
receiving server, the server's reply, and the status that reply was filed
as. Rows outlive `mail.log`, so an older message's lines may have rotated
away; the page says so. Retention is set under **Server › Settings** (global
as. The lines are kept in the database with the row, so they stay on the
page after `mail.log` has rotated and are deleted together with the row; a
message accepted before `2.2.0` shows them only while they are still in the
current file, and the page says so when there are none. Retention is set under **Server › Settings** (global
role); `SEND_LOG_RETENTION_DAYS` in `.env` is only the initial default until it
is changed there.
docs/plans/delivery-log-storage.md
+1 -1
View File
@@ -146,7 +146,7 @@ Target version cut: **`2.2.0`** (MINOR). One commit per step;
- [x] Log tailer: store queue-id lines, write once while catching up, prune with the send-log sweep and hourly — **Opus**
- [x] The two texts, mockup first: the *Outbound log* help topic and the empty note (`design:` commit) — **Opus** (the help topic is in the mockups; the empty note is not drawn there - the `out-message` mockup shows a message with lines - and changes with the page)
- [x] Message page reads the table, falls back to the current `mail.log`; help text in `help.go` — **Opus**
- [ ] [architecture.md](../architecture.md), [guide.md](../guide.md), [schema-migrations.md](../schema-migrations.md) — **Opus**
- [x] [architecture.md](../architecture.md), [guide.md](../guide.md), [schema-migrations.md](../schema-migrations.md) — **Opus** (schema-migrations.md went in with the migration)
- [ ] `go vet`, `go test`, e2e: a message's page still lists its lines after a forced rotation — **Opus**
- [ ] Technical review — **Opus**
- [ ] Security review — **Fable**
docs/roadmap.md
+1 -1
View File
@@ -36,7 +36,7 @@ in `git log` and [CHANGELOG.md](../CHANGELOG.md).
| structured-logging | `log/slog` with levels and fields | candidate | — | — |
| review-2026-08-followups | Remaining findings of the 2026-08-19 code review | candidate | — | — |
| panel-notifications | E-mail notifications about important events | candidate | — | — |
| delivery-log-storage | Keep a message's `mail.log` lines with its send-log row | in progress | 4/8 | [plans/delivery-log-storage.md](plans/delivery-log-storage.md) |
| delivery-log-storage | Keep a message's `mail.log` lines with its send-log row | in progress | 5/8 | [plans/delivery-log-storage.md](plans/delivery-log-storage.md) |
| password-reset | Password reset by e-mail | candidate | — | — |
| phone-desktop-mode | A phone browser in "desktop site" mode gets the phone navigation | candidate | — | — |
| panel-redesign | Panel redesign (accepted mockups, design contract) | shipped `2.0.0` | 27/27 | [plans/panel-redesign.md](plans/panel-redesign.md) |