diff --git a/CHANGELOG.md b/CHANGELOG.md index 78ffb87..d29cbc1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,12 @@ milter chain, DMARC reports are validated before they are stored, and the auto rate limit is correct for a non-default level-1 window. Schema migration `0010` (index only). Upgrading is a tag bump. +### Removed + +- Plan documents for already-shipped features: `domain-stats-auto-ratelimit`, + `dmarc-reports`, `inbound-relay`, `panel-docs`, `queue-retries`, + `send-log-retention`. Only unimplemented plans remain in `docs/plans/`. + ### Added - Roadmap item and plan for `/preflight` installation check page diff --git a/docs/plans/dmarc-reports.md b/docs/plans/dmarc-reports.md deleted file mode 100644 index 82c8dad..0000000 --- a/docs/plans/dmarc-reports.md +++ /dev/null @@ -1,79 +0,0 @@ -# Plan: dmarc-reports - -**Status:** done — shipped in `[1.7.0]` (2026-08-18); security review (Fable) -of the ingest path pending. - ---- - -## Goal - -SelfPost **receives** DMARC aggregate reports on SMTP, parses them inside the -image, and **shows summaries in the panel** — pass/fail by source, hints when -`tighten p=` is reasonable. No external DMARC SaaS and no IMAP workflow for the -operator. - -## Scope - -**In:** -- Inbound SMTP for configured report addresses only (not a general backup-MX). -- gzip + XML aggregate parsing → SQLite summaries per sending domain. -- Panel page and/or per-domain section: recent reports, third-party senders, - delivery health of report ingestion. -- Reuse the `dmarc_report_email` setting (moved off the old `admin` table into - `settings` by migration `0005`) and `domains.dmarc_rua` for DNS templates; - when enabled, suggest a SelfPost-hosted report address. - -**Out:** -- Forensic reports (`ruf=`). -- Full dashboards, APIs, email alerting. -- Mailboxes for people (IMAP/POP3/webmail). - -## Architecture (sketch) - -1. Receiving MTAs → SMTP to SelfPost (hub MX). -2. Postfix virtual alias or dedicated listener → panel ingest worker. -3. Parse XML → `dmarc_reports` table (domain, reporter, counts, date). -4. Panel reads SQLite; links from domain DNS card. - -May share port-25 plumbing with [inbound-relay.md](inbound-relay.md) but must -remain a separate, opt-in feature that does not forward mail upstream. - -## Done when - -- Operator can point `rua=` at an address SelfPost accepts and see parsed - summaries in the panel within one reporting cycle. -- With the feature off, outbound-only behaviour is unchanged. -- Documented in [guide.md](../guide.md); migrations are backward-compatible. - -## Risks - -- Attack surface of accepting mail (mitigate: strict recipient allow-list). -- Report volume and retention (mitigate: caps + pruning). - -## Implementation checklist - -**Ingest path.** `DMARC_REPORTS_ENABLE=true` enables `smtp/inet` on 25 (shared -with inbound relay when both are on). Postfix `relay_domains` + -`transport_maps` route allow-listed recipients to a `dmarc-ingest` pipe -(`panel -dmarc-ingest`). `check_recipient_access` on `dmarc_recipients` is the -allow-list; no SASL, no local mailboxes. - -**Schema.** Migration `0008_dmarc_reports.sql`: `dmarc_reports` (summary per -aggregate) + `dmarc_report_records` (per-source rows). Dedup on -`(reporter, report_id, domain)`. - -**Retention.** Max 500 reports; drop older than 90 days; prune after each -ingest. - -Target version cut: **`1.7.0`** (MINOR). One commit per step; code only after -roadmap status is **agreed**. Expand the sketch sections above before step 1 -if still thin. See [development.md](../development.md) § Plan checklists. - -- [x] Expand plan: ingest path, `dmarc_reports` schema, retention caps — **Sonnet** -- [x] Opt-in inbound SMTP for report addresses only (allow-list) — **Opus** -- [x] Worker: gzip/XML parse → SQLite — **Opus** -- [x] Panel: domain roll-up + parsed report (panel-ui mockups) — **Sonnet** -- [x] Tie-in `dmarc_report_email` / `domains.dmarc_rua` — **Sonnet** -- [x] Tests and [guide.md](../guide.md) — **Sonnet** -- [ ] Security review ingest path — **Fable** -- [x] `go vet`, `go test` on touched packages — **Haiku** diff --git a/docs/plans/domain-stats-auto-ratelimit.md b/docs/plans/domain-stats-auto-ratelimit.md deleted file mode 100644 index 3c5d366..0000000 --- a/docs/plans/domain-stats-auto-ratelimit.md +++ /dev/null @@ -1,205 +0,0 @@ -# Plan: domain-stats-auto-ratelimit - -**Status:** done — shipped in `[1.6.0]` (2026-08-18); security review (Fable) pending -**Date:** 2026-08-17 -**Version:** `1.x` MINOR; migrations must stay compatible with `1.0.0`. - ---- - -## Goal - -Give the operator **30-day sending statistics** per domain and per application -(total volume, peak and average rate), and an optional **auto** level-2 rate -limit that sets `max_messages` from the average rate (avg × multiplier) over the -level-1 window. - -## Scope - -**In:** - -- Rolling **30-day** stats on each domain page (domain aggregate + per-app rows): - message count, peak msg/h, average msg/h. -- Level-2 rate limit mode **manual** (today) or **auto** for domain and - application scopes. -- Auto formula: `max_messages = ceil(avg_hourly × multiplier)`, window = - level-1 window (`RATE_LIMIT_WINDOW_SECONDS`); capped at level 1. -- Background recalculation (e.g. every 6 h, alongside send-log prune); milter - reads stored `max_messages` / `window_seconds` only (no aggregates on the - hot path). -- Panel UI: stats card, manual/auto toggle, multiplier field, read-only computed - limit in auto mode, optional «Recalculate now». -- RBAC: domain-admin sees stats and may configure auto/manual for assigned - domains only; same authz as existing rate-limit handlers. -- Tests, [guide.md](../guide.md), [CHANGELOG.md](../../CHANGELOG.md); security - review (Fable) for rate-limit path changes. - -**Out:** - -- Changing level 1 (Postfix env) — auto only fills level 2 under the L1 cap. -- Automated IP warmup schedules ([guide.md](../guide.md) § IP warmup stays - operator-driven). -- Prometheus/Grafana, alerting, APIs. -- Counting level-1 refusals or `rejected` rows as sent volume. -- Per-client-IP analytics. - -## Data source - -All metrics come from SQLite `send_log`, same rules as -[`CountMessages`](../../internal/store/ratelimits.go): - -- One message = one distinct `queue_id` (many recipients = one count). -- `status != rejected` (level-2 refusals never queued). -- Level-1 refusals are **not** in `send_log` — stats under-count refusals; - document in UI copy. - -Retention today is env `SEND_LOG_RETENTION_DAYS` (default 90). Stats use the -last **30 days** of rows still present. If retention < 30 days (after -[send-log-retention](send-log-retention.md)), the stats window is -`min(30, retention)` with a warning. - -### Metrics - -| Metric | Definition | -|---|---| -| **total** | `COUNT(DISTINCT queue_id)` in the stats window | -| **peak rate** | maximum messages in any **hourly** bucket in that window (msg/h) | -| **avg rate** | `total / hours_in_window`, where `hours_in_window = min(720, age of domain/app in hours, retention hours)` | - -Keys: `send_log.domain` (domain scope), `send_log.app_login` (application scope). - -## Architecture - -```mermaid -flowchart LR - milter["journal-milter"] --> sendLog["send_log"] - sendLog --> statsQuery["StatsQuery 30d"] - statsQuery --> domainPage["domain page stats card"] - statsQuery --> autoJob["auto recalc job"] - autoJob --> rateLimits["rate_limits"] - rateLimits --> milter -``` - -1. **[`internal/store/stats.go`](../../internal/store/stats.go)** — `SendStats` - with `Total`, `PeakPerHour`, `AvgPerHour`; `DomainSendStats(name, since)` / - `AppSendStats(login, since)`. -2. Hourly buckets: `strftime('%Y-%m-%d %H', created_at)` + `GROUP BY`; subquery - for peak; total via distinct `queue_id`. -3. Indexes `idx_send_log_domain` and `idx_send_log_created_at` exist; add - composite `(domain, created_at)` only if profiling shows need. -4. **Auto recalc** — panel goroutine (same interval as send-log prune): for each - `rate_limits` row with `mode = auto`, recompute `max_messages`, set - `auto_updated_at`. Milter unchanged except reading new columns via existing - `RateLimit` lookup. - -### Auto rate limit - -Extend [`RateLimit`](../../internal/store/ratelimits.go): - -```go -type RateLimit struct { - // existing: Scope, RefID, AllowedIPs, MaxMessages, WindowSeconds - Mode string // "manual" | "auto" - AutoMultiplier float64 // default 2.5 when Mode == "auto" - AutoUpdatedAt time.Time -} -``` - -**Formula:** - -``` -avg_hourly = total_messages_in_window / hours_in_window -max_messages = ceil(avg_hourly * auto_multiplier) -window_seconds = L1 window (not editable in auto mode) -max_messages = min(max_messages, L1 max) -``` - -When `total == 0`: auto limit stays **inactive** (same as empty manual limit); -UI explains that traffic is required before auto can apply. - -**Application overrides** ([`handlers_ratelimit.go`](../../internal/web/handlers/handlers_ratelimit.go)): - -- Trusted IPs required. -- Auto app ceiling **strictly above** domain limit when domain limit is active. -- Ceiling ≤ L1. - -**Fail-open:** store errors during recalc must not weaken enforcement of the -last successfully written limit; recalc failures are logged only. - -### Migration (`0006_rate_limit_auto.sql`) - -```sql -ALTER TABLE rate_limits ADD COLUMN mode TEXT NOT NULL DEFAULT 'manual' - CHECK (mode IN ('manual', 'auto')); -ALTER TABLE rate_limits ADD COLUMN auto_multiplier REAL; -ALTER TABLE rate_limits ADD COLUMN auto_updated_at TEXT; -``` - -Existing rows → `manual`. - -### Domain export - -Today rate limits are **not** exported. This plan adds them (including -`mode`, `auto_multiplier`) to domain transfer JSON — document as a boundary -change in [guide.md](../guide.md) § Export. - -## Panel UI - -- Domain page ([`domain_detail.html`](../../internal/web/view/templates/domain_detail.html)): - - **Sending statistics (30 days)** — total, peak msg/h, avg msg/h. - - Per-application stats in the app list. - - Rate limit: Manual / Auto, multiplier (e.g. 1.5–5.0, default 2.5), read-only - computed max/window in auto mode, «Recalculate now». -- Optional later: «30d» column on domain list (global admin only). - -## Tests - -- `internal/store/stats_test.go` — fixtures → total / peak / avg. -- `internal/store/ratelimits_test.go` — auto recalc, L1 cap, app > domain. -- Handler tests — auto form validation, multiplier bounds. -- Milter tests — enforced limit matches last recalculated values. - -`go test` / `go vet` on touched packages. - -## Done when - -- Domain and app 30-day stats visible on the domain page; domain-admin scoping - enforced. -- Manual/auto toggle works for domain and app; auto recalc updates `rate_limits` - and milter enforces stored ceilings. -- Zero-traffic auto stays inactive with clear UI copy. -- [guide.md](../guide.md) and [CHANGELOG.md](../../CHANGELOG.md) updated; - security review passed. - -## Risks - -- Heavy aggregation on large `send_log` tables — mitigate with indexes or - nightly rollups (phase 2). -- Stats without level-1 visibility — mitigate with operator-facing caveat. -- Auto limit too tight after a spike — multiplier is operator-tuned; show peak - alongside avg in auto UI. - -## Dependencies - -- [`send-log-retention`](send-log-retention.md) is a separate roadmap item but - should land before or in parallel so operators can set retention ≥ 30 days - from the panel. - -**Version:** `1.x` MINOR. - -## Implementation checklist - -Target version cut: **`1.6.0`** (MINOR). One commit per step; code only after -roadmap status is **agreed**. See [development.md](../development.md) § Plan -checklists. - -- [x] Migration `0007_rate_limit_auto.sql` (`mode`, `auto_multiplier`, `auto_updated_at`) — **Opus** -- [x] `internal/store/stats.go`: total / peak / avg over 30 days — **Opus** -- [x] Auto recalc job (6h): `ceil(avg × multiplier)`, L1 cap, fail-open on error — **Opus** -- [x] Extend `RateLimit` + handler forms (manual/auto) — **Opus** -- [x] Domain page stats card + per-app stats (`domain_detail.html`) — **Sonnet** -- [x] Domain export JSON includes rate limits — **Opus** -- [x] Milter tests — enforced limit matches stored ceiling — **Opus** -- [x] Store and handler tests — **Sonnet** -- [x] [guide.md](../guide.md) — **Sonnet** -- [ ] Security review rate-limit path — **Fable** (pending post-release) -- [x] `go vet`, `go test` on touched packages — **Haiku** diff --git a/docs/plans/inbound-relay.md b/docs/plans/inbound-relay.md deleted file mode 100644 index e660b27..0000000 --- a/docs/plans/inbound-relay.md +++ /dev/null @@ -1,189 +0,0 @@ -# Plan: inbound-relay (inbound relay) - -**Status:** done — shipped in `[1.4.0]` (2026-08-17) -**Version:** 1.4.0 MINOR (flag off is compatible; not a 2.x break). -**Order:** after queue-retries `[1.3.1]`. One checklist step remains: -security review of the inbound path (Fable). - ---- - -## Goal - -The ability to accept mail on port 25 for explicitly configured domains and -forward it to a given upstream backend (a backup-MX / relay-forwarder role), as -a **module disabled by default** that changes neither the behaviour nor the -attack surface of the base outbound relay. - -## What it is for (scenarios) - -- **Backup-MX** — accept mail while the domain's primary mail server is - temporarily unreachable, and hand it over when it comes back. -- **A front for a server without a public IP** — the operator runs their own - mail server which, for whatever reason, **cannot accept mail from the - internet itself** (no static or public IP, behind NAT, a private address, - inbound port 25 blocked, and so on). SelfPost, with a public IP and a correct - PTR, acts as the domain's public entry node (the MX points at it) and - forwards mail to that internal or otherwise unreachable server. - -## Scope boundary (critical — what this is NOT) - -- **IT IS:** acceptance on 25 for domains from an explicit list, plus - forwarding (relay/forward) to an upstream (`relay_domains` + - `transport_maps` + `relay_recipient_maps`). Postfix here is a pure forwarder, - with no local delivery. -- **IT IS NOT (out of scope, [product.md](../product.md)):** local delivery to - mailboxes, IMAP/POP3, webmail, Dovecot. No mailboxes at all. SelfPost also - **neither implements nor bundles** an anti-spam or anti-virus engine - (rspamd/ClamAV) — but, unlike the earlier wording, it does **not** push - filtering onto the backend either (see the "Anti-spam" section below): it - provides an attachment point for an external filter. - -## Why as an option / plugin - -- Accepting on port 25 changes the threat model (open relay for inbound, - backscatter, spam ingress). So it is **off** by default behind the - `INBOUND_RELAY_ENABLE=false` env flag; turning it on is a deliberate step by - the operator. -- Isolation: separate SQLite tables, separate panel handlers and pages, a - separate branch of config generation. With the flag off, the inbound - listener, the tables and the UI are absent — the base outbound path is - byte-for-byte unchanged. - -## What to do - -- The `INBOUND_RELAY_ENABLE` env flag (default false); when `true`, generate - the inbound service and its config from panel state the same way the rest of - the config is generated (`postfix-config.sh`). -- **`master.cf`:** an inbound `smtp inet` on 25 for accepting from the internet - (today 25 is used only for outbound delivery). Separate from 465/587: on 25 - SASL is **not** offered and sending outwards is **not** allowed — inbound - only, for `relay_domains`. -- **Anti-open-relay for inbound (mandatory):** the inbound smtpd's - `smtpd_relay_restrictions` / `smtpd_recipient_restrictions` accept mail - **only** for domains in `relay_domains` and **only** for known recipients - (`relay_recipient_maps`); everything else gets - `reject_unauth_destination` / `reject_unlisted_recipient`. An open relay, or - accepting "for anyone", is impossible. -- **Backscatter:** knowing the valid recipients is preferable (reject unknown - recipient at RCPT stage) so that bounces to non-existent addresses are never - generated. -- **The panel manages:** the list of inbound domains; for each one the upstream - destination (`host:port`, transport), an optional list of valid recipients, - and optional TLS to the upstream. Strict validation of domain, host and port - (whitelist), injection-safe writing of map files (as with - `sender_login_maps` in Phase 4), `os/exec` without a shell - ([security.md](../security.md)). -- **Milters:** OpenDKIM is not needed on the inbound path (we do not sign - someone else's inbound mail). The journal-milter can optionally be reused for - an inbound journal (extra work), or the inbound path can go without it in the - first stage; fail-open behaviour is preserved. -- **Rate limit / size:** a coarse per-client-IP limit (`anvil`, as L1) and - `message_size_limit` on the inbound smtpd. - -## Anti-spam (important, but optional) - -This is a valuable option, but it is **not mandatory**: some operators will be -perfectly served by **blind forwarding without filtering** — when the backend -can filter on content itself, when the upstream is trusted, or when the volume -and risk are low. So the anti-spam hook is **off** by default (an empty -`INBOUND_ANTISPAM_MILTER`), and the inbound relay is fully functional without -it. - -What matters is something else: where filtering is technically possible. With a -"blind" relay the destination backend sees **SelfPost's** address as the -connecting IP, not the original sender's, so everything on the backend that -depends on the origin IP breaks (DNSBL and reputation are checked against -SelfPost's IP; SPF returns fail, since SelfPost is not in the sending domain's -SPF). **The only point where the real client IP is still visible is the inbound -hop at SelfPost** — so for those who need filtering, it has to be *attachable -right here*, not delegated to a backend that has already lost the information. - -The attachment design: - -- **The anti-spam engine is a separate optional container** (rspamd or - similar), which the operator runs **only if this option is wanted** (the same - principle as the reverse proxy — a separate container outside the SelfPost - image). SelfPost **neither contains nor starts it** — the image and the "one - container, three processes" principle are unchanged, and - [product.md](../product.md)'s out-of-scope list is not violated (SelfPost - does not implement anti-spam). -- **SelfPost provides the attachment point:** a milter hook on the inbound - smtpd. The engine's address is set via env (for example, - `INBOUND_ANTISPAM_MILTER=inet:antispam:11332`, empty → the hook is off) and - is added to `smtpd_milters` for the **inbound path only** (not on 465/587). - Postfix passes the milter the real client IP, HELO and PTR — the filter sees - the true origin. `milter_default_action` for that milter is configurable - (fail-open vs tempfail); the default is to be decided during implementation. -- **A native backstop with no dependencies:** on that same inbound hop, - Postfix's own origin-IP facilities are available — `reject_rbl_client` - (DNSBL) and HELO/PTR checks — and they work even without an external - container. Plus preserving authentication results for downstream through ARC - or `Received`, where part of the filtering does remain on the backend. -- **docker-compose:** document an optional anti-spam sidecar fragment (like the - alternative reverse-proxy fragments) — the container comes up with the stack - only when the option is enabled. -- **Persistence:** new tables and map files under `/data` — they land in the - full backup automatically (Phase 9). Domain export/import can be extended - with the inbound configuration — optional, to be flagged. -- **DNS documentation:** an inbound domain needs an `MX` record pointing at the - server (unlike outbound, where no MX is required) — to be reflected in the - README's DNS section. - -## Security - -[security.md](../security.md): server-side input validation, escaped writes to -config files, `exec` without interpolation, no open relay, protection against -backscatter. - -## Done when - -With `INBOUND_RELAY_ENABLE=true` and a configured domain, mail arriving on port -25 for that domain is forwarded to the given upstream; mail for unconfigured -domains or recipients is rejected (not an open relay, no backscatter); with -`INBOUND_ANTISPAM_MILTER` set, inbound mail passes through the external filter -with the real origin IP (verified with a sidecar container), and with it empty -the hook stays out of the way; with `INBOUND_RELAY_ENABLE=false` the inbound -port, tables and UI are absent and the base outbound relay is unchanged; -`build`/`vet`/`test`/image green. - -## Risks - -- open relay / backscatter — removed by `relay_domains` + - `relay_recipient_maps` + `reject_unauth_destination`; -- the loss of the origin IP for filtering on the backend when forwarding — - removed by the anti-spam milter hook plus native DNSBL on the inbound hop, - where the origin IP is still visible; -- port 25 accepting mail widens the attack surface (off by default); -- semver: if the contract turns out incompatible (ports, backup, behaviour with - the flag off) a major `2.x` is possible; the decision comes after the - implementation. - -**External deployment dependency:** the optional anti-spam container — outside -the SelfPost image, brought up by the operator when the option is enabled. - -## Dependencies - -A finished outbound path (already implemented). Agreement obtained — see the -status above. - -## Implementation checklist - -Target version cut: **`1.4.0`** (MINOR). One commit per step; see -[development.md](../development.md) § Plan checklists. UI reference: -[panel-ui inbound mockups](../assets/panel-ui/inbound.html). - -- [x] Migration: inbound domain / recipient / transport tables under `/data` — **Opus** -- [x] `INBOUND_RELAY_ENABLE` (default false) in entrypoint + `postfix-config.sh` — **Opus** -- [x] `master.cf`: inbound `smtp inet` on 25; separate from 465/587 — **Opus** -- [x] Generate `relay_domains`, `transport_maps`, `relay_recipient_maps` (injection-safe) — **Opus** -- [x] `smtpd_relay_restrictions` / recipient maps — no open relay, no backscatter — **Opus** -- [x] `internal/store` CRUD + validation (domain, host, port) — **Opus** -- [x] Panel: list, add, domain detail, recipients, danger zone (mockups) — **Sonnet** -- [x] Rate limit + `message_size_limit` on inbound smtpd — **Opus** -- [x] Optional `INBOUND_ANTISPAM_MILTER` + compose fragment — **Opus** -- [x] DNS MX copy in README/guide; `.env.example` — **Sonnet** -- [x] Backup/export inbound config (per plan optional flag) — **Opus** -- [x] Unit + handler tests; image build and container smoke — **Opus** -- [x] [guide.md](../guide.md) and [security.md](../security.md) — **Sonnet** -- [ ] Security review inbound path — **Fable** -- [x] `go vet`, `go test`, e2e if applicable — **Haiku** diff --git a/docs/plans/panel-docs.md b/docs/plans/panel-docs.md deleted file mode 100644 index 3250079..0000000 --- a/docs/plans/panel-docs.md +++ /dev/null @@ -1,54 +0,0 @@ -# Plan: panel-docs (in-panel operator documentation) - -**Status:** agreed -**Date:** 2026-08-17 -**Version:** `1.x` MINOR; `candidate` until explicitly agreed. - ---- - -## Goal - -Built-in operator documentation in the panel — short pages or a help drawer that -explain what each Status check and other controls mean, without sending the -operator out to [guide.md](../guide.md). - -## Scope - -**In:** - -- Help drawer or short help pages (CSS-checkbox pattern from - [panel-ui mockups](../assets/panel-ui/system.html)). -- Seed content: Status blurbs removed from cards for a denser layout — Machine - (kernel counters / rate window), TLS certificate (port 465, reverse-proxy - mount), Hostname / reverse DNS (FCrDNS, PTR at the hosting provider), and - similar notes for other surfaces as inline commentary is removed. -- «?» entry points on domain cards (mockups). - -**Out:** - -- A second copy of the full operator guide. -- Translation workflow beyond English (same as the rest of the panel). - -## Done when - -An operator can open help from the panel for those topics; the removed Status -blurbs are preserved there (or equivalent); no requirement to read the git -tree for day-to-day meaning of a card. - -## Risks - -Copy ownership and keeping help in sync when checks change; not bloating every -page with a second column of prose. - -## Implementation checklist - -Target version cut: **`1.8.0`** (MINOR). One commit per step; code only after -roadmap status is **agreed**. See [development.md](../development.md) § Plan -checklists. - -- [x] Help drawer / pages shell (CSS checkbox pattern from mockups) — **Sonnet** -- [x] Seed Status blurbs (Machine, TLS, PTR, …) — **Sonnet** -- [x] «?» entry points on domain cards — **Sonnet** -- [x] [guide.md](../guide.md) boundary: in-panel help vs full guide — **Sonnet** -- [x] Template tests — **Sonnet** -- [x] `go vet`, `go test` on touched packages — **Haiku** diff --git a/docs/plans/queue-retries.md b/docs/plans/queue-retries.md deleted file mode 100644 index 4c0227e..0000000 --- a/docs/plans/queue-retries.md +++ /dev/null @@ -1,156 +0,0 @@ -# Plan: queue-retries (Postfix retry policy in the panel) - -**Status:** done — shipped in `[1.3.1]` (2026-08-17) -**Date:** 2026-08-13 -**Version:** patch; no schema, no configuration surface. -**Order:** small panel item; does not wait on inbound-relay. - ---- - -## Goal - -Show the operator how Postfix retries deferred mail: first retry delay, later -backoff cap, and how long a message stays in the queue before it bounces. The -numbers come from this container's effective Postfix config, not from -hard-coded copy. - -## Scope - -**In:** - -- A static «How delivery retries work» card on `/mail-queue` (global - administrator). Not inside the HTMX poll fragment — the snapshot is taken at - panel start. -- The same human-readable intervals in `/deliveries/{id}` history for - `deferred` and `bounced` (domain administrators never see Mail queue). -- Operator docs: [guide.md](../guide.md) Mail queue bullet; - [architecture.md](../architecture.md) notes the one-shot `postconf -h` at - panel start. [CHANGELOG.md](../../CHANGELOG.md) `### Added`. - -**Out:** - -- Changing Postfix retry parameters, or exposing them as panel settings - ([product.md](../product.md): Postfix is used as-is). -- An `attempts` column on `send_log`, or «attempt 3 of N» — Postfix has no - attempt budget; it is time-based. -- Counting `status=deferred` lines in `mail.log` or reading `postcat`. Each - attempt is already on the delivery page's log table; the journal stores only - the last status. -- Duplicating the card on Status (summary + link to Mail queue already exist). -- Re-reading `postconf` on every HTTP request. - -## Architecture - -Acceptance is still synchronous SMTP. Delivery stays in Postfix's on-disk -queue. SelfPost does not enqueue, retry, or deliver. - -```mermaid -flowchart LR - smtpAccept["SMTP 250 accepted"] --> postfixQueue["Postfix queue"] - postfixQueue -->|"deferred: backoff until queue lifetime"| retry["Retry MX"] - retry -->|sent| done["sent"] - retry -->|lifetime elapsed or 5xx| bounce["bounced"] - postfixQueue --> mailQueuePage["/mail-queue policy card"] - postfixQueue --> deliveryPage["/deliveries/id history copy"] -``` - -[build/postfix-config.sh](../../build/postfix-config.sh) does not set -`queue_run_delay`, `minimal_backoff_time`, `maximal_backoff_time`, -`maximal_queue_lifetime`, `bounce_queue_lifetime`, or `delay_warning_time`. -Debian/Postfix 3.x compiled-in defaults therefore apply unless the operator -overrides them (`postconf -e` inside the container). - -### Loading the numbers - -Once, when the HTTP role starts ([cmd/panel/httpserver.go](../../cmd/panel/httpserver.go), -after `postfix-config.sh` has run): - -``` -postconf -h queue_run_delay minimal_backoff_time maximal_backoff_time maximal_queue_lifetime bounce_queue_lifetime delay_warning_time -``` - -`postconf -h`, not a parse of `/etc/postfix/main.cf`: stock values are not -written to the file. `postconf` is the effective config, including a manual -override. - -- Fixed argv, no user input — same pattern as - [postfix.Queue](../../internal/postfix/queue.go) / - [security.md](../security.md). -- Cache on `handlers.Config` (via `web.Config`). The HTMX fragment does not - call `postconf`. -- A live `postconf -e` is visible after the next panel (or container) restart. - While the process is up, the panel shows the start-up snapshot. -- Parse Postfix time units (`300s`, `5d`, `1h`, a bare number is seconds) in - `internal/postfix`. Format human strings (`5 minutes`, `5 days`, - `about 1 hour 7 minutes`) in one place so the Mail queue card and - `deliveryEvents` cannot drift. -- If `postconf` is missing (unit tests on Windows, binary outside the - container): log a warning, fall back to Postfix 3.x compiled-in defaults - (`300s` / `4000s` / `5d` / `0`), and put a muted note on the card. Tests - stub the lookup (as `queueIDs` in the log-tailer) or pass a fixture on - `Config`. Do not fail panel start. - -Typical stock values, for orientation only — the UI prints whatever -`postconf` returned: - -| Parameter | Stock | Meaning | -|---|---|---| -| `queue_run_delay` / `minimal_backoff_time` | `300s` | First retry and deferred-queue scan | -| `maximal_backoff_time` | `4000s` | Cap on the doubling gap (~1 h 7 min) | -| `maximal_queue_lifetime` | `5d` | Then bounce | -| `delay_warning_time` | `0` | No delay warning to the sender | - -## Panel copy - -Mail queue card facts: first retry; later retries (doubling, capped); -kept in queue; then bounced. Short prose: there is no fixed attempt count; a -`deferred` message stays in this listing until it is delivered or the queue -lifetime runs out. - -`deliveryEvents(row, policy)`: - -- `deferred`: retries, first after X, then with increasing gaps up to Y, for - up to Z. -- `bounced`: or Postfix gave up after Z in the queue. - -## Tests - -- Duration parser: `5d`, `300s`, `4000s`, `1h`, `0`, bare number. -- `/mail-queue` handler: card shows the fixture policy's human strings, not a - live `postconf`. -- Delivery page / `deliveryEvents`: `deferred` and `bounced` contain those - strings ([handlers_monitor_test.go](../../internal/web/handlers/handlers_monitor_test.go)). -- [templates_test.go](../../internal/web/view/templates_test.go): pass the new - fields if rendering `mail_queue` requires them. - -`go test` / `go vet` on the touched packages. - -## Done when - -- `/mail-queue` states this Postfix's first retry, backoff cap, and queue - lifetime. -- A `deferred` / `bounced` delivery page uses the same intervals. -- A manual `postconf -e maximal_queue_lifetime=2d` followed by a panel restart - changes what the panel prints. -- Guide and architecture describe the snapshot; CHANGELOG has an Added entry. - -## Risks - -- Showing compiled-in fallbacks when `postconf` failed would mislead if the - operator had overridden them — mitigate with the muted note on the card. -- Inventing a max-attempt count would be false; the copy must stay time-based. - -## Implementation checklist - -Target version cut: **`1.3.1`** (PATCH). One commit per step; see -[development.md](../development.md) § Plan checklists. - -- [x] `internal/postfix`: parse Postfix time units (`5d`, `300s`, bare seconds) + tests — **Opus** -- [x] `internal/postfix`: one-shot `postconf -h` (six keys), fallback + warn — **Opus** -- [x] Load policy at HTTP start in `cmd/panel/httpserver.go`; cache on handlers config — **Opus** -- [x] Human-readable duration formatter (shared by Mail queue card and delivery history) — **Sonnet** -- [x] «How delivery retries work» card on `/mail-queue` (outside HTMX fragment) — **Sonnet** -- [x] `deliveryEvents(row, policy)` — intervals in deferred/bounced copy — **Sonnet** -- [x] Handler and template tests (`handlers_monitor_test.go`, `templates_test.go`) — **Sonnet** -- [x] [guide.md](../guide.md) and [architecture.md](../architecture.md) — **Sonnet** -- [x] `go vet`, `go test` on touched packages — **Haiku** diff --git a/docs/plans/send-log-retention.md b/docs/plans/send-log-retention.md deleted file mode 100644 index 9493d5f..0000000 --- a/docs/plans/send-log-retention.md +++ /dev/null @@ -1,139 +0,0 @@ -# Plan: send-log-retention - -**Status:** done — shipped in `[1.5.0]` (2026-08-18) -**Date:** 2026-08-17 -**Version:** `1.5.0` MINOR; no schema migration required (uses existing `settings` table). - ---- - -## Goal - -Let the **global administrator** change how long delivery journal rows -(`send_log`, `/deliveries`) are kept, from the panel — without editing `.env`. - -## Context (as-built) - -Retention **already exists**, but only via environment: - -- `SEND_LOG_RETENTION_DAYS` (default **90**) in `.env` / Compose. -- [`cmd/panel/main.go`](../../cmd/panel/main.go) passes it to - [`logtail.Run`](../../internal/logtail/logtail.go). -- [`retentionLoop`](../../internal/logtail/logtail.go) prunes via - [`DeleteSendLogBefore`](../../internal/store/sendlog.go) every **6 hours**. -- No panel control; [`handlers_monitor.go`](../../internal/web/handlers/handlers_monitor.go) - hardcodes «ninety days» in copy. -- Migration `0001_init.sql` describes `settings` as the place for «retention - overrides», but no UI writes that key yet. - -This plan moves the **effective** retention into SQLite `settings`, with env as -bootstrap only. - -## Scope - -**In:** - -- Settings card on `/settings` (global administrator only): **Send log - retention (days)**. -- Key `send_log_retention_days` in [`settings`](../../internal/store/settings.go). -- Validation: integer range **7–365** (exact bounds fixed at implementation). -- On first use: if setting missing, seed from env - (`SEND_LOG_RETENTION_DAYS`, default 90) at panel start or first save. -- Log-tailer reads the setting **each prune cycle** (no container restart). -- Delivery pages and guide copy show the **current** retention, not a hardcoded - 90. -- Tests; [guide.md](../guide.md); [CHANGELOG.md](../../CHANGELOG.md). - -**Out:** - -- Per-domain retention (instance-wide only). -- `mail.log` rotation (logrotate, 14 daily files — unchanged). -- Immediate prune on save when lowering retention (next 6 h cycle is enough; - optional «Prune now» not in v1). -- Domain-admin access to this setting. - -## Architecture - -```mermaid -flowchart LR - settingsPage["/settings form"] --> sqlite["settings.send_log_retention_days"] - env["SEND_LOG_RETENTION_DAYS bootstrap"] --> sqlite - sqlite --> retentionLoop["logtail retentionLoop"] - retentionLoop --> prune["DeleteSendLogBefore"] -``` - -1. **Read path** — `GetSendLogRetentionDays()`: settings value if valid, else env - default. -2. **Write path** — POST `/settings` (global admin): validate, `SetSetting`, - flash confirmation. -3. **Prune path** — change [`logtail.retentionLoop`](../../internal/logtail/logtail.go) - to accept `func() int` or `RetentionReader` that queries settings each cycle - (same 6 h ticker). -4. **Copy** — inject retention days into delivery list/detail templates and - remove hardcoded «ninety days». - -`SEND_LOG_RETENTION_DAYS` remains documented in [guide.md](../guide.md) as the -**initial default** until changed in Settings. - -## Relation to domain-stats-auto-ratelimit - -[domain-stats-auto-ratelimit](domain-stats-auto-ratelimit.md) uses a **30-day** -stats window. Requires effective retention ≥ 30 for full accuracy. When -retention < 30: - -- Stats UI shows a warning and uses `min(30, retention)` as the window, or -- Settings validation warns when saving a value below 30 while stats/auto are - enabled (pick one at implementation; document in guide). - -Recommended roadmap order: **send-log-retention** before or parallel with -domain-stats-auto-ratelimit. - -## Panel UI - -New card on [`settings.html`](../../internal/web/view/templates/settings.html) -(global admin block, near rate limits or under a «Deliveries» heading): - -- Number input: retention days (7–365). -- Muted copy: rows older than this are deleted from `/deliveries`; main driver - of `/data` growth; does not affect `mail.log` rotation. - -Domain administrators keep the narrow credentials-only settings page. - -## Tests - -- Save/load setting; reject out-of-range values. -- `retentionLoop` uses updated value without process restart (mock reader). -- Bootstrap: empty settings → env default used for prune. -- Template/delivery copy reflects configured days. - -`go test` / `go vet` on touched packages. - -## Done when - -- Global admin can set retention on `/settings`; value persists in SQLite. -- Prune uses the panel value on the next cycle; env remains bootstrap default. -- Guide documents panel vs env; CHANGELOG entry added. -- Hardcoded «ninety days» removed from delivery UI. - -## Risks - -- Operator lowers retention while bookmarking old delivery URLs — existing - behaviour; copy already notes pruned rows are gone. -- Settings change without restart — must be tested so log-tailer never keeps a - stale int from panel start only. - -**Version:** `1.x` MINOR. - -## Implementation checklist - -Target version cut: **`1.5.0`** (MINOR). One commit per step; code only after -roadmap status is **agreed**. See [development.md](../development.md) § Plan -checklists. - -- [x] `GetSendLogRetentionDays` / `SetSetting` key `send_log_retention_days` (7–365) — **Opus** -- [x] Bootstrap from `SEND_LOG_RETENTION_DAYS` when settings empty — **Opus** -- [x] `logtail.retentionLoop`: read setting each prune cycle — **Opus** -- [x] Settings card on `/settings` (`settings.html`) — **Sonnet** -- [x] Remove hardcoded «ninety days» in handlers and templates — **Sonnet** -- [x] Tests: save/load, range, loop without restart — **Sonnet** -- [x] [guide.md](../guide.md) — **Sonnet** -- [x] `go vet`, `go test` on touched packages — **Haiku**