release: 1.3.0: 1.3.0-1.3.1: security pass, queue persist, full backup layout
[1.3.0] Security and quality after 1.2.5: domain-admin send-log authorization, fail-closed sign-in and application delete, level-2 rate-limit race fix, restore Resync, expanded tests, operator docs, release CI, and OFL for IBM Plex. Upgrading from 1.2.x is a tag bump; no migration. Added: - licence: the SIL Open Font License 1.1 text now travels with the IBM Plex - docs: agreed roadmap item **queue-retries** — show this Postfix's retry Security: - The independent security review of the send-log authorization and Fixed: - test (e2e): send-log status scrapers follow the badge markup in - security (panel): the Deliveries list is scoped to a domain administrator's - mail (level-2 rate limit): the ceiling is no longer overshot by messages that - panel (sign-in): a session that cannot be written to the database no longer - panel (applications): deleting an application removes its SASL credentials … (2 more in CHANGELOG) Changed: - docs: operator and as-built docs aligned with the code after a full - docs: [guide.md](docs/guide.md) reorganised into **Installation**, **Instance - docs: the 2026-08-13 full-tree review plan is complete — every phase (P0–P7) - licence: [NOTICE](NOTICE) tells modifiers to update `SourceURL` in - ci: the release image is published only for a **published** GitHub Release … (9 more in CHANGELOG) [1.3.1] Retry policy in the panel, persistent Postfix queue, and self-contained full backups. Mail queue and delivery history show this Postfix's first retry, backoff cap, and queue lifetime. Deferred mail survives container recreate. Full backups now archive `data/`, compose, `.env`, and `certs/`; the archive layout breaks 1.3.0 flat backups (unpack those with `tar xzf backup.tar.gz -C ./data` as before). Upgrading from 1.3.0 is a tag bump; no schema migration. Added: - Mail queue and a delivery's history show this Postfix's retry policy - docs: **Plan checklists** in [development.md](docs/development.md) — format, - docs: **Implementation checklists** with model routing in - docs: roadmap candidates **send-log-retention** (panel Settings for delivery - Postfix queue under `/data/postfix/queue` — deferred and active mail survive … (1 more in CHANGELOG) Changed: - **Breaking:** full-backup archive layout — paths are prefixed with `data/`; - docs: operator guide, architecture, security, and backup UI updated for - docs: HTML mockups for a full panel UI refresh Changed: - docs: panel UI mockups after review — Status restored to live readings - docs: panel UI design system and mockups rebuilt as separate pages - docs: panel UI mockups — add-domain sits in the list card (Domains and Fixed: - ci (GHCR): per-arch package tags (`X.Y.Z-amd64`, `X.Y.Z-arm64`) are dropped Squashed CHANGELOG sections: [1.3.0], [1.3.1]. Full detail: CHANGELOG.md (1.3.0, 1.3.1).
This commit is contained in:
@@ -18,7 +18,8 @@ operator.
|
||||
- gzip + XML aggregate parsing → SQLite summaries per sending domain.
|
||||
- Panel page and/or per-domain section: recent reports, third-party senders,
|
||||
delivery health of report ingestion.
|
||||
- Reuse `admin.dmarc_report_email` and `domains.dmarc_rua` for DNS templates;
|
||||
- Reuse the `dmarc_report_email` setting (moved off the old `admin` table into
|
||||
`settings` by migration `0005`) and `domains.dmarc_rua` for DNS templates;
|
||||
when enabled, suggest a SelfPost-hosted report address.
|
||||
|
||||
**Out:**
|
||||
@@ -47,3 +48,18 @@ remain a separate, opt-in feature that does not forward mail upstream.
|
||||
|
||||
- Attack surface of accepting mail (mitigate: strict recipient allow-list).
|
||||
- Report volume and retention (mitigate: caps + pruning).
|
||||
|
||||
## Implementation checklist
|
||||
|
||||
Target version cut: **`1.7.0`** (MINOR). One commit per step; code only after
|
||||
roadmap status is **agreed**. Expand the sketch sections above before step 1
|
||||
if still thin. See [development.md](../development.md) § Plan checklists.
|
||||
|
||||
- [ ] Expand plan: ingest path, `dmarc_reports` schema, retention caps — **Sonnet**
|
||||
- [ ] Opt-in inbound SMTP for report addresses only (allow-list) — **Opus**
|
||||
- [ ] Worker: gzip/XML parse → SQLite — **Opus**
|
||||
- [ ] Panel: domain roll-up + parsed report (panel-ui mockups) — **Sonnet**
|
||||
- [ ] Tie-in `dmarc_report_email` / `domains.dmarc_rua` — **Sonnet**
|
||||
- [ ] Tests and [guide.md](../guide.md) — **Sonnet**
|
||||
- [ ] Security review ingest path — **Fable**
|
||||
- [ ] `go vet`, `go test` on touched packages — **Haiku**
|
||||
|
||||
@@ -0,0 +1,205 @@
|
||||
# Plan: domain-stats-auto-ratelimit
|
||||
|
||||
**Status:** candidate
|
||||
**Date:** 2026-08-17
|
||||
**Version:** `1.x` MINOR; migrations must stay compatible with `1.0.0`.
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Give the operator **30-day sending statistics** per domain and per application
|
||||
(total volume, peak and average rate), and an optional **auto** level-2 rate
|
||||
limit that sets `max_messages` from the average rate (avg × multiplier) over the
|
||||
level-1 window.
|
||||
|
||||
## Scope
|
||||
|
||||
**In:**
|
||||
|
||||
- Rolling **30-day** stats on each domain page (domain aggregate + per-app rows):
|
||||
message count, peak msg/h, average msg/h.
|
||||
- Level-2 rate limit mode **manual** (today) or **auto** for domain and
|
||||
application scopes.
|
||||
- Auto formula: `max_messages = ceil(avg_hourly × multiplier)`, window =
|
||||
level-1 window (`RATE_LIMIT_WINDOW_SECONDS`); capped at level 1.
|
||||
- Background recalculation (e.g. every 6 h, alongside send-log prune); milter
|
||||
reads stored `max_messages` / `window_seconds` only (no aggregates on the
|
||||
hot path).
|
||||
- Panel UI: stats card, manual/auto toggle, multiplier field, read-only computed
|
||||
limit in auto mode, optional «Recalculate now».
|
||||
- RBAC: domain-admin sees stats and may configure auto/manual for assigned
|
||||
domains only; same authz as existing rate-limit handlers.
|
||||
- Tests, [guide.md](../guide.md), [CHANGELOG.md](../../CHANGELOG.md); security
|
||||
review (Fable) for rate-limit path changes.
|
||||
|
||||
**Out:**
|
||||
|
||||
- Changing level 1 (Postfix env) — auto only fills level 2 under the L1 cap.
|
||||
- Automated IP warmup schedules ([guide.md](../guide.md) § IP warmup stays
|
||||
operator-driven).
|
||||
- Prometheus/Grafana, alerting, APIs.
|
||||
- Counting level-1 refusals or `rejected` rows as sent volume.
|
||||
- Per-client-IP analytics.
|
||||
|
||||
## Data source
|
||||
|
||||
All metrics come from SQLite `send_log`, same rules as
|
||||
[`CountMessages`](../../internal/store/ratelimits.go):
|
||||
|
||||
- One message = one distinct `queue_id` (many recipients = one count).
|
||||
- `status != rejected` (level-2 refusals never queued).
|
||||
- Level-1 refusals are **not** in `send_log` — stats under-count refusals;
|
||||
document in UI copy.
|
||||
|
||||
Retention today is env `SEND_LOG_RETENTION_DAYS` (default 90). Stats use the
|
||||
last **30 days** of rows still present. If retention < 30 days (after
|
||||
[send-log-retention](send-log-retention.md)), the stats window is
|
||||
`min(30, retention)` with a warning.
|
||||
|
||||
### Metrics
|
||||
|
||||
| Metric | Definition |
|
||||
|---|---|
|
||||
| **total** | `COUNT(DISTINCT queue_id)` in the stats window |
|
||||
| **peak rate** | maximum messages in any **hourly** bucket in that window (msg/h) |
|
||||
| **avg rate** | `total / hours_in_window`, where `hours_in_window = min(720, age of domain/app in hours, retention hours)` |
|
||||
|
||||
Keys: `send_log.domain` (domain scope), `send_log.app_login` (application scope).
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
milter["journal-milter"] --> sendLog["send_log"]
|
||||
sendLog --> statsQuery["StatsQuery 30d"]
|
||||
statsQuery --> domainPage["domain page stats card"]
|
||||
statsQuery --> autoJob["auto recalc job"]
|
||||
autoJob --> rateLimits["rate_limits"]
|
||||
rateLimits --> milter
|
||||
```
|
||||
|
||||
1. **[`internal/store/stats.go`](../../internal/store/stats.go)** — `SendStats`
|
||||
with `Total`, `PeakPerHour`, `AvgPerHour`; `DomainSendStats(name, since)` /
|
||||
`AppSendStats(login, since)`.
|
||||
2. Hourly buckets: `strftime('%Y-%m-%d %H', created_at)` + `GROUP BY`; subquery
|
||||
for peak; total via distinct `queue_id`.
|
||||
3. Indexes `idx_send_log_domain` and `idx_send_log_created_at` exist; add
|
||||
composite `(domain, created_at)` only if profiling shows need.
|
||||
4. **Auto recalc** — panel goroutine (same interval as send-log prune): for each
|
||||
`rate_limits` row with `mode = auto`, recompute `max_messages`, set
|
||||
`auto_updated_at`. Milter unchanged except reading new columns via existing
|
||||
`RateLimit` lookup.
|
||||
|
||||
### Auto rate limit
|
||||
|
||||
Extend [`RateLimit`](../../internal/store/ratelimits.go):
|
||||
|
||||
```go
|
||||
type RateLimit struct {
|
||||
// existing: Scope, RefID, AllowedIPs, MaxMessages, WindowSeconds
|
||||
Mode string // "manual" | "auto"
|
||||
AutoMultiplier float64 // default 2.5 when Mode == "auto"
|
||||
AutoUpdatedAt time.Time
|
||||
}
|
||||
```
|
||||
|
||||
**Formula:**
|
||||
|
||||
```
|
||||
avg_hourly = total_messages_in_window / hours_in_window
|
||||
max_messages = ceil(avg_hourly * auto_multiplier)
|
||||
window_seconds = L1 window (not editable in auto mode)
|
||||
max_messages = min(max_messages, L1 max)
|
||||
```
|
||||
|
||||
When `total == 0`: auto limit stays **inactive** (same as empty manual limit);
|
||||
UI explains that traffic is required before auto can apply.
|
||||
|
||||
**Application overrides** ([`handlers_ratelimit.go`](../../internal/web/handlers/handlers_ratelimit.go)):
|
||||
|
||||
- Trusted IPs required.
|
||||
- Auto app ceiling **strictly above** domain limit when domain limit is active.
|
||||
- Ceiling ≤ L1.
|
||||
|
||||
**Fail-open:** store errors during recalc must not weaken enforcement of the
|
||||
last successfully written limit; recalc failures are logged only.
|
||||
|
||||
### Migration (`0006_rate_limit_auto.sql`)
|
||||
|
||||
```sql
|
||||
ALTER TABLE rate_limits ADD COLUMN mode TEXT NOT NULL DEFAULT 'manual'
|
||||
CHECK (mode IN ('manual', 'auto'));
|
||||
ALTER TABLE rate_limits ADD COLUMN auto_multiplier REAL;
|
||||
ALTER TABLE rate_limits ADD COLUMN auto_updated_at TEXT;
|
||||
```
|
||||
|
||||
Existing rows → `manual`.
|
||||
|
||||
### Domain export
|
||||
|
||||
Today rate limits are **not** exported. This plan adds them (including
|
||||
`mode`, `auto_multiplier`) to domain transfer JSON — document as a boundary
|
||||
change in [guide.md](../guide.md) § Export.
|
||||
|
||||
## Panel UI
|
||||
|
||||
- Domain page ([`domain_detail.html`](../../internal/web/view/templates/domain_detail.html)):
|
||||
- **Sending statistics (30 days)** — total, peak msg/h, avg msg/h.
|
||||
- Per-application stats in the app list.
|
||||
- Rate limit: Manual / Auto, multiplier (e.g. 1.5–5.0, default 2.5), read-only
|
||||
computed max/window in auto mode, «Recalculate now».
|
||||
- Optional later: «30d» column on domain list (global admin only).
|
||||
|
||||
## Tests
|
||||
|
||||
- `internal/store/stats_test.go` — fixtures → total / peak / avg.
|
||||
- `internal/store/ratelimits_test.go` — auto recalc, L1 cap, app > domain.
|
||||
- Handler tests — auto form validation, multiplier bounds.
|
||||
- Milter tests — enforced limit matches last recalculated values.
|
||||
|
||||
`go test` / `go vet` on touched packages.
|
||||
|
||||
## Done when
|
||||
|
||||
- Domain and app 30-day stats visible on the domain page; domain-admin scoping
|
||||
enforced.
|
||||
- Manual/auto toggle works for domain and app; auto recalc updates `rate_limits`
|
||||
and milter enforces stored ceilings.
|
||||
- Zero-traffic auto stays inactive with clear UI copy.
|
||||
- [guide.md](../guide.md) and [CHANGELOG.md](../../CHANGELOG.md) updated;
|
||||
security review passed.
|
||||
|
||||
## Risks
|
||||
|
||||
- Heavy aggregation on large `send_log` tables — mitigate with indexes or
|
||||
nightly rollups (phase 2).
|
||||
- Stats without level-1 visibility — mitigate with operator-facing caveat.
|
||||
- Auto limit too tight after a spike — multiplier is operator-tuned; show peak
|
||||
alongside avg in auto UI.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- [`send-log-retention`](send-log-retention.md) is a separate roadmap item but
|
||||
should land before or in parallel so operators can set retention ≥ 30 days
|
||||
from the panel.
|
||||
|
||||
**Version:** `1.x` MINOR.
|
||||
|
||||
## Implementation checklist
|
||||
|
||||
Target version cut: **`1.6.0`** (MINOR). One commit per step; code only after
|
||||
roadmap status is **agreed**. See [development.md](../development.md) § Plan
|
||||
checklists.
|
||||
|
||||
- [ ] Migration `0006_rate_limit_auto.sql` (`mode`, `auto_multiplier`, `auto_updated_at`) — **Opus**
|
||||
- [ ] `internal/store/stats.go`: total / peak / avg over 30 days — **Opus**
|
||||
- [ ] Auto recalc job (6h): `ceil(avg × multiplier)`, L1 cap, fail-open on error — **Opus**
|
||||
- [ ] Extend `RateLimit` + handler forms (manual/auto) — **Opus**
|
||||
- [ ] Domain page stats card + per-app stats (`domain_detail.html`) — **Sonnet**
|
||||
- [ ] Domain export JSON includes rate limits — **Opus**
|
||||
- [ ] Milter tests — enforced limit matches stored ceiling — **Opus**
|
||||
- [ ] Store and handler tests — **Sonnet**
|
||||
- [ ] [guide.md](../guide.md) — **Sonnet**
|
||||
- [ ] Security review rate-limit path — **Fable**
|
||||
- [ ] `go vet`, `go test` on touched packages — **Haiku**
|
||||
@@ -3,9 +3,10 @@
|
||||
**Status:** agreed
|
||||
**Version:** target bump **1.x** MINOR; **`2.x` possible** — to be settled once
|
||||
the implementation lands (do not fix a major in advance).
|
||||
**Order:** recommended next among agreed 1.x+ items (after `internal/web` split
|
||||
and the domain-admin role, both shipped — see [CHANGELOG](../CHANGELOG.md)
|
||||
`[1.2.0]`).
|
||||
**Order:** queue-retries shipped in `[1.3.1]`; this is next as the largest
|
||||
remaining 1.x+ **feature**. The 2026-08-13 full-tree review follow-ups closed
|
||||
in `[1.3.0]`. The domain-admin role and `internal/web` split have shipped —
|
||||
see [CHANGELOG](../../CHANGELOG.md) `[1.2.0]`.
|
||||
|
||||
---
|
||||
|
||||
@@ -167,3 +168,25 @@ the SelfPost image, brought up by the operator when the option is enabled.
|
||||
|
||||
A finished outbound path (already implemented). Agreement obtained — see the
|
||||
status above.
|
||||
|
||||
## Implementation checklist
|
||||
|
||||
Target version cut: **`1.4.0`** (MINOR). One commit per step; see
|
||||
[development.md](../development.md) § Plan checklists. UI reference:
|
||||
[panel-ui inbound mockups](../assets/panel-ui/inbound.html).
|
||||
|
||||
- [ ] Migration: inbound domain / recipient / transport tables under `/data` — **Opus**
|
||||
- [ ] `INBOUND_RELAY_ENABLE` (default false) in entrypoint + `postfix-config.sh` — **Opus**
|
||||
- [ ] `master.cf`: inbound `smtp inet` on 25; separate from 465/587 — **Opus**
|
||||
- [ ] Generate `relay_domains`, `transport_maps`, `relay_recipient_maps` (injection-safe) — **Opus**
|
||||
- [ ] `smtpd_relay_restrictions` / recipient maps — no open relay, no backscatter — **Opus**
|
||||
- [ ] `internal/store` CRUD + validation (domain, host, port) — **Opus**
|
||||
- [ ] Panel: list, add, domain detail, recipients, danger zone (mockups) — **Sonnet**
|
||||
- [ ] Rate limit + `message_size_limit` on inbound smtpd — **Opus**
|
||||
- [ ] Optional `INBOUND_ANTISPAM_MILTER` + compose fragment — **Opus**
|
||||
- [ ] DNS MX copy in README/guide; `.env.example` — **Sonnet**
|
||||
- [ ] Backup/export inbound config (per plan optional flag) — **Opus**
|
||||
- [ ] Unit + handler tests; image build and container smoke — **Opus**
|
||||
- [ ] [guide.md](../guide.md) and [security.md](../security.md) — **Sonnet**
|
||||
- [ ] Security review inbound path — **Fable**
|
||||
- [ ] `go vet`, `go test`, e2e if applicable — **Haiku**
|
||||
|
||||
@@ -1,28 +0,0 @@
|
||||
# Plan: logrotate-mode (mail.log stops rotating in some images)
|
||||
|
||||
**Status:** done
|
||||
**Version:** patch; no schema, no configuration surface.
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
`mail.log` stopped rotating when `/etc/logrotate.d/mail` landed in the image
|
||||
with group/other write permission (e.g. build context from a Windows tar sync).
|
||||
logrotate ignores such configs but exits 0, so the loop looked healthy while
|
||||
the log grew without bound.
|
||||
|
||||
## What shipped
|
||||
|
||||
- [`build/Dockerfile`](../../build/Dockerfile): `COPY --chmod` pins config
|
||||
(`0644`) and script (`0755`) modes so the image no longer depends on checkout
|
||||
file modes.
|
||||
- [`build/logrotate-loop.sh`](../../build/logrotate-loop.sh): preflight and
|
||||
per-iteration checks refuse group/other-writable configs; logrotate stderr
|
||||
mentioning `Ignoring` is fatal.
|
||||
- [`test/e2e/logrotate_check.go`](../../test/e2e/logrotate_check.go): asserts
|
||||
mode `644`, forced `logrotate -f`, and that a group-writable context file
|
||||
still produces `644` in the image.
|
||||
|
||||
`create 0640 postfix selfpost` in [`build/logrotate-mail.conf`](../../build/logrotate-mail.conf)
|
||||
is unchanged — required for panel readability after rotation.
|
||||
@@ -0,0 +1,54 @@
|
||||
# Plan: panel-docs (in-panel operator documentation)
|
||||
|
||||
**Status:** candidate
|
||||
**Date:** 2026-08-17
|
||||
**Version:** `1.x` MINOR; `candidate` until explicitly agreed.
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Built-in operator documentation in the panel — short pages or a help drawer that
|
||||
explain what each Status check and other controls mean, without sending the
|
||||
operator out to [guide.md](../guide.md).
|
||||
|
||||
## Scope
|
||||
|
||||
**In:**
|
||||
|
||||
- Help drawer or short help pages (CSS-checkbox pattern from
|
||||
[panel-ui mockups](../assets/panel-ui/system.html)).
|
||||
- Seed content: Status blurbs removed from cards for a denser layout — Machine
|
||||
(kernel counters / rate window), TLS certificate (port 465, reverse-proxy
|
||||
mount), Hostname / reverse DNS (FCrDNS, PTR at the hosting provider), and
|
||||
similar notes for other surfaces as inline commentary is removed.
|
||||
- «?» entry points on domain cards (mockups).
|
||||
|
||||
**Out:**
|
||||
|
||||
- A second copy of the full operator guide.
|
||||
- Translation workflow beyond English (same as the rest of the panel).
|
||||
|
||||
## Done when
|
||||
|
||||
An operator can open help from the panel for those topics; the removed Status
|
||||
blurbs are preserved there (or equivalent); no requirement to read the git
|
||||
tree for day-to-day meaning of a card.
|
||||
|
||||
## Risks
|
||||
|
||||
Copy ownership and keeping help in sync when checks change; not bloating every
|
||||
page with a second column of prose.
|
||||
|
||||
## Implementation checklist
|
||||
|
||||
Target version cut: **`1.8.0`** (MINOR). One commit per step; code only after
|
||||
roadmap status is **agreed**. See [development.md](../development.md) § Plan
|
||||
checklists.
|
||||
|
||||
- [ ] Help drawer / pages shell (CSS checkbox pattern from mockups) — **Sonnet**
|
||||
- [ ] Seed Status blurbs (Machine, TLS, PTR, …) — **Sonnet**
|
||||
- [ ] «?» entry points on domain cards — **Sonnet**
|
||||
- [ ] [guide.md](../guide.md) boundary: in-panel help vs full guide — **Sonnet**
|
||||
- [ ] Template tests — **Sonnet**
|
||||
- [ ] `go vet`, `go test` on touched packages — **Haiku**
|
||||
@@ -0,0 +1,156 @@
|
||||
# Plan: queue-retries (Postfix retry policy in the panel)
|
||||
|
||||
**Status:** done — shipped in `[1.3.1]` (2026-08-17)
|
||||
**Date:** 2026-08-13
|
||||
**Version:** patch; no schema, no configuration surface.
|
||||
**Order:** small panel item; does not wait on inbound-relay.
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Show the operator how Postfix retries deferred mail: first retry delay, later
|
||||
backoff cap, and how long a message stays in the queue before it bounces. The
|
||||
numbers come from this container's effective Postfix config, not from
|
||||
hard-coded copy.
|
||||
|
||||
## Scope
|
||||
|
||||
**In:**
|
||||
|
||||
- A static «How delivery retries work» card on `/mail-queue` (global
|
||||
administrator). Not inside the HTMX poll fragment — the snapshot is taken at
|
||||
panel start.
|
||||
- The same human-readable intervals in `/deliveries/{id}` history for
|
||||
`deferred` and `bounced` (domain administrators never see Mail queue).
|
||||
- Operator docs: [guide.md](../guide.md) Mail queue bullet;
|
||||
[architecture.md](../architecture.md) notes the one-shot `postconf -h` at
|
||||
panel start. [CHANGELOG.md](../../CHANGELOG.md) `### Added`.
|
||||
|
||||
**Out:**
|
||||
|
||||
- Changing Postfix retry parameters, or exposing them as panel settings
|
||||
([product.md](../product.md): Postfix is used as-is).
|
||||
- An `attempts` column on `send_log`, or «attempt 3 of N» — Postfix has no
|
||||
attempt budget; it is time-based.
|
||||
- Counting `status=deferred` lines in `mail.log` or reading `postcat`. Each
|
||||
attempt is already on the delivery page's log table; the journal stores only
|
||||
the last status.
|
||||
- Duplicating the card on Status (summary + link to Mail queue already exist).
|
||||
- Re-reading `postconf` on every HTTP request.
|
||||
|
||||
## Architecture
|
||||
|
||||
Acceptance is still synchronous SMTP. Delivery stays in Postfix's on-disk
|
||||
queue. SelfPost does not enqueue, retry, or deliver.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
smtpAccept["SMTP 250 accepted"] --> postfixQueue["Postfix queue"]
|
||||
postfixQueue -->|"deferred: backoff until queue lifetime"| retry["Retry MX"]
|
||||
retry -->|sent| done["sent"]
|
||||
retry -->|lifetime elapsed or 5xx| bounce["bounced"]
|
||||
postfixQueue --> mailQueuePage["/mail-queue policy card"]
|
||||
postfixQueue --> deliveryPage["/deliveries/id history copy"]
|
||||
```
|
||||
|
||||
[build/postfix-config.sh](../../build/postfix-config.sh) does not set
|
||||
`queue_run_delay`, `minimal_backoff_time`, `maximal_backoff_time`,
|
||||
`maximal_queue_lifetime`, `bounce_queue_lifetime`, or `delay_warning_time`.
|
||||
Debian/Postfix 3.x compiled-in defaults therefore apply unless the operator
|
||||
overrides them (`postconf -e` inside the container).
|
||||
|
||||
### Loading the numbers
|
||||
|
||||
Once, when the HTTP role starts ([cmd/panel/httpserver.go](../../cmd/panel/httpserver.go),
|
||||
after `postfix-config.sh` has run):
|
||||
|
||||
```
|
||||
postconf -h queue_run_delay minimal_backoff_time maximal_backoff_time maximal_queue_lifetime bounce_queue_lifetime delay_warning_time
|
||||
```
|
||||
|
||||
`postconf -h`, not a parse of `/etc/postfix/main.cf`: stock values are not
|
||||
written to the file. `postconf` is the effective config, including a manual
|
||||
override.
|
||||
|
||||
- Fixed argv, no user input — same pattern as
|
||||
[postfix.Queue](../../internal/postfix/queue.go) /
|
||||
[security.md](../security.md).
|
||||
- Cache on `handlers.Config` (via `web.Config`). The HTMX fragment does not
|
||||
call `postconf`.
|
||||
- A live `postconf -e` is visible after the next panel (or container) restart.
|
||||
While the process is up, the panel shows the start-up snapshot.
|
||||
- Parse Postfix time units (`300s`, `5d`, `1h`, a bare number is seconds) in
|
||||
`internal/postfix`. Format human strings (`5 minutes`, `5 days`,
|
||||
`about 1 hour 7 minutes`) in one place so the Mail queue card and
|
||||
`deliveryEvents` cannot drift.
|
||||
- If `postconf` is missing (unit tests on Windows, binary outside the
|
||||
container): log a warning, fall back to Postfix 3.x compiled-in defaults
|
||||
(`300s` / `4000s` / `5d` / `0`), and put a muted note on the card. Tests
|
||||
stub the lookup (as `queueIDs` in the log-tailer) or pass a fixture on
|
||||
`Config`. Do not fail panel start.
|
||||
|
||||
Typical stock values, for orientation only — the UI prints whatever
|
||||
`postconf` returned:
|
||||
|
||||
| Parameter | Stock | Meaning |
|
||||
|---|---|---|
|
||||
| `queue_run_delay` / `minimal_backoff_time` | `300s` | First retry and deferred-queue scan |
|
||||
| `maximal_backoff_time` | `4000s` | Cap on the doubling gap (~1 h 7 min) |
|
||||
| `maximal_queue_lifetime` | `5d` | Then bounce |
|
||||
| `delay_warning_time` | `0` | No delay warning to the sender |
|
||||
|
||||
## Panel copy
|
||||
|
||||
Mail queue card facts: first retry; later retries (doubling, capped);
|
||||
kept in queue; then bounced. Short prose: there is no fixed attempt count; a
|
||||
`deferred` message stays in this listing until it is delivered or the queue
|
||||
lifetime runs out.
|
||||
|
||||
`deliveryEvents(row, policy)`:
|
||||
|
||||
- `deferred`: retries, first after X, then with increasing gaps up to Y, for
|
||||
up to Z.
|
||||
- `bounced`: or Postfix gave up after Z in the queue.
|
||||
|
||||
## Tests
|
||||
|
||||
- Duration parser: `5d`, `300s`, `4000s`, `1h`, `0`, bare number.
|
||||
- `/mail-queue` handler: card shows the fixture policy's human strings, not a
|
||||
live `postconf`.
|
||||
- Delivery page / `deliveryEvents`: `deferred` and `bounced` contain those
|
||||
strings ([handlers_monitor_test.go](../../internal/web/handlers/handlers_monitor_test.go)).
|
||||
- [templates_test.go](../../internal/web/view/templates_test.go): pass the new
|
||||
fields if rendering `mail_queue` requires them.
|
||||
|
||||
`go test` / `go vet` on the touched packages.
|
||||
|
||||
## Done when
|
||||
|
||||
- `/mail-queue` states this Postfix's first retry, backoff cap, and queue
|
||||
lifetime.
|
||||
- A `deferred` / `bounced` delivery page uses the same intervals.
|
||||
- A manual `postconf -e maximal_queue_lifetime=2d` followed by a panel restart
|
||||
changes what the panel prints.
|
||||
- Guide and architecture describe the snapshot; CHANGELOG has an Added entry.
|
||||
|
||||
## Risks
|
||||
|
||||
- Showing compiled-in fallbacks when `postconf` failed would mislead if the
|
||||
operator had overridden them — mitigate with the muted note on the card.
|
||||
- Inventing a max-attempt count would be false; the copy must stay time-based.
|
||||
|
||||
## Implementation checklist
|
||||
|
||||
Target version cut: **`1.3.1`** (PATCH). One commit per step; see
|
||||
[development.md](../development.md) § Plan checklists.
|
||||
|
||||
- [x] `internal/postfix`: parse Postfix time units (`5d`, `300s`, bare seconds) + tests — **Opus**
|
||||
- [x] `internal/postfix`: one-shot `postconf -h` (six keys), fallback + warn — **Opus**
|
||||
- [x] Load policy at HTTP start in `cmd/panel/httpserver.go`; cache on handlers config — **Opus**
|
||||
- [x] Human-readable duration formatter (shared by Mail queue card and delivery history) — **Sonnet**
|
||||
- [x] «How delivery retries work» card on `/mail-queue` (outside HTMX fragment) — **Sonnet**
|
||||
- [x] `deliveryEvents(row, policy)` — intervals in deferred/bounced copy — **Sonnet**
|
||||
- [x] Handler and template tests (`handlers_monitor_test.go`, `templates_test.go`) — **Sonnet**
|
||||
- [x] [guide.md](../guide.md) and [architecture.md](../architecture.md) — **Sonnet**
|
||||
- [x] `go vet`, `go test` on touched packages — **Haiku**
|
||||
@@ -0,0 +1,139 @@
|
||||
# Plan: send-log-retention
|
||||
|
||||
**Status:** candidate
|
||||
**Date:** 2026-08-17
|
||||
**Version:** `1.x` MINOR; no schema migration required (uses existing `settings` table).
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Let the **global administrator** change how long delivery journal rows
|
||||
(`send_log`, `/deliveries`) are kept, from the panel — without editing `.env`.
|
||||
|
||||
## Context (as-built)
|
||||
|
||||
Retention **already exists**, but only via environment:
|
||||
|
||||
- `SEND_LOG_RETENTION_DAYS` (default **90**) in `.env` / Compose.
|
||||
- [`cmd/panel/main.go`](../../cmd/panel/main.go) passes it to
|
||||
[`logtail.Run`](../../internal/logtail/logtail.go).
|
||||
- [`retentionLoop`](../../internal/logtail/logtail.go) prunes via
|
||||
[`DeleteSendLogBefore`](../../internal/store/sendlog.go) every **6 hours**.
|
||||
- No panel control; [`handlers_monitor.go`](../../internal/web/handlers/handlers_monitor.go)
|
||||
hardcodes «ninety days» in copy.
|
||||
- Migration `0001_init.sql` describes `settings` as the place for «retention
|
||||
overrides», but no UI writes that key yet.
|
||||
|
||||
This plan moves the **effective** retention into SQLite `settings`, with env as
|
||||
bootstrap only.
|
||||
|
||||
## Scope
|
||||
|
||||
**In:**
|
||||
|
||||
- Settings card on `/settings` (global administrator only): **Send log
|
||||
retention (days)**.
|
||||
- Key `send_log_retention_days` in [`settings`](../../internal/store/settings.go).
|
||||
- Validation: integer range **7–365** (exact bounds fixed at implementation).
|
||||
- On first use: if setting missing, seed from env
|
||||
(`SEND_LOG_RETENTION_DAYS`, default 90) at panel start or first save.
|
||||
- Log-tailer reads the setting **each prune cycle** (no container restart).
|
||||
- Delivery pages and guide copy show the **current** retention, not a hardcoded
|
||||
90.
|
||||
- Tests; [guide.md](../guide.md); [CHANGELOG.md](../../CHANGELOG.md).
|
||||
|
||||
**Out:**
|
||||
|
||||
- Per-domain retention (instance-wide only).
|
||||
- `mail.log` rotation (logrotate, 14 daily files — unchanged).
|
||||
- Immediate prune on save when lowering retention (next 6 h cycle is enough;
|
||||
optional «Prune now» not in v1).
|
||||
- Domain-admin access to this setting.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
settingsPage["/settings form"] --> sqlite["settings.send_log_retention_days"]
|
||||
env["SEND_LOG_RETENTION_DAYS bootstrap"] --> sqlite
|
||||
sqlite --> retentionLoop["logtail retentionLoop"]
|
||||
retentionLoop --> prune["DeleteSendLogBefore"]
|
||||
```
|
||||
|
||||
1. **Read path** — `GetSendLogRetentionDays()`: settings value if valid, else env
|
||||
default.
|
||||
2. **Write path** — POST `/settings` (global admin): validate, `SetSetting`,
|
||||
flash confirmation.
|
||||
3. **Prune path** — change [`logtail.retentionLoop`](../../internal/logtail/logtail.go)
|
||||
to accept `func() int` or `RetentionReader` that queries settings each cycle
|
||||
(same 6 h ticker).
|
||||
4. **Copy** — inject retention days into delivery list/detail templates and
|
||||
remove hardcoded «ninety days».
|
||||
|
||||
`SEND_LOG_RETENTION_DAYS` remains documented in [guide.md](../guide.md) as the
|
||||
**initial default** until changed in Settings.
|
||||
|
||||
## Relation to domain-stats-auto-ratelimit
|
||||
|
||||
[domain-stats-auto-ratelimit](domain-stats-auto-ratelimit.md) uses a **30-day**
|
||||
stats window. Requires effective retention ≥ 30 for full accuracy. When
|
||||
retention < 30:
|
||||
|
||||
- Stats UI shows a warning and uses `min(30, retention)` as the window, or
|
||||
- Settings validation warns when saving a value below 30 while stats/auto are
|
||||
enabled (pick one at implementation; document in guide).
|
||||
|
||||
Recommended roadmap order: **send-log-retention** before or parallel with
|
||||
domain-stats-auto-ratelimit.
|
||||
|
||||
## Panel UI
|
||||
|
||||
New card on [`settings.html`](../../internal/web/view/templates/settings.html)
|
||||
(global admin block, near rate limits or under a «Deliveries» heading):
|
||||
|
||||
- Number input: retention days (7–365).
|
||||
- Muted copy: rows older than this are deleted from `/deliveries`; main driver
|
||||
of `/data` growth; does not affect `mail.log` rotation.
|
||||
|
||||
Domain administrators keep the narrow credentials-only settings page.
|
||||
|
||||
## Tests
|
||||
|
||||
- Save/load setting; reject out-of-range values.
|
||||
- `retentionLoop` uses updated value without process restart (mock reader).
|
||||
- Bootstrap: empty settings → env default used for prune.
|
||||
- Template/delivery copy reflects configured days.
|
||||
|
||||
`go test` / `go vet` on touched packages.
|
||||
|
||||
## Done when
|
||||
|
||||
- Global admin can set retention on `/settings`; value persists in SQLite.
|
||||
- Prune uses the panel value on the next cycle; env remains bootstrap default.
|
||||
- Guide documents panel vs env; CHANGELOG entry added.
|
||||
- Hardcoded «ninety days» removed from delivery UI.
|
||||
|
||||
## Risks
|
||||
|
||||
- Operator lowers retention while bookmarking old delivery URLs — existing
|
||||
behaviour; copy already notes pruned rows are gone.
|
||||
- Settings change without restart — must be tested so log-tailer never keeps a
|
||||
stale int from panel start only.
|
||||
|
||||
**Version:** `1.x` MINOR.
|
||||
|
||||
## Implementation checklist
|
||||
|
||||
Target version cut: **`1.5.0`** (MINOR). One commit per step; code only after
|
||||
roadmap status is **agreed**. See [development.md](../development.md) § Plan
|
||||
checklists.
|
||||
|
||||
- [ ] `GetSendLogRetentionDays` / `SetSetting` key `send_log_retention_days` (7–365) — **Opus**
|
||||
- [ ] Bootstrap from `SEND_LOG_RETENTION_DAYS` when settings empty — **Opus**
|
||||
- [ ] `logtail.retentionLoop`: read setting each prune cycle — **Opus**
|
||||
- [ ] Settings card on `/settings` (`settings.html`) — **Sonnet**
|
||||
- [ ] Remove hardcoded «ninety days» in handlers and templates — **Sonnet**
|
||||
- [ ] Tests: save/load, range, loop without restart — **Sonnet**
|
||||
- [ ] [guide.md](../guide.md) — **Sonnet**
|
||||
- [ ] `go vet`, `go test` on touched packages — **Haiku**
|
||||
Reference in New Issue
Block a user