From b99f60c1d6b09d096f1233483a59d8963d36cc6e Mon Sep 17 00:00:00 2001 From: Mikhail Yenuchenko Date: Mon, 17 Aug 2026 14:53:39 +0300 Subject: [PATCH] docs: add roadmap plans for domain stats, auto rate limit, and send-log retention Co-authored-by: Cursor --- CHANGELOG.md | 5 + docs/plans/domain-stats-auto-ratelimit.md | 187 ++++++++++++++++++++++ docs/plans/send-log-retention.md | 124 ++++++++++++++ docs/product.md | 4 +- docs/roadmap.md | 44 ++++- 5 files changed, 362 insertions(+), 2 deletions(-) create mode 100644 docs/plans/domain-stats-auto-ratelimit.md create mode 100644 docs/plans/send-log-retention.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 69a08f8..d77d22d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,11 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); version ### Added +- docs: roadmap candidates **send-log-retention** (panel Settings for delivery + journal retention) and **domain-stats-auto-ratelimit** (30-day send stats per + domain/application, auto level-2 rate limit from avg × multiplier) — plans in + [docs/plans/](docs/plans/). + - Postfix queue under `/data/postfix/queue` — deferred and active mail survive container recreate and are included in full backups. - Full backup archives the whole operator project: `data/`, `docker-compose.yml`, diff --git a/docs/plans/domain-stats-auto-ratelimit.md b/docs/plans/domain-stats-auto-ratelimit.md new file mode 100644 index 0000000..4301eb5 --- /dev/null +++ b/docs/plans/domain-stats-auto-ratelimit.md @@ -0,0 +1,187 @@ +# Plan: domain-stats-auto-ratelimit + +**Status:** candidate +**Date:** 2026-08-17 +**Version:** `1.x` MINOR; migrations must stay compatible with `1.0.0`. + +--- + +## Goal + +Give the operator **30-day sending statistics** per domain and per application +(total volume, peak and average rate), and an optional **auto** level-2 rate +limit that sets `max_messages` from the average rate (avg × multiplier) over the +level-1 window. + +## Scope + +**In:** + +- Rolling **30-day** stats on each domain page (domain aggregate + per-app rows): + message count, peak msg/h, average msg/h. +- Level-2 rate limit mode **manual** (today) or **auto** for domain and + application scopes. +- Auto formula: `max_messages = ceil(avg_hourly × multiplier)`, window = + level-1 window (`RATE_LIMIT_WINDOW_SECONDS`); capped at level 1. +- Background recalculation (e.g. every 6 h, alongside send-log prune); milter + reads stored `max_messages` / `window_seconds` only (no aggregates on the + hot path). +- Panel UI: stats card, manual/auto toggle, multiplier field, read-only computed + limit in auto mode, optional «Recalculate now». +- RBAC: domain-admin sees stats and may configure auto/manual for assigned + domains only; same authz as existing rate-limit handlers. +- Tests, [guide.md](../guide.md), [CHANGELOG.md](../../CHANGELOG.md); security + review (Fable) for rate-limit path changes. + +**Out:** + +- Changing level 1 (Postfix env) — auto only fills level 2 under the L1 cap. +- Automated IP warmup schedules ([guide.md](../guide.md) § IP warmup stays + operator-driven). +- Prometheus/Grafana, alerting, APIs. +- Counting level-1 refusals or `rejected` rows as sent volume. +- Per-client-IP analytics. + +## Data source + +All metrics come from SQLite `send_log`, same rules as +[`CountMessages`](../../internal/store/ratelimits.go): + +- One message = one distinct `queue_id` (many recipients = one count). +- `status != rejected` (level-2 refusals never queued). +- Level-1 refusals are **not** in `send_log` — stats under-count refusals; + document in UI copy. + +Retention today is env `SEND_LOG_RETENTION_DAYS` (default 90). Stats use the +last **30 days** of rows still present. If retention < 30 days (after +[send-log-retention](send-log-retention.md)), the stats window is +`min(30, retention)` with a warning. + +### Metrics + +| Metric | Definition | +|---|---| +| **total** | `COUNT(DISTINCT queue_id)` in the stats window | +| **peak rate** | maximum messages in any **hourly** bucket in that window (msg/h) | +| **avg rate** | `total / hours_in_window`, where `hours_in_window = min(720, age of domain/app in hours, retention hours)` | + +Keys: `send_log.domain` (domain scope), `send_log.app_login` (application scope). + +## Architecture + +```mermaid +flowchart LR + milter["journal-milter"] --> sendLog["send_log"] + sendLog --> statsQuery["StatsQuery 30d"] + statsQuery --> domainPage["domain page stats card"] + statsQuery --> autoJob["auto recalc job"] + autoJob --> rateLimits["rate_limits"] + rateLimits --> milter +``` + +1. **[`internal/store/stats.go`](../../internal/store/stats.go)** — `SendStats` + with `Total`, `PeakPerHour`, `AvgPerHour`; `DomainSendStats(name, since)` / + `AppSendStats(login, since)`. +2. Hourly buckets: `strftime('%Y-%m-%d %H', created_at)` + `GROUP BY`; subquery + for peak; total via distinct `queue_id`. +3. Indexes `idx_send_log_domain` and `idx_send_log_created_at` exist; add + composite `(domain, created_at)` only if profiling shows need. +4. **Auto recalc** — panel goroutine (same interval as send-log prune): for each + `rate_limits` row with `mode = auto`, recompute `max_messages`, set + `auto_updated_at`. Milter unchanged except reading new columns via existing + `RateLimit` lookup. + +### Auto rate limit + +Extend [`RateLimit`](../../internal/store/ratelimits.go): + +```go +type RateLimit struct { + // existing: Scope, RefID, AllowedIPs, MaxMessages, WindowSeconds + Mode string // "manual" | "auto" + AutoMultiplier float64 // default 2.5 when Mode == "auto" + AutoUpdatedAt time.Time +} +``` + +**Formula:** + +``` +avg_hourly = total_messages_in_window / hours_in_window +max_messages = ceil(avg_hourly * auto_multiplier) +window_seconds = L1 window (not editable in auto mode) +max_messages = min(max_messages, L1 max) +``` + +When `total == 0`: auto limit stays **inactive** (same as empty manual limit); +UI explains that traffic is required before auto can apply. + +**Application overrides** ([`handlers_ratelimit.go`](../../internal/web/handlers/handlers_ratelimit.go)): + +- Trusted IPs required. +- Auto app ceiling **strictly above** domain limit when domain limit is active. +- Ceiling ≤ L1. + +**Fail-open:** store errors during recalc must not weaken enforcement of the +last successfully written limit; recalc failures are logged only. + +### Migration (`0006_rate_limit_auto.sql`) + +```sql +ALTER TABLE rate_limits ADD COLUMN mode TEXT NOT NULL DEFAULT 'manual' + CHECK (mode IN ('manual', 'auto')); +ALTER TABLE rate_limits ADD COLUMN auto_multiplier REAL; +ALTER TABLE rate_limits ADD COLUMN auto_updated_at TEXT; +``` + +Existing rows → `manual`. + +### Domain export + +Today rate limits are **not** exported. This plan adds them (including +`mode`, `auto_multiplier`) to domain transfer JSON — document as a boundary +change in [guide.md](../guide.md) § Export. + +## Panel UI + +- Domain page ([`domain_detail.html`](../../internal/web/view/templates/domain_detail.html)): + - **Sending statistics (30 days)** — total, peak msg/h, avg msg/h. + - Per-application stats in the app list. + - Rate limit: Manual / Auto, multiplier (e.g. 1.5–5.0, default 2.5), read-only + computed max/window in auto mode, «Recalculate now». +- Optional later: «30d» column on domain list (global admin only). + +## Tests + +- `internal/store/stats_test.go` — fixtures → total / peak / avg. +- `internal/store/ratelimits_test.go` — auto recalc, L1 cap, app > domain. +- Handler tests — auto form validation, multiplier bounds. +- Milter tests — enforced limit matches last recalculated values. + +`go test` / `go vet` on touched packages. + +## Done when + +- Domain and app 30-day stats visible on the domain page; domain-admin scoping + enforced. +- Manual/auto toggle works for domain and app; auto recalc updates `rate_limits` + and milter enforces stored ceilings. +- Zero-traffic auto stays inactive with clear UI copy. +- [guide.md](../guide.md) and [CHANGELOG.md](../../CHANGELOG.md) updated; + security review passed. + +## Risks + +- Heavy aggregation on large `send_log` tables — mitigate with indexes or + nightly rollups (phase 2). +- Stats without level-1 visibility — mitigate with operator-facing caveat. +- Auto limit too tight after a spike — multiplier is operator-tuned; show peak + alongside avg in auto UI. + +## Dependencies + +- [`send-log-retention`](send-log-retention.md) is a separate roadmap item but + should land before or in parallel so operators can set retention ≥ 30 days + from the panel. + +**Version:** `1.x` MINOR. diff --git a/docs/plans/send-log-retention.md b/docs/plans/send-log-retention.md new file mode 100644 index 0000000..d3d484b --- /dev/null +++ b/docs/plans/send-log-retention.md @@ -0,0 +1,124 @@ +# Plan: send-log-retention + +**Status:** candidate +**Date:** 2026-08-17 +**Version:** `1.x` MINOR; no schema migration required (uses existing `settings` table). + +--- + +## Goal + +Let the **global administrator** change how long delivery journal rows +(`send_log`, `/deliveries`) are kept, from the panel — without editing `.env`. + +## Context (as-built) + +Retention **already exists**, but only via environment: + +- `SEND_LOG_RETENTION_DAYS` (default **90**) in `.env` / Compose. +- [`cmd/panel/main.go`](../../cmd/panel/main.go) passes it to + [`logtail.Run`](../../internal/logtail/logtail.go). +- [`retentionLoop`](../../internal/logtail/logtail.go) prunes via + [`DeleteSendLogBefore`](../../internal/store/sendlog.go) every **6 hours**. +- No panel control; [`handlers_monitor.go`](../../internal/web/handlers/handlers_monitor.go) + hardcodes «ninety days» in copy. +- Migration `0001_init.sql` describes `settings` as the place for «retention + overrides», but no UI writes that key yet. + +This plan moves the **effective** retention into SQLite `settings`, with env as +bootstrap only. + +## Scope + +**In:** + +- Settings card on `/settings` (global administrator only): **Send log + retention (days)**. +- Key `send_log_retention_days` in [`settings`](../../internal/store/settings.go). +- Validation: integer range **7–365** (exact bounds fixed at implementation). +- On first use: if setting missing, seed from env + (`SEND_LOG_RETENTION_DAYS`, default 90) at panel start or first save. +- Log-tailer reads the setting **each prune cycle** (no container restart). +- Delivery pages and guide copy show the **current** retention, not a hardcoded + 90. +- Tests; [guide.md](../guide.md); [CHANGELOG.md](../../CHANGELOG.md). + +**Out:** + +- Per-domain retention (instance-wide only). +- `mail.log` rotation (logrotate, 14 daily files — unchanged). +- Immediate prune on save when lowering retention (next 6 h cycle is enough; + optional «Prune now» not in v1). +- Domain-admin access to this setting. + +## Architecture + +```mermaid +flowchart LR + settingsPage["/settings form"] --> sqlite["settings.send_log_retention_days"] + env["SEND_LOG_RETENTION_DAYS bootstrap"] --> sqlite + sqlite --> retentionLoop["logtail retentionLoop"] + retentionLoop --> prune["DeleteSendLogBefore"] +``` + +1. **Read path** — `GetSendLogRetentionDays()`: settings value if valid, else env + default. +2. **Write path** — POST `/settings` (global admin): validate, `SetSetting`, + flash confirmation. +3. **Prune path** — change [`logtail.retentionLoop`](../../internal/logtail/logtail.go) + to accept `func() int` or `RetentionReader` that queries settings each cycle + (same 6 h ticker). +4. **Copy** — inject retention days into delivery list/detail templates and + remove hardcoded «ninety days». + +`SEND_LOG_RETENTION_DAYS` remains documented in [guide.md](../guide.md) as the +**initial default** until changed in Settings. + +## Relation to domain-stats-auto-ratelimit + +[domain-stats-auto-ratelimit](domain-stats-auto-ratelimit.md) uses a **30-day** +stats window. Requires effective retention ≥ 30 for full accuracy. When +retention < 30: + +- Stats UI shows a warning and uses `min(30, retention)` as the window, or +- Settings validation warns when saving a value below 30 while stats/auto are + enabled (pick one at implementation; document in guide). + +Recommended roadmap order: **send-log-retention** before or parallel with +domain-stats-auto-ratelimit. + +## Panel UI + +New card on [`settings.html`](../../internal/web/view/templates/settings.html) +(global admin block, near rate limits or under a «Deliveries» heading): + +- Number input: retention days (7–365). +- Muted copy: rows older than this are deleted from `/deliveries`; main driver + of `/data` growth; does not affect `mail.log` rotation. + +Domain administrators keep the narrow credentials-only settings page. + +## Tests + +- Save/load setting; reject out-of-range values. +- `retentionLoop` uses updated value without process restart (mock reader). +- Bootstrap: empty settings → env default used for prune. +- Template/delivery copy reflects configured days. + +`go test` / `go vet` on touched packages. + +## Done when + +- Global admin can set retention on `/settings`; value persists in SQLite. +- Prune uses the panel value on the next cycle; env remains bootstrap default. +- Guide documents panel vs env; CHANGELOG entry added. +- Hardcoded «ninety days» removed from delivery UI. + +## Risks + +- Operator lowers retention while bookmarking old delivery URLs — existing + behaviour; copy already notes pruned rows are gone. +- Settings change without restart — must be tested so log-tailer never keeps a + stale int from panel start only. + +**Version:** `1.x` MINOR. diff --git a/docs/product.md b/docs/product.md index 95ce410..fe8ba1a 100644 --- a/docs/product.md +++ b/docs/product.md @@ -59,7 +59,9 @@ domain administrators with assigned domains). The optional **inbound relay** is the main agreed **1.x+** extension still on the [roadmap](roadmap.md) — it targets a 1.x MINOR bump by default; a 2.x major remains possible pending implementation. Items marked *candidate* in the -roadmap require explicit approval before coding. +roadmap require explicit approval before coding (including **send-log retention +in Settings**, **30-day send statistics**, and **auto level-2 rate limits** — +see linked plans there). --- diff --git a/docs/roadmap.md b/docs/roadmap.md index 48099b0..62476f6 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -30,6 +30,8 @@ in `git log` and [CHANGELOG.md](../CHANGELOG.md). |---|---|---|---| | queue-retries | Postfix retry policy in the panel (queue lifetime, backoff) | **agreed** | [plans/queue-retries.md](plans/queue-retries.md) | | inbound-relay | Inbound relay (backup-MX / forwarding) | **agreed** | [plans/inbound-relay.md](plans/inbound-relay.md) | +| send-log-retention | Send-log retention days in panel Settings | candidate | [plans/send-log-retention.md](plans/send-log-retention.md) | +| domain-stats-auto-ratelimit | 30-day send stats + auto level-2 rate limit | candidate | [plans/domain-stats-auto-ratelimit.md](plans/domain-stats-auto-ratelimit.md) | | contributing | `CONTRIBUTING.md` | candidate | — | | dmarc-reports | DMARC aggregate report ingestion and panel UI | candidate | [plans/dmarc-reports.md](plans/dmarc-reports.md) | | panel-docs | In-panel operator documentation | candidate | — | @@ -37,7 +39,9 @@ in `git log` and [CHANGELOG.md](../CHANGELOG.md). **Recommended order** (not binding): **queue-retries** is a small panel item that can land first or in parallel; the next feature is **inbound-relay**. -The 2026-08-13 full-tree review follow-ups (send-log authorization, +**send-log-retention** can land before or beside **domain-stats-auto-ratelimit** +(panel retention ≥ 30 days helps the stats window). The 2026-08-13 full-tree +review follow-ups (send-log authorization, fail-closed paths, docs, GUI, tests, licence) are closed — history in [CHANGELOG.md](../CHANGELOG.md) `[Unreleased]` and git. Candidates need explicit agreement before they join the queue. @@ -89,6 +93,44 @@ implementation lands. --- +## send-log-retention + +**Goal:** global administrator sets how many days of delivery journal rows +(`send_log`, `/deliveries`) are kept, from `/settings` — stored in SQLite +`settings`, with `SEND_LOG_RETENTION_DAYS` as bootstrap default only. + +**Boundary:** instance-wide retention; not per-domain. Does not change +`mail.log` rotation. Today retention is env-only (default 90 days); the panel +has no control. + +**Done when:** see [plans/send-log-retention.md](plans/send-log-retention.md). + +**Dependencies / risks:** log-tailer must re-read settings each prune cycle; +delivery UI must stop hardcoding «ninety days». +**Version:** `1.x` MINOR; `candidate` until explicitly agreed. + +--- + +## domain-stats-auto-ratelimit + +**Goal:** 30-day sending statistics per domain and application (total, peak and +average msg/h) on the domain page, plus optional **auto** level-2 rate limits +computed as `ceil(avg_hourly × multiplier)` over the level-1 window. + +**Boundary:** extension of v1.0 manual L2 limits; does not change Postfix +level 1. Stats exclude level-1 refusals (not in send log). Auto with zero +traffic stays inactive. + +**Done when:** see +[plans/domain-stats-auto-ratelimit.md](plans/domain-stats-auto-ratelimit.md). + +**Dependencies / risks:** send log data and preferably +[send-log-retention](#send-log-retention) ≥ 30 days; aggregate query cost on +large logs; rate-limit path needs security review. +**Version:** `1.x` MINOR; `candidate` until explicitly agreed. + +--- + ## contributing **Goal:** `CONTRIBUTING.md` in the root — the dev loop, the checks to run