docs: add roadmap plans for domain stats, auto rate limit, and send-log retention

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-17 14:53:39 +03:00
parent 41c3e6e896
commit b99f60c1d6
5 changed files with 362 additions and 2 deletions
@@ -0,0 +1,187 @@
+# Plan: domain-stats-auto-ratelimit
+
+**Status:** candidate  
+**Date:** 2026-08-17  
+**Version:** `1.x` MINOR; migrations must stay compatible with `1.0.0`.
+
+---
+
+## Goal
+
+Give the operator **30-day sending statistics** per domain and per application
+(total volume, peak and average rate), and an optional **auto** level-2 rate
+limit that sets `max_messages` from the average rate (avg × multiplier) over the
+level-1 window.
+
+## Scope
+
+**In:**
+
+- Rolling **30-day** stats on each domain page (domain aggregate + per-app rows):
+  message count, peak msg/h, average msg/h.
+- Level-2 rate limit mode **manual** (today) or **auto** for domain and
+  application scopes.
+- Auto formula: `max_messages = ceil(avg_hourly × multiplier)`, window =
+  level-1 window (`RATE_LIMIT_WINDOW_SECONDS`); capped at level 1.
+- Background recalculation (e.g. every 6 h, alongside send-log prune); milter
+  reads stored `max_messages` / `window_seconds` only (no aggregates on the
+  hot path).
+- Panel UI: stats card, manual/auto toggle, multiplier field, read-only computed
+  limit in auto mode, optional «Recalculate now».
+- RBAC: domain-admin sees stats and may configure auto/manual for assigned
+  domains only; same authz as existing rate-limit handlers.
+- Tests, [guide.md](../guide.md), [CHANGELOG.md](../../CHANGELOG.md); security
+  review (Fable) for rate-limit path changes.
+
+**Out:**
+
+- Changing level 1 (Postfix env) — auto only fills level 2 under the L1 cap.
+- Automated IP warmup schedules ([guide.md](../guide.md) § IP warmup stays
+  operator-driven).
+- Prometheus/Grafana, alerting, APIs.
+- Counting level-1 refusals or `rejected` rows as sent volume.
+- Per-client-IP analytics.
+
+## Data source
+
+All metrics come from SQLite `send_log`, same rules as
+[`CountMessages`](../../internal/store/ratelimits.go):
+
+- One message = one distinct `queue_id` (many recipients = one count).
+- `status != rejected` (level-2 refusals never queued).
+- Level-1 refusals are **not** in `send_log` — stats under-count refusals;
+  document in UI copy.
+
+Retention today is env `SEND_LOG_RETENTION_DAYS` (default 90). Stats use the
+last **30 days** of rows still present. If retention &lt; 30 days (after
+[send-log-retention](send-log-retention.md)), the stats window is
+`min(30, retention)` with a warning.
+
+### Metrics
+
+| Metric | Definition |
+|---|---|
+| **total** | `COUNT(DISTINCT queue_id)` in the stats window |
+| **peak rate** | maximum messages in any **hourly** bucket in that window (msg/h) |
+| **avg rate** | `total / hours_in_window`, where `hours_in_window = min(720, age of domain/app in hours, retention hours)` |
+
+Keys: `send_log.domain` (domain scope), `send_log.app_login` (application scope).
+
+## Architecture
+
+```mermaid
+flowchart LR
+  milter["journal-milter"] --> sendLog["send_log"]
+  sendLog --> statsQuery["StatsQuery 30d"]
+  statsQuery --> domainPage["domain page stats card"]
+  statsQuery --> autoJob["auto recalc job"]
+  autoJob --> rateLimits["rate_limits"]
+  rateLimits --> milter
+```
+
+1. **[`internal/store/stats.go`](../../internal/store/stats.go)** — `SendStats`
+   with `Total`, `PeakPerHour`, `AvgPerHour`; `DomainSendStats(name, since)` /
+   `AppSendStats(login, since)`.
+2. Hourly buckets: `strftime('%Y-%m-%d %H', created_at)` + `GROUP BY`; subquery
+   for peak; total via distinct `queue_id`.
+3. Indexes `idx_send_log_domain` and `idx_send_log_created_at` exist; add
+   composite `(domain, created_at)` only if profiling shows need.
+4. **Auto recalc** — panel goroutine (same interval as send-log prune): for each
+   `rate_limits` row with `mode = auto`, recompute `max_messages`, set
+   `auto_updated_at`. Milter unchanged except reading new columns via existing
+   `RateLimit` lookup.
+
+### Auto rate limit
+
+Extend [`RateLimit`](../../internal/store/ratelimits.go):
+
+```go
+type RateLimit struct {
+    // existing: Scope, RefID, AllowedIPs, MaxMessages, WindowSeconds
+    Mode           string  // "manual" | "auto"
+    AutoMultiplier float64 // default 2.5 when Mode == "auto"
+    AutoUpdatedAt  time.Time
+}
+```
+
+**Formula:**
+
+```
+avg_hourly = total_messages_in_window / hours_in_window
+max_messages = ceil(avg_hourly * auto_multiplier)
+window_seconds = L1 window (not editable in auto mode)
+max_messages = min(max_messages, L1 max)
+```
+
+When `total == 0`: auto limit stays **inactive** (same as empty manual limit);
+UI explains that traffic is required before auto can apply.
+
+**Application overrides** ([`handlers_ratelimit.go`](../../internal/web/handlers/handlers_ratelimit.go)):
+
+- Trusted IPs required.
+- Auto app ceiling **strictly above** domain limit when domain limit is active.
+- Ceiling ≤ L1.
+
+**Fail-open:** store errors during recalc must not weaken enforcement of the
+last successfully written limit; recalc failures are logged only.
+
+### Migration (`0006_rate_limit_auto.sql`)
+
+```sql
+ALTER TABLE rate_limits ADD COLUMN mode TEXT NOT NULL DEFAULT 'manual'
+  CHECK (mode IN ('manual', 'auto'));
+ALTER TABLE rate_limits ADD COLUMN auto_multiplier REAL;
+ALTER TABLE rate_limits ADD COLUMN auto_updated_at TEXT;
+```
+
+Existing rows → `manual`.
+
+### Domain export
+
+Today rate limits are **not** exported. This plan adds them (including
+`mode`, `auto_multiplier`) to domain transfer JSON — document as a boundary
+change in [guide.md](../guide.md) § Export.
+
+## Panel UI
+
+- Domain page ([`domain_detail.html`](../../internal/web/view/templates/domain_detail.html)):
+  - **Sending statistics (30 days)** — total, peak msg/h, avg msg/h.
+  - Per-application stats in the app list.
+  - Rate limit: Manual / Auto, multiplier (e.g. 1.5–5.0, default 2.5), read-only
+    computed max/window in auto mode, «Recalculate now».
+- Optional later: «30d» column on domain list (global admin only).
+
+## Tests
+
+- `internal/store/stats_test.go` — fixtures → total / peak / avg.
+- `internal/store/ratelimits_test.go` — auto recalc, L1 cap, app &gt; domain.
+- Handler tests — auto form validation, multiplier bounds.
+- Milter tests — enforced limit matches last recalculated values.
+
+`go test` / `go vet` on touched packages.
+
+## Done when
+
+- Domain and app 30-day stats visible on the domain page; domain-admin scoping
+  enforced.
+- Manual/auto toggle works for domain and app; auto recalc updates `rate_limits`
+  and milter enforces stored ceilings.
+- Zero-traffic auto stays inactive with clear UI copy.
+- [guide.md](../guide.md) and [CHANGELOG.md](../../CHANGELOG.md) updated;
+  security review passed.
+
+## Risks
+
+- Heavy aggregation on large `send_log` tables — mitigate with indexes or
+  nightly rollups (phase 2).
+- Stats without level-1 visibility — mitigate with operator-facing caveat.
+- Auto limit too tight after a spike — multiplier is operator-tuned; show peak
+  alongside avg in auto UI.
+
+## Dependencies
+
+- [`send-log-retention`](send-log-retention.md) is a separate roadmap item but
+  should land before or in parallel so operators can set retention ≥ 30 days
+  from the panel.
+
+**Version:** `1.x` MINOR.
@@ -0,0 +1,124 @@
+# Plan: send-log-retention
+
+**Status:** candidate  
+**Date:** 2026-08-17  
+**Version:** `1.x` MINOR; no schema migration required (uses existing `settings` table).
+
+---
+
+## Goal
+
+Let the **global administrator** change how long delivery journal rows
+(`send_log`, `/deliveries`) are kept, from the panel — without editing `.env`.
+
+## Context (as-built)
+
+Retention **already exists**, but only via environment:
+
+- `SEND_LOG_RETENTION_DAYS` (default **90**) in `.env` / Compose.
+- [`cmd/panel/main.go`](../../cmd/panel/main.go) passes it to
+  [`logtail.Run`](../../internal/logtail/logtail.go).
+- [`retentionLoop`](../../internal/logtail/logtail.go) prunes via
+  [`DeleteSendLogBefore`](../../internal/store/sendlog.go) every **6 hours**.
+- No panel control; [`handlers_monitor.go`](../../internal/web/handlers/handlers_monitor.go)
+  hardcodes «ninety days» in copy.
+- Migration `0001_init.sql` describes `settings` as the place for «retention
+  overrides», but no UI writes that key yet.
+
+This plan moves the **effective** retention into SQLite `settings`, with env as
+bootstrap only.
+
+## Scope
+
+**In:**
+
+- Settings card on `/settings` (global administrator only): **Send log
+  retention (days)**.
+- Key `send_log_retention_days` in [`settings`](../../internal/store/settings.go).
+- Validation: integer range **7–365** (exact bounds fixed at implementation).
+- On first use: if setting missing, seed from env
+  (`SEND_LOG_RETENTION_DAYS`, default 90) at panel start or first save.
+- Log-tailer reads the setting **each prune cycle** (no container restart).
+- Delivery pages and guide copy show the **current** retention, not a hardcoded
+  90.
+- Tests; [guide.md](../guide.md); [CHANGELOG.md](../../CHANGELOG.md).
+
+**Out:**
+
+- Per-domain retention (instance-wide only).
+- `mail.log` rotation (logrotate, 14 daily files — unchanged).
+- Immediate prune on save when lowering retention (next 6 h cycle is enough;
+  optional «Prune now» not in v1).
+- Domain-admin access to this setting.
+
+## Architecture
+
+```mermaid
+flowchart LR
+  settingsPage["/settings form"] --> sqlite["settings.send_log_retention_days"]
+  env["SEND_LOG_RETENTION_DAYS bootstrap"] --> sqlite
+  sqlite --> retentionLoop["logtail retentionLoop"]
+  retentionLoop --> prune["DeleteSendLogBefore"]
+```
+
+1. **Read path** — `GetSendLogRetentionDays()`: settings value if valid, else env
+   default.
+2. **Write path** — POST `/settings` (global admin): validate, `SetSetting`,
+   flash confirmation.
+3. **Prune path** — change [`logtail.retentionLoop`](../../internal/logtail/logtail.go)
+   to accept `func() int` or `RetentionReader` that queries settings each cycle
+   (same 6 h ticker).
+4. **Copy** — inject retention days into delivery list/detail templates and
+   remove hardcoded «ninety days».
+
+`SEND_LOG_RETENTION_DAYS` remains documented in [guide.md](../guide.md) as the
+**initial default** until changed in Settings.
+
+## Relation to domain-stats-auto-ratelimit
+
+[domain-stats-auto-ratelimit](domain-stats-auto-ratelimit.md) uses a **30-day**
+stats window. Requires effective retention ≥ 30 for full accuracy. When
+retention &lt; 30:
+
+- Stats UI shows a warning and uses `min(30, retention)` as the window, or
+- Settings validation warns when saving a value below 30 while stats/auto are
+  enabled (pick one at implementation; document in guide).
+
+Recommended roadmap order: **send-log-retention** before or parallel with
+domain-stats-auto-ratelimit.
+
+## Panel UI
+
+New card on [`settings.html`](../../internal/web/view/templates/settings.html)
+(global admin block, near rate limits or under a «Deliveries» heading):
+
+- Number input: retention days (7–365).
+- Muted copy: rows older than this are deleted from `/deliveries`; main driver
+  of `/data` growth; does not affect `mail.log` rotation.
+
+Domain administrators keep the narrow credentials-only settings page.
+
+## Tests
+
+- Save/load setting; reject out-of-range values.
+- `retentionLoop` uses updated value without process restart (mock reader).
+- Bootstrap: empty settings → env default used for prune.
+- Template/delivery copy reflects configured days.
+
+`go test` / `go vet` on touched packages.
+
+## Done when
+
+- Global admin can set retention on `/settings`; value persists in SQLite.
+- Prune uses the panel value on the next cycle; env remains bootstrap default.
+- Guide documents panel vs env; CHANGELOG entry added.
+- Hardcoded «ninety days» removed from delivery UI.
+
+## Risks
+
+- Operator lowers retention while bookmarking old delivery URLs — existing
+  behaviour; copy already notes pruned rows are gone.
+- Settings change without restart — must be tested so log-tailer never keeps a
+  stale int from panel start only.
+
+**Version:** `1.x` MINOR.
@@ -59,7 +59,9 @@ domain administrators with assigned domains). The optional **inbound relay** is
 the main agreed **1.x+** extension still on the
 [roadmap](roadmap.md) — it targets a 1.x MINOR bump by default; a 2.x major
 remains possible pending implementation. Items marked *candidate* in the
-roadmap require explicit approval before coding.
+roadmap require explicit approval before coding (including **send-log retention
+in Settings**, **30-day send statistics**, and **auto level-2 rate limits** —
+see linked plans there).

 ---

@@ -30,6 +30,8 @@ in `git log` and [CHANGELOG.md](../CHANGELOG.md).
 |---|---|---|---|
 | queue-retries | Postfix retry policy in the panel (queue lifetime, backoff) | **agreed** | [plans/queue-retries.md](plans/queue-retries.md) |
 | inbound-relay | Inbound relay (backup-MX / forwarding) | **agreed** | [plans/inbound-relay.md](plans/inbound-relay.md) |
+| send-log-retention | Send-log retention days in panel Settings | candidate | [plans/send-log-retention.md](plans/send-log-retention.md) |
+| domain-stats-auto-ratelimit | 30-day send stats + auto level-2 rate limit | candidate | [plans/domain-stats-auto-ratelimit.md](plans/domain-stats-auto-ratelimit.md) |
 | contributing | `CONTRIBUTING.md` | candidate | — |
 | dmarc-reports | DMARC aggregate report ingestion and panel UI | candidate | [plans/dmarc-reports.md](plans/dmarc-reports.md) |
 | panel-docs | In-panel operator documentation | candidate | — |
@@ -37,7 +39,9 @@ in `git log` and [CHANGELOG.md](../CHANGELOG.md).

 **Recommended order** (not binding): **queue-retries** is a small panel item
 that can land first or in parallel; the next feature is **inbound-relay**.
-The 2026-08-13 full-tree review follow-ups (send-log authorization,
+**send-log-retention** can land before or beside **domain-stats-auto-ratelimit**
+(panel retention ≥ 30 days helps the stats window). The 2026-08-13 full-tree
+review follow-ups (send-log authorization,
 fail-closed paths, docs, GUI, tests, licence) are closed — history in
 [CHANGELOG.md](../CHANGELOG.md) `[Unreleased]` and git. Candidates need
 explicit agreement before they join the queue.
@@ -89,6 +93,44 @@ implementation lands.

 ---

+## send-log-retention
+
+**Goal:** global administrator sets how many days of delivery journal rows
+(`send_log`, `/deliveries`) are kept, from `/settings` — stored in SQLite
+`settings`, with `SEND_LOG_RETENTION_DAYS` as bootstrap default only.
+
+**Boundary:** instance-wide retention; not per-domain. Does not change
+`mail.log` rotation. Today retention is env-only (default 90 days); the panel
+has no control.
+
+**Done when:** see [plans/send-log-retention.md](plans/send-log-retention.md).
+
+**Dependencies / risks:** log-tailer must re-read settings each prune cycle;
+delivery UI must stop hardcoding «ninety days».
+**Version:** `1.x` MINOR; `candidate` until explicitly agreed.
+
+---
+
+## domain-stats-auto-ratelimit
+
+**Goal:** 30-day sending statistics per domain and application (total, peak and
+average msg/h) on the domain page, plus optional **auto** level-2 rate limits
+computed as `ceil(avg_hourly × multiplier)` over the level-1 window.
+
+**Boundary:** extension of v1.0 manual L2 limits; does not change Postfix
+level 1. Stats exclude level-1 refusals (not in send log). Auto with zero
+traffic stays inactive.
+
+**Done when:** see
+[plans/domain-stats-auto-ratelimit.md](plans/domain-stats-auto-ratelimit.md).
+
+**Dependencies / risks:** send log data and preferably
+[send-log-retention](#send-log-retention) ≥ 30 days; aggregate query cost on
+large logs; rate-limit path needs security review.
+**Version:** `1.x` MINOR; `candidate` until explicitly agreed.
+
+---
+
 ## contributing

 **Goal:** `CONTRIBUTING.md` in the root — the dev loop, the checks to run