diff --git a/CHANGELOG.md b/CHANGELOG.md index 67cbfa8..ddad24f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,42 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); version ## [Unreleased] +## [1.9.1] - 2026-08-19 + +Port-25 Postfix startup when inbound relay, DMARC ingest, or the inbound antispam +milter are enabled. Schema-migration reference and roadmap plans for inbound +antispam and quarantine. Upgrading from 1.9.0 is a tag bump; no migration. + +### Added + +- [docs/plans/inbound-antispam-panel.md](docs/plans/inbound-antispam-panel.md) — + agreed plan for inbound antispam journal (inbound-journal milter + mail.log + rejects) and instance-wide allow/deny lists synced to rspamd maps; target + `1.10.0`; Composer → Opus → Fable workflow. +- [docs/plans/inbound-quarantine.md](docs/plans/inbound-quarantine.md) — candidate + plan for held-mail review/release (design TBD). +- [docs/schema-migrations.md](docs/schema-migrations.md) — living reference for the + SQLite migration chain (`user_version` head, per-file history, legacy artefacts, + 1.x rules, and planned 2.x squash gate). + +### Changed + +- [docs/roadmap.md](docs/roadmap.md) — `inbound-antispam-panel` (agreed, 1/12) + and `inbound-quarantine` (candidate) in the index. +- [docs/development.md](docs/development.md) — plan checklists may override + model routing with Composer → Opus → Fable. +- [docs/development.md](docs/development.md), [docs/roadmap.md](docs/roadmap.md), + and [docs/architecture.md](docs/architecture.md) link to the new file; + `schema-squash` defers gate thresholds to it instead of a stale v5 snapshot. + +### Fixed + +- `build/postfix-config.sh`: enabling inbound relay, DMARC ingest, and/or the + inbound antispam milter no longer crash-loops on start — `postconf -P` cannot + set `master.cf` `-o` values that contain whitespace, so port-25 + `smtpd_recipient_restrictions` and `smtpd_milters` are patched directly; DMARC + pipe `argv` uses a `dmarc-ingest` symlink instead of `panel -dmarc-ingest`. + ## [1.9.0] - 2026-08-18 Application sending controls: client IP allow-list for authorization, and diff --git a/build/postfix-config.sh b/build/postfix-config.sh index 66f9800..0c3075f 100644 --- a/build/postfix-config.sh +++ b/build/postfix-config.sh @@ -13,6 +13,37 @@ # a fixed literal or a container environment variable (spec 7.6.3). set -eu +# postconf -P refuses master.cf -o values that contain whitespace; inbound port +# 25 needs restriction lists and milter specs that include spaces. Patch the +# smtp/inet service stanza directly instead. +master_cf_set_smtp_inet_o() { + param="$1" + value="$2" + awk -v p="$param" -v v="$value" ' + /^smtp[ \t]+inet[ \t]/ && !done { + print + print " -o " p "=" v + done = 1 + in_smtp_inet = 1 + next + } + in_smtp_inet && /^[ \t]+-o / { + line = $0 + sub(/^[ \t]+-o /, "", line) + eq = index(line, "=") + on = substr(line, 1, eq - 1) + if (on == p) { + next + } + } + in_smtp_inet && /^[^ \t#-]/ { + in_smtp_inet = 0 + } + { print } + ' /etc/postfix/master.cf > /etc/postfix/master.cf.new + mv /etc/postfix/master.cf.new /etc/postfix/master.cf +} + # --- environment knobs (spec 8) ---------------------------------------------- # Server hostname: used as HELO name AND, crucially, as the Cyrus SASL realm the # sasldb2 accounts are looked up under. The panel creates accounts under realm @@ -235,17 +266,18 @@ if [ "${INBOUND_RELAY_ENABLE}" = "true" ] || [ "${DMARC_REPORTS_ENABLE}" = "true done RELAY_DOMAINS_SETTING="texthash:${RELAY_DOMAINS_MAP}" TRANSPORT_SETTING="texthash:${TRANSPORT_MAP}" - RECIPIENT_RESTRICTIONS="reject_unauth_destination, reject_unlisted_recipient" + RECIPIENT_RESTRICTIONS="reject_unauth_destination,reject_unlisted_recipient" fi if [ "${DMARC_REPORTS_ENABLE}" = "true" ]; then for f in "$DMARC_RECIPIENTS_MAP" "$DMARC_TRANSPORT_MAP" "$DMARC_RELAY_DOMAINS_MAP"; do [ -e "$f" ] || : > "$f" done + ln -sf /usr/local/bin/panel /usr/local/bin/dmarc-ingest postconf -M "dmarc-ingest/unix-pipe=dmarc-ingest unix - n n - - pipe" postconf -P \ "dmarc-ingest/unix-pipe/flags=Rq" \ "dmarc-ingest/unix-pipe/user=panel" \ - "dmarc-ingest/unix-pipe/argv=/usr/local/bin/panel -dmarc-ingest" + "dmarc-ingest/unix-pipe/argv=/usr/local/bin/dmarc-ingest" if [ -n "$RELAY_DOMAINS_SETTING" ]; then RELAY_DOMAINS_SETTING="${RELAY_DOMAINS_SETTING} texthash:${DMARC_RELAY_DOMAINS_MAP}" TRANSPORT_SETTING="${TRANSPORT_SETTING} texthash:${DMARC_TRANSPORT_MAP}" @@ -253,7 +285,7 @@ if [ "${INBOUND_RELAY_ENABLE}" = "true" ] || [ "${DMARC_REPORTS_ENABLE}" = "true RELAY_DOMAINS_SETTING="texthash:${DMARC_RELAY_DOMAINS_MAP}" TRANSPORT_SETTING="texthash:${DMARC_TRANSPORT_MAP}" fi - RECIPIENT_RESTRICTIONS="check_recipient_access texthash:${DMARC_RECIPIENTS_MAP}, ${RECIPIENT_RESTRICTIONS}" + RECIPIENT_RESTRICTIONS="check_recipient_access texthash:${DMARC_RECIPIENTS_MAP},${RECIPIENT_RESTRICTIONS}" fi postconf -e \ "relay_domains=${RELAY_DOMAINS_SETTING}" \ @@ -285,7 +317,7 @@ if [ "${INBOUND_RELAY_ENABLE}" = "true" ] || [ "${DMARC_REPORTS_ENABLE}" = "true exit 1 ;; esac - INBOUND_MILTERS="{ ${INBOUND_ANTISPAM_MILTER}, default_action=${INBOUND_ANTISPAM_MILTER_ACTION} }" + INBOUND_MILTERS="{${INBOUND_ANTISPAM_MILTER},default_action=${INBOUND_ANTISPAM_MILTER_ACTION}}" fi PORT25_RATE="${INBOUND_RATE_LIMIT_MESSAGES_PER_IP}" @@ -311,10 +343,12 @@ if [ "${INBOUND_RELAY_ENABLE}" = "true" ] || [ "${DMARC_REPORTS_ENABLE}" = "true "smtp/inet/smtpd_sender_restrictions=" \ "smtp/inet/smtpd_client_restrictions=" \ "smtp/inet/smtpd_relay_restrictions=reject_unauth_destination" \ - "smtp/inet/smtpd_recipient_restrictions=${RECIPIENT_RESTRICTIONS}" \ - "smtp/inet/smtpd_milters=${INBOUND_MILTERS}" \ "smtp/inet/smtpd_client_message_rate_limit=${PORT25_RATE}" \ "smtp/inet/message_size_limit=${PORT25_SIZE}" + master_cf_set_smtp_inet_o smtpd_recipient_restrictions "${RECIPIENT_RESTRICTIONS}" + if [ -n "${INBOUND_MILTERS}" ]; then + master_cf_set_smtp_inet_o smtpd_milters "${INBOUND_MILTERS}" + fi else postconf -MX "smtp/inet" 2>/dev/null || true postconf -MX "dmarc-ingest/unix-pipe" 2>/dev/null || true diff --git a/deploy/docker-compose.yml b/deploy/docker-compose.yml index b565a70..474d1e7 100644 --- a/deploy/docker-compose.yml +++ b/deploy/docker-compose.yml @@ -22,7 +22,7 @@ services: selfpost: - image: ghcr.io/mixeme/selfpost:1.9.0 + image: ghcr.io/mixeme/selfpost:1.9.1 restart: unless-stopped environment: SELFPOST_HOSTNAME: "${SELFPOST_HOSTNAME:?set the mail/panel hostname, e.g. mail.example.com}" diff --git a/docs/architecture.md b/docs/architecture.md index afbc435..d99ee35 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -318,7 +318,7 @@ single-connection trade-off that follows from it. | Path | Contents | |---|---| -| `selfpost.db` | SQLite: domains, apps, admin, sessions, send log, L2 limits, log-tailer offset, inbound relay domains | +| `selfpost.db` | SQLite panel state (domains, apps, users, sessions, send log, L2 limits, log-tailer offset, inbound relay, DMARC reports) — see [schema-migrations.md](schema-migrations.md) | | `setup-token` | First-run setup token file | | `opendkim/` | DKIM keys + tables | | `sasl/sasldb2` | Application SASL credentials | diff --git a/docs/development.md b/docs/development.md index 20abe97..50edd3c 100644 --- a/docs/development.md +++ b/docs/development.md @@ -64,6 +64,11 @@ obligation — not because the convention around AI authorship shifts. Default rule: risk-critical → Opus; UI / docs / boilerplate → Sonnet; trivial mechanics → Haiku. Reviewers must not be the author of the code under review. +**Composer → Opus → Fable** (when a plan specifies it): **Composer** writes all +implementation for that item (code, tests, docs); **Opus** does the technical +review of the result; **Fable** does the security review. A plan's checklist +overrides the default routing table for that item only. + --- ## Plan checklists @@ -424,6 +429,7 @@ There is no `docs/archive/` directory. | As-built design | [architecture.md](architecture.md) | | Development process (this file) | [development.md](development.md) | | Security requirements and accepted risks | [security.md](security.md) | +| SQLite schema and migrations | [schema-migrations.md](schema-migrations.md) | | Roadmap (1.x+) | [roadmap.md](roadmap.md) | | Active design plans | [plans/](plans/) | | Release history | [CHANGELOG.md](../CHANGELOG.md) | @@ -453,6 +459,8 @@ Out of scope for v1.x: `CONTRIBUTING.md`, man pages, a separate docs site fails on an undocumented `loadConfig` or build-script key. 3. **New gaps** go into [roadmap.md](roadmap.md) (or the active plan file), not silent drive-by edits. +4. **Schema changes:** update [schema-migrations.md](schema-migrations.md) in the + same change as a new `internal/store/migrations/*.sql` file. ### Verifying docs against code diff --git a/docs/plans/inbound-antispam-panel.md b/docs/plans/inbound-antispam-panel.md new file mode 100644 index 0000000..bb2c5da --- /dev/null +++ b/docs/plans/inbound-antispam-panel.md @@ -0,0 +1,251 @@ +# Plan: inbound-antispam-panel + +**Status:** agreed +**Date:** 2026-08-19 +**Version:** `1.10.0` MINOR (opt-in; no change to outbound-only or inbound-without-filter paths). + +--- + +## Goal + +Give the operator a **panel view of inbound anti-spam decisions** and **editable +allow/deny lists** that tune filter behaviour — without SelfPost shipping or +starting an anti-spam engine ([product.md](../product.md) out-of-scope list +unchanged). + +Typical journal row: + +| When | From | To | Subject | Decision | +|------|------|----|---------|----------| +| 2026-08-19 09:14 | `spammer@evil.example` | `user@inbound.example` | Cheap pills | reject (rspamd: BAYES_SPAM +0.95) | + +Lists let the operator correct false positives/negatives (e.g. always allow +`billing@vendor.example`, block `192.0.2.0/24`). + +## Context (as-built) + +- Inbound relay ships in `[1.4.0]` (`INBOUND_RELAY_ENABLE`); optional + `INBOUND_ANTISPAM_MILTER` attaches an **external** filter (documented rspamd + sidecar in [deploy/antispam/docker-compose.antispam.yml](../../deploy/antispam/docker-compose.antispam.yml)). +- The inbound smtpd has **no journal-milter** today — only outbound 465/587 + writes `send_log` ([architecture.md](../architecture.md)). +- Inbound UI and handlers are **global-administrator only** today + (`requireInbound` → `requireGlobal`). + +## Quarantine + +**Not in this plan.** Held-mail review/release is tracked separately as roadmap +candidate [inbound-quarantine](inbound-quarantine.md). The journal may still +record a `quarantine` **decision label** when the external filter reports it, +without SelfPost storing the message. + +**DMARC `p=quarantine`** remains unrelated (outbound policy for receivers). + +## Decisions (closed 2026-08-19) + +### Capture mechanism + +**Inbound journal-milter in `panel` only** — no rspamd history API polling in v1. +A separate unix socket (`INBOUND_JOURNAL_MILTER_SOCKET`, default +`/run/selfpost/inbound-journal.sock`) on the inbound smtpd only; fail-open +(`default_action=accept`). + +**Milter order on port 25** (when antispam is enabled): + +```text +smtpd_milters = { antispam, inbound-journal } +``` + +Antispam runs first; inbound-journal runs at end-of-message on the **accept** +path and reads rspamd-added headers (`X-Rspamd-Action`, `X-Rspamd-Score`, symbol +summary when present). + +**Antispam rejects** do not reach end-of-message on the journal milter. v1 also +tails `mail.log` for inbound `postfix/smtpd` **`milter-reject`** / **`reject`** +lines and inserts journal rows (client IP, envelope from/to when logged, reason +text, subject empty if rejected before DATA). DMARC-ingest pipe traffic is out +of scope for this tailer. + +### Journal without `INBOUND_ANTISPAM_MILTER` + +Journal stays **on** whenever `INBOUND_RELAY_ENABLE=true` (not tied to the +antispam hook). Without an external filter: + +- **accept** rows — message relayed (engine `selfpost`, detail empty); +- **reject** / **tempfail** rows — from the mail.log tailer (Postfix policy, + size limit, unknown recipient, etc.). + +Panel copy explains that rspamd symbols appear only when the antispam milter is +configured. + +### RBAC + +**Global administrator only** for journal and lists — same as inbound relay. No +domain-admin access in v1. + +### Retention + +Mirror send-log retention: + +- Settings key `inbound_spam_log_retention_days` on `/settings` (global admin). +- Default **90** days; validation **7–365**; seed from env + `INBOUND_SPAM_LOG_RETENTION_DAYS` when the setting is missing. +- Prune every **6 hours** (same cadence as send-log retention). +- Hard cap **10 000** rows after age prune (drop oldest) — busier than DMARC + reports, lighter than unbounded growth. + +### Allow/deny lists + +| Topic | Decision | +|-------|----------| +| Entry types | sender address, sender domain (`@domain` suffix), client IP / CIDR | +| Precedence | **deny overrides allow** | +| Scope | instance-wide (not per inbound domain in v1) | +| Where applied | **rspamd map files** under `/data/antispam/` (allow.map, deny.map); documented compose fragment mounts them into the sidecar | +| SelfPost pre-check | **out of v1** — lists live in rspamd only | +| Edit access | global administrator only | +| When inactive | lists editable in the panel but UI warns that sync applies only when `INBOUND_ANTISPAM_MILTER` is set; `postfix reload` / rspamd reload documented in guide | + +List change → atomic map rewrite (same pattern as Postfix maps) → rspamd reload +via documented operator step or sidecar `SIGHUP` in the compose fragment. + +### Activation + +| Condition | Behaviour | +|-----------|-----------| +| `INBOUND_RELAY_ENABLE=false` | no inbound journal milter, no UI, no tables written | +| inbound on, antispam empty | journal + UI; lists visible but marked inactive for filter sync | +| inbound on + antispam set | full journal (symbols on accept) + list sync | + +No separate feature flag beyond inbound relay + existing antispam env vars. + +### UI + +New nav item **Inbound spam** (or subsection under **Inbound**) — journal table +with filters (decision, domain, date range), list management on the same page or +a tab. Mockup: add `docs/assets/panel-ui/inbound_spam.html` in the panel step. + +## Schema (migration `0010_inbound_spam_log.sql`) + +**`inbound_spam_log`** + +| Column | Type | Notes | +|--------|------|-------| +| `id` | INTEGER PK | | +| `inbound_domain` | TEXT NOT NULL | recipient domain (from `relay_domains`) | +| `client_ip` | TEXT NOT NULL | | +| `from_addr` | TEXT NOT NULL | envelope from | +| `to_addr` | TEXT NOT NULL | envelope to (one row per recipient) | +| `subject` | TEXT NOT NULL | truncated with `mailhdr.SubjectMaxRunes` (200) | +| `decision` | TEXT NOT NULL | `accept`, `reject`, `tempfail`, `quarantine` | +| `engine` | TEXT NOT NULL | `rspamd`, `postfix`, `selfpost` | +| `detail` | TEXT NOT NULL | symbol/score summary or log-line reason; may be empty | +| `created_at` | TEXT NOT NULL | RFC3339 UTC | + +Index on `(created_at)`, `(inbound_domain, created_at)`. + +**`inbound_spam_list`** + +| Column | Type | Notes | +|--------|------|-------| +| `id` | INTEGER PK | | +| `list_type` | TEXT NOT NULL | `allow` or `deny` | +| `entry_type` | TEXT NOT NULL | `address`, `domain`, `ip` | +| `value` | TEXT NOT NULL | normalized entry | +| `note` | TEXT NOT NULL | operator comment; default `''` | +| `created_at` | TEXT NOT NULL | | + +Unique on `(list_type, entry_type, value)`. + +## Scope + +**In:** + +- Schema, store CRUD, retention loop, mail.log tailer extension for inbound + rejects. +- Inbound journal-milter (fail-open) + Postfix `postfix-config.sh` wiring. +- rspamd map sync under `/data/antispam/`. +- Panel journal + list UI (global admin). +- Full backup includes `/data/antispam/` and new SQLite tables. +- Tests; [guide.md](../guide.md); [architecture.md](../architecture.md). + +**Out:** + +- Shipping rspamd, ClamAV, or any filter binary inside the SelfPost image. +- Outbound spam filtering. +- [inbound-quarantine.md](inbound-quarantine.md) storage/release. +- rspamd rule editing, Bayes training UI, antivirus. +- MIME replay from the panel (metadata only). +- Per-domain lists; domain-admin RBAC. +- rspamd history API polling. + +## Architecture + +```mermaid +flowchart LR + internet["Internet :25"] --> inbound["Postfix inbound smtpd"] + inbound --> antispam["External antispam milter optional"] + antispam --> journalM["inbound-journal milter fail-open"] + journalM --> upstream["smtp upstream transport"] + journalM --> sqlite["inbound_spam_log"] + mailog["mail.log tailer"] --> sqlite + listsUI["Panel allow/deny lists"] --> maps["/data/antispam/*.map"] + maps --> antispam + sqlite --> journalUI["Panel inbound spam journal"] +``` + +## Security + +- Same validation whitelists as inbound relay ([security.md](../security.md)). +- Journal rows contain PII — retention and global-admin-only access. +- Inbound journal milter fail-open must not block mail (backup-MX role). +- List map writes injection-safe; deny wins over allow. + +## Done when + +- With inbound relay + antispam sidecar, panel shows accept rows with symbols + and reject rows from mail.log tailing. +- Operator can add/remove list entries; a test message reflects allow/deny. +- With antispam hook off, journal still shows Postfix-level decisions; list UI + shows inactive sync notice. +- `go vet`, `go test`, image build green; guide and architecture updated. + +## Risks + +- rspamd header/symbol format drift — thin parser, pin sidecar tag in compose. +- Journal volume — retention + 10k cap. +- mail.log parse brittleness — unit tests on sample lines; inbound smtpd only. + +## Dependencies + +- [inbound-relay.md](inbound-relay.md) (shipped). + +## Model routing (this plan) + +| Role | Model | +|------|-------| +| All implementation (code, tests, docs) | **Composer** | +| Technical review (reviewer ≠ author) | **Opus** | +| Security review | **Fable** | + +Composer ships each step; Opus reviews the accumulated diff once implementation +steps are done (fix loop: Composer addresses Opus findings, Opus re-checks). +Fable runs after Opus sign-off. + +## Implementation checklist + +Target version cut: **`1.10.0`** (MINOR). One commit per step; +[development.md](../development.md) § Plan checklists. + +- [x] Agree journal fields, retention, and RBAC (this plan § Decisions) — **Composer** +- [ ] Migration `0010_inbound_spam_log.sql` — **Composer** +- [ ] Inbound journal-milter + Postfix wiring — **Composer** +- [ ] mail.log tailer: inbound reject rows — **Composer** +- [ ] List CRUD + validation + atomic rspamd map sync — **Composer** +- [ ] Panel: journal + lists UI (+ mockup) — **Composer** +- [ ] Backup includes `/data/antispam/` — **Composer** +- [ ] Unit + handler tests — **Composer** +- [ ] [guide.md](../guide.md), [architecture.md](../architecture.md) — **Composer** +- [ ] `go vet`, `go test`, e2e if applicable — **Composer** +- [ ] Technical review of inbound antispam changes — **Opus** +- [ ] Security review inbound antispam path — **Fable** diff --git a/docs/plans/inbound-quarantine.md b/docs/plans/inbound-quarantine.md new file mode 100644 index 0000000..fd0ec8c --- /dev/null +++ b/docs/plans/inbound-quarantine.md @@ -0,0 +1,72 @@ +# Plan: inbound-quarantine + +**Status:** candidate — design TBD; needs explicit agreement before any code +**Version:** TBD (`1.x` MINOR if opt-in and backward-compatible; may warrant `2.x` +if it changes inbound storage or backup contract). + +--- + +## Goal (draft) + +Let the operator **hold suspicious inbound mail** instead of only accept/reject +at the milter — review in the panel, then release to the configured upstream, +delete, or add sender to allow/deny lists. + +This is a **product boundary question**, not a settled design. The item exists +so we can compare approaches before committing. + +## Why it is separate from inbound-antispam-panel + +[inbound-antispam-panel.md](inbound-antispam-panel.md) covers **metadata** +(journal + lists). Quarantine implies **storing message bodies** (or a durable +reference to them) under `/data`, retention, release workflow, and a wider +attack/backup surface — closer to "mini-mailbox" than pure relay. + +## Open questions (to decide before a checklist) + +1. **Where mail lives** + - SelfPost store under `/data/quarantine/` (panel owns lifecycle), or + - delegate to rspamd/Redis and panel only proxies release (less storage in + SelfPost, tighter rspamd coupling). +2. **What "release" means** + - inject into Postfix as a new delivery to the domain's upstream transport; + - or manual download only (no automatic forward). +3. **Retention and caps** + - max age, max total bytes, per-domain limits; interaction with backup size. +4. **RBAC** + - global admin only vs domain-admin for assigned inbound domains. +5. **Threat model** + - malware in stored MIME, path traversal on extract, quota DoS on busy MX. +6. **Product fit** + - does this violate [product.md](../product.md) "no mailboxes" if we only + hold spam suspects briefly? Explicit product decision required. + +## Context (as-built) + +- Inbound relay forwards or rejects; no local delivery ([inbound-relay.md](inbound-relay.md)). +- Optional `INBOUND_ANTISPAM_MILTER` — external rspamd may quarantine on its + own today, without panel integration. +- DMARC `p=quarantine` is unrelated (outbound policy for receivers). + +## Likely out of scope (until revisited) + +- IMAP/webmail for quarantined mail. +- End-user self-service quarantine (non-admin recipients). +- Outbound quarantine. + +## Dependencies + +- [inbound-relay.md](inbound-relay.md) (shipped). +- Sensible to decide **after** or **alongside** + [inbound-antispam-panel.md](inbound-antispam-panel.md) (journal/lists), but + not blocked on it for design discussion. + +## Done when (placeholder) + +TBD once approach is chosen. Minimum bar would include: opt-in flag, panel +list/detail, release or discard action, retention, backup inclusion documented, +security review. + +## Implementation checklist + +None yet — expand this plan after the open questions above are answered. diff --git a/docs/roadmap.md b/docs/roadmap.md index 4f85c3f..40a6343 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -30,6 +30,8 @@ in `git log` and [CHANGELOG.md](../CHANGELOG.md). |---|---|---|---|---| | contributing | `CONTRIBUTING.md` | candidate | — | — | | panel-docs | In-panel operator documentation | agreed | 6/6 | [plans/panel-docs.md](plans/panel-docs.md) | +| inbound-antispam-panel | Inbound antispam journal + allow/deny lists | agreed | 1/12 | [plans/inbound-antispam-panel.md](plans/inbound-antispam-panel.md) | +| inbound-quarantine | Inbound spam quarantine (hold / review / release) | candidate | — | [plans/inbound-quarantine.md](plans/inbound-quarantine.md) | | schema-squash | Squash SQLite migrations into a 2.x baseline | **2.x** | — | — | **Recommended order** (not binding): the next feature is **panel-docs** once @@ -89,38 +91,74 @@ when checks change; not bloating every page with a second column of prose. --- +## inbound-antispam-panel + +**Goal:** panel view of inbound anti-spam decisions (when, from, to, subject, +action) plus editable allow/deny lists that tune the external filter — without +shipping rspamd or another engine inside the image. + +**Boundary:** metadata journal and list sync only; SelfPost does not run the +filter ([product.md](product.md)). Quarantine: +[inbound-quarantine](#inbound-quarantine). Design closed in +[plans/inbound-antispam-panel.md](plans/inbound-antispam-panel.md) § Decisions. + +**Done when:** with inbound relay enabled, the operator sees journal rows +(accepts via inbound-journal milter, rejects via mail.log tailer) and can +maintain instance-wide allow/deny lists synced to rspamd maps when the antispam +hook is set. + +**Dependencies / risks:** inbound relay `[1.4.0]`; rspamd map/header coupling; +journal PII; global-admin-only RBAC. + +**Version:** `1.10.0` MINOR; **agreed**. + +--- + +## inbound-quarantine + +**Goal:** optional hold for suspicious inbound mail — review in the panel, then +release to upstream, discard, or tune lists. Design not settled. + +**Boundary:** expands inbound from pure relay toward short-lived message +storage; may conflict with "no mailboxes" in [product.md](product.md) until +product explicitly agrees. Alternatives (rspamd-only quarantine vs SelfPost +store) are listed in [plans/inbound-quarantine.md](plans/inbound-quarantine.md). + +**Done when:** TBD after open questions in the plan are answered. + +**Dependencies / risks:** inbound relay `[1.4.0]`; storage/retention/backup +size; malware in held MIME; overlap with +[inbound-antispam-panel](#inbound-antispam-panel). No implementation checklist +yet. + +**Version:** TBD; `candidate` until explicitly agreed. + +--- + ## schema-squash -**Goal:** when 2.x is cut, stop shipping the 1.x migration files -(`0001_init.sql` … `0005_panel_users.sql`) in the binary and replace them with -one baseline that is the schema as of `user_version = 5`. Fresh 2.x data -directories no longer create-then-drop the historical `admin` table. +**Goal:** when 2.x is cut, stop shipping the 1.x migration chain in the binary +and replace it with one baseline equal to the schema at the then-current head. +Fresh 2.x data directories no longer create-then-drop the historical `admin` +table. Current chain, legacy notes, and gate thresholds: +[schema-migrations.md](schema-migrations.md) (update that file when migrations +ship; adjust the squash gate to its head at cut time). -**Boundary:** 1.x keeps the full chain so a 1.0.0 data directory still boots. -Do not delete, rename, or reorder those files while MINOR compatibility with -`1.0.0` holds. `migrate()` maps **file order** to `PRAGMA user_version` (`target -= i + 1`); dropping a file in 1.x would skip or mis-apply steps on existing -databases. Git history keeps the old files either way; only the embedded set -in the 2.x image changes. - -**Upgrade gate (required with the squash):** - -| `user_version` | 2.x behaviour | -|---|---| -| `0` (empty file) | Apply the baseline; set `user_version` to the new chain’s head | -| `>= 5` (fully migrated 1.x) | Skip; schema is already the baseline | -| `1`…`4` (mid-chain 1.x) | **Refuse to start** — boot the last 1.x once, then 2.x | +**Boundary:** 1.x keeps the full chain so a `1.0.0` data directory still boots. +Do not delete, rename, or reorder migration files while MINOR compatibility with +`1.0.0` holds. Git history keeps the old files either way; only the embedded +set in the 2.x image changes. Restore remains a separate lock: the backup manifest version must match the running binary ([architecture.md](architecture.md) § Persistence). It does not -replace this gate. +replace the squash upgrade gate. **Done when:** 2.x embeds a single baseline (plus any 2.x-only migrations after -it); the gate above is tested; the operator guide says a 2.x image will not -open an unfinished 1.x database. +it); the gate in [schema-migrations.md](schema-migrations.md) is tested; the +operator guide says a 2.x image will not open an unfinished 1.x database. **Dependencies / risks:** a decided 2.x cut (another breaking change, or an -explicit major). Squashing five short files is not a reason to cut 2.x on its -own. A missed gate leaves a `user_version = 3` database silently stuck. +explicit major). Squashing the current short chain is not a reason to cut 2.x on +its own. A missed gate leaves a mid-chain 1.x database silently stuck. **Version:** `2.x` major only; not a 1.x item. diff --git a/docs/schema-migrations.md b/docs/schema-migrations.md new file mode 100644 index 0000000..ffcc07b --- /dev/null +++ b/docs/schema-migrations.md @@ -0,0 +1,196 @@ +# SQLite schema and migrations + +**What this file is.** A living reference for the embedded SQLite migration chain, +historical legacy that still runs on fresh 1.x installs, and the planned 2.x +schema squash. When you add or review a migration, update this file in the same +change. + +**Source of truth in code:** `internal/store/migrations/*.sql`, applied by +`internal/store/store.go` (`migrate()`). + +**Related:** [architecture.md](architecture.md) § Persistence; +[roadmap.md](roadmap.md) § schema-squash; [development.md](development.md) § +Documentation. + +--- + +## Current snapshot + +| Field | Value | +|---|---| +| Chain head | `user_version = 9` | +| Files | `0001_init.sql` … `0009_application_auth_ips.sql` (9 files) | +| Database file | `/data/selfpost.db` (bind mount) | +| Compatibility | 1.x MINOR releases must boot a `1.0.0` data directory | + +Last updated with release **1.9.0** (`0009_application_auth_ips`). + +--- + +## How migrations run + +1. Migrations are embedded at build time (`//go:embed migrations/*.sql`). +2. Filenames are sorted lexicographically; **file order = version number**. +3. `PRAGMA user_version` records progress: after file *i* (0-based), version is + *i + 1*. +4. Each pending migration runs in its own transaction, then bumps `user_version`. +5. There is no down-migration; fixes ship as a new `00NN_*.sql` file. + +```text +0001_init.sql → user_version 1 +0002_sessions.sql → user_version 2 +… +0009_application_auth_ips.sql → user_version 9 +``` + +**Implication:** never delete, rename, or reorder migration files while 1.x must +stay compatible with `1.0.0`. Git history keeps old files; only a 2.x cut may +replace the embedded set (see [Planned 2.x squash](#planned-2x-squash)). + +--- + +## Migration chain + +| Ver | File | Shipped | Kind | Summary | +|-----|------|---------|------|---------| +| 1 | `0001_init.sql` | ≤ 1.0.0 | DDL | Core schema: `admin`, `settings`, `domains`, `applications`, `application_addresses`, `send_log`, `rate_limits` | +| 2 | `0002_sessions.sql` | ≤ 1.0.0 | DDL | `sessions` (token hash, sliding idle expiry) | +| 3 | `0003_logtail_state.sql` | 0.5.0 | DDL | `logtail_state` (log-tailer read offset + fingerprint) | +| 4 | `0004_dmarc_report_email.sql` | 1.1.0 | DDL | `admin.dmarc_report_email`, `domains.dmarc_rua` | +| 5 | `0005_panel_users.sql` | 1.2.0 | DDL + data | `users`, `user_domains`; migrate single `admin` row; copy profile `dmarc_report_email` to `settings`; **DROP `admin`** | +| 6 | `0006_inbound_relay.sql` | 1.4.0 | DDL | `inbound_domains`, `inbound_transports`, `inbound_recipients` | +| 7 | `0007_rate_limit_auto.sql` | 1.6.0 | DDL | `rate_limits.mode`, `auto_multiplier`, `auto_updated_at` | +| 8 | `0008_dmarc_reports.sql` | 1.7.0 | DDL | `dmarc_reports`, `dmarc_report_records` | +| 9 | `0009_application_auth_ips.sql` | 1.9.0 | DDL + data | `applications.auth_ip_restrict`, `auth_allowed_ips`; move legacy app `rate_limits.allowed_ips` into auth columns; clear those IPs on rate limits | + +**Kind:** *DDL* — schema only; *data* — `INSERT`/`UPDATE` that must stay correct +for operators upgrading from older 1.x images. + +Feature plans that introduced schema work: [inbound-relay](plans/inbound-relay.md), +[domain-stats-auto-ratelimit](plans/domain-stats-auto-ratelimit.md), +[dmarc-reports](plans/dmarc-reports.md). + +--- + +## Schema after head (v9) + +Tables present in a fully migrated database: + +| Table | Introduced | Role | +|-------|------------|------| +| `settings` | 0001 | Key/value panel settings (retention, profile flags) | +| `domains` | 0001 (+ `dmarc_rua` in 0004) | Sending domains | +| `applications` | 0001 (+ auth IP cols in 0009) | SASL applications per domain | +| `application_addresses` | 0001 | Explicit From addresses (`list` mode) | +| `send_log` | 0001 | Delivery journal | +| `rate_limits` | 0001 (+ auto cols in 0007) | Level-2 limits per domain/application | +| `sessions` | 0002 | Panel login sessions | +| `logtail_state` | 0003 | Log-tailer persistence | +| `users` | 0005 | Panel users (`global`, `domain_admin`) | +| `user_domains` | 0005 | Domain-admin assignments | +| `inbound_domains` | 0006 | Inbound relay domains | +| `inbound_transports` | 0006 | Upstream host/port/TLS per inbound domain | +| `inbound_recipients` | 0006 | Allow-list when `recipient_mode = list` | +| `dmarc_reports` | 0008 | Parsed aggregate report summaries | +| `dmarc_report_records` | 0008 | Per-source rows inside a report | + +**Not present after v9:** `admin` (dropped in 0005). + +--- + +## Legacy and fresh-install artefacts + +These are intentional in 1.x; they are the main motivation for +[schema-squash](roadmap.md#schema-squash) at 2.x. + +### `admin` table (0001 → 0005) + +On a **new** 1.x data directory the chain still: + +1. Creates `admin` (0001), +2. Adds `admin.dmarc_report_email` (0004), +3. Copies the row into `users` and drops `admin` (0005). + +Functionally harmless; confusing when reading migrations or inferring schema from +code. A 2.x baseline should define `users` directly and omit `admin`. + +### Application trusted IPs (0009) + +Before 1.9.0, client IP restriction for an application lived in +`rate_limits.allowed_ips` (`scope = application`). Migration 0009 copies non-empty +values into `applications.auth_allowed_ips`, sets `auth_ip_restrict = 1`, and +clears `rate_limits.allowed_ips` for application scope. Level-2 limits no longer +carry IP bindings; auth and rate limiting are separate concerns. + +### Profile DMARC email (0004 → 0005) + +`0004` adds `admin.dmarc_report_email`. `0005` copies it into the migrated global +`users` row and into `settings` key `dmarc_report_email`. Per-domain overrides +remain on `domains.dmarc_rua`. + +### Documentation references to migration numbers + +Other docs cite migrations by number (e.g. architecture § sessions → `0002`). +After a 2.x squash, those references remain valid for **upgrade history** and +git; fresh 2.x installs only run the baseline plus post-2.x files. + +--- + +## Rules for 1.x changes + +1. **Add** the next `00NN_descriptive_name.sql`; do not edit shipped migrations. +2. **Bump this file:** snapshot table, chain row, schema table if needed. +3. **CHANGELOG** under the release that ships the migration. +4. **Operator impact:** if upgrade behaviour matters, note it in [guide.md](guide.md). +5. **Backup manifest** is separate: restore requires matching binary version + ([architecture.md](architecture.md) § Persistence); it does not replace + running pending SQLite migrations. + +### Checklist for a new migration + +- [ ] File name is next integer, zero-padded four digits. +- [ ] SQL is idempotent in spirit (runs once per DB; guard with schema state, not + “IF NOT EXISTS” everywhere unless needed). +- [ ] Data migrations handle empty/partial state (e.g. no `admin` row on re-run is + impossible; mid-upgrade failure is recovered by re-running the same file + only if the transaction failed before `user_version` bump). +- [ ] Row added to [Migration chain](#migration-chain) and [Current snapshot](#current-snapshot). +- [ ] `go test ./...` (store and dependents). + +--- + +## Planned 2.x squash + +Tracked as [schema-squash](roadmap.md#schema-squash). **Not** a reason to cut 2.x +on its own — only bundled with another breaking change or an explicit major. + +**Goal:** embed one baseline SQL file equal to the v9 schema (plus any later 1.x +migrations if 2.x is cut later), instead of the full 1.x chain. Fresh 2.x +`/data` directories skip create-then-drop `admin`. + +**Upgrade gate (required when squash ships):** + +| `user_version` | 2.x behaviour | +|---|---| +| `0` (empty DB) | Apply baseline; set `user_version` to new chain head | +| `>= 9` (fully migrated 1.x) | Skip; schema already matches baseline | +| `1`…`8` (mid-chain 1.x) | **Refuse to start** — run the last 1.x image once, then 2.x | + +Adjust the `>= N` and `1`…`N-1` thresholds to the chain head at cut time. + +**Done when:** baseline embedded; gate tested; [guide.md](guide.md) states that +2.x will not open an unfinished 1.x database. + +--- + +## Restore vs migrate + +| Mechanism | What it checks | +|-----------|----------------| +| SQLite `user_version` | Which embedded migrations have run on this `selfpost.db` | +| Backup `manifest.json` | Binary/image version after full restore | + +An operator can have a matching manifest after restore but still need migrations +if they restored an older DB snapshot with a newer binary — normal `migrate()` +applies pending files. The 2.x gate adds a **refusal** for half-upgraded 1.x DBs +when the old chain is no longer embedded.