mix/selfpost
1
0
Fork 0
Code Issues Pull Requests Actions 21 Packages Projects Releases Wiki Activity

release: 1.9.1: Port-25 Postfix startup with inbound/DMARC/antispam flags

Browse Source 
Port-25 Postfix startup when inbound relay, DMARC ingest, or the inbound antispam milter are enabled. Schema-migration reference and roadmap plans for inbound antispam and quarantine. Upgrading from 1.9.0 is a tag bump; no migration.
Added:
  - [docs/plans/inbound-antispam-panel.md](docs/plans/inbound-antispam-panel.md) —
  - [docs/plans/inbound-quarantine.md](docs/plans/inbound-quarantine.md) — candidate
  - [docs/schema-migrations.md](docs/schema-migrations.md) — living reference for the
Changed:
  - [docs/roadmap.md](docs/roadmap.md) — `inbound-antispam-panel` (agreed, 1/12)
  - [docs/development.md](docs/development.md) — plan checklists may override
  - [docs/development.md](docs/development.md), [docs/roadmap.md](docs/roadmap.md),
Fixed:
  - `build/postfix-config.sh`: enabling inbound relay, DMARC ingest, and/or the

Full detail: CHANGELOG.md (1.9.1).
This commit is contained in:
mix
2026-09-22 15:24:53 +03:00
parent 7b8685d038
commit 967d3d95de
9 changed files with 666 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+1 -1
View File
@@ -318,7 +318,7 @@ single-connection trade-off that follows from it.
| Path | Contents |
|---|---|
| `selfpost.db` | SQLite: domains, apps, admin, sessions, send log, L2 limits, log-tailer offset, inbound relay domains |
| `selfpost.db` | SQLite panel state (domains, apps, users, sessions, send log, L2 limits, log-tailer offset, inbound relay, DMARC reports) — see [schema-migrations.md](schema-migrations.md) |
| `setup-token` | First-run setup token file |
| `opendkim/` | DKIM keys + tables |
| `sasl/sasldb2` | Application SASL credentials |
docs/development.md
+8
View File
@@ -64,6 +64,11 @@ obligation — not because the convention around AI authorship shifts.
Default rule: risk-critical → Opus; UI / docs / boilerplate → Sonnet; trivial
mechanics → Haiku. Reviewers must not be the author of the code under review.
**Composer → Opus → Fable** (when a plan specifies it): **Composer** writes all
implementation for that item (code, tests, docs); **Opus** does the technical
review of the result; **Fable** does the security review. A plan's checklist
overrides the default routing table for that item only.
---
## Plan checklists
@@ -424,6 +429,7 @@ There is no `docs/archive/` directory.
| As-built design | [architecture.md](architecture.md) |
| Development process (this file) | [development.md](development.md) |
| Security requirements and accepted risks | [security.md](security.md) |
| SQLite schema and migrations | [schema-migrations.md](schema-migrations.md) |
| Roadmap (1.x+) | [roadmap.md](roadmap.md) |
| Active design plans | [plans/](plans/) |
| Release history | [CHANGELOG.md](../CHANGELOG.md) |
@@ -453,6 +459,8 @@ Out of scope for v1.x: `CONTRIBUTING.md`, man pages, a separate docs site
fails on an undocumented `loadConfig` or build-script key.
3. **New gaps** go into [roadmap.md](roadmap.md) (or the active plan file), not
silent drive-by edits.
4. **Schema changes:** update [schema-migrations.md](schema-migrations.md) in the
same change as a new `internal/store/migrations/*.sql` file.
### Verifying docs against code
docs/plans/inbound-antispam-panel.md
+251
View File
@@ -0,0 +1,251 @@
# Plan: inbound-antispam-panel
**Status:** agreed
**Date:** 2026-08-19
**Version:** `1.10.0` MINOR (opt-in; no change to outbound-only or inbound-without-filter paths).
---
## Goal
Give the operator a **panel view of inbound anti-spam decisions** and **editable
allow/deny lists** that tune filter behaviour — without SelfPost shipping or
starting an anti-spam engine ([product.md](../product.md) out-of-scope list
unchanged).
Typical journal row:
| When | From | To | Subject | Decision |
|------|------|----|---------|----------|
| 2026-08-19 09:14 | `spammer@evil.example` | `user@inbound.example` | Cheap pills | reject (rspamd: BAYES_SPAM +0.95) |
Lists let the operator correct false positives/negatives (e.g. always allow
`billing@vendor.example`, block `192.0.2.0/24`).
## Context (as-built)
- Inbound relay ships in `[1.4.0]` (`INBOUND_RELAY_ENABLE`); optional
`INBOUND_ANTISPAM_MILTER` attaches an **external** filter (documented rspamd
sidecar in [deploy/antispam/docker-compose.antispam.yml](../../deploy/antispam/docker-compose.antispam.yml)).
- The inbound smtpd has **no journal-milter** today — only outbound 465/587
writes `send_log` ([architecture.md](../architecture.md)).
- Inbound UI and handlers are **global-administrator only** today
(`requireInbound` → `requireGlobal`).
## Quarantine
**Not in this plan.** Held-mail review/release is tracked separately as roadmap
candidate [inbound-quarantine](inbound-quarantine.md). The journal may still
record a `quarantine` **decision label** when the external filter reports it,
without SelfPost storing the message.
**DMARC `p=quarantine`** remains unrelated (outbound policy for receivers).
## Decisions (closed 2026-08-19)
### Capture mechanism
**Inbound journal-milter in `panel` only** — no rspamd history API polling in v1.
A separate unix socket (`INBOUND_JOURNAL_MILTER_SOCKET`, default
`/run/selfpost/inbound-journal.sock`) on the inbound smtpd only; fail-open
(`default_action=accept`).
**Milter order on port 25** (when antispam is enabled):
```text
smtpd_milters = { antispam, inbound-journal }
```
Antispam runs first; inbound-journal runs at end-of-message on the **accept**
path and reads rspamd-added headers (`X-Rspamd-Action`, `X-Rspamd-Score`, symbol
summary when present).
**Antispam rejects** do not reach end-of-message on the journal milter. v1 also
tails `mail.log` for inbound `postfix/smtpd` **`milter-reject`** / **`reject`**
lines and inserts journal rows (client IP, envelope from/to when logged, reason
text, subject empty if rejected before DATA). DMARC-ingest pipe traffic is out
of scope for this tailer.
### Journal without `INBOUND_ANTISPAM_MILTER`
Journal stays **on** whenever `INBOUND_RELAY_ENABLE=true` (not tied to the
antispam hook). Without an external filter:
- **accept** rows — message relayed (engine `selfpost`, detail empty);
- **reject** / **tempfail** rows — from the mail.log tailer (Postfix policy,
size limit, unknown recipient, etc.).
Panel copy explains that rspamd symbols appear only when the antispam milter is
configured.
### RBAC
**Global administrator only** for journal and lists — same as inbound relay. No
domain-admin access in v1.
### Retention
Mirror send-log retention:
- Settings key `inbound_spam_log_retention_days` on `/settings` (global admin).
- Default **90** days; validation **7–365**; seed from env
`INBOUND_SPAM_LOG_RETENTION_DAYS` when the setting is missing.
- Prune every **6 hours** (same cadence as send-log retention).
- Hard cap **10 000** rows after age prune (drop oldest) — busier than DMARC
reports, lighter than unbounded growth.
### Allow/deny lists
| Topic | Decision |
|-------|----------|
| Entry types | sender address, sender domain (`@domain` suffix), client IP / CIDR |
| Precedence | **deny overrides allow** |
| Scope | instance-wide (not per inbound domain in v1) |
| Where applied | **rspamd map files** under `/data/antispam/` (allow.map, deny.map); documented compose fragment mounts them into the sidecar |
| SelfPost pre-check | **out of v1** — lists live in rspamd only |
| Edit access | global administrator only |
| When inactive | lists editable in the panel but UI warns that sync applies only when `INBOUND_ANTISPAM_MILTER` is set; `postfix reload` / rspamd reload documented in guide |
List change → atomic map rewrite (same pattern as Postfix maps) → rspamd reload
via documented operator step or sidecar `SIGHUP` in the compose fragment.
### Activation
| Condition | Behaviour |
|-----------|-----------|
| `INBOUND_RELAY_ENABLE=false` | no inbound journal milter, no UI, no tables written |
| inbound on, antispam empty | journal + UI; lists visible but marked inactive for filter sync |
| inbound on + antispam set | full journal (symbols on accept) + list sync |
No separate feature flag beyond inbound relay + existing antispam env vars.
### UI
New nav item **Inbound spam** (or subsection under **Inbound**) — journal table
with filters (decision, domain, date range), list management on the same page or
a tab. Mockup: add `docs/assets/panel-ui/inbound_spam.html` in the panel step.
## Schema (migration `0010_inbound_spam_log.sql`)
**`inbound_spam_log`**
| Column | Type | Notes |
|--------|------|-------|
| `id` | INTEGER PK | |
| `inbound_domain` | TEXT NOT NULL | recipient domain (from `relay_domains`) |
| `client_ip` | TEXT NOT NULL | |
| `from_addr` | TEXT NOT NULL | envelope from |
| `to_addr` | TEXT NOT NULL | envelope to (one row per recipient) |
| `subject` | TEXT NOT NULL | truncated with `mailhdr.SubjectMaxRunes` (200) |
| `decision` | TEXT NOT NULL | `accept`, `reject`, `tempfail`, `quarantine` |
| `engine` | TEXT NOT NULL | `rspamd`, `postfix`, `selfpost` |
| `detail` | TEXT NOT NULL | symbol/score summary or log-line reason; may be empty |
| `created_at` | TEXT NOT NULL | RFC3339 UTC |
Index on `(created_at)`, `(inbound_domain, created_at)`.
**`inbound_spam_list`**
| Column | Type | Notes |
|--------|------|-------|
| `id` | INTEGER PK | |
| `list_type` | TEXT NOT NULL | `allow` or `deny` |
| `entry_type` | TEXT NOT NULL | `address`, `domain`, `ip` |
| `value` | TEXT NOT NULL | normalized entry |
| `note` | TEXT NOT NULL | operator comment; default `''` |
| `created_at` | TEXT NOT NULL | |
Unique on `(list_type, entry_type, value)`.
## Scope
**In:**
- Schema, store CRUD, retention loop, mail.log tailer extension for inbound
rejects.
- Inbound journal-milter (fail-open) + Postfix `postfix-config.sh` wiring.
- rspamd map sync under `/data/antispam/`.
- Panel journal + list UI (global admin).
- Full backup includes `/data/antispam/` and new SQLite tables.
- Tests; [guide.md](../guide.md); [architecture.md](../architecture.md).
**Out:**
- Shipping rspamd, ClamAV, or any filter binary inside the SelfPost image.
- Outbound spam filtering.
- [inbound-quarantine.md](inbound-quarantine.md) storage/release.
- rspamd rule editing, Bayes training UI, antivirus.
- MIME replay from the panel (metadata only).
- Per-domain lists; domain-admin RBAC.
- rspamd history API polling.
## Architecture
```mermaid
flowchart LR
internet["Internet :25"] --> inbound["Postfix inbound smtpd"]
inbound --> antispam["External antispam milter optional"]
antispam --> journalM["inbound-journal milter fail-open"]
journalM --> upstream["smtp upstream transport"]
journalM --> sqlite["inbound_spam_log"]
mailog["mail.log tailer"] --> sqlite
listsUI["Panel allow/deny lists"] --> maps["/data/antispam/*.map"]
maps --> antispam
sqlite --> journalUI["Panel inbound spam journal"]
```
## Security
- Same validation whitelists as inbound relay ([security.md](../security.md)).
- Journal rows contain PII — retention and global-admin-only access.
- Inbound journal milter fail-open must not block mail (backup-MX role).
- List map writes injection-safe; deny wins over allow.
## Done when
- With inbound relay + antispam sidecar, panel shows accept rows with symbols
and reject rows from mail.log tailing.
- Operator can add/remove list entries; a test message reflects allow/deny.
- With antispam hook off, journal still shows Postfix-level decisions; list UI
shows inactive sync notice.
- `go vet`, `go test`, image build green; guide and architecture updated.
## Risks
- rspamd header/symbol format drift — thin parser, pin sidecar tag in compose.
- Journal volume — retention + 10k cap.
- mail.log parse brittleness — unit tests on sample lines; inbound smtpd only.
## Dependencies
- [inbound-relay.md](inbound-relay.md) (shipped).
## Model routing (this plan)
| Role | Model |
|------|-------|
| All implementation (code, tests, docs) | **Composer** |
| Technical review (reviewer ≠ author) | **Opus** |
| Security review | **Fable** |
Composer ships each step; Opus reviews the accumulated diff once implementation
steps are done (fix loop: Composer addresses Opus findings, Opus re-checks).
Fable runs after Opus sign-off.
## Implementation checklist
Target version cut: **`1.10.0`** (MINOR). One commit per step;
[development.md](../development.md) § Plan checklists.
- [x] Agree journal fields, retention, and RBAC (this plan § Decisions) — **Composer**
- [ ] Migration `0010_inbound_spam_log.sql` — **Composer**
- [ ] Inbound journal-milter + Postfix wiring — **Composer**
- [ ] mail.log tailer: inbound reject rows — **Composer**
- [ ] List CRUD + validation + atomic rspamd map sync — **Composer**
- [ ] Panel: journal + lists UI (+ mockup) — **Composer**
- [ ] Backup includes `/data/antispam/` — **Composer**
- [ ] Unit + handler tests — **Composer**
- [ ] [guide.md](../guide.md), [architecture.md](../architecture.md) — **Composer**
- [ ] `go vet`, `go test`, e2e if applicable — **Composer**
- [ ] Technical review of inbound antispam changes — **Opus**
- [ ] Security review inbound antispam path — **Fable**
docs/plans/inbound-quarantine.md
+72
View File
@@ -0,0 +1,72 @@
# Plan: inbound-quarantine
**Status:** candidate — design TBD; needs explicit agreement before any code
**Version:** TBD (`1.x` MINOR if opt-in and backward-compatible; may warrant `2.x`
if it changes inbound storage or backup contract).
---
## Goal (draft)
Let the operator **hold suspicious inbound mail** instead of only accept/reject
at the milter — review in the panel, then release to the configured upstream,
delete, or add sender to allow/deny lists.
This is a **product boundary question**, not a settled design. The item exists
so we can compare approaches before committing.
## Why it is separate from inbound-antispam-panel
[inbound-antispam-panel.md](inbound-antispam-panel.md) covers **metadata**
(journal + lists). Quarantine implies **storing message bodies** (or a durable
reference to them) under `/data`, retention, release workflow, and a wider
attack/backup surface — closer to "mini-mailbox" than pure relay.
## Open questions (to decide before a checklist)
1. **Where mail lives**
- SelfPost store under `/data/quarantine/` (panel owns lifecycle), or
- delegate to rspamd/Redis and panel only proxies release (less storage in
SelfPost, tighter rspamd coupling).
2. **What "release" means**
- inject into Postfix as a new delivery to the domain's upstream transport;
- or manual download only (no automatic forward).
3. **Retention and caps**
- max age, max total bytes, per-domain limits; interaction with backup size.
4. **RBAC**
- global admin only vs domain-admin for assigned inbound domains.
5. **Threat model**
- malware in stored MIME, path traversal on extract, quota DoS on busy MX.
6. **Product fit**
- does this violate [product.md](../product.md) "no mailboxes" if we only
hold spam suspects briefly? Explicit product decision required.
## Context (as-built)
- Inbound relay forwards or rejects; no local delivery ([inbound-relay.md](inbound-relay.md)).
- Optional `INBOUND_ANTISPAM_MILTER` — external rspamd may quarantine on its
own today, without panel integration.
- DMARC `p=quarantine` is unrelated (outbound policy for receivers).
## Likely out of scope (until revisited)
- IMAP/webmail for quarantined mail.
- End-user self-service quarantine (non-admin recipients).
- Outbound quarantine.
## Dependencies
- [inbound-relay.md](inbound-relay.md) (shipped).
- Sensible to decide **after** or **alongside**
[inbound-antispam-panel.md](inbound-antispam-panel.md) (journal/lists), but
not blocked on it for design discussion.
## Done when (placeholder)
TBD once approach is chosen. Minimum bar would include: opt-in flag, panel
list/detail, release or discard action, retention, backup inclusion documented,
security review.
## Implementation checklist
None yet — expand this plan after the open questions above are answered.
docs/roadmap.md
+61 -23
View File
@@ -30,6 +30,8 @@ in `git log` and [CHANGELOG.md](../CHANGELOG.md).
|---|---|---|---|---|
| contributing | `CONTRIBUTING.md` | candidate | — | — |
| panel-docs | In-panel operator documentation | agreed | 6/6 | [plans/panel-docs.md](plans/panel-docs.md) |
| inbound-antispam-panel | Inbound antispam journal + allow/deny lists | agreed | 1/12 | [plans/inbound-antispam-panel.md](plans/inbound-antispam-panel.md) |
| inbound-quarantine | Inbound spam quarantine (hold / review / release) | candidate | — | [plans/inbound-quarantine.md](plans/inbound-quarantine.md) |
| schema-squash | Squash SQLite migrations into a 2.x baseline | **2.x** | — | — |
**Recommended order** (not binding): the next feature is **panel-docs** once
@@ -89,38 +91,74 @@ when checks change; not bloating every page with a second column of prose.
---
## inbound-antispam-panel
**Goal:** panel view of inbound anti-spam decisions (when, from, to, subject,
action) plus editable allow/deny lists that tune the external filter — without
shipping rspamd or another engine inside the image.
**Boundary:** metadata journal and list sync only; SelfPost does not run the
filter ([product.md](product.md)). Quarantine:
[inbound-quarantine](#inbound-quarantine). Design closed in
[plans/inbound-antispam-panel.md](plans/inbound-antispam-panel.md) § Decisions.
**Done when:** with inbound relay enabled, the operator sees journal rows
(accepts via inbound-journal milter, rejects via mail.log tailer) and can
maintain instance-wide allow/deny lists synced to rspamd maps when the antispam
hook is set.
**Dependencies / risks:** inbound relay `[1.4.0]`; rspamd map/header coupling;
journal PII; global-admin-only RBAC.
**Version:** `1.10.0` MINOR; **agreed**.
---
## inbound-quarantine
**Goal:** optional hold for suspicious inbound mail — review in the panel, then
release to upstream, discard, or tune lists. Design not settled.
**Boundary:** expands inbound from pure relay toward short-lived message
storage; may conflict with "no mailboxes" in [product.md](product.md) until
product explicitly agrees. Alternatives (rspamd-only quarantine vs SelfPost
store) are listed in [plans/inbound-quarantine.md](plans/inbound-quarantine.md).
**Done when:** TBD after open questions in the plan are answered.
**Dependencies / risks:** inbound relay `[1.4.0]`; storage/retention/backup
size; malware in held MIME; overlap with
[inbound-antispam-panel](#inbound-antispam-panel). No implementation checklist
yet.
**Version:** TBD; `candidate` until explicitly agreed.
---
## schema-squash
**Goal:** when 2.x is cut, stop shipping the 1.x migration files
(`0001_init.sql` … `0005_panel_users.sql`) in the binary and replace them with
one baseline that is the schema as of `user_version = 5`. Fresh 2.x data
directories no longer create-then-drop the historical `admin` table.
**Goal:** when 2.x is cut, stop shipping the 1.x migration chain in the binary
and replace it with one baseline equal to the schema at the then-current head.
Fresh 2.x data directories no longer create-then-drop the historical `admin`
table. Current chain, legacy notes, and gate thresholds:
[schema-migrations.md](schema-migrations.md) (update that file when migrations
ship; adjust the squash gate to its head at cut time).
**Boundary:** 1.x keeps the full chain so a 1.0.0 data directory still boots.
Do not delete, rename, or reorder those files while MINOR compatibility with
`1.0.0` holds. `migrate()` maps **file order** to `PRAGMA user_version` (`target
= i + 1`); dropping a file in 1.x would skip or mis-apply steps on existing
databases. Git history keeps the old files either way; only the embedded set
in the 2.x image changes.
**Upgrade gate (required with the squash):**
| `user_version` | 2.x behaviour |
|---|---|
| `0` (empty file) | Apply the baseline; set `user_version` to the new chain’s head |
| `>= 5` (fully migrated 1.x) | Skip; schema is already the baseline |
| `1`…`4` (mid-chain 1.x) | **Refuse to start** — boot the last 1.x once, then 2.x |
**Boundary:** 1.x keeps the full chain so a `1.0.0` data directory still boots.
Do not delete, rename, or reorder migration files while MINOR compatibility with
`1.0.0` holds. Git history keeps the old files either way; only the embedded
set in the 2.x image changes.
Restore remains a separate lock: the backup manifest version must match the
running binary ([architecture.md](architecture.md) § Persistence). It does not
replace this gate.
replace the squash upgrade gate.
**Done when:** 2.x embeds a single baseline (plus any 2.x-only migrations after
it); the gate above is tested; the operator guide says a 2.x image will not
open an unfinished 1.x database.
it); the gate in [schema-migrations.md](schema-migrations.md) is tested; the
operator guide says a 2.x image will not open an unfinished 1.x database.
**Dependencies / risks:** a decided 2.x cut (another breaking change, or an
explicit major). Squashing five short files is not a reason to cut 2.x on its
own. A missed gate leaves a `user_version = 3` database silently stuck.
explicit major). Squashing the current short chain is not a reason to cut 2.x on
its own. A missed gate leaves a mid-chain 1.x database silently stuck.
**Version:** `2.x` major only; not a 1.x item.
docs/schema-migrations.md
+196
View File
@@ -0,0 +1,196 @@
# SQLite schema and migrations
**What this file is.** A living reference for the embedded SQLite migration chain,
historical legacy that still runs on fresh 1.x installs, and the planned 2.x
schema squash. When you add or review a migration, update this file in the same
change.
**Source of truth in code:** `internal/store/migrations/*.sql`, applied by
`internal/store/store.go` (`migrate()`).
**Related:** [architecture.md](architecture.md) § Persistence;
[roadmap.md](roadmap.md) § schema-squash; [development.md](development.md) §
Documentation.
---
## Current snapshot
| Field | Value |
|---|---|
| Chain head | `user_version = 9` |
| Files | `0001_init.sql` … `0009_application_auth_ips.sql` (9 files) |
| Database file | `/data/selfpost.db` (bind mount) |
| Compatibility | 1.x MINOR releases must boot a `1.0.0` data directory |
Last updated with release **1.9.0** (`0009_application_auth_ips`).
---
## How migrations run
1. Migrations are embedded at build time (`//go:embed migrations/*.sql`).
2. Filenames are sorted lexicographically; **file order = version number**.
3. `PRAGMA user_version` records progress: after file *i* (0-based), version is
*i + 1*.
4. Each pending migration runs in its own transaction, then bumps `user_version`.
5. There is no down-migration; fixes ship as a new `00NN_*.sql` file.
```text
0001_init.sql → user_version 1
0002_sessions.sql → user_version 2
…
0009_application_auth_ips.sql → user_version 9
```
**Implication:** never delete, rename, or reorder migration files while 1.x must
stay compatible with `1.0.0`. Git history keeps old files; only a 2.x cut may
replace the embedded set (see [Planned 2.x squash](#planned-2x-squash)).
---
## Migration chain
| Ver | File | Shipped | Kind | Summary |
|-----|------|---------|------|---------|
| 1 | `0001_init.sql` | ≤ 1.0.0 | DDL | Core schema: `admin`, `settings`, `domains`, `applications`, `application_addresses`, `send_log`, `rate_limits` |
| 2 | `0002_sessions.sql` | ≤ 1.0.0 | DDL | `sessions` (token hash, sliding idle expiry) |
| 3 | `0003_logtail_state.sql` | 0.5.0 | DDL | `logtail_state` (log-tailer read offset + fingerprint) |
| 4 | `0004_dmarc_report_email.sql` | 1.1.0 | DDL | `admin.dmarc_report_email`, `domains.dmarc_rua` |
| 5 | `0005_panel_users.sql` | 1.2.0 | DDL + data | `users`, `user_domains`; migrate single `admin` row; copy profile `dmarc_report_email` to `settings`; **DROP `admin`** |
| 6 | `0006_inbound_relay.sql` | 1.4.0 | DDL | `inbound_domains`, `inbound_transports`, `inbound_recipients` |
| 7 | `0007_rate_limit_auto.sql` | 1.6.0 | DDL | `rate_limits.mode`, `auto_multiplier`, `auto_updated_at` |
| 8 | `0008_dmarc_reports.sql` | 1.7.0 | DDL | `dmarc_reports`, `dmarc_report_records` |
| 9 | `0009_application_auth_ips.sql` | 1.9.0 | DDL + data | `applications.auth_ip_restrict`, `auth_allowed_ips`; move legacy app `rate_limits.allowed_ips` into auth columns; clear those IPs on rate limits |
**Kind:** *DDL* — schema only; *data* — `INSERT`/`UPDATE` that must stay correct
for operators upgrading from older 1.x images.
Feature plans that introduced schema work: [inbound-relay](plans/inbound-relay.md),
[domain-stats-auto-ratelimit](plans/domain-stats-auto-ratelimit.md),
[dmarc-reports](plans/dmarc-reports.md).
---
## Schema after head (v9)
Tables present in a fully migrated database:
| Table | Introduced | Role |
|-------|------------|------|
| `settings` | 0001 | Key/value panel settings (retention, profile flags) |
| `domains` | 0001 (+ `dmarc_rua` in 0004) | Sending domains |
| `applications` | 0001 (+ auth IP cols in 0009) | SASL applications per domain |
| `application_addresses` | 0001 | Explicit From addresses (`list` mode) |
| `send_log` | 0001 | Delivery journal |
| `rate_limits` | 0001 (+ auto cols in 0007) | Level-2 limits per domain/application |
| `sessions` | 0002 | Panel login sessions |
| `logtail_state` | 0003 | Log-tailer persistence |
| `users` | 0005 | Panel users (`global`, `domain_admin`) |
| `user_domains` | 0005 | Domain-admin assignments |
| `inbound_domains` | 0006 | Inbound relay domains |
| `inbound_transports` | 0006 | Upstream host/port/TLS per inbound domain |
| `inbound_recipients` | 0006 | Allow-list when `recipient_mode = list` |
| `dmarc_reports` | 0008 | Parsed aggregate report summaries |
| `dmarc_report_records` | 0008 | Per-source rows inside a report |
**Not present after v9:** `admin` (dropped in 0005).
---
## Legacy and fresh-install artefacts
These are intentional in 1.x; they are the main motivation for
[schema-squash](roadmap.md#schema-squash) at 2.x.
### `admin` table (0001 → 0005)
On a **new** 1.x data directory the chain still:
1. Creates `admin` (0001),
2. Adds `admin.dmarc_report_email` (0004),
3. Copies the row into `users` and drops `admin` (0005).
Functionally harmless; confusing when reading migrations or inferring schema from
code. A 2.x baseline should define `users` directly and omit `admin`.
### Application trusted IPs (0009)
Before 1.9.0, client IP restriction for an application lived in
`rate_limits.allowed_ips` (`scope = application`). Migration 0009 copies non-empty
values into `applications.auth_allowed_ips`, sets `auth_ip_restrict = 1`, and
clears `rate_limits.allowed_ips` for application scope. Level-2 limits no longer
carry IP bindings; auth and rate limiting are separate concerns.
### Profile DMARC email (0004 → 0005)
`0004` adds `admin.dmarc_report_email`. `0005` copies it into the migrated global
`users` row and into `settings` key `dmarc_report_email`. Per-domain overrides
remain on `domains.dmarc_rua`.
### Documentation references to migration numbers
Other docs cite migrations by number (e.g. architecture § sessions → `0002`).
After a 2.x squash, those references remain valid for **upgrade history** and
git; fresh 2.x installs only run the baseline plus post-2.x files.
---
## Rules for 1.x changes
1. **Add** the next `00NN_descriptive_name.sql`; do not edit shipped migrations.
2. **Bump this file:** snapshot table, chain row, schema table if needed.
3. **CHANGELOG** under the release that ships the migration.
4. **Operator impact:** if upgrade behaviour matters, note it in [guide.md](guide.md).
5. **Backup manifest** is separate: restore requires matching binary version
([architecture.md](architecture.md) § Persistence); it does not replace
running pending SQLite migrations.
### Checklist for a new migration
- [ ] File name is next integer, zero-padded four digits.
- [ ] SQL is idempotent in spirit (runs once per DB; guard with schema state, not
“IF NOT EXISTS” everywhere unless needed).
- [ ] Data migrations handle empty/partial state (e.g. no `admin` row on re-run is
impossible; mid-upgrade failure is recovered by re-running the same file
only if the transaction failed before `user_version` bump).
- [ ] Row added to [Migration chain](#migration-chain) and [Current snapshot](#current-snapshot).
- [ ] `go test ./...` (store and dependents).
---
## Planned 2.x squash
Tracked as [schema-squash](roadmap.md#schema-squash). **Not** a reason to cut 2.x
on its own — only bundled with another breaking change or an explicit major.
**Goal:** embed one baseline SQL file equal to the v9 schema (plus any later 1.x
migrations if 2.x is cut later), instead of the full 1.x chain. Fresh 2.x
`/data` directories skip create-then-drop `admin`.
**Upgrade gate (required when squash ships):**
| `user_version` | 2.x behaviour |
|---|---|
| `0` (empty DB) | Apply baseline; set `user_version` to new chain head |
| `>= 9` (fully migrated 1.x) | Skip; schema already matches baseline |
| `1`…`8` (mid-chain 1.x) | **Refuse to start** — run the last 1.x image once, then 2.x |
Adjust the `>= N` and `1`…`N-1` thresholds to the chain head at cut time.
**Done when:** baseline embedded; gate tested; [guide.md](guide.md) states that
2.x will not open an unfinished 1.x database.
---
## Restore vs migrate
| Mechanism | What it checks |
|-----------|----------------|
| SQLite `user_version` | Which embedded migrations have run on this `selfpost.db` |
| Backup `manifest.json` | Binary/image version after full restore |
An operator can have a matching manifest after restore but still need migrations
if they restored an older DB snapshot with a newer binary — normal `migrate()`
applies pending files. The 2.x gate adds a **refusal** for half-upgraded 1.x DBs
when the old chain is no longer embedded.