selfpost/docs/schema-migrations.md

SQLite schema and migrations

What this file is. A living reference for the embedded SQLite migration chain, historical legacy that still runs on fresh 1.x installs, and the planned 2.x schema squash. When you add or review a migration, update this file in the same change.

Source of truth in code: internal/store/migrations/*.sql, applied by internal/store/store.go (migrate()).

Related: architecture.md § Persistence; roadmap.md § schema-squash; development.md § Documentation.

---

Current snapshot

Field	Value
Chain head	user_version = 10
Files	0001_init.sql … 0010_send_log_app_login_index.sql (10 files)
Database file	/data/selfpost.db (bind mount)
Compatibility	1.x MINOR releases must boot a 1.0.0 data directory

Last updated in [Unreleased] (0010_send_log_app_login_index); 1.9.1 had no schema change. The release column below records the cut a migration shipped in — fill 0010's in at the next version cut.

---

How migrations run

1. Migrations are embedded at build time (//go:embed migrations/*.sql).
2. Filenames are sorted lexicographically; file order = version number.
3. PRAGMA user_version records progress: after file i (0-based), version is i + 1.
4. Each pending migration runs in its own transaction, then bumps user_version.
5. There is no down-migration; fixes ship as a new 00NN_*.sql file.

0001_init.sql              → user_version 1
0002_sessions.sql          → user_version 2
…
0009_application_auth_ips.sql → user_version 9
0010_send_log_app_login_index.sql → user_version 10

Implication: never delete, rename, or reorder migration files while 1.x must stay compatible with 1.0.0. Git history keeps old files; only a 2.x cut may replace the embedded set (see Planned 2.x squash).

---

Migration chain

Ver	File	Shipped	Kind	Summary
1	0001_init.sql	≤ 1.0.0	DDL	Core schema: admin, settings, domains, applications, application_addresses, send_log, rate_limits
2	0002_sessions.sql	≤ 1.0.0	DDL	sessions (token hash, sliding idle expiry)
3	0003_logtail_state.sql	0.5.0	DDL	logtail_state (log-tailer read offset + fingerprint)
4	0004_dmarc_report_email.sql	1.1.0	DDL	admin.dmarc_report_email, domains.dmarc_rua
5	0005_panel_users.sql	1.2.0	DDL + data	users, user_domains; migrate single admin row; copy profile dmarc_report_email to settings; DROP admin
6	0006_inbound_relay.sql	1.4.0	DDL	inbound_domains, inbound_transports, inbound_recipients
7	0007_rate_limit_auto.sql	1.6.0	DDL	rate_limits.mode, auto_multiplier, auto_updated_at
8	0008_dmarc_reports.sql	1.7.0	DDL	dmarc_reports, dmarc_report_records
9	0009_application_auth_ips.sql	1.9.0	DDL + data	applications.auth_ip_restrict, auth_allowed_ips; move legacy app rate_limits.allowed_ips into auth columns; clear those IPs on rate limits
10	0010_send_log_app_login_index.sql	Unreleased	DDL	idx_send_log_app_login_created_at — app-scoped rate-limit counts and per-application send statistics were scanning the created_at range

Kind: DDL — schema only; data — INSERT/UPDATE that must stay correct for operators upgrading from older 1.x images.

Feature plans that introduced schema work: inbound-relay, domain-stats-auto-ratelimit, dmarc-reports.

---

Schema after head (v9)

Tables present in a fully migrated database:

Table	Introduced	Role
settings	0001	Key/value panel settings (retention, profile flags)
domains	0001 (+ dmarc_rua in 0004)	Sending domains
applications	0001 (+ auth IP cols in 0009)	SASL applications per domain
application_addresses	0001	Explicit From addresses (list mode)
send_log	0001	Delivery journal
rate_limits	0001 (+ auto cols in 0007)	Level-2 limits per domain/application
sessions	0002	Panel login sessions
logtail_state	0003	Log-tailer persistence
users	0005	Panel users (global, domain_admin)
user_domains	0005	Domain-admin assignments
inbound_domains	0006	Inbound relay domains
inbound_transports	0006	Upstream host/port/TLS per inbound domain
inbound_recipients	0006	Allow-list when recipient_mode = list
dmarc_reports	0008	Parsed aggregate report summaries
dmarc_report_records	0008	Per-source rows inside a report

Not present after v9: admin (dropped in 0005).

---

Legacy and fresh-install artefacts

These are intentional in 1.x; they are the main motivation for schema-squash at 2.x.

admin table (0001 → 0005)

On a new 1.x data directory the chain still:

1. Creates admin (0001),
2. Adds admin.dmarc_report_email (0004),
3. Copies the row into users and drops admin (0005).

Functionally harmless; confusing when reading migrations or inferring schema from code. A 2.x baseline should define users directly and omit admin.

Application trusted IPs (0009)

Before 1.9.0, client IPs for an application lived in rate_limits.allowed_ips (scope = application). Migration 0009 copies non-empty values into applications.auth_allowed_ips, sets auth_ip_restrict = 1, and clears rate_limits.allowed_ips for application scope. Level-2 limits no longer carry IP bindings; auth and rate limiting are separate concerns.

Note that the two columns do not mean the same thing: the old list was permissive (those IPs got the application ceiling, every other IP fell back to the domain limit), the new one is restrictive (every IP not on the list is refused for that application). The conversion branch only fires for a database that carried application-scope allowed_ips from before 1.9.0.

Profile DMARC email (0004 → 0005)

0004 adds admin.dmarc_report_email. 0005 copies it into the migrated global users row and into settings key dmarc_report_email. Per-domain overrides remain on domains.dmarc_rua.

Documentation references to migration numbers

Other docs cite migrations by number (e.g. architecture § sessions → 0002). After a 2.x squash, those references remain valid for upgrade history and git; fresh 2.x installs only run the baseline plus post-2.x files.

---

Rules for 1.x changes

1. Add the next 00NN_descriptive_name.sql; do not edit shipped migrations.
2. Bump this file: snapshot table, chain row, schema table if needed.
3. CHANGELOG under the release that ships the migration.
4. Operator impact: if upgrade behaviour matters, note it in guide.md.
5. Backup manifest is separate: restore requires matching binary version (architecture.md § Persistence); it does not replace running pending SQLite migrations.

Checklist for a new migration

• File name is next integer, zero-padded four digits.
• SQL is idempotent in spirit (runs once per DB; guard with schema state, not “IF NOT EXISTS” everywhere unless needed).
• Data migrations handle empty/partial state (e.g. no admin row on re-run is impossible; mid-upgrade failure is recovered by re-running the same file only if the transaction failed before user_version bump).
• Row added to Migration chain and Current snapshot.
• go test ./... (store and dependents).

---

Planned 2.x squash

Tracked as schema-squash. Not a reason to cut 2.x on its own — only bundled with another breaking change or an explicit major.

Goal: embed one baseline SQL file equal to the v9 schema (plus any later 1.x migrations if 2.x is cut later), instead of the full 1.x chain. Fresh 2.x /data directories skip create-then-drop admin.

Upgrade gate (required when squash ships):

user_version	2.x behaviour
0 (empty DB)	Apply baseline; set user_version to new chain head
>= 9 (fully migrated 1.x)	Skip; schema already matches baseline
1…8 (mid-chain 1.x)	Refuse to start — run the last 1.x image once, then 2.x

Adjust the >= N and 1…N-1 thresholds to the chain head at cut time.

Done when: baseline embedded; gate tested; guide.md states that 2.x will not open an unfinished 1.x database.

---

Restore vs migrate

Mechanism	What it checks
SQLite user_version	Which embedded migrations have run on this selfpost.db
Backup manifest.json	Binary/image version after full restore

An operator can have a matching manifest after restore but still need migrations if they restored an older DB snapshot with a newer binary — normal migrate() applies pending files. The 2.x gate adds a refusal for half-upgraded 1.x DBs when the old chain is no longer embedded.