docs: add schema-migrations reference for SQLite chain and legacy.
test / test (push) Waiting to run
test / test (push) Waiting to run
Centralises migration history, 1.x rules, and the 2.x squash gate so roadmap and architecture no longer carry a stale v5 snapshot. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
+12
-1
@@ -5,7 +5,18 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); version
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [1.9.0] - 2026-08-18
|
||||
### Added
|
||||
|
||||
- [docs/schema-migrations.md](docs/schema-migrations.md) — living reference for the
|
||||
SQLite migration chain (`user_version` head, per-file history, legacy artefacts,
|
||||
1.x rules, and planned 2.x squash gate).
|
||||
|
||||
### Changed
|
||||
|
||||
- [docs/development.md](docs/development.md), [docs/roadmap.md](docs/roadmap.md),
|
||||
and [docs/architecture.md](docs/architecture.md) link to the new file;
|
||||
`schema-squash` defers gate thresholds to it instead of a stale v5 snapshot.
|
||||
|
||||
|
||||
Application sending controls: client IP allow-list for authorization, and
|
||||
level-2 rate limits that override the domain ceiling per application.
|
||||
|
||||
@@ -318,7 +318,7 @@ single-connection trade-off that follows from it.
|
||||
|
||||
| Path | Contents |
|
||||
|---|---|
|
||||
| `selfpost.db` | SQLite: domains, apps, admin, sessions, send log, L2 limits, log-tailer offset, inbound relay domains |
|
||||
| `selfpost.db` | SQLite panel state (domains, apps, users, sessions, send log, L2 limits, log-tailer offset, inbound relay, DMARC reports) — see [schema-migrations.md](schema-migrations.md) |
|
||||
| `setup-token` | First-run setup token file |
|
||||
| `opendkim/` | DKIM keys + tables |
|
||||
| `sasl/sasldb2` | Application SASL credentials |
|
||||
|
||||
@@ -424,6 +424,7 @@ There is no `docs/archive/` directory.
|
||||
| As-built design | [architecture.md](architecture.md) |
|
||||
| Development process (this file) | [development.md](development.md) |
|
||||
| Security requirements and accepted risks | [security.md](security.md) |
|
||||
| SQLite schema and migrations | [schema-migrations.md](schema-migrations.md) |
|
||||
| Roadmap (1.x+) | [roadmap.md](roadmap.md) |
|
||||
| Active design plans | [plans/](plans/) |
|
||||
| Release history | [CHANGELOG.md](../CHANGELOG.md) |
|
||||
@@ -453,6 +454,8 @@ Out of scope for v1.x: `CONTRIBUTING.md`, man pages, a separate docs site
|
||||
fails on an undocumented `loadConfig` or build-script key.
|
||||
3. **New gaps** go into [roadmap.md](roadmap.md) (or the active plan file), not
|
||||
silent drive-by edits.
|
||||
4. **Schema changes:** update [schema-migrations.md](schema-migrations.md) in the
|
||||
same change as a new `internal/store/migrations/*.sql` file.
|
||||
|
||||
### Verifying docs against code
|
||||
|
||||
|
||||
+15
-23
@@ -91,36 +91,28 @@ when checks change; not bloating every page with a second column of prose.
|
||||
|
||||
## schema-squash
|
||||
|
||||
**Goal:** when 2.x is cut, stop shipping the 1.x migration files
|
||||
(`0001_init.sql` … `0005_panel_users.sql`) in the binary and replace them with
|
||||
one baseline that is the schema as of `user_version = 5`. Fresh 2.x data
|
||||
directories no longer create-then-drop the historical `admin` table.
|
||||
**Goal:** when 2.x is cut, stop shipping the 1.x migration chain in the binary
|
||||
and replace it with one baseline equal to the schema at the then-current head.
|
||||
Fresh 2.x data directories no longer create-then-drop the historical `admin`
|
||||
table. Current chain, legacy notes, and gate thresholds:
|
||||
[schema-migrations.md](schema-migrations.md) (update that file when migrations
|
||||
ship; adjust the squash gate to its head at cut time).
|
||||
|
||||
**Boundary:** 1.x keeps the full chain so a 1.0.0 data directory still boots.
|
||||
Do not delete, rename, or reorder those files while MINOR compatibility with
|
||||
`1.0.0` holds. `migrate()` maps **file order** to `PRAGMA user_version` (`target
|
||||
= i + 1`); dropping a file in 1.x would skip or mis-apply steps on existing
|
||||
databases. Git history keeps the old files either way; only the embedded set
|
||||
in the 2.x image changes.
|
||||
|
||||
**Upgrade gate (required with the squash):**
|
||||
|
||||
| `user_version` | 2.x behaviour |
|
||||
|---|---|
|
||||
| `0` (empty file) | Apply the baseline; set `user_version` to the new chain’s head |
|
||||
| `>= 5` (fully migrated 1.x) | Skip; schema is already the baseline |
|
||||
| `1`…`4` (mid-chain 1.x) | **Refuse to start** — boot the last 1.x once, then 2.x |
|
||||
**Boundary:** 1.x keeps the full chain so a `1.0.0` data directory still boots.
|
||||
Do not delete, rename, or reorder migration files while MINOR compatibility with
|
||||
`1.0.0` holds. Git history keeps the old files either way; only the embedded
|
||||
set in the 2.x image changes.
|
||||
|
||||
Restore remains a separate lock: the backup manifest version must match the
|
||||
running binary ([architecture.md](architecture.md) § Persistence). It does not
|
||||
replace this gate.
|
||||
replace the squash upgrade gate.
|
||||
|
||||
**Done when:** 2.x embeds a single baseline (plus any 2.x-only migrations after
|
||||
it); the gate above is tested; the operator guide says a 2.x image will not
|
||||
open an unfinished 1.x database.
|
||||
it); the gate in [schema-migrations.md](schema-migrations.md) is tested; the
|
||||
operator guide says a 2.x image will not open an unfinished 1.x database.
|
||||
|
||||
**Dependencies / risks:** a decided 2.x cut (another breaking change, or an
|
||||
explicit major). Squashing five short files is not a reason to cut 2.x on its
|
||||
own. A missed gate leaves a `user_version = 3` database silently stuck.
|
||||
explicit major). Squashing the current short chain is not a reason to cut 2.x on
|
||||
its own. A missed gate leaves a mid-chain 1.x database silently stuck.
|
||||
**Version:** `2.x` major only; not a 1.x item.
|
||||
|
||||
|
||||
@@ -0,0 +1,196 @@
|
||||
# SQLite schema and migrations
|
||||
|
||||
**What this file is.** A living reference for the embedded SQLite migration chain,
|
||||
historical legacy that still runs on fresh 1.x installs, and the planned 2.x
|
||||
schema squash. When you add or review a migration, update this file in the same
|
||||
change.
|
||||
|
||||
**Source of truth in code:** `internal/store/migrations/*.sql`, applied by
|
||||
`internal/store/store.go` (`migrate()`).
|
||||
|
||||
**Related:** [architecture.md](architecture.md) § Persistence;
|
||||
[roadmap.md](roadmap.md) § schema-squash; [development.md](development.md) §
|
||||
Documentation.
|
||||
|
||||
---
|
||||
|
||||
## Current snapshot
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Chain head | `user_version = 9` |
|
||||
| Files | `0001_init.sql` … `0009_application_auth_ips.sql` (9 files) |
|
||||
| Database file | `/data/selfpost.db` (bind mount) |
|
||||
| Compatibility | 1.x MINOR releases must boot a `1.0.0` data directory |
|
||||
|
||||
Last updated with release **1.9.0** (`0009_application_auth_ips`).
|
||||
|
||||
---
|
||||
|
||||
## How migrations run
|
||||
|
||||
1. Migrations are embedded at build time (`//go:embed migrations/*.sql`).
|
||||
2. Filenames are sorted lexicographically; **file order = version number**.
|
||||
3. `PRAGMA user_version` records progress: after file *i* (0-based), version is
|
||||
*i + 1*.
|
||||
4. Each pending migration runs in its own transaction, then bumps `user_version`.
|
||||
5. There is no down-migration; fixes ship as a new `00NN_*.sql` file.
|
||||
|
||||
```text
|
||||
0001_init.sql → user_version 1
|
||||
0002_sessions.sql → user_version 2
|
||||
…
|
||||
0009_application_auth_ips.sql → user_version 9
|
||||
```
|
||||
|
||||
**Implication:** never delete, rename, or reorder migration files while 1.x must
|
||||
stay compatible with `1.0.0`. Git history keeps old files; only a 2.x cut may
|
||||
replace the embedded set (see [Planned 2.x squash](#planned-2x-squash)).
|
||||
|
||||
---
|
||||
|
||||
## Migration chain
|
||||
|
||||
| Ver | File | Shipped | Kind | Summary |
|
||||
|-----|------|---------|------|---------|
|
||||
| 1 | `0001_init.sql` | ≤ 1.0.0 | DDL | Core schema: `admin`, `settings`, `domains`, `applications`, `application_addresses`, `send_log`, `rate_limits` |
|
||||
| 2 | `0002_sessions.sql` | ≤ 1.0.0 | DDL | `sessions` (token hash, sliding idle expiry) |
|
||||
| 3 | `0003_logtail_state.sql` | 0.5.0 | DDL | `logtail_state` (log-tailer read offset + fingerprint) |
|
||||
| 4 | `0004_dmarc_report_email.sql` | 1.1.0 | DDL | `admin.dmarc_report_email`, `domains.dmarc_rua` |
|
||||
| 5 | `0005_panel_users.sql` | 1.2.0 | DDL + data | `users`, `user_domains`; migrate single `admin` row; copy profile `dmarc_report_email` to `settings`; **DROP `admin`** |
|
||||
| 6 | `0006_inbound_relay.sql` | 1.4.0 | DDL | `inbound_domains`, `inbound_transports`, `inbound_recipients` |
|
||||
| 7 | `0007_rate_limit_auto.sql` | 1.6.0 | DDL | `rate_limits.mode`, `auto_multiplier`, `auto_updated_at` |
|
||||
| 8 | `0008_dmarc_reports.sql` | 1.7.0 | DDL | `dmarc_reports`, `dmarc_report_records` |
|
||||
| 9 | `0009_application_auth_ips.sql` | 1.9.0 | DDL + data | `applications.auth_ip_restrict`, `auth_allowed_ips`; move legacy app `rate_limits.allowed_ips` into auth columns; clear those IPs on rate limits |
|
||||
|
||||
**Kind:** *DDL* — schema only; *data* — `INSERT`/`UPDATE` that must stay correct
|
||||
for operators upgrading from older 1.x images.
|
||||
|
||||
Feature plans that introduced schema work: [inbound-relay](plans/inbound-relay.md),
|
||||
[domain-stats-auto-ratelimit](plans/domain-stats-auto-ratelimit.md),
|
||||
[dmarc-reports](plans/dmarc-reports.md).
|
||||
|
||||
---
|
||||
|
||||
## Schema after head (v9)
|
||||
|
||||
Tables present in a fully migrated database:
|
||||
|
||||
| Table | Introduced | Role |
|
||||
|-------|------------|------|
|
||||
| `settings` | 0001 | Key/value panel settings (retention, profile flags) |
|
||||
| `domains` | 0001 (+ `dmarc_rua` in 0004) | Sending domains |
|
||||
| `applications` | 0001 (+ auth IP cols in 0009) | SASL applications per domain |
|
||||
| `application_addresses` | 0001 | Explicit From addresses (`list` mode) |
|
||||
| `send_log` | 0001 | Delivery journal |
|
||||
| `rate_limits` | 0001 (+ auto cols in 0007) | Level-2 limits per domain/application |
|
||||
| `sessions` | 0002 | Panel login sessions |
|
||||
| `logtail_state` | 0003 | Log-tailer persistence |
|
||||
| `users` | 0005 | Panel users (`global`, `domain_admin`) |
|
||||
| `user_domains` | 0005 | Domain-admin assignments |
|
||||
| `inbound_domains` | 0006 | Inbound relay domains |
|
||||
| `inbound_transports` | 0006 | Upstream host/port/TLS per inbound domain |
|
||||
| `inbound_recipients` | 0006 | Allow-list when `recipient_mode = list` |
|
||||
| `dmarc_reports` | 0008 | Parsed aggregate report summaries |
|
||||
| `dmarc_report_records` | 0008 | Per-source rows inside a report |
|
||||
|
||||
**Not present after v9:** `admin` (dropped in 0005).
|
||||
|
||||
---
|
||||
|
||||
## Legacy and fresh-install artefacts
|
||||
|
||||
These are intentional in 1.x; they are the main motivation for
|
||||
[schema-squash](roadmap.md#schema-squash) at 2.x.
|
||||
|
||||
### `admin` table (0001 → 0005)
|
||||
|
||||
On a **new** 1.x data directory the chain still:
|
||||
|
||||
1. Creates `admin` (0001),
|
||||
2. Adds `admin.dmarc_report_email` (0004),
|
||||
3. Copies the row into `users` and drops `admin` (0005).
|
||||
|
||||
Functionally harmless; confusing when reading migrations or inferring schema from
|
||||
code. A 2.x baseline should define `users` directly and omit `admin`.
|
||||
|
||||
### Application trusted IPs (0009)
|
||||
|
||||
Before 1.9.0, client IP restriction for an application lived in
|
||||
`rate_limits.allowed_ips` (`scope = application`). Migration 0009 copies non-empty
|
||||
values into `applications.auth_allowed_ips`, sets `auth_ip_restrict = 1`, and
|
||||
clears `rate_limits.allowed_ips` for application scope. Level-2 limits no longer
|
||||
carry IP bindings; auth and rate limiting are separate concerns.
|
||||
|
||||
### Profile DMARC email (0004 → 0005)
|
||||
|
||||
`0004` adds `admin.dmarc_report_email`. `0005` copies it into the migrated global
|
||||
`users` row and into `settings` key `dmarc_report_email`. Per-domain overrides
|
||||
remain on `domains.dmarc_rua`.
|
||||
|
||||
### Documentation references to migration numbers
|
||||
|
||||
Other docs cite migrations by number (e.g. architecture § sessions → `0002`).
|
||||
After a 2.x squash, those references remain valid for **upgrade history** and
|
||||
git; fresh 2.x installs only run the baseline plus post-2.x files.
|
||||
|
||||
---
|
||||
|
||||
## Rules for 1.x changes
|
||||
|
||||
1. **Add** the next `00NN_descriptive_name.sql`; do not edit shipped migrations.
|
||||
2. **Bump this file:** snapshot table, chain row, schema table if needed.
|
||||
3. **CHANGELOG** under the release that ships the migration.
|
||||
4. **Operator impact:** if upgrade behaviour matters, note it in [guide.md](guide.md).
|
||||
5. **Backup manifest** is separate: restore requires matching binary version
|
||||
([architecture.md](architecture.md) § Persistence); it does not replace
|
||||
running pending SQLite migrations.
|
||||
|
||||
### Checklist for a new migration
|
||||
|
||||
- [ ] File name is next integer, zero-padded four digits.
|
||||
- [ ] SQL is idempotent in spirit (runs once per DB; guard with schema state, not
|
||||
“IF NOT EXISTS” everywhere unless needed).
|
||||
- [ ] Data migrations handle empty/partial state (e.g. no `admin` row on re-run is
|
||||
impossible; mid-upgrade failure is recovered by re-running the same file
|
||||
only if the transaction failed before `user_version` bump).
|
||||
- [ ] Row added to [Migration chain](#migration-chain) and [Current snapshot](#current-snapshot).
|
||||
- [ ] `go test ./...` (store and dependents).
|
||||
|
||||
---
|
||||
|
||||
## Planned 2.x squash
|
||||
|
||||
Tracked as [schema-squash](roadmap.md#schema-squash). **Not** a reason to cut 2.x
|
||||
on its own — only bundled with another breaking change or an explicit major.
|
||||
|
||||
**Goal:** embed one baseline SQL file equal to the v9 schema (plus any later 1.x
|
||||
migrations if 2.x is cut later), instead of the full 1.x chain. Fresh 2.x
|
||||
`/data` directories skip create-then-drop `admin`.
|
||||
|
||||
**Upgrade gate (required when squash ships):**
|
||||
|
||||
| `user_version` | 2.x behaviour |
|
||||
|---|---|
|
||||
| `0` (empty DB) | Apply baseline; set `user_version` to new chain head |
|
||||
| `>= 9` (fully migrated 1.x) | Skip; schema already matches baseline |
|
||||
| `1`…`8` (mid-chain 1.x) | **Refuse to start** — run the last 1.x image once, then 2.x |
|
||||
|
||||
Adjust the `>= N` and `1`…`N-1` thresholds to the chain head at cut time.
|
||||
|
||||
**Done when:** baseline embedded; gate tested; [guide.md](guide.md) states that
|
||||
2.x will not open an unfinished 1.x database.
|
||||
|
||||
---
|
||||
|
||||
## Restore vs migrate
|
||||
|
||||
| Mechanism | What it checks |
|
||||
|-----------|----------------|
|
||||
| SQLite `user_version` | Which embedded migrations have run on this `selfpost.db` |
|
||||
| Backup `manifest.json` | Binary/image version after full restore |
|
||||
|
||||
An operator can have a matching manifest after restore but still need migrations
|
||||
if they restored an older DB snapshot with a newer binary — normal `migrate()`
|
||||
applies pending files. The 2.x gate adds a **refusal** for half-upgraded 1.x DBs
|
||||
when the old chain is no longer embedded.
|
||||
Reference in New Issue
Block a user