mix/selfpost
1
0
Fork 0
Code Issues Pull Requests Actions 17 Packages Projects Releases Wiki Activity
Files
2f7b5d95c370db5e7503067fef0587034a889711
selfpost/docs/schema-migrations.md
T
mixandClaude Opus 5 ab2cd885c4
test / test (push) Has been cancelled
Details
review follow-up: close the gaps left by the phase1-6 pass 
Verified the tree against docs/plans/code-review-2026-08.md and finished the
tasks that were only partly done, plus two defects the pass introduced.

- bcrypt: the setup account was still hashed at DefaultCost (10). The work
  factor now has one definition, auth.BcryptCost = 12, used everywhere.
- rate limiter: protecting blocked buckets from eviction was unbounded — an
  attacker with many source addresses could grow the map by making each one
  fail its way to blocked. Blocked buckets are now evicted past
  maxBuckets * 4, nearest expiry first, logged once per window. Tests cover
  both halves.
- postqueue: the 5s timeout left the output size unbounded; reads are now
  capped at 4 MiB and report truncation.
- spec references: the ~76 `spec N` citations CHANGELOG [0.5.0] claimed were
  gone are removed from build/, deploy/, the first two migrations and the
  release workflow, with the affected comment paragraphs re-wrapped. Comments
  only; shell/YAML/SQL syntax re-checked.
- schema-migrations.md: schema section was headed v9 with the chain at v10,
  and three links pointed at deleted plan files.
- internal/configsafe: the shared empty/forbidden-character check behind the
  OpenDKIM and Postfix config writers; each writer keeps its own forbidden set.
- dropped the parseDomainRateLimitForm/parseAppRateLimitForm pass-throughs
  left behind by the parser merge (only their own tests still called them).
- plan doc now carries a status table: what shipped, what was already in
  place, what was declined (//go:build linux for the /proc reader) and the
  open CSRF-token question.

Not done, each its own phase: typed template data structs, log/slog, DMARC
service and handler tests, error-page/corrupted-backup tests, backup re-auth.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-19 20:03:05 +03:00

208 lines
9.1 KiB
Markdown
Raw Blame History

# SQLite schema and migrations
**What this file is.** A living reference for the embedded SQLite migration chain,
historical legacy that still runs on fresh 1.x installs, and the planned 2.x
schema squash. When you add or review a migration, update this file in the same
change.
**Source of truth in code:** `internal/store/migrations/*.sql`, applied by
`internal/store/store.go` (`migrate()`).
**Related:** [architecture.md](architecture.md) § Persistence;
[roadmap.md](roadmap.md) § schema-squash; [development.md](development.md) §
Documentation.
---
## Current snapshot
| Field | Value |
|---|---|
| Chain head | `user_version = 10` |
| Files | `0001_init.sql` … `0010_send_log_app_login_index.sql` (10 files) |
| Database file | `/data/selfpost.db` (bind mount) |
| Compatibility | 1.x MINOR releases must boot a `1.0.0` data directory |
Last updated in **[Unreleased]** (`0010_send_log_app_login_index`); **1.9.1**
had no schema change. The release column below records the cut a migration
shipped in — fill `0010`'s in at the next version cut.
---
## How migrations run
1. Migrations are embedded at build time (`//go:embed migrations/*.sql`).
2. Filenames are sorted lexicographically; **file order = version number**.
3. `PRAGMA user_version` records progress: after file *i* (0-based), version is
*i + 1*.
4. Each pending migration runs in its own transaction, then bumps `user_version`.
5. There is no down-migration; fixes ship as a new `00NN_*.sql` file.
```text
0001_init.sql → user_version 1
0002_sessions.sql → user_version 2
…
0009_application_auth_ips.sql → user_version 9
0010_send_log_app_login_index.sql → user_version 10
```
**Implication:** never delete, rename, or reorder migration files while 1.x must
stay compatible with `1.0.0`. Git history keeps old files; only a 2.x cut may
replace the embedded set (see [Planned 2.x squash](#planned-2x-squash)).
---
## Migration chain
| Ver | File | Shipped | Kind | Summary |
|-----|------|---------|------|---------|
| 1 | `0001_init.sql` | ≤ 1.0.0 | DDL | Core schema: `admin`, `settings`, `domains`, `applications`, `application_addresses`, `send_log`, `rate_limits` |
| 2 | `0002_sessions.sql` | ≤ 1.0.0 | DDL | `sessions` (token hash, sliding idle expiry) |
| 3 | `0003_logtail_state.sql` | 0.5.0 | DDL | `logtail_state` (log-tailer read offset + fingerprint) |
| 4 | `0004_dmarc_report_email.sql` | 1.1.0 | DDL | `admin.dmarc_report_email`, `domains.dmarc_rua` |
| 5 | `0005_panel_users.sql` | 1.2.0 | DDL + data | `users`, `user_domains`; migrate single `admin` row; copy profile `dmarc_report_email` to `settings`; **DROP `admin`** |
| 6 | `0006_inbound_relay.sql` | 1.4.0 | DDL | `inbound_domains`, `inbound_transports`, `inbound_recipients` |
| 7 | `0007_rate_limit_auto.sql` | 1.6.0 | DDL | `rate_limits.mode`, `auto_multiplier`, `auto_updated_at` |
| 8 | `0008_dmarc_reports.sql` | 1.7.0 | DDL | `dmarc_reports`, `dmarc_report_records` |
| 9 | `0009_application_auth_ips.sql` | 1.9.0 | DDL + data | `applications.auth_ip_restrict`, `auth_allowed_ips`; move legacy app `rate_limits.allowed_ips` into auth columns; clear those IPs on rate limits |
| 10 | `0010_send_log_app_login_index.sql` | Unreleased | DDL | `idx_send_log_app_login_created_at` — app-scoped rate-limit counts and per-application send statistics were scanning the `created_at` range |
**Kind:** *DDL* — schema only; *data* — `INSERT`/`UPDATE` that must stay correct
for operators upgrading from older 1.x images.
The feature plans that introduced schema work (`inbound-relay`,
`domain-stats-auto-ratelimit`, `dmarc-reports`) were deleted once they
shipped; what each migration was for is the table above, and the release it
came with is in [CHANGELOG.md](../CHANGELOG.md).
---
## Schema after head (v10)
Tables present in a fully migrated database:
| Table | Introduced | Role |
|-------|------------|------|
| `settings` | 0001 | Key/value panel settings (retention, profile flags) |
| `domains` | 0001 (+ `dmarc_rua` in 0004) | Sending domains |
| `applications` | 0001 (+ auth IP cols in 0009) | SASL applications per domain |
| `application_addresses` | 0001 | Explicit From addresses (`list` mode) |
| `send_log` | 0001 | Delivery journal |
| `rate_limits` | 0001 (+ auto cols in 0007) | Level-2 limits per domain/application |
| `sessions` | 0002 | Panel login sessions |
| `logtail_state` | 0003 | Log-tailer persistence |
| `users` | 0005 | Panel users (`global`, `domain_admin`) |
| `user_domains` | 0005 | Domain-admin assignments |
| `inbound_domains` | 0006 | Inbound relay domains |
| `inbound_transports` | 0006 | Upstream host/port/TLS per inbound domain |
| `inbound_recipients` | 0006 | Allow-list when `recipient_mode = list` |
| `dmarc_reports` | 0008 | Parsed aggregate report summaries |
| `dmarc_report_records` | 0008 | Per-source rows inside a report |
**Not present after v10:** `admin` (dropped in 0005).
---
## Legacy and fresh-install artefacts
These are intentional in 1.x; they are the main motivation for
[schema-squash](roadmap.md#schema-squash) at 2.x.
### `admin` table (0001 → 0005)
On a **new** 1.x data directory the chain still:
1. Creates `admin` (0001),
2. Adds `admin.dmarc_report_email` (0004),
3. Copies the row into `users` and drops `admin` (0005).
Functionally harmless; confusing when reading migrations or inferring schema from
code. A 2.x baseline should define `users` directly and omit `admin`.
### Application trusted IPs (0009)
Before 1.9.0, client IPs for an application lived in `rate_limits.allowed_ips`
(`scope = application`). Migration 0009 copies non-empty values into
`applications.auth_allowed_ips`, sets `auth_ip_restrict = 1`, and clears
`rate_limits.allowed_ips` for application scope. Level-2 limits no longer carry
IP bindings; auth and rate limiting are separate concerns.
Note that the two columns do not mean the same thing: the old list was
*permissive* (those IPs got the application ceiling, every other IP fell back
to the domain limit), the new one is *restrictive* (every IP not on the list is
refused for that application). The conversion branch only fires for a database
that carried application-scope `allowed_ips` from before 1.9.0.
### Profile DMARC email (0004 → 0005)
`0004` adds `admin.dmarc_report_email`. `0005` copies it into the migrated global
`users` row and into `settings` key `dmarc_report_email`. Per-domain overrides
remain on `domains.dmarc_rua`.
### Documentation references to migration numbers
Other docs cite migrations by number (e.g. architecture § sessions → `0002`).
After a 2.x squash, those references remain valid for **upgrade history** and
git; fresh 2.x installs only run the baseline plus post-2.x files.
---
## Rules for 1.x changes
1. **Add** the next `00NN_descriptive_name.sql`; do not edit shipped migrations.
2. **Bump this file:** snapshot table, chain row, schema table if needed.
3. **CHANGELOG** under the release that ships the migration.
4. **Operator impact:** if upgrade behaviour matters, note it in [guide.md](guide.md).
5. **Backup manifest** is separate: restore requires matching binary version
([architecture.md](architecture.md) § Persistence); it does not replace
running pending SQLite migrations.
### Checklist for a new migration
- [ ] File name is next integer, zero-padded four digits.
- [ ] SQL is idempotent in spirit (runs once per DB; guard with schema state, not
“IF NOT EXISTS” everywhere unless needed).
- [ ] Data migrations handle empty/partial state (e.g. no `admin` row on re-run is
impossible; mid-upgrade failure is recovered by re-running the same file
only if the transaction failed before `user_version` bump).
- [ ] Row added to [Migration chain](#migration-chain) and [Current snapshot](#current-snapshot).
- [ ] `go test ./...` (store and dependents).
---
## Planned 2.x squash
Tracked as [schema-squash](roadmap.md#schema-squash). **Not** a reason to cut 2.x
on its own — only bundled with another breaking change or an explicit major.
**Goal:** embed one baseline SQL file equal to the schema at whatever the 1.x
chain head is when the squash ships (`v10` today), instead of the full 1.x
chain. Fresh 2.x `/data` directories skip create-then-drop `admin`.
**Upgrade gate (required when squash ships):**
| `user_version` | 2.x behaviour |
|---|---|
| `0` (empty DB) | Apply baseline; set `user_version` to new chain head |
| `>= N` (fully migrated 1.x) | Skip; schema already matches baseline |
| `1`…`N-1` (mid-chain 1.x) | **Refuse to start** — run the last 1.x image once, then 2.x |
`N` is the chain head at cut time (`10` today).
**Done when:** baseline embedded; gate tested; [guide.md](guide.md) states that
2.x will not open an unfinished 1.x database.
---
## Restore vs migrate
| Mechanism | What it checks |
|-----------|----------------|
| SQLite `user_version` | Which embedded migrations have run on this `selfpost.db` |
| Backup `manifest.json` | Binary/image version after full restore |
An operator can have a matching manifest after restore but still need migrations
if they restored an older DB snapshot with a newer binary — normal `migrate()`
applies pending files. The 2.x gate adds a **refusal** for half-upgraded 1.x DBs
when the old chain is no longer embedded.
Reference in New Issue View Git Blame Copy Permalink