mix/selfpost
1
0
Fork 0
Code Issues Pull Requests Actions 21 Packages Projects Releases Wiki Activity
Files
f09b60aaa219e153ce7013aa5e32fe102ca604ce
selfpost/docs/schema-migrations.md
T
mixandClaude Opus 5.5 f09b60aaa2
test / test (push) Canceled after 0s
Details
panel-notifications: findings of the technical and security reviews 
Neither review (read-only agents, Opus and Fable, over 972a9f8..400aa76) has
a Critical finding. Both found the same serious one, and more:

- The notification about refused mail was built on what a rejected row says
  of itself. Its domain is the one in MAIL FROM, which the client typed and
  Postfix has not yet held against the login when the milter refuses: an
  application over its limit could have had the panel write a message per
  made-up domain, to another tenant's users among others. And the same
  status is written when an application is refused for its client address,
  which is no limit. Refusals are now counted by the domain the login is
  registered under, only for logins that exist and only while a limit is set
  on the login or its domain; one subject per sending domain. They are read
  through a partial index of their own (migration 0005): the query walked
  the whole send log every minute.
- Account and user e-mails were checked with the DMARC report validator,
  which refuses Gmail and the like - the owner's own kind of address. A new
  validate.AccountEmail accepts them and refuses a local part beginning "-".
- "Said" now means sent: what arises while nobody can be told, or while
  Postfix does not take the message, is told at a later check. Its ending
  with nobody left to tell is simply over.
- Two DNS checks must be 45 minutes apart as well as two: the last check
  before a restart and the first after it were minutes apart.
- The notifier's own mail waiting in the queue, and its bounce coming back
  to postmaster@ this server, are not the queue's subject. A listing cut at
  its size limit or with a line too long to read is no listing; postqueue -j
  gets 30 seconds.
- The tables must name the selector the panel shows, not only have the line.
- An a/mx lookup that did not answer inside the SPF check made the record
  "not authorising"; it is "cannot tell" now, for the pages too.
- A ceiling of 60 messages an hour; a pass stops between domains when the
  process is asked to stop; sendmail gets a WaitDelay; days are rounded as a
  person counts; a failed read of the switch shows it on, not off.

Recorded as accepted in security.md, with a roadmap candidate
(from-header-binding): an application can have a message signed as the
server's own name - or, as always, as another registered domain - by writing
it into the From: header, which nothing holds against the login.

Left as found: a recipient whose copy failed while another's went out is not
tried again; a certificate that lives under 14 days is reported at each
renewal.

On the test server, for this tree: go vet clean, go test ./... 23 packages
ok, make e2e ok (326 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 17:00:01 +03:00

146 lines
8.6 KiB
Markdown
Raw Blame History

# SQLite schema and migrations
**What this file is.** A living reference for the embedded SQLite schema: the
2.0 baseline, the migrations added after it, and how a database is opened.
When you add or review a migration, update this file in the same change.
**Source of truth in code:** `internal/store/migrations/*.sql`, applied by
`internal/store/store.go` (`migrate()`).
**Related:** [architecture.md](architecture.md) § Persistence;
[plans/panel-redesign.md](plans/panel-redesign.md) § No compatibility;
[development.md](development.md) § Documentation.
---
## Current snapshot
| Field | Value |
|---|---|
| Chain head | `user_version = 5` |
| Files | `0001_init.sql` (the 2.0 baseline), `0002_inbound_spam_log.sql`, `0003_send_log_lines.sql`, `0004_notifications.sql`, `0005_send_log_rejected_index.sql` |
| File mark | `PRAGMA application_id = 0x53503230` ("SP20") |
| Database file | `/data/selfpost.db` (bind mount) |
| Compatibility | 2.x MINOR releases must boot a `2.0.0` data directory. **Nothing is carried over from 1.x.** |
---
## 2.0 starts from an empty data directory
Decided by the owner on 2026-09-21: the redesign is a breaking release, the
1.x migration chain (`0001` … `0010`, with its `admin` table created in the
first file and dropped in the fifth) is deleted from the binary, and there is
no upgrade path. The only live 1.x installation is the owner's and is set up
again on 2.0. The 1.x chain stays in git history (last present at `1.9.5`).
What that means in practice:
- A fresh `/data` gets the baseline and nothing else.
- **A 1.x database is refused, not migrated.** `store.Open` returns
`ErrForeignSchema` for a file that has a schema version but not the 2.x
`application_id`. Without that check a 1.x file at `user_version = 10`
would read as "already migrated" and the panel would start on tables it
does not know. The file is not modified. The same applies to restoring a
1.x full backup into a 2.0 container.
- A database whose version is higher than the build knows is refused too.
- Domains can still be carried by hand: a 1.x **domain export** (JSON, with
the DKIM key and applications) imports into 2.0 — the transfer format is
not the database.
---
## How migrations run
1. Migrations are embedded at build time (`//go:embed migrations/*.sql`).
2. Filenames are sorted lexicographically; **file order = version number**.
3. `PRAGMA user_version` records progress: after file *i* (0-based), version is
*i + 1*.
4. Each pending migration runs in its own transaction, then bumps `user_version`.
5. There is no down-migration; fixes ship as a new `00NN_*.sql` file.
**Implication:** once `2.0.0` is released, `0001_init.sql` is never edited
again and no file is deleted, renamed or reordered — every 2.x database has
already run it. `2.0.0` and `2.0.1` shipped with the baseline alone; the
features after them add files, the first being `0002_inbound_spam_log.sql` of
inbound-antispam-panel (`2.1.0`), then `0003_send_log_lines.sql` of
delivery-log-storage (`2.2.0`), and `0004_notifications.sql` and
`0005_send_log_rejected_index.sql` of panel-notifications (`2.3.0`).
---
## Migration chain
| Ver | File | Shipped | Kind | Summary |
|-----|------|---------|------|---------|
| 1 | `0001_init.sql` | 2.0.0 | DDL | The whole 2.0 schema, below |
| 2 | `0002_inbound_spam_log.sql` | 2.1.0 | DDL | `inbound_spam_log`, `inbound_spam_list` and two indexes; nothing existing changes |
| 3 | `0003_send_log_lines.sql` | 2.2.0 | DDL | `send_log_lines` and one index; nothing existing changes |
| 4 | `0004_notifications.sql` | 2.3.0 | DDL | `users.notify` (default 1: every user starts with notifications on) and `notify_state` |
| 5 | `0005_send_log_rejected_index.sql` | 2.3.0 | DDL | `idx_send_log_rejected`, a partial index of the refused rows of `send_log` by time; a file of its own because the test installation had run 0004 before the review asked for it |
**Kind:** *DDL* — schema only; *data* — `INSERT`/`UPDATE` that must stay correct
for operators upgrading from older 2.x images.
---
## Schema at head (v2)
| Table | Role |
|-------|------|
| `settings` | Key/value settings of the instance — what is true whoever is signed in (send-log retention). Nothing that belongs to a user |
| `users` | Panel logins. `role` is `global` or `domain`; `email` is the user's own address; `notify` is the user's own switch for notification e-mails (v4); `all_domains` / `all_inbound_domains` widen a `domain` user to every domain of that direction |
| `sessions` | Panel login sessions (token hash, owning user by id and deleted with them, sliding idle expiry) |
| `domains` | Sending (outbound) domains. `dmarc_rua` is where aggregate reports go (`''` = no reports) |
| `user_domains` | Outbound domains assigned to a `domain` user |
| `applications` | SASL applications per domain, with the client-IP restriction (`auth_ip_restrict`, `auth_allowed_ips`) |
| `application_addresses` | Explicit From addresses (`list` mode) |
| `rate_limits` | Level-2 limits per domain / application, manual or auto |
| `send_log` | Delivery journal |
| `logtail_state` | Log-tailer read position |
| `inbound_domains` | Inbound relay domains |
| `inbound_transports` | Upstream host / port / TLS per inbound domain |
| `inbound_recipients` | Allow-list when `recipient_mode = list` |
| `user_inbound_domains` | Inbound domains assigned to a `domain` user — granted separately from the outbound ones |
| `dmarc_reports` | Parsed aggregate report summaries |
| `dmarc_report_records` | Per-source rows inside a report |
| `inbound_spam_log` | Inbound antispam journal: one row per recipient of a message offered on port 25, with the decision (`accept`, `reject`, `tempfail`, `quarantine`) and who made it (`rspamd`, `postfix`, `selfpost`). Metadata only (v2) |
| `inbound_spam_list` | Allow / deny entries for the external filter (`address`, `domain`, `ip`), instance-wide; unique per list (v2) |
| `notify_state` | What the panel has already written to its users about, one row per subject (`dns:<domain>`, `cert`, `queue`, `limit:<domain>`), so that each thing is said once and a restart says nothing again (v4) |
| `send_log_lines` | The `mail.log` lines Postfix wrote under a queue id, with the line's own time; what a message's page shows once the log has rotated. No reference to `send_log`: a line belongs to a row by queue id and time, and goes when no row carries its id (v3) |
`TestBaselineSchema` (`internal/store/schema_test.go`) pins every table, column
and index of this list; `TestOpenMigratesA20Database` opens a database that
holds the baseline alone.
### What changed against the last 1.x schema
| 1.x | 2.0 | Why |
|---|---|---|
| `admin` created in 0001, dropped in 0005 | never exists | the reason for the squash |
| `users.role` = `global` / `domain_admin` | `global` / `domain` | the role is a reach, not a rank — everyone in the panel is an administrator |
| `users.dmarc_report_email` | `users.email` | a user's e-mail is theirs; it is what the panel writes to, and a domain's report address can be filled with it |
| instance setting `dmarc_report_email` (mirror of the global user's field); `domains.dmarc_rua` NULL = inherit the instance default | `domains.dmarc_rua` alone: the address, `''` = no reports | a domain's report address is one of three explicit choices — no reports, hosted, a typed address; nothing is inherited and no user's default exists (2026-10-10) |
| — | `user_inbound_domains` | inbound is delegated per user, separately from outbound |
| — | `users.all_domains`, `users.all_inbound_domains` | "All" includes domains added later |
| `rate_limits.allowed_ips` (unused since 1.9.0) | gone | client IPs belong to the application (`auth_allowed_ips`) |
### Where a domain's DMARC reports go
`domains.dmarc_rua` is the address itself and the only thing stored; `''` means
no reports, and it is what a new domain has. The choice shown in Domain
settings is derived from it: `''` is no reports, the address SelfPost derives
for the domain and the server's hostname (`dmarc.HostedReportAddress`, offered
while hosted reports are enabled) is hosted, anything else is an address typed
for the domain. Nothing refers to a user: deleting one changes no domain.
---
## Adding a migration
1. Add `internal/store/migrations/00NN_short_name.sql` (next number).
2. Update the table in [Migration chain](#migration-chain), the snapshot at the
top, and the affected rows of [Schema at head](#schema-at-head-v2).
3. Update `TestBaselineSchema` if tables, columns or indexes change, and add a
test that opens a database at the previous version and migrates it.
4. A *data* migration states in a comment which 2.x versions it is written for.
Reference in New Issue View Git Blame Copy Permalink