mixandClaude Opus 5.5
b8e156c9f4
Agreed by the owner on 2026-10-13 ("the e-mail must be confirmed");
notifications to an unconfirmed address settled on 2026-10-17: "Без
подтверждения слать". Mockups 2f5f5b2, route guard b7bbb2c, panel guard
0774da2.
Migration 0013: users.email_confirmed_at ('' for every address saved
before it) and user_links - the SHA-256 of a link's token, its user,
purpose (confirm, reset), address, created and expiry. A changed address
loses its mark (keepConfirmed, in UpdateUser and SaveUser).
internal/web/auth/links.go. A link is 256 bits from crypto/rand, mailed as
https://SELFPOST_HOSTNAME/... - never the request's Host. Confirmation:
24 h, sent when an address is saved (Account, or the global role's user
form) and by Account's Send the link again; its page /confirm/<token> has
a button (a GET changes nothing) and confirms only for a session of its
own account. Reset: /login/reset asks for a username and answers the same
for every name (the sign-in page, "If that account has a confirmed e-mail,
the link is on its way"), mailing after the answer; only a confirmed
address gets one, none within ten minutes of the last, five links an hour
per account, five requests in 15 minutes per client address;
/login/reset/<token> (30 min) sets the password with the link taken in
the same transaction that checks the address again
(store.TakeResetLink), then ends every session. Changing the e-mail on
Account asks for the current password. A password set on Account or the
user form removes the reset links; a restore removes every link; the stop
waits for links being mailed (Server.Drain); a rejected request's log line
hides a link's token (loggedPath).
Reviewed for security before the commit (2026-10-17): no high; two
medium and five low, all fixed here - confirming needs the account's
session and the mail does not name the account (a mistyped address was a
takeover); a reset request neither replaces the links already sent nor
spends the account's quota inside ten minutes; take-and-set atomic; the
confirmation mail sent after the answer; the stop waits for mail; a
restore deletes the links; the token kept out of the log.
Tests: store TestUserLinks, TestUserLinkTakenOnce, TestEmailConfirmation,
TestTakeResetLink; auth TestResetSendsALinkOnlyToAConfirmedAddress,
TestResetLinkSetsThePasswordOnce, TestResetLinkDiesWithTheAddress,
TestResetRequestsAreLimited, TestResetRequestsAreSpaced, TestConfirmLink,
TestLinkPathsWithoutAToken; handlers TestAccountEmailIsConfirmedByItsLink,
TestUserFormSendsANewAddressItsLink; web TestLoggedPathHidesTokens; the
view's pages and fixtures; e2e password_reset_by_e_mail runs it through
the sink (and checks a link from before the restore is dead). go vet +
go test (26 packages) and make e2e on the test server for this tree.
Docs: guide § Account, security.md (the bullet and what ends sessions),
architecture (routes), schema-migrations (row 13, users, user_links),
help #account (word for word with the mockup), roadmap row done,
CHANGELOG [Unreleased].
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
164 lines
14 KiB
Markdown
164 lines
14 KiB
Markdown
# SQLite schema and migrations
|
|
|
|
**What this file is.** A living reference for the embedded SQLite schema: the
|
|
2.0 baseline, the migrations added after it, and how a database is opened.
|
|
When you add or review a migration, update this file in the same change.
|
|
|
|
**Source of truth in code:** `internal/store/migrations/*.sql`, applied by
|
|
`internal/store/store.go` (`migrate()`).
|
|
|
|
**Related:** [architecture.md](architecture.md) § Persistence;
|
|
[plans/panel-redesign.md](plans/panel-redesign.md) § No compatibility;
|
|
[development.md](development.md) § Documentation.
|
|
|
|
---
|
|
|
|
## Current snapshot
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Chain head | `user_version = 13` |
|
|
| Files | `0001_init.sql` (the 2.0 baseline), `0002_inbound_spam_log.sql`, `0003_send_log_lines.sql`, `0004_notifications.sql`, `0005_send_log_rejected_index.sql`, `0006_sender_check.sql`, `0007_delivery_log_days.sql`, `0008_tenant_boundary.sql`, `0009_refusal_line.sql`, `0010_send_log_indexes.sql`, `0011_notify_addressees.sql`, `0012_inbound_upstream.sql` |
|
|
| File mark | `PRAGMA application_id = 0x53503230` ("SP20") |
|
|
| Database file | `/data/selfpost.db` (bind mount) |
|
|
| Compatibility | 2.x MINOR releases must boot a `2.0.0` data directory. **Nothing is carried over from 1.x.** |
|
|
|
|
---
|
|
|
|
## 2.0 starts from an empty data directory
|
|
|
|
Decided by the owner on 2026-09-21: the redesign is a breaking release, the
|
|
1.x migration chain (`0001` … `0010`, with its `admin` table created in the
|
|
first file and dropped in the fifth) is deleted from the binary, and there is
|
|
no upgrade path. The only live 1.x installation is the owner's and is set up
|
|
again on 2.0. The 1.x chain stays in git history (last present at `1.9.5`).
|
|
|
|
What that means in practice:
|
|
|
|
- A fresh `/data` gets the baseline and nothing else.
|
|
- **A 1.x database is refused, not migrated.** `store.Open` returns
|
|
`ErrForeignSchema` for a file that has a schema version but not the 2.x
|
|
`application_id`. Without that check a 1.x file at `user_version = 10`
|
|
would read as "already migrated" and the panel would start on tables it
|
|
does not know. The file is not modified. The same applies to restoring a
|
|
1.x full backup into a 2.0 container.
|
|
- A database whose version is higher than the build knows is refused too.
|
|
- Domains can still be carried by hand: a 1.x **domain export** (JSON, with
|
|
the DKIM key and applications) imports into 2.0 — the transfer format is
|
|
not the database.
|
|
|
|
---
|
|
|
|
## How migrations run
|
|
|
|
1. Migrations are embedded at build time (`//go:embed migrations/*.sql`).
|
|
2. Filenames are sorted lexicographically; **file order = version number**.
|
|
3. `PRAGMA user_version` records progress: after file *i* (0-based), version is
|
|
*i + 1*.
|
|
4. Each pending migration runs in its own transaction, then bumps `user_version`.
|
|
5. There is no down-migration; fixes ship as a new `00NN_*.sql` file.
|
|
|
|
**Implication:** once `2.0.0` is released, `0001_init.sql` is never edited
|
|
again and no file is deleted, renamed or reordered — every 2.x database has
|
|
already run it. `2.0.0` and `2.0.1` shipped with the baseline alone; the
|
|
features after them add files, the first being `0002_inbound_spam_log.sql` of
|
|
inbound-antispam-panel (`2.1.0`), then `0003_send_log_lines.sql` of
|
|
delivery-log-storage (`2.2.0`), `0004_notifications.sql` and
|
|
`0005_send_log_rejected_index.sql` of panel-notifications (`2.3.0`), and
|
|
`0006_sender_check.sql` of from-header-binding (`2.4.0`), and
|
|
`0007_delivery_log_days.sql` and `0008_tenant_boundary.sql` of the 2026-10-12
|
|
audit (`2.5.0`), `0009_refusal_line.sql` (`2.5.1`) and
|
|
`0010_send_log_indexes.sql` (`2.5.3`) and `0011_notify_addressees.sql`
|
|
(`2.5.4`), and `0012_inbound_upstream.sql` of inbound-upstream-result
|
|
(`2.10.0`).
|
|
|
|
---
|
|
|
|
## Migration chain
|
|
|
|
| Ver | File | Shipped | Kind | Summary |
|
|
|-----|------|---------|------|---------|
|
|
| 1 | `0001_init.sql` | 2.0.0 | DDL | The whole 2.0 schema, below |
|
|
| 2 | `0002_inbound_spam_log.sql` | 2.1.0 | DDL | `inbound_spam_log`, `inbound_spam_list` and two indexes; nothing existing changes |
|
|
| 3 | `0003_send_log_lines.sql` | 2.2.0 | DDL | `send_log_lines` and one index; nothing existing changes |
|
|
| 4 | `0004_notifications.sql` | 2.3.0 | DDL | `users.notify` (default 1: every user starts with notifications on) and `notify_state` |
|
|
| 5 | `0005_send_log_rejected_index.sql` | 2.3.0 | DDL | `idx_send_log_rejected`, a partial index of the refused rows of `send_log` by time; a file of its own because the test installation had run 0004 before the review asked for it |
|
|
| 6 | `0006_sender_check.sql` | 2.4.0 | DDL | `send_log.reason` and `send_log.header_from` (both default `''`: a row refused earlier keeps no reason), and `sender_check_days` |
|
|
| 7 | `0007_delivery_log_days.sql` | 2.5.0 | DDL | `delivery_log_days`: what the reconcile sweep and the log tailer did, a row per UTC day, kept a year, shown nowhere - the measurement that decides the sweep's fate (audit 2026-10-12, stage 1). `send_log.reason` gains the values `reconcile` and `reconcile-contradicted` without a schema change |
|
|
| 8 | `0008_tenant_boundary.sql` | 2.5.0 | DDL + data | `idx_applications_login_nocase`: an application login is unique without regard to case (audit 2026-10-12, H4). `send_log.domain_id` and `inbound_spam_log.inbound_domain_id` (`ON DELETE SET NULL`), each with an index, filled once by name from the registry as it was then: a row belongs to a domain by id, and a domain deleted and added again starts with an empty log (L12) |
|
|
| 9 | `0009_refusal_line.sql` | 2.5.1 | DDL | `send_log.refusal_line`: Postfix's line about a refusal at MAIL FROM (a level-2 limit, a client address off the list), which has no queue id; the log tailer puts it on the row, found by `send_log.client_ip`, the client the refusal came from; `send_log.recovered`, a refusal row the tailer wrote from `mail.log` because the panel stopped before writing it; `sender_check_days.messages` (messages deferred mid-message, counted apart from the connections turned away) and `sender_check_days.estimated` (a day whose downtime is an upper bound) (audit 2026-10-12, stage 2b) |
|
|
| 10 | `0010_send_log_indexes.sql` | 2.5.3 | DDL | Four indexes of `send_log`, each for a question that walked the table or sorted a domain's log (audit 2026-10-12, M2, stage 3): `idx_send_log_open`, partial, the queued and deferred rows by time (the tailer's sweep every 5 min); `idx_send_log_domain_id` and `idx_send_log_app_login`, a page of one domain's or one application's log in id order; `idx_send_log_refusal_line`, partial, whether a refusal line is already on a row. `TestSendLogQueriesUseTheirIndexes` checks the plans of these and of the other by-time and by-scope statements the store runs |
|
|
| 11 | `0011_notify_addressees.sql` | 2.5.4 | DDL | `notify_addressees (key, address, told_at)`: who a subject's notification is owed to and who was handed it, so a copy that failed or was held back by the hourly ceiling goes out at a later check and a subject's end goes to those told (audit 2026-10-12, stage 4); deleted with its `notify_state` row |
|
|
| 12 | `0012_inbound_upstream.sql` | 2.10.0 | DDL | `inbound_spam_log` gains `queue_id`, `upstream_status` (`''`, `queued`, `held`, `sent`, `deferred`, `bounced`; CHECK), `upstream_relay`, `upstream_reply` and `upstream_at`, and the partial index `idx_inbound_spam_log_queue_id` on rows that have a queue id: what the upstream answered about mail accepted on port 25, put on the row by the log tailer (inbound-upstream-result). Rows from before have none |
|
|
| 13 | `0013_email_links.sql` | 2.14.0 | DDL | `users.email_confirmed_at` (`''` for every address saved before it: all unconfirmed) and `user_links`, the one-time links that confirm an address and reset a password, with `idx_user_links_user` (password-reset; owner, 2026-10-13 and 2026-10-17) |
|
|
|
|
**Kind:** *DDL* — schema only; *data* — `INSERT`/`UPDATE` that must stay correct
|
|
for operators upgrading from older 2.x images.
|
|
|
|
---
|
|
|
|
## Schema at head
|
|
|
|
| Table | Role |
|
|
|-------|------|
|
|
| `settings` | Key/value settings of the instance — what is true whoever is signed in (send-log retention). Nothing that belongs to a user |
|
|
| `users` | Panel logins. `role` is `global` or `domain`; `email` is the user's own address; `notify` is the user's own switch for notification e-mails (v4); `all_domains` / `all_inbound_domains` widen a `domain` user to every domain of that direction; `email_confirmed_at` is when `email` was confirmed, `''` while it is not, cleared when it changes (v13) |
|
|
| `user_links` | The one-time links of password-reset (v13): the SHA-256 of the token, the user, `purpose` (`confirm`, `reset`; CHECK), the address it was sent to, when, and until when; one live link per user and purpose |
|
|
| `sessions` | Panel login sessions (token hash, owning user by id and deleted with them, sliding idle expiry) |
|
|
| `domains` | Sending (outbound) domains. `dmarc_rua` is where aggregate reports go (`''` = no reports) |
|
|
| `user_domains` | Outbound domains assigned to a `domain` user |
|
|
| `applications` | SASL applications per domain, with the client-IP restriction (`auth_ip_restrict`, `auth_allowed_ips`) |
|
|
| `application_addresses` | Explicit From addresses (`list` mode) |
|
|
| `rate_limits` | Level-2 limits per domain / application, manual or auto |
|
|
| `send_log` | Delivery journal. `reason` says why a row was refused (`limit`, `client`, `from`, and `sender` since 2.8.0) or that the sweep closed it (`reconcile`, `reconcile-contradicted`), `header_from` what the From: or Sender: line of a message refused for it said (v6), `refusal_line`, `client_ip` and `recovered` (v9), and `domain_id` (v8: the row belongs to a domain by id, `ON DELETE SET NULL`) |
|
|
| `logtail_state` | Log-tailer read position |
|
|
| `inbound_domains` | Inbound relay domains |
|
|
| `inbound_transports` | Upstream host / port / TLS per inbound domain |
|
|
| `inbound_recipients` | Allow-list when `recipient_mode = list` |
|
|
| `user_inbound_domains` | Inbound domains assigned to a `domain` user — granted separately from the outbound ones |
|
|
| `dmarc_reports` | Parsed aggregate report summaries |
|
|
| `dmarc_report_records` | Per-source rows inside a report |
|
|
| `inbound_spam_log` | Inbound antispam journal: one row per recipient of a message offered on port 25, with the decision (`accept`, `reject`, `tempfail`, `quarantine`) and who made it (`rspamd`, `postfix`, `selfpost`). Metadata only (v2). `inbound_domain_id` (v8) ties a row to its domain by id; `queue_id` and `upstream_status`, `upstream_relay`, `upstream_reply`, `upstream_at` (v12) are what the upstream answered for an accepted message |
|
|
| `inbound_spam_list` | Allow / deny entries for the external filter (`address`, `domain`, `ip`), instance-wide; unique per list (v2) |
|
|
| `notify_state` | What the panel has already written to its users about, one row per subject (`dns:<domain>`, `cert`, `queue`, `limit:<domain>`), so that each thing is said once and a restart says nothing again (v4) |
|
|
| `sender_check_days` | What the sender check costs, a row per day of the container's zone (`TZ`) that has something to say: seconds the panel did not answer while Postfix ran, connections turned away (`deferred`), messages deferred mid-message (`messages`), and whether the day's downtime is an upper bound (`estimated`). Kept a year (v6, v9) |
|
|
| `delivery_log_days` | What the reconcile sweep and the log tailer did, a row per day of the container's zone, kept a year. Read off a copy of the database; no page shows it (v7) |
|
|
| `notify_addressees` | Who a subject's notification is owed to and whether each was handed it; goes with the subject's `notify_state` row (v11) |
|
|
| `send_log_lines` | The `mail.log` lines Postfix wrote under a queue id, with the line's own time; what a message's page shows once the log has rotated. No reference to `send_log`: a line belongs to a row by queue id and time, and goes when no row carries its id (v3) |
|
|
|
|
`TestBaselineSchema` (`internal/store/schema_test.go`) pins every table, column
|
|
and index of this list; `TestOpenMigratesA20Database` opens a database that
|
|
holds the baseline alone.
|
|
|
|
### What changed against the last 1.x schema
|
|
|
|
| 1.x | 2.0 | Why |
|
|
|---|---|---|
|
|
| `admin` created in 0001, dropped in 0005 | never exists | the reason for the squash |
|
|
| `users.role` = `global` / `domain_admin` | `global` / `domain` | the role is a reach, not a rank — everyone in the panel is an administrator |
|
|
| `users.dmarc_report_email` | `users.email` | a user's e-mail is theirs; it is what the panel writes to, and a domain's report address can be filled with it |
|
|
| instance setting `dmarc_report_email` (mirror of the global user's field); `domains.dmarc_rua` NULL = inherit the instance default | `domains.dmarc_rua` alone: the address, `''` = no reports | a domain's report address is one of three explicit choices — no reports, hosted, a typed address; nothing is inherited and no user's default exists (2026-10-10) |
|
|
| — | `user_inbound_domains` | inbound is delegated per user, separately from outbound |
|
|
| — | `users.all_domains`, `users.all_inbound_domains` | "All" includes domains added later |
|
|
| `rate_limits.allowed_ips` (unused since 1.9.0) | gone | client IPs belong to the application (`auth_allowed_ips`) |
|
|
|
|
### Where a domain's DMARC reports go
|
|
|
|
`domains.dmarc_rua` is the address itself and the only thing stored; `''` means
|
|
no reports, and it is what a new domain has. The choice shown in Domain
|
|
settings is derived from it: `''` is no reports, the address SelfPost derives
|
|
for the domain and the server's hostname (`dmarc.HostedReportAddress`, offered
|
|
while hosted reports are enabled) is hosted, anything else is an address typed
|
|
for the domain. Nothing refers to a user: deleting one changes no domain.
|
|
|
|
---
|
|
|
|
## Adding a migration
|
|
|
|
1. Add `internal/store/migrations/00NN_short_name.sql` (next number).
|
|
2. Update the table in [Migration chain](#migration-chain), the snapshot at the
|
|
top, and the affected rows of [Schema at head](#schema-at-head).
|
|
3. Update `TestBaselineSchema` if tables, columns or indexes change, and add a
|
|
test that opens a database at the previous version and migrates it.
|
|
4. A *data* migration states in a comment which 2.x versions it is written for.
|