inbound-antispam-panel: migration 0002, journal and filter-list tables

Step 2/12 of docs/plans/inbound-antispam-panel.md, the first migration
after the 2.0 baseline (2.0.1 was cut before it, as agreed).

0002_inbound_spam_log.sql adds inbound_spam_log (one row per recipient:
domain, client IP, envelope, subject, decision, engine, detail) with
indexes on created_at and (inbound_domain, created_at), and
inbound_spam_list (allow / deny entries, unique per list). DDL only;
decision, engine, list_type and entry_type carry CHECKs, as the
baseline does for its own enumerations. Nothing reads or writes the
tables yet.

Tests: TestBaselineSchema pins both tables and user_version 2;
TestOpenMigratesA20Database opens a file holding the baseline alone and
finds its user kept; TestInboundSpamConstraints.

go test ./internal/store/ ok; go vet ./... ok.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
mix
2026-10-11 22:05:20 +03:00
co-authored by Claude Opus 5.5
parent d65d632505
commit a3cc52d6d4
7 changed files with 161 additions and 17 deletions
+7
View File
@@ -5,6 +5,13 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); version
## [Unreleased]
### Added
- Schema: migration `0002_inbound_spam_log.sql`, the first after the 2.0
baseline - two empty tables for the inbound antispam journal and the filter
lists (inbound-antispam-panel). A 2.0.x database is migrated when the panel
starts; nothing reads or writes the tables yet.
## [2.0.1] - 2026-10-11
### Changed
+5 -4
View File
@@ -531,16 +531,17 @@ full backups also archive the operator's `./certs` PEM files when present.
### Database
The schema is one embedded baseline, `internal/store/migrations/0001_init.sql`,
applied by `store.Open` into an empty file; `PRAGMA user_version` is `1` and
The schema is an embedded baseline, `internal/store/migrations/0001_init.sql`,
and the migrations after it (`0002_inbound_spam_log.sql`), applied by
`store.Open` in file order; `PRAGMA user_version` counts the files applied and
`PRAGMA application_id` is `0x53503230` ("SP20"). There is no 1.x migration
chain. **2.0 starts from an empty data directory:** a database with a schema
version but without the 2.x `application_id` — what every 1.x file looks like —
is refused at start with `ErrForeignSchema` and left untouched, as is one whose
version is newer than the build knows. A restored 1.x backup fails the same
way. Domains are carried across with a domain export, whose format is not the
database. Later schema changes are new files (`0002_*.sql`, …); once `2.0.0` is
released the baseline is never edited. History and the per-table list:
database. A schema change is a new file; the baseline has not been edited
since `2.0.0`. History and the per-table list:
[schema-migrations.md](schema-migrations.md).
The tables that carry the panel's model:
+1 -1
View File
@@ -243,7 +243,7 @@ Target version cut: **`2.1.0`** (MINOR), after `2.0.0`. One commit per step;
[development.md](../development.md) § Plan checklists.
- [x] Agree journal fields, retention, and RBAC (this plan § Decisions) — **Composer**
- [ ] Migration `0002_inbound_spam_log.sql` on the 2.0 baseline — **Composer**
- [x] Migration `0002_inbound_spam_log.sql` on the 2.0 baseline — **Composer**
- [ ] Inbound journal-milter + Postfix wiring — **Composer**
- [ ] mail.log tailer: inbound reject rows — **Composer**
- [ ] List CRUD + validation + atomic rspamd map sync — **Composer**
+1 -1
View File
@@ -30,7 +30,7 @@ in `git log` and [CHANGELOG.md](../CHANGELOG.md).
|---|---|---|---|---|
| preflight | Installation check page (`/preflight`) | candidate | — | [plans/preflight.md](plans/preflight.md) |
| contributing | `CONTRIBUTING.md` | candidate | — | — |
| inbound-antispam-panel | Inbound antispam journal + allow/deny lists | agreed | 1/12 | [plans/inbound-antispam-panel.md](plans/inbound-antispam-panel.md) |
| inbound-antispam-panel | Inbound antispam journal + allow/deny lists | in progress | 2/12 | [plans/inbound-antispam-panel.md](plans/inbound-antispam-panel.md) |
| inbound-quarantine | Inbound spam quarantine (hold / review / release) | candidate | — | [plans/inbound-quarantine.md](plans/inbound-quarantine.md) |
| csrf-tokens | Per-session CSRF tokens on state-changing forms | candidate | — | — |
| template-data-typing | Typed structs for template data instead of `map[string]any` | candidate | — | — |
+12 -9
View File
@@ -17,8 +17,8 @@ When you add or review a migration, update this file in the same change.
| Field | Value |
|---|---|
| Chain head | `user_version = 1` |
| Files | `0001_init.sql` (the 2.0 baseline) |
| Chain head | `user_version = 2` |
| Files | `0001_init.sql` (the 2.0 baseline), `0002_inbound_spam_log.sql` |
| File mark | `PRAGMA application_id = 0x53503230` ("SP20") |
| Database file | `/data/selfpost.db` (bind mount) |
| Compatibility | 2.x MINOR releases must boot a `2.0.0` data directory. **Nothing is carried over from 1.x.** |
@@ -60,10 +60,9 @@ What that means in practice:
**Implication:** once `2.0.0` is released, `0001_init.sql` is never edited
again and no file is deleted, renamed or reordered — every 2.x database has
already run it. Until that release the baseline is still being written:
later steps of the redesign and the features queued behind it (the next one
is `0002_inbound_spam_log.sql` of inbound-antispam-panel, `2.1.0`) add files
after it.
already run it. `2.0.0` and `2.0.1` shipped with the baseline alone; the
features after them add files, the first being `0002_inbound_spam_log.sql` of
inbound-antispam-panel (`2.1.0`).
---
@@ -72,13 +71,14 @@ after it.
| Ver | File | Shipped | Kind | Summary |
|-----|------|---------|------|---------|
| 1 | `0001_init.sql` | 2.0.0 | DDL | The whole 2.0 schema, below |
| 2 | `0002_inbound_spam_log.sql` | 2.1.0 (unreleased) | DDL | `inbound_spam_log`, `inbound_spam_list` and two indexes; nothing existing changes |
**Kind:** *DDL* — schema only; *data* — `INSERT`/`UPDATE` that must stay correct
for operators upgrading from older 2.x images.
---
## Schema at head (v1)
## Schema at head (v2)
| Table | Role |
|-------|------|
@@ -98,9 +98,12 @@ for operators upgrading from older 2.x images.
| `user_inbound_domains` | Inbound domains assigned to a `domain` user — granted separately from the outbound ones |
| `dmarc_reports` | Parsed aggregate report summaries |
| `dmarc_report_records` | Per-source rows inside a report |
| `inbound_spam_log` | Inbound antispam journal: one row per recipient of a message offered on port 25, with the decision (`accept`, `reject`, `tempfail`, `quarantine`) and who made it (`rspamd`, `postfix`, `selfpost`). Metadata only (v2) |
| `inbound_spam_list` | Allow / deny entries for the external filter (`address`, `domain`, `ip`), instance-wide; unique per list (v2) |
`TestBaselineSchema` (`internal/store/schema_test.go`) pins every table, column
and index of this list.
and index of this list; `TestOpenMigratesA20Database` opens a database that
holds the baseline alone.
### What changed against the last 1.x schema
@@ -129,7 +132,7 @@ for the domain. Nothing refers to a user: deleting one changes no domain.
1. Add `internal/store/migrations/00NN_short_name.sql` (next number).
2. Update the table in [Migration chain](#migration-chain), the snapshot at the
top, and the affected rows of [Schema at head](#schema-at-head-v1).
top, and the affected rows of [Schema at head](#schema-at-head-v2).
3. Update `TestBaselineSchema` if tables, columns or indexes change, and add a
test that opens a database at the previous version and migrates it.
4. A *data* migration states in a comment which 2.x versions it is written for.
@@ -0,0 +1,40 @@
-- Inbound antispam journal and filter lists (inbound-antispam-panel, 2.1.0).
-- The first migration after the 2.0 baseline: DDL only, so a 2.0.x database
-- gains two empty tables and nothing in it changes.
-- What happened to a message offered on port 25, one row per recipient.
-- Metadata only - SelfPost relays inbound mail and stores none of it. Accepted
-- messages are written by the inbound journal milter, refused ones by the
-- mail.log tailer, which is why a row can lack a subject (refused before DATA)
-- and why engine says who decided: the external filter, Postfix's own policy,
-- or nobody (selfpost: relayed with no filter attached).
CREATE TABLE inbound_spam_log (
id INTEGER PRIMARY KEY AUTOINCREMENT,
inbound_domain TEXT NOT NULL,
client_ip TEXT NOT NULL,
from_addr TEXT NOT NULL,
to_addr TEXT NOT NULL,
subject TEXT NOT NULL,
decision TEXT NOT NULL CHECK (decision IN ('accept', 'reject', 'tempfail', 'quarantine')),
engine TEXT NOT NULL CHECK (engine IN ('rspamd', 'postfix', 'selfpost')),
detail TEXT NOT NULL,
created_at TEXT NOT NULL
);
-- Retention prunes by age; the Log page lists newest first, for the whole
-- server or for one domain.
CREATE INDEX idx_inbound_spam_log_created_at ON inbound_spam_log (created_at);
CREATE INDEX idx_inbound_spam_log_domain_created_at ON inbound_spam_log (inbound_domain, created_at);
-- Allow / deny entries for the external filter, instance-wide. The panel
-- writes them out as map files under /data/antispam/; SelfPost itself does not
-- consult them. value is stored normalised, so the same entry cannot be added
-- twice to one list.
CREATE TABLE inbound_spam_list (
id INTEGER PRIMARY KEY AUTOINCREMENT,
list_type TEXT NOT NULL CHECK (list_type IN ('allow', 'deny')),
entry_type TEXT NOT NULL CHECK (entry_type IN ('address', 'domain', 'ip')),
value TEXT NOT NULL,
note TEXT NOT NULL DEFAULT '',
created_at TEXT NOT NULL,
UNIQUE (list_type, entry_type, value)
);
+95 -2
View File
@@ -55,6 +55,8 @@ func TestBaselineSchema(t *testing.T) {
"user_inbound_domains": "user_id inbound_domain_id",
"dmarc_reports": "id domain reporter report_id period_begin period_end received_at contact_email policy_p policy_sp policy_pct policy_adkim policy_aspf pass_count fail_count recipient",
"dmarc_report_records": "id report_row_id source_ip count disposition spf_result dkim_result header_from",
"inbound_spam_log": "id inbound_domain client_ip from_addr to_addr subject decision engine detail created_at",
"inbound_spam_list": "id list_type entry_type value note created_at",
}
rows, err := st.db.Query("SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%' ORDER BY name")
if err != nil {
@@ -99,6 +101,7 @@ func TestBaselineSchema(t *testing.T) {
rows.Close()
wantIdx := []string{
"idx_dmarc_report_records_report", "idx_dmarc_reports_domain", "idx_dmarc_reports_received",
"idx_inbound_spam_log_created_at", "idx_inbound_spam_log_domain_created_at",
"idx_send_log_app_login_created_at", "idx_send_log_created_at", "idx_send_log_domain", "idx_send_log_queue_id",
"idx_sessions_expires_at",
}
@@ -108,8 +111,8 @@ func TestBaselineSchema(t *testing.T) {
}
var version, appID int64
if err := st.db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 1 {
t.Errorf("user_version = %d, %v; want 1", version, err)
if err := st.db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 2 {
t.Errorf("user_version = %d, %v; want 2", version, err)
}
if err := st.db.QueryRow("PRAGMA application_id").Scan(&appID); err != nil || appID != applicationID {
t.Errorf("application_id = %#x, %v; want %#x", appID, err, applicationID)
@@ -156,6 +159,96 @@ func TestReopenKeepsTheDatabase(t *testing.T) {
}
}
// A 2.0.x data directory - the baseline and nothing after it - is brought to
// the head by opening it, with what it holds left alone.
func TestOpenMigratesA20Database(t *testing.T) {
path := filepath.Join(t.TempDir(), "test.db")
baseline, err := migrationsFS.ReadFile("migrations/0001_init.sql")
if err != nil {
t.Fatalf("read the baseline: %v", err)
}
db, err := sql.Open("sqlite", "file:"+path)
if err != nil {
t.Fatalf("create a 2.0 db: %v", err)
}
for _, stmt := range []string{
string(baseline),
"PRAGMA user_version = 1",
"INSERT INTO users (username, password_hash, role, created_at) VALUES ('admin', 'hash', 'global', '2026-10-11T00:00:00Z')",
} {
if _, err := db.Exec(stmt); err != nil {
t.Fatalf("prepare a 2.0 db: %v", err)
}
}
db.Close()
st, err := Open(path)
if err != nil {
t.Fatalf("open a 2.0 database: %v", err)
}
defer st.Close()
var version int
if err := st.db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 2 {
t.Errorf("user_version = %d, %v; want 2", version, err)
}
if ok, err := st.UserExists(); err != nil || !ok {
t.Errorf("the user of the 2.0 database: exists = %v, %v", ok, err)
}
for _, table := range []string{"inbound_spam_log", "inbound_spam_list"} {
if len(tableColumns(t, st, table)) == 0 {
t.Errorf("table %s was not created", table)
}
}
}
// What the antispam tables refuse: a decision, engine or list the panel does
// not know, and the same entry twice in one list.
func TestInboundSpamConstraints(t *testing.T) {
st := openTestStore(t)
logRow := func(decision, engine string) error {
_, err := st.db.Exec(
"INSERT INTO inbound_spam_log (inbound_domain, client_ip, from_addr, to_addr, subject, decision, engine, detail, created_at)"+
" VALUES ('in.example.com', '192.0.2.1', 'a@example.net', 'b@in.example.com', '', ?, ?, '', '2026-10-11T00:00:00Z')",
decision, engine)
return err
}
if err := logRow("reject", "postfix"); err != nil {
t.Fatalf("a reject by postfix: %v", err)
}
if err := logRow("bounce", "postfix"); err == nil {
t.Error("decision bounce was accepted")
}
if err := logRow("accept", "spamassassin"); err == nil {
t.Error("engine spamassassin was accepted")
}
entry := func(list, kind, value string) error {
_, err := st.db.Exec(
"INSERT INTO inbound_spam_list (list_type, entry_type, value, created_at) VALUES (?, ?, ?, '2026-10-11T00:00:00Z')",
list, kind, value)
return err
}
if err := entry("allow", "address", "billing@vendor.example"); err != nil {
t.Fatalf("an allow entry: %v", err)
}
if err := entry("allow", "address", "billing@vendor.example"); err == nil {
t.Error("the same entry was added to one list twice")
}
if err := entry("deny", "address", "billing@vendor.example"); err != nil {
t.Errorf("the same value on the other list: %v", err)
}
if err := entry("block", "address", "x@example.net"); err == nil {
t.Error("list type block was accepted")
}
if err := entry("deny", "subject", "pills"); err == nil {
t.Error("entry type subject was accepted")
}
var note string
if err := st.db.QueryRow("SELECT note FROM inbound_spam_list WHERE list_type = 'allow'").Scan(&note); err != nil || note != "" {
t.Errorf("note of an entry added without one = %q, %v", note, err)
}
}
// A 1.x data directory must be refused, loudly. Its user_version (10 at the
// last 1.x release) would otherwise read as "the baseline is already applied",
// and the panel would start on tables it does not know.