inbound-antispam-panel: migration 0002, journal and filter-list tables
Step 2/12 of docs/plans/inbound-antispam-panel.md, the first migration after the 2.0 baseline (2.0.1 was cut before it, as agreed). 0002_inbound_spam_log.sql adds inbound_spam_log (one row per recipient: domain, client IP, envelope, subject, decision, engine, detail) with indexes on created_at and (inbound_domain, created_at), and inbound_spam_list (allow / deny entries, unique per list). DDL only; decision, engine, list_type and entry_type carry CHECKs, as the baseline does for its own enumerations. Nothing reads or writes the tables yet. Tests: TestBaselineSchema pins both tables and user_version 2; TestOpenMigratesA20Database opens a file holding the baseline alone and finds its user kept; TestInboundSpamConstraints. go test ./internal/store/ ok; go vet ./... ok. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -5,6 +5,13 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); version
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
- Schema: migration `0002_inbound_spam_log.sql`, the first after the 2.0
|
||||
baseline - two empty tables for the inbound antispam journal and the filter
|
||||
lists (inbound-antispam-panel). A 2.0.x database is migrated when the panel
|
||||
starts; nothing reads or writes the tables yet.
|
||||
|
||||
## [2.0.1] - 2026-10-11
|
||||
|
||||
### Changed
|
||||
|
||||
@@ -531,16 +531,17 @@ full backups also archive the operator's `./certs` PEM files when present.
|
||||
|
||||
### Database
|
||||
|
||||
The schema is one embedded baseline, `internal/store/migrations/0001_init.sql`,
|
||||
applied by `store.Open` into an empty file; `PRAGMA user_version` is `1` and
|
||||
The schema is an embedded baseline, `internal/store/migrations/0001_init.sql`,
|
||||
and the migrations after it (`0002_inbound_spam_log.sql`), applied by
|
||||
`store.Open` in file order; `PRAGMA user_version` counts the files applied and
|
||||
`PRAGMA application_id` is `0x53503230` ("SP20"). There is no 1.x migration
|
||||
chain. **2.0 starts from an empty data directory:** a database with a schema
|
||||
version but without the 2.x `application_id` — what every 1.x file looks like —
|
||||
is refused at start with `ErrForeignSchema` and left untouched, as is one whose
|
||||
version is newer than the build knows. A restored 1.x backup fails the same
|
||||
way. Domains are carried across with a domain export, whose format is not the
|
||||
database. Later schema changes are new files (`0002_*.sql`, …); once `2.0.0` is
|
||||
released the baseline is never edited. History and the per-table list:
|
||||
database. A schema change is a new file; the baseline has not been edited
|
||||
since `2.0.0`. History and the per-table list:
|
||||
[schema-migrations.md](schema-migrations.md).
|
||||
|
||||
The tables that carry the panel's model:
|
||||
|
||||
@@ -243,7 +243,7 @@ Target version cut: **`2.1.0`** (MINOR), after `2.0.0`. One commit per step;
|
||||
[development.md](../development.md) § Plan checklists.
|
||||
|
||||
- [x] Agree journal fields, retention, and RBAC (this plan § Decisions) — **Composer**
|
||||
- [ ] Migration `0002_inbound_spam_log.sql` on the 2.0 baseline — **Composer**
|
||||
- [x] Migration `0002_inbound_spam_log.sql` on the 2.0 baseline — **Composer**
|
||||
- [ ] Inbound journal-milter + Postfix wiring — **Composer**
|
||||
- [ ] mail.log tailer: inbound reject rows — **Composer**
|
||||
- [ ] List CRUD + validation + atomic rspamd map sync — **Composer**
|
||||
|
||||
+1
-1
@@ -30,7 +30,7 @@ in `git log` and [CHANGELOG.md](../CHANGELOG.md).
|
||||
|---|---|---|---|---|
|
||||
| preflight | Installation check page (`/preflight`) | candidate | — | [plans/preflight.md](plans/preflight.md) |
|
||||
| contributing | `CONTRIBUTING.md` | candidate | — | — |
|
||||
| inbound-antispam-panel | Inbound antispam journal + allow/deny lists | agreed | 1/12 | [plans/inbound-antispam-panel.md](plans/inbound-antispam-panel.md) |
|
||||
| inbound-antispam-panel | Inbound antispam journal + allow/deny lists | in progress | 2/12 | [plans/inbound-antispam-panel.md](plans/inbound-antispam-panel.md) |
|
||||
| inbound-quarantine | Inbound spam quarantine (hold / review / release) | candidate | — | [plans/inbound-quarantine.md](plans/inbound-quarantine.md) |
|
||||
| csrf-tokens | Per-session CSRF tokens on state-changing forms | candidate | — | — |
|
||||
| template-data-typing | Typed structs for template data instead of `map[string]any` | candidate | — | — |
|
||||
|
||||
@@ -17,8 +17,8 @@ When you add or review a migration, update this file in the same change.
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Chain head | `user_version = 1` |
|
||||
| Files | `0001_init.sql` (the 2.0 baseline) |
|
||||
| Chain head | `user_version = 2` |
|
||||
| Files | `0001_init.sql` (the 2.0 baseline), `0002_inbound_spam_log.sql` |
|
||||
| File mark | `PRAGMA application_id = 0x53503230` ("SP20") |
|
||||
| Database file | `/data/selfpost.db` (bind mount) |
|
||||
| Compatibility | 2.x MINOR releases must boot a `2.0.0` data directory. **Nothing is carried over from 1.x.** |
|
||||
@@ -60,10 +60,9 @@ What that means in practice:
|
||||
|
||||
**Implication:** once `2.0.0` is released, `0001_init.sql` is never edited
|
||||
again and no file is deleted, renamed or reordered — every 2.x database has
|
||||
already run it. Until that release the baseline is still being written:
|
||||
later steps of the redesign and the features queued behind it (the next one
|
||||
is `0002_inbound_spam_log.sql` of inbound-antispam-panel, `2.1.0`) add files
|
||||
after it.
|
||||
already run it. `2.0.0` and `2.0.1` shipped with the baseline alone; the
|
||||
features after them add files, the first being `0002_inbound_spam_log.sql` of
|
||||
inbound-antispam-panel (`2.1.0`).
|
||||
|
||||
---
|
||||
|
||||
@@ -72,13 +71,14 @@ after it.
|
||||
| Ver | File | Shipped | Kind | Summary |
|
||||
|-----|------|---------|------|---------|
|
||||
| 1 | `0001_init.sql` | 2.0.0 | DDL | The whole 2.0 schema, below |
|
||||
| 2 | `0002_inbound_spam_log.sql` | 2.1.0 (unreleased) | DDL | `inbound_spam_log`, `inbound_spam_list` and two indexes; nothing existing changes |
|
||||
|
||||
**Kind:** *DDL* — schema only; *data* — `INSERT`/`UPDATE` that must stay correct
|
||||
for operators upgrading from older 2.x images.
|
||||
|
||||
---
|
||||
|
||||
## Schema at head (v1)
|
||||
## Schema at head (v2)
|
||||
|
||||
| Table | Role |
|
||||
|-------|------|
|
||||
@@ -98,9 +98,12 @@ for operators upgrading from older 2.x images.
|
||||
| `user_inbound_domains` | Inbound domains assigned to a `domain` user — granted separately from the outbound ones |
|
||||
| `dmarc_reports` | Parsed aggregate report summaries |
|
||||
| `dmarc_report_records` | Per-source rows inside a report |
|
||||
| `inbound_spam_log` | Inbound antispam journal: one row per recipient of a message offered on port 25, with the decision (`accept`, `reject`, `tempfail`, `quarantine`) and who made it (`rspamd`, `postfix`, `selfpost`). Metadata only (v2) |
|
||||
| `inbound_spam_list` | Allow / deny entries for the external filter (`address`, `domain`, `ip`), instance-wide; unique per list (v2) |
|
||||
|
||||
`TestBaselineSchema` (`internal/store/schema_test.go`) pins every table, column
|
||||
and index of this list.
|
||||
and index of this list; `TestOpenMigratesA20Database` opens a database that
|
||||
holds the baseline alone.
|
||||
|
||||
### What changed against the last 1.x schema
|
||||
|
||||
@@ -129,7 +132,7 @@ for the domain. Nothing refers to a user: deleting one changes no domain.
|
||||
|
||||
1. Add `internal/store/migrations/00NN_short_name.sql` (next number).
|
||||
2. Update the table in [Migration chain](#migration-chain), the snapshot at the
|
||||
top, and the affected rows of [Schema at head](#schema-at-head-v1).
|
||||
top, and the affected rows of [Schema at head](#schema-at-head-v2).
|
||||
3. Update `TestBaselineSchema` if tables, columns or indexes change, and add a
|
||||
test that opens a database at the previous version and migrates it.
|
||||
4. A *data* migration states in a comment which 2.x versions it is written for.
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
-- Inbound antispam journal and filter lists (inbound-antispam-panel, 2.1.0).
|
||||
-- The first migration after the 2.0 baseline: DDL only, so a 2.0.x database
|
||||
-- gains two empty tables and nothing in it changes.
|
||||
|
||||
-- What happened to a message offered on port 25, one row per recipient.
|
||||
-- Metadata only - SelfPost relays inbound mail and stores none of it. Accepted
|
||||
-- messages are written by the inbound journal milter, refused ones by the
|
||||
-- mail.log tailer, which is why a row can lack a subject (refused before DATA)
|
||||
-- and why engine says who decided: the external filter, Postfix's own policy,
|
||||
-- or nobody (selfpost: relayed with no filter attached).
|
||||
CREATE TABLE inbound_spam_log (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
inbound_domain TEXT NOT NULL,
|
||||
client_ip TEXT NOT NULL,
|
||||
from_addr TEXT NOT NULL,
|
||||
to_addr TEXT NOT NULL,
|
||||
subject TEXT NOT NULL,
|
||||
decision TEXT NOT NULL CHECK (decision IN ('accept', 'reject', 'tempfail', 'quarantine')),
|
||||
engine TEXT NOT NULL CHECK (engine IN ('rspamd', 'postfix', 'selfpost')),
|
||||
detail TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL
|
||||
);
|
||||
-- Retention prunes by age; the Log page lists newest first, for the whole
|
||||
-- server or for one domain.
|
||||
CREATE INDEX idx_inbound_spam_log_created_at ON inbound_spam_log (created_at);
|
||||
CREATE INDEX idx_inbound_spam_log_domain_created_at ON inbound_spam_log (inbound_domain, created_at);
|
||||
|
||||
-- Allow / deny entries for the external filter, instance-wide. The panel
|
||||
-- writes them out as map files under /data/antispam/; SelfPost itself does not
|
||||
-- consult them. value is stored normalised, so the same entry cannot be added
|
||||
-- twice to one list.
|
||||
CREATE TABLE inbound_spam_list (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
list_type TEXT NOT NULL CHECK (list_type IN ('allow', 'deny')),
|
||||
entry_type TEXT NOT NULL CHECK (entry_type IN ('address', 'domain', 'ip')),
|
||||
value TEXT NOT NULL,
|
||||
note TEXT NOT NULL DEFAULT '',
|
||||
created_at TEXT NOT NULL,
|
||||
UNIQUE (list_type, entry_type, value)
|
||||
);
|
||||
@@ -55,6 +55,8 @@ func TestBaselineSchema(t *testing.T) {
|
||||
"user_inbound_domains": "user_id inbound_domain_id",
|
||||
"dmarc_reports": "id domain reporter report_id period_begin period_end received_at contact_email policy_p policy_sp policy_pct policy_adkim policy_aspf pass_count fail_count recipient",
|
||||
"dmarc_report_records": "id report_row_id source_ip count disposition spf_result dkim_result header_from",
|
||||
"inbound_spam_log": "id inbound_domain client_ip from_addr to_addr subject decision engine detail created_at",
|
||||
"inbound_spam_list": "id list_type entry_type value note created_at",
|
||||
}
|
||||
rows, err := st.db.Query("SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%' ORDER BY name")
|
||||
if err != nil {
|
||||
@@ -99,6 +101,7 @@ func TestBaselineSchema(t *testing.T) {
|
||||
rows.Close()
|
||||
wantIdx := []string{
|
||||
"idx_dmarc_report_records_report", "idx_dmarc_reports_domain", "idx_dmarc_reports_received",
|
||||
"idx_inbound_spam_log_created_at", "idx_inbound_spam_log_domain_created_at",
|
||||
"idx_send_log_app_login_created_at", "idx_send_log_created_at", "idx_send_log_domain", "idx_send_log_queue_id",
|
||||
"idx_sessions_expires_at",
|
||||
}
|
||||
@@ -108,8 +111,8 @@ func TestBaselineSchema(t *testing.T) {
|
||||
}
|
||||
|
||||
var version, appID int64
|
||||
if err := st.db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 1 {
|
||||
t.Errorf("user_version = %d, %v; want 1", version, err)
|
||||
if err := st.db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 2 {
|
||||
t.Errorf("user_version = %d, %v; want 2", version, err)
|
||||
}
|
||||
if err := st.db.QueryRow("PRAGMA application_id").Scan(&appID); err != nil || appID != applicationID {
|
||||
t.Errorf("application_id = %#x, %v; want %#x", appID, err, applicationID)
|
||||
@@ -156,6 +159,96 @@ func TestReopenKeepsTheDatabase(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A 2.0.x data directory - the baseline and nothing after it - is brought to
|
||||
// the head by opening it, with what it holds left alone.
|
||||
func TestOpenMigratesA20Database(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "test.db")
|
||||
baseline, err := migrationsFS.ReadFile("migrations/0001_init.sql")
|
||||
if err != nil {
|
||||
t.Fatalf("read the baseline: %v", err)
|
||||
}
|
||||
db, err := sql.Open("sqlite", "file:"+path)
|
||||
if err != nil {
|
||||
t.Fatalf("create a 2.0 db: %v", err)
|
||||
}
|
||||
for _, stmt := range []string{
|
||||
string(baseline),
|
||||
"PRAGMA user_version = 1",
|
||||
"INSERT INTO users (username, password_hash, role, created_at) VALUES ('admin', 'hash', 'global', '2026-10-11T00:00:00Z')",
|
||||
} {
|
||||
if _, err := db.Exec(stmt); err != nil {
|
||||
t.Fatalf("prepare a 2.0 db: %v", err)
|
||||
}
|
||||
}
|
||||
db.Close()
|
||||
|
||||
st, err := Open(path)
|
||||
if err != nil {
|
||||
t.Fatalf("open a 2.0 database: %v", err)
|
||||
}
|
||||
defer st.Close()
|
||||
var version int
|
||||
if err := st.db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 2 {
|
||||
t.Errorf("user_version = %d, %v; want 2", version, err)
|
||||
}
|
||||
if ok, err := st.UserExists(); err != nil || !ok {
|
||||
t.Errorf("the user of the 2.0 database: exists = %v, %v", ok, err)
|
||||
}
|
||||
for _, table := range []string{"inbound_spam_log", "inbound_spam_list"} {
|
||||
if len(tableColumns(t, st, table)) == 0 {
|
||||
t.Errorf("table %s was not created", table)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// What the antispam tables refuse: a decision, engine or list the panel does
|
||||
// not know, and the same entry twice in one list.
|
||||
func TestInboundSpamConstraints(t *testing.T) {
|
||||
st := openTestStore(t)
|
||||
logRow := func(decision, engine string) error {
|
||||
_, err := st.db.Exec(
|
||||
"INSERT INTO inbound_spam_log (inbound_domain, client_ip, from_addr, to_addr, subject, decision, engine, detail, created_at)"+
|
||||
" VALUES ('in.example.com', '192.0.2.1', 'a@example.net', 'b@in.example.com', '', ?, ?, '', '2026-10-11T00:00:00Z')",
|
||||
decision, engine)
|
||||
return err
|
||||
}
|
||||
if err := logRow("reject", "postfix"); err != nil {
|
||||
t.Fatalf("a reject by postfix: %v", err)
|
||||
}
|
||||
if err := logRow("bounce", "postfix"); err == nil {
|
||||
t.Error("decision bounce was accepted")
|
||||
}
|
||||
if err := logRow("accept", "spamassassin"); err == nil {
|
||||
t.Error("engine spamassassin was accepted")
|
||||
}
|
||||
|
||||
entry := func(list, kind, value string) error {
|
||||
_, err := st.db.Exec(
|
||||
"INSERT INTO inbound_spam_list (list_type, entry_type, value, created_at) VALUES (?, ?, ?, '2026-10-11T00:00:00Z')",
|
||||
list, kind, value)
|
||||
return err
|
||||
}
|
||||
if err := entry("allow", "address", "billing@vendor.example"); err != nil {
|
||||
t.Fatalf("an allow entry: %v", err)
|
||||
}
|
||||
if err := entry("allow", "address", "billing@vendor.example"); err == nil {
|
||||
t.Error("the same entry was added to one list twice")
|
||||
}
|
||||
if err := entry("deny", "address", "billing@vendor.example"); err != nil {
|
||||
t.Errorf("the same value on the other list: %v", err)
|
||||
}
|
||||
if err := entry("block", "address", "x@example.net"); err == nil {
|
||||
t.Error("list type block was accepted")
|
||||
}
|
||||
if err := entry("deny", "subject", "pills"); err == nil {
|
||||
t.Error("entry type subject was accepted")
|
||||
}
|
||||
var note string
|
||||
if err := st.db.QueryRow("SELECT note FROM inbound_spam_list WHERE list_type = 'allow'").Scan(¬e); err != nil || note != "" {
|
||||
t.Errorf("note of an entry added without one = %q, %v", note, err)
|
||||
}
|
||||
}
|
||||
|
||||
// A 1.x data directory must be refused, loudly. Its user_version (10 at the
|
||||
// last 1.x release) would otherwise read as "the baseline is already applied",
|
||||
// and the panel would start on tables it does not know.
|
||||
|
||||
Reference in New Issue
Block a user