diff --git a/CHANGELOG.md b/CHANGELOG.md index b65a5e6..5890273 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,13 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); version ## [Unreleased] +### Added + +- Schema: migration `0002_inbound_spam_log.sql`, the first after the 2.0 + baseline - two empty tables for the inbound antispam journal and the filter + lists (inbound-antispam-panel). A 2.0.x database is migrated when the panel + starts; nothing reads or writes the tables yet. + ## [2.0.1] - 2026-10-11 ### Changed diff --git a/docs/architecture.md b/docs/architecture.md index ba8e1ac..2ee7383 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -531,16 +531,17 @@ full backups also archive the operator's `./certs` PEM files when present. ### Database -The schema is one embedded baseline, `internal/store/migrations/0001_init.sql`, -applied by `store.Open` into an empty file; `PRAGMA user_version` is `1` and +The schema is an embedded baseline, `internal/store/migrations/0001_init.sql`, +and the migrations after it (`0002_inbound_spam_log.sql`), applied by +`store.Open` in file order; `PRAGMA user_version` counts the files applied and `PRAGMA application_id` is `0x53503230` ("SP20"). There is no 1.x migration chain. **2.0 starts from an empty data directory:** a database with a schema version but without the 2.x `application_id` — what every 1.x file looks like — is refused at start with `ErrForeignSchema` and left untouched, as is one whose version is newer than the build knows. A restored 1.x backup fails the same way. Domains are carried across with a domain export, whose format is not the -database. Later schema changes are new files (`0002_*.sql`, …); once `2.0.0` is -released the baseline is never edited. History and the per-table list: +database. A schema change is a new file; the baseline has not been edited +since `2.0.0`. History and the per-table list: [schema-migrations.md](schema-migrations.md). The tables that carry the panel's model: diff --git a/docs/plans/inbound-antispam-panel.md b/docs/plans/inbound-antispam-panel.md index 4f7c49b..99bb4f2 100644 --- a/docs/plans/inbound-antispam-panel.md +++ b/docs/plans/inbound-antispam-panel.md @@ -243,7 +243,7 @@ Target version cut: **`2.1.0`** (MINOR), after `2.0.0`. One commit per step; [development.md](../development.md) § Plan checklists. - [x] Agree journal fields, retention, and RBAC (this plan § Decisions) — **Composer** -- [ ] Migration `0002_inbound_spam_log.sql` on the 2.0 baseline — **Composer** +- [x] Migration `0002_inbound_spam_log.sql` on the 2.0 baseline — **Composer** - [ ] Inbound journal-milter + Postfix wiring — **Composer** - [ ] mail.log tailer: inbound reject rows — **Composer** - [ ] List CRUD + validation + atomic rspamd map sync — **Composer** diff --git a/docs/roadmap.md b/docs/roadmap.md index 857f963..68bfec3 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -30,7 +30,7 @@ in `git log` and [CHANGELOG.md](../CHANGELOG.md). |---|---|---|---|---| | preflight | Installation check page (`/preflight`) | candidate | — | [plans/preflight.md](plans/preflight.md) | | contributing | `CONTRIBUTING.md` | candidate | — | — | -| inbound-antispam-panel | Inbound antispam journal + allow/deny lists | agreed | 1/12 | [plans/inbound-antispam-panel.md](plans/inbound-antispam-panel.md) | +| inbound-antispam-panel | Inbound antispam journal + allow/deny lists | in progress | 2/12 | [plans/inbound-antispam-panel.md](plans/inbound-antispam-panel.md) | | inbound-quarantine | Inbound spam quarantine (hold / review / release) | candidate | — | [plans/inbound-quarantine.md](plans/inbound-quarantine.md) | | csrf-tokens | Per-session CSRF tokens on state-changing forms | candidate | — | — | | template-data-typing | Typed structs for template data instead of `map[string]any` | candidate | — | — | diff --git a/docs/schema-migrations.md b/docs/schema-migrations.md index acd0ec0..050d9fa 100644 --- a/docs/schema-migrations.md +++ b/docs/schema-migrations.md @@ -17,8 +17,8 @@ When you add or review a migration, update this file in the same change. | Field | Value | |---|---| -| Chain head | `user_version = 1` | -| Files | `0001_init.sql` (the 2.0 baseline) | +| Chain head | `user_version = 2` | +| Files | `0001_init.sql` (the 2.0 baseline), `0002_inbound_spam_log.sql` | | File mark | `PRAGMA application_id = 0x53503230` ("SP20") | | Database file | `/data/selfpost.db` (bind mount) | | Compatibility | 2.x MINOR releases must boot a `2.0.0` data directory. **Nothing is carried over from 1.x.** | @@ -60,10 +60,9 @@ What that means in practice: **Implication:** once `2.0.0` is released, `0001_init.sql` is never edited again and no file is deleted, renamed or reordered — every 2.x database has -already run it. Until that release the baseline is still being written: -later steps of the redesign and the features queued behind it (the next one -is `0002_inbound_spam_log.sql` of inbound-antispam-panel, `2.1.0`) add files -after it. +already run it. `2.0.0` and `2.0.1` shipped with the baseline alone; the +features after them add files, the first being `0002_inbound_spam_log.sql` of +inbound-antispam-panel (`2.1.0`). --- @@ -72,13 +71,14 @@ after it. | Ver | File | Shipped | Kind | Summary | |-----|------|---------|------|---------| | 1 | `0001_init.sql` | 2.0.0 | DDL | The whole 2.0 schema, below | +| 2 | `0002_inbound_spam_log.sql` | 2.1.0 (unreleased) | DDL | `inbound_spam_log`, `inbound_spam_list` and two indexes; nothing existing changes | **Kind:** *DDL* — schema only; *data* — `INSERT`/`UPDATE` that must stay correct for operators upgrading from older 2.x images. --- -## Schema at head (v1) +## Schema at head (v2) | Table | Role | |-------|------| @@ -98,9 +98,12 @@ for operators upgrading from older 2.x images. | `user_inbound_domains` | Inbound domains assigned to a `domain` user — granted separately from the outbound ones | | `dmarc_reports` | Parsed aggregate report summaries | | `dmarc_report_records` | Per-source rows inside a report | +| `inbound_spam_log` | Inbound antispam journal: one row per recipient of a message offered on port 25, with the decision (`accept`, `reject`, `tempfail`, `quarantine`) and who made it (`rspamd`, `postfix`, `selfpost`). Metadata only (v2) | +| `inbound_spam_list` | Allow / deny entries for the external filter (`address`, `domain`, `ip`), instance-wide; unique per list (v2) | `TestBaselineSchema` (`internal/store/schema_test.go`) pins every table, column -and index of this list. +and index of this list; `TestOpenMigratesA20Database` opens a database that +holds the baseline alone. ### What changed against the last 1.x schema @@ -129,7 +132,7 @@ for the domain. Nothing refers to a user: deleting one changes no domain. 1. Add `internal/store/migrations/00NN_short_name.sql` (next number). 2. Update the table in [Migration chain](#migration-chain), the snapshot at the - top, and the affected rows of [Schema at head](#schema-at-head-v1). + top, and the affected rows of [Schema at head](#schema-at-head-v2). 3. Update `TestBaselineSchema` if tables, columns or indexes change, and add a test that opens a database at the previous version and migrates it. 4. A *data* migration states in a comment which 2.x versions it is written for. diff --git a/internal/store/migrations/0002_inbound_spam_log.sql b/internal/store/migrations/0002_inbound_spam_log.sql new file mode 100644 index 0000000..4ae0286 --- /dev/null +++ b/internal/store/migrations/0002_inbound_spam_log.sql @@ -0,0 +1,40 @@ +-- Inbound antispam journal and filter lists (inbound-antispam-panel, 2.1.0). +-- The first migration after the 2.0 baseline: DDL only, so a 2.0.x database +-- gains two empty tables and nothing in it changes. + +-- What happened to a message offered on port 25, one row per recipient. +-- Metadata only - SelfPost relays inbound mail and stores none of it. Accepted +-- messages are written by the inbound journal milter, refused ones by the +-- mail.log tailer, which is why a row can lack a subject (refused before DATA) +-- and why engine says who decided: the external filter, Postfix's own policy, +-- or nobody (selfpost: relayed with no filter attached). +CREATE TABLE inbound_spam_log ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + inbound_domain TEXT NOT NULL, + client_ip TEXT NOT NULL, + from_addr TEXT NOT NULL, + to_addr TEXT NOT NULL, + subject TEXT NOT NULL, + decision TEXT NOT NULL CHECK (decision IN ('accept', 'reject', 'tempfail', 'quarantine')), + engine TEXT NOT NULL CHECK (engine IN ('rspamd', 'postfix', 'selfpost')), + detail TEXT NOT NULL, + created_at TEXT NOT NULL +); +-- Retention prunes by age; the Log page lists newest first, for the whole +-- server or for one domain. +CREATE INDEX idx_inbound_spam_log_created_at ON inbound_spam_log (created_at); +CREATE INDEX idx_inbound_spam_log_domain_created_at ON inbound_spam_log (inbound_domain, created_at); + +-- Allow / deny entries for the external filter, instance-wide. The panel +-- writes them out as map files under /data/antispam/; SelfPost itself does not +-- consult them. value is stored normalised, so the same entry cannot be added +-- twice to one list. +CREATE TABLE inbound_spam_list ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + list_type TEXT NOT NULL CHECK (list_type IN ('allow', 'deny')), + entry_type TEXT NOT NULL CHECK (entry_type IN ('address', 'domain', 'ip')), + value TEXT NOT NULL, + note TEXT NOT NULL DEFAULT '', + created_at TEXT NOT NULL, + UNIQUE (list_type, entry_type, value) +); diff --git a/internal/store/schema_test.go b/internal/store/schema_test.go index e602462..d05b7cb 100644 --- a/internal/store/schema_test.go +++ b/internal/store/schema_test.go @@ -55,6 +55,8 @@ func TestBaselineSchema(t *testing.T) { "user_inbound_domains": "user_id inbound_domain_id", "dmarc_reports": "id domain reporter report_id period_begin period_end received_at contact_email policy_p policy_sp policy_pct policy_adkim policy_aspf pass_count fail_count recipient", "dmarc_report_records": "id report_row_id source_ip count disposition spf_result dkim_result header_from", + "inbound_spam_log": "id inbound_domain client_ip from_addr to_addr subject decision engine detail created_at", + "inbound_spam_list": "id list_type entry_type value note created_at", } rows, err := st.db.Query("SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%' ORDER BY name") if err != nil { @@ -99,6 +101,7 @@ func TestBaselineSchema(t *testing.T) { rows.Close() wantIdx := []string{ "idx_dmarc_report_records_report", "idx_dmarc_reports_domain", "idx_dmarc_reports_received", + "idx_inbound_spam_log_created_at", "idx_inbound_spam_log_domain_created_at", "idx_send_log_app_login_created_at", "idx_send_log_created_at", "idx_send_log_domain", "idx_send_log_queue_id", "idx_sessions_expires_at", } @@ -108,8 +111,8 @@ func TestBaselineSchema(t *testing.T) { } var version, appID int64 - if err := st.db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 1 { - t.Errorf("user_version = %d, %v; want 1", version, err) + if err := st.db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 2 { + t.Errorf("user_version = %d, %v; want 2", version, err) } if err := st.db.QueryRow("PRAGMA application_id").Scan(&appID); err != nil || appID != applicationID { t.Errorf("application_id = %#x, %v; want %#x", appID, err, applicationID) @@ -156,6 +159,96 @@ func TestReopenKeepsTheDatabase(t *testing.T) { } } +// A 2.0.x data directory - the baseline and nothing after it - is brought to +// the head by opening it, with what it holds left alone. +func TestOpenMigratesA20Database(t *testing.T) { + path := filepath.Join(t.TempDir(), "test.db") + baseline, err := migrationsFS.ReadFile("migrations/0001_init.sql") + if err != nil { + t.Fatalf("read the baseline: %v", err) + } + db, err := sql.Open("sqlite", "file:"+path) + if err != nil { + t.Fatalf("create a 2.0 db: %v", err) + } + for _, stmt := range []string{ + string(baseline), + "PRAGMA user_version = 1", + "INSERT INTO users (username, password_hash, role, created_at) VALUES ('admin', 'hash', 'global', '2026-10-11T00:00:00Z')", + } { + if _, err := db.Exec(stmt); err != nil { + t.Fatalf("prepare a 2.0 db: %v", err) + } + } + db.Close() + + st, err := Open(path) + if err != nil { + t.Fatalf("open a 2.0 database: %v", err) + } + defer st.Close() + var version int + if err := st.db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 2 { + t.Errorf("user_version = %d, %v; want 2", version, err) + } + if ok, err := st.UserExists(); err != nil || !ok { + t.Errorf("the user of the 2.0 database: exists = %v, %v", ok, err) + } + for _, table := range []string{"inbound_spam_log", "inbound_spam_list"} { + if len(tableColumns(t, st, table)) == 0 { + t.Errorf("table %s was not created", table) + } + } +} + +// What the antispam tables refuse: a decision, engine or list the panel does +// not know, and the same entry twice in one list. +func TestInboundSpamConstraints(t *testing.T) { + st := openTestStore(t) + logRow := func(decision, engine string) error { + _, err := st.db.Exec( + "INSERT INTO inbound_spam_log (inbound_domain, client_ip, from_addr, to_addr, subject, decision, engine, detail, created_at)"+ + " VALUES ('in.example.com', '192.0.2.1', 'a@example.net', 'b@in.example.com', '', ?, ?, '', '2026-10-11T00:00:00Z')", + decision, engine) + return err + } + if err := logRow("reject", "postfix"); err != nil { + t.Fatalf("a reject by postfix: %v", err) + } + if err := logRow("bounce", "postfix"); err == nil { + t.Error("decision bounce was accepted") + } + if err := logRow("accept", "spamassassin"); err == nil { + t.Error("engine spamassassin was accepted") + } + + entry := func(list, kind, value string) error { + _, err := st.db.Exec( + "INSERT INTO inbound_spam_list (list_type, entry_type, value, created_at) VALUES (?, ?, ?, '2026-10-11T00:00:00Z')", + list, kind, value) + return err + } + if err := entry("allow", "address", "billing@vendor.example"); err != nil { + t.Fatalf("an allow entry: %v", err) + } + if err := entry("allow", "address", "billing@vendor.example"); err == nil { + t.Error("the same entry was added to one list twice") + } + if err := entry("deny", "address", "billing@vendor.example"); err != nil { + t.Errorf("the same value on the other list: %v", err) + } + if err := entry("block", "address", "x@example.net"); err == nil { + t.Error("list type block was accepted") + } + if err := entry("deny", "subject", "pills"); err == nil { + t.Error("entry type subject was accepted") + } + var note string + if err := st.db.QueryRow("SELECT note FROM inbound_spam_list WHERE list_type = 'allow'").Scan(¬e); err != nil || note != "" { + t.Errorf("note of an entry added without one = %q, %v", note, err) + } +} + // A 1.x data directory must be refused, loudly. Its user_version (10 at the // last 1.x release) would otherwise read as "the baseline is already applied", // and the panel would start on tables it does not know.