audit 3: every per-message question of the send log has its index
test / test (push) Canceled after 0s
test / test (push) Canceled after 0s
Migration 0010 (DDL only; 0009 went to stage 2b). EXPLAIN QUERY PLAN of
every send_log statement showed, beyond M2 as written (stage 2 had already
moved the by-domain counts onto (domain_id, created_at)):
- the tailer's sweep of open rows (every 5 min) and its count scanned the
table: partial idx_send_log_open (created_at) WHERE status IN ('queued',
'deferred'), the statuses literal in the statements, ORDER BY +id (else
SQLite walks the table in id order to skip a sort);
- AttachRefusalLine's NOT EXISTS (... refusal_line = ?1) scanned the table
for every refusal line: partial idx_send_log_refusal_line, the query
carrying the index's WHERE; its row found by idx_send_log_rejected
(status literal, +queue_id) instead of every refusal without a queue id;
- an application's level-2 count went through its domain's index:
+domain_id in appByLogin;
- a page of one domain's or one application's log sorted all of it:
idx_send_log_domain_id, idx_send_log_app_login (id order; several domains
still sort, top-N). idx_send_log_domain stays for the global filter.
The statements are constants and builders (countMessagesQuery,
peakPerHourQuery, listOpenQuery, countOpenQuery, sendLogPageQuery,
attachRefusalLineQuery, limitRefusalsQuery) and
TestSendLogQueriesUseTheirIndexes explains exactly those: the index each
must use, no SCAN of send_log, no sort where none is wanted; the literal
statuses checked against the constants. It replaces the hand-written plan
check of TestLimitRefusalsSince.
Tested on the test server: go vet + go test of 24 packages, make e2e (all
pass) on this tree.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -1092,12 +1092,28 @@ those after it move up one patch number each.
|
||||
untouched, no `-wal`), a damaged key named; the domain import of a cut
|
||||
plain export, a cut and an altered `.spde` (each refused, nothing
|
||||
created). The CLI's `-decrypt -o` is step 3.
|
||||
- [ ] **Indexes**: `idx_send_log_domain_created_at (domain_id, created_at)` —
|
||||
- [x] **Indexes**: `idx_send_log_domain_created_at (domain_id, created_at)` —
|
||||
on the id column stage 2 adds, replacing `idx_send_log_domain`;
|
||||
partial `idx_send_log_queued (created_at) WHERE status = 'queued'`;
|
||||
consider `(domain, id)` for the scoped page. Migration `0009` DDL only;
|
||||
`TestLimitRefusalsSince`-style EXPLAIN tests against the **real** query
|
||||
strings. — **Opus**
|
||||
*Done 2026-10-15,* migration `0010` (0009 went to stage 2b). Stage 2
|
||||
already counted by `(domain_id, created_at)`; `idx_send_log_domain` stays,
|
||||
the global log's filter by name reads its page off it in id order. The
|
||||
plans of every `send_log` statement showed four more: the sweep now asks
|
||||
for queued *and* deferred rows - `idx_send_log_open (created_at) WHERE
|
||||
status IN ('queued', 'deferred')`, the statuses literal and `ORDER BY +id`
|
||||
(else SQLite walks the table in id order); `NOT EXISTS (... refusal_line
|
||||
= ?1)` of stage 2b scanned the table per refusal line - partial
|
||||
`idx_send_log_refusal_line`, and its row is found by
|
||||
`idx_send_log_rejected` (status literal, `+queue_id`); the application's
|
||||
level-2 count went by the domain's index - `+domain_id`; a page of one
|
||||
domain or one application - `idx_send_log_domain_id`,
|
||||
`idx_send_log_app_login` (id order, no sort; several domains still sort,
|
||||
top-N). The statements are constants/builders the test explains
|
||||
(`TestSendLogQueriesUseTheirIndexes`, which replaces the hand-written
|
||||
plan check of `TestLimitRefusalsSince`).
|
||||
- [ ] **CLI backup**: open the source `mode=ro`, refuse a missing file; password
|
||||
floor of 12 shared with the panel (`validate.MinSecretFilePasswordLen`);
|
||||
`-decrypt -o` verifies the password (first chunk) before opening the
|
||||
|
||||
+1
-1
@@ -42,7 +42,7 @@ in `git log` and [CHANGELOG.md](../CHANGELOG.md).
|
||||
| from-header-binding | Hold a message's `From:` header against the application that sends it | shipped `2.4.0` | 9/9 | [plans/from-header-binding.md](plans/from-header-binding.md) |
|
||||
| panel-redesign | Panel redesign (accepted mockups, design contract) | shipped `2.0.0` | 27/27 | [plans/panel-redesign.md](plans/panel-redesign.md) |
|
||||
| schema-squash | Squash SQLite migrations into a 2.x baseline | built in panel-redesign (stage 1) | — | [plans/panel-redesign.md](plans/panel-redesign.md) |
|
||||
| audit-2026-10-12 | Follow-ups of the 2026-10-12 full-tree code audit, seven stages | in progress (stage 0 shipped `2.4.2`, stages 1+2 `2.5.0`, 2b `2.5.1`, 2c `2.5.2`; next stage 3 → `2.5.3`) | 50/81 | [audit-2026-10-12.md](audit-2026-10-12.md) |
|
||||
| audit-2026-10-12 | Follow-ups of the 2026-10-12 full-tree code audit, seven stages | in progress (stage 0 shipped `2.4.2`, stages 1+2 `2.5.0`, 2b `2.5.1`, 2c `2.5.2`; next stage 3 → `2.5.3`) | 51/81 | [audit-2026-10-12.md](audit-2026-10-12.md) |
|
||||
|
||||
**Recommended order** (not binding; owner, 2026-10-09: feature order follows
|
||||
development convenience): **panel-redesign** shipped as `2.0.0` (2026-10-11) and
|
||||
|
||||
@@ -17,8 +17,8 @@ When you add or review a migration, update this file in the same change.
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Chain head | `user_version = 9` |
|
||||
| Files | `0001_init.sql` (the 2.0 baseline), `0002_inbound_spam_log.sql`, `0003_send_log_lines.sql`, `0004_notifications.sql`, `0005_send_log_rejected_index.sql`, `0006_sender_check.sql`, `0007_delivery_log_days.sql`, `0008_tenant_boundary.sql`, `0009_refusal_line.sql` |
|
||||
| Chain head | `user_version = 10` |
|
||||
| Files | `0001_init.sql` (the 2.0 baseline), `0002_inbound_spam_log.sql`, `0003_send_log_lines.sql`, `0004_notifications.sql`, `0005_send_log_rejected_index.sql`, `0006_sender_check.sql`, `0007_delivery_log_days.sql`, `0008_tenant_boundary.sql`, `0009_refusal_line.sql`, `0010_send_log_indexes.sql` |
|
||||
| File mark | `PRAGMA application_id = 0x53503230` ("SP20") |
|
||||
| Database file | `/data/selfpost.db` (bind mount) |
|
||||
| Compatibility | 2.x MINOR releases must boot a `2.0.0` data directory. **Nothing is carried over from 1.x.** |
|
||||
@@ -67,7 +67,8 @@ delivery-log-storage (`2.2.0`), `0004_notifications.sql` and
|
||||
`0005_send_log_rejected_index.sql` of panel-notifications (`2.3.0`), and
|
||||
`0006_sender_check.sql` of from-header-binding (`2.4.0`), and
|
||||
`0007_delivery_log_days.sql` and `0008_tenant_boundary.sql` of the 2026-10-12
|
||||
audit (`2.5.0`).
|
||||
audit (`2.5.0`), `0009_refusal_line.sql` (`2.5.1`) and
|
||||
`0010_send_log_indexes.sql` (`2.5.3`).
|
||||
|
||||
---
|
||||
|
||||
@@ -84,6 +85,7 @@ audit (`2.5.0`).
|
||||
| 7 | `0007_delivery_log_days.sql` | 2.5.0 | DDL | `delivery_log_days`: what the reconcile sweep and the log tailer did, a row per UTC day, kept a year, shown nowhere - the measurement that decides the sweep's fate (audit 2026-10-12, stage 1). `send_log.reason` gains the values `reconcile` and `reconcile-contradicted` without a schema change |
|
||||
| 8 | `0008_tenant_boundary.sql` | 2.5.0 | DDL + data | `idx_applications_login_nocase`: an application login is unique without regard to case (audit 2026-10-12, H4). `send_log.domain_id` and `inbound_spam_log.inbound_domain_id` (`ON DELETE SET NULL`), each with an index, filled once by name from the registry as it was then: a row belongs to a domain by id, and a domain deleted and added again starts with an empty log (L12) |
|
||||
| 9 | `0009_refusal_line.sql` | 2.5.1 | DDL | `send_log.refusal_line`: Postfix's line about a refusal at MAIL FROM (a level-2 limit, a client address off the list), which has no queue id; the log tailer puts it on the row; `send_log.recovered`, a refusal row the tailer wrote from `mail.log` because the panel stopped before writing it; `sender_check_days.messages` (messages deferred mid-message, counted apart from the connections turned away) and `sender_check_days.estimated` (a day whose downtime is an upper bound) (audit 2026-10-12, stage 2b) |
|
||||
| 10 | `0010_send_log_indexes.sql` | 2.5.3 | DDL | Four indexes of `send_log`, each for a question that walked the table or sorted a domain's log (audit 2026-10-12, M2, stage 3): `idx_send_log_open`, partial, the queued and deferred rows by time (the tailer's sweep every 5 min); `idx_send_log_domain_id` and `idx_send_log_app_login`, a page of one domain's or one application's log in id order; `idx_send_log_refusal_line`, partial, whether a refusal line is already on a row. `TestSendLogQueriesUseTheirIndexes` checks the plans of the statements the store runs |
|
||||
|
||||
**Kind:** *DDL* — schema only; *data* — `INSERT`/`UPDATE` that must stay correct
|
||||
for operators upgrading from older 2.x images.
|
||||
|
||||
Reference in New Issue
Block a user