audit 3: backup CLI, envelope budget, directory sync, docs; findings of the reviews
Steps 3-6 of stage 3 and what the two reviews found in steps 1-6, in one commit: the review fixes sit in the same hunks. Step 3, CLI backup (M3, M5, L1, L2; L4 of the stage's list): - backup.snapshotDB refuses a database that is not there and opens the source mode=ro (measured: mode=ro creates no file). - selfpost-backup holds an encrypting password to validate.MinSecretFilePasswordLen; -o of both directions writes a temporary file beside the name and renames it whole; -decrypt peeks the first chunk (password and type authenticated) before the output exists. - what a backup leaves out (a link under data/, a link under certs/ to a directory, a socket...) is told to OnWarn; certs/ with no file is backup.ErrNoCerts, refused by Check and Create, named by the panel. - parseTrustedProxies: a bare address takes /128 when written with a colon (::ffff:10.0.0.1/32 was ::/32). Step 4, secretfile (M4, L3): scrypt's V at most 128 MiB, r <= 64, p <= 2; bytes after the last chunk are ErrCorrupt. Step 5 (L7): atomicfile.SyncDir after the rename, used by Write, replaceSASLDB and the CLI; atomicfile_test.go. Step 6: security.md and guide on backup and restore. Reviews (technical, security): - H1: a restore of a server with DMARC on was refused on every start - the archive records no owner, tar as root left the DMARC maps root's, and the entrypoint only chowned the maps it named. It now hands every file beside queue/ in /data/postfix to panel:selfpost 0640; e2e writes an extra map before the backup and finds it panel:selfpost 640 after a root unpack. - the comparison reads regular files only (Lstat, size, LimitReader); a record unpacked again wins over the earlier check's mark; ConsumeRestore removes both; the mark is synced; a journal file beside a freshly unpacked database is refused. - AttachRefusalLine refuses an empty line as the old statement did; skipped entries named in words; -decrypt of an empty payload is no failure; guide wording on the shell's '>' and a killed CLI; the in-place restore's 'mv .' (older than this stage) fixed. Tested on the test server: go vet + go test of 25 packages, make e2e (all pass, 643 s) on this tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
@@ -1114,17 +1114,46 @@ those after it move up one patch number each.
|
||||
top-N). The statements are constants/builders the test explains
|
||||
(`TestSendLogQueriesUseTheirIndexes`, which replaces the hand-written
|
||||
plan check of `TestLimitRefusalsSince`).
|
||||
- [ ] **CLI backup**: open the source `mode=ro`, refuse a missing file; password
|
||||
- [x] **CLI backup**: open the source `mode=ro`, refuse a missing file; password
|
||||
floor of 12 shared with the panel (`validate.MinSecretFilePasswordLen`);
|
||||
`-decrypt -o` verifies the password (first chunk) before opening the
|
||||
output; `OnWarn` for skipped symlinks and `Check` fails when `certs/` is
|
||||
named but empty. — **Opus**
|
||||
- [ ] **`secretfile` caps**: bound `128·N·r` to a stated budget (e.g. 128 MiB)
|
||||
*Done 2026-10-15.* The source check is in `backup.snapshotDB`, so the
|
||||
panel has it too; `mode=ro` measured not to create a missing file
|
||||
(`TestSnapshotRefusesAMissingDatabase`). `-o` of both directions writes
|
||||
a temporary file beside the name and renames it whole (a backup that
|
||||
fails half way left a partial file too); `-decrypt` peeks the first
|
||||
chunk - the password, and the envelope's type, authenticated - before
|
||||
it opens the output. Since 2.5.2 links under `certs/` are followed, so
|
||||
what is still left out and now warned of: a link under `data/`, a link
|
||||
under `certs/` to a directory, a socket or the like; `certs/` with no
|
||||
file refused by Check and Create alike. L4, in this stage's list without
|
||||
a line: a bare address takes /128 when written with a colon, /32
|
||||
otherwise (`::ffff:10.0.0.1/32` was `::/32`).
|
||||
- [x] **`secretfile` caps**: bound `128·N·r` to a stated budget (e.g. 128 MiB)
|
||||
and `p` to 1-2; refuse the rest with `ErrHeader`; reject trailing bytes
|
||||
after the last chunk. Tests. — **Opus**
|
||||
- [ ] **`atomicfile`**: fsync the directory after rename; a test file for the
|
||||
*Done 2026-10-15:* 128 MiB of scrypt's `V` (x/crypto v0.58.0 `Key`:
|
||||
`make([]uint32, 32*N*r)`), `p` ≤ 2, `logN` ≤ 30 for the arithmetic; the
|
||||
refusal is the existing `ErrCorrupt` (there is no `ErrHeader`), which the
|
||||
import shows as "damaged or incomplete". One byte is read after the last
|
||||
chunk; any is `ErrCorrupt`. `TestKDFBudget` (the old cap's 4 GiB among
|
||||
the refused), `TestBytesAfterTheLastChunkRejected`.
|
||||
- [x] **`atomicfile`**: fsync the directory after rename; a test file for the
|
||||
package. — **Opus**
|
||||
- [ ] Docs: security.md § Backup; guide § Backup. — **Sonnet**
|
||||
*Done 2026-10-15:* `atomicfile.SyncDir`, best effort as sasl.go's (the
|
||||
file is in place either way; Go on Windows cannot sync a directory), used
|
||||
by `Write`, `app.replaceSASLDB` (its inline copy replaced) and the CLI's
|
||||
`-o`. `atomicfile_test.go`: create, replace, mode, nothing left beside;
|
||||
an open reader keeps the old file; a failed write (missing directory,
|
||||
a directory in the way) leaves the old state; SyncDir works on Linux.
|
||||
- [x] Docs: security.md § Backup; guide § Backup. — **Sonnet**
|
||||
*Done 2026-10-15:* security.md § Backup and domain export (what is warned
|
||||
of, the missing database, the restore's record - integrity, not
|
||||
authenticity -, the KDF budget, trailing bytes, the CLI's floor and
|
||||
`-o`); guide § Restore (step 1), § Full backup (certs/, `-o`, the
|
||||
database) and § Encrypting (the first block, a cut file, the floor).
|
||||
|
||||
### Stage 4 — panel behaviour (M8, M11, L10–L19) → `2.5.4`
|
||||
|
||||
|
||||
+35
-8
@@ -724,7 +724,10 @@ into `/etc/letsencrypt`, the commands of "Reverse proxy" above give certbot's
|
||||
own files the group). A key readable by root alone is enough for Postfix, but
|
||||
then the panel refuses to make a backup and names the file: it does not hand
|
||||
over an archive without the key. Links under `certs/` are followed: the
|
||||
archive holds the files they lead to.
|
||||
archive holds the files they lead to. A `certs/` that is there with no file
|
||||
in it is refused for the same reason; what a backup leaves out - a link to
|
||||
a directory under `certs/`, a link under `data/` - it names in a warning (the
|
||||
CLI's standard error, the panel's log).
|
||||
|
||||
**2. `selfpost-backup` CLI — while the container is running (no downtime).**
|
||||
The panel button and this command call the same code path: a consistent SQLite
|
||||
@@ -737,9 +740,12 @@ docker exec <container> selfpost-backup > selfpost-backup.tar.gz
|
||||
```
|
||||
|
||||
Use `-o /path/inside/container` to write inside the container instead of
|
||||
stdout. Optional encryption: set `SELFPOST_BACKUP_PASSWORD` or pass
|
||||
`-password-file` — see [Encrypting a backup or
|
||||
export](#encrypting-a-backup-or-export).
|
||||
stdout; the file is written beside that name and takes it only when it is
|
||||
whole, so a backup that fails leaves no half-written file and whatever had
|
||||
the name before. A database that is not where `SELFPOST_DB_PATH` points is
|
||||
refused, rather than archived empty. Optional encryption: set
|
||||
`SELFPOST_BACKUP_PASSWORD` or pass `-password-file` — see [Encrypting a
|
||||
backup or export](#encrypting-a-backup-or-export).
|
||||
|
||||
**3. `tar` of the whole project directory — with the service stopped.** Stop
|
||||
the instance so nothing is writing to SQLite, then archive the operator
|
||||
@@ -832,7 +838,7 @@ back after a bad change):
|
||||
docker compose down
|
||||
|
||||
# 2. Move the current project aside rather than deleting it
|
||||
mv . ../selfpost.before-restore
|
||||
cd .. && mv selfpost selfpost.before-restore
|
||||
mkdir selfpost && cd selfpost
|
||||
|
||||
# 3. Unpack the backup into the fresh directory
|
||||
@@ -871,7 +877,15 @@ The last entry of every panel and CLI archive, `data/contents.json`, records
|
||||
each file under `data/` with its size and SHA-256, and the first start
|
||||
compares `/data` with it; a file that differs is named in the message.
|
||||
`docker-compose.yml`, `.env` and `certs/` are not compared — they are yours to
|
||||
edit before that start.
|
||||
edit before that start. A journal file beside the database
|
||||
(`selfpost.db-wal`, `-shm`, `-journal`) is refused too: no archive holds one,
|
||||
so it is left from what the directory held before, and SQLite would take it
|
||||
up. To unpack again, unpack into an **empty** directory.
|
||||
|
||||
The files are compared once: a first start that gets past the comparison
|
||||
and then fails (its Resync, say) leaves the manifest, and the next
|
||||
`docker compose up -d` runs the Resync without comparing again - that start
|
||||
had the database open, so its files are no longer the archive's.
|
||||
|
||||
**Moving to a different host** is the same flow: create an empty project
|
||||
directory, unpack the backup there, edit `.env` (and `docker-compose.yml` if
|
||||
@@ -957,9 +971,22 @@ unpacked into `/data`, which the CLI does with the same password:
|
||||
docker exec -i <container> selfpost-backup -decrypt < backup.spbk > backup.tar.gz
|
||||
```
|
||||
|
||||
The password is checked on the file's first block, before the command
|
||||
writes anything - but the shell's `>` has already emptied `backup.tar.gz`
|
||||
before the command starts, so a wrong password leaves it empty; `-o` keeps
|
||||
a file that was there. A file cut short fails at its end, with the command's
|
||||
exit status non-zero, after what came before has been written: an archive
|
||||
cut so is refused at the restore when it lost any file (see
|
||||
[Restore](#restore) - the record is the archive's last entry). With `-o` the
|
||||
output takes its name only when it is whole; a command killed outright
|
||||
leaves a `.NAME.tmp-*` file beside it, readable by its owner alone, to
|
||||
remove by hand.
|
||||
|
||||
The CLI also *writes* encrypted backups for scripted/cron use. The password
|
||||
comes from `SELFPOST_BACKUP_PASSWORD` or `-password-file <path>` (first line),
|
||||
never from a command-line argument, which would be visible in the process list.
|
||||
must be at least 12 characters, as on the panel's forms (decrypting takes
|
||||
any), and it comes from `SELFPOST_BACKUP_PASSWORD` or `-password-file <path>`
|
||||
(first line), never from a command-line argument, which would be visible in
|
||||
the process list.
|
||||
`-e` with the name only takes the value from your shell's environment, so it is
|
||||
not written on the command line:
|
||||
|
||||
|
||||
+1
-1
@@ -42,7 +42,7 @@ in `git log` and [CHANGELOG.md](../CHANGELOG.md).
|
||||
| from-header-binding | Hold a message's `From:` header against the application that sends it | shipped `2.4.0` | 9/9 | [plans/from-header-binding.md](plans/from-header-binding.md) |
|
||||
| panel-redesign | Panel redesign (accepted mockups, design contract) | shipped `2.0.0` | 27/27 | [plans/panel-redesign.md](plans/panel-redesign.md) |
|
||||
| schema-squash | Squash SQLite migrations into a 2.x baseline | built in panel-redesign (stage 1) | — | [plans/panel-redesign.md](plans/panel-redesign.md) |
|
||||
| audit-2026-10-12 | Follow-ups of the 2026-10-12 full-tree code audit, seven stages | in progress (stage 0 shipped `2.4.2`, stages 1+2 `2.5.0`, 2b `2.5.1`, 2c `2.5.2`; next stage 3 → `2.5.3`) | 51/81 | [audit-2026-10-12.md](audit-2026-10-12.md) |
|
||||
| audit-2026-10-12 | Follow-ups of the 2026-10-12 full-tree code audit, seven stages | in progress (stage 0 shipped `2.4.2`, stages 1+2 `2.5.0`, 2b `2.5.1`, 2c `2.5.2`; next stage 3 → `2.5.3`) | 55/81 | [audit-2026-10-12.md](audit-2026-10-12.md) |
|
||||
|
||||
**Recommended order** (not binding; owner, 2026-10-09: feature order follows
|
||||
development convenience): **panel-redesign** shipped as `2.0.0` (2026-10-11) and
|
||||
|
||||
@@ -85,7 +85,7 @@ audit (`2.5.0`), `0009_refusal_line.sql` (`2.5.1`) and
|
||||
| 7 | `0007_delivery_log_days.sql` | 2.5.0 | DDL | `delivery_log_days`: what the reconcile sweep and the log tailer did, a row per UTC day, kept a year, shown nowhere - the measurement that decides the sweep's fate (audit 2026-10-12, stage 1). `send_log.reason` gains the values `reconcile` and `reconcile-contradicted` without a schema change |
|
||||
| 8 | `0008_tenant_boundary.sql` | 2.5.0 | DDL + data | `idx_applications_login_nocase`: an application login is unique without regard to case (audit 2026-10-12, H4). `send_log.domain_id` and `inbound_spam_log.inbound_domain_id` (`ON DELETE SET NULL`), each with an index, filled once by name from the registry as it was then: a row belongs to a domain by id, and a domain deleted and added again starts with an empty log (L12) |
|
||||
| 9 | `0009_refusal_line.sql` | 2.5.1 | DDL | `send_log.refusal_line`: Postfix's line about a refusal at MAIL FROM (a level-2 limit, a client address off the list), which has no queue id; the log tailer puts it on the row; `send_log.recovered`, a refusal row the tailer wrote from `mail.log` because the panel stopped before writing it; `sender_check_days.messages` (messages deferred mid-message, counted apart from the connections turned away) and `sender_check_days.estimated` (a day whose downtime is an upper bound) (audit 2026-10-12, stage 2b) |
|
||||
| 10 | `0010_send_log_indexes.sql` | 2.5.3 | DDL | Four indexes of `send_log`, each for a question that walked the table or sorted a domain's log (audit 2026-10-12, M2, stage 3): `idx_send_log_open`, partial, the queued and deferred rows by time (the tailer's sweep every 5 min); `idx_send_log_domain_id` and `idx_send_log_app_login`, a page of one domain's or one application's log in id order; `idx_send_log_refusal_line`, partial, whether a refusal line is already on a row. `TestSendLogQueriesUseTheirIndexes` checks the plans of the statements the store runs |
|
||||
| 10 | `0010_send_log_indexes.sql` | 2.5.3 | DDL | Four indexes of `send_log`, each for a question that walked the table or sorted a domain's log (audit 2026-10-12, M2, stage 3): `idx_send_log_open`, partial, the queued and deferred rows by time (the tailer's sweep every 5 min); `idx_send_log_domain_id` and `idx_send_log_app_login`, a page of one domain's or one application's log in id order; `idx_send_log_refusal_line`, partial, whether a refusal line is already on a row. `TestSendLogQueriesUseTheirIndexes` checks the plans of these and of the other by-time and by-scope statements the store runs |
|
||||
|
||||
**Kind:** *DDL* — schema only; *data* — `INSERT`/`UPDATE` that must stay correct
|
||||
for operators upgrading from older 2.x images.
|
||||
|
||||
+30
-2
@@ -308,6 +308,23 @@ operator can act on it.
|
||||
that the panel's user may read. `certs/` is the operator's, mounted
|
||||
read-only; only someone who can write the project directory places a link
|
||||
there, and only a global user can download the archive (`2.5.2`).
|
||||
What a backup still leaves out - a link under `data/`, a link under
|
||||
`certs/` to a directory - it names in a warning, and a `certs/` with no
|
||||
file in it is refused rather than archived empty (`2.5.3`).
|
||||
- A backup is made only of the database that is there: a wrong
|
||||
`SELFPOST_DB_PATH` is refused, and the snapshot opens the source
|
||||
read-only. An archive of an empty database would restore a server that
|
||||
offers its first-run setup - a new global administrator - while it holds
|
||||
live DKIM keys and `sasldb2` (`2.5.3`).
|
||||
- The first start after a restore opens nothing until `/data` matches the
|
||||
archive's last entry, `data/contents.json` (size and SHA-256 of each file
|
||||
under `data/`): an archive unpacked in part - its manifest there, half its
|
||||
database - is refused (`2.5.3`). The check is integrity, not authenticity:
|
||||
whoever can write the project directory can write that record too. It
|
||||
reads only regular files at the recorded names (a link or a pipe there is
|
||||
refused, not followed), refuses a database journal file beside the
|
||||
restored database, and is made once per restore: a start after one that
|
||||
passed it and failed is not compared again while the manifest is the same.
|
||||
- Since 2.2.0 a full backup also carries, in the database, what Postfix logged
|
||||
about each message still in the send log (`send_log_lines`): the submitting
|
||||
client's address and SASL login, the message id, the receiving server and
|
||||
@@ -318,7 +335,12 @@ operator can act on it.
|
||||
scrypt (N=2¹⁵, r=8, p=1) → AES-256-GCM, streamed in 64 KiB chunks, each
|
||||
authenticated with the header, the chunk number, and an end-of-stream flag —
|
||||
a truncated or substituted file fails to open instead of silently restoring a
|
||||
partial "tail". Format and wrapper:
|
||||
partial "tail", and bytes after the last chunk are refused (`2.5.3`). The
|
||||
key-derivation parameters travel in the header, so a reader is bounded in
|
||||
what a header may ask before the password is checked: scrypt's array at
|
||||
most 128 MiB (four times what SelfPost writes), r ≤ 64 so the rest stays
|
||||
under 32 KiB, p ≤ 2 - *Import a domain* reads an uploaded file, and the
|
||||
old cap let a header ask for 4 GiB (`2.5.3`). Format and wrapper:
|
||||
[internal/secretfile](../internal/secretfile/secretfile.go).
|
||||
- Extensions: `.spbk` (**S**elf**P**ost **b**ac**k**up — full backup), `.spde`
|
||||
(**S**elf**P**ost **d**omain **e**xport — domain export); unencrypted files
|
||||
@@ -329,7 +351,13 @@ operator can act on it.
|
||||
`-password-file`, never as an argument (the process list is readable by any
|
||||
process in the container).
|
||||
- Minimum password length matches the administrator password (12): the file
|
||||
sits offline and can be attacked without a time limit.
|
||||
sits offline and can be attacked without a time limit. The panel's forms and,
|
||||
since `2.5.3`, the CLI hold an encrypting password to it; decrypting takes
|
||||
any.
|
||||
- The CLI's `-o FILE` is written beside its name and takes it only when
|
||||
whole, so a failure - a wrong password to `-decrypt` among them, which is
|
||||
checked on the first chunk before the output is touched - leaves the file
|
||||
that was there (`2.5.3`).
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user