audit 3: backup CLI, envelope budget, directory sync, docs; findings of the reviews

Steps 3-6 of stage 3 and what the two reviews found in steps 1-6, in one
commit: the review fixes sit in the same hunks.

Step 3, CLI backup (M3, M5, L1, L2; L4 of the stage's list):
- backup.snapshotDB refuses a database that is not there and opens the
  source mode=ro (measured: mode=ro creates no file).
- selfpost-backup holds an encrypting password to
  validate.MinSecretFilePasswordLen; -o of both directions writes a
  temporary file beside the name and renames it whole; -decrypt peeks the
  first chunk (password and type authenticated) before the output exists.
- what a backup leaves out (a link under data/, a link under certs/ to a
  directory, a socket...) is told to OnWarn; certs/ with no file is
  backup.ErrNoCerts, refused by Check and Create, named by the panel.
- parseTrustedProxies: a bare address takes /128 when written with a
  colon (::ffff:10.0.0.1/32 was ::/32).
Step 4, secretfile (M4, L3): scrypt's V at most 128 MiB, r <= 64, p <= 2;
bytes after the last chunk are ErrCorrupt.
Step 5 (L7): atomicfile.SyncDir after the rename, used by Write,
replaceSASLDB and the CLI; atomicfile_test.go.
Step 6: security.md and guide on backup and restore.

Reviews (technical, security):
- H1: a restore of a server with DMARC on was refused on every start - the
  archive records no owner, tar as root left the DMARC maps root's, and the
  entrypoint only chowned the maps it named. It now hands every file beside
  queue/ in /data/postfix to panel:selfpost 0640; e2e writes an extra map
  before the backup and finds it panel:selfpost 640 after a root unpack.
- the comparison reads regular files only (Lstat, size, LimitReader); a
  record unpacked again wins over the earlier check's mark; ConsumeRestore
  removes both; the mark is synced; a journal file beside a freshly unpacked
  database is refused.
- AttachRefusalLine refuses an empty line as the old statement did; skipped
  entries named in words; -decrypt of an empty payload is no failure; guide
  wording on the shell's '>' and a killed CLI; the in-place restore's
  'mv .' (older than this stage) fixed.

Tested on the test server: go vet + go test of 25 packages, make e2e (all
pass, 643 s) on this tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
mix
2026-10-15 10:38:39 +03:00
co-authored by Claude Opus 5.5
parent 508ac96c79
commit 242515c305
24 changed files with 901 additions and 90 deletions
+49 -4
View File
@@ -18,7 +18,46 @@ Migration `0010` (indexes only).
domain's rows; a page of a domain administrator's log sorted the domain's
whole log. Migration `0010` adds four indexes, and a test now holds each
of these statements to its index (audit 2026-10-12, M2).
- **`selfpost-backup` with a wrong `SELFPOST_DB_PATH` made a backup of
nothing.** SQLite created an empty database where the path pointed, and
the archive of it passed as a backup - one that restores a server offering
its first-run setup while it holds live DKIM keys. A database that is not
there is now refused, and the snapshot opens it read-only (audit
2026-10-12, M3).
- **The CLI encrypted a backup with any password.** It now holds the
password to the panel's floor of 12 characters (audit 2026-10-12, M5).
Decrypting takes any password, as before.
- **`selfpost-backup -o FILE` left a half-written file** when it failed,
and `-decrypt -o FILE` with a wrong password emptied a file that may have
been a good archive. The output is written beside its name and takes it
only when whole; `-decrypt` checks the password on the first block before
it touches the output (audit 2026-10-12, L1).
- **A backup left links out without a word, and an empty `certs/` passed.**
An entry a backup does not carry - a link under `data/`, a link under
`certs/` to a directory - is now named in a warning (the CLI's standard
error, the panel's log), and a `certs/` with no file in it is refused: the
archive would restore a server without its certificate (audit
2026-10-12, L2).
- **`TRUSTED_PROXY_CIDR=::ffff:10.0.0.1` trusted the wrong addresses.** A
bare IPv4-mapped address written as IPv6 was given "/32", which made it
`::/32`: every address beginning with 32 zero bits, `::1` among them, and
not the proxy meant. A bare address is now one host however it is written
(audit 2026-10-12, L4).
- **An encrypted file could make the panel allocate 4 GiB before its
password was checked.** The key-derivation parameters an encrypted backup
or export carries were capped so loosely that a crafted header, uploaded
to *Import a domain*, asked scrypt for 4 GiB and sixteen times the work.
A header may now ask for 128 MiB at most - four times what SelfPost
writes, with the rest of scrypt's memory under 32 KiB - and is refused as
damaged otherwise. Bytes after an envelope's
last block, such as a second file appended to it, are refused too (audit
2026-10-12, M4, L3).
- **A power loss could bring back a map the panel had just replaced.** The
files the panel writes for Postfix and OpenDKIM were synced, but not the
directory their rename changed, so the old file could return while the
database said otherwise. The directory is synced after the rename now, as
`sasldb2` has been since 2.5.2, and the CLI's `-o` file too (audit
2026-10-12, L7).
- **A restore whose first start failed never ran its Resync.** The panel
removed the restore manifest as soon as it had checked the version, before
it opened the database and regenerated the mail maps from it; a start that
@@ -31,9 +70,15 @@ Migration `0010` (indexes only).
there. Every archive now ends with `data/contents.json`, the size and
SHA-256 of each file under `data/`, and the first start after a restore
compares `/data` with it before it opens anything: an archive unpacked in
part, or a file that differs, is refused with a message that names it, and
the directory is left as it was (guide § Restore). A damaged or cut domain
export was already refused before anything is created; a test now says so.
part, a file that differs, or a database journal file left beside the
database from before, is refused with a message that names it, and the
directory is left as it was (guide § Restore). The entrypoint now hands
every map under `data/postfix` to the panel, whatever its name - a restore
unpacked as root left the DMARC maps root's, and the comparison could not
read them. A damaged or cut domain export was already refused before
anything is created; a test now says so.
- **The guide's in-place restore moved `.`**, which `mv` refuses; it now
steps out of the project directory first.
## [2.5.2] - 2026-10-15
+9 -4
View File
@@ -114,9 +114,16 @@ mkdir -p /data/postfix/queue
# Empty files keep postfix check happy if the flag is on before any domain exists.
for f in relay_domains transport relay_recipients tls_policy; do
[ -e "/data/postfix/$f" ] || : > "/data/postfix/$f"
chown panel:selfpost "/data/postfix/$f"
chmod 0640 "/data/postfix/$f"
done
# Every map here is the panel's to write and Postfix's to read - the sender
# map, the inbound and the DMARC maps alike - so each file beside queue/ is
# handed to the panel, group selfpost, whatever its name. A restored archive's
# files arrive owned by whoever unpacked them (a backup records no owner, and
# tar run as root makes them root's), and the first start after a restore
# reads every one of them to compare it with the archive's record: one left
# to root alone is a restore refused on every start (review of audit stage 3,
# H1 - the DMARC maps were not among the names listed here).
find /data/postfix -mindepth 1 -maxdepth 1 -type f -exec chown panel:selfpost {} + -exec chmod 0640 {} +
chown panel:selfpost /data/postfix
chmod 2750 /data/postfix
@@ -128,8 +135,6 @@ chmod 2750 /data/postfix
mkdir -p /data/antispam
chown panel:selfpost /data/antispam
chmod 0755 /data/antispam
chown panel:selfpost /data/postfix/sender_login_maps
chmod 0640 /data/postfix/sender_login_maps
# Delivery log (architecture.md § Log tailer). postlogd writes it as user
# `postfix`; the panel reads it for the log-tailer and the System log page. It
+10 -3
View File
@@ -265,10 +265,17 @@ func parseTrustedProxies(raw string) []*net.IPNet {
continue
}
if !strings.Contains(cidr, "/") {
if ip := net.ParseIP(cidr); ip != nil && ip.To4() != nil {
cidr += "/32"
} else {
// The prefix length goes with how the address is written: an
// IPv4-mapped address written as IPv6 (::ffff:10.0.0.1) is a
// 128-bit address, and "/32" on it is ::/32 - every address
// whose first 32 bits are zero (::1 among them) trusted, and the
// host meant not at all (audit 2026-10-12, L4). Go 1.26
// net.IPNet.Contains compares an IPv4 client with the mapped
// /128 as with 10.0.0.1/32 (net/ip.go, networkNumberAndMask).
if strings.Contains(cidr, ":") {
cidr += "/128"
} else {
cidr += "/32"
}
}
_, n, err := net.ParseCIDR(cidr)
+36
View File
@@ -0,0 +1,36 @@
package main
import (
"net"
"testing"
)
// A bare address is one host however it is written; an IPv4-mapped one in
// IPv6 notation used to become ::/32 (audit 2026-10-12, L4).
func TestParseTrustedProxiesTakesABareAddressAsOneHost(t *testing.T) {
for _, c := range []struct {
entry string
trusts []string
refuses []string
}{
{"10.0.0.1", []string{"10.0.0.1"}, []string{"10.0.0.2"}},
{"::ffff:10.0.0.1", []string{"10.0.0.1", "::ffff:10.0.0.1"}, []string{"10.0.0.2", "::1", "0:0:1::1"}},
{"2001:db8::1", []string{"2001:db8::1"}, []string{"2001:db8::2"}},
{"10.0.0.0/8", []string{"10.1.2.3"}, []string{"11.0.0.1"}},
} {
nets := parseTrustedProxies(c.entry)
if len(nets) != 1 {
t.Fatalf("%q parsed as %v", c.entry, nets)
}
for _, ip := range c.trusts {
if !nets[0].Contains(net.ParseIP(ip)) {
t.Errorf("%q (%v) does not trust %s", c.entry, nets[0], ip)
}
}
for _, ip := range c.refuses {
if nets[0].Contains(net.ParseIP(ip)) {
t.Errorf("%q (%v) trusts %s", c.entry, nets[0], ip)
}
}
}
}
+83 -13
View File
@@ -21,16 +21,21 @@
package main
import (
"bufio"
"errors"
"flag"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"unicode/utf8"
"github.com/mixeme/selfpost/internal/atomicfile"
"github.com/mixeme/selfpost/internal/backup"
"github.com/mixeme/selfpost/internal/buildinfo"
"github.com/mixeme/selfpost/internal/secretfile"
"github.com/mixeme/selfpost/internal/web/validate"
)
// passwordEnv names the environment variable holding the encryption password.
@@ -70,13 +75,20 @@ func run(outPath, password string) error {
dataDir := envDefault("SELFPOST_DATA_DIR", "/data")
dbPath := envDefault("SELFPOST_DB_PATH", filepath.Join(dataDir, "selfpost.db"))
w, closeOut, err := openOutput(outPath)
// The floor the panel holds the same files to: an encrypted backup is
// attacked offline, at leisure (security.md; audit 2026-10-12, M5). An
// empty password is no encryption, by design.
if password != "" && utf8.RuneCountInString(password) < validate.MinSecretFilePasswordLen {
return fmt.Errorf("the encryption password must be at least %d characters", validate.MinSecretFilePasswordLen)
}
w, err := openOutput(outPath)
if err != nil {
return err
}
defer closeOut()
defer w.abort()
sink := w
var sink io.Writer = w
var env *secretfile.Writer
if password != "" {
env, err = secretfile.NewWriter(w, secretfile.TypeFullBackup, password)
@@ -102,6 +114,9 @@ func run(outPath, password string) error {
return err
}
}
if err := w.commit(); err != nil {
return err
}
if outPath != "" {
kind := "plain"
if password != "" {
@@ -133,17 +148,28 @@ func runDecrypt(inPath, outPath, password string) error {
if err != nil {
return err
}
// The first chunk is opened before anything is written: a wrong password
// is told before the output is touched, and the envelope's type is
// authenticated only once a chunk opens (secretfile.Reader.Type; audit
// 2026-10-12, L1).
plain := bufio.NewReaderSize(env, 64<<10)
if _, err := plain.Peek(1); err != nil && !errors.Is(err, io.EOF) {
return err
}
if env.Type() != secretfile.TypeFullBackup {
return fmt.Errorf("that file is a %s, not a full backup", env.Type())
}
w, closeOut, err := openOutput(outPath)
w, err := openOutput(outPath)
if err != nil {
return err
}
defer closeOut()
defer w.abort()
if _, err := io.Copy(w, env); err != nil {
if _, err := io.Copy(w, plain); err != nil {
return err
}
if err := w.commit(); err != nil {
return err
}
if outPath != "" {
@@ -152,17 +178,61 @@ func runDecrypt(inPath, outPath, password string) error {
return nil
}
// openOutput returns stdout, or a freshly created owner-only file: both plain
// and encrypted backups are secret enough to keep off other users' eyes.
func openOutput(outPath string) (io.Writer, func(), error) {
// output is where the CLI writes: stdout, or a file that takes its name only
// once it is whole. A backup or a decryption that fails half way - a wrong
// password, an envelope cut short - leaves whatever had the name before, and
// no half-written file under it (audit 2026-10-12, L1 and stage 3). The file
// is the owner's alone (os.CreateTemp makes it 0600): plain and encrypted
// backups are secret enough to keep off other users' eyes.
type output struct {
io.Writer
tmp *os.File // nil for stdout
path string
}
func openOutput(outPath string) (*output, error) {
if outPath == "" {
return os.Stdout, func() {}, nil
return &output{Writer: os.Stdout}, nil
}
f, err := os.OpenFile(outPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o600)
f, err := os.CreateTemp(filepath.Dir(outPath), "."+filepath.Base(outPath)+".tmp-*")
if err != nil {
return nil, nil, err
return nil, err
}
return f, func() { _ = f.Close() }, nil
return &output{Writer: f, tmp: f, path: outPath}, nil
}
// commit puts the whole file under its name.
func (o *output) commit() error {
if o.tmp == nil {
return nil
}
f := o.tmp
o.tmp = nil
if err := f.Sync(); err != nil {
f.Close()
os.Remove(f.Name())
return err
}
if err := f.Close(); err != nil {
os.Remove(f.Name())
return err
}
if err := os.Rename(f.Name(), o.path); err != nil {
os.Remove(f.Name())
return err
}
_ = atomicfile.SyncDir(filepath.Dir(o.path))
return nil
}
// abort removes a file that was not committed; after commit it does nothing.
func (o *output) abort() {
if o.tmp == nil {
return
}
o.tmp.Close()
os.Remove(o.tmp.Name())
o.tmp = nil
}
// readPassword takes the password from the given file (first line) or, when no
+73
View File
@@ -162,3 +162,76 @@ func TestReadPassword(t *testing.T) {
t.Error("-decrypt without a password was accepted")
}
}
// An encrypted backup is held to the panel's password floor (audit
// 2026-10-12, M5); a refused one writes nothing.
func TestEncryptionPasswordFloor(t *testing.T) {
seedProject(t)
out := filepath.Join(t.TempDir(), "backup.spbk")
err := run(out, "eleven-char")
if err == nil || !strings.Contains(err.Error(), "at least 12 characters") {
t.Fatalf("an 11-character password = %v, want refused", err)
}
if _, statErr := os.Stat(out); !os.IsNotExist(statErr) {
t.Errorf("the refused backup wrote %s", out)
}
}
// A wrong SELFPOST_DB_PATH makes no backup and no database (audit
// 2026-10-12, M3).
func TestBackupOfAMissingDatabaseIsRefused(t *testing.T) {
_, dataDir := seedProject(t)
missing := filepath.Join(dataDir, "typo.db")
t.Setenv("SELFPOST_DB_PATH", missing)
out := filepath.Join(t.TempDir(), "backup.tar.gz")
if err := run(out, ""); err == nil {
t.Fatal("a backup of a database that is not there was made")
}
for _, p := range []string{missing, out} {
if _, err := os.Stat(p); !os.IsNotExist(err) {
t.Errorf("the refused backup left %s (stat err = %v)", p, err)
}
}
}
// -decrypt -o writes its file whole or not at all: a wrong password or an
// envelope cut short leaves the file that had the name before, and nothing
// else in its directory (audit 2026-10-12, L1 and stage 3).
func TestDecryptLeavesTheOutputAloneOnFailure(t *testing.T) {
seedProject(t)
dir := t.TempDir()
encrypted := filepath.Join(dir, "backup.spbk")
const password = "a long enough password"
if err := run(encrypted, password); err != nil {
t.Fatalf("create encrypted backup: %v", err)
}
sealed, err := os.ReadFile(encrypted)
if err != nil {
t.Fatal(err)
}
cut := filepath.Join(dir, "cut.spbk")
writeFile(t, cut, string(sealed[:len(sealed)-100]))
outDir := t.TempDir()
out := filepath.Join(outDir, "backup.tar.gz")
const before = "the archive that was here"
for _, c := range []struct{ name, in, password, want string }{
{"wrong password", encrypted, "the wrong password", "wrong password"},
{"cut short", cut, password, "corrupted"},
} {
t.Run(c.name, func(t *testing.T) {
writeFile(t, out, before)
err := runDecrypt(c.in, out, c.password)
if err == nil || !strings.Contains(err.Error(), c.want) {
t.Fatalf("runDecrypt = %v, want %q", err, c.want)
}
if got, _ := os.ReadFile(out); string(got) != before {
t.Errorf("the failed decryption changed %s to %d bytes", out, len(got))
}
entries, _ := os.ReadDir(outDir)
if len(entries) != 1 {
t.Errorf("the failed decryption left files beside its output: %v", entries)
}
})
}
}
+33 -4
View File
@@ -1114,17 +1114,46 @@ those after it move up one patch number each.
top-N). The statements are constants/builders the test explains
(`TestSendLogQueriesUseTheirIndexes`, which replaces the hand-written
plan check of `TestLimitRefusalsSince`).
- [ ] **CLI backup**: open the source `mode=ro`, refuse a missing file; password
- [x] **CLI backup**: open the source `mode=ro`, refuse a missing file; password
floor of 12 shared with the panel (`validate.MinSecretFilePasswordLen`);
`-decrypt -o` verifies the password (first chunk) before opening the
output; `OnWarn` for skipped symlinks and `Check` fails when `certs/` is
named but empty. — **Opus**
- [ ] **`secretfile` caps**: bound `128·N·r` to a stated budget (e.g. 128 MiB)
*Done 2026-10-15.* The source check is in `backup.snapshotDB`, so the
panel has it too; `mode=ro` measured not to create a missing file
(`TestSnapshotRefusesAMissingDatabase`). `-o` of both directions writes
a temporary file beside the name and renames it whole (a backup that
fails half way left a partial file too); `-decrypt` peeks the first
chunk - the password, and the envelope's type, authenticated - before
it opens the output. Since 2.5.2 links under `certs/` are followed, so
what is still left out and now warned of: a link under `data/`, a link
under `certs/` to a directory, a socket or the like; `certs/` with no
file refused by Check and Create alike. L4, in this stage's list without
a line: a bare address takes /128 when written with a colon, /32
otherwise (`::ffff:10.0.0.1/32` was `::/32`).
- [x] **`secretfile` caps**: bound `128·N·r` to a stated budget (e.g. 128 MiB)
and `p` to 1-2; refuse the rest with `ErrHeader`; reject trailing bytes
after the last chunk. Tests. — **Opus**
- [ ] **`atomicfile`**: fsync the directory after rename; a test file for the
*Done 2026-10-15:* 128 MiB of scrypt's `V` (x/crypto v0.58.0 `Key`:
`make([]uint32, 32*N*r)`), `p` ≤ 2, `logN` ≤ 30 for the arithmetic; the
refusal is the existing `ErrCorrupt` (there is no `ErrHeader`), which the
import shows as "damaged or incomplete". One byte is read after the last
chunk; any is `ErrCorrupt`. `TestKDFBudget` (the old cap's 4 GiB among
the refused), `TestBytesAfterTheLastChunkRejected`.
- [x] **`atomicfile`**: fsync the directory after rename; a test file for the
package. — **Opus**
- [ ] Docs: security.md § Backup; guide § Backup. — **Sonnet**
*Done 2026-10-15:* `atomicfile.SyncDir`, best effort as sasl.go's (the
file is in place either way; Go on Windows cannot sync a directory), used
by `Write`, `app.replaceSASLDB` (its inline copy replaced) and the CLI's
`-o`. `atomicfile_test.go`: create, replace, mode, nothing left beside;
an open reader keeps the old file; a failed write (missing directory,
a directory in the way) leaves the old state; SyncDir works on Linux.
- [x] Docs: security.md § Backup; guide § Backup. — **Sonnet**
*Done 2026-10-15:* security.md § Backup and domain export (what is warned
of, the missing database, the restore's record - integrity, not
authenticity -, the KDF budget, trailing bytes, the CLI's floor and
`-o`); guide § Restore (step 1), § Full backup (certs/, `-o`, the
database) and § Encrypting (the first block, a cut file, the floor).
### Stage 4 — panel behaviour (M8, M11, L10–L19) → `2.5.4`
+35 -8
View File
@@ -724,7 +724,10 @@ into `/etc/letsencrypt`, the commands of "Reverse proxy" above give certbot's
own files the group). A key readable by root alone is enough for Postfix, but
then the panel refuses to make a backup and names the file: it does not hand
over an archive without the key. Links under `certs/` are followed: the
archive holds the files they lead to.
archive holds the files they lead to. A `certs/` that is there with no file
in it is refused for the same reason; what a backup leaves out - a link to
a directory under `certs/`, a link under `data/` - it names in a warning (the
CLI's standard error, the panel's log).
**2. `selfpost-backup` CLI — while the container is running (no downtime).**
The panel button and this command call the same code path: a consistent SQLite
@@ -737,9 +740,12 @@ docker exec <container> selfpost-backup > selfpost-backup.tar.gz
```
Use `-o /path/inside/container` to write inside the container instead of
stdout. Optional encryption: set `SELFPOST_BACKUP_PASSWORD` or pass
`-password-file` — see [Encrypting a backup or
export](#encrypting-a-backup-or-export).
stdout; the file is written beside that name and takes it only when it is
whole, so a backup that fails leaves no half-written file and whatever had
the name before. A database that is not where `SELFPOST_DB_PATH` points is
refused, rather than archived empty. Optional encryption: set
`SELFPOST_BACKUP_PASSWORD` or pass `-password-file` — see [Encrypting a
backup or export](#encrypting-a-backup-or-export).
**3. `tar` of the whole project directory — with the service stopped.** Stop
the instance so nothing is writing to SQLite, then archive the operator
@@ -832,7 +838,7 @@ back after a bad change):
docker compose down
# 2. Move the current project aside rather than deleting it
mv . ../selfpost.before-restore
cd .. && mv selfpost selfpost.before-restore
mkdir selfpost && cd selfpost
# 3. Unpack the backup into the fresh directory
@@ -871,7 +877,15 @@ The last entry of every panel and CLI archive, `data/contents.json`, records
each file under `data/` with its size and SHA-256, and the first start
compares `/data` with it; a file that differs is named in the message.
`docker-compose.yml`, `.env` and `certs/` are not compared — they are yours to
edit before that start.
edit before that start. A journal file beside the database
(`selfpost.db-wal`, `-shm`, `-journal`) is refused too: no archive holds one,
so it is left from what the directory held before, and SQLite would take it
up. To unpack again, unpack into an **empty** directory.
The files are compared once: a first start that gets past the comparison
and then fails (its Resync, say) leaves the manifest, and the next
`docker compose up -d` runs the Resync without comparing again - that start
had the database open, so its files are no longer the archive's.
**Moving to a different host** is the same flow: create an empty project
directory, unpack the backup there, edit `.env` (and `docker-compose.yml` if
@@ -957,9 +971,22 @@ unpacked into `/data`, which the CLI does with the same password:
docker exec -i <container> selfpost-backup -decrypt < backup.spbk > backup.tar.gz
```
The password is checked on the file's first block, before the command
writes anything - but the shell's `>` has already emptied `backup.tar.gz`
before the command starts, so a wrong password leaves it empty; `-o` keeps
a file that was there. A file cut short fails at its end, with the command's
exit status non-zero, after what came before has been written: an archive
cut so is refused at the restore when it lost any file (see
[Restore](#restore) - the record is the archive's last entry). With `-o` the
output takes its name only when it is whole; a command killed outright
leaves a `.NAME.tmp-*` file beside it, readable by its owner alone, to
remove by hand.
The CLI also *writes* encrypted backups for scripted/cron use. The password
comes from `SELFPOST_BACKUP_PASSWORD` or `-password-file <path>` (first line),
never from a command-line argument, which would be visible in the process list.
must be at least 12 characters, as on the panel's forms (decrypting takes
any), and it comes from `SELFPOST_BACKUP_PASSWORD` or `-password-file <path>`
(first line), never from a command-line argument, which would be visible in
the process list.
`-e` with the name only takes the value from your shell's environment, so it is
not written on the command line:
+1 -1
View File
@@ -42,7 +42,7 @@ in `git log` and [CHANGELOG.md](../CHANGELOG.md).
| from-header-binding | Hold a message's `From:` header against the application that sends it | shipped `2.4.0` | 9/9 | [plans/from-header-binding.md](plans/from-header-binding.md) |
| panel-redesign | Panel redesign (accepted mockups, design contract) | shipped `2.0.0` | 27/27 | [plans/panel-redesign.md](plans/panel-redesign.md) |
| schema-squash | Squash SQLite migrations into a 2.x baseline | built in panel-redesign (stage 1) | — | [plans/panel-redesign.md](plans/panel-redesign.md) |
| audit-2026-10-12 | Follow-ups of the 2026-10-12 full-tree code audit, seven stages | in progress (stage 0 shipped `2.4.2`, stages 1+2 `2.5.0`, 2b `2.5.1`, 2c `2.5.2`; next stage 3 → `2.5.3`) | 51/81 | [audit-2026-10-12.md](audit-2026-10-12.md) |
| audit-2026-10-12 | Follow-ups of the 2026-10-12 full-tree code audit, seven stages | in progress (stage 0 shipped `2.4.2`, stages 1+2 `2.5.0`, 2b `2.5.1`, 2c `2.5.2`; next stage 3 → `2.5.3`) | 55/81 | [audit-2026-10-12.md](audit-2026-10-12.md) |
**Recommended order** (not binding; owner, 2026-10-09: feature order follows
development convenience): **panel-redesign** shipped as `2.0.0` (2026-10-11) and
+1 -1
View File
@@ -85,7 +85,7 @@ audit (`2.5.0`), `0009_refusal_line.sql` (`2.5.1`) and
| 7 | `0007_delivery_log_days.sql` | 2.5.0 | DDL | `delivery_log_days`: what the reconcile sweep and the log tailer did, a row per UTC day, kept a year, shown nowhere - the measurement that decides the sweep's fate (audit 2026-10-12, stage 1). `send_log.reason` gains the values `reconcile` and `reconcile-contradicted` without a schema change |
| 8 | `0008_tenant_boundary.sql` | 2.5.0 | DDL + data | `idx_applications_login_nocase`: an application login is unique without regard to case (audit 2026-10-12, H4). `send_log.domain_id` and `inbound_spam_log.inbound_domain_id` (`ON DELETE SET NULL`), each with an index, filled once by name from the registry as it was then: a row belongs to a domain by id, and a domain deleted and added again starts with an empty log (L12) |
| 9 | `0009_refusal_line.sql` | 2.5.1 | DDL | `send_log.refusal_line`: Postfix's line about a refusal at MAIL FROM (a level-2 limit, a client address off the list), which has no queue id; the log tailer puts it on the row; `send_log.recovered`, a refusal row the tailer wrote from `mail.log` because the panel stopped before writing it; `sender_check_days.messages` (messages deferred mid-message, counted apart from the connections turned away) and `sender_check_days.estimated` (a day whose downtime is an upper bound) (audit 2026-10-12, stage 2b) |
| 10 | `0010_send_log_indexes.sql` | 2.5.3 | DDL | Four indexes of `send_log`, each for a question that walked the table or sorted a domain's log (audit 2026-10-12, M2, stage 3): `idx_send_log_open`, partial, the queued and deferred rows by time (the tailer's sweep every 5 min); `idx_send_log_domain_id` and `idx_send_log_app_login`, a page of one domain's or one application's log in id order; `idx_send_log_refusal_line`, partial, whether a refusal line is already on a row. `TestSendLogQueriesUseTheirIndexes` checks the plans of the statements the store runs |
| 10 | `0010_send_log_indexes.sql` | 2.5.3 | DDL | Four indexes of `send_log`, each for a question that walked the table or sorted a domain's log (audit 2026-10-12, M2, stage 3): `idx_send_log_open`, partial, the queued and deferred rows by time (the tailer's sweep every 5 min); `idx_send_log_domain_id` and `idx_send_log_app_login`, a page of one domain's or one application's log in id order; `idx_send_log_refusal_line`, partial, whether a refusal line is already on a row. `TestSendLogQueriesUseTheirIndexes` checks the plans of these and of the other by-time and by-scope statements the store runs |
**Kind:** *DDL* — schema only; *data* — `INSERT`/`UPDATE` that must stay correct
for operators upgrading from older 2.x images.
+30 -2
View File
@@ -308,6 +308,23 @@ operator can act on it.
that the panel's user may read. `certs/` is the operator's, mounted
read-only; only someone who can write the project directory places a link
there, and only a global user can download the archive (`2.5.2`).
What a backup still leaves out - a link under `data/`, a link under
`certs/` to a directory - it names in a warning, and a `certs/` with no
file in it is refused rather than archived empty (`2.5.3`).
- A backup is made only of the database that is there: a wrong
`SELFPOST_DB_PATH` is refused, and the snapshot opens the source
read-only. An archive of an empty database would restore a server that
offers its first-run setup - a new global administrator - while it holds
live DKIM keys and `sasldb2` (`2.5.3`).
- The first start after a restore opens nothing until `/data` matches the
archive's last entry, `data/contents.json` (size and SHA-256 of each file
under `data/`): an archive unpacked in part - its manifest there, half its
database - is refused (`2.5.3`). The check is integrity, not authenticity:
whoever can write the project directory can write that record too. It
reads only regular files at the recorded names (a link or a pipe there is
refused, not followed), refuses a database journal file beside the
restored database, and is made once per restore: a start after one that
passed it and failed is not compared again while the manifest is the same.
- Since 2.2.0 a full backup also carries, in the database, what Postfix logged
about each message still in the send log (`send_log_lines`): the submitting
client's address and SASL login, the message id, the receiving server and
@@ -318,7 +335,12 @@ operator can act on it.
scrypt (N=2¹⁵, r=8, p=1) → AES-256-GCM, streamed in 64 KiB chunks, each
authenticated with the header, the chunk number, and an end-of-stream flag —
a truncated or substituted file fails to open instead of silently restoring a
partial "tail". Format and wrapper:
partial "tail", and bytes after the last chunk are refused (`2.5.3`). The
key-derivation parameters travel in the header, so a reader is bounded in
what a header may ask before the password is checked: scrypt's array at
most 128 MiB (four times what SelfPost writes), r ≤ 64 so the rest stays
under 32 KiB, p ≤ 2 - *Import a domain* reads an uploaded file, and the
old cap let a header ask for 4 GiB (`2.5.3`). Format and wrapper:
[internal/secretfile](../internal/secretfile/secretfile.go).
- Extensions: `.spbk` (**S**elf**P**ost **b**ac**k**up — full backup), `.spde`
(**S**elf**P**ost **d**omain **e**xport — domain export); unencrypted files
@@ -329,7 +351,13 @@ operator can act on it.
`-password-file`, never as an argument (the process list is readable by any
process in the container).
- Minimum password length matches the administrator password (12): the file
sits offline and can be attacked without a time limit.
sits offline and can be attacked without a time limit. The panel's forms and,
since `2.5.3`, the CLI hold an encrypting password to it; decrypting takes
any.
- The CLI's `-o FILE` is written beside its name and takes it only when
whole, so a failure - a wrong password to `-decrypt` among them, which is
checked on the first chunk before the output is touched - leaves the file
that was there (`2.5.3`).
---
+3 -4
View File
@@ -12,6 +12,8 @@ import (
"path/filepath"
"strings"
"sync"
"github.com/mixeme/selfpost/internal/atomicfile"
)
// SASLDB manages the Cyrus SASL account database (sasldb2) the panel maintains
@@ -118,10 +120,7 @@ func replaceSASLDB(path string, edit func(path string) error) error {
// loss may bring the old database back, out of step with the panel's
// records (review of stage 2c, F10). A directory that cannot be synced
// - not on every system - leaves the change as durable as before.
if dir, err := os.Open(filepath.Dir(path)); err == nil {
_ = dir.Sync()
dir.Close()
}
_ = atomicfile.SyncDir(filepath.Dir(path))
return nil
}
+23 -1
View File
@@ -1,3 +1,5 @@
// Package atomicfile writes a file so that readers see the old file or the
// new one, whole, and a power loss after Write returns keeps the new one.
package atomicfile
import (
@@ -7,7 +9,13 @@ import (
)
// Write writes data to path via a temp file in the same directory followed by
// a rename, so readers only ever see the complete old or new file.
// a rename, so readers only ever see the complete old or new file. The file
// is synced before the rename and the directory after it: the rename is a
// change of the directory, and without its own sync a power loss may bring
// the old file back (fsync(2): syncing a file "does not necessarily ensure
// that the entry in the directory containing the file has also reached
// disk"; audit 2026-10-12, L7). Every map the panel writes for Postfix and
// OpenDKIM goes through here.
func Write(path string, data []byte, perm os.FileMode) error {
dir := filepath.Dir(path)
tmp, err := os.CreateTemp(dir, ".tmp-*")
@@ -41,5 +49,19 @@ func Write(path string, data []byte, perm os.FileMode) error {
return fmt.Errorf("rename %s -> %s: %w", tmpName, path, err)
}
cleanup = false
_ = SyncDir(dir)
return nil
}
// SyncDir makes a rename in dir durable. It is best effort for its callers:
// the new file is in place and readable either way, and not every system
// syncs a directory - Go on Windows cannot open one for it - so a failure
// leaves the change as durable as it was before this existed.
func SyncDir(dir string) error {
d, err := os.Open(dir)
if err != nil {
return err
}
defer d.Close()
return d.Sync()
}
+113
View File
@@ -0,0 +1,113 @@
package atomicfile
import (
"os"
"path/filepath"
"runtime"
"testing"
)
// leftovers lists what is in dir besides the names given.
func leftovers(t *testing.T, dir string, names ...string) []string {
t.Helper()
entries, err := os.ReadDir(dir)
if err != nil {
t.Fatal(err)
}
keep := map[string]bool{}
for _, n := range names {
keep[n] = true
}
var out []string
for _, e := range entries {
if !keep[e.Name()] {
out = append(out, e.Name())
}
}
return out
}
// A new file and a replaced one hold what was written, with the mode asked
// for, and nothing is left beside them.
func TestWriteCreatesAndReplaces(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "map")
for _, body := range []string{"first\n", "second, longer than the first\n", ""} {
if err := Write(path, []byte(body), 0o640); err != nil {
t.Fatalf("Write %q: %v", body, err)
}
got, err := os.ReadFile(path)
if err != nil || string(got) != body {
t.Fatalf("file = %q, %v; want %q", got, err, body)
}
if runtime.GOOS != "windows" {
if info, _ := os.Stat(path); info.Mode().Perm() != 0o640 {
t.Errorf("mode = %v, want 0640", info.Mode().Perm())
}
}
if rest := leftovers(t, dir, "map"); len(rest) != 0 {
t.Errorf("Write left %v", rest)
}
}
}
// A reader that opened the old file reads the old file to its end: the new
// one takes the name, it does not overwrite the old one's bytes.
func TestAReaderKeepsTheOldFile(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("Windows does not rename over an open file; the container is Linux")
}
path := filepath.Join(t.TempDir(), "map")
if err := Write(path, []byte("old contents\n"), 0o640); err != nil {
t.Fatal(err)
}
f, err := os.Open(path)
if err != nil {
t.Fatal(err)
}
defer f.Close()
if err := Write(path, []byte("new\n"), 0o640); err != nil {
t.Fatal(err)
}
buf := make([]byte, 64)
n, _ := f.Read(buf)
if string(buf[:n]) != "old contents\n" {
t.Errorf("the open reader read %q", buf[:n])
}
}
// A write that fails leaves the old file and no temporary one.
func TestAFailedWriteLeavesTheOldFile(t *testing.T) {
dir := t.TempDir()
if err := Write(filepath.Join(dir, "missing", "map"), []byte("x"), 0o640); err == nil {
t.Error("Write into a directory that is not there succeeded")
}
// The name is a directory: the rename fails after the temporary file
// was written.
target := filepath.Join(dir, "taken")
if err := os.MkdirAll(filepath.Join(target, "inside"), 0o750); err != nil {
t.Fatal(err)
}
if err := Write(target, []byte("x"), 0o640); err == nil {
t.Error("Write over a directory succeeded")
}
if rest := leftovers(t, dir, "taken"); len(rest) != 0 {
t.Errorf("the failed Write left %v", rest)
}
if _, err := os.Stat(filepath.Join(target, "inside")); err != nil {
t.Errorf("the failed Write changed the directory in its way: %v", err)
}
}
// The directory is synced where the system allows it; on Linux, where the
// container runs, it does.
func TestSyncDir(t *testing.T) {
err := SyncDir(t.TempDir())
if runtime.GOOS == "linux" && err != nil {
t.Errorf("SyncDir on Linux = %v", err)
}
if err := SyncDir(filepath.Join(t.TempDir(), "missing")); err == nil {
t.Error("SyncDir of a directory that is not there = nil")
}
}
+82 -15
View File
@@ -166,7 +166,7 @@ func Create(w io.Writer, p Params) error {
return err
}
if err := addTree(a, p.DataDir, DataArchivePrefix, excludedFromArchive, skipQueue, false); err != nil {
if _, err := addTree(a, p.DataDir, DataArchivePrefix, excludedFromArchive, skipQueue, false, p.OnWarn); err != nil {
return err
}
@@ -239,7 +239,7 @@ func Check(p Params) error {
if err := ValidateDeployRoot(p.DeployRoot); err != nil {
return err
}
if err := addTree(nil, p.DataDir, DataArchivePrefix, excludedFromArchive, skipQueue, false); err != nil {
if _, err := addTree(nil, p.DataDir, DataArchivePrefix, excludedFromArchive, skipQueue, false, nil); err != nil {
return err
}
return addDeployFiles(nil, p.DeployRoot, nil)
@@ -297,20 +297,43 @@ func addDeployFiles(a *archive, deployRoot string, onWarn func(string)) error {
// are links again, relative, into archive/ (eff-certbot docs, "Where are
// my certificates?"; assumptions C-Q4, C-A1). The archive carries the
// files they lead to, under the links' names.
return addTree(a, certsDir, CertsDirName+"/", nil, nil, true)
files, err := addTree(a, certsDir, CertsDirName+"/", nil, nil, true, onWarn)
if err != nil {
return err
}
// certs/ is what Postfix serves its certificate from (the compose file
// mounts it at /etc/postfix/tls). There and empty - or holding only what
// the archive cannot carry - it would give an archive that restores a
// server without its certificate, so there is none (audit 2026-10-12,
// L2).
if files == 0 {
return ErrNoCerts
}
return nil
}
// ErrNoCerts is a certs/ that is there with no file a backup can carry.
var ErrNoCerts = errors.New("backup: " + CertsDirName + "/ holds no file this backup can carry, so the server it restores would have no certificate - put the PEM files (or links to them) there")
// addTree walks root and adds every regular file (and directory, to preserve
// empty ones and modes) to a under archivePrefix + path relative to root.
// empty ones and modes) to a under archivePrefix + path relative to root, and
// returns how many files it took.
// When exclude is non-nil, top-level names relative to root are skipped; skip,
// when non-nil, leaves out whatever it says. With a nil a nothing is written:
// every file is opened and closed, which is how Check finds what cannot be read.
// Non-regular, non-directory entries (symlinks, sockets) are skipped - except,
// with follow, a symbolic link to a regular file, which is archived as that
// file under the link's name; a link that leads nowhere this process can
// reach is an error, as an unreadable file is.
func addTree(a *archive, root, archivePrefix string, exclude map[string]bool, skip skipFunc, follow bool) error {
return filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
// reach is an error, as an unreadable file is. Each entry left out that way
// is told to warn, when it is not nil: a backup does not leave something out
// without saying so (audit 2026-10-12, L2).
func addTree(a *archive, root, archivePrefix string, exclude map[string]bool, skip skipFunc, follow bool, warn func(string)) (files int, err error) {
skipped := func(name, what string) {
if warn != nil {
warn(name + " is " + what + ", not a file: the backup does not carry it")
}
}
err = filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
@@ -362,6 +385,9 @@ func addTree(a *archive, root, archivePrefix string, exclude map[string]bool, sk
// the listing and the open: not part of /data's state.
return nil
}
if err == nil {
files++
}
return err
case follow && info.Mode()&fs.ModeSymlink != 0:
target, err := os.Stat(path)
@@ -369,13 +395,37 @@ func addTree(a *archive, root, archivePrefix string, exclude map[string]bool, sk
return fmt.Errorf("backup: %s is a link to a file this container cannot reach - with certbot, /etc/letsencrypt is mounted at the same path (guide § Reverse proxy): %w", name, err)
}
if !target.Mode().IsRegular() {
skipped(name, "a link to a directory")
return nil
}
return writeTarFile(a, name, target.Mode().Perm(), path)
if err := writeTarFile(a, name, target.Mode().Perm(), path); err != nil {
return err
}
files++
return nil
case info.Mode()&fs.ModeSymlink != 0:
skipped(name, "a link")
return nil
default:
skipped(name, kindOf(info.Mode()))
return nil
}
})
return files, err
}
// kindOf names what a non-regular, non-directory entry is, for a warning.
func kindOf(m fs.FileMode) string {
switch {
case m&fs.ModeSocket != 0:
return "a socket"
case m&fs.ModeNamedPipe != 0:
return "a named pipe"
case m&fs.ModeDevice != 0:
return "a device"
default:
return "neither a file nor a directory"
}
}
// writeTarBytes writes an in-memory file entry.
@@ -430,6 +480,21 @@ func writeTarFile(a *archive, name string, mode fs.FileMode, srcPath string) err
// applyQueryParams(): each _pragma is run as "pragma <value>" on connect,
// busy_timeout first.)
func snapshotDB(dbPath string) (path string, cleanup func(), err error) {
// The database must be there: SQLite creates a file that is not, and an
// archive of that empty database restores a panel that offers its
// first-run setup on a server holding live keys (audit 2026-10-12, M3).
// mode=ro makes the open fail rather than create one, should the file go
// between the two: SQLite 3.53.4 creates a database only under "rwc", the
// default (https://sqlite.org/uri.html § 3.3), and VACUUM INTO is not a
// write to the source (lang_vacuum.html § 2.1, § 3) - both measured by
// TestSnapshotRefusesAMissingDatabase.
info, err := os.Stat(dbPath)
if err != nil {
return "", nil, fmt.Errorf("backup: the database: %w", err)
}
if !info.Mode().IsRegular() {
return "", nil, fmt.Errorf("backup: the database %s is not a file", dbPath)
}
dir, err := os.MkdirTemp("", "selfpost-backup-")
if err != nil {
return "", nil, fmt.Errorf("backup: temp dir: %w", err)
@@ -439,7 +504,7 @@ func snapshotDB(dbPath string) (path string, cleanup func(), err error) {
// A short busy timeout lets VACUUM INTO wait out a brief writer rather than
// failing immediately if the panel happens to be mid-write.
db, err := sql.Open("sqlite", "file:"+dbPath+"?_pragma=busy_timeout(5000)")
db, err := sql.Open("sqlite", "file:"+dbPath+"?mode=ro&_pragma=busy_timeout(5000)")
if err != nil {
cleanup()
return "", nil, fmt.Errorf("backup: open database: %w", err)
@@ -497,16 +562,18 @@ func CheckRestore(manifestPath, binaryVersion string) (restored bool, err error)
// ConsumeRestore ends a restore once its first start has done what a restore
// start does: the manifest is removed, so later starts (and in-place
// upgrades) are not gated by it, and then the record of the checked files. A
// start stopped between the two leaves the record alone, which nothing reads
// without a manifest and no backup carries.
// upgrades) are not gated by it, and then the record of the files, checked or
// unpacked again since. A start stopped between the two leaves the record
// alone, which nothing reads without a manifest and no backup carries.
func ConsumeRestore(manifestPath string) error {
if err := os.Remove(manifestPath); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("backup: consume restore manifest: %w", err)
}
checked := filepath.Join(filepath.Dir(manifestPath), contentsCheckedName)
if err := os.Remove(checked); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("backup: consume restore manifest: %w", err)
for _, name := range []string{contentsCheckedName, ContentsName} {
err := os.Remove(filepath.Join(filepath.Dir(manifestPath), name))
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("backup: consume restore manifest: %w", err)
}
}
return nil
}
+141
View File
@@ -5,6 +5,7 @@ import (
"bytes"
"compress/gzip"
"crypto/sha256"
"database/sql"
"encoding/hex"
"encoding/json"
"fmt"
@@ -522,3 +523,143 @@ func TestCertsLinksAreFollowed(t *testing.T) {
t.Fatalf("Check with a dangling link: %v", err)
}
}
// A wrong SELFPOST_DB_PATH is refused, and nothing is created where it
// points: SQLite makes a database of a path that is not one, and an archive
// of that empty database restores a server that offers its first-run setup
// (audit 2026-10-12, M3). The snapshot opens the source read-only, which is
// what refuses it should the file go between the check and the open.
func TestSnapshotRefusesAMissingDatabase(t *testing.T) {
dataDir, _, deployRoot := seedDataDir(t)
missing := filepath.Join(dataDir, "elsewhere.db")
var buf bytes.Buffer
err := Create(&buf, Params{DataDir: dataDir, DBPath: missing, Version: "1.0.0", DeployRoot: deployRoot})
if err == nil {
t.Fatal("a backup of a database that is not there was made")
}
if _, statErr := os.Stat(missing); !os.IsNotExist(statErr) {
t.Fatalf("the refused backup created %s (stat err = %v)", missing, statErr)
}
db, err := sql.Open("sqlite", "file:"+missing+"?mode=ro")
if err != nil {
t.Fatal(err)
}
defer db.Close()
if err := db.Ping(); err == nil {
t.Fatal("mode=ro opened a database that is not there")
}
if _, statErr := os.Stat(missing); !os.IsNotExist(statErr) {
t.Fatalf("mode=ro created %s (stat err = %v)", missing, statErr)
}
}
// certs/ there and empty is refused by Check and Create alike: the archive
// would restore a server without its certificate (audit 2026-10-12, L2).
func TestEmptyCertsIsRefused(t *testing.T) {
dataDir, dbPath, deployRoot := seedDataDir(t)
for _, f := range []string{"fullchain.pem", "privkey.pem"} {
os.Remove(filepath.Join(deployRoot, CertsDirName, f))
}
p := Params{DataDir: dataDir, DBPath: dbPath, Version: "1.0.0", DeployRoot: deployRoot}
if err := Check(p); err == nil || !strings.Contains(err.Error(), "certs/ holds no file") {
t.Errorf("Check with an empty certs/ = %v", err)
}
if err := Create(io.Discard, p); err == nil || !strings.Contains(err.Error(), "certs/ holds no file") {
t.Errorf("Create with an empty certs/ = %v", err)
}
}
// What a backup leaves out it says it leaves out: a link under data/, and a
// link under certs/ to a directory (audit 2026-10-12, L2).
func TestBackupWarnsOfWhatItLeavesOut(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("symbolic links need privileges on Windows; the container is Linux")
}
dataDir, dbPath, deployRoot := seedDataDir(t)
if err := os.Symlink("sasl/sasldb2", filepath.Join(dataDir, "a-link")); err != nil {
t.Fatal(err)
}
if err := os.Symlink(t.TempDir(), filepath.Join(deployRoot, CertsDirName, "a-directory")); err != nil {
t.Fatal(err)
}
var warnings []string
p := Params{DataDir: dataDir, DBPath: dbPath, Version: "1.0.0", DeployRoot: deployRoot,
OnWarn: func(s string) { warnings = append(warnings, s) }}
if err := Create(io.Discard, p); err != nil {
t.Fatalf("Create: %v", err)
}
got := strings.Join(warnings, "\n")
for _, want := range []string{
"data/a-link is a link, not a file: the backup does not carry it",
"certs/a-directory is a link to a directory, not a file: the backup does not carry it",
} {
if !strings.Contains(got, want) {
t.Errorf("no warning %q in:\n%s", want, got)
}
}
}
// A link or a pipe where the record names a file is refused, not followed or
// read: a crafted archive must not make the start read outside /data, or
// wait (security review of audit stage 3).
func TestCheckRestoreReadsOnlyRegularFiles(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("symbolic links need privileges on Windows; the container is Linux")
}
path := unpackData(t, createArchive(t, "1.0.0"))
dir := filepath.Dir(path)
sasl := filepath.Join(dir, "sasl", "sasldb2")
outside := filepath.Join(t.TempDir(), "outside")
writeFile(t, outside, "SASLDB")
os.Remove(sasl)
if err := os.Symlink(outside, sasl); err != nil {
t.Fatal(err)
}
if _, err := CheckRestore(path, "1.0.0"); err == nil || !strings.Contains(err.Error(), "data/sasl/sasldb2 is not the file") {
t.Fatalf("a link to a file of the same contents = %v, want refused", err)
}
}
// A journal file beside the restored database is from what the directory
// held before; SQLite would take it up, so the restore is refused.
func TestCheckRestoreRefusesAJournalBesideTheDatabase(t *testing.T) {
for _, side := range []string{"-wal", "-shm", "-journal"} {
path := unpackData(t, createArchive(t, "1.0.0"))
writeFile(t, filepath.Join(filepath.Dir(path), "selfpost.db"+side), "old")
if _, err := CheckRestore(path, "1.0.0"); err == nil || !strings.Contains(err.Error(), "selfpost.db"+side+" is beside the database") {
t.Errorf("%s beside the database = %v, want refused", side, err)
}
}
}
// The archive unpacked again after a start that checked it is checked again:
// its record wins over the mark of the earlier check, and a file it cut short
// is refused. A finished restore leaves neither record.
func TestARecordUnpackedAgainIsChecked(t *testing.T) {
archive := createArchive(t, "1.0.0")
path := unpackData(t, archive)
dir := filepath.Dir(path)
if _, err := CheckRestore(path, "1.0.0"); err != nil {
t.Fatal(err)
}
files := readArchive(t, archive)
writeFile(t, filepath.Join(dir, ContentsName), files[DataArchivePrefix+ContentsName])
writeFile(t, filepath.Join(dir, "sasl", "sasldb2"), "SAS")
if _, err := CheckRestore(path, "1.0.0"); err == nil || !strings.Contains(err.Error(), "data/sasl/sasldb2 is not the file") {
t.Fatalf("a second unpack cut short = %v, want refused", err)
}
writeFile(t, filepath.Join(dir, "sasl", "sasldb2"), files[DataArchivePrefix+"sasl/sasldb2"])
if _, err := CheckRestore(path, "1.0.0"); err != nil {
t.Fatalf("the second unpack whole: %v", err)
}
writeFile(t, filepath.Join(dir, ContentsName), files[DataArchivePrefix+ContentsName])
if err := ConsumeRestore(path); err != nil {
t.Fatal(err)
}
for _, name := range []string{ManifestName, ContentsName, contentsCheckedName} {
if _, err := os.Stat(filepath.Join(dir, name)); !os.IsNotExist(err) {
t.Errorf("%s left after ConsumeRestore", name)
}
}
}
+44 -15
View File
@@ -12,6 +12,8 @@ import (
"path"
"path/filepath"
"strings"
"github.com/mixeme/selfpost/internal/atomicfile"
)
// FormatContents identifies the record of what an archive's data directory
@@ -93,34 +95,49 @@ func errUnpackedPart(detail string) error {
}
// checkContents compares the data directory with the archive's record, once
// per restore (see contentsCheckedName).
// per restore (see contentsCheckedName). A record unpacked anew wins over the
// mark of an earlier check: the archive was unpacked again, and that is what
// the start runs on (review of audit stage 3).
func checkContents(dataDir string) error {
checked := filepath.Join(dataDir, contentsCheckedName)
if c, err := readContents(checked); err == nil {
// A start before this one checked the files. It is the same restore
// when the manifest is still the one that record holds; a different
// archive unpacked over the directory since is checked anew.
if verifyFiles(dataDir, c, DataArchivePrefix+ManifestName) == nil {
return nil
}
} else if !errors.Is(err, os.ErrNotExist) {
return err
}
path := filepath.Join(dataDir, ContentsName)
checked := filepath.Join(dataDir, contentsCheckedName)
c, err := readContents(path)
if errors.Is(err, os.ErrNotExist) {
prev, perr := readContents(checked)
if perr == nil && verifyFiles(dataDir, prev, DataArchivePrefix+ManifestName) == nil {
// A start before this one checked the files, and the manifest is
// still the one that record holds: the same restore, whose
// files that start may have changed since.
return nil
}
if perr != nil && !errors.Is(perr, os.ErrNotExist) {
return perr
}
return errUnpackedPart(DataArchivePrefix + ContentsName + ", its last entry, is not there")
}
if err != nil {
return err
}
// No backup carries the database's journal files. One beside the
// restored database is left from the database the directory held
// before, and opening this one would take it up: the first connection
// after a crash recovers from the WAL, and a checkpoint "takes content
// from the WAL file and transfers it back into the original database
// file" (https://sqlite.org/wal.html § 9, § 2.2).
for _, side := range []string{"-wal", "-shm", "-journal"} {
if _, err := os.Lstat(filepath.Join(dataDir, "selfpost.db"+side)); err == nil {
return errUnpackedPart("selfpost.db" + side + " is beside the database, and no archive holds one - it is left from what was in the directory before")
}
}
if err := verifyFiles(dataDir, c, ""); err != nil {
return err
}
if err := os.Rename(path, checked); err != nil {
return fmt.Errorf("backup: mark the restored files checked: %w", err)
}
// The start goes on to change the files; the mark must not be lost to a
// power cut after that, or the next start compares changed files.
_ = atomicfile.SyncDir(dataDir)
return nil
}
@@ -151,15 +168,27 @@ func verifyFiles(dataDir string, c Contents, only string) error {
if !ok || rel == "" || path.Clean(rel) != rel || rel == ".." || strings.HasPrefix(rel, "../") || path.IsAbs(rel) {
return errUnpackedPart("the record names " + f.Name + ", which is not a file of the data directory")
}
file, err := os.Open(filepath.Join(dataDir, filepath.FromSlash(rel)))
// The archive holds regular files only (archive.writeFile); a link
// or a pipe under the name is not followed or read - a crafted
// archive must not make the start read a file outside /data, or
// wait on a pipe (security review of audit stage 3).
full := filepath.Join(dataDir, filepath.FromSlash(rel))
info, err := os.Lstat(full)
if errors.Is(err, os.ErrNotExist) {
return errUnpackedPart(f.Name + " is missing")
}
if err != nil {
return fmt.Errorf("backup: check %s: %w", f.Name, err)
}
if !info.Mode().IsRegular() || info.Size() != f.Size {
return errUnpackedPart(f.Name + " is not the file the archive holds")
}
file, err := os.Open(full)
if err != nil {
return fmt.Errorf("backup: check %s: %w", f.Name, err)
}
h := sha256.New()
n, err := io.Copy(h, file)
n, err := io.Copy(h, io.LimitReader(file, f.Size+1))
file.Close()
if err != nil {
return fmt.Errorf("backup: check %s: %w", f.Name, err)
+27 -8
View File
@@ -105,12 +105,19 @@ const (
defaultR = 8
defaultP = 1
// maxLogN caps what a file may ask for, so a hostile header cannot make the
// reader allocate its way to an out-of-memory kill before it ever fails
// authentication.
maxLogN = 20
maxR = 32
maxP = 16
// What a file may ask of the reader, so a hostile header cannot make it
// allocate its way to an out-of-memory kill, or spin, before it ever fails
// authentication (audit 2026-10-12, M4). golang.org/x/crypto v0.58.0,
// scrypt.Key, allocates V, 128·r·N bytes (make([]uint32, 32*N*r)), B,
// p·128·r bytes of PBKDF2 output, and XY, 256·r; its work is N·r·p. V is
// held to four times what the writer asks (128·2^15·8 = 32 MiB), r to 64
// - the writer's is 8 - so B and XY stay under 32 KiB together (a header
// with N=2 and a huge r asked for 384 MiB in B and XY: security review of
// stage 3), p to 2 - the writer's is 1. The domain import reads a file
// any global user uploads, the CLI a file of any origin.
maxScryptMemory = 128 << 20
maxR = 64
maxP = 2
)
var (
@@ -180,8 +187,11 @@ func parseHeader(buf []byte) (*header, error) {
h.salt = buf[i : i+saltLen]
h.noncePrefix = buf[i+saltLen : i+saltLen+noncePrefixLen]
// Reject absurd work factors before spending any memory on them.
if h.logN < 1 || h.logN > maxLogN || h.r < 1 || h.r > maxR || h.p < 1 || h.p > maxP {
// Reject absurd work factors before spending any memory on them. logN
// under 31 and r under 65 keep the shift and the product in range; then
// the product is held to the budget.
if h.logN < 1 || h.logN > 30 || h.r < 1 || h.r > maxR || h.p < 1 || h.p > maxP ||
128*(uint64(1)<<h.logN)*uint64(h.r) > maxScryptMemory {
return nil, fmt.Errorf("%w: unreasonable key-derivation parameters", ErrCorrupt)
}
return h, nil
@@ -450,6 +460,15 @@ func (d *Reader) next() error {
if err != nil {
return ErrWrongPassword
}
// The last chunk ends the file: bytes after it - another envelope
// appended, say - are damage, not something to ignore (audit
// 2026-10-12, L3).
var more [1]byte
if n, err := io.ReadFull(d.r, more[:]); n > 0 {
return fmt.Errorf("%w: bytes after the last chunk", ErrCorrupt)
} else if err != nil && !errors.Is(err, io.EOF) {
return fmt.Errorf("secretfile: read chunk: %w", err)
}
d.done = true
}
d.plain = plain
+65 -1
View File
@@ -2,6 +2,7 @@ package secretfile
import (
"bytes"
"encoding/binary"
"errors"
"io"
"testing"
@@ -155,8 +156,71 @@ func TestEmptyPasswordRejected(t *testing.T) {
func TestUnreasonableKDFParamsRejected(t *testing.T) {
enc := encrypt(t, TypeFullBackup, "password password", []byte("x"))
bad := bytes.Clone(enc)
bad[MagicLen+2] = 40 // logN far beyond maxLogN
bad[MagicLen+2] = 40 // logN far beyond the budget
if _, err := NewReader(bytes.NewReader(bad), "password password"); !errors.Is(err, ErrCorrupt) {
t.Fatalf("err = %v, want ErrCorrupt", err)
}
}
// A header may ask for no more than the stated budget: 128·N·r bytes of
// scrypt's array at most 128 MiB, r at most 64, p at most 2 (audit
// 2026-10-12, M4; security review of stage 3). The writer's own parameters
// are a quarter of it.
func TestKDFBudget(t *testing.T) {
enc := encrypt(t, TypeFullBackup, "password password", []byte("x"))
header := func(logN uint8, r, p uint32) []byte {
h := bytes.Clone(enc[:headerLen])
h[MagicLen+2] = logN
binary.BigEndian.PutUint32(h[MagicLen+3:], r)
binary.BigEndian.PutUint32(h[MagicLen+7:], p)
return h
}
for _, c := range []struct {
logN uint8
r, p uint32
ok bool
}{
{defaultLogN, defaultR, defaultP, true},
{17, 8, 2, true}, // 128 MiB, the budget
{20, 1, 1, true}, // 128 MiB another way
{18, 8, 1, false}, // 256 MiB
{15, 64, 1, false}, // 256 MiB
{20, 32, 16, false}, // what the old cap let through: 4 GiB
{15, 8, 3, false}, // p over 2
{0, 8, 1, false}, // N of 1
{31, 1, 1, false}, // past the shift
{1, 1 << 21, 1, false}, // r alone over the budget
{1, 1 << 19, 2, false}, // V small, B and XY 384 MiB
{11, 64, 2, true}, // r at its cap
{11, 65, 1, false}, // r past it
{30, 1 << 20, 1, false}, // the largest product, still refused
{15, 0, 1, false},
{15, 8, 0, false},
} {
_, err := parseHeader(header(c.logN, c.r, c.p))
if c.ok && err != nil {
t.Errorf("logN=%d r=%d p=%d refused: %v", c.logN, c.r, c.p, err)
}
if !c.ok && !errors.Is(err, ErrCorrupt) {
t.Errorf("logN=%d r=%d p=%d = %v, want ErrCorrupt", c.logN, c.r, c.p, err)
}
}
}
// Nothing may follow the last chunk: two envelopes end to end, or stray
// bytes, are damage rather than the first file (audit 2026-10-12, L3).
func TestBytesAfterTheLastChunkRejected(t *testing.T) {
first := encrypt(t, TypeFullBackup, "password password", []byte("first"))
second := encrypt(t, TypeFullBackup, "password password", []byte("second"))
for name, data := range map[string][]byte{
"two envelopes": append(bytes.Clone(first), second...),
"one byte": append(bytes.Clone(first), 0),
} {
if _, _, err := decrypt(t, data, "password password"); !errors.Is(err, ErrCorrupt) {
t.Errorf("%s: err = %v, want ErrCorrupt", name, err)
}
}
if plain, _, err := decrypt(t, first, "password password"); err != nil || string(plain) != "first" {
t.Errorf("the envelope alone = %q, %v", plain, err)
}
}
+9 -6
View File
@@ -25,12 +25,15 @@ func explain(t *testing.T, s *Store, query string, args ...any) string {
return strings.Join(plan, "\n")
}
// The send log is the one table that grows with the mail, and every question
// asked of it on a path that runs per message, per log line or every few
// minutes is answered from an index: measured with EXPLAIN QUERY PLAN against
// the statements the store runs, not copies of them (audit 2026-10-12, M2;
// stage 3). SQLite chooses without statistics here, as it does on a server -
// nothing runs ANALYZE.
// The send log is the one table that grows with the mail. The questions
// asked of it by time or by scope - per message (the level-2 count), per log
// line (a refusal line), every few minutes (the sweep, the notifier), per
// page and per statistics view - are answered from an index: measured with
// EXPLAIN QUERY PLAN against the statements the store runs, not copies of
// them (audit 2026-10-12, M2; stage 3). Those by queue id (UpdateStatus,
// ExpireMessage, RecoverRefusal) go by idx_send_log_queue_id and are not
// pinned here. SQLite chooses without statistics here, as it does on a
// server - nothing runs ANALYZE.
func TestSendLogQueriesUseTheirIndexes(t *testing.T) {
s := openTestStore(t)
const since = "2026-10-01T00:00:00Z"
+6
View File
@@ -328,6 +328,12 @@ const attachRefusalLineQuery = `UPDATE send_log SET refusal_line = ?1 WHERE id =
// § 2); NOCASE folds ASCII only (datatype3.html § 7.1); ?1 is bound once and
// used twice (lang_expr.html § 4).
func (s *Store) AttachRefusalLine(clientIP, login, from string, at time.Time, line string) (bool, error) {
// An empty line is what "no line" is on a row; the statement's second
// term would let it through, the old statement did not (review of stage
// 3). No caller passes one.
if line == "" {
return false, nil
}
res, err := s.db.Exec(attachRefusalLineQuery,
line, ReasonLimit, ReasonClient, from, clientIP,
at.Add(-refusalLineWindow).UTC().Format(time.RFC3339),
+4
View File
@@ -345,6 +345,10 @@ func unreadableBackupErr(err error, deployRoot string) string {
return "The panel is not allowed to read " + name + ", and a backup without it would not bring the server back. " +
"On the host, give the file to group 998 and make it group-readable (chgrp 998, chmod 640), then try again. Nothing was downloaded."
}
if errors.Is(err, backup.ErrNoCerts) {
return "The project's certs/ holds no certificate, so a backup would bring the server back without one. " +
"On the host, put fullchain.pem and privkey.pem - or links to certbot's - in certs/, then try again. Nothing was downloaded."
}
return "Could not read everything a backup holds. Nothing was downloaded; the reason is in the panel's log."
}
@@ -9,6 +9,7 @@ import (
"strings"
"testing"
"github.com/mixeme/selfpost/internal/backup"
"github.com/mixeme/selfpost/internal/secretfile"
"github.com/mixeme/selfpost/internal/web/validate"
)
@@ -123,3 +124,12 @@ func TestBackupPageOffersEncryption(t *testing.T) {
}
}
}
// An empty certs/ is not "could not read": the page says what is missing and
// where it goes (review of audit stage 3).
func TestEmptyCertsMessage(t *testing.T) {
msg := unreadableBackupErr(fmt.Errorf("check: %w", backup.ErrNoCerts), "/selfpost-deploy")
if !strings.Contains(msg, "certs/ holds no certificate") || !strings.Contains(msg, "fullchain.pem") {
t.Errorf("message = %q", msg)
}
}
+14
View File
@@ -41,6 +41,16 @@ func testFullBackupRestore(t *testing.T, sc *scenario) {
t.Fatal(err)
}
// A map under data/postfix by a name the entrypoint does not list - the
// DMARC maps were such, with DMARC on (review of audit stage 3, H1). The
// archive records no owner, and the unpack below runs as root, as an
// operator's tar does: the entrypoint has to hand the file back to the
// panel, which reads every recorded file on the first start.
if out, err := h.execIn("selfpost", "sh", "-c",
"echo 'x OK' > /data/postfix/e2e_extra_map && chown panel:selfpost /data/postfix/e2e_extra_map && chmod 0640 /data/postfix/e2e_extra_map"); err != nil {
t.Fatalf("write an extra map: %v\n%s", err, out)
}
resp, archive, err := sc.panel.postForm("/server/backup", url.Values{})
if err != nil {
t.Fatalf("full backup: %v", err)
@@ -104,6 +114,10 @@ func testFullBackupRestore(t *testing.T, sc *scenario) {
t.Fatalf("%v\n%s", err, logTail(h.logs("selfpost"), "backup"))
}
if out, _ := h.execIn("selfpost", "stat", "-c", "%U:%G %a", "/data/postfix/e2e_extra_map"); strings.TrimSpace(out) != "panel:selfpost 640" {
t.Errorf("the extra map after the restore is %q, want panel:selfpost 640", strings.TrimSpace(out))
}
p, err := newPanelClient()
if err != nil {
t.Fatal(err)