From 21e961cb29c939fb1432865502faf2d689cab36b Mon Sep 17 00:00:00 2001 From: Mix Date: Sun, 11 Oct 2026 20:06:41 +0300 Subject: [PATCH] backup: no full backup holds the Postfix queue Owner, 2026-10-11: the queue goes out of the command's archive too. The commit that made the panel's backup work left the queue in the archive of selfpost-backup, which runs as root and can read it. But that copy was never the queue: the archive is made while the instance runs, Postfix moves messages between the queue's directories all the time, and a message on its way could be in the archive twice or not at all. One rule now - QueueDir is left out of every archive - and Params.Unprivileged is only about a certs/ file the panel may not open. What is not carried over is mail still waiting for delivery; a restored instance builds an empty queue (the e2e restore scenario starts one). To keep the queue: stop the instance and tar the project directory, as the guide's methods 3 and 4 already describe. Guide: the description of a full backup, methods 1 and 2 and the comparison table. Help follows the design commit before this one. go vet ./internal/backup/: ok go test ./internal/backup/ ./cmd/... ./internal/web/...: ok Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 18 +++++++++----- docs/guide.md | 30 +++++++++++------------ internal/backup/backup.go | 44 ++++++++++++++++------------------ internal/backup/backup_test.go | 6 ++--- internal/web/view/help.go | 4 ++-- 5 files changed, 52 insertions(+), 50 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 844ece2..7222e43 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,12 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); version as one parse failure and the reason is in the mail log, on the message's own delivery line (Server › System log, `dropped: …`). Only a failure of the database is still retried. +- **A full backup no longer holds the Postfix queue**, from the panel or from + `selfpost-backup`. It is taken while the instance runs, and a copy of a + queue Postfix is working on can hold a message twice or not at all. Mail + still waiting for delivery is not carried over; a restored instance starts + with an empty queue. To keep the queue, stop the instance and `tar` the + project directory. - DMARC reports are kept 500 **for each domain** (and 90 days), not 500 for the whole server - see Security. @@ -26,12 +32,12 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); version includes the queue, Postfix keeps the queue readable by its own user alone, and the panel is not that user ("open /data/postfix/queue/active: permission denied" in the log). `selfpost-backup` run with `docker exec` - was not affected - it runs as root. The panel's archive now leaves the queue - out, and likewise a file in `certs/` it may not open (usually the private - key, named in the log); the page, Help and the guide say so, and the guide - says to put the key back before starting a restored instance. Everything is - read through before the download starts, so a failure is a message on the - page and not a file cut short. + was not affected - it runs as root. The queue is no longer part of a backup + (see Changed), and the panel's archive leaves out a file in `certs/` it may + not open (usually the private key, named in the log); the page, Help and + the guide say so, and the guide says to put the key back before starting a + restored instance. Everything is read through before the download starts, + so a failure is a message on the page and not a file cut short. ### Security diff --git a/docs/guide.md b/docs/guide.md index 6cda3d2..b0cb6ae 100644 --- a/docs/guide.md +++ b/docs/guide.md @@ -542,10 +542,14 @@ journal-milter (level 2) is down. There is no per-IP bypass. ### Full backup and restore **Full backup** is a self-contained project archive: `data/` (SQLite, all -domains' DKIM keys, all applications' SASL credentials, the Postfix queue, -`manifest.json` with the version that created it), plus `docker-compose.yml`, -`.env`, and `certs/` from the operator directory next to `./data`. Delivery -logs under `data/log/` are excluded. The base compose file mounts the project +domains' DKIM keys, all applications' SASL credentials, `manifest.json` with +the version that created it), plus `docker-compose.yml`, `.env`, and `certs/` +from the operator directory next to `./data`. Delivery logs under `data/log/` +are excluded, and so is the Postfix queue: the archive is taken while the +instance runs, and a copy of a queue Postfix is working on can hold a message +twice or not at all. Mail still waiting for delivery is therefore not carried +over, and a restored instance starts with an empty queue; to keep the queue, +stop the instance and use `tar` (methods 3 and 4 below). The base compose file mounts the project directory read-only at `/selfpost-deploy` so the panel and CLI can read those deploy files — without that mount, *Full backup* refuses with an error. @@ -556,23 +560,17 @@ instance keeps running, what goes into the archive, and what happens on restore. **1. Panel — while the container is running (no downtime).** *Server › Backup* → *Full backup*. *Encrypt with a password* is ticked; untick it for a plain -archive. The archive of the CLI below, less two things the panel may not read: -it runs as its own user, not as root. **The Postfix queue** is left out — -Postfix keeps it readable by its own user alone — so mail that was waiting for -delivery at that moment is not in the archive; the restored instance starts -with an empty queue. **A file in `certs/` the panel cannot open** is left out -too, which is usually the private key (the panel's log names it); put the key +archive. The archive of the CLI below, less one thing the panel may not read: +it runs as its own user, not as root, and **a file in `certs/` it cannot open** +is left out — usually the private key (the panel's log names it). Put the key back into `certs/` before starting a restored instance. If anything else cannot be read, the panel says so and downloads nothing. **2. `selfpost-backup` CLI — while the container is running (no downtime).** The panel button and this command call the same code path: a consistent SQLite snapshot via `VACUUM INTO`, then a gzip tar of `data/` (minus diagnostics) plus -deploy files. `docker exec` runs the command as root, so this archive is the -complete one: it has the Postfix queue and everything in `certs/`. The queue is -copied while Postfix works on it, so a message on its way through may be in the -archive twice or not at all; stop the service first (method 3) if the queue -matters. +deploy files. `docker exec` runs the command as root, so this archive has +everything in `certs/`, the private key included. ```sh docker exec selfpost-backup > selfpost-backup.tar.gz @@ -627,7 +625,7 @@ methods avoid it by stopping first. | Self-contained archive | Yes | Yes | No — deploy files separate | | SQLite consistency | `VACUUM INTO` snapshot | Safe when stopped | Safe when stopped | | `data/log/` | Excluded | Included (optional exclude) | Included (optional exclude) | -| Postfix queue | CLI: yes, copied live. Panel: no | Yes | Yes | +| Postfix queue | No | Yes | Yes | | `certs/` private key | CLI: yes. Panel: only if the panel may read it | Yes | No — deploy files separate | | `data/manifest.json` | Written (version stamp) | Absent | Absent | | Restore version check | Yes — must match image tag | No | No | diff --git a/internal/backup/backup.go b/internal/backup/backup.go index 2be15d3..307c5ed 100644 --- a/internal/backup/backup.go +++ b/internal/backup/backup.go @@ -1,16 +1,16 @@ // Package backup implements SelfPost's full-server backup and the restore // version guard (architecture.md § Persistence). A full backup is a // gzip-compressed tar of the consolidated persistent state under /data — the -// SQLite database (as a consistent snapshot), the per-domain DKIM keys, the -// SASL database, and the Postfix queue — plus docker-compose.yml, .env, and -// certs/ from the operator's deploy directory, and a manifest recording the -// SelfPost version that produced it. Postfix delivery logs under log/ are -// excluded (diagnostics, not state). +// SQLite database (as a consistent snapshot), the per-domain DKIM keys and the +// SASL database — plus docker-compose.yml, .env, and certs/ from the +// operator's deploy directory, and a manifest recording the SelfPost version +// that produced it. Postfix delivery logs under log/ are excluded +// (diagnostics, not state), and so is the Postfix queue (see QueueDir). // // Restore is not a separate code path in the panel: a backup is extracted into // the operator's project directory (data/, docker-compose.yml, .env, certs/) // before first start. The archive carries everything needed to bring the -// instance back — DKIM keys, sasldb2, sender map, queue, and deploy files — +// instance back — DKIM keys, sasldb2, sender map, and deploy files — // so the operator only adjusts hostname or proxy settings on a new host. The // restore-specific steps the panel runs are CheckRestore, which refuses to // boot if the manifest's version does not match the running binary so @@ -81,22 +81,23 @@ type Params struct { DeployRoot string OnWarn func(string) // Unprivileged says the archive is made by the panel, which runs as its - // own user and not as root. Two things under the roots are then not its to - // read, and are left out instead of failing the whole archive: - // - // - the Postfix queue (QueueDir). Postfix keeps every queue file and - // directory readable by its own user alone, and that cannot be - // configured. What is lost is mail not yet delivered at that moment; - // a start without a queue directory builds an empty one. - // - a file under certs/ the panel may not open, which is usually the - // private key (root and Postfix's group only). OnWarn names it. - // - // The selfpost-backup command, run as root, leaves this unset and takes - // both. + // own user and not as root. A file under certs/ it may not open - usually + // the private key, root and Postfix's group only - is then left out + // instead of failing the whole archive, and OnWarn names it. The + // selfpost-backup command, run as root, leaves this unset and takes it. Unprivileged bool } -// QueueDir is the Postfix queue, relative to the data directory. +// QueueDir is the Postfix queue, relative to the data directory. No backup +// holds it (owner, 2026-10-11). An archive is made while the instance runs, +// and Postfix moves messages between the queue's directories all the time: a +// copy taken then can hold a message twice or not at all, so what it would +// bring back is not the queue. Nor could the panel read it - Postfix keeps +// every queue file readable by its own user alone, which is why the panel's +// backup failed outright from 1.3.0 to 2.0.0. What is not carried over is mail +// still waiting for delivery; a start without a queue directory builds an +// empty one. Whoever needs the queue stops the instance and archives the +// project directory (guide § Taking a full backup). const QueueDir = "postfix/queue" // excludedFromArchive lists the data-directory entries a backup never carries. @@ -203,11 +204,8 @@ func validateDeployRoot(deployRoot string) error { // left out of the archive. A directory left out is not descended into. type skipFunc func(path, rel string, d fs.DirEntry) bool -// skipData leaves the Postfix queue out of an unprivileged archive. +// skipData leaves the Postfix queue out of every archive. func (p Params) skipData() skipFunc { - if !p.Unprivileged { - return nil - } return func(_, rel string, _ fs.DirEntry) bool { return filepath.ToSlash(rel) == QueueDir } diff --git a/internal/backup/backup_test.go b/internal/backup/backup_test.go index 75a3281..01d6ab3 100644 --- a/internal/backup/backup_test.go +++ b/internal/backup/backup_test.go @@ -112,7 +112,6 @@ func TestCreateIncludesStateExcludesTransient(t *testing.T) { DataArchivePrefix + "opendkim/keys/example.com/selfpost.private", DataArchivePrefix + "sasl/sasldb2", DataArchivePrefix + "postfix/sender_login_maps", - DataArchivePrefix + "postfix/queue/deferred/sample", ComposeFileName, EnvFileName, CertsDirName + "/fullchain.pem", @@ -129,6 +128,7 @@ func TestCreateIncludesStateExcludesTransient(t *testing.T) { DataArchivePrefix + "selfpost.db-shm", DataArchivePrefix + "log/mail.log", DataArchivePrefix + "log/mail.log.1", + DataArchivePrefix + "postfix/queue/deferred/sample", } { if _, ok := files[name]; ok { t.Errorf("archive should not contain %s", name) @@ -236,8 +236,8 @@ func TestCheckRestoreWrongFormatRejected(t *testing.T) { } } -// The panel's archive: no Postfix queue, everything else as before, and Check -// agrees that it can be made. +// The panel's archive: like any other it has no Postfix queue, the rest is +// there, and Check agrees that it can be made. func TestCreateUnprivilegedLeavesTheQueueOut(t *testing.T) { dataDir, dbPath, deployRoot := seedDataDir(t) p := Params{DataDir: dataDir, DBPath: dbPath, Version: "1.2.3", DeployRoot: deployRoot, Unprivileged: true} diff --git a/internal/web/view/help.go b/internal/web/view/help.go index 13b7f05..e3fec11 100644 --- a/internal/web/view/help.go +++ b/internal/web/view/help.go @@ -190,8 +190,8 @@ var serverTopics = []HelpTopic{ {ID: HelpBackup, Title: "Backup and moving a domain", Body: []Text{ Rich("Full backup downloads one archive of the whole instance: ", Code("data/"), " (the database, DKIM keys and application credentials), ", Code("docker-compose.yml"), ", ", Code(".env"), " and ", Code("certs/"), - ". The panel cannot read the Postfix queue, nor usually the private key in ", Code("certs/"), - ", and leaves them out; ", Code("selfpost-backup"), " run in the container takes both. It holds secrets, so leave Encrypt with a password ticked: the file is sealed as ", Code(".spbk"), + ". The Postfix queue is not in it: mail still waiting for delivery is not carried over. Nor, usually, is the private key in ", Code("certs/"), + ", which the panel cannot read; ", Code("selfpost-backup"), " run in the container takes the key. It holds secrets, so leave Encrypt with a password ticked: the file is sealed as ", Code(".spbk"), ", the password is not stored, and without it nothing can be recovered. To restore, unpack it into an empty project directory (decrypt a ", Code(".spbk"), " first with ", Code("selfpost-backup -decrypt"), ") and start the same SelfPost version that made it; the panel refuses data from another version. The reverse-proxy configuration and ",