mix/selfpost
1
0
Fork 0
Code Issues Pull Requests Actions 21 Packages Projects Releases Wiki Activity

backup: no full backup holds the Postfix queue

Browse Source 
Owner, 2026-10-11: the queue goes out of the command's archive too.

The commit that made the panel's backup work left the queue in the
archive of selfpost-backup, which runs as root and can read it. But that
copy was never the queue: the archive is made while the instance runs,
Postfix moves messages between the queue's directories all the time, and
a message on its way could be in the archive twice or not at all. One
rule now - QueueDir is left out of every archive - and
Params.Unprivileged is only about a certs/ file the panel may not open.

What is not carried over is mail still waiting for delivery; a restored
instance builds an empty queue (the e2e restore scenario starts one).
To keep the queue: stop the instance and tar the project directory, as
the guide's methods 3 and 4 already describe.

Guide: the description of a full backup, methods 1 and 2 and the
comparison table. Help follows the design commit before this one.

  go vet ./internal/backup/: ok
  go test ./internal/backup/ ./cmd/... ./internal/web/...: ok

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
mix
2026-10-11 20:06:41 +03:00
co-authored by Claude Opus 5.5
parent 82f212ab39
commit 21e961cb29
5 changed files with 52 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+12 -6
View File
@@ -16,6 +16,12 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); version
as one parse failure and the reason is in the mail log, on the message's
own delivery line (Server › System log, `dropped: …`). Only a failure of
the database is still retried.
- **A full backup no longer holds the Postfix queue**, from the panel or from
`selfpost-backup`. It is taken while the instance runs, and a copy of a
queue Postfix is working on can hold a message twice or not at all. Mail
still waiting for delivery is not carried over; a restored instance starts
with an empty queue. To keep the queue, stop the instance and `tar` the
project directory.
- DMARC reports are kept 500 **for each domain** (and 90 days), not 500 for
the whole server - see Security.
@@ -26,12 +32,12 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); version
includes the queue, Postfix keeps the queue readable by its own user alone,
and the panel is not that user ("open /data/postfix/queue/active:
permission denied" in the log). `selfpost-backup` run with `docker exec`
was not affected - it runs as root. The panel's archive now leaves the queue
out, and likewise a file in `certs/` it may not open (usually the private
key, named in the log); the page, Help and the guide say so, and the guide
says to put the key back before starting a restored instance. Everything is
read through before the download starts, so a failure is a message on the
page and not a file cut short.
was not affected - it runs as root. The queue is no longer part of a backup
(see Changed), and the panel's archive leaves out a file in `certs/` it may
not open (usually the private key, named in the log); the page, Help and
the guide say so, and the guide says to put the key back before starting a
restored instance. Everything is read through before the download starts,
so a failure is a message on the page and not a file cut short.
### Security
docs/guide.md
+14 -16
View File
@@ -542,10 +542,14 @@ journal-milter (level 2) is down. There is no per-IP bypass.
### Full backup and restore
**Full backup** is a self-contained project archive: `data/` (SQLite, all
domains' DKIM keys, all applications' SASL credentials, the Postfix queue,
`manifest.json` with the version that created it), plus `docker-compose.yml`,
`.env`, and `certs/` from the operator directory next to `./data`. Delivery
logs under `data/log/` are excluded. The base compose file mounts the project
domains' DKIM keys, all applications' SASL credentials, `manifest.json` with
the version that created it), plus `docker-compose.yml`, `.env`, and `certs/`
from the operator directory next to `./data`. Delivery logs under `data/log/`
are excluded, and so is the Postfix queue: the archive is taken while the
instance runs, and a copy of a queue Postfix is working on can hold a message
twice or not at all. Mail still waiting for delivery is therefore not carried
over, and a restored instance starts with an empty queue; to keep the queue,
stop the instance and use `tar` (methods 3 and 4 below). The base compose file mounts the project
directory read-only at `/selfpost-deploy` so the panel and CLI can read those
deploy files — without that mount, *Full backup* refuses with an error.
@@ -556,23 +560,17 @@ instance keeps running, what goes into the archive, and what happens on restore.
**1. Panel — while the container is running (no downtime).** *Server › Backup*
→ *Full backup*. *Encrypt with a password* is ticked; untick it for a plain
archive. The archive of the CLI below, less two things the panel may not read:
it runs as its own user, not as root. **The Postfix queue** is left out —
Postfix keeps it readable by its own user alone — so mail that was waiting for
delivery at that moment is not in the archive; the restored instance starts
with an empty queue. **A file in `certs/` the panel cannot open** is left out
too, which is usually the private key (the panel's log names it); put the key
archive. The archive of the CLI below, less one thing the panel may not read:
it runs as its own user, not as root, and **a file in `certs/` it cannot open**
is left out — usually the private key (the panel's log names it). Put the key
back into `certs/` before starting a restored instance. If anything else
cannot be read, the panel says so and downloads nothing.
**2. `selfpost-backup` CLI — while the container is running (no downtime).**
The panel button and this command call the same code path: a consistent SQLite
snapshot via `VACUUM INTO`, then a gzip tar of `data/` (minus diagnostics) plus
deploy files. `docker exec` runs the command as root, so this archive is the
complete one: it has the Postfix queue and everything in `certs/`. The queue is
copied while Postfix works on it, so a message on its way through may be in the
archive twice or not at all; stop the service first (method 3) if the queue
matters.
deploy files. `docker exec` runs the command as root, so this archive has
everything in `certs/`, the private key included.
```sh
docker exec <container> selfpost-backup > selfpost-backup.tar.gz
@@ -627,7 +625,7 @@ methods avoid it by stopping first.
| Self-contained archive | Yes | Yes | No — deploy files separate |
| SQLite consistency | `VACUUM INTO` snapshot | Safe when stopped | Safe when stopped |
| `data/log/` | Excluded | Included (optional exclude) | Included (optional exclude) |
| Postfix queue | CLI: yes, copied live. Panel: no | Yes | Yes |
| Postfix queue | No | Yes | Yes |
| `certs/` private key | CLI: yes. Panel: only if the panel may read it | Yes | No — deploy files separate |
| `data/manifest.json` | Written (version stamp) | Absent | Absent |
| Restore version check | Yes — must match image tag | No | No |
internal/backup/backup.go
+21 -23
View File
@@ -1,16 +1,16 @@
// Package backup implements SelfPost's full-server backup and the restore
// version guard (architecture.md § Persistence). A full backup is a
// gzip-compressed tar of the consolidated persistent state under /data — the
// SQLite database (as a consistent snapshot), the per-domain DKIM keys, the
// SASL database, and the Postfix queue — plus docker-compose.yml, .env, and
// certs/ from the operator's deploy directory, and a manifest recording the
// SelfPost version that produced it. Postfix delivery logs under log/ are
// excluded (diagnostics, not state).
// SQLite database (as a consistent snapshot), the per-domain DKIM keys and the
// SASL database — plus docker-compose.yml, .env, and certs/ from the
// operator's deploy directory, and a manifest recording the SelfPost version
// that produced it. Postfix delivery logs under log/ are excluded
// (diagnostics, not state), and so is the Postfix queue (see QueueDir).
//
// Restore is not a separate code path in the panel: a backup is extracted into
// the operator's project directory (data/, docker-compose.yml, .env, certs/)
// before first start. The archive carries everything needed to bring the
// instance back — DKIM keys, sasldb2, sender map, queue, and deploy files —
// instance back — DKIM keys, sasldb2, sender map, and deploy files —
// so the operator only adjusts hostname or proxy settings on a new host. The
// restore-specific steps the panel runs are CheckRestore, which refuses to
// boot if the manifest's version does not match the running binary so
@@ -81,22 +81,23 @@ type Params struct {
DeployRoot string
OnWarn func(string)
// Unprivileged says the archive is made by the panel, which runs as its
// own user and not as root. Two things under the roots are then not its to
// read, and are left out instead of failing the whole archive:
//
// - the Postfix queue (QueueDir). Postfix keeps every queue file and
// directory readable by its own user alone, and that cannot be
// configured. What is lost is mail not yet delivered at that moment;
// a start without a queue directory builds an empty one.
// - a file under certs/ the panel may not open, which is usually the
// private key (root and Postfix's group only). OnWarn names it.
//
// The selfpost-backup command, run as root, leaves this unset and takes
// both.
// own user and not as root. A file under certs/ it may not open - usually
// the private key, root and Postfix's group only - is then left out
// instead of failing the whole archive, and OnWarn names it. The
// selfpost-backup command, run as root, leaves this unset and takes it.
Unprivileged bool
}
// QueueDir is the Postfix queue, relative to the data directory.
// QueueDir is the Postfix queue, relative to the data directory. No backup
// holds it (owner, 2026-10-11). An archive is made while the instance runs,
// and Postfix moves messages between the queue's directories all the time: a
// copy taken then can hold a message twice or not at all, so what it would
// bring back is not the queue. Nor could the panel read it - Postfix keeps
// every queue file readable by its own user alone, which is why the panel's
// backup failed outright from 1.3.0 to 2.0.0. What is not carried over is mail
// still waiting for delivery; a start without a queue directory builds an
// empty one. Whoever needs the queue stops the instance and archives the
// project directory (guide § Taking a full backup).
const QueueDir = "postfix/queue"
// excludedFromArchive lists the data-directory entries a backup never carries.
@@ -203,11 +204,8 @@ func validateDeployRoot(deployRoot string) error {
// left out of the archive. A directory left out is not descended into.
type skipFunc func(path, rel string, d fs.DirEntry) bool
// skipData leaves the Postfix queue out of an unprivileged archive.
// skipData leaves the Postfix queue out of every archive.
func (p Params) skipData() skipFunc {
if !p.Unprivileged {
return nil
}
return func(_, rel string, _ fs.DirEntry) bool {
return filepath.ToSlash(rel) == QueueDir
}
internal/backup/backup_test.go
+3 -3
View File
@@ -112,7 +112,6 @@ func TestCreateIncludesStateExcludesTransient(t *testing.T) {
DataArchivePrefix + "opendkim/keys/example.com/selfpost.private",
DataArchivePrefix + "sasl/sasldb2",
DataArchivePrefix + "postfix/sender_login_maps",
DataArchivePrefix + "postfix/queue/deferred/sample",
ComposeFileName,
EnvFileName,
CertsDirName + "/fullchain.pem",
@@ -129,6 +128,7 @@ func TestCreateIncludesStateExcludesTransient(t *testing.T) {
DataArchivePrefix + "selfpost.db-shm",
DataArchivePrefix + "log/mail.log",
DataArchivePrefix + "log/mail.log.1",
DataArchivePrefix + "postfix/queue/deferred/sample",
} {
if _, ok := files[name]; ok {
t.Errorf("archive should not contain %s", name)
@@ -236,8 +236,8 @@ func TestCheckRestoreWrongFormatRejected(t *testing.T) {
}
}
// The panel's archive: no Postfix queue, everything else as before, and Check
// agrees that it can be made.
// The panel's archive: like any other it has no Postfix queue, the rest is
// there, and Check agrees that it can be made.
func TestCreateUnprivilegedLeavesTheQueueOut(t *testing.T) {
dataDir, dbPath, deployRoot := seedDataDir(t)
p := Params{DataDir: dataDir, DBPath: dbPath, Version: "1.2.3", DeployRoot: deployRoot, Unprivileged: true}
internal/web/view/help.go
+2 -2
View File
@@ -190,8 +190,8 @@ var serverTopics = []HelpTopic{
{ID: HelpBackup, Title: "Backup and moving a domain", Body: []Text{
Rich("Full backup downloads one archive of the whole instance: ", Code("data/"), " (the database, DKIM keys and application credentials), ",
Code("docker-compose.yml"), ", ", Code(".env"), " and ", Code("certs/"),
". The panel cannot read the Postfix queue, nor usually the private key in ", Code("certs/"),
", and leaves them out; ", Code("selfpost-backup"), " run in the container takes both. It holds secrets, so leave Encrypt with a password ticked: the file is sealed as ", Code(".spbk"),
". The Postfix queue is not in it: mail still waiting for delivery is not carried over. Nor, usually, is the private key in ", Code("certs/"),
", which the panel cannot read; ", Code("selfpost-backup"), " run in the container takes the key. It holds secrets, so leave Encrypt with a password ticked: the file is sealed as ", Code(".spbk"),
", the password is not stored, and without it nothing can be recovered. To restore, unpack it into an empty project directory (decrypt a ",
Code(".spbk"), " first with ", Code("selfpost-backup -decrypt"),
") and start the same SelfPost version that made it; the panel refuses data from another version. The reverse-proxy configuration and ",