audit 3: a restore is ended by its Resync, and half an archive is refused
test / test (push) Canceled after 0s

The restore manifest was consumed as soon as its version was checked,
before store.Open and the Resync; a start that failed in between left a
data directory the next start took for an ordinary one (M1). openData now
runs CheckRestore, store.Open, the Resync, and only then ConsumeRestore.

A truncated archive had nothing to refuse it: the manifest is the first
entry, so a download cut short or a tar stopped by a damaged block was a
restore on half a database. Every archive now ends with data/contents.json,
the size and SHA-256 of each data/ file (deploy files are the operator's to
edit), and CheckRestore compares /data with it before the database is
opened, then renames it contents.checked.json so a start after a failed one
does not compare again unless the manifest changed.

Tests: a start whose Resync fails, then a start that resyncs and consumes;
an archive cut in the database, before the deploy files and in its last
entry (refused, database untouched, no -wal); a damaged key named; the
record's names confined to data/; the domain import of a cut plain export,
a cut and an altered .spde (refused, nothing created). e2e on the test
server: full_backup_restore now also sees contents.json in the archive and
gone after the first start, through the real entrypoint and daemons (all
pass, 762 s); go vet + go test of 24 packages pass there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
mix
2026-10-15 09:53:46 +03:00
co-authored by Claude Opus 5.5
parent 68b3a7b23f
commit 8e3241ef98
13 changed files with 785 additions and 101 deletions
+2 -1
View File
@@ -960,7 +960,8 @@ single-connection trade-off that follows from it.
| `postfix/dmarc_relay_domains` | Domains accepted for DMARC report delivery |
| `postfix/queue/` | Postfix transit mail (deferred/active); survives container recreate |
| `log/mail.log` | Postfix delivery log + rotated copies (excluded from backups) |
| `manifest.json` | Backup version stamp (consumed on restore) |
| `manifest.json` | Backup version stamp; removed by the first start after a restore once its Resync is done |
| `contents.json` | A backup's last entry: the size and SHA-256 of each file under `data/`, compared by the first start after a restore before it opens the database (renamed `contents.checked.json` once compared, removed with the manifest) |
Not in `/data`: TLS certificates for the panel (reverse-proxy mount) — though
full backups also archive the operator's `./certs` PEM files when present.
+14 -1
View File
@@ -1074,11 +1074,24 @@ those after it move up one patch number each.
### Stage 3 — persistence and restore (M1–M5, L1–L4, L7) → `2.5.3`
- [ ] **Consume the manifest after the resync succeeds** (`CheckRestore` reports;
- [x] **Consume the manifest after the resync succeeds** (`CheckRestore` reports;
`run` deletes after `resyncAfterRestore` returns nil). Test: Resync fails →
manifest kept → second boot resyncs. And a corrupted or truncated archive
is refused on restore and on import with its message, nothing half
written (a row of review-2026-08-followups, placed here 2026-10-14). — **Opus**
*Done 2026-10-15:* `cmd/panel` `openData` runs CheckRestore, `store.Open`,
the Resync and then `backup.ConsumeRestore`. A truncated archive had
nothing to refuse it - the manifest is the first entry, so half an
archive was a restore on half a database: every archive now ends with
`data/contents.json` (size and SHA-256 of each `data/` file; the deploy
files are the operator's to edit), CheckRestore compares `/data` with it
before the database is opened and renames it `contents.checked.json`, so
a start after a failed one (which had the database open) does not compare
again unless the manifest changed. Tests: the start that fails, a cut in
the database / before the deploy files / in the last entry (database
untouched, no `-wal`), a damaged key named; the domain import of a cut
plain export, a cut and an altered `.spde` (each refused, nothing
created). The CLI's `-decrypt -o` is step 3.
- [ ] **Indexes**: `idx_send_log_domain_created_at (domain_id, created_at)` —
on the id column stage 2 adds, replacing `idx_send_log_domain`;
partial `idx_send_log_queued (created_at) WHERE status = 'queued'`;
+17 -1
View File
@@ -809,7 +809,9 @@ blocked. On that same first boot the panel also runs one **Resync** — OpenDKIM
tables and Postfix's sender map are re-derived from SQLite (inbound relay maps
when `INBOUND_RELAY_ENABLE=true`; DMARC maps when `DMARC_REPORTS_ENABLE=true`)
and both daemons are reloaded, healing any drift between the extracted files and
the database. The *Reload configuration* button on Server › Health resyncs
the database. The manifest is deleted only after that Resync is done: a first
start that fails on the way leaves it, and the next start is a restore again
and runs the Resync. The *Reload configuration* button on Server › Health resyncs
OpenDKIM, the sender map, and inbound maps only — not DMARC maps. This is why the
compose file pins a fixed tag rather than `:latest`: without a known version,
there'd be no way to tell which image restoring a given backup actually requires
@@ -857,6 +859,20 @@ tag](#fixed-image-tag) if it is not published), then `docker compose up -d`
again — the manifest is still there because the failed boot never got to
delete it.
An archive unpacked **in part** — a download cut short, or a `tar` that
stopped at a damaged block (it says so and exits non-zero) — is refused at
step 5 as well, before the database is opened:
```
backup: the archive was not unpacked whole (data/contents.json, its last entry, is not there) - it was cut short or is damaged; nothing was opened. Unpack it again, from an intact copy, into an empty directory
```
The last entry of every panel and CLI archive, `data/contents.json`, records
each file under `data/` with its size and SHA-256, and the first start
compares `/data` with it; a file that differs is named in the message.
`docker-compose.yml`, `.env` and `certs/` are not compared — they are yours to
edit before that start.
**Moving to a different host** is the same flow: create an empty project
directory, unpack the backup there, edit `.env` (and `docker-compose.yml` if
needed) for the new hostname or proxy, then `docker compose up -d`. The archive
+1 -1
View File
@@ -42,7 +42,7 @@ in `git log` and [CHANGELOG.md](../CHANGELOG.md).
| from-header-binding | Hold a message's `From:` header against the application that sends it | shipped `2.4.0` | 9/9 | [plans/from-header-binding.md](plans/from-header-binding.md) |
| panel-redesign | Panel redesign (accepted mockups, design contract) | shipped `2.0.0` | 27/27 | [plans/panel-redesign.md](plans/panel-redesign.md) |
| schema-squash | Squash SQLite migrations into a 2.x baseline | built in panel-redesign (stage 1) | — | [plans/panel-redesign.md](plans/panel-redesign.md) |
| audit-2026-10-12 | Follow-ups of the 2026-10-12 full-tree code audit, seven stages | in progress (stage 0 shipped `2.4.2`, stages 1+2 `2.5.0`, 2b `2.5.1`, 2c `2.5.2`; next stage 3 → `2.5.3`) | 49/81 | [audit-2026-10-12.md](audit-2026-10-12.md) |
| audit-2026-10-12 | Follow-ups of the 2026-10-12 full-tree code audit, seven stages | in progress (stage 0 shipped `2.4.2`, stages 1+2 `2.5.0`, 2b `2.5.1`, 2c `2.5.2`; next stage 3 → `2.5.3`) | 50/81 | [audit-2026-10-12.md](audit-2026-10-12.md) |
**Recommended order** (not binding; owner, 2026-10-09: feature order follows
development convenience): **panel-redesign** shipped as `2.0.0` (2026-10-11) and