242515c305cdb30dcf383773930d37707c60a4bb
57
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  mixandClaude Opus 5.5
|
242515c305 |
audit 3: backup CLI, envelope budget, directory sync, docs; findings of the reviews
Steps 3-6 of stage 3 and what the two reviews found in steps 1-6, in one commit: the review fixes sit in the same hunks. Step 3, CLI backup (M3, M5, L1, L2; L4 of the stage's list): - backup.snapshotDB refuses a database that is not there and opens the source mode=ro (measured: mode=ro creates no file). - selfpost-backup holds an encrypting password to validate.MinSecretFilePasswordLen; -o of both directions writes a temporary file beside the name and renames it whole; -decrypt peeks the first chunk (password and type authenticated) before the output exists. - what a backup leaves out (a link under data/, a link under certs/ to a directory, a socket...) is told to OnWarn; certs/ with no file is backup.ErrNoCerts, refused by Check and Create, named by the panel. - parseTrustedProxies: a bare address takes /128 when written with a colon (::ffff:10.0.0.1/32 was ::/32). Step 4, secretfile (M4, L3): scrypt's V at most 128 MiB, r <= 64, p <= 2; bytes after the last chunk are ErrCorrupt. Step 5 (L7): atomicfile.SyncDir after the rename, used by Write, replaceSASLDB and the CLI; atomicfile_test.go. Step 6: security.md and guide on backup and restore. Reviews (technical, security): - H1: a restore of a server with DMARC on was refused on every start - the archive records no owner, tar as root left the DMARC maps root's, and the entrypoint only chowned the maps it named. It now hands every file beside queue/ in /data/postfix to panel:selfpost 0640; e2e writes an extra map before the backup and finds it panel:selfpost 640 after a root unpack. - the comparison reads regular files only (Lstat, size, LimitReader); a record unpacked again wins over the earlier check's mark; ConsumeRestore removes both; the mark is synced; a journal file beside a freshly unpacked database is refused. - AttachRefusalLine refuses an empty line as the old statement did; skipped entries named in words; -decrypt of an empty payload is no failure; guide wording on the shell's '>' and a killed CLI; the in-place restore's 'mv .' (older than this stage) fixed. Tested on the test server: go vet + go test of 25 packages, make e2e (all pass, 643 s) on this tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
508ac96c79 |
audit 3: every per-message question of the send log has its index
test / test (push) Canceled after 0s
Migration 0010 (DDL only; 0009 went to stage 2b). EXPLAIN QUERY PLAN of
every send_log statement showed, beyond M2 as written (stage 2 had already
moved the by-domain counts onto (domain_id, created_at)):
- the tailer's sweep of open rows (every 5 min) and its count scanned the
table: partial idx_send_log_open (created_at) WHERE status IN ('queued',
'deferred'), the statuses literal in the statements, ORDER BY +id (else
SQLite walks the table in id order to skip a sort);
- AttachRefusalLine's NOT EXISTS (... refusal_line = ?1) scanned the table
for every refusal line: partial idx_send_log_refusal_line, the query
carrying the index's WHERE; its row found by idx_send_log_rejected
(status literal, +queue_id) instead of every refusal without a queue id;
- an application's level-2 count went through its domain's index:
+domain_id in appByLogin;
- a page of one domain's or one application's log sorted all of it:
idx_send_log_domain_id, idx_send_log_app_login (id order; several domains
still sort, top-N). idx_send_log_domain stays for the global filter.
The statements are constants and builders (countMessagesQuery,
peakPerHourQuery, listOpenQuery, countOpenQuery, sendLogPageQuery,
attachRefusalLineQuery, limitRefusalsQuery) and
TestSendLogQueriesUseTheirIndexes explains exactly those: the index each
must use, no SCAN of send_log, no sort where none is wanted; the literal
statuses checked against the constants. It replaces the hand-written plan
check of TestLimitRefusalsSince.
Tested on the test server: go vet + go test of 24 packages, make e2e (all
pass) on this tree.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
mixandClaude Opus 5.5
|
8e3241ef98 |
audit 3: a restore is ended by its Resync, and half an archive is refused
test / test (push) Canceled after 0s
The restore manifest was consumed as soon as its version was checked, before store.Open and the Resync; a start that failed in between left a data directory the next start took for an ordinary one (M1). openData now runs CheckRestore, store.Open, the Resync, and only then ConsumeRestore. A truncated archive had nothing to refuse it: the manifest is the first entry, so a download cut short or a tar stopped by a damaged block was a restore on half a database. Every archive now ends with data/contents.json, the size and SHA-256 of each data/ file (deploy files are the operator's to edit), and CheckRestore compares /data with it before the database is opened, then renames it contents.checked.json so a start after a failed one does not compare again unless the manifest changed. Tests: a start whose Resync fails, then a start that resyncs and consumes; an archive cut in the database, before the deploy files and in its last entry (refused, database untouched, no -wal); a damaged key named; the record's names confined to data/; the domain import of a cut plain export, a cut and an altered .spde (refused, nothing created). e2e on the test server: full_backup_restore now also sees contents.json in the archive and gone after the first start, through the real entrypoint and daemons (all pass, 762 s); go vet + go test of 24 packages pass there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
68b3a7b23f |
release: 2.5.2: audit stage 2c - what SelfPost relies on, read in the sources
test / test (push) Canceled after 0s
The owner's word, 2026-10-15: "Да, режем."
Every place SelfPost relies on another program was read in that program's
source, of the version the image runs, and is cited in the code (621
entries, docs/assumptions-2026-10-14.md); what the sources contradicted is
fixed: rspamd signing a sender's own verdict, the sender map's first key,
queue expiry and Postfix's time values, refusal and recipient lines as
Postfix writes them, the rotated log read until postlogd lets it go,
inbound mail over 10 MB, local delivery to an address literal, supervision
(Postfix stopped, give-ups FATAL, the reload's exit read), sasldb2 through
a copy, Health dialling sockets and showing the certificate Postfix
serves, Referrer-Policy same-origin, migrations under BEGIN IMMEDIATE,
certbot's links followed, about forty sentences of the documents, and ten
measurements on the stand. The sender check reads From: with OpenDKIM's
own parser, ported and checked against the C and against OpenDKIM's
signing; L8 is closed; an address OpenDKIM would cut short is refused.
Plus the findings of both reviews of the stage (
|
||
|
mixandClaude Opus 5.5
|
ec08c8606b |
audit 2c: the From: parser port checked against OpenDKIM's signing itself
test / test (push) Canceled after 0s
The last check of the port, and the plan's last line of stage 2c. Through
Postfix the sender check refuses most lines before OpenDKIM sees them, so
TestPortAgainstOpenDKIM (internal/milter) talks to OpenDKIM directly:
test/upstream/opendkim-probe.sh starts OpenDKIM from the SelfPost image in
a throwaway container, with keys and a refile SigningTable for three
names, and the test plays it each From: line through go-milter's client,
as a client OpenDKIM trusts (127.0.0.1, InternalHosts' default). For each
line the d= of the signature OpenDKIM adds must be the domain the port
predicts - its address, the domain cut to DKIM_MAXHOSTNAMELEN, the whole
cut to MAXADDRESS, matched against "^.*@<domain>$" case-insensitive as
dkimf_apply_signtable() matches - or neither signs.
Run on the test server, 2026-10-15 (OpenDKIM Filter v2.11.0 from the
image): 385 lines - both shared files and lines composed from display
names plain, quoted, escaped and encoded, comments nested and open,
groups, lists, brackets, source routes, domain literals and quoted local
parts, with the other sender's name and the server's in each place - 385
agree, 287 signed. The two lines padded so that, cut at 256 bytes, they
end in "@other.e2e.test" are signed by OpenDKIM with other.e2e.test's key:
the danger the sender check's 256-byte rule (
|
||
|
mixandClaude Opus 5.5
|
3584adcbc2 |
sender check: From: read with OpenDKIM's own parser; L8 closed
test / test (push) Canceled after 0s
Audit 2026-10-12, stage 2c, the owner's decision of 2026-10-14: the check
decides by the address OpenDKIM signs for, whatever a display name or a
comment holds.
internal/milter/mailparse.go ports libopendkim's dkim_mail_parse() with
dkim_mail_first_special(), dkim_mail_token() and
dkim_mail_matching_paren() from Debian's opendkim 2.11.0~beta2-8+deb12u1,
line for line: the '<' form, group and list punctuation, an '@' before the
first ':', ';' or ',' deciding where the address starts, source routes
dropped, quoted strings and domain literals kept with their delimiters,
comments skipped, whitespace between atoms one space, no encoded word
decoded; and the C's own quirks ("@d" gives d as the local part too).
testdata/mailparse_harness.c compiles the C from the image's source; the
test feeds it and the port 200 019 lines (the hand-written cases and
200 000 random ones over the specials) - 93 443 parse and agree byte for
byte, 106 576 fail and agree on the error (on an error the C leaves its
in-place result unterminated, and OpenDKIM uses neither part then).
judge(): one From: field and one address stay required (moreAfter: only
list or group punctuation may follow); what OpenDKIM cannot read is
refused (mlfi_eoh() would not sign it); the address is lower-cased in
ASCII only, as dkimf_lowercase() and Postfix's tables do. Found reading
dkimf_apply_signtable(): OpenDKIM matches its refile SigningTable against
"user@domain" printed into MAXADDRESS + 1 = 257 bytes, so a longer address
is matched cut short - a quoted local part padded to end there in
"@another.tenant" would be signed with that tenant's key while its real
domain is the application's. Such an address (over 256 bytes) is refused;
one that fits is matched whole. The display-name and comment scan, the
"one @ outside quotes" rule and the SigningTable reader (Signers) go; the
check asks for header values with their leading space as OpenDKIM does
(OptHeaderLeadingSpace; Postfix keeps a v2 milter's event mask, milter8.c
milter8_negotiate()).
L8 closed: refused or passed no longer depends on other domains of the
server; security.md's accepted risk is struck. The shared other-sender
lines are now held to "never signed for the other sender": unit test (a
passed line is read at the application's domain) and e2e (a passed line
arrives signed for the application's domain, none for the other).
Documents: architecture "How the header is read", security's sender
check paragraph, the guide; the help topic per
|
||
|
mixandClaude Opus 5.5
|
cad1182150 |
audit 2c: what only the stand could answer, measured and written down
test / test (push) Canceled after 0s
All 2026-10-15; each written beside the code or text it settles. - A-43: /usr/sbin/postlog is -rwxr-xr-x root:root in the image (postfix 3.7.11-0+deb12u1; dpkg-statoverride lists postdrop and postqueue only): Debian does not install it set-gid, so it is no use to dmarc-ingest (cmd/panel/dmarc_ingest.go). - C-X1: OpenSSL 3.0.22 in the image prints the same public key from `x509 -pubkey` and `pkey -pubout` for RSA-2048, P-256, P-384 and a SEC1 P-256 key - cert-reload's same_pair holds for certbot's ECDSA keys. - C-K5, D-guide:933/:233: on the live server (Docker 29.6.1, userland proxy on 0.0.0.0 and ::, no IPv6 on the network) an outside IPv4 client and the host's own public IPv4 address are logged as themselves; the host's IPv6 address and 127.0.0.1 arrive as 172.18.0.1. Over IPv6 every client is the gateway: one level-1 budget, an application IP list cannot name them, rspamd sees the gateway. Guide paragraph under level 1, notes at the milter and allow-list, comments at the anvil setup and authip.go, and a Security entry in the CHANGELOG (an allow-list naming the gateway would admit every IPv6 client). - D-guide:61: a published port nothing listens on (a throwaway container, -p 20999:25) is refused from outside over IPv4 and from the host's public IPv4, accepted and closed through 127.0.0.1 - Docker's proxy, which IPv6 goes through too (not probed over IPv6 for this). - C-M8: the live installation has received one DMARC report, of one record: nothing to set maxReportRecords by yet - said at the constant. - D-S7, D-arch:516, D-arch:532: a new database filled through the store to 120 days at 1 900 messages a day (228 000 messages, 1.37 million lines, 464 MB) on the test server (2 cores): the longest write is the line prune after a 30-day gap, 1.9 s for 342 000 lines (0.4 s with nothing to delete); the send-log delete 0.6 s; 2 000 lines of a tailer pass 18 ms; VACUUM INTO 1.7 s; a second connection writing every 50 ms waited 1.8 s at most, no SQLITE_BUSY - inside busy_timeout's 5 s. architecture.md's "about a second per million lines" was five times too fast; its "six to ten lines of 200 bytes" is 6.0 lines of 146 bytes on the live data. - D-B16: which favicon is fetched, by the probe server's log: Edge 155 the SVG, the desktop app's embedded Chromium 152 the PNG, with or without sizes="any" - layout.html's "vector wins where supported, raster catches the rest" holds; Firefox and Safari not measured. - D-arch:121: the allow/deny trial again, in rspamd/rspamd:3.11.1-3 with today's multimap.conf: every mix of allow and deny entries matching one message ended reject; a changed map was used about 10 s later. - D-sec:424: rspamd runs as _rspamd, uid 11333; the panel is uid 999. Stand: go vet and go test ./... pass on the server (24 packages); make e2e exit 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
fbd0acd035 |
audit 2c: the rest the sources contradict, and the documents
test / test (push) Canceled after 0s
Code and config: - C-M7: a DMARC report's explicit <pct>0</pct> stays 0; only an absent pct takes RFC 7489 § 6.3's default 100 (Pct *int). - C-D1: docs/development.md now tables the Debian versions the citations were read at (postfix 3.7.11-0+deb12u1, opendkim and libopendkim11 2.11.0~beta2-8+deb12u1, libmilter1.0.1 8.17.1.9-2+deb12u2, libsasl2-2 and sasl2-bin 2.1.28+dfsg-10, supervisor 4.2.5-1, logrotate 3.21.0-1 - read off the image with dpkg-query, 2026-10-15); the e2e stand's new checkCitedPackageVersions fails when the image it built differs. - C-E2: the e2e zone's SOA serial only grows (CoreDNS v1.14.6's file plugin reloads on a grown serial, not on mtime); Corefile comment says so; TestZoneSerialGrows. - C-K8, C-K9: the e2e compose and stage comments say how --project-directory and .env resolve. - D-arch:154: postfix-config.sh's "Postfix insists" comment says what TLS_README advises and that smtpd reads the key as root. - B-G7: inflight.go's reservationTTL comment: a dropped client does give ABORT (smtpd_proto() -> mail_reset() -> milter_abort()); the TTL is for smtpd dying or the milter connection breaking. - D-guide:301: the nginx fragment's first-certificate command now reaches certbot (--entrypoint certbot, --standalone, port 80 free), and the renewal loop passes --webroot -w /var/www/certbot - a standalone certificate would otherwise renew standalone while nginx holds port 80 (certbot docs, "Renewing certificates": the issuing plugin is used "unless you specify other plugins or options", and options of a successful renewal are saved). The register's proposal missed that. Documents: every D-arch, D-guide and D-sec claim the register's Answers E found false or partly true, rewritten - by three Sonnet agents, one per document, then reviewed: two proposals were obsolete after this stage's fixes and not used (a bounce's From: line - extract_from = "smtp" now; "Origin: null" - Referrer-Policy is same-origin now); a stand observation the agent had dated from a commit is now tied to the e2e test that checks it; the SQLite writers paragraph describes _txlock=immediate; the log tailer's rotation paragraph describes the minute's grace; the guide's nginx steps were put in working order (links and modes before `up`, no reconfigure step). "Deferred" is kept for Postfix's queue in the guide too. guide.md's CRLF from the agent normalised. Stand (2026-10-15, make e2e, exit 0, the version gate and zone serial test included); go vet and go test ./... pass on the server (24 packages). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
2e1544dac5 |
certbot: certs/ links into /etc/letsencrypt, mounted at the same path; the backup follows them
test / test (push) Canceled after 0s
C-A1, C-Q4, D-guide:258, :291, :722. certbot's live/<host>/*.pem are relative links into ../../archive/<host>/ (eff-certbot docs, "Where are my certificates?": point a server at them "or create symlinks"). The guide had `ln -s /etc/letsencrypt/live/<host> certs` and the nginx fragment mounted live/<host> alone: inside the container the links led into a directory it did not have - postfix-config.sh's [ -f ] false, no copy, Postfix without a certificate - and the panel's backup, reading certs/ through /selfpost-deploy, skipped links silently. Now certs/fullchain.pem and certs/privkey.pem are links to /etc/letsencrypt/live/<host>/, and /etc/letsencrypt is mounted read-only at the same path: a commented line in deploy/docker-compose.yml for the Apache scenario, mounted in the nginx fragment (./nginx/certbot-etc). The guide's steps add certbot's own commands for a server that is not root - chmod 0755 live archive, chgrp 998 + chmod 0640 the key, which certbot keeps on renewal - so the panel reads them (Health, backup). Postfix's side follows links already ([ -f ], cp, cmp, openssl). The vhost comment says the same. The backup table: links are followed by the panel's backup; a stopped tar stores them as links unless -h (GNU tar). internal/backup: addTree follows, for certs/ only, a link to a regular file and archives that file under the link's name; a link to nothing reachable is an error naming it and the mount. Tests: TestCertsLinksAreFollowed (certs/ -> live/ -> archive/, the archive holds the files; the tree gone, Check names the dangling link; Linux). Stand (2026-10-15, make e2e, exit 0): the stand mounts ./letsencrypt at /etc/letsencrypt; new certbot_layout_followed puts a new pair into letsencrypt/archive/, relative links in live/, absolute links in certs/, recreates the container: 465 serves the new serial, Health names no renewal waiting, and the full backup holds certs/*.pem as regular files with the archive's bytes. go vet and go test ./... pass on the server (24 packages); the Linux-only tests of this session run there and pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
3c63ecc1ad |
store: migrations under BEGIN IMMEDIATE, the version read again inside
test / test (push) Canceled after 0s
D-S3. A deferred BEGIN starts nothing until the first access (SQLite, lang_transaction.html § 2.2), and migrate() read user_version outside the transaction: after an upgrade the panel and dmarc-ingest, opening the file at once, could both read the old version, both begin, and the second applied the next migration again on the migrated file - "already exists", that Open failing (nothing damaged: its transaction rolled back). The DSN now has _txlock=immediate (modernc.org/sqlite v1.60.1, sqlite.go applyQueryParams() sets beginMode, tx.go newTx() issues "begin immediate" for any transaction not read-only), so a migration holds the write lock from its BEGIN, and migrate() reads user_version again inside it and skips one another process applied meanwhile. Test TestTwoOpensMigrateOnce: a raw connection holds BEGIN IMMEDIATE on a new file while two Opens read version 0 and queue on their first BEGIN; released, both succeed and user_version is the number of migrations. Without the fix it fails as the register said: "apply migration 0001_init.sql: SQL logic error: table settings already exists (1)". Stand (2026-10-15): go vet and go test ./... pass on the server (24 packages); make e2e exit 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
9ca50d33fd |
web: Referrer-Policy same-origin; a password shown once emptied on pagehide
test / test (push) Canceled after 0s
D-B3. Fetch Standard § 3.2 "append a request `Origin` header": under Referrer-Policy no-referrer a request that is not GET/HEAD and not CORS - a form POST, a navigation - gets Origin "null". originAllowed() refuses "null" whenever Sec-Fetch-Site is absent, and Fetch Metadata § 3 sends Sec-Fetch-Site only to a potentially trustworthy URL: every form, sign-in included, failed in Safari before 16.4 and in any browser on a plain-HTTP panel at a name. same-origin nulls Origin only for a request not same origin with its URL and still sends no referrer to another site. The comments of the origin check say so. D-B13. noStore keeps the page out of HTTP caches (RFC 9111 § 5.2.2.5); a history mechanism "might honor" it (§ 6), and Pragma in a response "was never specified" (§ 5.4) - it goes. CopyField.Secret marks the password of out-app-created data-secret, and panel.js empties such fields on pagehide (HTML § 7.5.9, "unload a Document", step 10.2: fired whether or not the document is kept for history traversal). Measured 2026-10-15 in Chrome 152 (the desktop app's browser), a local probe page with Cache-Control: no-store and a data-secret field: Back reloads it (pageshow persisted=false, a new load), so this browser keeps no-store pages out of its back/forward cache; the handler, fired on the same page, empties the field. Firefox and Safari not measured. Tests: the header value; the created page has the password data-secret and the login not, no Pragma. Stand (2026-10-15, make e2e, exit 0): new form_post_with_the_browsers_origin - Referrer-Policy: same-origin on a page; a POST carrying the page's own Origin and no Sec-Fetch-Site, what a browser sends now, passes; one carrying Origin: null, what it sent before, is 403. (A first run failed where the e2e client and a view test scraped the old input markup; both read data-secret now.) go vet and go test ./... pass on the server (24 packages). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
98e09fef4a |
health: sockets dialled, the certificate Postfix serves, "deferred" only for the queue
test / test (push) Canceled after 0s
Audit 2026-10-12, stage 2c, "Health says what is served"; the pages of
ea8b6bf's mockup.
Sockets (A-38). A unix socket's file outlives a killed OpenDKIM or panel
and refuses every connect() (unix(7)); Health said "Listening" from a
stat. CheckSocket and CheckMilter's unix case now dial: refused is
"Nothing listens on the ... socket: the program that made it has
stopped", EACCES is unknown (the panel may not connect), anything else
"does not answer". The probe closes before a byte, which no milter logs:
libmilter takes the EOF as SMFIC_EOF without a line (sendmail
8.17.1.9-2+deb12u2, the image's libmilter1.0.1, libmilter/comm.c
mi_rd_cmd(), engine.c mi_engine() - its source is now in dev/upstream),
OpenDKIM's mlfi_close() has no connection to free, go-milter v0.4.1
returns quietly on io.EOF (session.go, HandleMilterCommands()). The page
polls every 5 s while in view, so this mattered.
Certificate (A-65). Postfix serves build/postfix-config.sh's copy,
refreshed by cert-reload every TLS_RELOAD_INTERVAL_SECONDS. The panel now
reads smtpd_tls_cert_file from postconf -h at its start (as the retry
policy) and TLS_RELOAD_INTERVAL_SECONDS; CheckServedCertificate checks the
served copy and, when the mounted file differs byte for byte (cert-reload's
own cmp -s), names it as Renewal: its expiry and mtime, two facts on the
card. Waiting is fine for the interval: a nearing expiry of the served one
is then no warning (an expiry still an error); past interval + 1 h it is a
warning pointing at cert-reload's lines. The notifier sends no expiry
notice while a renewal waits on time.
Wording (D-P1, D-P3, and the same for the spam filter): Sender check
note, help topics sender-check, upstream, filter, certificate - word for
word as in the mockup; the filter row's detail too.
Tests: stale socket file (Linux) refused for CheckSocket and CheckMilter;
CheckServedCertificate (same, waiting, late, expired + waiting, no served
path); intervalText; the card's facts; the notifier quiet while a renewal
waits, speaking once it is late. Stand (2026-10-15, make e2e, exit 0):
new health_shows_what_is_served - the three sockets dialled ok by the
panel's user; a certificate mounted without its key is refused by
cert-reload ("do not belong together") and the card shows the served
expiry, "Renewal expires" with the mounted one's, "Renewal mounted", and
"Postfix still serves the certificate above."; a matching pair is then
served again. go vet and go test ./... pass on the server (24 packages).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
mixandClaude Opus 5.5
|
d20009c0f0 |
sasldb2 changed through a copy renamed over it, never in place
test / test (push) Canceled after 0s
Cyrus SASL 2.1.28 opens sasldb2 with no Berkeley DB environment, so with
no locks (sasldb/db_berkeley.c, berkeleydb_open(): db_create(mbdb, NULL,
0)); saslpasswd2 puts and deletes in the file itself (_sasldb_putdata()),
while smtpd opens and closes it at every AUTH (_sasldb_getdata()) and the
backup copies it whenever it runs. Either could read a half-written page:
a spurious 535, or a torn sasldb2 in a backup whose restore breaks every
application's AUTH (assumptions C-S3, C-Q3).
SASLDB.Set and Delete now run saslpasswd2 on sasldb2.new - a byte copy of
the database, or no file when there is none yet (saslpasswd2 creates it,
DB_CREATE) - then chmod 0640, fsync, rename over sasldb2; a failed run
removes the copy and leaves the database; a copy a killed panel left is
removed first. A mutex keeps two changes of the panel from copying the
same old file. The backup needs no lock after all (the plan said one): it
reads each file through one descriptor, and a rename leaves that file
whole; the walk now leaves out a file that vanished between listing and
open (a writer's temporary file), while the SQLite snapshot's absence
stays an error. CHANGELOG also gains the entries of
|
||
|
mixandClaude Opus 5.5
|
ee37d16dc7 |
audit: stage 2c - seven lines built, ticked
|
||
|
mixandClaude Opus 5.5
|
a658da11f7 |
audit: stage 2c - the answers, and what they make to fix
test / test (push) Canceled after 0s
docs/assumptions-2026-10-14.md gets its second half: every entry of the inventory settled in the source of the version the image runs, with the citation now beside it in the code. 559 verdicts (the documents' claims that repeat one another answered once): 415 hold, 77 hold in part, 55 do not hold, 12 need a measurement on the stand. The plan's heuristics line is done; what the sources contradict becomes fourteen lines of stage 2c, worst first: rspamd signs a forged verdict for a client it counts as local; the sender check takes any key of the sender map where Postfix takes the first; expiry, the forced-expire flag and Postfix's time units; refusals and recipients matched by what Postfix really writes; postlogd after a rename; inbound mail over 10 MB refused by cleanup; master outside supervisord's reach; sasldb written in place; Health showing a socket file and a certificate Postfix does not serve; the Origin check refusing every POST where Sec-Fetch-Site is missing; two processes migrating at once; certbot's symlinks; the documents; and the measurements. Roadmap 31/81. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
0f984ff8d8 |
development: code relying on another component cites its source; stage 2c: the heuristics review
test / test (push) Canceled after 0s
Owner, 2026-10-14: "нам нужно полное ревью кода на предмет всей этой эвристики. Её не должно там, где можно посмотреть код и ответить точно и однозначно", and "в коде таких наших блоков должно быть обоснование со ссылкой на код компонента". - development.md § Code that relies on another component's behaviour: the rule, with the form of the comment (component, the image's version, file, function; or a document's URL and section; or "measured on the stand, date" said as such), and fixtures saying whether they were seen or composed. - Audit stage 2c: the line "what was taken on trust" becomes the heuristics review - an inventory over the whole tree written down as docs/assumptions-2026-10-14.md, then each answered from the source of the version the image runs, the comment added, the code fixed where it contradicts. Plan still 66 lines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
3c8628074f |
release: 2.5.1: audit stage 2b - the log tailer and what the panel shows of a refusal
test / test (push) Canceled after 0s
A log rotated while the panel was down is read to its end; the deferred counter remembers a place in the log, not a second; a refused message's page shows what Postfix wrote; a refusal the panel did not get to write is written from mail.log; downtime is estimated (≈) when supervisord is silent; the Sender check card counts connections and messages apart (the owner's decision); a message the sweep closed says the panel closed it. Plus the findings of both reviews of the stage. Rolled out to the test server as 2.5.0-post.e1ddf12 (the same code): migration 0009 ran on the live database - schema 9, the new columns in place, 68/68 send-log rows still tied to their domain, integrity ok. CHANGELOG cut; pin 2.5.1 in deploy/docker-compose.yml, README (with the local-build note), guide; development.md "from pin"; roadmap row and the recommended order; stage 2b marked shipped in the plan. No tag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
e1ddf12744 |
audit: stage 2c - what was taken on trust, read in Postfix's source
test / test (push) Canceled after 0s
Owner, 2026-10-14, after the rule "do not guess how code works when its documentation or source can be read": "Есть ли ещё у нас места, где мы угадывали?". The inventory found four: qmgr's expiry against the queue lifetime (a margin in queued_over_lifetime), postlogd after a rename (the rotation window), the stale smtpd after a reload (measured, not understood), and the NOQUEUE line's fields (two fixtures carry a sasl_username Postfix does not write, one presented as seen on the stand). A line in stage 2c reads Postfix 3.7.11 for them, records what is read, fixes the fixtures and the code where they disagree. Plan 66 lines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
5dcff8c2b9 |
audit: stage 2c - the sender check reads From: as OpenDKIM does (owner's decision)
test / test (push) Canceled after 0s
Owner, 2026-10-14, after reading OpenDKIM's source together (dkim_mail_parse, called in mlfi_eoh on the first From:; SenderHeaders is unset in build/opendkim.conf): "Да" to porting the parser so that the check decides by the address OpenDKIM will take. The scan of the display name and comments for another sender of the server - a guard against two parsers parting ways, and the reason L8 is half closed - then goes. New stage 2c (three lines) after 2.5.1 and before stage 3: the port from the version the image runs, checked against OpenDKIM on the stand with generated lines, then docs and wording. Stages 3 and 4 move to 2.5.3 and 2.5.4 (development.md's planned cuts). Plan 65 lines, roadmap 30/65. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
57142b21b5 |
security: L8's remaining probe accepted by the owner
test / test (push) Canceled after 0s
Owner, 2026-10-14, asked whether to refuse every foreign address in a display name to close what L8 leaves (refused or passed still tells an application which domains are hosted here): "Ответ 1" - leave it. The probe is a trusted application of this server guessing names one at a time, as it already had to before L8, when the reply said in words what the line itself had named. security.md (Mail path and accepted risks, with the trigger to revisit: untrusted tenants, or the display-name rule decided anew) and the audit plan record it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
5154bba61c |
docs: stage 2b of the audit - the log tailer, refusals, the Sender check card
test / test (push) Canceled after 0s
architecture.md - § Log tailer: the fingerprint carries the inode; a log rotated while the panel was down is read from mail.log.1 to its end, or the gap is logged; a refusal's line (the queue id of a From: refusal, the NOQUEUE line put on a MAIL FROM refusal's row) and the refusal written from mail.log when the panel had not. - § Sender check, what it costs: supervisord asked for half a minute, then the estimate and ≈; connections and messages in two columns; the count's saved place in the log. guide.md - § Outbound log: a refused message's page shows Postfix's line(s); rows from before, and a row written from mail.log, say so. - § Overview and Health: the two columns and ≈. schema-migrations row 9 was written with the migration. Audit plan line ticked: stage 2b's lines are all done. Roadmap 30/62. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
415334c2cc |
message page: a row the sweep closed says the panel closed it (stage 2b)
The History step of every bounced row said the receiving server refused
the message permanently or Postfix gave up, "the reason is in the delivery
log below" - for a row the reconcile sweep closed (reason "reconcile")
neither is known and the log has nothing (review of stage 1). Such a row's
step is now "Closed by the panel - Postfix no longer held the message and
mail.log never said what became of it ... rather than show a delivery it
cannot evidence." The help topic log names the case (design
|
||
|
mixandClaude Opus 5.5
|
ac985d03a8 |
sender check: connections and messages counted apart; downtime estimated when supervisord is silent (stage 2b)
Following the design commit
|
||
|
mixandClaude Opus 5.5
|
65d916f26a |
logtail: a refusal the panel did not get to write is written from mail.log (stage 2b)
test / test (push) Canceled after 0s
The sender check answers 550 5.7.1 and writes the refused row through a
buffer a moment later; a panel stopped in between (a rollout, a kill) left
an application refused and a log that did not know it. A refusal at MAIL
FROM needs nothing: the journal milter writes its row before it answers.
- logtail.refusalLines remembers each queue id's login from smtpd's
"<id>: client=..., sasl_username=<login>" (4096 ids at most) and, on
cleanup's "<id>: milter-reject: END-OF-MESSAGE ...: 5.7.1 From: <line> -
<why>; from=<...> to=<...>", waits 30 s and hands it to
store.RecoverRefusal - an INSERT ... WHERE NOT EXISTS a refused row of that
queue id within two minutes: the row the panel wrote, or one recovered
before, means nothing is written. It is read from the saved offset on, so
a kill is covered as well as a clean stop - which a flush of the buffer at
shutdown would not.
- Migration 0009 (not deployed yet) gains send_log.recovered; the row is
written with the line's time, the login's domain id, reason "from", the
From: line as the check printed it (to the last " - " before ";").
- The message page: a fact "Written: from mail.log after a restart: the
panel stopped before it wrote the refusal".
Tests: store (not written when the panel wrote it; written once however
often); logtail (the login from client=, the From: line with a " - " in it,
nothing before the wait is over); view (the fact). e2e
refusal_recovered_from_the_log: a From: refusal, the panel stopped, its row
deleted and the tailer's saved place moved before the message's first line
(python3 in the container, as the panel's user - a kill cannot be timed to
that moment from outside); started again, the row is back, recovered, once.
Also fixed: handlers TestMessagePageOfARefusedMessage still expected the
old sentence; it failed on the server after
|
||
|
mixandClaude Opus 5.5
|
e166a2ebd6 |
refused messages: the page shows what Postfix wrote (stage 2b, 0009)
The Delivery log box of every refused row said "This message never reached
the queue, so Postfix wrote no delivery lines for it" - untrue: Postfix
wrote a line for each. The owner, 2026-10-13: the box is required on every
message page. Pictured for him first (dev/shots, the refused variant of
out-message: wrong From with its queue id lines, rate-limited with the
NOQUEUE line); the variant has no mockup screen of its own - the guard ties
screens to pages, and its outline is out-message's.
- Wrong From: cleanup gives the message a queue id before the headers are
judged; the sender check reads {i} at end-of-headers and writes it on the
refused row. Its lines - stored under the id like any other's - are on the
page, but only those from 10 minutes before to 1 minute after the refusal
(store.SendLogLines for a rejected row): Postfix frees the id at once, and
a later message with it lends nothing. No mail.log fallback for a refused
row, for the same reason.
- Rate-limited and wrong client: refused at MAIL FROM, no queue id. Migration
0009 adds send_log.refusal_line; logtail.refusalLines matches smtpd's
"NOQUEUE: milter-reject: MAIL from ...; from=<...>" (not port 25's, not
CONNECT/EHLO) and store.AttachRefusalLine puts it on the newest refused row
of that sender (and login, when the line names it) within two minutes; a
line whose row is not written yet waits up to 30 s (retried at each pass).
- The page: the queue id fact says "given, the message was not queued" or
"none - refused at MAIL FROM, before one is given"; the box's caption is
"The line from mail.log for this refusal" for the latter; a row from
before says the line was not kept. LogLineLevel colours a refusal by its
reply: 5xx red, 4xx amber (IPv6 clients included). The id in the fact is
not muted as in the picture: the kit has no muted inline.
Tests: store (attach by sender, login, time; not a queued row, not one an
hour off; a refused row's lines are its own minutes); logtail (attach, wait
and retry, port 25 and other lines ignored); milter (the refused row carries
{i}); view (facts, caption, line levels); handlers (the line on the page,
the sentence without it). e2e refusals_show_postfixs_line: a client-address
refusal made there, and the pages of a wrong From, a rate-limited and a
wrong client refusal each show Postfix's line.
schema-migrations row 9, CHANGELOG [Unreleased], audit plan line ticked,
roadmap 25/62.
Evidence on the test server for this tree: make e2e exit 0 (the new case
in 5.4 s); go test ./... ok but for two schema tests, fixed after the run
(the schema at 0009) - store tests ok locally; the full run follows before
the push.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
mixandClaude Opus 5.5
|
c452606175 |
logtail: the deferred counter remembers a place in the log, not a second (stage 2b)
test / test (push) Canceled after 0s
The counter passed over, while catching up after a start, every warning no later than the last one counted - by time; mail.log stamps whole seconds, so a second warning of that second, later in the file, was lost. - The tailer gives each line its place (Place: the file's fingerprint and the offset just past the line): follow tracks the bytes the reader hands over, readRotated likewise in mail.log.1. Run hands tr.line to the counter. - store.AddSenderCheckDeferred(at, n, place) counts, moves the last-deferral mark (never back - an upsert WHERE) and saves the place (settings sender_check_counted_place) in one transaction. - Catching up, a warning in the file of the last one counted is passed over by place (end <= counted end); in another file - a rotated log read again - by time, as before. The fingerprint carries the inode, so the place stays true across a rotation. Tests: two warnings of one second, an unclean stop between them, all lines read again - each counted once; the store saves the place with the count and keeps the latest deferral time. Existing counter tests pass a zero place (time rule). CHANGELOG [Unreleased], audit plan line ticked, roadmap 24/62. Evidence: go vet ./... clean; go test ./internal/store, ./internal/logtail ok (but for the Windows-only test); the server run follows before the push. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
4678015bbd |
logtail: a log rotated while the panel was down is read to its end (stage 2b)
At start, a saved offset whose fingerprint was not the current file's meant "start on the new file": what Postfix wrote between the panel's stop and the rotation was never read - delivery statuses, stored lines, refusals, inbound journal rows, the deferred counter. - tracker.resume marks the case (behind, prev); follow calls readRotated before the new file: it opens mail.log.1 (delaycompress leaves it plain), checks by fingerprint that it is the file the place belongs to, reads it from the saved offset to its end through handle (catching up: the once-protections apply) and flushes the stored lines. - Not found, not that file, or the offset past its end (rotated twice): logged, with the saved place's time - from when nothing was read. - The fingerprint gains the file's identity (device.inode, fileid_unix.go; empty elsewhere): two logs that begin alike are two files - the review-2026-08-followups row on collisions. A fingerprint saved before (hash alone) still matches by its hash (sameLog), so the first start after the upgrade resumes where it was. - store.LogtailState reads back updated_at. Tests: stop, append, rotate, append, start - both halves read, once each, the earlier line not again; a mail.log.1 that is another file is not read from the place; sameLog with an old and a new fingerprint. CHANGELOG [Unreleased], audit plan line ticked, roadmap 23/62. Evidence: GOOS=linux go vet ./internal/logtail clean; go test ./internal/logtail ok on Windows but for TestFollowTailsAndRotates; the server run follows before the push. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
5be109b871 |
release: 2.5.0: audit stages 1 and 2 - the delivery log and the tenant boundary
test / test (push) Canceled after 0s
Stage 1: a cut queue listing no longer bounces held mail; the reconcile sweep closes only queued rows, by id, marked "reconcile", and its fate is measured (delivery_log_days, migration 0007); delivery lines are read at the head of the line; a renewed TLS certificate is served without a restart; Health counts the message a hung panel deferred. Stage 2: logins folded to lower case and unique without regard to case; the last global administrator kept inside the transaction, users saved whole; a domain import held to the forms' rules; log rows by domain id (migration 0008); the 550 reply names no other domain; which key OpenDKIM signs with, tested. Plus the findings of both reviews of each stage. Rolled out to the test server as 2.4.2-post.ef18ef2 (the same code): migrations 0007 and 0008 ran on the live database - schema 8, 68/68 send-log and 10/10 inbound rows tied to their domain, integrity ok. CHANGELOG cut; pin 2.5.0 in deploy/docker-compose.yml, README (with the local-build note), guide; development.md "from pin"; roadmap row and the recommended order; stages 1 and 2 marked shipped in the plan. No tag (the owner's command only). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
ef18ef21a0 |
findings of the reviews of audit stage 2
test / test (push) Canceled after 0s
Technical review (Opus) and security review (Fable) of a59e3d4..cb5a4a2;
reports in the session's scratchpad. Nothing critical, high or medium; no
outright bug. Fixed:
- F2, F3 (0008 backfill, before it runs on the live database): rows are
filled by the rule new rows follow - the application's domain, then the
name's - and only by a domain and application that existed when the row
was written (created_at); the inbound log likewise. Test: the migration's
UPDATEs re-run on prepared rows (before the domain, after, null sender,
application gone, a stranger's name). Dry run on a copy of the live
database: 68/68 and 10/10 rows filled, none disagreeing with its login's
domain, integrity ok.
- V2, F4: an application's statistics and level-2 count are its login's
rows of its domain (appByLogin), and StatsWindow starts no earlier than
the entity's creation; the message page's "may send as" only for a login
still of the row's domain. Test: a login taken again in another domain
starts at zero.
- F7: MAIL FROM:<> is held to the domain limit of its login
(ApplicationByLogin fills DomainName).
- V3: an imported DKIM selector is held to DNS labels (validate.Selector).
- F5, F6: an imported auto limit's numbers are dropped and recalculated
right after the import; its mode is stored trimmed.
- V6: the store refuses a role that is neither global nor domain.
- F9: SaveUser keeps the stored hash when no new password is given (the
form no longer writes back the hash it read). F10: a reach naming a
domain deleted meanwhile is ErrNoSuchDomain, answered on the form.
- V4, F12: from-lines-other.txt - lines naming another sender ({other}) in
a quoted string, a quoted pair, capitals, a comment, an encoded word, a
quoted local part; the unit test wants each refused, the e2e
(lines_naming_another_sender_never_signed_for_it) wants each refused and
none signed for it. The passed list gained a.{DOMAIN}, a comment after
the angle address, a quoted local part, and "b@ {other}" - passed by
design (security.md), and the e2e shows it signed for the own domain.
- F13: e2e login_in_other_capitals_does_not_authenticate - AUTH with the
login upper-cased gets 535.
Not changed, written down: V1/F1 - L8 is half closed (refused or passed
still tells an application which domains are hosted here); security.md's
accepted risks and the CHANGELOG say so, and the owner is asked whether to
refuse every foreign address in a display name (changes the 2.4.0 rule).
F11 - deleting a domain rewrites its rows' domain_id (seconds at 1e5 rows),
accepted at this size. F8 (test cannot tell write-first from read-first;
the reviewer verified it with two stores, 30/30), V5, V7, P2: noted.
Evidence on the test server for this tree: go vet clean, go test ./... 23
packages ok, make e2e exit 0 with the three new cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
mixandClaude Opus 5.5
|
8b53d63160 |
audit: the owner's decision on the Sender check columns (stage 2b)
test / test (push) Canceled after 0s
Owner, 2026-10-14: show connections turned away and messages deferred mid-message apart, not mixed. The stage 2b step now says: smtpd's warning to the existing column (Connections turned away), cleanup's to a new column of sender_check_days in the stage's migration (Messages deferred), both on the card, the help topic saying what each counts; mockup first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
cb5a4a22e1 |
docs: stage 2 of the audit - roles, the guard, logs by id, import, logins
test / test (push) Canceled after 0s
security.md - § Roles: the last-global guard is inside the transaction that deletes or demotes (write, then count); setup is one statement; a user is saved in one transaction. A log row, outbound or inbound, belongs to a domain by id; a domain added again starts with an empty log. The Inbound › Log row of the table says by id. - § Mail path, sender check: the agreement of the check and OpenDKIM is tested (the shared list of passed From: lines); the refusal does not say which other sender a line names. - Accepted risks: "a row of the inbound log belongs to a domain by name" struck (L12). guide.md - Domain export and import: the file is checked like the forms before anything is created; a login keeps its case and a case clash is refused. - The application form: a new login is stored in lower case. schema-migrations.md was updated with 0008 in its commits. Audit plan line ticked; stage 2's lines are all done. Roadmap 22/62. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
5747f0b25c |
sender check: the reply names no other domain (L8); which key signs, tested (stage 2)
L8: a From: line naming another sender of the server was refused with
"it names another sender of this server, <domain>" - a confirmation, to
whoever tries names, that the domain is hosted here. The reason now stops
before the domain; the line as written stays in the reply (the client's own
text) and on the refused message's page (header_from). Unit test: the reason
of every refusal names none of the server's other domains; e2e: the reason
of each sender-check refusal does not contain other.e2e.test.
Which key OpenDKIM signs with (owner, 2026-10-13): the check and OpenDKIM
read From: with two parsers, and their agreement was argued, never run.
internal/milter/testdata/from-lines-passed.txt holds the lines the check
passes for an application - bare and bracketed, quoted names with a comma,
escaped quotes and an address inside, an extension, three encoded charsets,
two folded lines, comments (nested too), a stranger's address in the name -
as Go-quoted strings with {domain}. The unit test wants each passed; the new
e2e case passed_from_lines_signed_for_their_domain_only sends each as app1
and wants every DKIM signature at the sink to be for the sender domain,
with one valid. All fifteen were. "(the shop) a@domain" - a comment before
the address, valid RFC 5322 - is refused by the check ("it must be one
address"); strict, not a hole, so it is not in the file. Groups are refused
too.
Audit plan lines ticked (two) with what was found, CHANGELOG [Unreleased],
roadmap 21/62.
Evidence on the test server for this tree (with
|
||
|
mixandClaude Opus 5.5
|
a618ffb0e4 |
send log: rows belong to a domain by id (audit L12, 0008)
A domain user's log, the message page, a domain's statistics, the level-2 count and the notifier's refusal count picked send-log rows by domain name, so a domain deleted and re-added under its old name showed its earlier mail to the next assignee and went on counting it. Migration 0008 (not run on the live database yet, so extended here): send_log.domain_id and inbound_spam_log.inbound_domain_id, REFERENCES ... ON DELETE SET NULL, filled once by name from the registry as it is at migration time, each with an (id, created_at) index. - Writes: InsertQueued / InsertRejected set domain_id to the domain of the row's login (loginDomainID) - the journal milter and the sender check's refusal rows both go through them; the inbound journal sets inbound_domain_id from the inbound domain's name at insert. - Reads: SendLogFilter.DomainIDs (was Domains, by name) and InboundSpamFilter.DomainIDs are the scope; the global role still reads everything. The message page checks row.DomainID against the user's domains. DomainSendStats and CountMessages keep their name argument and select domain_id = (SELECT id FROM domains WHERE name = ?). The notifier joins send_log.domain_id to the domain and its application. - DMARC reports stay by name on purpose (a report is about the DNS name). Tests: store - scope by ids; a re-added domain's log, level-2 count and statistics hold its own row only, the global log both; inbound scope by ids. Handlers - the new assignee gets 404 for the old row's page and does not see it in the log, the global role sees both. Tests that wrote rows for a login that is no application now register it (a row is a domain's through its application). Schema at 0008 with the two columns and indexes. schema-migrations row 8, CHANGELOG [Unreleased], audit plan line ticked, roadmap 19/62. Evidence: go vet ./... clean; go test ./... ok locally but for the two Windows-only tests; the server run follows before the push. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
e469034477 |
import: the export file held to the forms' rules before anything is created (audit M10)
test / test (push) Canceled after 0s
domain.Import trusted the file for everything but the domain name, the selector and the password: DMARCRua was not validate.Email'ed, client IPs not parsed, a rate limit's mode, window and count not checked - a window of 0 counted nothing, a garbage IP made the application refuse every client. handlers.checkDomainImport runs in HandleImportDomain after the domain name check and before Import, and refuses the file with "Invalid <what> in export file: <the form's own text>": the report address (validate.Email), each rate limit (parseRateLimitMode; auto - checkAutoMultiplier, 0 meaning the default as an empty field does; manual - checkManualLimit, which the form now uses too: count > 0, <= the level-1 backstop, window > 0; both 0 is an export without a limit of its own), each application's client addresses and the legacy ones on its rate limit (parseIPList, written back as the form stores them; allow-list on with none is refused). Login, sending addresses and password stay checked in app.ImportApplication. Tests: a good export passes and its addresses come out normalised; nine broken fields each refused with their message; a refused file posted to the handler creates no domain. CHANGELOG [Unreleased], audit plan line ticked, roadmap 18/62. Evidence: go vet clean; the import, rate-limit and application tests of ./internal/web/handlers ok; the server run follows before the push. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
f9e75de8d1 |
store: users saved in one transaction, the last global kept inside it (audit M7, M9)
M7: DeleteUser counted the global users and then deleted; the handler's demotion and CreateGlobalUser checked and then wrote. MaxOpenConns(1) serialises statements, not pairs: two globals deleting or demoting each other at once could leave none. Now: - DeleteUser: a transaction that deletes first (taking the write lock) and then requires a global left (requireAGlobal) - ErrLastGlobal rolls back. - SaveUser: likewise for a demotion, the count inside the transaction after the write. The handler's early count stays only to answer before hashing a password; SaveUser decides, and ErrLastGlobal maps to the same message. - CreateGlobalUser: INSERT ... SELECT ... WHERE NOT EXISTS (SELECT 1 FROM users), one statement. M9: submitUserUpdate made five writes in a row (UpdateUser, sessions, SetUserRole, ClearUserReach with its error only logged, SetUserReach). store.SaveUser(UserChange) writes name, hash, e-mail, role and reach in one transaction (a global user's reach cleared); sessions end after it commits. The create path had the same flaw - user, reach and then e-mail in three writes - and now goes through AddUser, one transaction; CreateUser stays as a wrapper for its many test callers. SetUserRole is gone; writeUserReach is the reach write inside a transaction. Tests: store - two globals demoting, and deleting, each other from two goroutines: exactly one refused, one global left; a refused SaveUser (taken name, a reach to no domain, nothing assigned) changes nothing; promoted to global loses the reach; AddUser with a bad reach leaves no user; four setups at once create one user. Handlers (the users pages, direct): the whole change saved, back to domain with a reach; the last global not demoted and the typed rename not written; a domain user with nothing assigned refused, nothing changed; create with e-mail and reach. CHANGELOG [Unreleased], audit plan lines ticked (two), roadmap 17/62. Evidence: go vet ./... clean; go test ./internal/store ok; the users tests of ./internal/web/handlers ok; the server run follows before the push. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
60c7b85f84 |
findings of the reviews of audit stage 1
test / test (push) Canceled after 0s
Technical review (Opus) and security review (Fable) of 9efc9c7..a59e3d4; reports in the session's scratchpad. Fixed: - B1 (verified bug): store.UpdateStatus read before writing in its transaction; a commit by another connection in between (the dmarc-ingest process opens the same file) failed it at once with "database is locked" and the delivery line was lost. The first statement is now a write. - R3: a delivery line updated every row of its queue id and recipient; it now updates only the newest (Postfix reuses short ids). The sweep closes by row id (QueuedDelivery.ID, CloseUnreported(id)). - R4: a contradicted sweep keeps a mark, reason "reconcile-contradicted", instead of losing which row it was. - R1: contradicted cannot see a lost line (never read late). New counter reconcile_missed: at close, the sweep looks for the row's delivery line in mail.log and mail.log.1 (QueueLines). 0007 edited - it has run on no database but throwaway stands. - R2: queued_over_lifetime cannot measure "what removing the sweep would leave" while the sweep runs; its text now says rows stuck despite the sweep, and the rule uses closed - missed - contradicted. - cert-reload (R5, R6, R7, security medium/low): both files copied to .new and the copies checked as a pair before both are renamed; acts on the copy only when postconf says Postfix reads it; a failed reload is retried next pass; umask 077; says so when the mounted files cannot be read. The e2e case asserts Postfix reads the copy, the copy equals the mount, the key is 600 root. - Security low: a recipient with a quoted local part - the journal cut "a b"@x at the space, deliveryRe cut "a>b"@x at the ">" - matched no delivery line and was closed by the sweep. cleanAddress ends a bracketed address at the first ">" outside quotes; deliveryRe takes the recipient up to the ">, " before relay= (test fixtures given the relay= every real delivery line has). - Security low: postqueue -p's error carries at most 512 bytes of output. - Nits: addDeliveryLogDay refuses a column not on its list; the truncation test checks its message. - To stage 2b (page wording, mockup-first): the history step of a bounced row says the receiving server refused it, false for a row the sweep closed. To the owner when stage 2b is reached: since M13 the "deferred" figure counts whole messages too, which "Connections turned away" would not name. Found while verifying: the e2e's main positive path failed once with "553 not owned by user app1" - an smtpd idle from before the panel's reload took the connection in the reload's second. Measured on the stand (one-off probe, not committed): sending 0, 10, 40, 110 s after the save (two seconds at the soonest) always met an smtpd started after the reload. The e2e now waits for the sender map after each application it creates (waitForSenderMap: AUTH, MAIL, RCPT, RSET - no message, no row), and the level-1 case uses it instead of its own loop. Docs: architecture § Queue reconcile, schema-migrations (rows 7 and 8, chain head 8), CHANGELOG, audit plan (stage 1 note, the measured window, a new stage 2b line: 62 lines), roadmap 15/62. Evidence on the test server for this tree: go vet clean, go test ./... 23 packages ok, make e2e exit 0 (renewed certificate, hung panel, level 1 at 49/60). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
f51f5db245 |
app: logins folded to lower case, unique without regard to case (audit H4, 0008)
Postfix's reject_sender_login_mismatch compares the SASL login with the sender map without regard to case - seen on the e2e stand this time (a one-off probe, not committed: map rewritten to "APP1", client authenticated as "app1", RCPT accepted). The panel stored logins as typed under a BINARY UNIQUE, so "Ops" and "ops" were two applications to the panel and one sender to Postfix: a tenant could send with another tenant's envelope sender. - app.foldLogin: Service.Create lower-cases the login (owner: fold, do not refuse); CreateWithSettings rolls back the stored (folded) login. - ImportApplication keeps the login as written: folding it would rename the SASL account and break the imported application's working credentials; the index below refuses a case clash there too. - Migration 0008 (DDL): CREATE UNIQUE INDEX idx_applications_login_nocase ON applications (login COLLATE NOCASE). The column's own BINARY UNIQUE stays (SQLite cannot change a collation without a table rebuild, which a migration transaction cannot do with foreign keys on). Checked on the live installation first: 4 logins, no case clash; "Quieter3C" keeps its case. 0008 is not run anywhere yet; send_log.domain_id joins it later in stage 2. Tests: "Ops" created as ops, SASL account and map binding in lower case, OPS in another domain and an imported oPs refused; the store refuses ops beside Ops; schema at user_version 8 with the index. CHANGELOG [Unreleased], audit plan line ticked, roadmap 15/61. Evidence on the test server for this tree (with the probe): go vet clean, go test ./... 23 packages ok, make e2e exit 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
a59e3d4d61 |
docs: stage 1 of the audit - log tailer, reconcile, sender check, guide
test / test (push) Canceled after 0s
architecture.md - § Log tailer: the rotation drain does not close the gap entirely (postlogd writes to the renamed file until the reload reaches it); the fix waits for the measurement. Delivery lines read at the head of the line (M12). - § Queue reconcile: reason "reconcile", the 4 MiB cap, the conditional close; the measurement (delivery_log_days, its four counters) and the decision rule written before the data. - § Sender check, what it costs: cleanup meets a check that hangs mid-message, with the warning seen on the stand; counted once per message. guide.md - § Outbound log: `bounced` is the panel's word when Postfix no longer holds a message and the log never said; it is counted. - § Overview and Health, Sender check: the message a hung panel deferred is counted too. Audit plan line ticked, roadmap 14/61. Stage 1's code items are done (the two deferred tailer fixes wait for the data, as planned). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
db36aa2c0e |
logtail: count the message a hung sender check defers in cleanup (audit M13)
Seen on the e2e stand, the panel frozen with SIGSTOP once DATA was
answered: the client gets 451 4.7.1 after 30 s, and the warning that names
the socket is cleanup's, not smtpd's:
postfix/cleanup[1339]: warning: milter unix:/run/selfpost/sender.sock:
can't read SMFIC_HEADER reply packet header: Connection timed out
postfix/cleanup[1339]: 7BCAF3C75DA0: milter-reject: END-OF-MESSAGE ...
4.7.1 Service unavailable - try again later; ...
The counter read smtpd lines only, so Health's "messages deferred" missed
the one case that tells a hung panel from a stopped one. milterLineRe now
takes smtpd and cleanup; cleanup's "milter unix:<socket>:" warning counts a
message each time (one cleanup process serves message after message, and the
first warning ends the message); smtpd keeps once per connection. The
composed fixture is replaced by the seen lines, with two messages through one
cleanup process and OpenDKIM hung in cleanup (not counted).
e2e panel_hung_mid_message_is_counted: freezes the panel after DATA's 354,
wants a 4xx at the end of DATA, logs what mail.log says, and waits for
Health's today count to grow. kill through sh -c: the image has no kill(1).
CHANGELOG [Unreleased], audit plan line ticked, roadmap 13/61.
Evidence on the test server for this tree (with the next commit's docs):
go vet clean, go test ./... 23 packages ok, make e2e exit 0 -
renewed_certificate_picked_up 1.2 s, panel_hung_mid_message_is_counted
31.7 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
mixandClaude Opus 5.5
|
7af37654a1 |
cert-reload: renew Postfix's copy of a renewed certificate (audit H2)
postfix-config.sh copies the mounted certificate and key into /etc/postfix/tls-internal at start (root-owned, key 0600) and Postfix reads the copy; cert-reload only ran `postfix reload` daily, which re-read the stale copy, so a certificate renewed while the container ran expired on 465/587 with the old one. postfix-cert-reload.sh now, every TLS_RELOAD_INTERVAL_SECONDS: compares the mount with the copy (cmp); when they differ and the certificate and key belong together (openssl public keys equal - a renewer writes them one after the other), writes each file beside its target, root-owned with the start's modes, renames it over the target, and reloads Postfix. Nothing changed - no reload. No copy (files missing at start) - the old periodic reload. Never exits non-zero. e2e renewed_certificate_picked_up: writes a new certificate into the stand's mount and waits (30 s) for port 465 to present its serial; compose.override.yml sets TLS_RELOAD_INTERVAL_SECONDS=2. supervisord.conf comment, architecture.md (process table, § Postfix TLS certs, the env knob), guide (after the proxies: picked up within a day). CHANGELOG [Unreleased], audit plan line ticked, roadmap 12/61. Evidence: GOOS=linux go vet in test/e2e clean; the server run of the stage follows before the push. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
63520f1678 |
logtail: deliveryRe anchored at the head of the line (audit M12)
The pattern that writes a row's status matched anywhere in a line: a port-25 reject line ending helo=<ID: to=<victim@...>, status=sent> parsed as (ID, victim, sent). parseDelivery now takes the stamp off with SplitTimestamp and matches `^host program[pid]: <id>: to=<...>, ... status=`, as queueLineRe and the inbound patterns do. Tests: the probed HELO line, the same text in a quoted sender on a qmgr line, the text after an smtpd line's queue id - none parse; a syslog-stamped line as Postfix 3.7 writes it parses. The existing cases unchanged. CHANGELOG [Unreleased], audit plan line ticked, roadmap 11/61. Evidence: go test ./internal/logtail ok but for TestFollowTailsAndRotates (Windows); the server run of the stage follows before the push. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
ffefa9d6df |
delivery log: measure the sweep before deciding its fate (audit H3, 0007)
Migration 0007 (DDL only): delivery_log_days, a row per UTC day, kept a year, shown nowhere - read off a copy of the database after a month by the decision rule written in the audit plan. - reconcile_closed: the sweep adds the rows it actually closed. - reconcile_contradicted: store.UpdateStatus, in one transaction, counts the rows with reason "reconcile" a delivery line now speaks of, writes the log's status and clears the reason - so a re-read line counts once. - tailer_blind_seconds: follow() tracks stretches in which the file is gone or reopening fails (logtail.blindness); stored when a stretch ends and every minute while it lasts; under 3 s not counted (the rotation's tick between rename and create loses nothing). - queued_over_lifetime: every sweep overwrites the day's level with the rows queued past LongestLifetime + MaximalBackoff (a message Postfix still works on stays queued until its first retry after the lifetime). - retentionLoop prunes days older than a year. Tests: store (sums per day, the level overwritten, under a second nothing, prune; a contradicted sweep counted once and its reason cleared; count of queued rows); logtail (the sweep counts what it closed; countStuck records the level; blindness - a one-tick gap not counted, 75 s stored as 60 + 15 from where the last part ended). Schema tests at user_version 7. schema-migrations.md: chain head 7 (it still said 5 since 2.4.0), row 7. CHANGELOG [Unreleased], audit plan line ticked, roadmap 10/61. Evidence: go vet ./... clean; go test ./internal/store ok, ./internal/logtail ok but for TestFollowTailsAndRotates (Windows); the server run of the stage follows before the push. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
9adb44845c |
logtail: the sweep closes only queued rows, with reason "reconcile" (audit H3)
reconcile() closed a row through UpdateStatus (WHERE queue_id AND to_addr), which could overwrite a "sent" the tailer wrote between the listing and the close, and rewrite an old row of a reused queue id. store.CloseUnreported sets status bounced and reason "reconcile" (store.ReasonReconcile; the 0006 column, no schema change) WHERE ... AND status = 'queued'; the sweep counts rows actually closed. Tests: store - a queued row is closed with the reason, a sent row is left alone (0 rows); logtail - the sweep's close goes through CloseUnreported. CHANGELOG [Unreleased], audit plan line ticked, roadmap 9/61. Evidence: go vet ./... clean; go test ./internal/store ok, ./internal/logtail ok but for TestFollowTailsAndRotates (Windows rename of an open file, fails here always); the server run of the stage follows before the push. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
dfe10f6c27 |
postfix: a truncated postqueue -p listing is no listing for the sweep (audit H3)
QueueIDs parsed Queue()'s text, which caps postqueue -p at 4 MiB and appends a note, and returned the ids before the cut with no error: the reconcile sweep then closed every queued row past the cut as bounced while Postfix still held the mail. listQueue() now returns the text and whether it was cut; QueueIDs refuses a cut listing with an error (reconcile already treats an error as no information), the way postqueueJSON does for the notifier. Queue() keeps showing the cut listing with its note. postqueueP is a var so tests run a fake postqueue (the test binary as a helper process): 5 MiB -> QueueIDs errors, Queue ends with the note; 1000 bytes -> ids read. CHANGELOG [Unreleased], audit plan line ticked, roadmap 8/61. Evidence: go test ./internal/postfix ok (Windows); the server run of the stage follows before the push. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
9efc9c7005 |
release: 2.4.2: audit stage 0, release hygiene
test / test (push) Canceled after 0s
Stage 0 of docs/audit-2026-10-12.md, 7/7: the release image says which release it is (H1), the e2e module is compiled on every push (M20), test/ is out of the build context (L33), the documents say the 2.x image is not published and show the local build (M6), dependencies raised, staticcheck clean, the negative e2e cases held to their own refusals (which showed the level-1 rate-limit case had never seen the limit). CHANGELOG cut; pin 2.4.2 in deploy/docker-compose.yml, README (with the local-build note), guide; development.md "from pin" and the planned cuts of the audit's stages (2.5.0 ... 2.6.0); roadmap row in progress and the recommended order; stage 0 marked shipped in the plan. No tag (the owner's command only). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
96425dd515 |
e2e: a negative case passes only on its own refusal (audit stage 0)
sendResult.refusedAt(stage, code, text) and replyIs(err, code, text):
nil only for the server's reply with that code and text at that stage;
a reset, a timeout or another reply is described for the failure.
- level 1: only anvil's "450 4.7.1 Error: too much mail from", only after
a message of the loop was accepted. Held to that, it failed twice on
the stand, each time for a real reason:
1. waitForSMTPSReady took a TCP connect for readiness; Docker's port
proxy accepts before Postfix listens and then resets - the reset
once taken for the limit. It now waits for the TLS 220 greeting.
2. the first message then got "553 ... not owned by user l1app": an
smtpd idle from before the panel's postfix reload took the
connection with the old sender_login_maps. The level-1 limit had
never been what the case saw. The first message now waits (15 s at
most) for the map, failing at once on any other refusal.
Now: "level-1 limit tripped on attempt 51/60: 450 4.7.1 Error: too
much mail from 10.77.0.1".
- level 2: 451 4.7.1 at MAIL, and both MAIL FROM of one connection.
- sender mismatch: 553 at RCPT, "not owned by user".
- no AUTH and the relay case: 554 "Client host rejected: Access
denied" at RCPT; the relay case's comment claimed
reject_unauth_destination, which on 465 comes after the client
restriction - port 25 holds "554 Relay access denied".
- port 25: 550 5.1.1 for unknown recipients and report addresses.
- panel down: a 4xx reply (textproto), not just text starting with 4.
The product side (a second after an application is saved when its first
message can get 553) is written in the audit plan, not changed. Audit plan
line ticked with what was found, roadmap 7/61. No CHANGELOG line: tests only.
Evidence: make e2e on the test server for this tree, exit 0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
mixandClaude Opus 5.5
|
8b9798884a |
staticcheck: the four items of the audit (stage 0)
test / test (push) Canceled after 0s
- handlers/authz.go: domainIDSet removed, nothing called it (U1000). - handlers_outbound_pages_test.go: two refusals whose status code was assigned and never read (SA4006) now check it is 400 - a custom DMARC report mode without an address, export passwords that differ. - view/pages_test.go: !strings.Contains for a missing box (S1003). - Audit plan line ticked, roadmap 6/61. No CHANGELOG line: nothing an operator sees changes. staticcheck ./... clean afterwards. Evidence on the test server: see the previous commit (same tree run). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
80bab92c8c |
deps: modernc.org/sqlite 1.60.1, x/crypto 0.58.0, x/sys 0.49.0 (audit stage 0)
- go get ...@latest + go mod tidy: sqlite 1.53.0 -> 1.60.1 (libc 1.77.1, memory 1.12.1), x/crypto 0.54.0 -> 0.58.0, x/sys 0.47.0 -> 0.49.0, go-isatty 0.0.24. The go line reads 1.26.0 (tidy's spelling of 1.26); no dependency asks for a newer Go. - development.md: golang:1.26-bookworm floats to the newest patch, and the standard library's security fixes arrive that way - rebuild the image after each Go patch release; dependencies are raised by hand. - CHANGELOG [Unreleased], audit plan line ticked, roadmap 5/61. Evidence on the test server (Go 1.26.5) for this tree plus the next commit's staticcheck fixes: go vet clean, go test ./... 23 packages ok, make e2e exit 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
937b41ddb7 |
docs: say the 2.x image is not published; show the local build (audit M6)
README, guide and deploy/docker-compose.yml pinned ghcr.io/mixeme/selfpost:2.4.1, which does not exist: ghcr.io holds 1.0.0 and 1.3.0 only. The owner decided (2026-10-12) that old tags and images stay untouched and the pin points forward, made true by the next publication. Until then: - README Quick start and guide Fixed image tag say the 2.x line is not on ghcr.io and show `docker build ... --build-arg VERSION=2.4.1 -t ghcr.io/mixeme/selfpost:2.4.1 .`; with the image present locally, docker run and Compose (pull_policy missing) use it without pulling. - compose header says the same in two lines. - development.md release steps: bump the version in those notes at each cut, remove them at the first publication of the line. - CHANGELOG [Unreleased], audit plan line ticked, roadmap 4/61. Documents only; no test reads the pin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
cc2ca3c968 |
ci: compile the e2e on every push; test/ out of the build context (audit M20, L33)
test / test (push) Canceled after 0s
- test.yml: after go test, `go vet ./...` and `go test -c -o /dev/null .` in test/e2e - compiled and linked, not run. Not `go test -run xxx` as the plan said: TestMain would still run and bring up the stand. The Go cache now keys on test/e2e/go.sum too. - .dockerignore: test (e2e leftovers in test/e2e/.stage - DKIM keys, sasldb2 - no longer reach the builder; the sink image is built from test/e2e/sink as its own context, so it is unaffected). - development.md (test.yml line), CHANGELOG [Unreleased], audit plan lines ticked, roadmap 3/61. Evidence on the test server for this tree: make e2e (no SELFPOST_E2E_VERSION, so the "e2e" default path) exit 0, including image_stamped_with_version and the logrotate-mode build; in test/e2e go vet ./... and go test -c -o /dev/null . clean; docker build of build/Dockerfile with a file under test/e2e/.stage succeeds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |