32dfe4a2cfcaf289d72d7e68366f57b5cb638e29
29
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  mixandClaude Opus 5.5
|
9eddd49602 |
audit 2c: findings of the two reviews
test / test (push) Canceled after 0s
Technical review (Opus, 11 findings, 2 medium) and security review (Fable, no high or medium) of 3c86280..HEAD. Both confirm the port gives dkim_mail_parse()'s result on every input it parses, and that no line the check passes is signed with another domain's key. - F1 (medium): OpenDKIM's parser takes lines no mail program reads as one address - "ceo@bank.com:me@mine.com", "Name <ceo@bank.com:me@mine.com>", "<b@evil.com, a@mine.com>", "ceo@bank.com <me@mine.com>", "ceo@bank.com@mine.com" - and signs for the last while a lenient reader shows the first. The line must now also be one address in RFC 5322's grammar (net/mail, display names in any charset passed through), at the domain OpenDKIM signs for. A quoted display name may still hold anything. - F2 (medium): the bindings were asked about OpenDKIM's form of the address, quotes and all, so "x"@mine.com passed for an application holding @mine.com while x@mine.com belonged to another. They are now asked about the reader's address, its local part unquoted, as Postfix holds the envelope's. - F3 / security (low): Bindings.Allowed and DomainOf fold case in ASCII only: Postfix folds as Unicode (casefold.c, ICU), but every key and login the panel writes is ASCII, and the Unicode fold let the Kelvin sign pass for "k" (net/mail then refuses such a line on the domain test). - F4 (low): a TERM between "postfix start-fg &" and "child=$!" is no longer lost: the wrapper stops Postfix as soon as the pid is known. - F5 (low): for startsecs after a start any failed reload passed as "not running". postfix-reload is now build/postfix-reload.sh: exit 75 when master -t says no master runs, else exec postfix reload; RunOnce returns EndError with the status, and only 75 is taken as done. supervisor.State goes. - F6 (low): crashexit spares postfix-reload too - a reload that could not be spawned is reported to whoever asked, not a reason to stop the container. - F7 (low): a certs/ that is itself a link is resolved before the walk (filepath.WalkDir does not follow its root). - F8 (low): at shutdown the tailer reads what is left of the rotated file before saving its place. Not changed: a refusal line still waiting on the old file at shutdown is not kept - the panel writes refused rows itself, only a kill loses one, and that window is the minute's grace. - F9 (low): pct read as text - empty or unreadable is 100, the rest held to 0..100. - F10 (low): the directory is synced after sasldb2's rename. - F11 (low): POSTFIX_STARTSECS by expr, so "010" or "08" in MILTER_WAIT_TIMEOUT are decimal, not octal or an error. - Security (low): pagehide also empties defaultValue - the input's value attribute and the textarea's text - not only .value. A sasldb2.new left by a killed panel is removed at start (entrypoint) and never archived. security.md says that links under certs/ are followed and by whom they can be placed. Tests: sender check cases for F1 (four lines), F2 (a bound address quoted, for its owner and for another), F3; EndError's state and status; pct cases. architecture.md and CHANGELOG describe the reader rule. Stand (2026-10-15, make e2e, exit 0); go vet and go test ./... pass on the server (24 packages). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
463d4ff3fe |
log tailer: the rotated mail.log read until postlogd lets it go
test / test (push) Canceled after 0s
postlogd opens maillog_file once per process (Postfix 3.7.11,
src/postlogd/postlogd.c, pre_jail_init()) and leaves it only when it
exits: at "postfix reload" master closes the service's status pipe
(src/master/master_service.c, master_restart_service() ->
master_status_cleanup()) and dgram_server_abort() exits on reading its end
(src/master/dgram_server.c). logrotate renames and creates, then runs the
reload (build/logrotate-mail.conf postrotate; logrotate 3.21.0,
logrotate.c, rotateLogSet(), after rotateSingleLog()), so lines go to the
renamed file in between, and the tailer, already on the new one, never
read them (assumptions A-33). follow() now keeps the renamed file open as a
rotatedTail and reads it on each tick for rotatedGrace (a minute, far
longer than the gap); its places are never saved as the resume point, so a
restart still reads the rotated file through readRotated. A truncated file
is not kept.
The RFC 3339 stamp goes (A-12): it was written for a maillog_file_format
said to arrive in Postfix 3.9; there is none in 3.7.11 nor in the 3.12
snapshot of 2026-09-25 (proto/postconf.proto; msg_logger.c,
msg_logger_print() keeps strftime("%b %d %H:%M:%S ")). Every fixture now
carries the stamp Postfix writes; logtail's TestMain fixes time.Local to
UTC, since the stamp is local time and this PC is UTC+3.
Tests: TestFollowReadsTheRotatedFileAfterTheSwitch (Linux: a line appended
to the renamed file after the switch reaches the store). Stand
(2026-10-14): go vet and go test ./... pass on the server (23 packages);
make e2e exit 0, delivery_log_survives_rotation included. The tested tree
was this commit plus the previous one; the CHANGELOG entry was written
after the run, and a comment's logrotate citation corrected
(rotateLogSet runs postrotate, not rotateSingleLog).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
mixandClaude Opus 5.5
|
c8f669714e |
journal milter: recipients as Postfix logs them, once each
test / test (push) Canceled after 0s
The row's recipient has to be the address Postfix writes in to=<..> or orig_to=<..>, or the delivery line finds no row and the sweep closes it as bounced. Postfix 3.7.11 hands the milter the RCPT argument as the client wrote it (smtpd.c, milter_argv(); extract_addr() leaves it), go-milter v0.4.1 trims "<>" and drops the ESMTP parameters (session.go, case CodeRcpt) - so cleanAddress's cut at the first blank broke a quoted local part (B-G5). Postfix keeps the address in internal form (tok822: quotes off, backslash escapes taken) and logs it external (info_log_addr_form.c, quote_822_local(): quoted again only when not a dot-string); recipientAsLogged does the same, "ab"@x -> ab@x, "a b"@x stays. A recipient given twice in any case is one row: cleanup drops the duplicate, comparing folded (cleanup_out_recipient.c, been_here(), BH_FLAG_FOLD; A-31). A rewritten recipient is matched by orig_to=, which log_adhoc() writes only when it differs (A-9). The store's case-insensitive match stays, now with its reason from the source (D-S20). Stand (2026-10-14, make e2e, exit 0): new recipients_as_postfix_logs_them - RCPT TO "x y"@, "ab"@ and "X Y"@ sink.e2e.test give two rows, "x y"@sink.e2e.test and ab@sink.e2e.test, both sent. go vet and go test ./... pass on the server; a unit test drives the milter through go-milter's own client. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
0ed03b2094 |
log tailer: expiry, Postfix's time values and refusal lines as Postfix writes them
test / test (push) Canceled after 0s
Expiry (assumptions A-8). Postfix 3.7.11 logs the end of a message's queue lifetime once per message, "<id>: from=<..>, status=expired, returned to sender" (src/qmgr/qmgr_active.c, qmgr_active_done_25_generic()), with no line per recipient; the tailer waited for "to=..., status=expired", which no Postfix writes, and the sweep closed only queued rows, so such rows stayed deferred. The tailer now reads qmgr's line (ExpireMessage: each recipient's newest open row of the id becomes bounced); the sweep closes a deferred row too when Postfix no longer holds the message, and counts as missed only a final line (sent, bounced, expiry). The queue listing's '#' (forced expiry, showq_compat.c showq_message()) is read as a flag (B-PQ1). Time values (B-PQ10, B-PQ11): conv_time() ported - one number, one lower-case unit, the parameter's default unit when none (get_def_time_unit: d for the lifetimes, h for delay_warning_time), the rest ignored; postconf -h -x expands $name; bounce_queue_lifetime capped at maximal_queue_lifetime as qmgr_post_init() caps it, so LongestLifetime is the queue lifetime. Refusals (A-21, A-23, A-24, A-15): smtpd_whatsup() ends the NOQUEUE line with sasl_username=<login> on an authenticated session; the line is now matched to its row by client, login and sender, and a line without a login is no application's. The client= line's login ends at the comma. The "IPv6:" trim goes (namaddr is built from state->addr). A row recovered from cleanup's line says that line names the last recipient only. Stand (2026-10-14, make e2e, exit 0): new expired_message_is_closed - defer_transports=smtp makes the retry service defer the message, postsuper -e and postqueue -i force its expiry; mail.log has "<id>: from=<alerts@sender.e2e.test>, status=force-expired, returned to sender" and the row goes deferred -> bounced; refusals_show_postfixs_line passes with the login match. go vet and go test ./... pass on the server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
ce363f48dd |
logtail, health, panel: cite the source of what they rely on (assumptions part A)
Comments only, no behaviour changed. Each place the log tailer, Health and cmd/panel rely on another program now says where that is written, for the version the image runs: Postfix 3.7.11 (log_adhoc(), msg_logger_print(), smtpd_whatsup(), cleanup_milter.c, qmgr_active.c, postlogd.c, showq.c, pipe.c), logrotate 3.21.0, supervisor 4.2.5, proc(5). Measurements stay measurements and say where and when; test fixtures say whether they were seen or composed. Where the source contradicts the code the comment does not bless it but names the register entry (A-8 expiry, A-12 the ISO stamp no Postfix writes, A-15 the last recipient, A-21/A-23 sasl_username, A-33 postlogd's old inode, A-38 a socket file is not a listener, A-65 the certificate served); those are fixed in their own commits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
af130e268a |
findings of the reviews of audit stage 2b
Technical review (Opus) and security review (Fable) of 5be109b..5154bba;
reports in the session's scratchpad. Nothing above low from security; one
bug and three risks from the technical review. Migration 0009 is not
deployed, so it was extended.
- Bug: two refusals of one second swapped their NOQUEUE lines (each line
went to the newest lineless row). AttachRefusalLine now matches the
client address too (send_log.client_ip, written by the journal milter -
Postfix's line names the client and the sender, never the login: the
sasl_username part of the pattern never matched and could be fed by HELO,
security F1/F2), takes the nearest row in time and the oldest of equals,
and a line already on a row goes on no other (security F6).
- Waiting lines and recoveries lived only in memory while the saved offset
moved past them: a stop within 30 s lost them. The tracker now never saves
an offset past the oldest waiting item in its file (tracker.hold,
refusalLines.oldest) - for a recovery, the client= line that names its
login - so they are read again; the "once" guards make that harmless.
- A recovered row whose login the tailer did not see was nobody's: its
domain_id falls back to the envelope domain's.
- The fingerprint's file identity is the inode alone: the device number can
change across a host reboot and make the same log look rotated.
- A refused row carries its queue id; UpdateStatus's "newest row" now skips
refused rows, so a later message with that id does not flip it to sent.
- The alive mark: its age is read first and kept from then on, so asking
supervisord for 30 s cannot make it stale; the entrypoint removes it with
the sockets - a container restart is not the check's downtime, and a mark
from before would have counted the whole outage as "≈".
- Security F3: the sender check prints no ";" of the client's in the From:
text of its reply, and the tailer reads the envelope after the first ";"
- Postfix's own text.
- A refused row with a queue id and no stored lines says so (it was told the
mail.log text of 2.2.0); the fact's em dash is the house hyphen.
- Docs: architecture (client address, no login), guide § Health (warns on
either count), help topics log and sender-check (design
|
||
|
mixandClaude Opus 5.5
|
ac985d03a8 |
sender check: connections and messages counted apart; downtime estimated when supervisord is silent (stage 2b)
Following the design commit
|
||
|
mixandClaude Opus 5.5
|
65d916f26a |
logtail: a refusal the panel did not get to write is written from mail.log (stage 2b)
test / test (push) Canceled after 0s
The sender check answers 550 5.7.1 and writes the refused row through a
buffer a moment later; a panel stopped in between (a rollout, a kill) left
an application refused and a log that did not know it. A refusal at MAIL
FROM needs nothing: the journal milter writes its row before it answers.
- logtail.refusalLines remembers each queue id's login from smtpd's
"<id>: client=..., sasl_username=<login>" (4096 ids at most) and, on
cleanup's "<id>: milter-reject: END-OF-MESSAGE ...: 5.7.1 From: <line> -
<why>; from=<...> to=<...>", waits 30 s and hands it to
store.RecoverRefusal - an INSERT ... WHERE NOT EXISTS a refused row of that
queue id within two minutes: the row the panel wrote, or one recovered
before, means nothing is written. It is read from the saved offset on, so
a kill is covered as well as a clean stop - which a flush of the buffer at
shutdown would not.
- Migration 0009 (not deployed yet) gains send_log.recovered; the row is
written with the line's time, the login's domain id, reason "from", the
From: line as the check printed it (to the last " - " before ";").
- The message page: a fact "Written: from mail.log after a restart: the
panel stopped before it wrote the refusal".
Tests: store (not written when the panel wrote it; written once however
often); logtail (the login from client=, the From: line with a " - " in it,
nothing before the wait is over); view (the fact). e2e
refusal_recovered_from_the_log: a From: refusal, the panel stopped, its row
deleted and the tailer's saved place moved before the message's first line
(python3 in the container, as the panel's user - a kill cannot be timed to
that moment from outside); started again, the row is back, recovered, once.
Also fixed: handlers TestMessagePageOfARefusedMessage still expected the
old sentence; it failed on the server after
|
||
|
mixandClaude Opus 5.5
|
e166a2ebd6 |
refused messages: the page shows what Postfix wrote (stage 2b, 0009)
The Delivery log box of every refused row said "This message never reached
the queue, so Postfix wrote no delivery lines for it" - untrue: Postfix
wrote a line for each. The owner, 2026-10-13: the box is required on every
message page. Pictured for him first (dev/shots, the refused variant of
out-message: wrong From with its queue id lines, rate-limited with the
NOQUEUE line); the variant has no mockup screen of its own - the guard ties
screens to pages, and its outline is out-message's.
- Wrong From: cleanup gives the message a queue id before the headers are
judged; the sender check reads {i} at end-of-headers and writes it on the
refused row. Its lines - stored under the id like any other's - are on the
page, but only those from 10 minutes before to 1 minute after the refusal
(store.SendLogLines for a rejected row): Postfix frees the id at once, and
a later message with it lends nothing. No mail.log fallback for a refused
row, for the same reason.
- Rate-limited and wrong client: refused at MAIL FROM, no queue id. Migration
0009 adds send_log.refusal_line; logtail.refusalLines matches smtpd's
"NOQUEUE: milter-reject: MAIL from ...; from=<...>" (not port 25's, not
CONNECT/EHLO) and store.AttachRefusalLine puts it on the newest refused row
of that sender (and login, when the line names it) within two minutes; a
line whose row is not written yet waits up to 30 s (retried at each pass).
- The page: the queue id fact says "given, the message was not queued" or
"none - refused at MAIL FROM, before one is given"; the box's caption is
"The line from mail.log for this refusal" for the latter; a row from
before says the line was not kept. LogLineLevel colours a refusal by its
reply: 5xx red, 4xx amber (IPv6 clients included). The id in the fact is
not muted as in the picture: the kit has no muted inline.
Tests: store (attach by sender, login, time; not a queued row, not one an
hour off; a refused row's lines are its own minutes); logtail (attach, wait
and retry, port 25 and other lines ignored); milter (the refused row carries
{i}); view (facts, caption, line levels); handlers (the line on the page,
the sentence without it). e2e refusals_show_postfixs_line: a client-address
refusal made there, and the pages of a wrong From, a rate-limited and a
wrong client refusal each show Postfix's line.
schema-migrations row 9, CHANGELOG [Unreleased], audit plan line ticked,
roadmap 25/62.
Evidence on the test server for this tree: make e2e exit 0 (the new case
in 5.4 s); go test ./... ok but for two schema tests, fixed after the run
(the schema at 0009) - store tests ok locally; the full run follows before
the push.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
mixandClaude Opus 5.5
|
c452606175 |
logtail: the deferred counter remembers a place in the log, not a second (stage 2b)
test / test (push) Canceled after 0s
The counter passed over, while catching up after a start, every warning no later than the last one counted - by time; mail.log stamps whole seconds, so a second warning of that second, later in the file, was lost. - The tailer gives each line its place (Place: the file's fingerprint and the offset just past the line): follow tracks the bytes the reader hands over, readRotated likewise in mail.log.1. Run hands tr.line to the counter. - store.AddSenderCheckDeferred(at, n, place) counts, moves the last-deferral mark (never back - an upsert WHERE) and saves the place (settings sender_check_counted_place) in one transaction. - Catching up, a warning in the file of the last one counted is passed over by place (end <= counted end); in another file - a rotated log read again - by time, as before. The fingerprint carries the inode, so the place stays true across a rotation. Tests: two warnings of one second, an unclean stop between them, all lines read again - each counted once; the store saves the place with the count and keeps the latest deferral time. Existing counter tests pass a zero place (time rule). CHANGELOG [Unreleased], audit plan line ticked, roadmap 24/62. Evidence: go vet ./... clean; go test ./internal/store, ./internal/logtail ok (but for the Windows-only test); the server run follows before the push. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
4678015bbd |
logtail: a log rotated while the panel was down is read to its end (stage 2b)
At start, a saved offset whose fingerprint was not the current file's meant "start on the new file": what Postfix wrote between the panel's stop and the rotation was never read - delivery statuses, stored lines, refusals, inbound journal rows, the deferred counter. - tracker.resume marks the case (behind, prev); follow calls readRotated before the new file: it opens mail.log.1 (delaycompress leaves it plain), checks by fingerprint that it is the file the place belongs to, reads it from the saved offset to its end through handle (catching up: the once-protections apply) and flushes the stored lines. - Not found, not that file, or the offset past its end (rotated twice): logged, with the saved place's time - from when nothing was read. - The fingerprint gains the file's identity (device.inode, fileid_unix.go; empty elsewhere): two logs that begin alike are two files - the review-2026-08-followups row on collisions. A fingerprint saved before (hash alone) still matches by its hash (sameLog), so the first start after the upgrade resumes where it was. - store.LogtailState reads back updated_at. Tests: stop, append, rotate, append, start - both halves read, once each, the earlier line not again; a mail.log.1 that is another file is not read from the place; sameLog with an old and a new fingerprint. CHANGELOG [Unreleased], audit plan line ticked, roadmap 23/62. Evidence: GOOS=linux go vet ./internal/logtail clean; go test ./internal/logtail ok on Windows but for TestFollowTailsAndRotates; the server run follows before the push. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
60c7b85f84 |
findings of the reviews of audit stage 1
test / test (push) Canceled after 0s
Technical review (Opus) and security review (Fable) of 9efc9c7..a59e3d4; reports in the session's scratchpad. Fixed: - B1 (verified bug): store.UpdateStatus read before writing in its transaction; a commit by another connection in between (the dmarc-ingest process opens the same file) failed it at once with "database is locked" and the delivery line was lost. The first statement is now a write. - R3: a delivery line updated every row of its queue id and recipient; it now updates only the newest (Postfix reuses short ids). The sweep closes by row id (QueuedDelivery.ID, CloseUnreported(id)). - R4: a contradicted sweep keeps a mark, reason "reconcile-contradicted", instead of losing which row it was. - R1: contradicted cannot see a lost line (never read late). New counter reconcile_missed: at close, the sweep looks for the row's delivery line in mail.log and mail.log.1 (QueueLines). 0007 edited - it has run on no database but throwaway stands. - R2: queued_over_lifetime cannot measure "what removing the sweep would leave" while the sweep runs; its text now says rows stuck despite the sweep, and the rule uses closed - missed - contradicted. - cert-reload (R5, R6, R7, security medium/low): both files copied to .new and the copies checked as a pair before both are renamed; acts on the copy only when postconf says Postfix reads it; a failed reload is retried next pass; umask 077; says so when the mounted files cannot be read. The e2e case asserts Postfix reads the copy, the copy equals the mount, the key is 600 root. - Security low: a recipient with a quoted local part - the journal cut "a b"@x at the space, deliveryRe cut "a>b"@x at the ">" - matched no delivery line and was closed by the sweep. cleanAddress ends a bracketed address at the first ">" outside quotes; deliveryRe takes the recipient up to the ">, " before relay= (test fixtures given the relay= every real delivery line has). - Security low: postqueue -p's error carries at most 512 bytes of output. - Nits: addDeliveryLogDay refuses a column not on its list; the truncation test checks its message. - To stage 2b (page wording, mockup-first): the history step of a bounced row says the receiving server refused it, false for a row the sweep closed. To the owner when stage 2b is reached: since M13 the "deferred" figure counts whole messages too, which "Connections turned away" would not name. Found while verifying: the e2e's main positive path failed once with "553 not owned by user app1" - an smtpd idle from before the panel's reload took the connection in the reload's second. Measured on the stand (one-off probe, not committed): sending 0, 10, 40, 110 s after the save (two seconds at the soonest) always met an smtpd started after the reload. The e2e now waits for the sender map after each application it creates (waitForSenderMap: AUTH, MAIL, RCPT, RSET - no message, no row), and the level-1 case uses it instead of its own loop. Docs: architecture § Queue reconcile, schema-migrations (rows 7 and 8, chain head 8), CHANGELOG, audit plan (stage 1 note, the measured window, a new stage 2b line: 62 lines), roadmap 15/62. Evidence on the test server for this tree: go vet clean, go test ./... 23 packages ok, make e2e exit 0 (renewed certificate, hung panel, level 1 at 49/60). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
db36aa2c0e |
logtail: count the message a hung sender check defers in cleanup (audit M13)
Seen on the e2e stand, the panel frozen with SIGSTOP once DATA was
answered: the client gets 451 4.7.1 after 30 s, and the warning that names
the socket is cleanup's, not smtpd's:
postfix/cleanup[1339]: warning: milter unix:/run/selfpost/sender.sock:
can't read SMFIC_HEADER reply packet header: Connection timed out
postfix/cleanup[1339]: 7BCAF3C75DA0: milter-reject: END-OF-MESSAGE ...
4.7.1 Service unavailable - try again later; ...
The counter read smtpd lines only, so Health's "messages deferred" missed
the one case that tells a hung panel from a stopped one. milterLineRe now
takes smtpd and cleanup; cleanup's "milter unix:<socket>:" warning counts a
message each time (one cleanup process serves message after message, and the
first warning ends the message); smtpd keeps once per connection. The
composed fixture is replaced by the seen lines, with two messages through one
cleanup process and OpenDKIM hung in cleanup (not counted).
e2e panel_hung_mid_message_is_counted: freezes the panel after DATA's 354,
wants a 4xx at the end of DATA, logs what mail.log says, and waits for
Health's today count to grow. kill through sh -c: the image has no kill(1).
CHANGELOG [Unreleased], audit plan line ticked, roadmap 13/61.
Evidence on the test server for this tree (with the next commit's docs):
go vet clean, go test ./... 23 packages ok, make e2e exit 0 -
renewed_certificate_picked_up 1.2 s, panel_hung_mid_message_is_counted
31.7 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
mixandClaude Opus 5.5
|
63520f1678 |
logtail: deliveryRe anchored at the head of the line (audit M12)
The pattern that writes a row's status matched anywhere in a line: a port-25 reject line ending helo=<ID: to=<victim@...>, status=sent> parsed as (ID, victim, sent). parseDelivery now takes the stamp off with SplitTimestamp and matches `^host program[pid]: <id>: to=<...>, ... status=`, as queueLineRe and the inbound patterns do. Tests: the probed HELO line, the same text in a quoted sender on a qmgr line, the text after an smtpd line's queue id - none parse; a syslog-stamped line as Postfix 3.7 writes it parses. The existing cases unchanged. CHANGELOG [Unreleased], audit plan line ticked, roadmap 11/61. Evidence: go test ./internal/logtail ok but for TestFollowTailsAndRotates (Windows); the server run of the stage follows before the push. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
ffefa9d6df |
delivery log: measure the sweep before deciding its fate (audit H3, 0007)
Migration 0007 (DDL only): delivery_log_days, a row per UTC day, kept a year, shown nowhere - read off a copy of the database after a month by the decision rule written in the audit plan. - reconcile_closed: the sweep adds the rows it actually closed. - reconcile_contradicted: store.UpdateStatus, in one transaction, counts the rows with reason "reconcile" a delivery line now speaks of, writes the log's status and clears the reason - so a re-read line counts once. - tailer_blind_seconds: follow() tracks stretches in which the file is gone or reopening fails (logtail.blindness); stored when a stretch ends and every minute while it lasts; under 3 s not counted (the rotation's tick between rename and create loses nothing). - queued_over_lifetime: every sweep overwrites the day's level with the rows queued past LongestLifetime + MaximalBackoff (a message Postfix still works on stays queued until its first retry after the lifetime). - retentionLoop prunes days older than a year. Tests: store (sums per day, the level overwritten, under a second nothing, prune; a contradicted sweep counted once and its reason cleared; count of queued rows); logtail (the sweep counts what it closed; countStuck records the level; blindness - a one-tick gap not counted, 75 s stored as 60 + 15 from where the last part ended). Schema tests at user_version 7. schema-migrations.md: chain head 7 (it still said 5 since 2.4.0), row 7. CHANGELOG [Unreleased], audit plan line ticked, roadmap 10/61. Evidence: go vet ./... clean; go test ./internal/store ok, ./internal/logtail ok but for TestFollowTailsAndRotates (Windows); the server run of the stage follows before the push. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
9adb44845c |
logtail: the sweep closes only queued rows, with reason "reconcile" (audit H3)
reconcile() closed a row through UpdateStatus (WHERE queue_id AND to_addr), which could overwrite a "sent" the tailer wrote between the listing and the close, and rewrite an old row of a reused queue id. store.CloseUnreported sets status bounced and reason "reconcile" (store.ReasonReconcile; the 0006 column, no schema change) WHERE ... AND status = 'queued'; the sweep counts rows actually closed. Tests: store - a queued row is closed with the reason, a sent row is left alone (0 rows); logtail - the sweep's close goes through CloseUnreported. CHANGELOG [Unreleased], audit plan line ticked, roadmap 9/61. Evidence: go vet ./... clean; go test ./internal/store ok, ./internal/logtail ok but for TestFollowTailsAndRotates (Windows rename of an open file, fails here always); the server run of the stage follows before the push. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
c817f0833c |
from-header-binding: what the reviews had left as found, and address extensions
test / test (push) Canceled after 0s
The owner on the three things the reviews' commit left as found: can they be fixed? They can. - Downtime that crossed a UTC midnight was all booked on the day of the restart: it is cut at the midnights between (byDay). - A message with the null envelope sender - a bounce an application writes itself - left a refusal row without a domain, which no domain administrator would see: the row takes the domain the login is bound to. - A log read again after an unclean stop counted its deferrals again: while the tailer catches up, a warning no later than the last one counted is passed over. The mark is kept to the fraction of a second the log gives. And on the check being stricter than Postfix for an address with an extension (invoices+x@ passed in the envelope, was refused in the header): it makes Postfix's second try now, the address without its extension. recipient_delimiter was the distribution's default; it is set in postfix-config.sh, since the check depends on it. e2e sends one through. Test server, this tree: go vet ok, go test 23 packages ok, make e2e ok (339 s). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
89fd0a6116 |
from-header-binding: findings of the technical and security reviews
test / test (push) Canceled after 0s
Both reviews were of
|
||
|
mixandClaude Opus 5.5
|
c171fb5bc7 |
from-header-binding: the sender check, what it costs, its pages; guide, architecture, security
test / test (push) Canceled after 0s
A message's From: line is held against the application that sends it, under the rule Postfix holds the envelope sender to. OpenDKIM chooses the key by that line, so until now an application could have mail signed as another domain of the server or as the server itself. The check (internal/milter/sender.go) - A role of the panel on a socket of its own (SENDER_CHECK_SOCKET), first in smtpd_milters and strict: default_action=tempfail. The journal milter keeps its socket and stays fail-open. postfix-wrapper waits for the new socket. - It reads no database: what an application may send as comes from Postfix's own sender_login_maps, what the server signs for from OpenDKIM's SigningTable. Both are read again when they change; one that cannot be read defers mail. - One From: field, one address, allowed by the envelope's lookup; then no other name of the SigningTable may appear anywhere in the raw line. Refused: 550 5.7.1 naming the line, reduced to printable ASCII. - A refusal is a send-log row with reason "from" and the header address (migration 0006), written by another goroutine and not waited for. The journal's refusals record their reason too (limit, client), and the notifier takes only a limit's. What it costs (docs/plans/from-header-binding.md, Statistics) - The role marks itself alive every 2 s; at start the gap since the mark is downtime, counted from Postfix's start if later and not at all when supervisord says Postfix is not running. - The log tailer counts the connections Postfix turned away: a warning that names the socket, then milter-reject "Service unavailable" from the same smtpd. Seen on the stand: the client is refused from CONNECT on and leaves at AUTH (454), so no MAIL line is ever written. - sender_check_days, a row per UTC day, kept a year. Pages - Health: the card "Sender check", the socket's row, and the cards layout - boxes numbered by importance in two stacks read in rows; the left stack polls, the three readings of the right one are swapped in by id. - Outbound log: a refused row says why in its tag (wrong From, wrong client, rate-limited); the message's page shows the From: line and what the application may send as. - Help: topic "Sender check", word for word the mockup's. Found by the stand, fixed here: the names for the second look came from the sender bindings, and a domain with a key and no application yet was not among them. e2e: negative_journal_milter_fail_open is gone - it proved that mail passes with the panel down, which is no longer so. In its place sender_check_holds_the_from_line (another domain's From:, the hostname's, a stranger's, an address off the list, another sender in the display name - refused; own addresses - delivered) and panel_down_defers_mail (4xx with the panel stopped, the container alive, mail again after, downtime and one deferred message on Health, the card warning). Test server, this tree before the last cosmetic edits (a fixture row, a fact's size, docs): go vet ok, go test 23 packages ok, make e2e ok (337 s). Locally after them: internal/web/view ok. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
a990e7cc23 |
delivery-log-storage: findings of the technical and security reviews
test / test (push) Canceled after 0s
Neither review (read-only agents, Opus and Fable, over ae06d8b..815236a) has a Critical or High finding. What they found and what was done: - A stranger on port 25 could grow the table: every queue-id line of inbound mail was stored and pruned an hour or more later, his own message-id among them. The tailer now leaves out the lines of a queue id that postfix/inbound/smtpd opened, until qmgr removes the message or another listener opens one under the id; 4096 such ids are remembered. - A commit, so a disk sync, per stored line on the connection the journal milter shares, and a backlog written line by line. Lines are now written in one transaction per pass over the log, 500 at most, before the read offset is saved (InsertSendLogLines; follow() calls the keeper's flush). - The prune asked the send log once per line and ran hourly. It asks once per queue id and runs at start and with the six-hourly sweep; with port-25 mail left out there is little without a row to wait for. - The 4096-byte cut could split a character; it keeps characters whole. - security.md did not say that a backup now carries these lines, and the plan said the time window excludes a reused queue id where it only limits the chance to the row's queue lifetime. Both are corrected; the collision is recorded as an accepted risk with its trigger. - Tests that did not prove their comments: the prune is now tested with the sweep, a rejected row next to the prune, batching and the change from catching up to following, port-25 mail left out and its id reused. Left as found and written down in the plan: lines the tailer never reads (a file rotated away while the panel was down), and a message in the queue across the upgrade, which shows only its later lines. On the test server, for this tree: go vet clean, go test ./... 22 packages ok, make e2e ok. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
81df131217 |
delivery-log-storage: the tailer keeps the lines of queued messages
Every mail.log line headed by a queue id is written to send_log_lines as the tailer reads it, dated by the line's own stamp. The id is taken from the head of the line only - the tail is the far end's text - and has to look like one: "NOQUEUE:" and "warning:" stand in the same place. The tailer does not check for a send-log row first: a message's first lines come before the journal milter writes it. Lines that never get a row are removed by the prune, which deletes lines older than an hour whose queue id no row carries. It runs at start, after each send-log sweep and hourly. While the tailer catches up at start a line is written only if it is not there already, as the inbound journal does: its place is saved every few seconds, so an unclean stop means reading some lines twice. Checked on Windows: go vet clean; go test ./internal/logtail passes except TestFollowTailsAndRotates, which fails there on renaming an open file and always did. New tests: which line heads are queue ids, a line dated by its stamp, no second copy after a rewound offset, the prune cadence and cutoff. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mixandClaude Opus 5.5
|
89774ab847 |
inbound-antispam-panel: three things the reviews left as found
test / test (push) Canceled after 0s
The owner asked for them to be put right (2026-10-12).
- A refusal is not logged twice after an unclean stop. The tailer
saves its place every five seconds, so on start it reads some
lines again; for the lines it reads on start a row is written only
if the same row is not there already (store.InsertInboundSpamOnce:
same moment, client, sender, recipient, decision, engine, detail).
Lines read as they are written are always new, so two identical
refusals in one second stay two rows.
- The Inbound log offers Deny / Allow sender only for a sender the
lists would take (inbound.Listable): an envelope sender may be
spelt in ways the form refuses, and the link led to a refusal.
- A network of one address is stored as the address: 192.0.2.7/32
and 192.0.2.7 were two entries of one list.
Checked on the test server, uncommitted tree: go vet ./... ok,
go test ./... ok (22 packages), make e2e ok.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
mixandClaude Opus 5.5
|
c731a98a0f |
inbound-antispam-panel: findings of the technical and security reviews
test / test (push) Canceled after 0s
Steps 11 and 12 of docs/plans/inbound-antispam-panel.md: 12/12. Two
read-only reviews of d65d632..HEAD by different models (Opus,
technical; Fable, security), each given the code and questions, not
conclusions. Nothing Critical or High in the feature; the High the
technical review found outside it is the commit before this one.
Log tailer (internal/logtail/inbound.go):
- the reply of a refusal ends at the first "; from=<", Postfix's own,
and the HELO name after it is whatever is left. Taking the longest
reply let a HELO name that spelt "; from=<..> to=<..> proto=.."
supply the row's sender, recipient and reply - and so make a
refusal read as the deny list's;
- an address with a quote, a bracket or a space is not kept: a quoted
sender could otherwise supply the recipient;
- the journal is cut to its row cap every 500 rows written, not only
every six hours;
- a queue id is forgotten when the message is queued or the id is
opened by another listener, and a stale place in the queue of
arrivals no longer drops a live id;
- milter-hold lines are not read: the held message reaches the
journal milter, which writes its row - it was written twice.
Lists (internal/inbound/lists.go): a change of the table and the
rewrite of the files are one step under a mutex; a failed rewrite puts
a removed entry back and rewrites after taking a new one out.
Store: the row cap removes the oldest by time, as the page orders them.
Page: on a sheet a refusal's text is a Reply whatever it starts with.
Help and the guide say that an allow entry for a sender address trusts
an address anyone can claim (design commit before this one).
docs/security.md: the two reviews; the roles table gains Inbound > Log
and Filter lists; Mail path describes the milters, the tailer and the
lists; four accepted risks - the filter's verdict while it is down, an
allow entry by sender address, the list files' mode, log rows held by
domain name. The plan's stale lines (size and rate refusals, who reads
the journal) are corrected.
Left as found, told to the owner: the journal's rows after an unclean
stop may be written twice (the tailer saves its place every five
seconds); a "Deny sender" link is offered for an address the form then
refuses; 1.2.3.4 and 1.2.3.4/32 are two entries.
Checked on the test server, uncommitted tree: go vet ./... ok,
go test ./... ok (22 packages), make e2e ok.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
mixandClaude Opus 5.5
|
27fd355e53 |
inbound-antispam-panel: refused inbound mail from mail.log; retention
Step 4/12 of docs/plans/inbound-antispam-panel.md. The log tailer writes an inbound_spam_log row for mail port 25 refused (internal/logtail/inbound.go): a "reject:" line of the port-25 smtpd is Postfix's own policy; a milter-reject / milter-hold / milter-discard line of cleanup is the filter's verdict. Only a recipient at an inbound domain is a row, so "Relay access denied" for a stranger's domain is not. The row is dated by its log line; the reply text is cut to 255 bytes of printable ASCII; a reply starting with 4 is a tempfail. To tell port 25 from 465 / 587 the listener now logs as postfix/inbound (syslog_name). cleanup logs under one name for every listener, so its verdict counts only for a queue id the port-25 smtpd was seen to open (4096 remembered). Lines are matched from the head, after timestamp and host: their tail is the sender's own text. Retention: PruneInboundSpam every six hours - older than the inbound_spam_log_retention_days setting (seeded from INBOUND_SPAM_LOG_RETENTION_DAYS, 90, range 7-365), then down to 10 000 rows. The setting has no field in the panel yet. Known limits, in architecture.md: a log line names one recipient, so a message refused at end-of-message is one row; a line re-read after an unclean stop would be a second row. Checked on the test server, uncommitted tree: go vet ./... ok, go test ./... ok (22 packages), make e2e ok - the unknown recipient at the inbound domain refused on port 25 has its row (reject by postfix), the stranger's domain has none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
mix
|
60b743e6c2 |
release: 1.6.0: 30-day send stats and optional auto level-2 rate limits
30-day sending statistics per domain and application, plus optional auto level-2 rate limits derived from average send rate. Added: - Domain page **Sending statistics** card: total messages, peak and average - Level-2 rate limit **Manual** / **Auto** mode on domain and application - Domain export JSON now includes rate limits (`mode`, `auto_multiplier`, Changed: - Auto mode uses the level-1 window (`RATE_LIMIT_WINDOW_SECONDS`); manual Full detail: CHANGELOG.md (1.6.0). |
||
|
mix
|
dca7d99c1b |
release: 1.5.0: Send-log retention configurable in panel Settings
Send-log retention is configurable from panel **Settings** (global administrator); `SEND_LOG_RETENTION_DAYS` seeds the initial default only. Added: - Panel **Settings** (global administrator): **Send log retention (days)** — Changed: - Deliveries list and delivery detail pages show the configured retention Full detail: CHANGELOG.md (1.5.0). |
||
|
mix
|
75207f0d1e |
release: 1.0.0: Public 1.0 image; delivery log; docs and SECURITY.md (incl. 1.0.1)
[1.0.0]
Added:
- A **Delivery log** on each delivery's page (`/deliveries/{id}`): the
- A **History** block on the same page: the journal's two timestamps stated as
Fixed:
- Postfix config: copy TLS cert/key from the (often `:ro`, host-owned) mount
- Postfix config: allow `maillog_file` under `/data` via
- E2e: read `/data/setup-token` via `docker compose exec` (file is `0600`
- Entrypoint: check `SELFPOST_HOSTNAME` before `/data` setup (so a bad identity
- E2e gate: wait for host-published `/healthz` before panel setup, and stop
… (3 more in CHANGELOG)
Changed:
- Document what `.spbk` and `.spde` stand for (SelfPost backup / SelfPost domain
- Docs aligned with the code: setup URL is `/setup/<token>` (README and
- Full backups no longer carry `/data/log`. It is Postfix's raw log plus its
- Monitoring screens (status, mail queue, system log, deliveries) use adaptive
- Documentation package consolidated into `docs/development.md`: Documentation
… (4 more in CHANGELOG)
[1.0.1]
A documentation and packaging release: no change to the mail path, the database, or the on-disk layout. Upgrading is a tag bump.
Added:
- `SECURITY.md` — how to report a vulnerability privately (GitHub private
- [docs/plans/](docs/plans/) — one document per agreed extension: the optional
Changed:
- The panel's **Account** entry is now called **Settings** — nav link, page
- The signed-in name in the panel's nav is now labelled `User:`, so it reads as
- [docs/product.md](docs/product.md) reframes the future line: the inbound
- [docs/security.md](docs/security.md) is now in English, matching the rest of
- The README documentation table now points at `SECURITY.md` for reporting a
… (2 more in CHANGELOG)
Fixed:
- The panel's static assets are served with a content ETag and
Squashed CHANGELOG sections: [1.0.0], [1.0.1].
Full detail: CHANGELOG.md (1.0.0, 1.0.1).
|
||
|
mix
|
5e5d157496 |
release: 0.5.0: Pre-1.0 security review, domain-admin role, doc consolidation
A bounce could be recorded as a successful delivery. The log-tailer's Fixed: - A bounce could be recorded as a successful delivery. The log-tailer's - Log-tailer resumes where it stopped instead of jumping to end-of-file on - Level-2 rate limit no longer overshoots under concurrency: messages that Changed: - The project has a single public home: `github.com/mixeme/selfpost`. Codeberg - Code comments no longer cite the archived specification. References like - `docs/code-review.md` is gone. Its plan is finished — phases 0 (bar the - `docs/architecture.md` gained a *Code layers* section: a diagram of - Phase 1 doc/code hygiene (`docs/code-review.md`): removed ~30 stale … (1 more in CHANGELOG) Security: - Pre-release security review (plan § D, model Fable, 2026-08-06): full pass - `saslpasswd2` argv: the application login is now passed after a `--` Added: - Optional password encryption for the two secret-bearing downloads (plan - Domain import accepts an encrypted export: the envelope is detected by its - `selfpost-backup` writes encrypted archives and reads them back: - `TestParseDelivery` covers the exotic mail.log shapes the review asked for - docs: README *Encrypting a backup or export*; `docs/security.md` § … (14 more in CHANGELOG) Removed: - docs: `security.md` — accepted risk «restore old backup revives session rows» Changed: - docs: `implementation-plan.md` trimmed to the sole open v1.x gate — pre-release - docs: `architecture.md` — sessions (SQLite, idle renew, password change), - docs: `development.md` — expanded e2e stack and `release.yml` matrix workflow. - docs: `security.md` — accepted risk for send-log rows stuck at `queued` after - docs: `roadmap.md` — optional send-log / `mail.log` follow-ups under v1.x tail. … (10 more in CHANGELOG) Full detail: CHANGELOG.md (0.5.0). |
||
|
mix
|
8ce050ef7f |
release: 0.1.0: Phases 0-11: panel, Postfix/DKIM, milter, backup, Docker image
Initial feature-complete implementation of the v1.0 specification (phases 0-11 of `docs/implementation-plan.md`). Added: - Panel (Go, single static binary) with SQLite persistence, one-time - Domain management with per-domain DKIM (RSA-2048, generated in pure Go) and - Application (sender identity) management: SASL credentials via `sasldb2`, - Full Postfix relay config generated from env at container start: SMTPS 465, - Journal milter (pure Go, `go-milter`) recording every send to `send_log`; … (6 more in CHANGELOG) Full detail: CHANGELOG.md (0.1.0). |