docs: stage 2b of the audit - the log tailer, refusals, the Sender check card
test / test (push) Canceled after 0s
test / test (push) Canceled after 0s
architecture.md - § Log tailer: the fingerprint carries the inode; a log rotated while the panel was down is read from mail.log.1 to its end, or the gap is logged; a refusal's line (the queue id of a From: refusal, the NOQUEUE line put on a MAIL FROM refusal's row) and the refusal written from mail.log when the panel had not. - § Sender check, what it costs: supervisord asked for half a minute, then the estimate and ≈; connections and messages in two columns; the count's saved place in the log. guide.md - § Outbound log: a refused message's page shows Postfix's line(s); rows from before, and a row written from mail.log, say so. - § Overview and Health: the two columns and ≈. schema-migrations row 9 was written with the migration. Audit plan line ticked: stage 2b's lines are all done. Roadmap 30/62. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+44
-16
@@ -278,7 +278,10 @@ server itself.
|
|||||||
starts without one). At start the time since the mark is the check's
|
starts without one). At start the time since the mark is the check's
|
||||||
downtime - counted from Postfix's own start if that was later, and not at
|
downtime - counted from Postfix's own start if that was later, and not at
|
||||||
all when supervisord says Postfix is not running, since mail was then
|
all when supervisord says Postfix is not running, since mail was then
|
||||||
stopped for another reason. The log tailer counts the mail: Postfix
|
stopped for another reason. Supervisord is asked for half a minute; when it
|
||||||
|
does not answer, the time since the mark is taken as an upper bound and the
|
||||||
|
day is marked estimated (`sender_check_days.estimated`), which the card
|
||||||
|
shows as ≈. The log tailer counts the mail: Postfix
|
||||||
writes a warning naming the socket and then, from the same smtpd process,
|
writes a warning naming the socket and then, from the same smtpd process,
|
||||||
`milter-reject: … 451 4.7.1 Service unavailable` from the connection on
|
`milter-reject: … 451 4.7.1 Service unavailable` from the connection on
|
||||||
(the client is told `454 4.3.0` at `AUTH` and leaves), so what is counted
|
(the client is told `454 4.3.0` at `AUTH` and leaves), so what is counted
|
||||||
@@ -291,14 +294,17 @@ server itself.
|
|||||||
unix:<socket>: can't read SMFIC_HEADER reply packet header` and refuses the
|
unix:<socket>: can't read SMFIC_HEADER reply packet header` and refuses the
|
||||||
message with `4.7.1` (seen on the e2e stand with the panel frozen). One
|
message with `4.7.1` (seen on the e2e stand with the panel frozen). One
|
||||||
cleanup process serves message after message, so each such warning is
|
cleanup process serves message after message, so each such warning is
|
||||||
counted as a message. Both go into
|
counted as a message. The two are counted apart (the owner, 2026-10-14):
|
||||||
`sender_check_days`, a
|
connections turned away in `deferred`, messages deferred mid-message in
|
||||||
row per UTC day kept a year, and Server › Health shows today, 30 days and
|
`messages` of `sender_check_days`, a row per UTC day kept a year, and
|
||||||
12 months. The second figure also sees a panel that is running and not
|
Server › Health shows today, 30 days and 12 months in a column each. The
|
||||||
answering, which the mark does not. Downtime is cut at the UTC midnights
|
counts also see a panel that is running and not answering, which the mark
|
||||||
it crossed. A log read again after an unclean stop does not count again:
|
does not. Downtime is cut at the UTC midnights it crossed. A log read again
|
||||||
while the tailer catches up, a warning no later than the last one counted
|
after an unclean stop does not count again: the count and the place in the
|
||||||
is passed over.
|
log it reached (the file's fingerprint and the offset past the line) are
|
||||||
|
saved in one transaction, and while the tailer catches up a warning at or
|
||||||
|
before that place in the same file is passed over - a second warning of
|
||||||
|
the same second is not lost; in another file the time decides.
|
||||||
- **Why it is not a process of its own.** Decided with the owner: measure
|
- **Why it is not a process of its own.** Decided with the owner: measure
|
||||||
first. The role has a socket of its own so that the journal stays
|
first. The role has a socket of its own so that the journal stays
|
||||||
fail-open, and moving it out later changes where it is started and nothing
|
fail-open, and moving it out later changes where it is started and nothing
|
||||||
@@ -396,13 +402,35 @@ name), and a pattern that could start there let a stranger's HELO name file a
|
|||||||
delivery (audit 2026-10-12, M12).
|
delivery (audit 2026-10-12, M12).
|
||||||
|
|
||||||
**Read offset is persisted** (`logtail_state` table, in the baseline schema): the
|
**Read offset is persisted** (`logtail_state` table, in the baseline schema): the
|
||||||
tailer stores its position plus a fingerprint of the log's first 512 bytes, and
|
tailer stores its position plus a fingerprint of the log - the hash of its first
|
||||||
on start resumes from it, parsing the tail written while the panel was down. If
|
512 bytes and the file's device and inode, so two logs that begin alike are two
|
||||||
the fingerprint no longer matches (rotated or recreated in the meantime) it reads
|
files and a renamed file keeps its fingerprint - and on start resumes from it,
|
||||||
the current file from the start; re-parsing lines is harmless because
|
parsing the tail written while the panel was down. If the fingerprint no longer
|
||||||
`UpdateStatus` writes the same status onto the same row. Only a first-ever start,
|
matches, the log was rotated in the meantime: the tailer first opens
|
||||||
with nothing stored, begins at end-of-file, so installing the panel does not
|
`mail.log.1` (left plain by `delaycompress`), and if that is the file the saved
|
||||||
replay a pre-existing log.
|
place belongs to it reads it from there to its end, then the new file from its
|
||||||
|
start - what Postfix wrote between the stop and the rotation is not lost. A
|
||||||
|
`mail.log.1` that is another file (rotated twice) is logged with the time from
|
||||||
|
which nothing was read. Re-parsing lines is harmless because `UpdateStatus`
|
||||||
|
writes the same status onto the same row. Only a first-ever start, with nothing
|
||||||
|
stored, begins at end-of-file, so installing the panel does not replay a
|
||||||
|
pre-existing log.
|
||||||
|
|
||||||
|
**A refusal's line, and a refusal the panel did not write.** A message refused
|
||||||
|
for its `From:` line has a queue id (cleanup gives one before the headers are
|
||||||
|
judged); the sender check writes it on the row, and the message's page shows
|
||||||
|
the lines stored under it - only those of the minutes around the refusal, since
|
||||||
|
Postfix frees the id at once. A message the journal milter refused at
|
||||||
|
`MAIL FROM` has none: the tailer reads smtpd's `NOQUEUE: milter-reject: MAIL
|
||||||
|
from …; from=<…>` and puts it on the newest refused row of that sender (and
|
||||||
|
login, when the line names it) within two minutes (`send_log.refusal_line`),
|
||||||
|
retrying for 30 s when the row is not written yet. The sender check writes its
|
||||||
|
row through a buffer after answering 550; a panel stopped in between wrote
|
||||||
|
none. The tailer remembers each queue id's login from smtpd's `client=` line,
|
||||||
|
and 30 s after cleanup's `milter-reject: END-OF-MESSAGE … 5.7.1 From:` line it
|
||||||
|
writes the row from the line if no refused row of that id exists
|
||||||
|
(`send_log.recovered`, shown on the page). It reads from its saved offset, so
|
||||||
|
a kill is covered as well as a clean stop.
|
||||||
|
|
||||||
**Queue reconcile** is the backstop for what the log cannot explain at all: a
|
**Queue reconcile** is the backstop for what the log cannot explain at all: a
|
||||||
row still `queued` more than two minutes after it was accepted, whose queue id
|
row still `queued` more than two minutes after it was accepted, whose queue id
|
||||||
|
|||||||
@@ -834,7 +834,7 @@ order - each stands on the one before.
|
|||||||
M13 fix on, not yet deployed when this was decided. The mockup `health`
|
M13 fix on, not yet deployed when this was decided. The mockup `health`
|
||||||
and help topic `sender-check` first (`design:` commit), then the page and
|
and help topic `sender-check` first (`design:` commit), then the page and
|
||||||
`view/help.go`, word for word. - **Sonnet**
|
`view/help.go`, word for word. - **Sonnet**
|
||||||
- [ ] Docs: architecture § Log tailer (the rotated file, the saved place, a
|
- [x] Docs: architecture § Log tailer (the rotated file, the saved place, a
|
||||||
refusal's line and a recovered row), guide § Outbound log and § Health,
|
refusal's line and a recovered row), guide § Outbound log and § Health,
|
||||||
schema-migrations.md for the mark on `sender_check_days`. — **Sonnet**
|
schema-migrations.md for the mark on `sender_check_days`. — **Sonnet**
|
||||||
|
|
||||||
|
|||||||
+14
-8
@@ -404,14 +404,16 @@ readings, in the order of what stops mail first:
|
|||||||
sockets** (OpenDKIM, the sender check and the journal-milter).
|
sockets** (OpenDKIM, the sender check and the journal-milter).
|
||||||
- **Sender check** — what the strict check of the `From:` line has cost
|
- **Sender check** — what the strict check of the `From:` line has cost
|
||||||
today, in the last 30 days and in the last 12 months: for how long the
|
today, in the last 30 days and in the last 12 months: for how long the
|
||||||
panel did not answer while Postfix was running, and how many messages
|
panel did not answer while Postfix was running, how many connections
|
||||||
Postfix deferred because of it (see [Who an application may send
|
Postfix turned away because of it, and how many messages it deferred in
|
||||||
|
the middle (see [Who an application may send
|
||||||
as](#who-an-application-may-send-as)). A restart of the whole container is
|
as](#who-an-application-may-send-as)). A restart of the whole container is
|
||||||
not counted: mail was stopped for another reason then. What is counted as
|
not counted: mail was stopped for another reason then. A connection turned
|
||||||
a message is a connection Postfix turned away - it refuses before a
|
away is refused before a message is offered, so a monitor that only
|
||||||
message is offered - so a monitor that only connects during an outage is
|
connects during an outage is counted too; a message deferred in the middle
|
||||||
counted too; and when the panel hangs instead of stopping, the message it
|
is one the panel stopped answering for after it had begun. A time marked ≈
|
||||||
hung on. Days are UTC days. The card warns when
|
is at most what it shows: at that start supervisord could not say since
|
||||||
|
when Postfix ran. Days are UTC days. The card warns when
|
||||||
a message was deferred in the last 24 hours. If these numbers do not stay
|
a message was deferred in the last 24 hours. If these numbers do not stay
|
||||||
at nothing, the panel is in the way of your mail more than it should be -
|
at nothing, the panel is in the way of your mail more than it should be -
|
||||||
report it.
|
report it.
|
||||||
@@ -1271,7 +1273,11 @@ refused row says why in its tag: `rate-limited` (a [level-2 rate
|
|||||||
limit](#rate-limiting--level-2-domain-and-application)), `wrong client` (the
|
limit](#rate-limiting--level-2-domain-and-application)), `wrong client` (the
|
||||||
application's client IP list) or `wrong From` (see [Who an application may
|
application's client IP list) or `wrong From` (see [Who an application may
|
||||||
send as](#who-an-application-may-send-as)); a row refused before `2.4.0`
|
send as](#who-an-application-may-send-as)); a row refused before `2.4.0`
|
||||||
says `rejected`;
|
says `rejected`. A refused message's page shows what Postfix wrote about the
|
||||||
|
refusal: for `wrong From` the lines of the queue id the message had, for the
|
||||||
|
other two the one `NOQUEUE` line; a row refused before `2.5.1` says the line
|
||||||
|
was not kept, and one the panel wrote from `mail.log` after a restart - it
|
||||||
|
had stopped before writing the refusal itself - says so;
|
||||||
*Details* opens that row's own page (`/outbound/log/{id}`). That page carries
|
*Details* opens that row's own page (`/outbound/log/{id}`). That page carries
|
||||||
the sending domain, the application it was submitted under, the Postfix
|
the sending domain, the application it was submitted under, the Postfix
|
||||||
queue id and the journal id, beside the message's history — when it was
|
queue id and the journal id, beside the message's history — when it was
|
||||||
|
|||||||
+1
-1
@@ -42,7 +42,7 @@ in `git log` and [CHANGELOG.md](../CHANGELOG.md).
|
|||||||
| from-header-binding | Hold a message's `From:` header against the application that sends it | shipped `2.4.0` | 9/9 | [plans/from-header-binding.md](plans/from-header-binding.md) |
|
| from-header-binding | Hold a message's `From:` header against the application that sends it | shipped `2.4.0` | 9/9 | [plans/from-header-binding.md](plans/from-header-binding.md) |
|
||||||
| panel-redesign | Panel redesign (accepted mockups, design contract) | shipped `2.0.0` | 27/27 | [plans/panel-redesign.md](plans/panel-redesign.md) |
|
| panel-redesign | Panel redesign (accepted mockups, design contract) | shipped `2.0.0` | 27/27 | [plans/panel-redesign.md](plans/panel-redesign.md) |
|
||||||
| schema-squash | Squash SQLite migrations into a 2.x baseline | built in panel-redesign (stage 1) | — | [plans/panel-redesign.md](plans/panel-redesign.md) |
|
| schema-squash | Squash SQLite migrations into a 2.x baseline | built in panel-redesign (stage 1) | — | [plans/panel-redesign.md](plans/panel-redesign.md) |
|
||||||
| audit-2026-10-12 | Follow-ups of the 2026-10-12 full-tree code audit, seven stages | in progress (stage 0 shipped `2.4.2`, stages 1+2 `2.5.0`; next stage 2b → `2.5.1`) | 29/62 | [audit-2026-10-12.md](audit-2026-10-12.md) |
|
| audit-2026-10-12 | Follow-ups of the 2026-10-12 full-tree code audit, seven stages | in progress (stage 0 shipped `2.4.2`, stages 1+2 `2.5.0`; next stage 2b → `2.5.1`) | 30/62 | [audit-2026-10-12.md](audit-2026-10-12.md) |
|
||||||
|
|
||||||
**Recommended order** (not binding; owner, 2026-10-09: feature order follows
|
**Recommended order** (not binding; owner, 2026-10-09: feature order follows
|
||||||
development convenience): **panel-redesign** shipped as `2.0.0` (2026-10-11) and
|
development convenience): **panel-redesign** shipped as `2.0.0` (2026-10-11) and
|
||||||
|
|||||||
Reference in New Issue
Block a user