55 Commits
Author SHA1 Message Date
mixandClaude Opus 5.5 b905f3e560 panel-notifications: Account, Server > Settings, Server > Health, help; e2e
test / test (push) Canceled after 0s
The pages of the mockups accepted in 87fa0a4, built from the kit's partials
(written by a Sonnet agent into the working tree, reviewed here):

- Account: "Send me notifications" in the Profile form, saved with the
  profile; an unticked box is off.
- Server > Settings: the instance switch (box 03), and box 04 Notification
  mail with the DKIM and SPF records of the server's own name, drawn with the
  record partial of a domain's DNS page and checked in DNS. Sending rate
  limits are box 02 in the left column. A record that is not in place is
  tagged as a warning, not a failure: notifications are sent without it.
- Server > Health: card 06 Notification mail with the state of each record
  and a link to Settings, beside Configuration (07); part of the polled
  fragment; Re-check DNS forces it too. Overview does not run this check.
- help.go: the Notifications topic and the clauses in Settings and Your
  account, word for word with help.html.

If the server's key cannot be read the pages still render: the DKIM record
is left out and SPF is shown unchecked.

e2e: the administrator gets a mailbox behind the sink and the server's DKIM
record from Settings is published; the level-2 refusal of the next step must
then arrive as a notification from postmaster@mail.e2e.test, signed as that
name with the key Settings shows, and must not be a row of the Outbound log.
The rotation step now waits for Postfix to be running: it follows a step that
restarts the container, and logrotate's `postfix reload` failed in between.

On the test server, for this tree: go vet clean, go test ./... 23 packages
ok, make e2e ok (147 s); the notification reached the sink 28 s after the
refusal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 16:21:48 +03:00
mixandClaude Opus 5.5 c9027381e5 panel-notifications: guide, architecture, security; refusals looked for every minute
guide.md gains Notifications: the four things, when each is said and said
again, who is told, the two switches, and the two DNS records of the server's
own name that make the mail arrive. architecture.md: the notifier as the
fifth role, its readings, state, recipients and sending; the server's own key
in OpenDKIM's tables; non_smtpd_milters. security.md: the panel submits mail
of its own - how, to whom, and what text from outside can reach a body.

That text - a queued message's sender, a DNS finding - is reduced to
printable ASCII and cut before it is printed. Refused mail is looked for
every minute instead of every five: it is one indexed query, and a refusal is
mail not being sent now. The image's note on the panel's postdrop membership
no longer calls it read-side only.

go test ./internal/notify passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 16:00:06 +03:00
mixandClaude Opus 5.5 50fd6b859f panel-notifications: the notifier, its mail and the server's own DKIM key
internal/notify applies the owner's rules. Each subject has a row of state,
so a thing is said once and a restart says nothing again:

- DNS of a sending domain: said after two hourly checks in a row found a
  record wrong, said to be over after two found it right. A lookup that did
  not answer moves nothing; a warning (no DMARC record) is not a subject.
- TLS certificate: at 14 days and at 3, per certificate.
- Queue: a message older than three days, once; once more when none is.
- Level-2 limit: from the rejected rows of the send log, every five minutes;
  at most once a day per domain and application.

Global users are told everything, domain users about their domains; one
message per recipient. With the instance switch off nothing is checked and no
state moves. A message that could not be handed to Postfix is tried again at
the next check.

Mail leaves by sendmail(1) as postmaster@<hostname>, addresses after "--",
no shell. For it to be believed the server's own name is signed like a
sending domain: domain.Service keeps a key for it and a line in OpenDKIM's
tables without a registry row, and non_smtpd_milters now names OpenDKIM. A
sending domain of the same name shares its key with the server.
dnscheck.ServerMail checks the name's DKIM and SPF records;
postfix.QueueMessages reads postqueue -j for arrival times.

The notifier is a sixth role of the panel process. Nothing in the panel
shows or switches it yet.

Checked on Windows: go vet clean; go test passes for notify, store, postfix,
dnscheck, cmd, and domain apart from TestWriteLoadPrivateKeyRoundtrip, which
fails there on file modes and always did. Not yet run in a container: whether
the unprivileged panel may submit is for the e2e step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 15:56:13 +03:00
mixandClaude Opus 5.5 06ad174fb9 panel-notifications: schema and store
Migration 0004_notifications.sql, DDL only: users.notify, a user's own
switch, on by default; notify_state, one row per subject the panel has
written about (dns:<domain>, cert, queue, limit:<domain>:<login>), so that
each thing is said once and a restart says nothing again.

Store: the instance switch as a settings key, on unless turned off;
SetUserNotify; NotifyRecipients - global users for everything, domain users
for the domains on their list or all of them, only with an e-mail and their
switch on, a shared mailbox once; the state's get, set, delete and list by
prefix; LimitRefusalsSince, the mail refused under a level-2 limit per domain
and application.

Nothing is sent yet. Checked on Windows: go vet clean, go test
./internal/store passes with the schema test at user_version 4. The page
guards fail from the design commit 87fa0a4 until the pages are built.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 15:48:56 +03:00
mixandClaude Opus 5.5 87fa0a4381 design: notifications on Account, Server > Settings and Server > Health
Accepted by the owner on 2026-10-12 from screenshots, after two changes of
his: DKIM for the server's own name as well as SPF ("we need the best
deliverability for these messages"), and the records themselves under
Settings with only their state on Health.

- account: a checkbox "Send me notifications" in the Profile box, under the
  e-mail it writes to. No box of its own, no choice per event.
- settings: box 03 Notifications with the instance switch; box 04
  Notification mail with the two DNS records of the server's own name, DKIM
  and SPF, drawn with the record component of a domain's DNS page. Sending
  rate limits move to the left column as box 02.
- health: card 06 Notification mail - the state of each record and a link to
  Settings - beside Configuration, now 07, each half the width.
- help: a Notifications topic (what is sent, when, to whom, what makes it
  arrive); Settings and Your account each gain a clause.

outlines.json rewritten with check.py --write: the structure of settings and
health changed. build.py: 35 pages; check.py: 32 screens, 52 sp- classes,
0 violations. The plan records both of his changes; step 1 of 10.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 15:45:06 +03:00
mixandClaude Opus 5.5 eff8010d39 panel-notifications: agreed, taken into work
The owner, 2026-10-12: "Делаем". Roadmap status agreed, 0/10; planned cut
2.3.0. The first step is the mockups, shown to him before any code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 15:27:20 +03:00
mixandClaude Opus 5.5 394eb67c4c panel-notifications: the design questions, closed with the owner
test / test (push) Canceled after 0s
Discussed on 2026-10-12, one question at a time; still a candidate.
plans/panel-notifications.md records the answers:

- Four events in the first version: a domain's DNS record stops matching,
  a level-2 limit refuses mail, the certificate is about to expire, a
  message is in the queue for more than three days (his number; the roadmap
  said one).
- From postmaster@<hostname>, not from a sending domain: with several the
  choice is arbitrary, and a certificate is no domain's subject. An SPF
  record for the hostname makes such mail pass; Server > Health will check it.
- One checkbox per user instead of a choice per event; one instance switch.
- Hourly checks; a DNS mismatch is sent after two checks in a row; "cleared"
  messages for DNS and the queue only; certificate at 14 and 3 days; a limit
  at most once a day.

roadmap.md follows, and no longer says the screens are drawn: the box was
taken out of the mockups on 2026-10-10.

Docs only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 15:05:33 +03:00
mixandClaude Opus 5.5 972a9f825c release: 2.2.0: a message's page keeps its delivery log
test / test (push) Canceled after 0s
delivery-log-storage (docs/plans/delivery-log-storage.md, 8/8), at the
owner's word (2026-10-12, "Да").

  - The log tailer stores what Postfix writes about queued outbound
    messages; a message's page reads it from the database, so the
    Delivery log box no longer empties when mail.log rotates. The lines
    live as long as the send-log row and go with it.
  - Mail that came in on port 25 is not stored.
  - Migration 0003 (send_log_lines), DDL only.

What an operator should know: the stored lines are 1-2 KB a message
and are in a full backup, which mail.log is not (security.md). A
message accepted before this version shows its lines only while they
are in the current file.

Cut: CHANGELOG [Unreleased] -> [2.2.0]; image pin 2.2.0 in compose,
README, guide; version table, roadmap, plan status. No tag: the owner
tags when he decides to (2026-10-11), so there is no image of this
version until then.

On the test server, on a990e7c: go vet ./... ok, go test ./... ok
(22 packages), make e2e ok.

Full detail: CHANGELOG.md (2.2.0).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 11:59:10 +03:00
mixandClaude Opus 5.5 a990e7cc23 delivery-log-storage: findings of the technical and security reviews
test / test (push) Canceled after 0s
Neither review (read-only agents, Opus and Fable, over ae06d8b..815236a) has
a Critical or High finding. What they found and what was done:

- A stranger on port 25 could grow the table: every queue-id line of inbound
  mail was stored and pruned an hour or more later, his own message-id among
  them. The tailer now leaves out the lines of a queue id that
  postfix/inbound/smtpd opened, until qmgr removes the message or another
  listener opens one under the id; 4096 such ids are remembered.
- A commit, so a disk sync, per stored line on the connection the journal
  milter shares, and a backlog written line by line. Lines are now written in
  one transaction per pass over the log, 500 at most, before the read offset
  is saved (InsertSendLogLines; follow() calls the keeper's flush).
- The prune asked the send log once per line and ran hourly. It asks once per
  queue id and runs at start and with the six-hourly sweep; with port-25 mail
  left out there is little without a row to wait for.
- The 4096-byte cut could split a character; it keeps characters whole.
- security.md did not say that a backup now carries these lines, and the plan
  said the time window excludes a reused queue id where it only limits the
  chance to the row's queue lifetime. Both are corrected; the collision is
  recorded as an accepted risk with its trigger.
- Tests that did not prove their comments: the prune is now tested with the
  sweep, a rejected row next to the prune, batching and the change from
  catching up to following, port-25 mail left out and its id reused.

Left as found and written down in the plan: lines the tailer never reads
(a file rotated away while the panel was down), and a message in the queue
across the upgrade, which shows only its later lines.

On the test server, for this tree: go vet clean, go test ./... 22 packages
ok, make e2e ok.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:41:04 +03:00
mixandClaude Opus 5.5 815236a607 delivery-log-storage: e2e holds the page to its lines after a rotation
New step delivery_log_survives_rotation: the page of the first delivered
message must list the line that opened its queue id and its delivery line,
then mail.log is rotated with logrotate -f, the current file is checked to
hold no delivery line, and the page must list both lines still. Before this
feature the page searched the current file and would show none.

The step runs after negative_level1_ratelimit: placed right after the first
delivery it rotated away the "connect from" line the level-2 step reads from
the current log, and that step failed.

On the test server, for this tree: go vet clean, go test ./... 22 packages
ok, make e2e ok (126 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:25:21 +03:00
mixandClaude Opus 5.5 a6845ae948 delivery-log-storage: architecture and guide
architecture.md § Log tailer gains "Stored lines": what is kept and why the
queue id is read from the head of a line, why the tailer does not check for a
row before writing, the one prune rule that is the table's retention, how a
row is given its lines when Postfix reuses queue ids, the write-once rule
while catching up, and the size. QueueLines is described as what it is now,
the fallback for rows without stored lines. guide.md: the message page keeps
its lines after rotation; retention also bounds the kept lines, which are in
a full backup.

Docs only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:15:47 +03:00
mixandClaude Opus 5.5 1e6836b3c1 delivery-log-storage: a message's page reads its stored lines
The Delivery log box on a message's page was filled by searching the current
mail.log on every view; that file is rotated daily, so the box was empty for
anything not sent today. deliveryLog now asks the store for the row's lines -
its queue id, dated within the row's queue lifetime plus a day
(RetryPolicy.LongestLifetime) - and shows the latest 200, oldest first.

When nothing is stored the page still searches the current file, which is
what shows a message accepted before 2.2.0. When that finds nothing either,
the note no longer blames rotation for the ordinary case: it says lines are
kept with the row since 2.2.0. The Outbound log help topic in help.go follows
the mockup changed in fc184ec, word for word.

Checked on Windows: go vet clean; go test ./internal/web/... ./internal/postfix
./internal/store pass, the guards among them. New test: lines shown from the
table when mail.log holds none of them, without another queue id's lines and
without those of an earlier message under the same id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:15:26 +03:00
mixandClaude Opus 5.5 fc184ecc01 design: the Outbound log help topic says the lines are kept
The topic said a message's lines are found in mail.log by queue id and that an
old message may show none, because that file is rotated. With
delivery-log-storage the lines are kept with the row: they stay on the page
for as long as the row does and go with it. A message accepted before 2.2.0
shows them only while they are in the current file.

Mockup source and its built page only; help.go follows with the page.
build.py: 35 pages; check.py: 32 screens, 52 sp- classes, 0 violations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:12:10 +03:00
mixandClaude Opus 5.5 81df131217 delivery-log-storage: the tailer keeps the lines of queued messages
Every mail.log line headed by a queue id is written to send_log_lines as the
tailer reads it, dated by the line's own stamp. The id is taken from the head
of the line only - the tail is the far end's text - and has to look like one:
"NOQUEUE:" and "warning:" stand in the same place.

The tailer does not check for a send-log row first: a message's first lines
come before the journal milter writes it. Lines that never get a row are
removed by the prune, which deletes lines older than an hour whose queue id
no row carries. It runs at start, after each send-log sweep and hourly.

While the tailer catches up at start a line is written only if it is not
there already, as the inbound journal does: its place is saved every few
seconds, so an unclean stop means reading some lines twice.

Checked on Windows: go vet clean; go test ./internal/logtail passes except
TestFollowTailsAndRotates, which fails there on renaming an open file and
always did. New tests: which line heads are queue ids, a line dated by its
stamp, no second copy after a rewound offset, the prune cadence and cutoff.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:11:28 +03:00
mixandClaude Opus 5.5 3d17206063 delivery-log-storage: the table for a message's log lines
Migration 0003_send_log_lines.sql, DDL only: queue id, the line's own time,
the line. No reference to send_log, because a message's first lines are
logged before the journal milter writes its row and one queue id is several
rows.

Store methods: InsertSendLogLine; InsertSendLogLineOnce for lines the tailer
reads twice after an unclean stop; SendLogLines, which gives a row the lines
of its queue id dated from an hour before the row to the queue lifetime after
it, so a later message under a reused id is not shown; PruneSendLogLines,
which deletes lines whose queue id has no send-log row - the whole of the
table's retention. A line is cut to 4096 bytes.

Nothing writes or reads the table yet. Checked on Windows: go vet and
go test ./internal/store pass, with the schema test pinning the new table,
its index and user_version 3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:08:56 +03:00
mixandClaude Opus 5.5 4468aa8dd7 delivery-log-storage: agreed, plan and checklist
The owner agreed the item on 2026-10-12 (option 1 of the roadmap entry: store
the lines). plans/delivery-log-storage.md records what that means in detail:
every mail.log line headed by a queue id goes into send_log_lines with the
line's own time; a line is a row's when the id matches and its time lies
within the row's queue lifetime, which is the bound against Postfix reusing
ids; a line whose id has no send-log row is deleted after an hour, which is
both the retention and the clean-up of inbound and bounce lines; the page
reads the table and falls back to the current file for rows older than it.
Eight checklist steps, cut 2.2.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:05:08 +03:00
mixandClaude Opus 5.5 977d63a2ea roadmap: inbound quarantine is an idea, not a candidate
The owner, 2026-10-12: take quarantine out of the main plan, it is an idea
to think about. Its index row and section leave roadmap.md; a new closing
section, "Ideas, not on the roadmap", keeps a paragraph and the link to the
notes in plans/inbound-quarantine.md. panel-notifications no longer lists
"mail is held in quarantine" among its events. The plans that called it a
candidate say what it is now. The drawn screens stay unbuilt, as they were.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:04:44 +03:00
mixandClaude Opus 5.5 ae06d8b47e release: 2.1.0: Inbound log and filter lists; a milter security fix
test / test (push) Canceled after 0s
inbound-antispam-panel (docs/plans/inbound-antispam-panel.md, 12/12),
at the owner's word (2026-10-12, "Да").

  - Inbound mail is recorded: accepted mail by a second, read-only
    milter on port 25 with the spam filter's verdict, refused mail from
    the mail log. The filter signs what it judged, and a verdict is
    shown only next to its signature.
  - Inbound > Log, with a sheet over the page that says who decided
    and why; Inbound > Filter lists, allow and deny entries written as
    rspamd maps, each with how often it decided a message.
  - Migration 0002 - the first after the 2.0 baseline.

Security, outside the feature and in every release since 1.9.0: the
level-2 rate limit and an application's client IP list held only for
the first message of a connection. Fixed; e2e sends two MAIL FROM on
one connection.

Upgrading an installation with the antispam sidecar: copy
deploy/antispam/ again (the fragment mounts two new things) and
restart the sidecar. Port 25 now logs as postfix/inbound.

Cut: CHANGELOG [Unreleased] -> [2.1.0]; image pin 2.1.0 in compose,
README, guide; version table, schema-migrations, roadmap, plan status.
No tag: the owner tags when he decides to (2026-10-11), so there is no
image of this version until then.

On the test server, on 1f44e40: go vet ./... ok, go test ./... ok
(22 packages), make e2e ok.

Full detail: CHANGELOG.md (2.1.0).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 09:12:16 +03:00
mixandClaude Opus 5.5 1f44e40382 inbound-antispam-panel: the filter reads its signing key for every message
test / test (push) Canceled after 0s
The routine kept the key after its first read. SelfPost makes a new key
when the file is lost or damaged, or a backup brings back another, and
rspamd would then have gone on signing with the old one - every verdict
unrecorded - until someone restarted the sidecar. The file is one short
line; reading it per message costs nothing worth keeping a copy for.

The guide says what never needs doing: nothing of rspamd's is cleared
after a change of the lists or the key; a restart is for a change of
its own configuration only.

Prompted by the owner's question whether rspamd's cache needs clearing
(2026-10-12).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 08:42:16 +03:00
mixandClaude Opus 5.5 d398f488e2 inbound-antispam-panel: the filter signs what it judged
test / test (push) Canceled after 0s
Closes what the security review left as an accepted risk: with the
spam filter down - it is attached fail-open - nothing removes verdict
headers a sender wrote himself, and the inbound journal recorded his
verdict as the filter's. The owner asked whether rspamd could add a
header that proves the others are ours (2026-10-12) and agreed to a
signature: a fixed mark could be written by a sender as easily as the
verdict.

  - inbound.FilterKey: a key of 32 random bytes, 64 hex characters, in
    /data/antispam/filter.key, made on the first start with the relay
    on. 0644 like the list files beside it: the filter is another user
    in another container.
  - deploy/antispam/rspamd/milter_headers.conf: a custom routine reads
    the key through the directory the sidecar already mounts and adds
    X-SelfPost-Filter, the HMAC-SHA256 of the message's queue id, hex;
    a header of that name from the sender is removed.
  - internal/milter/inbound.go: the verdict headers are kept as they
    are read, and at end-of-message - when the queue id is known - they
    count only if the signature fits (FilterSignature, hmac.Equal). No
    key, no queue id, no signature or a wrong one: the row is that of
    mail nobody judged (accept, selfpost, no detail).

The signature goes on to the upstream with the mail and is of no use
there: it fits one queue id.

Tried in a throwaway rspamd 3.11 container on the test server before
the Go side was written: the routine loads, signs
(2aa1ca6d... for key 0123...cdef and queue id 4A1B2C3D5E, the value
python's hmac gives and TestFilterSignatureIsWhatTheFilterWrites
pins), removes a sender's header of the same name, and adds nothing
without a queue id.

Tests: signed, unsigned, made-up, another message's, another key's,
the sender's followed by the real one, the next message on the
connection, no queue id; the key made once, read after, replaced when
damaged, left alone by the lists. e2e: filter.key is there, 644, 65
bytes.

docs/security.md: the accepted risk is replaced by a narrower one (the
key is readable on the host); Mail path describes the signature.

Checked on the test server, uncommitted tree: go vet ./... ok,
go test ./... ok (22 packages), make e2e ok.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 08:34:29 +03:00
mixandClaude Opus 5.5 89774ab847 inbound-antispam-panel: three things the reviews left as found
test / test (push) Canceled after 0s
The owner asked for them to be put right (2026-10-12).

  - A refusal is not logged twice after an unclean stop. The tailer
    saves its place every five seconds, so on start it reads some
    lines again; for the lines it reads on start a row is written only
    if the same row is not there already (store.InsertInboundSpamOnce:
    same moment, client, sender, recipient, decision, engine, detail).
    Lines read as they are written are always new, so two identical
    refusals in one second stay two rows.
  - The Inbound log offers Deny / Allow sender only for a sender the
    lists would take (inbound.Listable): an envelope sender may be
    spelt in ways the form refuses, and the link led to a refusal.
  - A network of one address is stored as the address: 192.0.2.7/32
    and 192.0.2.7 were two entries of one list.

Checked on the test server, uncommitted tree: go vet ./... ok,
go test ./... ok (22 packages), make e2e ok.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 08:17:23 +03:00
mixandClaude Opus 5.5 c731a98a0f inbound-antispam-panel: findings of the technical and security reviews
test / test (push) Canceled after 0s
Steps 11 and 12 of docs/plans/inbound-antispam-panel.md: 12/12. Two
read-only reviews of d65d632..HEAD by different models (Opus,
technical; Fable, security), each given the code and questions, not
conclusions. Nothing Critical or High in the feature; the High the
technical review found outside it is the commit before this one.

Log tailer (internal/logtail/inbound.go):
  - the reply of a refusal ends at the first "; from=<", Postfix's own,
    and the HELO name after it is whatever is left. Taking the longest
    reply let a HELO name that spelt "; from=<..> to=<..> proto=.."
    supply the row's sender, recipient and reply - and so make a
    refusal read as the deny list's;
  - an address with a quote, a bracket or a space is not kept: a quoted
    sender could otherwise supply the recipient;
  - the journal is cut to its row cap every 500 rows written, not only
    every six hours;
  - a queue id is forgotten when the message is queued or the id is
    opened by another listener, and a stale place in the queue of
    arrivals no longer drops a live id;
  - milter-hold lines are not read: the held message reaches the
    journal milter, which writes its row - it was written twice.
Lists (internal/inbound/lists.go): a change of the table and the
rewrite of the files are one step under a mutex; a failed rewrite puts
a removed entry back and rewrites after taking a new one out.
Store: the row cap removes the oldest by time, as the page orders them.
Page: on a sheet a refusal's text is a Reply whatever it starts with.

Help and the guide say that an allow entry for a sender address trusts
an address anyone can claim (design commit before this one).

docs/security.md: the two reviews; the roles table gains Inbound > Log
and Filter lists; Mail path describes the milters, the tailer and the
lists; four accepted risks - the filter's verdict while it is down, an
allow entry by sender address, the list files' mode, log rows held by
domain name. The plan's stale lines (size and rate refusals, who reads
the journal) are corrected.

Left as found, told to the owner: the journal's rows after an unclean
stop may be written twice (the tailer saves its place every five
seconds); a "Deny sender" link is offered for an address the form then
refuses; 1.2.3.4 and 1.2.3.4/32 are two entries.

Checked on the test server, uncommitted tree: go vet ./... ok,
go test ./... ok (22 packages), make e2e ok.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 07:51:50 +03:00
mixandClaude Opus 5.5 5f86457848 milter: the level-2 limit and the client IP list hold for every message
Security fix, outside the inbound antispam feature. Found by its
technical review (Opus, 2026-10-12), reproduced on the test stand
before the fix: over an application's level-2 limit, two MAIL FROM on
one connection were answered 451 and then accepted.

Cause. Postfix introduces a client to a milter once per connection.
The milter library (go-milter 0.4.1, session.go) replaces the milter's
session after any reply that is not "continue" - a refusal, or the
accept that used to end a message. The new session never saw Connect,
so it had no client address, and both checks passed a message without
one: overLimit returned early ("no client IP to key on") and
authIPAllowed did the same. So every message after the first on a
connection, and every MAIL FROM after a refused one, skipped the
level-2 limit and an application's client IP list. In the code since
1.9.0 for the list and longer for the limit.

Fix.
  - build/postfix-config.sh: milter_mail_macros carries {client_addr},
    so the address comes with every MAIL FROM; MailFrom reads it (both
    spellings of IPv6) in the journal milter and the inbound journal
    milter.
  - overLimit no longer asks for a client address: a ceiling is on a
    domain or an application.
  - authIPAllowed refuses an application that has a client IP list when
    no address is known.
  - Body replies "continue" in both milters, which lets the message
    through as "accept" did and keeps the session.

The inbound journal milter had the same cause with a smaller effect:
rows of a connection's later messages had no client address.

In the same file, the checks of INBOUND_ANTISPAM_MILTER and
INBOUND_JOURNAL_MILTER_SOCKET refuse any character outside their set
first: a glob's "*" matched anything after the first characters.

Tests: a session that never saw Connect is judged by the macro, a
listed and an unlisted address, no address, not an address; the limit
with no client address; the inbound milter answers only "continue".
test/e2e sends two MAIL FROM on one connection over the limit and
wants both refused.

Checked on the test server: e2e red before the fix on exactly that
check, green after; go vet ./... ok, go test ./... ok (22 packages).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 07:51:50 +03:00
mixandClaude Opus 5.5 00feb7c1c7 design: help, an allow entry by sender address trusts a claimed address
From the security review of 2026-10-12: the allow list matches the
envelope sender, which nobody authenticates. Help's Filter lists topic
says so and points to client addresses and networks.

check.py: 32 screens, 0 violations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 07:51:49 +03:00
mixandClaude Opus 5.5 d2a778edc0 inbound-antispam-panel: Filter lists accepted; steps 8-10 closed
test / test (push) Canceled after 0s
The owner accepted Inbound > Filter lists on 2026-10-12 ("Да"), after
a deny entry he added refused my test message on the live test server
with the list's own reply (554 5.7.1 Refused by this server's deny
list), mail.log carried the milter-reject line the count reads and the
journal its row.

Steps 8-10 of the plan were done with each earlier step and are ticked
with where. 10/12: the technical and the security review remain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 07:22:09 +03:00
mixandClaude Opus 5.5 7192c3861e inbound-antispam-panel: Filter lists, kind column, matches, note on a sheet
Follows the design commit before it (owner's words quoted there).

The lists' tables are Value, Kind, Matched and Remove. The kind tag
opens the entry's sheet: note, day added, how often it decided a
message in the days the log keeps, and the last match.

Matches are read off the journal on each view, nothing is stored
(inbound.Lists.Usage, store.ListSpamListHits). A row counts for an
entry when the list decided the message - a rejected row whose reply is
the deny rule's message, an accepted row whose verdict carries the
allow rule's symbol for that kind - and the entry fits the row's sender
or client address the way rspamd matches it: an address whole, a domain
as the sender's domain, a network containing the client (an
IPv4-mapped address as IPv4). A failure to count shows the lists with
"never" and logs why.

For that the journal milter keeps the lists' own symbols ahead of its
six-symbol cap: rspamd writes symbols heaviest first, so an allow
entry's -100 came last and was cut behind any six others.

The plan gains "Filter list usage": what counting from the journal
implies, and - the owner asked for it in writing - the conditions under
which a counter kept with each entry becomes worth a migration (the
journal cut by its row cap and not by age; a short retention; counts
that must outlive the journal; a slow page). multimap.conf says which
of its texts SelfPost reads back.

Tests: Usage over every kind on both lists and the rows that are
nobody's; the milter's symbol order; the store query; the page and its
sheets; a handler test from journal rows to the numbers on the page.

Seen: the fixture page with a sheet open, headless Edge at 1360 px -
the same as the accepted drawing.

Checked on the test server, uncommitted tree: go vet ./... ok,
go test ./... ok (22 packages), make e2e ok.

Not seen in a real run: a count above zero. It needs an entry added in
the panel, which I cannot sign in to, and then mail that the list
decides; in particular the deny rule's message arriving in mail.log as
the reply of a milter-reject line is what the trial with rspamc showed,
not yet what Postfix logged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 01:08:26 +03:00
mixandClaude Opus 5.5 e0ccff257d design: in-filter-lists, kind column, matches, the note on a sheet
Owner, 2026-10-12, on Inbound > Filter lists as built: "Note хочется
вынести в лист, подсказку, раскрывающийся блок. Тип в отдельную колонку.
Ещё полезно будет дать статистику использования правила, чтобы удалять
устаревшие." Drawn, then accepted ("Да") together with counting the
matches from the Inbound log.

  - Kind is a column of its own; the tag no longer wraps under the
    value.
  - Note and Added leave the table for the entry's sheet, opened by the
    kind tag (of his three - sheet, tooltip, disclosure - the sheet: it
    exists and works under a finger).
  - Matched: how often the entry decided a message, or "never". The
    sheet adds the window ("in the last 90 days") and the last match.
    A fifth column for the last match did not fit half the page width.

Help, Filter lists: what Matched counts and what "never" suggests.

check.py: 32 screens, 0 violations; outlines.json unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 00:54:04 +03:00
mixandClaude Opus 5.5 c1da784193 inbound-antispam-panel: the backup holds the lists; Inbound log accepted
Step 7/12 of docs/plans/inbound-antispam-panel.md. A full backup walks
all of /data, so data/antispam/ was in it from the day the directory
existed; TestCreateIncludesStateExcludesTransient now holds that. On
the first start after a restore the maps are written again from the
database (mailStack.Resync, already there). The guide says what is not
in a backup: the antispam/ directory with the compose fragment and the
rspamd configuration, which is installation and not state.

The plan records the owner's acceptance of Inbound > Log (2026-10-12,
"Этот экран утверждаем") and what he changed on the way to it.

go test ./internal/backup/ ok.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 00:36:59 +03:00
mixandClaude Opus 5.5 9d330bed77 inbound-antispam-panel: Inbound log, Why as a sheet; no Deny on a Postfix reject
Follows the two design commits before it (owner's decisions quoted
there).

Kit: a new component, Sheet / SheetRow (components.go), partials
sheet_tag and sheet (components.html), its rules in panel.css as in
theme.css, and its sample on the kit page next to status_tag. It is
the browser's popover: no script.

Inbound log: the Why column is gone; the decision tag opens a sheet
with who decided, then either the filter's verdict and its symbols one
a line (a detail the journal milter wrote) or the reply (a detail the
tailer read from mail.log - it starts with its code), and the subject
whole. The sheet's id is the journal row's. A row Postfix itself
refused no longer links to the filter lists.

Tests: the page fixture and view tests for the sheet's markup and
content (TestWhyRows), the link rule, three Deny and two Allow on the
mockup's six rows; e2e reads "Decided by" and "Reply" from the sheet
and wants no sender link on the refused row.

Seen: the fixture page with a sheet open, headless Edge at 1360 px -
the same as the accepted drawing.

Checked on the test server, uncommitted tree: go vet ./... ok,
go test ./... ok (22 packages), make e2e ok.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 00:23:51 +03:00
mixandClaude Opus 5.5 970c8ca8da design: the sheet on the components page is a table; no empty Subject
The sample sheet on the components page still held facts; it shows the
table of name and value the accepted sheet is. In in-log a refused
message's sheet has no Subject row - there is no subject to show whole.

check.py: 32 screens, 0 violations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 00:13:14 +03:00
mixandClaude Opus 5.5 19156814b8 design: in-log, Why is a sheet over the page; no Deny on a Postfix reject
Owner, 2026-10-11/12, on the Inbound log as built:
  - "Why сделать не столбцом, а листом поверх." Opened by the decision
    tag ("Тег."), drawn first, then: its content as a table ("В лист
    может быть таблицей?" - "Да. Но хорошо бы ещё её отцентрировать."),
    with a narrow column of names and only what the row does not show.
  - "Почему Deny у того, кто rejected?" - "Убирай": a message Postfix
    itself refused offers no Deny sender. Accepted mail offers Deny,
    mail the filter refused offers Allow.

A new component, sheet (theme.css, shown on the components page next to
status_tag): the browser's own popover, so no script - the tag is a
button with popovertarget; Esc, a click outside and the cross close it.
It rises from the bottom edge across the window; its content is a table
of name and value in a 44rem column in the middle. A tag that opens a
sheet carries a dotted underline to tell it from a plain tag.

in-log: the Why column is gone and rows are one height. A sheet holds
who decided, the verdict and the symbols (or the reply), and the
subject whole.

Guard: check.py WIDTH_OK gains ".sp-sheet" - the one component laid out
by the window and not by the page. The owner accepted the drawing with
that said.

check.py: 32 screens, 52 sp- classes, 0 violations; outlines.json
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 00:12:04 +03:00
mixandClaude Opus 5.5 d2b97b3963 inbound-antispam-panel: Inbound log, the sender column is headed From
Follows the design commit before it.

go test ./internal/web/view/ ok.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:59:47 +03:00
mixandClaude Opus 5.5 d4efdb483d design: in-log, the sender column is headed From
Owner, 2026-10-11, of "From · client": "Client в названии столбца
зачем? Мне кажется, оно лишним." The client address stays under the
sender, in mono and muted; an address of that shape needs no label.

check.py: 32 screens, 0 violations; outlines.json unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:59:18 +03:00
mixandClaude Opus 5.5 56c5dfee40 inbound-antispam-panel: Inbound > Log and Inbound > Filter lists
Step 6/12 of docs/plans/inbound-antispam-panel.md. Written by a Sonnet
subagent from the mockups, reviewed and corrected by me.

GET /inbound/log - the journal a page at a time (50), filtered by
domain, decision and day; QueryInboundSpam / CountInboundSpam with the
viewer's inbound domains as the scope. The global role reads all of it
and gets "Deny sender" / "Allow sender" links into the lists; a user
with the domain role reads the rows of the inbound domains assigned to
them, without the links. The plan said global-only for the journal;
the accepted mockup and 2.0's delegation of Inbound say otherwise -
recorded in the plan as built.

GET, POST /inbound/filter-lists and POST .../{id}/delete - global role,
404 otherwise. Add form (prefilled from the query), the two lists with
Remove as the kit's row action, and "Where the lists go" with the
filter's state and when the maps were last written
(Lists.LastWritten). A refused or duplicate entry is 422 with the
message and what was typed.

Server > Settings: "Keep inbound log rows, days" with the relay on.
Help: Inbound log, Filter lists. Navigation: Log for everyone who sees
Inbound, Filter lists for the global role (Shell.ShowFilterLists).
Status tags accept / reject / tempfail join the kit's mapping.

Tests: page fixtures for both screens, held to outlines.json; view and
handler tests (both roles, filters, paging, relay off, add / duplicate
/ invalid / delete, map file content, settings field, help). e2e reads
the journal from the page instead of the database and adds and removes
a deny entry through the panel, reading deny_domain.map after each.

Seen: both pages and Settings rendered from their fixtures and compared
with the mockups side by side (headless Edge, 1360 px) - same layout
and wording.

Checked on the test server, uncommitted tree: go vet ./... ok,
go test ./... ok (22 packages), make e2e ok.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:50:34 +03:00
mixandClaude Opus 5.5 82d4a209be design: in-log and in-filter-lists leave the deferred screens
Guards and mockups only, ahead of the pages (inbound-antispam-panel
6/12). The owner agreed the feature ("Следующая по дорожной карте -
inbound-antispam-panel" / "Да", 2026-10-11) and started this step
("Начинай"), so the two screens are no longer "drawn but not built":

  - guard_panel_test.go: in-log and in-filter-lists removed from
    deferredScreens. From here TestLegacyRatchetMatchesTheEngine wants
    the engine to have both pages - red until the next commit.
  - guard_routes_test.go: the four routes join routes.inbound, so they
    must exist with the relay on and answer 404 with it off.

Two mockup details brought to what the kit can draw, found when the
pages were built from it:
  - help: the two new topics are one paragraph with a break in it, as
    every topic is (panel.css has no gap between paragraphs of a topic);
  - in-filter-lists: the directory in "Files" is mono at the text size -
    the kit's inline code has no small variant.

check.py: 32 screens, 0 violations; outlines.json unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:50:33 +03:00
mixandClaude Opus 5.5 737611a07e design: Inbound log and Filter lists as built; inbound retention; Help
Mockup-first step of inbound-antispam-panel 6/12. The two screens were
drawn for the redesign before the feature existed; this brings them to
what steps 3-5 built.

in-log: the lead says where rows come from and how long they are kept;
a help link in the box head; Why is the engine and the detail as the
journal holds them (rspamd: action, score / threshold, weighted
symbols; postfix and a refusing filter: the reply text). No row
invents what the journal cannot hold: the "selfpost - upstream timed
out" tempfail and the Review link to a quarantine page that does not
exist are gone, a refused message has no subject, an allow-listed one
shows SELFPOST_ALLOW_FROM(-100.00).

in-filter-lists: Remove is a POST - the kit's row action, not a link;
a domain entry is shown as stored, without "@"; "Where the lists go"
names the directory and its six files instead of allow.map / deny.map,
says rspamd rereads them by itself and that Deny wins.

settings: "Keep inbound log rows, days" in the Log retention box.

help: two topics under Inbound, Inbound log and Filter lists, and a
sentence in Settings.

outlines.json rewritten: Help's Inbound box has six topics, was four.
check.py: 32 screens, 0 violations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:18:57 +03:00
mixandClaude Opus 5.5 eea40bcd49 inbound-antispam-panel: filter lists, written as rspamd maps
Step 5/12 of docs/plans/inbound-antispam-panel.md.

Store: AddSpamListEntry, ListSpamListEntries, DeleteSpamListEntry.
internal/inbound/lists.go validates and normalises an entry - sender
address, sender domain, client address or network (canonical text, an
IPv4-mapped address as IPv4, /0 refused) - and writes the lists as six
map files under /data/antispam (ANTISPAM_DIR), each atomically, 0644 in
a 0755 directory: the filter is another user in another container. All
six are written on every change and on every start with the relay on,
empty ones included. An entry whose maps could not be written is taken
back out of the table. The writer refuses a stored value with
whitespace, a newline or # in it.

The plan's allow.map / deny.map became one file per list and entry
type: to rspamd an IP map and a sender map are different map types.

deploy/antispam/rspamd/multimap.conf holds the rules; the fragment
mounts it and ./data/antispam into the sidecar. Deny is a prefilter
reject, allow a symbol weighing -100. Tried in a throwaway rspamd 3.11
container on the test server:
  - with allow as a prefilter accept, an address on both lists was
    accepted and an IP on both was refused - the rule that ran first
    decided; with the weight, deny won in all five mixed cases;
  - IPv4, IPv6, address (any case), domain (not its subdomains) match;
  - a rewritten file is reread in about ten seconds, no restart;
  - files that appear after rspamd started are loaded the same way.

No page edits the lists yet (step 6).

Checked on the test server, uncommitted tree: go vet ./... ok,
go test ./... ok (22 packages), make e2e ok - with the relay on
/data/antispam is 755 and holds the six maps, 644, empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:07:10 +03:00
mixandClaude Opus 5.5 27fd355e53 inbound-antispam-panel: refused inbound mail from mail.log; retention
Step 4/12 of docs/plans/inbound-antispam-panel.md.

The log tailer writes an inbound_spam_log row for mail port 25 refused
(internal/logtail/inbound.go): a "reject:" line of the port-25 smtpd is
Postfix's own policy; a milter-reject / milter-hold / milter-discard
line of cleanup is the filter's verdict. Only a recipient at an inbound
domain is a row, so "Relay access denied" for a stranger's domain is
not. The row is dated by its log line; the reply text is cut to 255
bytes of printable ASCII; a reply starting with 4 is a tempfail.

To tell port 25 from 465 / 587 the listener now logs as
postfix/inbound (syslog_name). cleanup logs under one name for every
listener, so its verdict counts only for a queue id the port-25 smtpd
was seen to open (4096 remembered). Lines are matched from the head,
after timestamp and host: their tail is the sender's own text.

Retention: PruneInboundSpam every six hours - older than the
inbound_spam_log_retention_days setting (seeded from
INBOUND_SPAM_LOG_RETENTION_DAYS, 90, range 7-365), then down to 10 000
rows. The setting has no field in the panel yet.

Known limits, in architecture.md: a log line names one recipient, so a
message refused at end-of-message is one row; a line re-read after an
unclean stop would be a second row.

Checked on the test server, uncommitted tree: go vet ./... ok,
go test ./... ok (22 packages), make e2e ok - the unknown recipient at
the inbound domain refused on port 25 has its row (reject by postfix),
the stranger's domain has none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 22:42:25 +03:00
mixandClaude Opus 5.5 94e94490af roadmap: phone in "desktop site" mode gets the phone navigation
A candidate, recorded at the owner's word (2026-10-11, screenshot of
Server > Health on selfpost.mixfed.ru): the body is laid out in columns
while the navigation is the phone stack with every section open. The
item states what was seen and a first guess at the cause (980 px falls
between Bulma's 769 and 1024 px breakpoints); nothing is investigated
yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 22:42:08 +03:00
mixandClaude Opus 5.5 28ec784ae2 inbound-antispam-panel: inbound journal milter on port 25
Step 3/12 of docs/plans/inbound-antispam-panel.md.

The panel serves a second milter while the inbound relay is on
(internal/milter/inbound.go, socket INBOUND_JOURNAL_MILTER_SOCKET,
default /run/selfpost/inbound-journal.sock). build/postfix-config.sh
puts it last on port 25, after the optional antispam milter, with
default_action=accept; postfix-wrapper.sh waits for its socket. At
end-of-message it writes one inbound_spam_log row per recipient at an
inbound domain - other port-25 recipients (DMARC report addresses) get
none - and always accepts. With a filter attached it reads the verdict
from X-Rspamd-Action and X-Spamd-Result (action, score, up to six
weighted symbols, only what the patterns recognise, 255 bytes); without
one those headers are the sender's and are not read. The plan named
X-Rspamd-Score, which rspamd does not have.

The antispam compose fragment mounts rspamd/milter_headers.conf into
the sidecar so that it adds the two headers (checked in rspamd 3.11:
both routines exist and remove a sender's own headers of those names).

Store: InsertInboundSpam, IsInboundDomain.

Checked on the test server, uncommitted tree: go vet ./... ok,
go test ./... ok (22 packages), make e2e ok - the relayed message of
inbound_relay_delivers_and_refuses has its journal row (accept by
selfpost) and the DMARC report sent through the same listener has none.

Not yet seen in a real run: a verdict written by rspamd reaching this
milter. The e2e stand has no filter, and the test server's only inbound
domain has no upstream, so it accepts no mail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 22:18:14 +03:00
mixandClaude Opus 5.5 a3cc52d6d4 inbound-antispam-panel: migration 0002, journal and filter-list tables
Step 2/12 of docs/plans/inbound-antispam-panel.md, the first migration
after the 2.0 baseline (2.0.1 was cut before it, as agreed).

0002_inbound_spam_log.sql adds inbound_spam_log (one row per recipient:
domain, client IP, envelope, subject, decision, engine, detail) with
indexes on created_at and (inbound_domain, created_at), and
inbound_spam_list (allow / deny entries, unique per list). DDL only;
decision, engine, list_type and entry_type carry CHECKs, as the
baseline does for its own enumerations. Nothing reads or writes the
tables yet.

Tests: TestBaselineSchema pins both tables and user_version 2;
TestOpenMigratesA20Database opens a file holding the baseline alone and
finds its user kept; TestInboundSpamConstraints.

go test ./internal/store/ ok; go vet ./... ok.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 22:05:20 +03:00
mixandClaude Opus 5.5 d65d632505 release: 2.0.1: DMARC ingest drops and bounds; Full backup from the panel
test / test (push) Canceled after 0s
A patch on 2.0.0, all of it found on the test server after the cut:
  - a message that is not a report this relay keeps is dropped at once
    instead of being retried for days;
  - a report is bounded before it is stored, and reports are kept per
    domain (security review of the ingest path, nothing Critical or High);
  - Full backup from the panel works (it had answered 200 with ten bytes
    since 1.3.0);
  - no backup holds the Postfix queue, and the panel says why.

Cut: CHANGELOG [Unreleased] -> [2.0.1]; image pin 2.0.1 in compose,
README, guide; version table.

On the test server, on dd0fefd: go vet ./... ok, go test ./... ok
(22 packages), make e2e ok.

Full detail: CHANGELOG.md (2.0.1).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 21:14:00 +03:00
mixandClaude Opus 5.5 dd0fefdcb7 backup: the panel's archive is the command's, private key included
Owner, 2026-10-11: the private key is needed, and a backup from the
panel and one from the console have to be the same.

The commit that made the panel's backup work skipped a file under
certs/ the panel could not open - on the test server the private key,
which the certificate hook leaves to root and Postfix's group. That gave
an archive with a certificate and no key, and two kinds of archive.

- Params.Unprivileged is gone. Nothing is left out on account of who
  makes the archive; the queue is out for everyone (previous commit).
- When the panel cannot read a file under certs/, backup.Check says so
  before the download and the panel refuses with what to do on the host:
  chgrp 998 certs/privkey.pem && chmod 640. backup.UnreadableCert tells
  that case from any other failure.
- 998 is the container's selfpost group, which holds the panel and
  Postfix. Its id was whatever groupadd gave; the Dockerfile now fixes
  it at the value it has always had, because an operator addresses it
  by number from the host.
- Guide: method 1 is the same archive as method 2, with the permission
  the key needs and a note for whatever renews the certificate. Help and
  the box follow the design commit before this one.
- e2e: the restore scenario first makes the key unreadable and expects
  the refusal naming the file and the way out, then takes the backup and
  expects the key in it; nothing is put back by hand before the restart.

  go vet ./internal/backup/ ./internal/web/handlers/: ok
  go test ./internal/backup/ ./cmd/... ./internal/web/...: ok
  make e2e on the test server: ok

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:43:10 +03:00
mixandClaude Opus 5.5 75ebbf6a57 design: the panel's backup needs the private key readable, and says so
Owner, 2026-10-11: the key is needed and the panel's archive must be the console's. The box no longer says a certificate file may be left out; Help says the key has to be readable by the panel (group 998 on the host) and that the panel refuses, naming the file, until it is.

check.py: 0 violations; outlines.json unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:43:10 +03:00
mixandClaude Opus 5.5 ede3783734 panel-redesign: Full backup and Help say why the queue is left out
Follows the design commit before it: the paragraph of the Full backup box and the Backup topic of Help.

  go test -count=1 ./internal/web/...: ok
  Backup page at 1280: no overflow, the box holds the longer paragraph.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:16:30 +03:00
mixandClaude Opus 5.5 d70371a50c design: Full backup and Help say why the queue is left out
Owner, 2026-10-11: the user has to be told why. The box gives the reason in a sentence - the queue cannot be copied reliably while the server is running; the Help topic explains it (Postfix keeps moving messages, a copy could hold one twice or miss it), what follows (waiting mail is not carried over, a restored server starts with an empty queue) and what to do to keep the queue (stop the container, archive the project directory).

check.py: 0 violations; outlines.json unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:16:30 +03:00
mixandClaude Opus 5.5 21e961cb29 backup: no full backup holds the Postfix queue
Owner, 2026-10-11: the queue goes out of the command's archive too.

The commit that made the panel's backup work left the queue in the
archive of selfpost-backup, which runs as root and can read it. But that
copy was never the queue: the archive is made while the instance runs,
Postfix moves messages between the queue's directories all the time, and
a message on its way could be in the archive twice or not at all. One
rule now - QueueDir is left out of every archive - and
Params.Unprivileged is only about a certs/ file the panel may not open.

What is not carried over is mail still waiting for delivery; a restored
instance builds an empty queue (the e2e restore scenario starts one).
To keep the queue: stop the instance and tar the project directory, as
the guide's methods 3 and 4 already describe.

Guide: the description of a full backup, methods 1 and 2 and the
comparison table. Help follows the design commit before this one.

  go vet ./internal/backup/: ok
  go test ./internal/backup/ ./cmd/... ./internal/web/...: ok

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:06:41 +03:00
mixandClaude Opus 5.5 82f212ab39 design: Help says no backup holds the Postfix queue
Owner, 2026-10-11: the queue is left out of the command's archive as well as the panel's. The Help topic no longer says selfpost-backup takes it; the command still takes the private key the panel cannot read.

check.py: 0 violations; outlines.json unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:06:41 +03:00
mixandClaude Opus 5.5 b55b33ae92 e2e: full backup from the panel, restore into an empty directory
After the rate-limit checks the run takes a full backup through the
panel, reads the archive through while the container can still say what
went wrong, then removes data/, certs/, docker-compose.yml and .env,
unpacks the archive into the project directory, puts the private key
back (the panel's archive comes without it and without the queue, both
asserted), and starts the stand again.

What has to hold: manifest.json is consumed by the first start, the
administrator signs in, the domain shows the same DKIM value, and a
message sent with the application's old password reaches the sink and
verifies against the record published before the backup.

The port-25 scenarios then run on the restored installation.

make e2e on the test server: ok.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 19:55:19 +03:00
mixandClaude Opus 5.5 8c9a71e6a0 backup: the panel's full backup works, without the queue
Found by the e2e restore scenario of the next commit: the Full backup
button answered 200 with ten bytes. The archive includes the Postfix
queue, Postfix keeps every queue file and directory readable by its own
user alone (not configurable), and the panel runs as its own user - the
walk stopped at /data/postfix/queue/active with "permission denied"
after the headers were out. So since the queue moved under /data in
1.3.0. Confirmed on the test server: selfpost-backup as root makes the
whole archive, the same command as user panel fails the same way.

Owner, 2026-10-11: the panel's archive comes without the queue.

- backup.Params.Unprivileged: the queue is left out, and so is a file
  under certs/ that cannot be opened (the private key on the test
  server is root and Postfix's group only), named through OnWarn. The
  selfpost-backup command leaves it unset and takes both, as before.
- backup.Check reads through everything Create would archive, writing
  nothing. The panel runs it before the first byte: a failure is a
  message on the page, not a download cut short.
- The box, Help (design commit before this one) and the guide say what
  the panel's archive leaves out; the guide's comparison table has rows
  for the queue and the private key, and tells to put the key back
  before starting a restored instance.

  go vet ./internal/backup/ ./internal/web/handlers/: ok
  go test ./internal/backup/ ./cmd/... ./internal/web/...: ok
  make e2e on the test server, with the restore scenario: ok

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 19:55:18 +03:00
mixandClaude Opus 5.5 646ec1e149 design: Full backup says what the panel's archive leaves out
Owner, 2026-10-11: the archive made from the panel comes without the Postfix queue (variant 1). The Full backup box and the Help topic say so, and name the other thing the panel may not read - a certificate file, usually the private key - and the command that takes both.

check.py: 0 violations; outlines.json unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 19:55:18 +03:00
mixandClaude Opus 5.5 8496b289ec e2e: the run ends on port 25, with a report, junk and inbound mail
Until now the suite only read back the configuration generated for the
two optional port-25 features. DMARC ingest shipped in 1.7.0 with a
delivery service Postfix could not start, and nothing sent a report
through it until a real one arrived on the test server on 2026-10-11.

The last steps recreate the container with INBOUND_RELAY_ENABLE and
DMARC_REPORTS_ENABLE on, over the same /data, and talk to port 25 as a
stranger's mail server:

- the container log has no postconf warning, the session survives;
- a domain takes the SelfPost-hosted report address; a gzip report
  mailed to it appears under Outbound > DMARC reports; an address nobody
  chose is refused at RCPT;
- a report on another domain and a message with no report are accepted,
  dropped with "dropped: ..." on their delivery lines, counted as two
  parse failures, absent from the queue and from the reports;
- an inbound domain with the sink as upstream and one listed recipient:
  mail for it reaches the sink, another recipient and another domain are
  refused at RCPT.

The stage directory now holds docker-compose.yml and .env, as an
operator's project directory does (a full backup needs them; the
restore scenario that uses it is not in this commit - the panel's full
backup turned out not to work, see the next one).

make e2e on the test server: ok.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 17:40:47 +03:00
mixandClaude Opus 5.5 4dd46ea355 dmarc: a report is bounded, kept per domain, and ingested one at a time
Security review of the DMARC report path (Fable, 2026-10-11), which ran
for the first time in 2.0.0. No Critical or High finding. Fixed:

- Medium: a report was bounded only as 8 MiB of XML - some 160 000
  records or one name of megabytes, all of it rows and table cells. At
  most 5 000 records (more is ErrRejected, dropped), text fields cut to
  255 bytes on a character boundary, a record's count capped.
- Medium: the cap of 500 reports was for the whole server, so a stranger
  filing under one domain's address evicted every other domain's
  reports. PruneDMARCReports counts per domain.
- Low: the pipe service ran up to default_process_limit (100) commands
  at once against the panel's database; maxproc is 1.
- Low: the reason logged for a dropped message carried the sender's
  bytes (header line, zip entry name) as they came; cut to 200 bytes and
  reduced to printable ASCII, zip entry names quoted.
- Low: SELFPOST_DATA_DIR / SELFPOST_DB_PATH with whitespace or "$" would
  reach the command through export_environment as another path; refused
  at start when DMARC ingest is on.

Left as they are, told to the owner: the parse-failure count is a
lifetime figure shown under "This week"; a report larger than the
command reads is dropped as unreadable; nested multipart is not searched.

The hub's Retention line follows the design commit before this one.
security.md: the review, and the accepted-risk entry says what a
stranger can and cannot do now. The roadmap no longer calls the review
pending.

  go vet ./cmd/... ./internal/dmarc/ ./internal/store/: ok
  go test ./cmd/... ./internal/dmarc/ ./internal/store/ ./internal/web/...: ok
  make e2e on the test server (with the port-25 scenarios of the next
  commit): ok

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 17:40:31 +03:00
mixandClaude Opus 5.5 505260e227 design: the DMARC hub says reports are kept 500 a domain
The retention cap is counted per domain (the security commit after this one); the Ingest box says so: 500 reports a domain or 90 days.

check.py: 0 violations; outlines.json unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 17:40:30 +03:00
mixandClaude Opus 5.5 09d9625f32 dmarc-ingest: what is not a report this relay keeps is dropped, not retried
Owner, 2026-10-11: foreign and malformed reports are turned away.

The command answered every failure with EX_TEMPFAIL, so Postfix kept a
message that could never be stored and offered it again for days, each
attempt adding to the parse-failure count. IngestMessage now marks those
failures (unreadable as an aggregate report, a domain not sent from
here, another domain's report address) with ErrRejected; the command
counts one failure, writes the reason where Postfix puts it on the delivery line of the mail log
and reports the delivery as done. No bounce: the envelope sender of such
a message is whatever its author wrote, and the report addresses are
public. A failing store is still EX_TEMPFAIL.

  go vet ./cmd/... ./internal/dmarc/: ok
  go test ./cmd/panel/ ./internal/dmarc/ ./internal/web/...: ok

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 16:28:12 +03:00
152 changed files with 12313 additions and 442 deletions
+242
View File
@@ -5,6 +5,248 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); version
## [Unreleased]
### Added
- **Schema for notifications** (migration `0004_notifications.sql`, DDL only):
a user's own switch, `users.notify`, on for everyone; `notify_state`, what
the panel has already written to its users about. The instance switch is a
settings key, on unless turned off
([plans/panel-notifications.md](docs/plans/panel-notifications.md)).
- **The panel writes to its users when something needs them.** Four things,
each said once: a sending domain's DKIM, SPF or DMARC record stops matching
(after two hourly checks in a row, and once more when it matches again); a
level-2 rate limit refuses mail (at most once a day per limit); the TLS
certificate is 14 days from expiry, and 3; a message has waited in the
queue for more than three days (and once more when none has). Global users
are told all of it, domain users about their own domains. Sent from
`postmaster@<SELFPOST_HOSTNAME>` by local submission, one message per
recipient; not rows of the Outbound log and counted against no limit.
- **The server's own name is signed.** The panel generates a DKIM key for
`SELFPOST_HOSTNAME` and OpenDKIM signs mail submitted locally as that name
(`non_smtpd_milters`), so that notifications arrive. The name is not a
sending domain: no application can send as it.
- **Notifications in the panel.** Account has *Send me notifications* beside
the e-mail; Server › Settings has the switch for everyone and the two DNS
records of the server's own name, DKIM and SPF, with the values to publish;
Server › Health has a *Notification mail* card with the state of each. Help
has a Notifications topic. Every user starts with notifications on, and so
does the instance.
## [2.2.0] - 2026-10-12
### Added
- **Schema: `send_log_lines`** (migration `0003_send_log_lines.sql`, DDL only).
The table that will hold what Postfix wrote about a message for as long as
its send-log row lives, with the store methods to write a line, to write it
once, to read the lines of a row and to prune
([plans/delivery-log-storage.md](docs/plans/delivery-log-storage.md)).
- **The log tailer keeps what Postfix writes about queued messages.** A
`mail.log` line headed by a queue id is stored with the line's own time, one
transaction per pass over the log; after an unclean stop a line read twice
is stored once. Mail that came in on port 25 is left out: a stranger's
message never has a send-log row, and what he can make Postfix log is not
written to the database. Lines whose queue id has no send-log row - the row
was pruned, or the message never had one (bounces) - are deleted once they
are an hour old, at start and with every six-hourly send-log sweep.
### Changed
- **A message's page keeps its delivery log after `mail.log` rotates.** The
*Delivery log* box on Outbound › Log › a message was filled by searching
the current `mail.log`, which is rotated daily, so it was empty for anything
not sent today while the row itself lives for ninety days. The page now
reads the lines the tailer stored: they are there for as long as the row
is. Postfix reuses queue ids, so a row is given only the lines dated within
its own queue lifetime. A message accepted before this version shows its
lines only while they are still in the current file, as before; the note
shown when there are none and the *Outbound log* help topic say so.
### Security
- **A full backup now carries the log lines of messages in the send log** -
client addresses, SASL logins, message ids, receivers' replies - where
`mail.log` itself is not in a backup. Recorded in
[security.md](docs/security.md) with the accepted risk of a queue-id
collision showing one message's lines on another's page. Technical and
security reviews of 2026-10-12: no Critical or High finding.
## [2.1.0] - 2026-10-12
### Security
- **The level-2 rate limit and an application's client IP list held only for
the first message of a connection.** After a refused `MAIL FROM`, a second
one on the same connection was accepted, and so was every message after the
first that a connection carried: an application over its limit could go on
sending, and one restricted to certain client addresses could send from any
other with a retry. In the code since `1.9.0` (the client IP list) and
earlier (the limit); found by the technical review of 2026-10-12 and
reproduced on the test stand. Postfix now gives the panel's milters the
client address with every message, the limit no longer depends on it, and an
application with a client IP list is refused when no address is known.
- Reviews of the inbound antispam feature before its release (technical and
security, 2026-10-12; nothing Critical or High in the feature itself). Fixed:
a client's HELO name could supply the sender, recipient and reply of an
inbound log row; the log's row cap was applied only every six hours; rows of
a connection's later messages had no client address; two changes of the
filter lists at once could leave the filter's files behind the page; a held
message was logged twice; after an unclean stop of the panel the refusals of
its last seconds were logged twice; with the spam filter down, a sender's
own verdict headers were recorded as the filter's. Recorded as accepted in
[docs/security.md](docs/security.md): an allow entry for a sender address
trusts an address anyone can claim; the list files are readable by anyone on
the host.
- `INBOUND_ANTISPAM_MILTER` and `INBOUND_JOURNAL_MILTER_SOCKET` are checked
character by character at start; the check looked at their first characters
only.
### Changed
- **Port 25 logs as `postfix/inbound`.** Its lines in the mail log (Server ›
System log) now read `postfix/inbound/smtpd[…]` where they read
`postfix/smtpd[…]`; the submission ports keep `postfix/smtpd`. This is how a
refusal on port 25 is told from one on 465 / 587.
### Added
- Schema: migration `0002_inbound_spam_log.sql`, the first after the 2.0
baseline - two empty tables for the inbound antispam journal and the filter
lists (inbound-antispam-panel). A 2.0.x database is migrated when the panel
starts.
- **Inbound mail is recorded.** With the inbound relay on, the panel runs a
second milter, the last one on port 25, and writes a row for every recipient
of a message that was accepted for an inbound domain: client address,
envelope, subject and - with a spam filter attached - the filter's action,
score and heaviest symbols. It only watches: it refuses nothing and Postfix
accepts mail when it is down.
- **Refused inbound mail is recorded too**, from the mail log: a recipient
Postfix refused at an inbound domain (unknown user, sender domain not
found), and a message the spam filter rejected, deferred, held or
discarded. The row is dated by its log line and carries the reply text. A
refusal to relay for a domain that is not an inbound one is not recorded.
- The inbound journal is pruned every six hours: rows older than 90 days
(`INBOUND_SPAM_LOG_RETENTION_DAYS`, 7-365) and anything beyond 10 000 rows.
- The antispam compose fragment mounts `antispam/rspamd/milter_headers.conf`
into the rspamd sidecar, which makes it write its verdict into
`X-Rspamd-Action` and `X-Spamd-Result`. **Copy `deploy/antispam/rspamd/`
next to the fragment before starting it** - Docker creates a directory in
place of a missing file and the sidecar then fails to read its
configuration.
- **Allow and deny lists for the spam filter**, kept by SelfPost and applied
by rspamd: sender addresses, sender domains, client addresses and networks.
They are written as six files under `data/antispam/`, which the fragment
mounts into the sidecar together with `antispam/rspamd/multimap.conf`;
rspamd rereads a changed file by itself. Deny wins over allow. The files
are there from the first start with the inbound relay on.
- **Inbound › Log**: the journal as a page - time, sender and client address,
recipient, subject and decision, filtered by domain, decision and day. The
decision opens a sheet over the page with who decided and why: the filter's
verdict and symbols, or the reply a refusal was given. A user with the
*domain* role sees the rows of the inbound domains assigned to them. For the
global role a row links to the filter lists with the sender filled in:
*Deny sender* on accepted mail, *Allow sender* on mail the filter refused -
unless the sender is spelt in a way the lists do not take.
- **Inbound › Filter lists** (global role): add and remove allow and deny
entries; the page says where the lists go and whether the filter is on to
read them. Each entry shows how often it decided a message - *Matched*, or
*never* - and its kind opens a sheet with the note, the day it was added and
the last match, so that entries nothing matches any more can be found and
removed. The count is read from the inbound log and reaches back as far as
the log does. A network of one address (`192.0.2.7/32`) is stored as the
address, so it cannot be on a list twice.
- **Server › Settings**: *Keep inbound log rows, days*, shown with the inbound
relay on.
- Help: *Inbound log* and *Filter lists*.
- New setting `ANTISPAM_DIR` (default `/data/antispam`).
- **The spam filter signs what it judged.** rspamd adds `X-SelfPost-Filter` to
every message it lets through - an HMAC of the message's queue id under a
key SelfPost creates in `data/antispam/filter.key` - and the inbound log
shows a verdict only next to a signature that fits. Nothing to configure,
but the sidecar needs the new `antispam/rspamd/milter_headers.conf` and a
restart to read it; until then accepted mail is logged as *selfpost*, with
no verdict.
- A full backup carries the inbound log, the filter lists and their files
under `data/antispam/`. It does not carry the `antispam/` directory with the
compose fragment and the rspamd configuration - copy that from
`deploy/antispam/` when restoring.
- New setting `INBOUND_JOURNAL_MILTER_SOCKET` (default
`/run/selfpost/inbound-journal.sock`).
## [2.0.1] - 2026-10-11
### Changed
- **A message that is not a report this relay keeps is dropped at once.**
One that cannot be read as an aggregate report, reports on a domain not
sent from here, or came to another domain's report address used to be put
off and offered again by Postfix for days, counted as a parse failure each
time. It is now accepted and discarded - no retry, and no bounce, since the
envelope sender of such a message is whatever its author wrote. It counts
as one parse failure and the reason is in the mail log, on the message's
own delivery line (Server › System log, `dropped: …`). Only a failure of
the database is still retried.
- **A full backup no longer holds the Postfix queue**, from the panel or from
`selfpost-backup`. It is taken while the instance runs, and a copy of a
queue Postfix is working on can hold a message twice or not at all. Mail
still waiting for delivery is not carried over; a restored instance starts
with an empty queue. To keep the queue, stop the instance and `tar` the
project directory.
- DMARC reports are kept 500 **for each domain** (and 90 days), not 500 for
the whole server - see Security.
### Fixed
- **Full backup from the panel works.** Since the Postfix queue moved under
`/data` in 1.3.0 the button answered with a 200 and ten bytes: the archive
includes the queue, Postfix keeps the queue readable by its own user alone,
and the panel is not that user ("open /data/postfix/queue/active:
permission denied" in the log). `selfpost-backup` run with `docker exec`
was not affected - it runs as root. The queue is no longer part of a backup
(see Changed), so the two archives are the same again. Everything is read
through before the download starts, so a failure is a message on the page
and not a file cut short. One thing the panel may still be unable to read
is the private key in `certs/`: it then refuses and says what to do -
`chgrp 998 certs/privkey.pem && chmod 640 certs/privkey.pem` on the host,
998 being the container's `selfpost` group, whose id is now fixed in the
image.
### Security
Review of the DMARC report path (Fable, 2026-10-11), which started to run for
the first time in 2.0.0. Nothing Critical or High.
- **A report is bounded before it is stored.** The XML comes from anyone who
knows a report address, and its 8 MiB limit left room for some 160 000
records or a single name of megabytes - rows in the database and cells of
the report page. A report holds at most 5 000 records (a larger one is
dropped), every text field is cut to 255 bytes, and a record's count is
capped so the sums cannot wrap.
- **One domain's reports cannot push out another's.** The cap of 500 was for
the whole server, so a stranger filing reports under one domain's address
could evict every other domain's history. It is counted per domain now.
- The delivery command runs one at a time, not up to a hundred in parallel.
- The reason written to the mail log for a dropped message is cut to 200
bytes and anything outside printable ASCII is replaced: parts of it are the
sender's own bytes (a header line, a file name inside an archive).
- `SELFPOST_DATA_DIR` / `SELFPOST_DB_PATH` with whitespace or `$` are refused
at start when DMARC ingest is on, instead of handing the delivery command
another path and so an empty database of its own.
### Added
- e2e: the run now ends on port 25 with both optional features on. A DMARC
report is mailed to a domain's hosted address and has to appear under
Outbound › DMARC reports; a report on another domain and a message with no
report are accepted and dropped; mail for an inbound domain reaches its
upstream and other recipients are refused. Until now the suite only read
back the generated configuration of these features.
- e2e: full backup and restore. The run takes a backup from the panel, throws
the installation away, unpacks the archive into the empty project directory
and starts it: the administrator signs in, the domain has the same DKIM key,
and mail sent by the application verifies against the record published
before the backup.
## [2.0.0] - 2026-10-11
The panel redesign. A breaking release: every panel URL changed and the
+1 -1
View File
@@ -112,7 +112,7 @@ docker run --rm -d --name selfpost-try \
-e SELFPOST_HOSTNAME=mail.local.test \
-e PANEL_COOKIE_SECURE=false \
-v selfpost-try-data:/data \
ghcr.io/mixeme/selfpost:2.0.0
ghcr.io/mixeme/selfpost:2.2.0
```
**Get the setup URL** (pick one):
+7 -2
View File
@@ -62,7 +62,10 @@ RUN useradd --system --no-create-home --shell /usr/sbin/nologin panel
# entrypoint.sh) — lets OpenDKIM read the panel-owned keys and lets Postfix read
# the sasldb2/sender map, and lets the panel reach the supervisor control socket
# to signal OpenDKIM/Postfix reloads without any process running as root.
RUN groupadd --system selfpost \
# The id is fixed, and it is the one the group has always had: an operator
# gives the TLS private key on the host to this group by number, so that the
# panel can put it into a full backup (guide § Taking a full backup).
RUN groupadd --system --gid 998 selfpost \
&& usermod -aG selfpost panel \
&& usermod -aG selfpost opendkim \
&& usermod -aG selfpost postfix
@@ -73,7 +76,9 @@ RUN groupadd --system selfpost \
# with `no-new-privileges`, which disables setgid transitions, so the panel
# would always see "Permission denied". Making `panel` a real member of postdrop
# grants the same access without relying on a setgid escalation the hardening
# deliberately forbids. postdrop membership is read-side only: it does not let
# deliberately forbids. The same membership is what lets the panel submit its
# own notifications with sendmail(1) - postdrop, too, is setgid and would fail
# to write the maildrop directory (internal/notify/mailer.go). It does not let
# the panel bypass any Postfix restriction that a local user does not already
# have through the world-executable sendmail.
RUN usermod -aG postdrop panel
+9
View File
@@ -112,6 +112,15 @@ for f in relay_domains transport relay_recipients tls_policy; do
done
chown panel:selfpost /data/postfix
chmod 2750 /data/postfix
# Filter lists for the external spam filter (written by the panel when
# INBOUND_RELAY_ENABLE=true, read by the antispam sidecar, which mounts this
# directory). The sidecar runs under a user of its own in another container,
# so the directory and its files are readable by anyone: they hold sender
# addresses and networks, no secrets.
mkdir -p /data/antispam
chown panel:selfpost /data/antispam
chmod 0755 /data/antispam
chown panel:selfpost /data/postfix/sender_login_maps
chmod 0640 /data/postfix/sender_login_maps
+64 -3
View File
@@ -117,6 +117,9 @@ SUBMISSION_ENABLE="${SUBMISSION_ENABLE:-false}"
INBOUND_RELAY_ENABLE="${INBOUND_RELAY_ENABLE:-false}"
INBOUND_ANTISPAM_MILTER="${INBOUND_ANTISPAM_MILTER:-}"
INBOUND_ANTISPAM_MILTER_ACTION="${INBOUND_ANTISPAM_MILTER_ACTION:-accept}"
# The panel's inbound journal milter (monitoring, fail-open), on port 25 only
# and only while the inbound relay is on. Matched by postfix-wrapper.sh.
INBOUND_JOURNAL_SOCK="${INBOUND_JOURNAL_MILTER_SOCKET:-/run/selfpost/inbound-journal.sock}"
INBOUND_RATE_LIMIT_MESSAGES_PER_IP="${INBOUND_RATE_LIMIT_MESSAGES_PER_IP:-20}"
INBOUND_MESSAGE_SIZE_LIMIT="${INBOUND_MESSAGE_SIZE_LIMIT:-26214400}"
RELAY_DOMAINS_MAP="${POSTFIX_RELAY_DOMAINS:-/data/postfix/relay_domains}"
@@ -221,11 +224,25 @@ postconf -e \
# if it is unreachable, defer rather than send unsigned). The journal-milter is
# monitoring only and is fail-open (default_action=accept): its failure must
# never block the relay. Per-milter settings use Postfix 3.0+ brace syntax.
#
# non_smtpd_milters is the chain for mail submitted locally with sendmail(1) -
# the panel's own notifications, sent as postmaster@ the hostname. OpenDKIM
# signs them with the key of the server's own name; the journal-milter is not
# on this chain, since they are not an application's mail. Bounces Postfix
# generates itself do not pass it (internal_mail_filter_classes is empty).
postconf -e \
"milter_protocol=6" \
"milter_default_action=tempfail" \
"smtpd_milters={ unix:${OPENDKIM_SOCK}, default_action=tempfail }, { unix:${JOURNAL_SOCK}, default_action=accept }" \
"non_smtpd_milters="
"non_smtpd_milters={ unix:${OPENDKIM_SOCK}, default_action=tempfail }"
# What Postfix tells a milter with MAIL FROM: its defaults plus the client's
# address. The panel's milters judge every message by that address (an
# application's client-address list) and record it (the inbound journal), and
# the address given once at connect is not there for a connection's later
# messages - see MailFrom in internal/milter/milter.go.
postconf -e \
"milter_mail_macros=i {auth_type} {auth_authen} {auth_author} {mail_addr} {mail_host} {mail_mailer} {client_addr}"
# Bounded milter timeouts: a *hung* milter (socket accepts but never replies)
# must fail open just like a crash, not stall mail acceptance until the Postfix
@@ -291,11 +308,23 @@ if [ "${INBOUND_RELAY_ENABLE}" = "true" ] || [ "${DMARC_REPORTS_ENABLE}" = "true
# attribute" on every report. They go into the service line itself.
# ${recipient} is expanded by pipe(8), not by the shell: the command
# learns which report address the message came to from its argument.
postconf -M 'dmarc-ingest/unix=dmarc-ingest unix - n n - - pipe flags=Rq user=panel argv=/usr/local/bin/dmarc-ingest ${recipient}'
# One at a time (the 1 before "pipe"): reports are in no hurry, every run
# writes to the database the panel writes to, and a hundred of them in
# parallel - Postfix's default - is what a flood of reports would start.
postconf -M 'dmarc-ingest/unix=dmarc-ingest unix - n n - 1 pipe flags=Rq user=panel argv=/usr/local/bin/dmarc-ingest ${recipient}'
# One recipient per run, so ${recipient} is one argument.
postconf -e "dmarc-ingest_destination_recipient_limit=1"
# pipe(8) starts the command with export_environment only, so what the
# command reads from the container's environment is named there.
# export_environment is a list split at whitespace in which Postfix
# expands "$": a path with either would reach the command as another
# path, and it would quietly open a database of its own.
case "${SELFPOST_DATA_DIR:-}${SELFPOST_DB_PATH:-}" in
*[[:space:]]* | *'$'*)
echo "postfix-config: SELFPOST_DATA_DIR and SELFPOST_DB_PATH must not contain whitespace or \$ when DMARC_REPORTS_ENABLE=true" >&2
exit 1
;;
esac
DMARC_INGEST_ENV="DMARC_REPORTS_ENABLE=true"
[ -z "${SELFPOST_DATA_DIR:-}" ] || DMARC_INGEST_ENV="${DMARC_INGEST_ENV} SELFPOST_DATA_DIR=${SELFPOST_DATA_DIR}"
[ -z "${SELFPOST_DB_PATH:-}" ] || DMARC_INGEST_ENV="${DMARC_INGEST_ENV} SELFPOST_DB_PATH=${SELFPOST_DB_PATH}"
@@ -345,10 +374,18 @@ if [ "${INBOUND_RELAY_ENABLE}" = "true" ] || [ "${DMARC_REPORTS_ENABLE}" = "true
# would file inbound mail into the outbound send log under the *sender's*
# domain — polluting Deliveries and letting a forged From: consume that
# domain's level-2 rate-limit budget. So smtpd_milters is always overridden
# here: either the optional antispam milter, or nothing at all.
# here: the optional antispam milter, then - with the inbound relay on - the
# panel's inbound journal milter, or nothing at all.
INBOUND_MILTERS=""
if [ "${INBOUND_RELAY_ENABLE}" = "true" ] && [ -n "${INBOUND_ANTISPAM_MILTER}" ]; then
# A glob's "*" matches anything, so the shape alone would pass
# "inet:a:1},{unix:/elsewhere": characters outside the set are refused
# first, then the shape is checked.
case "${INBOUND_ANTISPAM_MILTER}" in
*[!A-Za-z0-9._:/-]*)
echo "FATAL: INBOUND_ANTISPAM_MILTER must be inet:host:port or unix:/path, got: ${INBOUND_ANTISPAM_MILTER}" >&2
exit 1
;;
inet:[A-Za-z0-9._-]*:[0-9]* | unix:/[A-Za-z0-9._/-]* ) ;;
*)
echo "FATAL: INBOUND_ANTISPAM_MILTER must be inet:host:port or unix:/path, got: ${INBOUND_ANTISPAM_MILTER}" >&2
@@ -364,6 +401,26 @@ if [ "${INBOUND_RELAY_ENABLE}" = "true" ] || [ "${DMARC_REPORTS_ENABLE}" = "true
esac
INBOUND_MILTERS="{${INBOUND_ANTISPAM_MILTER},default_action=${INBOUND_ANTISPAM_MILTER_ACTION}}"
fi
if [ "${INBOUND_RELAY_ENABLE}" = "true" ]; then
# Last in the chain: Postfix hands a message to its milters one after
# another at end-of-message, so this one reads the verdict headers the
# filter added and sees only what the filter let through. It watches and
# never refuses, and must never hold up someone else's mail: accept when
# it is down. A comma, not a space, between the two - see
# master_cf_set_smtp_inet_o.
case "${INBOUND_JOURNAL_SOCK}" in
*[!A-Za-z0-9._/-]*)
echo "FATAL: INBOUND_JOURNAL_MILTER_SOCKET must be an absolute path, got: ${INBOUND_JOURNAL_SOCK}" >&2
exit 1
;;
/[A-Za-z0-9._/-]*) ;;
*)
echo "FATAL: INBOUND_JOURNAL_MILTER_SOCKET must be an absolute path, got: ${INBOUND_JOURNAL_SOCK}" >&2
exit 1
;;
esac
INBOUND_MILTERS="${INBOUND_MILTERS:+${INBOUND_MILTERS},}{unix:${INBOUND_JOURNAL_SOCK},default_action=accept}"
fi
PORT25_RATE="${INBOUND_RATE_LIMIT_MESSAGES_PER_IP}"
PORT25_SIZE="${INBOUND_MESSAGE_SIZE_LIMIT}"
@@ -380,8 +437,12 @@ if [ "${INBOUND_RELAY_ENABLE}" = "true" ] || [ "${DMARC_REPORTS_ENABLE}" = "true
fi
fi
# syslog_name: this listener's lines read postfix/inbound/smtpd, the
# submission ports' postfix/smtpd. The log tailer files a refusal on port 25
# into the inbound journal and has to tell the two apart.
postconf -M "smtp/inet=smtp inet n - n - - smtpd"
postconf -P \
"smtp/inet/syslog_name=postfix/inbound" \
"smtp/inet/smtpd_sasl_auth_enable=no" \
"smtp/inet/smtpd_tls_auth_only=no" \
"smtp/inet/smtpd_sender_login_maps=" \
+10 -4
View File
@@ -2,8 +2,9 @@
# Postfix start wrapper: solves the cold-start race where Postfix would try to
# reach the milter sockets before they are listening.
#
# It blocks until BOTH milter sockets — OpenDKIM and the panel's journal-milter
# — are present, then execs `postfix start-fg`. If they are not ready within the
# It blocks until the milter sockets — OpenDKIM, the panel's journal-milter
# and, with the inbound relay on, its inbound journal milter — are present,
# then execs `postfix start-fg`. If they are not ready within the
# timeout it exits non-zero WITHOUT starting Postfix, so supervisord/Docker see
# an explicit startup failure instead of a relay running blind.
#
@@ -16,8 +17,13 @@ JOURNAL_SOCK="${JOURNAL_MILTER_SOCKET:-/run/selfpost/journal.sock}"
TIMEOUT="${MILTER_WAIT_TIMEOUT:-30}"
INTERVAL=1
SOCKS="$OPENDKIM_SOCK $JOURNAL_SOCK"
if [ "${INBOUND_RELAY_ENABLE:-false}" = "true" ]; then
SOCKS="$SOCKS ${INBOUND_JOURNAL_MILTER_SOCKET:-/run/selfpost/inbound-journal.sock}"
fi
elapsed=0
for sock in "$OPENDKIM_SOCK" "$JOURNAL_SOCK"; do
for sock in $SOCKS; do
while [ ! -S "$sock" ]; do
if [ "$elapsed" -ge "$TIMEOUT" ]; then
echo "postfix-wrapper: timed out after ${TIMEOUT}s waiting for milter socket $sock" >&2
@@ -29,5 +35,5 @@ for sock in "$OPENDKIM_SOCK" "$JOURNAL_SOCK"; do
echo "postfix-wrapper: milter socket ready: $sock"
done
echo "postfix-wrapper: both milter sockets ready, starting postfix"
echo "postfix-wrapper: milter sockets ready, starting postfix"
exec postfix start-fg
+35 -1
View File
@@ -1,6 +1,7 @@
package main
import (
"errors"
"fmt"
"io"
"log"
@@ -28,14 +29,47 @@ func runDMARCIngestMode() error {
recipient := ingestRecipient(os.Args[1:])
if err := dmarc.IngestMessage(st, io.LimitReader(os.Stdin, maxIngestStdin), recipient, time.Now().UTC()); err != nil {
if !errors.Is(err, dmarc.ErrRejected) {
return err
}
// Not a report this relay keeps. It is dropped: deferring would have
// Postfix try it again for days, and a bounce would go to an envelope
// sender anyone can write - the report addresses are public.
if incrErr := st.IncrDMARCParseFailures(); incrErr != nil {
log.Printf("dmarc ingest: record failure: %v", incrErr)
}
return err
logRejected(err)
}
return nil
}
// logRejected leaves the reason in the mail log. Postfix puts the start of
// what a delivery command wrote - about a hundred bytes - into its own line
// for the message ("status=sent (delivered via dmarc-ingest service (...))"),
// so the reason goes out first and bare: no prefix, no date, no recipient,
// all of which that line already has. postlog is no use here: pipe(8) starts
// the command without the group that may write to Postfix's log socket.
func logRejected(reason error) {
fmt.Fprintln(os.Stderr, "dropped: "+logSafe(strings.TrimPrefix(reason.Error(), dmarc.ErrRejected.Error()+": ")))
}
// logSafe makes a reason fit for a log line. Parts of it are the sender's own
// bytes - a header line, a file name inside an archive, a piece of XML - so it
// is cut short and anything that is not printable ASCII becomes "?", whatever
// Postfix would have done with it.
func logSafe(s string) string {
const max = 200
if len(s) > max {
s = s[:max]
}
return strings.Map(func(r rune) rune {
if r < 0x20 || r > 0x7e {
return '?'
}
return r
}, s)
}
// ingestRecipient is the report address the message was delivered to. Postfix's
// pipe(8) hands it over as an argument (${recipient} in the service's argv,
// build/postfix-config.sh) - it sets no RECIPIENT variable, that is local(8).
+14
View File
@@ -18,3 +18,17 @@ func TestIngestRecipient(t *testing.T) {
}
}
}
func TestLogSafe(t *testing.T) {
got := logSafe("zip entry \"a\nstatus=bounced\tx\" \x1b[31m é")
if want := "zip entry \"a?status=bounced?x\" ?[31m ?"; got != want {
t.Errorf("logSafe = %q, want %q", got, want)
}
long := make([]byte, 1000)
for i := range long {
long[i] = 'a'
}
if got := logSafe(string(long)); len(got) != 200 {
t.Errorf("logSafe kept %d bytes of 1000, want 200", len(got))
}
}
+14 -2
View File
@@ -24,6 +24,7 @@ type mailStack struct {
Domains *domain.Service
Apps *app.Service
Inbound *inbound.Service
Lists *inbound.Lists
DMARC *dmarc.Service
pf *postfix.Postfix
odk *domain.OpenDKIM
@@ -36,11 +37,15 @@ func newMailStack(cfg config, st *store.Store) *mailStack {
odk := domain.NewOpenDKIM(cfg.opendkimDir)
apps := app.NewService(st, app.NewSASLDB(cfg.saslDBPath, cfg.saslRealm), pf)
domains := domain.NewService(st, odk, apps, cfg.dkimSelectorDef)
// The panel's notifications are sent as postmaster@ the server's own name,
// and signed as it.
domains.SetServerName(cfg.hostname)
inb := inbound.NewService(st, pf)
dmarcSvc := dmarc.NewService(st, pf, cfg.hostname, cfg.dmarcEnabled)
return &mailStack{
Domains: domains, Apps: apps, Inbound: inb, DMARC: dmarcSvc,
pf: pf, odk: odk, inboundEnabled: cfg.inboundEnabled, dmarcEnabled: cfg.dmarcEnabled,
Lists: inbound.NewLists(st, cfg.antispamDir),
pf: pf, odk: odk, inboundEnabled: cfg.inboundEnabled, dmarcEnabled: cfg.dmarcEnabled,
}
}
@@ -57,6 +62,11 @@ func (m *mailStack) Resync() error {
if err := m.Inbound.Resync(); err != nil {
return fmt.Errorf("inbound maps resync: %w", err)
}
// The filter's configuration names all six list files; they exist from
// the first start, empty until an entry is added.
if err := m.Lists.Sync(); err != nil {
return fmt.Errorf("filter lists resync: %w", err)
}
}
if m.dmarcEnabled {
if err := m.DMARC.Resync(); err != nil {
@@ -91,7 +101,7 @@ func newPanel(cfg config, st *store.Store) (*web.Server, error) {
// effective config, including a manual override; the panel keeps this
// snapshot for the process lifetime (architecture.md).
retryPolicy := postfix.LoadRetryPolicy()
return web.New(st, ms.Domains, ms.Apps, ms.Inbound, ms.DMARC, web.Config{
return web.New(st, ms.Domains, ms.Apps, ms.Inbound, ms.Lists, ms.DMARC, web.Config{
Hostname: cfg.hostname,
CookieSecure: cfg.cookieSecure,
SubmissionEnabled: cfg.submissionEnabled,
@@ -114,6 +124,8 @@ func newPanel(cfg config, st *store.Store) (*web.Server, error) {
InboundAntispamAction: cfg.inboundAntispamAction,
DMARCEnabled: cfg.dmarcEnabled,
SendLogRetentionEnvDefault: cfg.retentionDays,
InboundSpamLogRetentionEnvDefault: cfg.inboundSpamLogRetentionDays,
}, cfg.setupTokenPath)
}
+39 -9
View File
@@ -8,6 +8,7 @@ import (
"os"
"path/filepath"
"github.com/mixeme/selfpost/internal/inbound"
"github.com/mixeme/selfpost/internal/milter"
"github.com/mixeme/selfpost/internal/store"
)
@@ -15,21 +16,52 @@ import (
// serveJournal opens the journal-milter Unix socket and runs the real milter
// (architecture.md § Mail path), recording accepted messages into the send
// log. Socket lifecycle (creation, stale cleanup, group permissions) lives
// here; the protocol handler lives in internal/milter.
// in listenMilter; the protocol handler lives in internal/milter.
func serveJournal(ctx context.Context, cfg config, st *store.Store) error {
socketPath := cfg.journalSocket
if err := os.MkdirAll(filepath.Dir(socketPath), 0o755); err != nil {
ln, err := listenMilter(cfg.journalSocket)
if err != nil {
return err
}
log.Printf("journal-milter listening on %s", cfg.journalSocket)
return milter.Serve(ctx, ln, st)
}
// serveInboundJournal runs the inbound journal milter, the last milter on
// port 25 (internal/milter/inbound.go). It exists only while the inbound relay
// is on: build/postfix-config.sh attaches the socket under the same condition.
func serveInboundJournal(ctx context.Context, cfg config, st *store.Store) error {
ln, err := listenMilter(cfg.inboundJournalSocket)
if err != nil {
return err
}
// With a filter attached, the key it signs with. Without the key no
// verdict can be told from a sender's own, so none is read: the journal
// still records the mail, and the log says why it shows no verdicts.
key := ""
if cfg.inboundAntispamMilter != "" {
if key, err = inbound.FilterKey(cfg.antispamDir); err != nil {
log.Printf("inbound-journal-milter: no key for the filter's signature, verdicts will not be recorded: %v", err)
key = ""
}
}
log.Printf("inbound-journal-milter listening on %s", cfg.inboundJournalSocket)
return milter.ServeInbound(ctx, ln, st, key)
}
// listenMilter opens a milter's Unix socket where Postfix can reach it.
func listenMilter(socketPath string) (net.Listener, error) {
if err := os.MkdirAll(filepath.Dir(socketPath), 0o755); err != nil {
return nil, err
}
// Clear a stale socket left behind by an unclean shutdown, otherwise the
// listen below fails with "address already in use".
if err := os.Remove(socketPath); err != nil && !errors.Is(err, os.ErrNotExist) {
return err
return nil, err
}
ln, err := net.Listen("unix", socketPath)
if err != nil {
return err
return nil, err
}
// Postfix (user `postfix`) connects to this socket as a milter and needs
// group access. The socket inherits group `selfpost` from the setgid parent
@@ -37,9 +69,7 @@ func serveJournal(ctx context.Context, cfg config, st *store.Store) error {
// it (connecting to a Unix socket needs write permission on the node).
if err := os.Chmod(socketPath, 0o660); err != nil {
ln.Close()
return err
return nil, err
}
log.Printf("journal-milter listening on %s", socketPath)
return milter.Serve(ctx, ln, st)
return ln, nil
}
+60 -4
View File
@@ -23,6 +23,7 @@ import (
"github.com/mixeme/selfpost/internal/backup"
"github.com/mixeme/selfpost/internal/buildinfo"
"github.com/mixeme/selfpost/internal/dnscheck"
"github.com/mixeme/selfpost/internal/inbound"
"github.com/mixeme/selfpost/internal/logtail"
"github.com/mixeme/selfpost/internal/store"
)
@@ -89,6 +90,9 @@ type config struct {
saslRealm string
postfixDir string
deployRoot string
// Where the filter lists are written as map files for the external spam
// filter; the antispam compose fragment mounts it into the sidecar.
antispamDir string
inboundEnabled bool
// The optional milter on the inbound listener (INBOUND_ANTISPAM_MILTER) and
@@ -96,7 +100,12 @@ type config struct {
// what attaches it; the panel only says so. See antispamAction.
inboundAntispamMilter string
inboundAntispamAction string
dmarcEnabled bool
// The inbound journal milter's socket; the default mirrors
// build/postfix-config.sh. Served only while the inbound relay is on.
inboundJournalSocket string
// Retention of the inbound journal until Server > Settings says otherwise.
inboundSpamLogRetentionDays int
dmarcEnabled bool
}
func loadConfig() config {
@@ -157,11 +166,16 @@ func loadConfig() config {
saslDBPath: envDefault("SASL_DB_PATH", filepath.Join(dataDir, "sasl", "sasldb2")),
saslRealm: saslRealm(),
postfixDir: envDefault("POSTFIX_DIR", filepath.Join(dataDir, "postfix")),
deployRoot: envDefault("SELFPOST_DEPLOY_ROOT", "/selfpost-deploy"),
antispamDir: envDefault("ANTISPAM_DIR", filepath.Join(dataDir, "antispam")),
deployRoot: envDefault("SELFPOST_DEPLOY_ROOT", "/selfpost-deploy"),
// Optional inbound relay (backup-MX / forwarder). Off unless exactly "true".
inboundEnabled: os.Getenv("INBOUND_RELAY_ENABLE") == "true",
inboundAntispamMilter: os.Getenv("INBOUND_ANTISPAM_MILTER"),
inboundAntispamAction: antispamAction(os.Getenv("INBOUND_ANTISPAM_MILTER_ACTION")),
inboundJournalSocket: envDefault("INBOUND_JOURNAL_MILTER_SOCKET", "/run/selfpost/inbound-journal.sock"),
inboundSpamLogRetentionDays: envInt("INBOUND_SPAM_LOG_RETENTION_DAYS", store.InboundSpamLogRetentionDaysDefault),
// Optional DMARC aggregate ingest on port 25. Off unless exactly "true".
dmarcEnabled: os.Getenv("DMARC_REPORTS_ENABLE") == "true",
}
@@ -289,12 +303,46 @@ func run() error {
}
}
// The filter lists as files, on every start: the spam filter's
// configuration names all six, and a data directory from before the lists
// existed has none. Not fatal - the lists are the filter's business, and
// the panel has mail to relay either way.
if cfg.inboundEnabled && !restored {
if err := inbound.NewLists(st, cfg.antispamDir).Sync(); err != nil {
log.Printf("filter lists bootstrap: %v", err)
}
}
// The key the filter signs with has to be in the shared directory before
// the filter reads it, whether or not a filter is attached yet.
if cfg.inboundEnabled {
if _, err := inbound.FilterKey(cfg.antispamDir); err != nil {
log.Printf("filter key bootstrap: %v", err)
}
}
if err := st.EnsureSendLogRetentionDays(cfg.retentionDays); err != nil {
return fmt.Errorf("bootstrap send-log retention: %w", err)
}
// With the inbound relay on, the tailer also writes the inbound journal's
// rows for refused mail and prunes that journal.
var inboundTail *logtail.Inbound
if cfg.inboundEnabled {
if err := st.EnsureInboundSpamLogRetentionDays(cfg.inboundSpamLogRetentionDays); err != nil {
return fmt.Errorf("bootstrap inbound journal retention: %w", err)
}
inboundTail = &logtail.Inbound{Store: st, Retention: func() int {
days, err := st.GetInboundSpamLogRetentionDays(cfg.inboundSpamLogRetentionDays)
if err != nil {
log.Printf("inbound journal retention: %v", err)
return cfg.inboundSpamLogRetentionDays
}
return days
}}
}
var wg sync.WaitGroup
errc := make(chan error, 4)
// Room for every role to report a failure without blocking: six at most.
errc := make(chan error, 6)
roles := []struct {
name string
@@ -310,12 +358,20 @@ func run() error {
return cfg.retentionDays
}
return days
})
}, inboundTail)
}},
{"rate-limit-recalc", func(ctx context.Context) error {
runAutoRateLimitRecalc(ctx, cfg, st)
return nil
}},
{"notifier", func(ctx context.Context) error { return runNotifier(ctx, cfg, st) }},
}
if cfg.inboundEnabled {
roles = append(roles, struct {
name string
fn func(context.Context) error
}{"inbound-journal-milter", func(ctx context.Context) error { return serveInboundJournal(ctx, cfg, st) }})
}
for _, r := range roles {
+42
View File
@@ -0,0 +1,42 @@
package main
import (
"context"
"github.com/mixeme/selfpost/internal/dnscheck"
"github.com/mixeme/selfpost/internal/health"
"github.com/mixeme/selfpost/internal/notify"
"github.com/mixeme/selfpost/internal/postfix"
"github.com/mixeme/selfpost/internal/store"
)
// runNotifier writes to the panel's users when something needs them
// (architecture.md § Notifications), until ctx is cancelled. It reads what the
// pages read - a domain's published records against the key this server signs
// with, the certificate file, Postfix's queue - with a DNS checker of its own,
// so that its hourly look is never a page's cached one.
func runNotifier(ctx context.Context, cfg config, st *store.Store) error {
ms := newMailStack(cfg, st)
dns := dnscheck.New(cfg.dnsResolvers)
n := notify.New(st, notify.Checks{
Domain: func(d store.Domain) (dnscheck.Domain, error) {
rec, err := ms.Domains.DKIMRecord(d)
if err != nil {
return dnscheck.Domain{}, err
}
srv := dns.Server(cfg.hostname, true)
return dns.Domain(dnscheck.Query{
Name: d.Name,
Selector: d.DKIMSelector,
ExpectedDKIM: rec.Value,
Hostname: srv.Hostname,
ServerIPs: srv.IPs,
DMARCReportEmail: d.DMARCRua,
}, true), nil
},
Certificate: func() health.Certificate { return health.CheckCertificate(cfg.tlsCertFile) },
Queue: postfix.QueueMessages,
}, notify.NewMailer(cfg.hostname), cfg.hostname)
return n.Run(ctx)
}
+4
View File
@@ -30,6 +30,10 @@ DMARC_REPORTS_ENABLE=false
# INBOUND_ANTISPAM_MILTER=inet:antispam:11332
# INBOUND_ANTISPAM_MILTER_ACTION=accept
# Days the inbound journal keeps a row (7-365) until the panel says otherwise;
# it never holds more than 10 000 rows.
# INBOUND_SPAM_LOG_RETENTION_DAYS=90
# Coarse per-client-IP limit and message size on inbound smtpd (port 25).
# INBOUND_RATE_LIMIT_MESSAGES_PER_IP=20
# INBOUND_MESSAGE_SIZE_LIMIT=26214400
@@ -27,3 +27,10 @@ services:
image: rspamd/rspamd:3.11
restart: unless-stopped
# No host ports: only SelfPost talks to the milter on the compose network.
volumes:
# The verdict headers SelfPost's inbound journal reads.
- ./antispam/rspamd/milter_headers.conf:/etc/rspamd/local.d/milter_headers.conf:ro
# The allow and deny lists kept in the panel: the rules, and the files
# SelfPost writes them to.
- ./antispam/rspamd/multimap.conf:/etc/rspamd/local.d/multimap.conf:ro
- ./data/antispam:/etc/rspamd/selfpost:ro
@@ -0,0 +1,45 @@
# Mounted into the sidecar as /etc/rspamd/local.d/milter_headers.conf.
#
# rspamd writes its verdict into two headers of every message it lets through:
# the action it chose and the score with the symbols behind it. SelfPost's
# inbound journal reads them (Inbound > Log shows them next to the message),
# and the upstream mailbox server receives them with the mail.
#
# rspamd removes headers of these names that a sender wrote himself before it
# adds its own. Without this file the journal still lists accepted mail, with
# no verdict next to it.
#
# The third header is the proof that the other two are rspamd's. The filter is
# attached so that mail still flows when it is down, and then nothing removes
# what a sender wrote: he could send his own verdict. So rspamd signs each
# message - an HMAC-SHA256 of the message's queue id under a key SelfPost
# generates and leaves in the directory it shares with this container - and
# SelfPost reads a verdict only next to a signature it can recompute. The
# signature travels on with the mail and is of no use to whoever sees it: it
# fits that one queue id.
use = ["x-spamd-result", "x-rspamd-action", "selfpost-signature"];
custom {
selfpost-signature = <<EOD
return function(task, common_meta)
local remove = { ['X-SelfPost-Filter'] = 0 }
-- The key is read for every message and kept nowhere: SelfPost may make a
-- new one (the file was lost, a backup was restored), and a key remembered
-- here would go on signing what SelfPost no longer accepts until someone
-- restarted this container. The file is one short line.
local key
local f = io.open('/etc/rspamd/selfpost/filter.key', 'r')
if f then
key = f:read('*l')
f:close()
end
local qid = task:get_queue_id()
if not key or key == '' or not qid or qid == '' then
return nil, {}, remove, {}
end
local hash = require "rspamd_cryptobox_hash"
local sig = hash.create_specific_keyed(key, 'sha256', qid):hex()
return nil, { ['X-SelfPost-Filter'] = sig }, remove, {}
end
EOD;
}
+63
View File
@@ -0,0 +1,63 @@
# Mounted into the sidecar as /etc/rspamd/local.d/multimap.conf.
#
# The allow and deny lists kept in the panel (Inbound > Filter lists). SelfPost
# writes them as six files under data/antispam/, which the fragment mounts at
# /etc/rspamd/selfpost; rspamd rereads a file when it changes, no restart.
#
# Deny wins over allow, whichever entries match. A denied sender is refused
# before anything else is looked at (a prefilter). An allowed one is not
# accepted outright - rspamd would then let whichever of the two rules ran
# first decide - but given a weight no spam score outweighs, which a refusal
# by the deny list still overrides.
#
# A domain entry is that domain, not its subdomains.
#
# SelfPost counts how often an entry decided a message by what these rules
# leave behind: the deny message below and the names of the allow symbols.
# Change either and Inbound > Filter lists shows "never" for every entry.
SELFPOST_DENY_IP {
type = "ip";
map = "/etc/rspamd/selfpost/deny_ip.map";
prefilter = true;
action = "reject";
message = "Refused by this server's deny list";
description = "Client address on the deny list";
}
SELFPOST_DENY_FROM {
type = "from";
map = "/etc/rspamd/selfpost/deny_from.map";
prefilter = true;
action = "reject";
message = "Refused by this server's deny list";
description = "Sender address on the deny list";
}
SELFPOST_DENY_DOMAIN {
type = "from";
filter = "email:domain";
map = "/etc/rspamd/selfpost/deny_domain.map";
prefilter = true;
action = "reject";
message = "Refused by this server's deny list";
description = "Sender domain on the deny list";
}
SELFPOST_ALLOW_IP {
type = "ip";
map = "/etc/rspamd/selfpost/allow_ip.map";
score = -100.0;
description = "Client address on the allow list";
}
SELFPOST_ALLOW_FROM {
type = "from";
map = "/etc/rspamd/selfpost/allow_from.map";
score = -100.0;
description = "Sender address on the allow list";
}
SELFPOST_ALLOW_DOMAIN {
type = "from";
filter = "email:domain";
map = "/etc/rspamd/selfpost/allow_domain.map";
score = -100.0;
description = "Sender domain on the allow list";
}
+2 -1
View File
@@ -21,7 +21,7 @@
services:
selfpost:
image: ghcr.io/mixeme/selfpost:2.0.0
image: ghcr.io/mixeme/selfpost:2.2.0
restart: unless-stopped
environment:
SELFPOST_HOSTNAME: "${SELFPOST_HOSTNAME:?set the mail/panel hostname, e.g. mail.example.com}"
@@ -47,6 +47,7 @@ services:
# How long the send log keeps rows before the background sweep deletes
# them — the main driver of /data growth over time.
SEND_LOG_RETENTION_DAYS: "${SEND_LOG_RETENTION_DAYS:-90}"
INBOUND_SPAM_LOG_RETENTION_DAYS: "${INBOUND_SPAM_LOG_RETENTION_DAYS:-90}"
# Sliding idle timeout for the panel login session, in days.
PANEL_SESSION_IDLE_DAYS: "${PANEL_SESSION_IDLE_DAYS:-7}"
# Resolvers the panel's PTR/SPF/DKIM/DMARC checks query directly, so they
+175 -17
View File
@@ -27,8 +27,8 @@ Managed programs ([build/supervisord.conf](../build/supervisord.conf)):
| Program | User | Priority | Role |
|---|---|---|---|
| `opendkim` | root → `opendkim` (`UserID` in opendkim.conf) | 100 | DKIM signing milter |
| `panel` | panel | 200 | HTTP UI + journal-milter + log-tailer + rate-limit recalc |
| `postfix` | root (wrapper) | 300 | MTA — started only after both milter sockets exist |
| `panel` | panel | 200 | HTTP UI + journal-milter + log-tailer + rate-limit recalc; with the inbound relay on also the inbound journal milter |
| `postfix` | root (wrapper) | 300 | MTA — started only after the milter sockets exist (OpenDKIM's, the journal milter's and, with the inbound relay on, the inbound journal milter's) |
| `postfix-reload` | root | — | On-demand `postfix reload` (autostart off) |
| `cert-reload` | root | 400 | Daily `postfix reload` for renewed TLS certs |
| `logrotate` | root | 400 | Periodic `mail.log` rotation |
@@ -74,10 +74,61 @@ smtp/inet` removes Debian's stock smtpd). Outbound delivery still uses the
`smtpd_tls_security_level=encrypt`.
- **25/smtp inet** — when `INBOUND_RELAY_ENABLE=true` and/or
`DMARC_REPORTS_ENABLE=true`. No SASL, no OpenDKIM, no journal-milter —
`smtpd_milters` is always overridden on this service (to the antispam milter
or to nothing), because main.cf's submission chain would otherwise let a
signing outage defer relayed mail and would file inbound mail into the
outbound send log under the sender's domain.
`smtpd_milters` is always overridden on this service, because main.cf's
submission chain would otherwise let a signing outage defer relayed mail and
would file inbound mail into the outbound send log under the sender's
domain. With the inbound relay on the chain is the optional antispam milter,
then the panel's **inbound journal milter** (unix socket
`INBOUND_JOURNAL_MILTER_SOCKET`, default
`/run/selfpost/inbound-journal.sock`, `default_action=accept`); with DMARC
ingest alone it is empty. Postfix hands a message to its milters one after
another at end-of-message, each seeing what the one before added, so the
journal milter sees only mail the filter let through and reads the filter's
verdict headers (`X-Rspamd-Action`, `X-Spamd-Result`) off it. It writes one
`inbound_spam_log` row per recipient at an inbound domain - a DMARC report
address is not one - and never refuses anything
([internal/milter/inbound.go](../internal/milter/inbound.go)). The headers
are read only while a filter is attached, and only next to the filter's
signature: a third header, `X-SelfPost-Filter`, the HMAC-SHA256 of the
message's queue id under the key in `/data/antispam/filter.key`
(`inbound.FilterKey` makes it on first start; rspamd reads it through the
mounted directory and signs in a custom routine of `milter_headers.conf`).
With the filter down - it is attached fail-open - a sender's own verdict
headers carry no signature and the row is that of mail nobody judged.
Refused mail never reaches this milter.
Refusals come from `mail.log`: this listener logs as `postfix/inbound`
(`syslog_name`), and the log tailer turns a `reject:` line of
`postfix/inbound/smtpd` (Postfix's own policy) and a `milter-reject` /
`milter-hold` / `milter-discard` line of `cleanup` (the filter's verdict,
counted only for a queue id that listener opened) into a row, dated by the
line, when the recipient is at an inbound domain
([internal/logtail/inbound.go](../internal/logtail/inbound.go)). A line
names one recipient, so a message refused at end-of-message is one row
whatever its recipient count. The tailer saves its place every five seconds,
so after an unclean stop it reads some lines again; the lines it reads on
start are written only if the same row is not there already. The
journal is pruned every six hours: rows
older than the `inbound_spam_log_retention_days` setting, then down to
10 000.
**Filter lists** (`inbound_spam_list`): allow and deny entries for the
external filter - sender address, sender domain, client address or network -
validated and normalised by `internal/inbound` (`lists.go`) and written as
six map files under `/data/antispam/` (`ANTISPAM_DIR`), each atomically, on
every change and on every start with the relay on. SelfPost never consults
them. The sidecar mounts the directory and rspamd's multimap rules
([deploy/antispam/rspamd/multimap.conf](../deploy/antispam/rspamd/multimap.conf))
read them: a deny entry is a prefilter `reject`, an allow entry a symbol
weighing -100. Allow is deliberately not a prefilter `accept`: with two
prefilters rspamd lets the one that ran first decide, and deny then won or
lost by rule order (seen on rspamd 3.11). A refusal by the deny list
overrides any score, so deny wins whichever entries match.
How often an entry decided a message is not stored: `Lists.Usage` reads it
off the journal on each view of Filter lists - the rows a list decided (the
deny rule's reply, the allow rule's symbol, which the journal milter keeps
ahead of its six-symbol cap) matched against the entries as rspamd matches
them. When that stops being good enough is written in
[plans/inbound-antispam-panel.md](plans/inbound-antispam-panel.md) § Filter
list usage.
**Inbound relay** (`INBOUND_RELAY_ENABLE=true`): accepts only
`relay_domains` + `relay_recipient_maps` (`reject_unauth_destination`,
`reject_unlisted_recipient`). Maps under `/data/postfix/` (`relay_domains`,
@@ -105,9 +156,17 @@ smtp/inet` removes Debian's stock smtpd). Outbound delivery still uses the
Per-domain keys under `/data/opendkim/keys`; `KeyTable` / `SigningTable` maintained
by the panel. Socket `/run/opendkim/opendkim.sock`.
The server's own name (`SELFPOST_HOSTNAME`) has a key and a line in both
tables too, without being a row of the registry
(`domain.Service.SetServerName`): the panel's notifications are sent as
`postmaster@` that name and signed with it. No application can send as it -
the sender map is built from the registry. A sending domain of the same name
shares its key and selector with the server, and deleting that domain leaves
the key in place.
### Panel binary ([cmd/panel](../cmd/panel))
One process, four roles:
One process, five roles (six with the inbound relay on):
1. **HTTP server** — `:8080` (`PANEL_HTTP_ADDR`); HTTPS terminated by reverse
proxy only. On start it runs `postconf -h` once for the deferred-mail retry
@@ -133,10 +192,55 @@ One process, four roles:
3. **log-tailer** — follows `MAIL_LOG`, updates send-log delivery status by
queue-id. Send-log `queued → sent` transitions depend on this goroutine alone
(`UpdateStatus` is only called from [internal/logtail](../internal/logtail/logtail.go)).
It also keeps the lines Postfix writes under a queue id, for the message's
page (§ Log tailer, *Stored lines*).
4. **rate-limit recalc** — every six hours (and on demand from Domain settings and
the application form), recomputes level-2 **Auto** rate limits from send-log statistics.
5. **notifier** — writes to the panel's users when something needs them
(§ Notifications below).
Milter chain in Postfix: OpenDKIM (tempfail) then journal (accept on failure).
Mail submitted locally with `sendmail(1)` — the notifier's — passes OpenDKIM
alone (`non_smtpd_milters`): it is signed, and it is not an application's
mail to be journalled or limited. Bounces Postfix generates pass no milter.
### Notifications
[internal/notify](../internal/notify/notify.go) reports four things to the
users of the panel: a sending domain's DNS record that stopped matching, a
level-2 rate limit that refused mail, the TLS certificate about to expire, a
message more than three days in the queue. The rules of when a message goes
out are the owner's and are written in
[plans/panel-notifications.md](plans/panel-notifications.md) § When a message
is sent; the operator's view is guide § Notifications.
- **Readings.** Once an hour, starting three minutes after the process does:
every sending domain's records through a DNS checker of the notifier's own,
forced past the cache (`cmd/panel/notifier.go` builds the same query a
domain's page does); the certificate file; `postqueue -j` for the arrival
time of each queued message. Once a minute: the send log's `rejected` rows
of the last day, per domain and application.
- **State** is a row per subject in `notify_state` (`dns:<domain>`, `cert`,
`queue`, `limit:<domain>:<login>`), which is what makes each thing be said
once and a restart say nothing again. A DNS subject goes *pending* on the
first bad check and *open*, with a message, on the second; back the same
way. The certificate's row remembers which threshold was said for which
expiry. A deleted domain's row goes with it.
- **No information is not a finding.** A lookup that did not answer, a queue
that cannot be listed, a certificate that cannot be read move no state.
- **Recipients** come from the store at the moment of sending: global users
for everything, domain users for subjects that are a domain's; only users
with an e-mail and their own switch on. One message per recipient, so users
do not see each other's addresses.
- **The instance switch** (settings key `notifications`, on unless turned
off) stops the checks themselves, not only the mail: no state moves while
it is off.
- **Sending** is `sendmail -i -f postmaster@<hostname> -- <address>` as the
unprivileged panel user, which is a member of `postdrop` and so can write
the maildrop directory without the setgid bit `no-new-privileges` forbids.
A message that could not be handed over leaves the state where it was and
is tried at the next check. The text names the page to open and has no
link: the panel does not know its own address.
### Log tailer and `mail.log` rotation
@@ -183,15 +287,66 @@ leaves every row untouched rather than closing them all. Now that the log
survives the container, reaching this path means the lines are gone for good —
rotated past fourteen files while the panel was down, or deleted.
**Stored lines** (`send_log_lines`, since 2.2.0;
[internal/logtail/lines.go](../internal/logtail/lines.go)). A send-log row
lives for the retention window, ninety days by default, and `mail.log` for a
day before it is rotated; a message's page (`/outbound/log/{id}`) shows what
Postfix wrote about it, so the tailer keeps those lines in the database as it
reads them:
- **What is kept:** a line headed by a queue id — after the timestamp, the
host name and `program[pid]:` — with that id, the line's own time and the
line as written, cut to 4096 bytes. The id is read from the head of the line
only: the tail is the far end's text. `NOQUEUE` and `warning` stand in the
same place and are not ids; the pattern accepts Postfix's short form
(capital hexadecimal) and its long form (no vowels).
- **Port-25 mail is left out.** A queue id opened by `postfix/inbound/smtpd`
is a stranger's message and never gets a send-log row; none of its lines
are stored, up to qmgr's `removed` or until another listener opens a
message under the id (4096 such ids are remembered). What a stranger can
make Postfix log is therefore not written to the database at his rate.
- **No check for a row at write time.** A message's first lines (`client=`,
`message-id=`) are logged before the journal milter writes its row at
end-of-message, so the tailer cannot ask whether the id is in the send log.
Lines that never get a row — bounces Postfix generates, port-25 mail met in
mid-flight after a restart — are kept for a while.
- **Written per pass, not per line:** the lines of one pass over the log go
in one transaction (`InsertSendLogLines`), 500 at most, before the read
offset is saved. A commit per line would be a disk sync per line on the
connection the journal milter shares.
- **Retention is one rule:** a line older than an hour whose queue id no
send-log row carries is deleted (`PruneSendLogLines`), at start and after
every six-hourly send-log sweep. That removes the lines of rows the sweep
deleted and the lines that never had a row. The send log is asked once per
queue id; the statement still walks the table, about a second per million
lines, with the connection taken meanwhile.
- **Which lines are a row's.** Postfix reuses a queue id once the message has
left the queue, and a row outlives that by months. `SendLogLines` gives a row
the lines of its queue id dated from one hour before the row was written to
the queue lifetime (the larger of `maximal_queue_lifetime` and
`bounce_queue_lifetime`) plus a day after it. That limits a chance reuse of
the id to those days; it does not exclude it. A message with several
recipients is several rows under one id, and each row's page shows all of
the message's lines.
- **Read twice, stored once.** The read offset is saved every few seconds, so
after an unclean stop some lines are read again; while the tailer is
catching up at start it writes a line only if the same line is not there.
- **Not stored:** what the tailer never read — the tail of a file rotated away
while the panel was down.
- **Size:** six to ten lines of about 200 bytes per message — more than the row
itself. The table is in the database and therefore in a full backup, which
`log/` is not ([security.md](security.md) § Backup and domain export).
**Two one-shot reads** sit beside the follow loop and are unrelated to it, both
serving panel pages on request: `TailLines` (the last *n* lines, for
`/server/log`) and `QueueLines` (the lines carrying one queue-id, for
`/outbound/log/{id}`). `QueueLines` scans a bounded tail of the current file —
`/server/log`) and `QueueLines` (the lines carrying one queue-id). `QueueLines`
is what a message's page falls back to when no lines are stored for its row —
a message accepted before 2.2.0. It scans a bounded tail of the current file —
finding a message's lines means reading rather than seeking — and matches the id
anchored on the character before it, since queue ids are hexadecimal runs and a
shorter one is regularly the tail of a longer one. Send-log rows outlive the log
(retention 90 days, rotation 14 files), so an empty result is the expected end
state for an older message and the page reports it as such, not as a failure.
shorter one is regularly the tail of a longer one. For such a message an empty
result is the expected end state once the log has rotated, and the page reports
it as such, not as a failure.
---
@@ -531,16 +686,17 @@ full backups also archive the operator's `./certs` PEM files when present.
### Database
The schema is one embedded baseline, `internal/store/migrations/0001_init.sql`,
applied by `store.Open` into an empty file; `PRAGMA user_version` is `1` and
The schema is an embedded baseline, `internal/store/migrations/0001_init.sql`,
and the migrations after it (`0002_inbound_spam_log.sql`), applied by
`store.Open` in file order; `PRAGMA user_version` counts the files applied and
`PRAGMA application_id` is `0x53503230` ("SP20"). There is no 1.x migration
chain. **2.0 starts from an empty data directory:** a database with a schema
version but without the 2.x `application_id` — what every 1.x file looks like —
is refused at start with `ErrForeignSchema` and left untouched, as is one whose
version is newer than the build knows. A restored 1.x backup fails the same
way. Domains are carried across with a domain export, whose format is not the
database. Later schema changes are new files (`0002_*.sql`, …); once `2.0.0` is
released the baseline is never edited. History and the per-table list:
database. A schema change is a new file; the baseline has not been edited
since `2.0.0`. History and the per-table list:
[schema-migrations.md](schema-migrations.md).
The tables that carry the panel's model:
@@ -565,6 +721,7 @@ The tables that carry the panel's model:
(send-log retention). Nothing that belongs to a user.
- `applications` (with `auth_ip_restrict` / `auth_allowed_ips`),
`application_addresses`, `rate_limits` (no `allowed_ips` column), `send_log`,
`send_log_lines` (what Postfix logged under a queue id — § Log tailer),
`logtail_state`, `inbound_domains`, `inbound_transports`,
`inbound_recipients`, `dmarc_reports`, `dmarc_report_records`.
@@ -572,7 +729,8 @@ The panel is the only writer; SQLite runs in WAL mode on one connection (see
[Panel binary](#panel-binary-cmdpanel)), which is why a full backup snapshots
with `VACUUM INTO` instead of copying the file.
**Rotation:** send-log retention `SEND_LOG_RETENTION_DAYS` (default 90);
**Rotation:** send-log retention `SEND_LOG_RETENTION_DAYS` (default 90), which
the stored log lines follow;
`mail.log` via logrotate (14 rotated files, check every 6h, rename +
`postfix reload` in `postrotate` — see § Log tailer above).
@@ -82,6 +82,8 @@
<div class="field"><label class="label" for="username">Username</label><div class="control has-icons-left"><input class="input" id="username" name="username" value="admin" autocomplete="username" spellcheck="false" required><span class="icon is-left"><i class="ti ti-user"></i></span></div></div>
<div class="field"><label class="label" for="email">E-mail</label><div class="control has-icons-left"><input class="input" id="email" name="email" type="email" value="mix@example.org" autocomplete="email"><span class="icon is-left"><i class="ti ti-mail"></i></span></div>
<p class="help">Your own address. A domain's report address can be filled with it from Domain settings.</p></div>
<div class="field"><label class="checkbox"><input type="checkbox" name="notify" checked> Send me notifications</label>
<p class="help">To this address, when something needs you: a DNS record of a domain stops matching, a rate limit refuses mail, the TLS certificate is about to expire, a message has waited in the queue for three days. <a href="help.html#notifications">What is sent and when</a></p></div>
<button type="submit" class="button is-primary">Save profile</button>
</form>
</div>
+1 -1
View File
@@ -78,7 +78,7 @@
<div class="box">
<div class="sp-box-head"><span class="sp-no">01</span><h2>Full backup</h2><a class="sp-help" href="help.html#backup" title="Backups, encryption and restore"><i class="ti ti-help-circle"></i></a></div>
<div class="sp-box-body">
<p class="mb-3"><code>data/</code> (database, DKIM keys, credentials, the queue), <code>docker-compose.yml</code>, <code>.env</code> and <code>certs/</code> in one archive. Restore into an empty project directory and start the <strong>same SelfPost version</strong>. The reverse-proxy vhost is not included.</p>
<p class="mb-3"><code>data/</code> (database, DKIM keys, credentials), <code>docker-compose.yml</code>, <code>.env</code> and <code>certs/</code> in one archive. The Postfix queue is left out: it cannot be copied reliably while the server is running. Restore into an empty project directory and start the <strong>same SelfPost version</strong>. The reverse-proxy vhost is not included.</p>
<form class="sp-form" method="post" action="#">
<div class="field"><label class="checkbox"><input type="checkbox" name="encrypt" checked> Encrypt with a password (<code>.spbk</code>)</label></div>
<div class="sp-pair">
+1 -1
View File
@@ -37,7 +37,7 @@ SIGNED_OUT = {"sp-signin", "sp-signin-brand", "sp-signin-form", "sp-foot"}
# Selectors allowed to set a width. Everything else gets its width from the
# layout it sits in — that is the whole point of the redesign.
WIDTH_OK = (".navbar-brand img", ".sp-head .sp-lead", ".sp-head .sp-grow", ".sp-postmark", ".table td.sp-clip",
".sp-h-icon", ".sp-signin", ".sp-timeline li::before", ".sp-perf")
".sp-h-icon", ".sp-signin", ".sp-timeline li::before", ".sp-perf", ".sp-sheet")
class Node:
@@ -114,6 +114,7 @@
<tr><td>error, fail, mismatch, bounced</td><td><span class="tag is-danger is-light">fail</span></td><td class="sp-mono sp-small">tag is-danger is-light</td></tr>
<tr><td>unknown, off, neutral counts</td><td><span class="tag is-light">off</span></td><td class="sp-mono sp-small">tag is-light</td></tr>
<tr><td>held: quarantine · role: global</td><td><span class="tag is-primary is-light">quarantine</span></td><td class="sp-mono sp-small">tag is-primary is-light</td></tr>
<tr><td>a tag that opens a <code>sheet</code></td><td><button type="button" class="tag is-danger is-light" popovertarget="kit-sheet">reject</button><div class="sp-sheet" id="kit-sheet" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-danger is-light">reject</span><h2>sheet</h2><button type="button" popovertarget="kit-sheet" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>What it holds</th><td>What does not fit the row, as name and value</td></tr><tr><th>How it closes</th><td class="sp-mono sp-small">Esc<br>a click outside<br>the cross</td></tr></tbody></table></div></div></div></div></td><td class="sp-mono sp-small">button.tag + sp-sheet (popover)</td></tr>
</tbody>
</table></div>
</div>
+1 -1
View File
@@ -80,7 +80,7 @@
<div><dt>Last report</dt><dd>2026-09-21 06:12 UTC</dd></div>
<div><dt>This week</dt><dd>14 kept · 0 parse failures</dd></div>
<div><dt>Hosted addresses</dt><dd class="sp-mono sp-small">dmarc-reports+&lt;domain&gt;@mail.example.org</dd></div>
<div><dt>Retention</dt><dd>500 reports or 90 days</dd></div>
<div><dt>Retention</dt><dd>500 reports a domain or 90 days</dd></div>
</dl>
</div>
</div>
+24 -5
View File
@@ -145,11 +145,30 @@
</div>
</div>
<div class="box" id="configuration">
<div class="sp-box-head"><span class="sp-no">06</span><h2>Configuration</h2><div class="sp-end">Safe to run at any time</div></div>
<div class="sp-box-body">
<p class="mb-3">Regenerates the OpenDKIM and Postfix configuration from the database and reloads both daemons. Use it after editing the files by hand, after restoring a backup, or when the running configuration looks out of step with the domain and application lists. It does not touch the queue or the TLS certificate.</p>
<form method="post" action="#"><button type="submit" class="button"><i class="ti ti-reload"></i>Reload configuration</button></form>
<div class="columns">
<div class="column is-6">
<div class="box" id="notification-mail">
<div class="sp-box-head"><span class="sp-no">06</span><h2>Notification mail</h2><div class="sp-end"><span class="tag is-warning is-light">warn</span></div><a class="sp-help" href="help.html#notifications" title="What the panel sends and what makes it arrive"><i class="ti ti-help-circle"></i></a></div>
<div class="table-container"><table class="table is-fullwidth">
<thead><tr><th>Record</th><th>State</th></tr></thead>
<tbody>
<tr><td>DKIM<br><span class="sp-muted sp-mono sp-small">sp2026._domainkey.mail.example.org</span></td><td><span class="tag is-success is-light">published</span></td></tr>
<tr><td>SPF<br><span class="sp-muted sp-mono sp-small">mail.example.org</span></td><td><span class="tag is-warning is-light">missing</span></td></tr>
</tbody>
</table></div>
<div class="sp-box-body">
<p class="sp-muted">Sent from <span class="sp-mono">postmaster@mail.example.org</span>. What to publish is under <a href="settings.html#notification-mail">Settings</a>.</p>
</div>
</div>
</div>
<div class="column is-6">
<div class="box" id="configuration">
<div class="sp-box-head"><span class="sp-no">07</span><h2>Configuration</h2><div class="sp-end">Safe to run at any time</div></div>
<div class="sp-box-body">
<p class="mb-3">Regenerates the OpenDKIM and Postfix configuration from the database and reloads both daemons. Use it after editing the files by hand, after restoring a backup, or when the running configuration looks out of step with the domain and application lists. It does not touch the queue or the TLS certificate.</p>
<form method="post" action="#"><button type="submit" class="button"><i class="ti ti-reload"></i>Reload configuration</button></form>
</div>
</div>
</div>
</div>
+10 -4
View File
@@ -97,12 +97,15 @@
<li><a href="#upstream"><i class="ti ti-arrow-forward-up"></i>Upstream</a></li>
<li><a href="#recipients"><i class="ti ti-address-book"></i>Valid recipients</a></li>
<li><a href="#filter"><i class="ti ti-shield-check"></i>Spam filter</a></li>
<li><a href="#inbound-log"><i class="ti ti-list-details"></i>Inbound log</a></li>
<li><a href="#filter-lists"><i class="ti ti-filter"></i>Filter lists</a></li>
</ul>
<p class="menu-label">Server</p>
<ul class="menu-list">
<li><a href="#backup"><i class="ti ti-archive"></i>Backup and moving a domain</a></li>
<li><a href="#users"><i class="ti ti-users"></i>Users and roles</a></li>
<li><a href="#settings"><i class="ti ti-settings"></i>Settings</a></li>
<li><a href="#notifications"><i class="ti ti-mail"></i>Notifications</a></li>
</ul>
<p class="menu-label">Account</p>
<ul class="menu-list">
@@ -122,7 +125,7 @@
<div class="sp-help-topic" id="dns"><h3>DNS records</h3><p>DKIM proves the message was signed by this relay, SPF lists the addresses allowed to send for the domain, DMARC tells receivers what to do when both fail and where to report. Publish all three exactly as shown; merge SPF into an existing record rather than adding a second one. A domain's status is the worst of its three records, and results are cached for a few minutes, so use Re-check DNS after publishing. The SPF check is shallow: it looks for this server's address in the record itself and does not follow <code>include:</code> or <code>redirect=</code>, so a record that relies on them is reported as a warning, not a failure. <code>p=none</code> does not affect delivery; tighten to <code>quarantine</code>, then <code>reject</code>, once the reports look clean. A domain's report address is set in its Domain settings.</p></div>
<div class="sp-help-topic" id="apps"><h3>Applications</h3><p>An application is one SASL login, unique across all domains. It may send as any address of its domain or only as listed addresses, and, with a list of client IPs set, only from those addresses. The password is shown once at creation or regeneration and is not stored in readable form.</p></div>
<div class="sp-help-topic" id="limits"><h3>Rate limits</h3><p>Level 1 is a per-IP ceiling set in <code>.env</code>. Level 2 is optional, per domain and per application, entered by hand or computed from the last 30 days of sending, and can never exceed level 1. An application's own limit replaces its domain's.</p></div>
<div class="sp-help-topic" id="log"><h3>Outbound log</h3><p>One row per message and recipient, with the status Postfix last reported: queued (accepted, nothing reported yet), sent (the receiving server took it; what the mailbox does next is not reported back), deferred (not taken yet, Postfix tries again), bounced (failed for good, or the queue lifetime ran out) and rejected (refused before queueing under a level-2 rate limit, so it never reached Postfix). A message's page lists these steps and the lines Postfix wrote about it in <code>mail.log</code>, found by queue id; that file is rotated daily and about two weeks are kept, so an old message may show none. Rows are deleted after the retention set in Server › Settings, 90 days by default. A domain administrator sees only the rows of their own domains.</p></div>
<div class="sp-help-topic" id="log"><h3>Outbound log</h3><p>One row per message and recipient, with the status Postfix last reported: queued (accepted, nothing reported yet), sent (the receiving server took it; what the mailbox does next is not reported back), deferred (not taken yet, Postfix tries again), bounced (failed for good, or the queue lifetime ran out) and rejected (refused before queueing under a level-2 rate limit, so it never reached Postfix). A message's page lists these steps and the lines Postfix wrote about it in <code>mail.log</code>, found by queue id. The lines are kept with the row: they stay on the page after that file has rotated, for as long as the row does. A message accepted before version 2.2.0 shows them only while they are still in the current file. Rows are deleted after the retention set in Server › Settings, 90 days by default, and their lines with them. A domain administrator sees only the rows of their own domains.</p></div>
<div class="sp-help-topic" id="queue"><h3>Queue and retries</h3><p>The page lists, as <code>postqueue -p</code> prints it, the messages Postfix has accepted and not yet delivered: mail being sent right now and mail deferred for a later attempt. Only the global role can open it, and it only shows; nothing here deletes or forces a message. A deferred message is tried again after the first retry delay, then at gaps that double up to a cap, until the receiving server takes it or the queue lifetime runs out and it bounces; there is no attempt counter, and the box on the page shows this server's values. A queue that keeps growing means a receiver is refusing or not answering: open the message from the Outbound log to read the reason Postfix logged.</p></div>
<div class="sp-help-topic" id="dmarc"><h3>DMARC reports</h3><p>An aggregate report is the summary a receiving provider sends, usually daily, to the <code>rua=</code> address of your DMARC record: which servers sent mail as your domain and whether SPF or DKIM passed for it. A domain's report address is one of three choices, set in its Domain settings: no reports (the default for a new domain), the SelfPost hosted address (an address on this server whose reports are parsed and shown under DMARC reports; offered only while hosted reports are enabled on the server), or an address typed for the domain (an ordinary mailbox that receives the raw XML; a button fills in your own profile e-mail). When the typed address is on another domain than the one sending, that domain must publish a report-authorization record, or providers will not send; the domain's page shows it with the other DNS records and checks it. In the tables pass means SPF or DKIM passed and aligned, fail means neither did, and your own server shows as <em>this relay</em>. A fail from any other source is someone else sending as your domain — a forwarder, an application you forgot, or abuse — so do not tighten <code>p=</code> until that sender is fixed or removed. A domain administrator reads the reports of their own domains.</p></div>
</div>
@@ -132,16 +135,19 @@
<div class="sp-help-topic" id="upstream"><h3>Upstream</h3><p>The upstream is the server that actually holds the mailboxes. Give its host and port — used exactly as written, with no MX lookup — and choose the TLS for the hop to it: Opportunistic uses TLS when the upstream offers it and otherwise sends in the clear, Required holds the mail until a TLS session is possible, Off never uses TLS. While the upstream does not answer, Postfix keeps the mail and retries as for any other mail.</p></div>
<div class="sp-help-topic" id="recipients"><h3>Valid recipients</h3><p>Choose whom this domain accepts. Listed addresses only: the relay accepts exactly the addresses you list, one per line or comma-separated, and refuses any other while the sender is still connected, so it never has to send a bounce for them. Any recipient: everything at the domain is accepted and passed on, and an address the upstream then rejects comes back as a bounce from this server to a sender who is often forged — backscatter. Prefer a list, and keep it in step with the mailboxes upstream.</p></div>
<div class="sp-help-topic" id="filter"><h3>Spam filter</h3><p>The spam filter belongs to the server, not to a domain. The server's administrator attaches it in the server's <code>.env</code>, and from then on every inbound domain is screened by it before mail is handed on; the box shows on or off and has nothing to switch. Off means mail goes to the upstream unfiltered. While the filter is down, mail is either accepted unfiltered (the default) or deferred, so the sending server tries again later, as the server's setting says; the box states which.</p></div>
<div class="sp-help-topic" id="inbound-log"><h3>Inbound log</h3><p>One row per recipient of a message offered on port 25 for an inbound domain, with what was decided: <span class="tag is-success is-light">accept</span> — handed on to the upstream; <span class="tag is-danger is-light">reject</span> — refused for good; <span class="tag is-warning is-light">tempfail</span> — the sender was asked to come back later; <span class="tag is-primary is-light">quarantine</span> — held by the filter. <em>Why</em> names who decided. <code>rspamd</code> is the spam filter, with its action, the score against the reject threshold and the heaviest symbols; <code>postfix</code> is the relay's own rule — an unknown recipient, a sender domain that does not exist — with its reply; <code>selfpost</code> means nobody judged the message: it was relayed with no filter attached. Accept says what this server did, not what the upstream did with the message afterwards.<br><br>Accepted mail is recorded as it passes; refused mail is read from <code>mail.log</code>, so its row has no subject, and a message refused after its text was sent is one row however many recipients it had. A refusal to relay for a domain that is not an inbound one is every scanner on the internet and is not recorded. Rows are kept for the number of days set in Settings, and never more than 10 000.</p></div>
<div class="sp-help-topic" id="filter-lists"><h3>Filter lists</h3><p>Two lists for the whole server that correct the spam filter by hand. An entry is a sender address, a sender domain — that domain, not its subdomains — or a client address or network in CIDR form. A sender on the Deny list is refused outright. A sender on the Allow list is given a weight no spam score outweighs, so the filter lets it through. The sender's address is whatever the sending server says it is - anyone can claim to be a sender you allow - so allow a client address or network where you can, and a sender address only for mail you would rather receive forged than lose. When entries of both lists match one message, <strong>Deny wins</strong>.<br><br>SelfPost does not apply the lists itself: it writes them as files under <code>/data/antispam/</code>, and rspamd rereads a file within seconds of a change. With the spam filter off the lists are kept and can be edited, but nothing reads them. The sender matched is the envelope sender — the one in the log — not the From line a mail program shows. Matched counts the messages in the Inbound log that the entry decided, so it reaches back as far as the log does; never means not once in that time, and such an entry is one to consider removing.</p></div>
</div>
<div class="box">
<div class="sp-box-head"><span class="sp-no">04</span><h2>Server</h2></div>
<div class="sp-help-topic" id="backup"><h3>Backup and moving a domain</h3><p>Full backup downloads one archive of the whole instance: <code>data/</code> (the database, DKIM keys, application credentials and the queue), <code>docker-compose.yml</code>, <code>.env</code> and <code>certs/</code>. It holds secrets, so leave Encrypt with a password ticked: the file is sealed as <code>.spbk</code>, the password is not stored, and without it nothing can be recovered. To restore, unpack it into an empty project directory (decrypt a <code>.spbk</code> first with <code>selfpost-backup -decrypt</code>) and start the same SelfPost version that made it; the panel refuses data from another version. The reverse-proxy configuration and <code>mail.log</code> are not included. One domain moves on its own: Export domain on its settings page writes a <code>.json</code> (or encrypted <code>.spde</code>) with its DKIM key and its applications' passwords, and Import a domain here reads it into another instance, so the published DNS needs no change.</p></div>
<div class="sp-help-topic" id="backup"><h3>Backup and moving a domain</h3><p>Full backup downloads one archive of the whole instance: <code>data/</code> (the database, DKIM keys and application credentials), <code>docker-compose.yml</code>, <code>.env</code> and <code>certs/</code>. The Postfix queue is not in it. The backup is taken while the server is running, and Postfix keeps moving messages through the queue, so a copy could hold a message twice or miss it; mail still waiting for delivery is therefore not carried over, and a restored server starts with an empty queue. To keep the queue, stop the container and archive the project directory instead. The private key in <code>certs/</code> has to be readable by the panel: give it to group 998 on the host (<code>chgrp 998</code>, <code>chmod 640</code>). Until then the panel refuses to make a backup and names the file, rather than hand over an archive without the key. It holds secrets, so leave Encrypt with a password ticked: the file is sealed as <code>.spbk</code>, the password is not stored, and without it nothing can be recovered. To restore, unpack it into an empty project directory (decrypt a <code>.spbk</code> first with <code>selfpost-backup -decrypt</code>) and start the same SelfPost version that made it; the panel refuses data from another version. The reverse-proxy configuration and <code>mail.log</code> are not included. One domain moves on its own: Export domain on its settings page writes a <code>.json</code> (or encrypted <code>.spde</code>) with its DKIM key and its applications' passwords, and Import a domain here reads it into another instance, so the published DNS needs no change.</p></div>
<div class="sp-help-topic" id="users"><h3>Users and roles</h3><p>The two roles are a reach, not a rank: every user is an administrator and signs in with a username and password. Global reaches the whole server — Overview, Server, the queue and every domain — and is the only role that adds or deletes domains and users. Domain reaches only the domains assigned to it, outbound and inbound separately (the same name on both lists is two assignments); <em>All</em> includes domains added later, and a domain user needs at least one domain or <em>All</em> on either list. The only global user cannot be demoted or deleted, and you cannot delete the account you are signed in with. A new password set here signs that user out of their other sessions.</p></div>
<div class="sp-help-topic" id="settings"><h3>Settings</h3><p>Settings holds what is true of the whole instance. Log retention is how many days the Outbound log keeps its rows, from 7 to 365 and 90 by default; a sweep every six hours deletes older ones, and it is the main driver of how fast <code>/data</code> grows. It does not change the rotation of <code>mail.log</code>, which is daily with about two weeks kept. The sending rate limit shown under it is level 1, the per-client-IP ceiling set in <code>.env</code>: changing it needs a restart, and no level-2 limit can exceed it.</p></div>
<div class="sp-help-topic" id="settings"><h3>Settings</h3><p>Settings holds what is true of the whole instance. Log retention is how many days the Outbound log keeps its rows, from 7 to 365 and 90 by default; a sweep every six hours deletes older ones, and it is the main driver of how fast <code>/data</code> grows. With the inbound relay on there is a second number, for the Inbound log, with the same range. It does not change the rotation of <code>mail.log</code>, which is daily with about two weeks kept. The sending rate limit shown under it is level 1, the per-client-IP ceiling set in <code>.env</code>: changing it needs a restart, and no level-2 limit can exceed it. Notifications are turned on or off here for everyone.</p></div>
<div class="sp-help-topic" id="notifications"><h3>Notifications</h3><p>The panel writes to its users when something needs them, from <code>postmaster@</code> the server's hostname. Four things are reported. A sending domain's SPF, DKIM or DMARC record stops matching: sent once two hourly checks in a row have found it so, and once more when the record matches again. A level-2 rate limit refuses mail: sent at the first refusal, then at most once a day for the same limit. The TLS certificate is 14 days from expiry, and again at 3 days if it is still the same certificate. A message has waited in the queue for more than three days, with one more message when none is left. A global user is told all of it; a domain administrator is told about the DNS records and limits of their own domains.<br><br>Each user turns notifications on or off under Account and needs an e-mail there; Server › Settings turns them off for everyone. They are the panel's own mail: not rows of the Outbound log, and not counted against any limit.<br><br>For that mail to arrive, the server's own name needs two DNS records of its own, as each of your domains does: a DKIM key, which the panel generates and signs notifications with, and an SPF record, <code>v=spf1 a -all</code>. Server › Settings gives both with the values to publish, and Server › Health shows whether they are in place. Without them notifications are still sent, and a receiver may file them as spam. Mail of your domains does not depend on either.</p></div>
</div>
<div class="box">
<div class="sp-box-head"><span class="sp-no">05</span><h2>Account</h2></div>
<div class="sp-help-topic" id="account"><h3>Your account</h3><p>Account is your own, whatever your role: your username and e-mail, and your password. Changing the password signs out every other session of this account and keeps the one you are using; applications have their own logins and are not affected.</p></div>
<div class="sp-help-topic" id="account"><h3>Your account</h3><p>Account is your own, whatever your role: your username and e-mail, whether the panel sends you notifications, and your password. Changing the password signs out every other session of this account and keeps the one you are using; applications have their own logins and are not affected.</p></div>
</div>
</div>
</div>
@@ -74,7 +74,7 @@
</div>
<div class="box">
<div class="sp-box-head"><span class="sp-no">01</span><h2>Add an entry</h2><a class="sp-help" href="help.html#inbound" title="How the lists are applied"><i class="ti ti-help-circle"></i></a></div>
<div class="sp-box-head"><span class="sp-no">01</span><h2>Add an entry</h2><a class="sp-help" href="help.html#filter-lists" title="How the lists are applied"><i class="ti ti-help-circle"></i></a></div>
<div class="sp-box-body">
<form class="sp-form" method="post" action="#">
<div class="columns">
@@ -84,7 +84,7 @@
</div>
<div class="column is-3">
<div class="field"><label class="label" for="etype">Entry is</label>
<div class="control"><div class="select is-fullwidth"><select id="etype" name="entry_type"><option>a sender address</option><option>a sender domain</option><option>a client IP or CIDR</option></select></div></div></div>
<div class="control"><div class="select is-fullwidth"><select id="etype" name="entry_type"><option>a sender address</option><option>a sender domain</option><option>a client address or network</option></select></div></div></div>
</div>
<div class="column is-6">
<div class="field"><label class="label" for="value">Value</label>
@@ -103,11 +103,11 @@
<div class="box">
<div class="sp-box-head"><span class="sp-no">02</span><h2>Deny</h2><div class="sp-end">3 entries · wins over Allow</div></div>
<div class="table-container"><table class="table is-fullwidth is-hoverable">
<thead><tr><th>Value</th><th>Note</th><th>Added</th><th></th></tr></thead>
<thead><tr><th>Value</th><th>Kind</th><th>Matched</th><th></th></tr></thead>
<tbody>
<tr><td class="sp-mono sp-small">@bulk.example <span class="tag is-light">domain</span></td><td class="sp-muted">newsletter nobody subscribed to</td><td class="sp-muted sp-nowrap sp-small">2026-09-02</td><td class="sp-actions"><a class="has-text-danger" href="#">Remove</a></td></tr>
<tr><td class="sp-mono sp-small">198.51.100.0/24 <span class="tag is-light">ip</span></td><td class="sp-muted">dictionary attack on lists.example.org</td><td class="sp-muted sp-nowrap sp-small">2026-08-19</td><td class="sp-actions"><a class="has-text-danger" href="#">Remove</a></td></tr>
<tr><td class="sp-mono sp-small">win@lottery.example <span class="tag is-light">address</span></td><td class="sp-muted">—</td><td class="sp-muted sp-nowrap sp-small">2026-07-30</td><td class="sp-actions"><a class="has-text-danger" href="#">Remove</a></td></tr>
<tr><td class="sp-mono sp-small">bulk.example</td><td><button type="button" class="tag is-light" popovertarget="deny-1">domain</button><div class="sp-sheet" id="deny-1" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-light">domain</span><h2>bulk.example</h2><button type="button" popovertarget="deny-1" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Note</th><td>newsletter nobody subscribed to</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-09-02</td></tr><tr><th>Matched</th><td>41 in the last 90 days</td></tr><tr><th>Last match</th><td class="sp-mono sp-small">2026-09-21 13:57</td></tr></tbody></table></div></div></div></div></td><td class="sp-nowrap">41</td><td class="sp-actions"><form method="post" action="#"><button type="submit" class="has-text-danger">Remove</button></form></td></tr>
<tr><td class="sp-mono sp-small">198.51.100.0/24</td><td><button type="button" class="tag is-light" popovertarget="deny-2">ip</button><div class="sp-sheet" id="deny-2" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-light">ip</span><h2>198.51.100.0/24</h2><button type="button" popovertarget="deny-2" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Note</th><td>dictionary attack on lists.example.org</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-08-19</td></tr><tr><th>Matched</th><td>1203 in the last 90 days</td></tr><tr><th>Last match</th><td class="sp-mono sp-small">2026-09-21 09:12</td></tr></tbody></table></div></div></div></div></td><td class="sp-nowrap">1203</td><td class="sp-actions"><form method="post" action="#"><button type="submit" class="has-text-danger">Remove</button></form></td></tr>
<tr><td class="sp-mono sp-small">win@lottery.example</td><td><button type="button" class="tag is-light" popovertarget="deny-3">address</button><div class="sp-sheet" id="deny-3" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-light">address</span><h2>win@lottery.example</h2><button type="button" popovertarget="deny-3" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Note</th><td>—</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-07-30</td></tr><tr><th>Matched</th><td>never in the last 90 days</td></tr></tbody></table></div></div></div></div></td><td class="sp-muted sp-nowrap">never</td><td class="sp-actions"><form method="post" action="#"><button type="submit" class="has-text-danger">Remove</button></form></td></tr>
</tbody>
</table></div>
</div>
@@ -116,10 +116,10 @@
<div class="box">
<div class="sp-box-head"><span class="sp-no">03</span><h2>Allow</h2><div class="sp-end">2 entries</div></div>
<div class="table-container"><table class="table is-fullwidth is-hoverable">
<thead><tr><th>Value</th><th>Note</th><th>Added</th><th></th></tr></thead>
<thead><tr><th>Value</th><th>Kind</th><th>Matched</th><th></th></tr></thead>
<tbody>
<tr><td class="sp-mono sp-small">@partner.example <span class="tag is-light">domain</span></td><td class="sp-muted">invoices were scored as spam</td><td class="sp-muted sp-nowrap sp-small">2026-09-11</td><td class="sp-actions"><a class="has-text-danger" href="#">Remove</a></td></tr>
<tr><td class="sp-mono sp-small">noreply@github.example <span class="tag is-light">address</span></td><td class="sp-muted">security advisories</td><td class="sp-muted sp-nowrap sp-small">2026-06-04</td><td class="sp-actions"><a class="has-text-danger" href="#">Remove</a></td></tr>
<tr><td class="sp-mono sp-small">partner.example</td><td><button type="button" class="tag is-light" popovertarget="allow-1">domain</button><div class="sp-sheet" id="allow-1" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-light">domain</span><h2>partner.example</h2><button type="button" popovertarget="allow-1" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Note</th><td>invoices were scored as spam</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-09-11</td></tr><tr><th>Matched</th><td>7 in the last 90 days</td></tr><tr><th>Last match</th><td class="sp-mono sp-small">2026-09-19 16:40</td></tr></tbody></table></div></div></div></div></td><td class="sp-nowrap">7</td><td class="sp-actions"><form method="post" action="#"><button type="submit" class="has-text-danger">Remove</button></form></td></tr>
<tr><td class="sp-mono sp-small">noreply@github.example</td><td><button type="button" class="tag is-light" popovertarget="allow-2">address</button><div class="sp-sheet" id="allow-2" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-light">address</span><h2>noreply@github.example</h2><button type="button" popovertarget="allow-2" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Note</th><td>security advisories</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-06-04</td></tr><tr><th>Matched</th><td>18 in the last 90 days</td></tr><tr><th>Last match</th><td class="sp-mono sp-small">2026-09-21 13:30</td></tr></tbody></table></div></div></div></div></td><td class="sp-nowrap">18</td><td class="sp-actions"><form method="post" action="#"><button type="submit" class="has-text-danger">Remove</button></form></td></tr>
</tbody>
</table></div>
</div>
@@ -130,10 +130,10 @@
<div class="sp-box-head"><span class="sp-no">04</span><h2>Where the lists go</h2><div class="sp-end"><span class="tag is-success is-light">applied</span></div></div>
<div class="sp-box-body">
<dl class="sp-facts">
<div><dt>Applied by</dt><dd>rspamd, from map files</dd></div>
<div><dt>Files</dt><dd class="sp-mono sp-small">/data/antispam/allow.map<br>/data/antispam/deny.map</dd></div>
<div><dt>Applied by</dt><dd>rspamd, from map files it rereads within seconds of a change</dd></div>
<div><dt>Files</dt><dd><span class="sp-mono">/data/antispam/</span> — six, one for each list and kind of entry</dd></div>
<div><dt>Last written</dt><dd>2026-09-11 08:40 UTC</dd></div>
<div><dt>When the filter is off</dt><dd class="sp-muted">The lists are kept and editable, but nothing reads them until <code>INBOUND_ANTISPAM_MILTER</code> is set.</dd></div>
<div><dt>When both lists match</dt><dd>Deny wins, whichever entries they are.</dd></div>
</dl>
</div>
</div>
+10 -9
View File
@@ -69,7 +69,7 @@
<div class="sp-grow">
<p class="sp-kicker">Inbound</p>
<h1 class="title is-3">Inbound log</h1>
<p class="sp-lead">What port 25 decided about each message, one row per recipient. Accepts come from the inbound journal, rejects from <code>mail.log</code>; the filter's reasons appear when the spam filter is on.</p>
<p class="sp-lead">What port 25 decided about each message, one row per recipient. Accepted mail is recorded as it passes, refused mail is read from <code>mail.log</code>; the filter's verdict appears when the spam filter is on. Rows older than 90 days are deleted — change that in <a href="settings.html">Settings</a>.</p>
</div>
</div>
@@ -83,19 +83,20 @@
<input class="input" type="date" name="to" value="2026-09-21" aria-label="To date">
<button type="submit" class="button"><i class="ti ti-filter"></i>Filter</button>
</form>
<a class="sp-help" href="help.html#inbound-log" title="What the decisions mean"><i class="ti ti-help-circle"></i></a>
</div>
<div class="table-container"><table class="table is-fullwidth is-hoverable">
<thead><tr><th>Time, UTC</th><th>From · client</th><th>To</th><th>Subject</th><th>Decision</th><th>Why</th><th></th></tr></thead>
<thead><tr><th>Time, UTC</th><th>From</th><th>To</th><th>Subject</th><th>Decision</th><th></th></tr></thead>
<tbody>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 14:00:18</td><td>t.berg@<wbr>uni.example<br><span class="sp-mono sp-muted sp-small">192.0.2.77</span></td><td>dev@<wbr>lists.example.org</td><td class="sp-clip">Re: [dev] patch series v3</td><td><span class="tag is-success is-light">accept</span></td><td class="sp-muted sp-small">rspamd · score 0.4</td><td class="sp-actions"><a href="in-filter-lists.html">Deny sender</a></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:59:02</td><td>invoice@<wbr>partner-billing.example<br><span class="sp-mono sp-muted sp-small">203.0.113.140</span></td><td>owner@<wbr>lists.example.org</td><td class="sp-clip">Outstanding invoice — action required</td><td><span class="tag is-primary is-light">quarantine</span></td><td class="sp-muted sp-small">rspamd · score 9.8 · PHISHING, FORGED_SENDER</td><td class="sp-actions"><a href="in-quarantine-message.html">Review</a></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:57:41</td><td>promo@<wbr>bulk.example<br><span class="sp-mono sp-muted sp-small">198.51.100.90</span></td><td>sales@<wbr>lists.example.org</td><td class="sp-clip sp-muted">—</td><td><span class="tag is-danger is-light">reject</span></td><td class="sp-muted sp-small">postfix · 550 5.1.1 unknown recipient</td><td class="sp-actions"><a href="in-filter-lists.html">Deny sender</a></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:51:30</td><td>billing@<wbr>partner.example<br><span class="sp-mono sp-muted sp-small">198.51.100.12</span></td><td>owner@<wbr>lists.example.org</td><td class="sp-clip">Invoice 2026-0917</td><td><span class="tag is-danger is-light">reject</span></td><td class="sp-muted sp-small">rspamd · score 16.2 · BAYES_SPAM</td><td class="sp-actions"><a href="in-filter-lists.html">Allow sender</a></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:44:02</td><td>hr@<wbr>partner.example<br><span class="sp-mono sp-muted sp-small">192.0.2.31</span></td><td>jobs@<wbr>acme.io</td><td class="sp-clip">Application: backend engineer</td><td><span class="tag is-warning is-light">tempfail</span></td><td class="sp-muted sp-small">selfpost · upstream 10.0.4.12 timed out</td><td class="sp-actions"></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:30:55</td><td>noreply@<wbr>github.example<br><span class="sp-mono sp-muted sp-small">192.0.2.200</span></td><td>security@<wbr>lists.example.org</td><td class="sp-clip">[advisory] new report</td><td><span class="tag is-success is-light">accept</span></td><td class="sp-muted sp-small">rspamd · allow list</td><td class="sp-actions"></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 14:00:18</td><td>t.berg@<wbr>uni.example<br><span class="sp-mono sp-muted sp-small">192.0.2.77</span></td><td>dev@<wbr>lists.example.org</td><td class="sp-clip">Re: [dev] patch series v3</td><td><button type="button" class="tag is-success is-light" popovertarget="why-1">accept</button><div class="sp-sheet" id="why-1" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-success is-light">accept</span><h2>Why</h2><button type="button" popovertarget="why-1" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>rspamd</td></tr><tr><th>Verdict</th><td>no action, score 0.40 / 15.00</td></tr><tr><th>Subject</th><td>Re: [dev] patch series v3</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"><a href="in-filter-lists.html">Deny sender</a></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:59:02</td><td>invoice@<wbr>partner-billing.example<br><span class="sp-mono sp-muted sp-small">203.0.113.140</span></td><td>owner@<wbr>lists.example.org</td><td class="sp-clip">Outstanding invoice — action required</td><td><button type="button" class="tag is-success is-light" popovertarget="why-2">accept</button><div class="sp-sheet" id="why-2" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-success is-light">accept</span><h2>Why</h2><button type="button" popovertarget="why-2" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>rspamd</td></tr><tr><th>Verdict</th><td>add header, score 9.80 / 15.00</td></tr><tr><th>Symbols</th><td class="sp-mono sp-small">PHISHING(5.00)<br>FORGED_SENDER(3.00)<br>R_SPF_FAIL(1.00)</td></tr><tr><th>Subject</th><td>Outstanding invoice — action required</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"><a href="in-filter-lists.html">Deny sender</a></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:57:41</td><td>promo@<wbr>bulk.example<br><span class="sp-mono sp-muted sp-small">198.51.100.90</span></td><td>sales@<wbr>lists.example.org</td><td class="sp-clip sp-muted">—</td><td><button type="button" class="tag is-danger is-light" popovertarget="why-3">reject</button><div class="sp-sheet" id="why-3" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-danger is-light">reject</span><h2>Why</h2><button type="button" popovertarget="why-3" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>postfix</td></tr><tr><th>Reply</th><td class="sp-mono sp-small">550 5.1.1 &lt;sales@lists.example.org&gt;: Recipient address rejected: User unknown in relay recipient table</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:51:30</td><td>billing@<wbr>partner.example<br><span class="sp-mono sp-muted sp-small">198.51.100.12</span></td><td>owner@<wbr>lists.example.org</td><td class="sp-clip sp-muted">—</td><td><button type="button" class="tag is-danger is-light" popovertarget="why-4">reject</button><div class="sp-sheet" id="why-4" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-danger is-light">reject</span><h2>Why</h2><button type="button" popovertarget="why-4" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>rspamd</td></tr><tr><th>Reply</th><td class="sp-mono sp-small">5.7.1 Spam message rejected</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"><a href="in-filter-lists.html">Allow sender</a></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:44:02</td><td>hr@<wbr>partner.example<br><span class="sp-mono sp-muted sp-small">192.0.2.31</span></td><td>jobs@<wbr>acme.io</td><td class="sp-clip sp-muted">—</td><td><button type="button" class="tag is-warning is-light" popovertarget="why-5">tempfail</button><div class="sp-sheet" id="why-5" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-warning is-light">tempfail</span><h2>Why</h2><button type="button" popovertarget="why-5" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>rspamd</td></tr><tr><th>Reply</th><td class="sp-mono sp-small">4.7.1 Try again later</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"><a href="in-filter-lists.html">Allow sender</a></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:30:55</td><td>noreply@<wbr>github.example<br><span class="sp-mono sp-muted sp-small">192.0.2.200</span></td><td>security@<wbr>lists.example.org</td><td class="sp-clip">[advisory] new report</td><td><button type="button" class="tag is-success is-light" popovertarget="why-6">accept</button><div class="sp-sheet" id="why-6" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-success is-light">accept</span><h2>Why</h2><button type="button" popovertarget="why-6" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>rspamd</td></tr><tr><th>Verdict</th><td>no action, score -98.10 / 15.00</td></tr><tr><th>Symbols</th><td class="sp-mono sp-small">SELFPOST_ALLOW_FROM(-100.00)<br>R_DKIM_ALLOW(-0.20)</td></tr><tr><th>Subject</th><td>[advisory] new report</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"><a href="in-filter-lists.html">Deny sender</a></td></tr>
</tbody>
</table></div>
<div class="sp-box-foot"><span>Page 1 of 38</span><a href="#">Older →</a><span class="sp-end">Allow / Deny sender opens <a href="in-filter-lists.html">Filter lists</a> with the entry filled in · global administrator only</span></div>
<div class="sp-box-foot"><span>Page 1 of 38</span><a href="#">Older →</a><span class="sp-end">A decision opens its reason · Allow / Deny sender opens <a href="in-filter-lists.html">Filter lists</a> with the entry filled in</span></div>
</div>
<footer class="sp-foot">SelfPost 1.8.0 · © 2026 Mix · <a href="#">License (AGPL-3.0)</a> · <a href="#">Source</a> · No warranty</footer>
+20 -5
View File
@@ -108,7 +108,20 @@
}
]
},
"box[Configuration]:body"
{
"columns": [
{
"is-6": [
"box[Notification mail]:table+body"
]
},
{
"is-6": [
"box[Configuration]:body"
]
}
]
}
],
"help": [
"head",
@@ -123,8 +136,8 @@
"is-9": [
"box[Overview]:help×3",
"box[Outbound]:help×6",
"box[Inbound]:help×4",
"box[Server]:help×3",
"box[Inbound]:help×6",
"box[Server]:help×4",
"box[Account]:help"
]
}
@@ -406,12 +419,14 @@
"columns": [
{
"is-6": [
"box[Log retention]:body"
"box[Log retention]:body",
"box[Sending rate limits]:body"
]
},
{
"is-6": [
"box[Sending rate limits]:body"
"box[Notifications]:body",
"box[Notification mail]:record×2"
]
}
]
+40 -2
View File
@@ -82,11 +82,12 @@
<div class="field"><label class="label" for="ret">Keep outbound log rows, days</label>
<div class="control"><input class="input" id="ret" name="send_log_retention_days" type="number" min="7" max="365" value="30" required></div>
<p class="help">The main driver of <code>/data</code> growth. Does not affect <code>mail.log</code> rotation (fourteen daily files).</p></div>
<div class="field"><label class="label" for="inret">Keep inbound log rows, days</label>
<div class="control"><input class="input" id="inret" name="inbound_spam_log_retention_days" type="number" min="7" max="365" value="90" required></div>
<p class="help">The inbound log also never holds more than 10 000 rows: refused mail is logged at the rate strangers knock on port 25.</p></div>
</div>
</div>
</div>
<div class="column is-6">
<div class="box" id="rate-limits">
<div class="sp-box-head"><span class="sp-no">02</span><h2>Sending rate limits</h2><div class="sp-end">Read-only</div><a class="sp-help" href="help.html#limits" title="Levels 1 and 2"><i class="ti ti-help-circle"></i></a></div>
<div class="sp-box-body">
@@ -97,6 +98,43 @@
</div>
</div>
</div>
<div class="column is-6">
<div class="box" id="notifications">
<div class="sp-box-head"><span class="sp-no">03</span><h2>Notifications</h2><a class="sp-help" href="help.html#notifications" title="What is sent and when"><i class="ti ti-help-circle"></i></a></div>
<div class="sp-box-body">
<div class="field"><label class="checkbox"><input type="checkbox" name="notify" checked> Send notification e-mails</label>
<p class="help">From <span class="sp-mono">postmaster@mail.example.org</span>, to every user who has an e-mail and has not turned them off under <a href="account.html">Account</a>.</p></div>
</div>
</div>
<div class="box" id="notification-mail">
<div class="sp-box-head"><span class="sp-no">04</span><h2>Notification mail</h2><div class="sp-end">DNS records of the server's own name</div><a class="sp-help" href="help.html#notifications" title="What the panel sends and what makes it arrive"><i class="ti ti-help-circle"></i></a></div>
<div class="sp-record">
<div class="sp-record-head"><h3>DKIM</h3><span class="tag is-success is-light">published</span><small>Not a secret · selector <code>sp2026</code></small></div>
<div class="sp-row">
<div class="field"><label class="label">Host / name</label>
<div class="field has-addons"><div class="control is-expanded"><input class="input" readonly value="sp2026._domainkey.mail.example.org"></div><div class="control"><button type="button" class="button"><i class="ti ti-copy"></i>Copy</button></div></div></div>
<div class="field"><label class="label">Type</label><div class="control"><input class="input" readonly value="TXT"></div></div>
</div>
<div class="field"><label class="label">Value to publish</label>
<div class="field has-addons"><div class="control is-expanded"><textarea class="textarea" rows="5" readonly>v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAq7n2Lw0c5vXeR1dYt8uJb3mKs6pHf9aZoWg4iNy2TqCx0lVjE7rUh5sBk1dMa8fPz3oGc6yXw9tIe4nQv2uL…IDAQAB</textarea></div><div class="control"><button type="button" class="button"><i class="ti ti-copy"></i>Copy</button></div></div></div>
</div>
<div class="sp-record">
<div class="sp-record-head"><h3>SPF</h3><span class="tag is-warning is-light">missing</span><small>For the server's own name, not for your domains</small></div>
<div class="sp-row">
<div class="field"><label class="label">Host / name</label>
<div class="field has-addons"><div class="control is-expanded"><input class="input" readonly value="mail.example.org"></div><div class="control"><button type="button" class="button"><i class="ti ti-copy"></i>Copy</button></div></div></div>
<div class="field"><label class="label">Type</label><div class="control"><input class="input" readonly value="TXT"></div></div>
</div>
<div class="field"><label class="label">Value to publish</label>
<div class="field has-addons"><div class="control is-expanded"><input class="input" readonly value="v=spf1 a -all"></div><div class="control"><button type="button" class="button"><i class="ti ti-copy"></i>Copy</button></div></div></div>
<p class="help">Notifications are sent without it, and a receiver may file them as spam.</p>
</div>
</div>
</div>
</div>
<div class="buttons"><button type="submit" class="button is-primary">Save settings</button></div>
</form>
@@ -16,6 +16,8 @@
<div class="field"><label class="label" for="username">Username</label><div class="control has-icons-left"><input class="input" id="username" name="username" value="admin" autocomplete="username" spellcheck="false" required><span class="icon is-left"><i class="ti ti-user"></i></span></div></div>
<div class="field"><label class="label" for="email">E-mail</label><div class="control has-icons-left"><input class="input" id="email" name="email" type="email" value="mix@example.org" autocomplete="email"><span class="icon is-left"><i class="ti ti-mail"></i></span></div>
<p class="help">Your own address. A domain's report address can be filled with it from Domain settings.</p></div>
<div class="field"><label class="checkbox"><input type="checkbox" name="notify" checked> Send me notifications</label>
<p class="help">To this address, when something needs you: a DNS record of a domain stops matching, a rate limit refuses mail, the TLS certificate is about to expire, a message has waited in the queue for three days. <a href="help.html#notifications">What is sent and when</a></p></div>
<button type="submit" class="button is-primary">Save profile</button>
</form>
</div>
@@ -12,7 +12,7 @@
<div class="box">
<div class="sp-box-head"><span class="sp-no">01</span><h2>Full backup</h2><a class="sp-help" href="help.html#backup" title="Backups, encryption and restore"><i class="ti ti-help-circle"></i></a></div>
<div class="sp-box-body">
<p class="mb-3"><code>data/</code> (database, DKIM keys, credentials, the queue), <code>docker-compose.yml</code>, <code>.env</code> and <code>certs/</code> in one archive. Restore into an empty project directory and start the <strong>same SelfPost version</strong>. The reverse-proxy vhost is not included.</p>
<p class="mb-3"><code>data/</code> (database, DKIM keys, credentials), <code>docker-compose.yml</code>, <code>.env</code> and <code>certs/</code> in one archive. The Postfix queue is left out: it cannot be copied reliably while the server is running. Restore into an empty project directory and start the <strong>same SelfPost version</strong>. The reverse-proxy vhost is not included.</p>
<form class="sp-form" method="post" action="#">
<div class="field"><label class="checkbox"><input type="checkbox" name="encrypt" checked> Encrypt with a password (<code>.spbk</code>)</label></div>
<div class="sp-pair">
@@ -48,6 +48,7 @@
<tr><td>error, fail, mismatch, bounced</td><td><span class="tag is-danger is-light">fail</span></td><td class="sp-mono sp-small">tag is-danger is-light</td></tr>
<tr><td>unknown, off, neutral counts</td><td><span class="tag is-light">off</span></td><td class="sp-mono sp-small">tag is-light</td></tr>
<tr><td>held: quarantine · role: global</td><td><span class="tag is-primary is-light">quarantine</span></td><td class="sp-mono sp-small">tag is-primary is-light</td></tr>
<tr><td>a tag that opens a <code>sheet</code></td><td><button type="button" class="tag is-danger is-light" popovertarget="kit-sheet">reject</button><div class="sp-sheet" id="kit-sheet" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-danger is-light">reject</span><h2>sheet</h2><button type="button" popovertarget="kit-sheet" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>What it holds</th><td>What does not fit the row, as name and value</td></tr><tr><th>How it closes</th><td class="sp-mono sp-small">Esc<br>a click outside<br>the cross</td></tr></tbody></table></div></div></div></div></td><td class="sp-mono sp-small">button.tag + sp-sheet (popover)</td></tr>
</tbody>
</table></div>
</div>
@@ -14,7 +14,7 @@
<div><dt>Last report</dt><dd>2026-09-21 06:12 UTC</dd></div>
<div><dt>This week</dt><dd>14 kept · 0 parse failures</dd></div>
<div><dt>Hosted addresses</dt><dd class="sp-mono sp-small">dmarc-reports+&lt;domain&gt;@mail.example.org</dd></div>
<div><dt>Retention</dt><dd>500 reports or 90 days</dd></div>
<div><dt>Retention</dt><dd>500 reports a domain or 90 days</dd></div>
</dl>
</div>
</div>
@@ -79,10 +79,29 @@
</div>
</div>
<div class="box" id="configuration">
<div class="sp-box-head"><span class="sp-no">06</span><h2>Configuration</h2><div class="sp-end">Safe to run at any time</div></div>
<div class="sp-box-body">
<p class="mb-3">Regenerates the OpenDKIM and Postfix configuration from the database and reloads both daemons. Use it after editing the files by hand, after restoring a backup, or when the running configuration looks out of step with the domain and application lists. It does not touch the queue or the TLS certificate.</p>
<form method="post" action="#"><button type="submit" class="button"><i class="ti ti-reload"></i>Reload configuration</button></form>
<div class="columns">
<div class="column is-6">
<div class="box" id="notification-mail">
<div class="sp-box-head"><span class="sp-no">06</span><h2>Notification mail</h2><div class="sp-end"><span class="tag is-warning is-light">warn</span></div><a class="sp-help" href="help.html#notifications" title="What the panel sends and what makes it arrive"><i class="ti ti-help-circle"></i></a></div>
<div class="table-container"><table class="table is-fullwidth">
<thead><tr><th>Record</th><th>State</th></tr></thead>
<tbody>
<tr><td>DKIM<br><span class="sp-muted sp-mono sp-small">sp2026._domainkey.mail.example.org</span></td><td><span class="tag is-success is-light">published</span></td></tr>
<tr><td>SPF<br><span class="sp-muted sp-mono sp-small">mail.example.org</span></td><td><span class="tag is-warning is-light">missing</span></td></tr>
</tbody>
</table></div>
<div class="sp-box-body">
<p class="sp-muted">Sent from <span class="sp-mono">postmaster@mail.example.org</span>. What to publish is under <a href="settings.html#notification-mail">Settings</a>.</p>
</div>
</div>
</div>
<div class="column is-6">
<div class="box" id="configuration">
<div class="sp-box-head"><span class="sp-no">07</span><h2>Configuration</h2><div class="sp-end">Safe to run at any time</div></div>
<div class="sp-box-body">
<p class="mb-3">Regenerates the OpenDKIM and Postfix configuration from the database and reloads both daemons. Use it after editing the files by hand, after restoring a backup, or when the running configuration looks out of step with the domain and application lists. It does not touch the queue or the TLS certificate.</p>
<form method="post" action="#"><button type="submit" class="button"><i class="ti ti-reload"></i>Reload configuration</button></form>
</div>
</div>
</div>
</div>
+10 -4
View File
@@ -31,12 +31,15 @@
<li><a href="#upstream"><i class="ti ti-arrow-forward-up"></i>Upstream</a></li>
<li><a href="#recipients"><i class="ti ti-address-book"></i>Valid recipients</a></li>
<li><a href="#filter"><i class="ti ti-shield-check"></i>Spam filter</a></li>
<li><a href="#inbound-log"><i class="ti ti-list-details"></i>Inbound log</a></li>
<li><a href="#filter-lists"><i class="ti ti-filter"></i>Filter lists</a></li>
</ul>
<p class="menu-label">Server</p>
<ul class="menu-list">
<li><a href="#backup"><i class="ti ti-archive"></i>Backup and moving a domain</a></li>
<li><a href="#users"><i class="ti ti-users"></i>Users and roles</a></li>
<li><a href="#settings"><i class="ti ti-settings"></i>Settings</a></li>
<li><a href="#notifications"><i class="ti ti-mail"></i>Notifications</a></li>
</ul>
<p class="menu-label">Account</p>
<ul class="menu-list">
@@ -56,7 +59,7 @@
<div class="sp-help-topic" id="dns"><h3>DNS records</h3><p>DKIM proves the message was signed by this relay, SPF lists the addresses allowed to send for the domain, DMARC tells receivers what to do when both fail and where to report. Publish all three exactly as shown; merge SPF into an existing record rather than adding a second one. A domain's status is the worst of its three records, and results are cached for a few minutes, so use Re-check DNS after publishing. The SPF check is shallow: it looks for this server's address in the record itself and does not follow <code>include:</code> or <code>redirect=</code>, so a record that relies on them is reported as a warning, not a failure. <code>p=none</code> does not affect delivery; tighten to <code>quarantine</code>, then <code>reject</code>, once the reports look clean. A domain's report address is set in its Domain settings.</p></div>
<div class="sp-help-topic" id="apps"><h3>Applications</h3><p>An application is one SASL login, unique across all domains. It may send as any address of its domain or only as listed addresses, and, with a list of client IPs set, only from those addresses. The password is shown once at creation or regeneration and is not stored in readable form.</p></div>
<div class="sp-help-topic" id="limits"><h3>Rate limits</h3><p>Level 1 is a per-IP ceiling set in <code>.env</code>. Level 2 is optional, per domain and per application, entered by hand or computed from the last 30 days of sending, and can never exceed level 1. An application's own limit replaces its domain's.</p></div>
<div class="sp-help-topic" id="log"><h3>Outbound log</h3><p>One row per message and recipient, with the status Postfix last reported: queued (accepted, nothing reported yet), sent (the receiving server took it; what the mailbox does next is not reported back), deferred (not taken yet, Postfix tries again), bounced (failed for good, or the queue lifetime ran out) and rejected (refused before queueing under a level-2 rate limit, so it never reached Postfix). A message's page lists these steps and the lines Postfix wrote about it in <code>mail.log</code>, found by queue id; that file is rotated daily and about two weeks are kept, so an old message may show none. Rows are deleted after the retention set in Server › Settings, 90 days by default. A domain administrator sees only the rows of their own domains.</p></div>
<div class="sp-help-topic" id="log"><h3>Outbound log</h3><p>One row per message and recipient, with the status Postfix last reported: queued (accepted, nothing reported yet), sent (the receiving server took it; what the mailbox does next is not reported back), deferred (not taken yet, Postfix tries again), bounced (failed for good, or the queue lifetime ran out) and rejected (refused before queueing under a level-2 rate limit, so it never reached Postfix). A message's page lists these steps and the lines Postfix wrote about it in <code>mail.log</code>, found by queue id. The lines are kept with the row: they stay on the page after that file has rotated, for as long as the row does. A message accepted before version 2.2.0 shows them only while they are still in the current file. Rows are deleted after the retention set in Server › Settings, 90 days by default, and their lines with them. A domain administrator sees only the rows of their own domains.</p></div>
<div class="sp-help-topic" id="queue"><h3>Queue and retries</h3><p>The page lists, as <code>postqueue -p</code> prints it, the messages Postfix has accepted and not yet delivered: mail being sent right now and mail deferred for a later attempt. Only the global role can open it, and it only shows; nothing here deletes or forces a message. A deferred message is tried again after the first retry delay, then at gaps that double up to a cap, until the receiving server takes it or the queue lifetime runs out and it bounces; there is no attempt counter, and the box on the page shows this server's values. A queue that keeps growing means a receiver is refusing or not answering: open the message from the Outbound log to read the reason Postfix logged.</p></div>
<div class="sp-help-topic" id="dmarc"><h3>DMARC reports</h3><p>An aggregate report is the summary a receiving provider sends, usually daily, to the <code>rua=</code> address of your DMARC record: which servers sent mail as your domain and whether SPF or DKIM passed for it. A domain's report address is one of three choices, set in its Domain settings: no reports (the default for a new domain), the SelfPost hosted address (an address on this server whose reports are parsed and shown under DMARC reports; offered only while hosted reports are enabled on the server), or an address typed for the domain (an ordinary mailbox that receives the raw XML; a button fills in your own profile e-mail). When the typed address is on another domain than the one sending, that domain must publish a report-authorization record, or providers will not send; the domain's page shows it with the other DNS records and checks it. In the tables pass means SPF or DKIM passed and aligned, fail means neither did, and your own server shows as <em>this relay</em>. A fail from any other source is someone else sending as your domain — a forwarder, an application you forgot, or abuse — so do not tighten <code>p=</code> until that sender is fixed or removed. A domain administrator reads the reports of their own domains.</p></div>
</div>
@@ -66,16 +69,19 @@
<div class="sp-help-topic" id="upstream"><h3>Upstream</h3><p>The upstream is the server that actually holds the mailboxes. Give its host and port — used exactly as written, with no MX lookup — and choose the TLS for the hop to it: Opportunistic uses TLS when the upstream offers it and otherwise sends in the clear, Required holds the mail until a TLS session is possible, Off never uses TLS. While the upstream does not answer, Postfix keeps the mail and retries as for any other mail.</p></div>
<div class="sp-help-topic" id="recipients"><h3>Valid recipients</h3><p>Choose whom this domain accepts. Listed addresses only: the relay accepts exactly the addresses you list, one per line or comma-separated, and refuses any other while the sender is still connected, so it never has to send a bounce for them. Any recipient: everything at the domain is accepted and passed on, and an address the upstream then rejects comes back as a bounce from this server to a sender who is often forged — backscatter. Prefer a list, and keep it in step with the mailboxes upstream.</p></div>
<div class="sp-help-topic" id="filter"><h3>Spam filter</h3><p>The spam filter belongs to the server, not to a domain. The server's administrator attaches it in the server's <code>.env</code>, and from then on every inbound domain is screened by it before mail is handed on; the box shows on or off and has nothing to switch. Off means mail goes to the upstream unfiltered. While the filter is down, mail is either accepted unfiltered (the default) or deferred, so the sending server tries again later, as the server's setting says; the box states which.</p></div>
<div class="sp-help-topic" id="inbound-log"><h3>Inbound log</h3><p>One row per recipient of a message offered on port 25 for an inbound domain, with what was decided: <span class="tag is-success is-light">accept</span> — handed on to the upstream; <span class="tag is-danger is-light">reject</span> — refused for good; <span class="tag is-warning is-light">tempfail</span> — the sender was asked to come back later; <span class="tag is-primary is-light">quarantine</span> — held by the filter. <em>Why</em> names who decided. <code>rspamd</code> is the spam filter, with its action, the score against the reject threshold and the heaviest symbols; <code>postfix</code> is the relay's own rule — an unknown recipient, a sender domain that does not exist — with its reply; <code>selfpost</code> means nobody judged the message: it was relayed with no filter attached. Accept says what this server did, not what the upstream did with the message afterwards.<br><br>Accepted mail is recorded as it passes; refused mail is read from <code>mail.log</code>, so its row has no subject, and a message refused after its text was sent is one row however many recipients it had. A refusal to relay for a domain that is not an inbound one is every scanner on the internet and is not recorded. Rows are kept for the number of days set in Settings, and never more than 10 000.</p></div>
<div class="sp-help-topic" id="filter-lists"><h3>Filter lists</h3><p>Two lists for the whole server that correct the spam filter by hand. An entry is a sender address, a sender domain — that domain, not its subdomains — or a client address or network in CIDR form. A sender on the Deny list is refused outright. A sender on the Allow list is given a weight no spam score outweighs, so the filter lets it through. The sender's address is whatever the sending server says it is - anyone can claim to be a sender you allow - so allow a client address or network where you can, and a sender address only for mail you would rather receive forged than lose. When entries of both lists match one message, <strong>Deny wins</strong>.<br><br>SelfPost does not apply the lists itself: it writes them as files under <code>/data/antispam/</code>, and rspamd rereads a file within seconds of a change. With the spam filter off the lists are kept and can be edited, but nothing reads them. The sender matched is the envelope sender — the one in the log — not the From line a mail program shows. Matched counts the messages in the Inbound log that the entry decided, so it reaches back as far as the log does; never means not once in that time, and such an entry is one to consider removing.</p></div>
</div>
<div class="box">
<div class="sp-box-head"><span class="sp-no">04</span><h2>Server</h2></div>
<div class="sp-help-topic" id="backup"><h3>Backup and moving a domain</h3><p>Full backup downloads one archive of the whole instance: <code>data/</code> (the database, DKIM keys, application credentials and the queue), <code>docker-compose.yml</code>, <code>.env</code> and <code>certs/</code>. It holds secrets, so leave Encrypt with a password ticked: the file is sealed as <code>.spbk</code>, the password is not stored, and without it nothing can be recovered. To restore, unpack it into an empty project directory (decrypt a <code>.spbk</code> first with <code>selfpost-backup -decrypt</code>) and start the same SelfPost version that made it; the panel refuses data from another version. The reverse-proxy configuration and <code>mail.log</code> are not included. One domain moves on its own: Export domain on its settings page writes a <code>.json</code> (or encrypted <code>.spde</code>) with its DKIM key and its applications' passwords, and Import a domain here reads it into another instance, so the published DNS needs no change.</p></div>
<div class="sp-help-topic" id="backup"><h3>Backup and moving a domain</h3><p>Full backup downloads one archive of the whole instance: <code>data/</code> (the database, DKIM keys and application credentials), <code>docker-compose.yml</code>, <code>.env</code> and <code>certs/</code>. The Postfix queue is not in it. The backup is taken while the server is running, and Postfix keeps moving messages through the queue, so a copy could hold a message twice or miss it; mail still waiting for delivery is therefore not carried over, and a restored server starts with an empty queue. To keep the queue, stop the container and archive the project directory instead. The private key in <code>certs/</code> has to be readable by the panel: give it to group 998 on the host (<code>chgrp 998</code>, <code>chmod 640</code>). Until then the panel refuses to make a backup and names the file, rather than hand over an archive without the key. It holds secrets, so leave Encrypt with a password ticked: the file is sealed as <code>.spbk</code>, the password is not stored, and without it nothing can be recovered. To restore, unpack it into an empty project directory (decrypt a <code>.spbk</code> first with <code>selfpost-backup -decrypt</code>) and start the same SelfPost version that made it; the panel refuses data from another version. The reverse-proxy configuration and <code>mail.log</code> are not included. One domain moves on its own: Export domain on its settings page writes a <code>.json</code> (or encrypted <code>.spde</code>) with its DKIM key and its applications' passwords, and Import a domain here reads it into another instance, so the published DNS needs no change.</p></div>
<div class="sp-help-topic" id="users"><h3>Users and roles</h3><p>The two roles are a reach, not a rank: every user is an administrator and signs in with a username and password. Global reaches the whole server — Overview, Server, the queue and every domain — and is the only role that adds or deletes domains and users. Domain reaches only the domains assigned to it, outbound and inbound separately (the same name on both lists is two assignments); <em>All</em> includes domains added later, and a domain user needs at least one domain or <em>All</em> on either list. The only global user cannot be demoted or deleted, and you cannot delete the account you are signed in with. A new password set here signs that user out of their other sessions.</p></div>
<div class="sp-help-topic" id="settings"><h3>Settings</h3><p>Settings holds what is true of the whole instance. Log retention is how many days the Outbound log keeps its rows, from 7 to 365 and 90 by default; a sweep every six hours deletes older ones, and it is the main driver of how fast <code>/data</code> grows. It does not change the rotation of <code>mail.log</code>, which is daily with about two weeks kept. The sending rate limit shown under it is level 1, the per-client-IP ceiling set in <code>.env</code>: changing it needs a restart, and no level-2 limit can exceed it.</p></div>
<div class="sp-help-topic" id="settings"><h3>Settings</h3><p>Settings holds what is true of the whole instance. Log retention is how many days the Outbound log keeps its rows, from 7 to 365 and 90 by default; a sweep every six hours deletes older ones, and it is the main driver of how fast <code>/data</code> grows. With the inbound relay on there is a second number, for the Inbound log, with the same range. It does not change the rotation of <code>mail.log</code>, which is daily with about two weeks kept. The sending rate limit shown under it is level 1, the per-client-IP ceiling set in <code>.env</code>: changing it needs a restart, and no level-2 limit can exceed it. Notifications are turned on or off here for everyone.</p></div>
<div class="sp-help-topic" id="notifications"><h3>Notifications</h3><p>The panel writes to its users when something needs them, from <code>postmaster@</code> the server's hostname. Four things are reported. A sending domain's SPF, DKIM or DMARC record stops matching: sent once two hourly checks in a row have found it so, and once more when the record matches again. A level-2 rate limit refuses mail: sent at the first refusal, then at most once a day for the same limit. The TLS certificate is 14 days from expiry, and again at 3 days if it is still the same certificate. A message has waited in the queue for more than three days, with one more message when none is left. A global user is told all of it; a domain administrator is told about the DNS records and limits of their own domains.<br><br>Each user turns notifications on or off under Account and needs an e-mail there; Server › Settings turns them off for everyone. They are the panel's own mail: not rows of the Outbound log, and not counted against any limit.<br><br>For that mail to arrive, the server's own name needs two DNS records of its own, as each of your domains does: a DKIM key, which the panel generates and signs notifications with, and an SPF record, <code>v=spf1 a -all</code>. Server › Settings gives both with the values to publish, and Server › Health shows whether they are in place. Without them notifications are still sent, and a receiver may file them as spam. Mail of your domains does not depend on either.</p></div>
</div>
<div class="box">
<div class="sp-box-head"><span class="sp-no">05</span><h2>Account</h2></div>
<div class="sp-help-topic" id="account"><h3>Your account</h3><p>Account is your own, whatever your role: your username and e-mail, and your password. Changing the password signs out every other session of this account and keeps the one you are using; applications have their own logins and are not affected.</p></div>
<div class="sp-help-topic" id="account"><h3>Your account</h3><p>Account is your own, whatever your role: your username and e-mail, whether the panel sends you notifications, and your password. Changing the password signs out every other session of this account and keeps the one you are using; applications have their own logins and are not affected.</p></div>
</div>
</div>
</div>
@@ -8,7 +8,7 @@
</div>
<div class="box">
<div class="sp-box-head"><span class="sp-no">01</span><h2>Add an entry</h2><a class="sp-help" href="help.html#inbound" title="How the lists are applied"><i class="ti ti-help-circle"></i></a></div>
<div class="sp-box-head"><span class="sp-no">01</span><h2>Add an entry</h2><a class="sp-help" href="help.html#filter-lists" title="How the lists are applied"><i class="ti ti-help-circle"></i></a></div>
<div class="sp-box-body">
<form class="sp-form" method="post" action="#">
<div class="columns">
@@ -18,7 +18,7 @@
</div>
<div class="column is-3">
<div class="field"><label class="label" for="etype">Entry is</label>
<div class="control"><div class="select is-fullwidth"><select id="etype" name="entry_type"><option>a sender address</option><option>a sender domain</option><option>a client IP or CIDR</option></select></div></div></div>
<div class="control"><div class="select is-fullwidth"><select id="etype" name="entry_type"><option>a sender address</option><option>a sender domain</option><option>a client address or network</option></select></div></div></div>
</div>
<div class="column is-6">
<div class="field"><label class="label" for="value">Value</label>
@@ -37,11 +37,11 @@
<div class="box">
<div class="sp-box-head"><span class="sp-no">02</span><h2>Deny</h2><div class="sp-end">3 entries · wins over Allow</div></div>
<div class="table-container"><table class="table is-fullwidth is-hoverable">
<thead><tr><th>Value</th><th>Note</th><th>Added</th><th></th></tr></thead>
<thead><tr><th>Value</th><th>Kind</th><th>Matched</th><th></th></tr></thead>
<tbody>
<tr><td class="sp-mono sp-small">@bulk.example <span class="tag is-light">domain</span></td><td class="sp-muted">newsletter nobody subscribed to</td><td class="sp-muted sp-nowrap sp-small">2026-09-02</td><td class="sp-actions"><a class="has-text-danger" href="#">Remove</a></td></tr>
<tr><td class="sp-mono sp-small">198.51.100.0/24 <span class="tag is-light">ip</span></td><td class="sp-muted">dictionary attack on lists.example.org</td><td class="sp-muted sp-nowrap sp-small">2026-08-19</td><td class="sp-actions"><a class="has-text-danger" href="#">Remove</a></td></tr>
<tr><td class="sp-mono sp-small">win@lottery.example <span class="tag is-light">address</span></td><td class="sp-muted">—</td><td class="sp-muted sp-nowrap sp-small">2026-07-30</td><td class="sp-actions"><a class="has-text-danger" href="#">Remove</a></td></tr>
<tr><td class="sp-mono sp-small">bulk.example</td><td><button type="button" class="tag is-light" popovertarget="deny-1">domain</button><div class="sp-sheet" id="deny-1" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-light">domain</span><h2>bulk.example</h2><button type="button" popovertarget="deny-1" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Note</th><td>newsletter nobody subscribed to</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-09-02</td></tr><tr><th>Matched</th><td>41 in the last 90 days</td></tr><tr><th>Last match</th><td class="sp-mono sp-small">2026-09-21 13:57</td></tr></tbody></table></div></div></div></div></td><td class="sp-nowrap">41</td><td class="sp-actions"><form method="post" action="#"><button type="submit" class="has-text-danger">Remove</button></form></td></tr>
<tr><td class="sp-mono sp-small">198.51.100.0/24</td><td><button type="button" class="tag is-light" popovertarget="deny-2">ip</button><div class="sp-sheet" id="deny-2" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-light">ip</span><h2>198.51.100.0/24</h2><button type="button" popovertarget="deny-2" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Note</th><td>dictionary attack on lists.example.org</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-08-19</td></tr><tr><th>Matched</th><td>1203 in the last 90 days</td></tr><tr><th>Last match</th><td class="sp-mono sp-small">2026-09-21 09:12</td></tr></tbody></table></div></div></div></div></td><td class="sp-nowrap">1203</td><td class="sp-actions"><form method="post" action="#"><button type="submit" class="has-text-danger">Remove</button></form></td></tr>
<tr><td class="sp-mono sp-small">win@lottery.example</td><td><button type="button" class="tag is-light" popovertarget="deny-3">address</button><div class="sp-sheet" id="deny-3" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-light">address</span><h2>win@lottery.example</h2><button type="button" popovertarget="deny-3" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Note</th><td>—</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-07-30</td></tr><tr><th>Matched</th><td>never in the last 90 days</td></tr></tbody></table></div></div></div></div></td><td class="sp-muted sp-nowrap">never</td><td class="sp-actions"><form method="post" action="#"><button type="submit" class="has-text-danger">Remove</button></form></td></tr>
</tbody>
</table></div>
</div>
@@ -50,10 +50,10 @@
<div class="box">
<div class="sp-box-head"><span class="sp-no">03</span><h2>Allow</h2><div class="sp-end">2 entries</div></div>
<div class="table-container"><table class="table is-fullwidth is-hoverable">
<thead><tr><th>Value</th><th>Note</th><th>Added</th><th></th></tr></thead>
<thead><tr><th>Value</th><th>Kind</th><th>Matched</th><th></th></tr></thead>
<tbody>
<tr><td class="sp-mono sp-small">@partner.example <span class="tag is-light">domain</span></td><td class="sp-muted">invoices were scored as spam</td><td class="sp-muted sp-nowrap sp-small">2026-09-11</td><td class="sp-actions"><a class="has-text-danger" href="#">Remove</a></td></tr>
<tr><td class="sp-mono sp-small">noreply@github.example <span class="tag is-light">address</span></td><td class="sp-muted">security advisories</td><td class="sp-muted sp-nowrap sp-small">2026-06-04</td><td class="sp-actions"><a class="has-text-danger" href="#">Remove</a></td></tr>
<tr><td class="sp-mono sp-small">partner.example</td><td><button type="button" class="tag is-light" popovertarget="allow-1">domain</button><div class="sp-sheet" id="allow-1" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-light">domain</span><h2>partner.example</h2><button type="button" popovertarget="allow-1" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Note</th><td>invoices were scored as spam</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-09-11</td></tr><tr><th>Matched</th><td>7 in the last 90 days</td></tr><tr><th>Last match</th><td class="sp-mono sp-small">2026-09-19 16:40</td></tr></tbody></table></div></div></div></div></td><td class="sp-nowrap">7</td><td class="sp-actions"><form method="post" action="#"><button type="submit" class="has-text-danger">Remove</button></form></td></tr>
<tr><td class="sp-mono sp-small">noreply@github.example</td><td><button type="button" class="tag is-light" popovertarget="allow-2">address</button><div class="sp-sheet" id="allow-2" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-light">address</span><h2>noreply@github.example</h2><button type="button" popovertarget="allow-2" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Note</th><td>security advisories</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-06-04</td></tr><tr><th>Matched</th><td>18 in the last 90 days</td></tr><tr><th>Last match</th><td class="sp-mono sp-small">2026-09-21 13:30</td></tr></tbody></table></div></div></div></div></td><td class="sp-nowrap">18</td><td class="sp-actions"><form method="post" action="#"><button type="submit" class="has-text-danger">Remove</button></form></td></tr>
</tbody>
</table></div>
</div>
@@ -64,10 +64,10 @@
<div class="sp-box-head"><span class="sp-no">04</span><h2>Where the lists go</h2><div class="sp-end"><span class="tag is-success is-light">applied</span></div></div>
<div class="sp-box-body">
<dl class="sp-facts">
<div><dt>Applied by</dt><dd>rspamd, from map files</dd></div>
<div><dt>Files</dt><dd class="sp-mono sp-small">/data/antispam/allow.map<br>/data/antispam/deny.map</dd></div>
<div><dt>Applied by</dt><dd>rspamd, from map files it rereads within seconds of a change</dd></div>
<div><dt>Files</dt><dd><span class="sp-mono">/data/antispam/</span> — six, one for each list and kind of entry</dd></div>
<div><dt>Last written</dt><dd>2026-09-11 08:40 UTC</dd></div>
<div><dt>When the filter is off</dt><dd class="sp-muted">The lists are kept and editable, but nothing reads them until <code>INBOUND_ANTISPAM_MILTER</code> is set.</dd></div>
<div><dt>When both lists match</dt><dd>Deny wins, whichever entries they are.</dd></div>
</dl>
</div>
</div>
@@ -3,7 +3,7 @@
<div class="sp-grow">
<p class="sp-kicker">Inbound</p>
<h1 class="title is-3">Inbound log</h1>
<p class="sp-lead">What port 25 decided about each message, one row per recipient. Accepts come from the inbound journal, rejects from <code>mail.log</code>; the filter's reasons appear when the spam filter is on.</p>
<p class="sp-lead">What port 25 decided about each message, one row per recipient. Accepted mail is recorded as it passes, refused mail is read from <code>mail.log</code>; the filter's verdict appears when the spam filter is on. Rows older than 90 days are deleted — change that in <a href="settings.html">Settings</a>.</p>
</div>
</div>
@@ -17,17 +17,18 @@
<input class="input" type="date" name="to" value="2026-09-21" aria-label="To date">
<button type="submit" class="button"><i class="ti ti-filter"></i>Filter</button>
</form>
<a class="sp-help" href="help.html#inbound-log" title="What the decisions mean"><i class="ti ti-help-circle"></i></a>
</div>
<div class="table-container"><table class="table is-fullwidth is-hoverable">
<thead><tr><th>Time, UTC</th><th>From · client</th><th>To</th><th>Subject</th><th>Decision</th><th>Why</th><th></th></tr></thead>
<thead><tr><th>Time, UTC</th><th>From</th><th>To</th><th>Subject</th><th>Decision</th><th></th></tr></thead>
<tbody>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 14:00:18</td><td>t.berg@<wbr>uni.example<br><span class="sp-mono sp-muted sp-small">192.0.2.77</span></td><td>dev@<wbr>lists.example.org</td><td class="sp-clip">Re: [dev] patch series v3</td><td><span class="tag is-success is-light">accept</span></td><td class="sp-muted sp-small">rspamd · score 0.4</td><td class="sp-actions"><a href="in-filter-lists.html">Deny sender</a></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:59:02</td><td>invoice@<wbr>partner-billing.example<br><span class="sp-mono sp-muted sp-small">203.0.113.140</span></td><td>owner@<wbr>lists.example.org</td><td class="sp-clip">Outstanding invoice — action required</td><td><span class="tag is-primary is-light">quarantine</span></td><td class="sp-muted sp-small">rspamd · score 9.8 · PHISHING, FORGED_SENDER</td><td class="sp-actions"><a href="in-quarantine-message.html">Review</a></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:57:41</td><td>promo@<wbr>bulk.example<br><span class="sp-mono sp-muted sp-small">198.51.100.90</span></td><td>sales@<wbr>lists.example.org</td><td class="sp-clip sp-muted">—</td><td><span class="tag is-danger is-light">reject</span></td><td class="sp-muted sp-small">postfix · 550 5.1.1 unknown recipient</td><td class="sp-actions"><a href="in-filter-lists.html">Deny sender</a></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:51:30</td><td>billing@<wbr>partner.example<br><span class="sp-mono sp-muted sp-small">198.51.100.12</span></td><td>owner@<wbr>lists.example.org</td><td class="sp-clip">Invoice 2026-0917</td><td><span class="tag is-danger is-light">reject</span></td><td class="sp-muted sp-small">rspamd · score 16.2 · BAYES_SPAM</td><td class="sp-actions"><a href="in-filter-lists.html">Allow sender</a></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:44:02</td><td>hr@<wbr>partner.example<br><span class="sp-mono sp-muted sp-small">192.0.2.31</span></td><td>jobs@<wbr>acme.io</td><td class="sp-clip">Application: backend engineer</td><td><span class="tag is-warning is-light">tempfail</span></td><td class="sp-muted sp-small">selfpost · upstream 10.0.4.12 timed out</td><td class="sp-actions"></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:30:55</td><td>noreply@<wbr>github.example<br><span class="sp-mono sp-muted sp-small">192.0.2.200</span></td><td>security@<wbr>lists.example.org</td><td class="sp-clip">[advisory] new report</td><td><span class="tag is-success is-light">accept</span></td><td class="sp-muted sp-small">rspamd · allow list</td><td class="sp-actions"></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 14:00:18</td><td>t.berg@<wbr>uni.example<br><span class="sp-mono sp-muted sp-small">192.0.2.77</span></td><td>dev@<wbr>lists.example.org</td><td class="sp-clip">Re: [dev] patch series v3</td><td><button type="button" class="tag is-success is-light" popovertarget="why-1">accept</button><div class="sp-sheet" id="why-1" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-success is-light">accept</span><h2>Why</h2><button type="button" popovertarget="why-1" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>rspamd</td></tr><tr><th>Verdict</th><td>no action, score 0.40 / 15.00</td></tr><tr><th>Subject</th><td>Re: [dev] patch series v3</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"><a href="in-filter-lists.html">Deny sender</a></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:59:02</td><td>invoice@<wbr>partner-billing.example<br><span class="sp-mono sp-muted sp-small">203.0.113.140</span></td><td>owner@<wbr>lists.example.org</td><td class="sp-clip">Outstanding invoice — action required</td><td><button type="button" class="tag is-success is-light" popovertarget="why-2">accept</button><div class="sp-sheet" id="why-2" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-success is-light">accept</span><h2>Why</h2><button type="button" popovertarget="why-2" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>rspamd</td></tr><tr><th>Verdict</th><td>add header, score 9.80 / 15.00</td></tr><tr><th>Symbols</th><td class="sp-mono sp-small">PHISHING(5.00)<br>FORGED_SENDER(3.00)<br>R_SPF_FAIL(1.00)</td></tr><tr><th>Subject</th><td>Outstanding invoice — action required</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"><a href="in-filter-lists.html">Deny sender</a></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:57:41</td><td>promo@<wbr>bulk.example<br><span class="sp-mono sp-muted sp-small">198.51.100.90</span></td><td>sales@<wbr>lists.example.org</td><td class="sp-clip sp-muted">—</td><td><button type="button" class="tag is-danger is-light" popovertarget="why-3">reject</button><div class="sp-sheet" id="why-3" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-danger is-light">reject</span><h2>Why</h2><button type="button" popovertarget="why-3" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>postfix</td></tr><tr><th>Reply</th><td class="sp-mono sp-small">550 5.1.1 &lt;sales@lists.example.org&gt;: Recipient address rejected: User unknown in relay recipient table</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:51:30</td><td>billing@<wbr>partner.example<br><span class="sp-mono sp-muted sp-small">198.51.100.12</span></td><td>owner@<wbr>lists.example.org</td><td class="sp-clip sp-muted">—</td><td><button type="button" class="tag is-danger is-light" popovertarget="why-4">reject</button><div class="sp-sheet" id="why-4" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-danger is-light">reject</span><h2>Why</h2><button type="button" popovertarget="why-4" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>rspamd</td></tr><tr><th>Reply</th><td class="sp-mono sp-small">5.7.1 Spam message rejected</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"><a href="in-filter-lists.html">Allow sender</a></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:44:02</td><td>hr@<wbr>partner.example<br><span class="sp-mono sp-muted sp-small">192.0.2.31</span></td><td>jobs@<wbr>acme.io</td><td class="sp-clip sp-muted">—</td><td><button type="button" class="tag is-warning is-light" popovertarget="why-5">tempfail</button><div class="sp-sheet" id="why-5" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-warning is-light">tempfail</span><h2>Why</h2><button type="button" popovertarget="why-5" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>rspamd</td></tr><tr><th>Reply</th><td class="sp-mono sp-small">4.7.1 Try again later</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"><a href="in-filter-lists.html">Allow sender</a></td></tr>
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:30:55</td><td>noreply@<wbr>github.example<br><span class="sp-mono sp-muted sp-small">192.0.2.200</span></td><td>security@<wbr>lists.example.org</td><td class="sp-clip">[advisory] new report</td><td><button type="button" class="tag is-success is-light" popovertarget="why-6">accept</button><div class="sp-sheet" id="why-6" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-success is-light">accept</span><h2>Why</h2><button type="button" popovertarget="why-6" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>rspamd</td></tr><tr><th>Verdict</th><td>no action, score -98.10 / 15.00</td></tr><tr><th>Symbols</th><td class="sp-mono sp-small">SELFPOST_ALLOW_FROM(-100.00)<br>R_DKIM_ALLOW(-0.20)</td></tr><tr><th>Subject</th><td>[advisory] new report</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"><a href="in-filter-lists.html">Deny sender</a></td></tr>
</tbody>
</table></div>
<div class="sp-box-foot"><span>Page 1 of 38</span><a href="#">Older →</a><span class="sp-end">Allow / Deny sender opens <a href="in-filter-lists.html">Filter lists</a> with the entry filled in · global administrator only</span></div>
<div class="sp-box-foot"><span>Page 1 of 38</span><a href="#">Older →</a><span class="sp-end">A decision opens its reason · Allow / Deny sender opens <a href="in-filter-lists.html">Filter lists</a> with the entry filled in</span></div>
</div>
@@ -16,11 +16,12 @@
<div class="field"><label class="label" for="ret">Keep outbound log rows, days</label>
<div class="control"><input class="input" id="ret" name="send_log_retention_days" type="number" min="7" max="365" value="30" required></div>
<p class="help">The main driver of <code>/data</code> growth. Does not affect <code>mail.log</code> rotation (fourteen daily files).</p></div>
<div class="field"><label class="label" for="inret">Keep inbound log rows, days</label>
<div class="control"><input class="input" id="inret" name="inbound_spam_log_retention_days" type="number" min="7" max="365" value="90" required></div>
<p class="help">The inbound log also never holds more than 10 000 rows: refused mail is logged at the rate strangers knock on port 25.</p></div>
</div>
</div>
</div>
<div class="column is-6">
<div class="box" id="rate-limits">
<div class="sp-box-head"><span class="sp-no">02</span><h2>Sending rate limits</h2><div class="sp-end">Read-only</div><a class="sp-help" href="help.html#limits" title="Levels 1 and 2"><i class="ti ti-help-circle"></i></a></div>
<div class="sp-box-body">
@@ -31,6 +32,43 @@
</div>
</div>
</div>
<div class="column is-6">
<div class="box" id="notifications">
<div class="sp-box-head"><span class="sp-no">03</span><h2>Notifications</h2><a class="sp-help" href="help.html#notifications" title="What is sent and when"><i class="ti ti-help-circle"></i></a></div>
<div class="sp-box-body">
<div class="field"><label class="checkbox"><input type="checkbox" name="notify" checked> Send notification e-mails</label>
<p class="help">From <span class="sp-mono">postmaster@mail.example.org</span>, to every user who has an e-mail and has not turned them off under <a href="account.html">Account</a>.</p></div>
</div>
</div>
<div class="box" id="notification-mail">
<div class="sp-box-head"><span class="sp-no">04</span><h2>Notification mail</h2><div class="sp-end">DNS records of the server's own name</div><a class="sp-help" href="help.html#notifications" title="What the panel sends and what makes it arrive"><i class="ti ti-help-circle"></i></a></div>
<div class="sp-record">
<div class="sp-record-head"><h3>DKIM</h3><span class="tag is-success is-light">published</span><small>Not a secret · selector <code>sp2026</code></small></div>
<div class="sp-row">
<div class="field"><label class="label">Host / name</label>
<div class="field has-addons"><div class="control is-expanded"><input class="input" readonly value="sp2026._domainkey.mail.example.org"></div><div class="control"><button type="button" class="button"><i class="ti ti-copy"></i>Copy</button></div></div></div>
<div class="field"><label class="label">Type</label><div class="control"><input class="input" readonly value="TXT"></div></div>
</div>
<div class="field"><label class="label">Value to publish</label>
<div class="field has-addons"><div class="control is-expanded"><textarea class="textarea" rows="5" readonly>v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAq7n2Lw0c5vXeR1dYt8uJb3mKs6pHf9aZoWg4iNy2TqCx0lVjE7rUh5sBk1dMa8fPz3oGc6yXw9tIe4nQv2uL…IDAQAB</textarea></div><div class="control"><button type="button" class="button"><i class="ti ti-copy"></i>Copy</button></div></div></div>
</div>
<div class="sp-record">
<div class="sp-record-head"><h3>SPF</h3><span class="tag is-warning is-light">missing</span><small>For the server's own name, not for your domains</small></div>
<div class="sp-row">
<div class="field"><label class="label">Host / name</label>
<div class="field has-addons"><div class="control is-expanded"><input class="input" readonly value="mail.example.org"></div><div class="control"><button type="button" class="button"><i class="ti ti-copy"></i>Copy</button></div></div></div>
<div class="field"><label class="label">Type</label><div class="control"><input class="input" readonly value="TXT"></div></div>
</div>
<div class="field"><label class="label">Value to publish</label>
<div class="field has-addons"><div class="control is-expanded"><input class="input" readonly value="v=spf1 a -all"></div><div class="control"><button type="button" class="button"><i class="ti ti-copy"></i>Copy</button></div></div></div>
<p class="help">Notifications are sent without it, and a receiver may file them as spam.</p>
</div>
</div>
</div>
</div>
<div class="buttons"><button type="submit" class="button is-primary">Save settings</button></div>
</form>
@@ -270,3 +270,27 @@ code { background: var(--sp-well); color: var(--sp-ink); border-radius: 3px; pad
.sp-record-head small { margin-left: 0; text-align: left; flex-basis: 100%; }
.sp-postmark { display: none; }
}
/* sheet: what does not fit a table row, opened over the page from the row's
tag. It is the browser's own popover (the tag is a button with
popovertarget), so it needs no script: Esc, a click outside and the close
button all shut it, and focus returns to the tag. It rises from the bottom
edge across the whole window - the one component that is not laid out by the
page, hence the widths - with its content, a table of name and value, in a
column of reading width in the middle. */
.sp-sheet {
inset: auto 0 0 0; width: auto; margin: 0; padding: 0; border: 0; border-top: 3px solid var(--sp-brick);
max-height: 75vh; overflow: auto; background: #fff; color: inherit; box-shadow: 0 -12px 40px rgba(40, 20, 15, .22);
}
.sp-sheet-in { max-width: 44rem; margin: 0 auto; padding: 0 1.25rem; }
.sp-sheet::backdrop { background: rgba(40, 20, 15, .38); }
.sp-sheet-head { display: flex; align-items: center; gap: .7rem; padding: 1rem 0 .8rem; border-bottom: 1px solid var(--sp-rule); }
.sp-sheet-head h2 { font-size: 1.15rem; font-weight: 600; margin-right: auto; }
.sp-sheet-head button { border: 0; background: none; cursor: pointer; font-size: 1.3rem; color: var(--sp-muted); }
.sp-sheet-body { padding: .4rem 0 1.4rem; }
/* The sheet's content is a table of name and value: the name is a label, set like the dt of facts. */
.sp-sheet .table th { font-family: var(--sp-mono); font-size: var(--sp-fs-xs); font-weight: 500; letter-spacing: .08em; text-transform: uppercase; color: var(--sp-muted); white-space: nowrap; vertical-align: top; width: 1%; padding-right: 2rem; }
.sp-sheet .table td { overflow-wrap: anywhere; }
/* A tag that opens a sheet is a button: it keeps the tag's look and says it can
be pressed with the dotted underline links to more detail carry. */
button.tag { border: 0; cursor: pointer; text-decoration: underline dotted; text-underline-offset: 3px; }
+5 -2
View File
@@ -107,7 +107,7 @@ than the default «meaningful step» rule in [§ Commits and release build](#com
2. Git tag `vX.Y.Z` and publish the GitHub Release only on explicit request (see
[§ Release image](#release-image)).
**Planned version cuts** (from pin `2.0.0`; adjust if semver changes mid-track):
**Planned version cuts** (from pin `2.2.0`; adjust if semver changes mid-track):
| Stage | ID | Cut |
|---|---|---|
@@ -124,7 +124,10 @@ than the default «meaningful step» rule in [§ Commits and release build](#com
| 11 | panel-redesign plan: component kit, stages, reorder (docs only, owner-requested cut) | `1.9.4` (PATCH) |
| 12 | GHCR release fix: keep per-arch images, sweep orphaned versions (CI only) | `1.9.5` (PATCH) |
| 13 | panel-redesign | `2.0.0` (MAJOR, no compatibility) |
| 14 | inbound-antispam-panel | `2.1.0` (planned; was `1.10.0`, reordered 2026-10-09) |
| 14 | DMARC ingest drops and bounds, full backup from the panel, no queue in a backup | `2.0.1` (PATCH) |
| 15 | inbound-antispam-panel | `2.1.0` (was `1.10.0`, reordered 2026-10-09) |
| 16 | delivery-log-storage | `2.2.0` |
| 17 | panel-notifications | `2.3.0` |
Docs-only prep (checklists in plans, no product code) uses the same per-step
commit rule but **no** version cut until the next product stage ships.
+154 -16
View File
@@ -33,6 +33,7 @@ changed (see the CHANGELOG, *Removed*).
- [Queue and System log](#queue-and-system-log)
- [Account](#account)
- [Settings](#settings)
- [Notifications](#notifications)
- [Users](#users)
- [Sessions](#sessions)
- [Upgrading](#upgrading)
@@ -183,7 +184,7 @@ cat ./data/setup-token
#### Fixed image tag
`deploy/docker-compose.yml` pins an explicit version (`ghcr.io/mixeme/selfpost:X.Y.Z`),
deliberately never `:latest`. The current pin is `2.0.0`. Intermediate
deliberately never `:latest`. The current pin is `2.2.0`. Intermediate
CHANGELOG sections (`0.2.0`…`0.6.0`) record development cuts from before that
image was published. Pinning matters because of the backup version check (see
[Full backup and restore](#full-backup-and-restore)): the panel binary's
@@ -209,11 +210,13 @@ expected to set; defaults match the code exactly.
| `DMARC_MESSAGE_SIZE_LIMIT` | Maximum report message size in bytes when DMARC ingest is on. | `5242880` (5 MiB) | `.env` |
| `INBOUND_ANTISPAM_MILTER` | Optional milter on the inbound relay listener only (requires `INBOUND_RELAY_ENABLE=true`; not 465/587 or DMARC-only port 25). Empty = off. Format `inet:host:port` or `unix:/path`. Example with [deploy/antispam/docker-compose.antispam.yml](../deploy/antispam/docker-compose.antispam.yml): `inet:antispam:11332`. The panel reads it to report on the *Spam filter* box of the inbound pages and to check that the filter answers (Server › Health). | *(empty)* | `.env` |
| `INBOUND_ANTISPAM_MILTER_ACTION` | What Postfix does if that milter is down: `accept` (fail-open) or `tempfail` (defer). The *Spam filter* box states which. | `accept` | `.env` |
| `INBOUND_SPAM_LOG_RETENTION_DAYS` | How long the inbound journal keeps a row, 7-365 days; it also never holds more than 10 000 rows. Seeds the setting on the first start with the inbound relay on. | `90` | `.env` |
| `INBOUND_JOURNAL_MILTER_SOCKET` | Unix socket of the panel's inbound journal milter, the last milter on port 25 while the inbound relay is on. It records accepted inbound mail and never refuses any; there is no reason to change it. | `/run/selfpost/inbound-journal.sock` | compose |
| `INBOUND_RATE_LIMIT_MESSAGES_PER_IP` | Coarse per-client-IP cap on inbound smtpd (`smtpd_client_message_rate_limit`). Uses the same window as `RATE_LIMIT_WINDOW_SECONDS`. | `20` | `.env` |
| `INBOUND_MESSAGE_SIZE_LIMIT` | Maximum message size in bytes on inbound smtpd (`message_size_limit`). When both inbound relay and DMARC ingest are on, the shared listener uses the **greater** of this and `DMARC_MESSAGE_SIZE_LIMIT`. | `26214400` (25 MiB) | `.env` |
| `RATE_LIMIT_MESSAGES_PER_IP` | Level-1 backstop: maximum messages one client IP may submit per window (Postfix `smtpd_client_message_rate_limit`). See [Rate limiting — level 1](#rate-limiting--level-1-ip-backstop). | `100` | `.env` |
| `RATE_LIMIT_WINDOW_SECONDS` | Level-1 window length in seconds (Postfix `anvil_rate_time_unit`). | `3600` | `.env` |
| `SEND_LOG_RETENTION_DAYS` | Initial default for how many days of send-log history are kept before the background sweep deletes rows — the main driver of `/data` growth over time. After the first panel start, change retention under **Server › Settings** (global role); the env value is only used to seed SQLite when the setting has never been saved. | `90` | `.env` |
| `SEND_LOG_RETENTION_DAYS` | Initial default for how many days of send-log history are kept before the background sweep deletes rows, and with them the `mail.log` lines kept for each message (1–2 KB a message) — the main driver of `/data` growth over time and of the size of a full backup. After the first panel start, change retention under **Server › Settings** (global role); the env value is only used to seed SQLite when the setting has never been saved. | `90` | `.env` |
| `PANEL_SESSION_IDLE_DAYS` | Sliding idle timeout for the panel login session, in days. There is no absolute cap: a user who keeps coming back stays signed in indefinitely. | `7` | `.env` |
| `SELFPOST_DNS_RESOLVERS` | Comma-separated recursive resolvers the panel's PTR/SPF/DKIM/DMARC checks query directly (so they report what the internet sees, not what this host's stub resolver synthesises). | `1.1.1.1:53`, `8.8.8.8:53`, `9.9.9.9:53` when unset | `.env` |
| `TRUSTED_PROXY_CIDR` | Comma-separated CIDRs (bare IPs allowed) of reverse proxies allowed to supply `X-Forwarded-For` for login, setup, and account-change rate-limiting. **Leave unset unless you know the exact address of your reverse proxy.** A wrong value lets a client spoof its rate-limit key by sending a forged `X-Forwarded-For` header — the panel trusts the last hop only when the TCP peer matches one of these CIDRs. Behind the default Apache host-network setup this is typically the Docker bridge gateway, e.g. `172.18.0.1`. | *(empty — XFF ignored)* | `.env` |
@@ -417,10 +420,11 @@ readings:
**Account** (`/account`, every role) is the signed-in user's own: **Profile**
(username and e-mail) and **Password**. Saving the profile does not ask for the
password; changing the password asks for the current one and is rate-limited
like a sign-in. The e-mail is the user's own address. Nothing is sent to it
yet; it is what a domain's *Use my e-mail* button fills in (see [Domain-level
DNS](#domain-level-dns-spf-dkim-dmarc)). Application SASL logins are separate
and are not changed here.
like a sign-in. The e-mail is the user's own address: it is where the panel
sends [notifications](#notifications) while *Send me notifications* is ticked
beside it, and what a domain's *Use my e-mail* button fills in (see
[Domain-level DNS](#domain-level-dns-spf-dkim-dmarc)). Application SASL logins
are separate and are not changed here.
### Settings
@@ -429,7 +433,53 @@ the whole instance: how many days the **Outbound log** keeps its rows before the
background sweep deletes them (7–365 days; takes effect on the next six-hour
prune cycle without a container restart), and, read-only, the level-1 rate
limit that comes from `.env` ([Rate limiting — level 1](#rate-limiting--level-1-ip-backstop)).
There is no panel-wide DMARC report address: each domain has its own.
There is no panel-wide DMARC report address: each domain has its own. The page
also has the switch for [notifications](#notifications) and the two DNS records
that make them arrive.
### Notifications
The panel writes to its users when something needs them. Four things are
reported, each once:
| What | When a message is sent | A second message |
|---|---|---|
| A sending domain's DKIM, SPF or DMARC record stops matching | after two hourly checks in a row found it wrong. A resolver that did not answer is not a mismatch; a missing DMARC record is advice on the domain's page, not a notification | when two checks in a row find the records right again |
| A level-2 rate limit refuses mail | within a minute of the first refusal | at most once a day for the same domain and application, while it goes on |
| The TLS certificate is about to expire | 14 days before | 3 days before, if it is still the same certificate |
| A message has waited in the queue for more than three days | at the next hourly check | when no message that old is left |
A user with the **global** role is told all of it; a user with the **domain**
role is told about the DNS records and rate limits of the outbound domains
assigned to them. A message goes to the e-mail on the user's
[Account](#account) page and only while *Send me notifications* is ticked
there; a user without an e-mail gets none. **Server › Settings** has one switch
that turns notifications off for everyone. With it off nothing is checked, so
what is wrong when it is turned on again is reported then.
Notifications are the panel's own mail. They are sent from
`postmaster@<SELFPOST_HOSTNAME>`, one message per recipient, are not rows of
the Outbound log and count against no rate limit. They name the page of the
panel to open and carry no link: the panel does not know the address its
reverse proxy serves it under.
**Two DNS records make them arrive.** The server's own name is not one of your
sending domains, so it needs records of its own, shown with the values to
publish under **Server › Settings › Notification mail**:
- a **DKIM** key for the hostname, which the panel generates and signs
notifications with (`<selector>._domainkey.<hostname>`, TXT);
- an **SPF** record for the hostname: `v=spf1 a -all` (TXT).
**Server › Health** shows whether each is in place in the *Notification mail*
card. Without them notifications are still sent, and a receiver may file them
as spam or, if the parent domain publishes a strict DMARC policy, refuse them.
Mail of your sending domains depends on neither. If the hostname is itself one
of your sending domains, the records are that domain's and are already
published.
A relay that cannot send cannot say so: none of the four is "the relay is
down". Watch that from outside ([Container health](#container-health)).
### Users
@@ -542,10 +592,14 @@ journal-milter (level 2) is down. There is no per-IP bypass.
### Full backup and restore
**Full backup** is a self-contained project archive: `data/` (SQLite, all
domains' DKIM keys, all applications' SASL credentials, the Postfix queue,
`manifest.json` with the version that created it), plus `docker-compose.yml`,
`.env`, and `certs/` from the operator directory next to `./data`. Delivery
logs under `data/log/` are excluded. The base compose file mounts the project
domains' DKIM keys, all applications' SASL credentials, `manifest.json` with
the version that created it), plus `docker-compose.yml`, `.env`, and `certs/`
from the operator directory next to `./data`. Delivery logs under `data/log/`
are excluded, and so is the Postfix queue: the archive is taken while the
instance runs, and a copy of a queue Postfix is working on can hold a message
twice or not at all. Mail still waiting for delivery is therefore not carried
over, and a restored instance starts with an empty queue; to keep the queue,
stop the instance and use `tar` (methods 3 and 4 below). The base compose file mounts the project
directory read-only at `/selfpost-deploy` so the panel and CLI can read those
deploy files — without that mount, *Full backup* refuses with an error.
@@ -556,12 +610,25 @@ instance keeps running, what goes into the archive, and what happens on restore.
**1. Panel — while the container is running (no downtime).** *Server › Backup*
→ *Full backup*. *Encrypt with a password* is ticked; untick it for a plain
archive. Same archive as the CLI below.
archive. Same archive as the CLI below. The panel runs as its own user, not as
root, so **the private key in `certs/` has to be readable by it**: give the
file to the container's `selfpost` group, which has the fixed id 998 and holds
both the panel and Postfix —
```sh
chgrp 998 certs/privkey.pem && chmod 640 certs/privkey.pem
```
— and have whatever renews the certificate do the same (a certbot deploy hook,
for one). A key readable by root alone is enough for Postfix, but then the
panel refuses to make a backup and names the file: it does not hand over an
archive without the key.
**2. `selfpost-backup` CLI — while the container is running (no downtime).**
The panel button and this command call the same code path: a consistent SQLite
snapshot via `VACUUM INTO`, then a gzip tar of `data/` (minus diagnostics) plus
deploy files.
deploy files. `docker exec` runs the command as root, so it reads the private
key whatever its permissions.
```sh
docker exec <container> selfpost-backup > selfpost-backup.tar.gz
@@ -616,6 +683,8 @@ methods avoid it by stopping first.
| Self-contained archive | Yes | Yes | No — deploy files separate |
| SQLite consistency | `VACUUM INTO` snapshot | Safe when stopped | Safe when stopped |
| `data/log/` | Excluded | Included (optional exclude) | Included (optional exclude) |
| Postfix queue | No | Yes | Yes |
| `certs/` private key | Yes (the panel needs it readable by group 998) | Yes | No — deploy files separate |
| `data/manifest.json` | Written (version stamp) | Absent | Absent |
| Restore version check | Yes — must match image tag | No | No |
| Post-restore *Resync* | Yes, on first boot | No | No |
@@ -691,6 +760,15 @@ needed) for the new hostname or proxy, then `docker compose up -d`. The archive
carries `certs/` from the old host — re-issue certificates when the hostname or
IP changes. Set up the reverse-proxy vhost separately (not in the backup).
The inbound log and the filter lists are in the database and so in the
backup, and so are the list files under `data/antispam/`; the panel writes
those files again from the database on the first start after a restore. What
the backup does not hold is the spam filter's own side: the `antispam/`
directory with the compose fragment and the rspamd configuration. Like the
reverse-proxy fragments it is part of the installation, not of its state -
copy it from [deploy/antispam/](../deploy/antispam/) again, or the restored
instance runs with no filter.
**Restoring an encrypted (`.spbk`) backup** needs a running container to
decrypt it first — any container with the `selfpost-backup` CLI works; decryption
does not read `/data` and performs no version check. Start one normally
@@ -822,6 +900,58 @@ only sees SelfPost. Default action is fail-open (`accept`) so a down sidecar
does not block backup-MX; set `INBOUND_ANTISPAM_MILTER_ACTION=tempfail` to
defer instead.
The fragment mounts one file into the sidecar,
[deploy/antispam/rspamd/milter_headers.conf](../deploy/antispam/rspamd/milter_headers.conf):
it makes rspamd write its verdict - the action and the score with its symbols
- into two headers of every message it lets through. SelfPost records them
with the message, and the upstream receives them with the mail. Copy the
`antispam/rspamd/` directory next to the fragment; without the file accepted
mail is still recorded, with no verdict.
The same file makes rspamd **sign** each message, in a third header,
`X-SelfPost-Filter`. The filter is attached so that mail still flows when it
is down, and then a sender could write a verdict of his own into those
headers; SelfPost therefore records a verdict only next to a signature it can
recompute. The key is `data/antispam/filter.key`: SelfPost creates it on its
first start with the inbound relay on, and the sidecar reads it from the
directory it already mounts - there is nothing to configure. If Inbound › Log
shows *selfpost* for mail the filter did judge, the sidecar is running without
this file: copy `antispam/rspamd/` again and restart the sidecar. Nothing of
rspamd's needs clearing, then or after a change of the lists or of the key: it
rereads the list files by itself and reads the key anew for every message. A
restart is needed only for a change of its own configuration - the files of
`antispam/rspamd/`.
The same directory holds
[multimap.conf](../deploy/antispam/rspamd/multimap.conf), the rules for the
**allow and deny lists** kept in the panel. SelfPost does not apply the lists
itself: it writes them as six files under `data/antispam/` (`allow_ip.map`,
`allow_from.map`, `allow_domain.map` and the three `deny_` ones), the fragment
mounts that directory into the sidecar, and rspamd rereads a file within
seconds of a change - no restart. An entry is a sender address, a sender
domain (that domain, not its subdomains) or a client address or network. A
denied sender is refused outright. An allowed one is given a weight that no
spam score outweighs, so it is accepted - unless a deny entry matches the same
message: **deny wins**, whichever entries they are. **A sender address or
domain on the allow list is not proof of anything**: it is the envelope
sender, which the sending server states and nobody checks, so whoever knows an
allowed address can send as it. Allow a client address or network where you
can. The files are readable by
anyone on the host; they hold addresses, not secrets.
In the panel the lists are **Inbound › Filter lists** (global role), and what
port 25 decided about each message is **Inbound › Log**: accepted mail with the
filter's verdict, refused mail with the reply it was given. A row of the log
leads to the lists with its sender filled in. Beside each entry of a list is
how often it decided a message, counted from the log - so over the days the
log keeps - and *never* marks an entry that may have outlived its use. The
count depends on two things in
[multimap.conf](../deploy/antispam/rspamd/multimap.conf): the message of the
deny rules and the names of the allow symbols; change them and every entry
reads *never*. A user with the *domain* role
sees the log of the inbound domains assigned to them, and not the lists. How
long the log keeps a row is set under **Server › Settings**.
The panel only reports this setting; it does not change it. Every inbound
domain page has a **Spam filter** box — on or off, what happens to mail while
the filter is down, and that the filter belongs to the server, not to the
@@ -891,7 +1021,13 @@ sees the *Ingest* box — whether ingest works, the last report received, report
kept this week and parse failures. Open a domain's roll-up from the list or
from *DMARC reports* in the domain page's menu. A user with the *domain* role
sees the reports of the domains assigned to them, without the *Ingest* box.
Summaries are pruned (500 kept, 90 days max).
Summaries are pruned (500 kept for each domain, 90 days max).
A message that is not a report the relay keeps — unreadable as an aggregate
report, about a domain not sent from here, or delivered to another domain's
report address — is dropped without a retry and without a bounce. It adds one
to *parse failures*, and the reason is on the message's delivery line in the mail log
(*Server › System log*, `dropped: …`).
**Not an open relay.** The inbound smtpd offers no SASL. With DMARC ingest
alone, `check_recipient_access` permits only configured report addresses;
@@ -1056,8 +1192,10 @@ backoff cap, queue lifetime), the same numbers the Outbound queue shows; users
with the *domain* role see them here because they cannot open the queue. Under
both sit the `mail.log` lines for its queue id: the connection to the
receiving server, the server's reply, and the status that reply was filed
as. Rows outlive `mail.log`, so an older message's lines may have rotated
away; the page says so. Retention is set under **Server › Settings** (global
as. The lines are kept in the database with the row, so they stay on the
page after `mail.log` has rotated and are deleted together with the row; a
message accepted before `2.2.0` shows them only while they are still in the
current file, and the page says so when there are none. Retention is set under **Server › Settings** (global
role); `SEND_LOG_RETENTION_DAYS` in `.env` is only the initial default until it
is changed there.
+184
View File
@@ -0,0 +1,184 @@
# Plan: delivery-log-storage
**Status:** shipped in `2.2.0` (2026-10-12); agreed the same day (owner: option 1 of the roadmap item, "store the lines")
**Date:** 2026-10-12
**Version:** `2.2.0` MINOR (one DDL migration; a 2.1.0 data directory boots unchanged).
---
## Goal
A message's page (`/outbound/log/{id}`, mockup `out-message`) shows what
Postfix wrote about the message for as long as its row is in the send log.
Today the *Delivery log* box is filled by a search of the current `mail.log`
on every page view, so it is empty for anything sent before the last daily
rotation, while the row itself lives for ninety days.
## Context (as-built)
- The log tailer ([internal/logtail](../../internal/logtail/logtail.go))
already reads every `mail.log` line once, from a persisted offset, to
advance delivery statuses.
- `logtail.QueueLines` searches the last 4 MB of the current file for one
queue id; `Handlers.deliveryLog` calls it per page view.
- A message with several recipients is several send-log rows under one queue
id; the page of each shows every line of the message, as the search does
today.
## Decisions
### What is stored
A `mail.log` line whose head is a queue id - after the timestamp, the host
name and `program[pid]:` - goes into a new table with that id, the line's own
time (`lineTime`, the same reading the inbound journal uses) and the line as
written, cut to 4096 bytes on a character boundary. The head is matched from
the start of the line: the tail is the far end's words. `NOQUEUE` and
`warning` are not queue ids - the pattern takes Postfix's short form
(hexadecimal) and its long form (no vowels) and nothing else.
The tailer does not ask whether the id belongs to a send-log row before it
writes. The first lines of a message (`client=`, `message-id=`) are logged
before the journal milter writes the row at end-of-message, so such a check
would drop exactly them.
**Mail that came in on port 25 is left out** (security review, 2026-10-12). A
queue id that `postfix/inbound/smtpd` opened is a stranger's message; it never
gets a send-log row, and what a stranger can make Postfix log - his own
`message-id`, up to the cut - is not stored to be pruned later. The tailer
remembers such ids (4096 at most) until qmgr's `removed` or until another
listener opens a message under the id. What is still stored without a row is
little: the bounces Postfix generates, and port-25 mail met in mid-flight
after a restart.
Lines are written in one transaction per pass over the log, 500 at most
(technical review): a commit per line is a disk sync per line on the one
connection the journal milter writes through.
### Which lines are a row's
A line is shown on a row's page when the queue id is the same **and** the
line's time lies between one hour before the row was written and the queue
lifetime (the larger of `maximal_queue_lifetime` and `bounce_queue_lifetime`,
as `postconf` reports them) plus a day after it. Postfix reuses a short queue
id once the message has left the queue - the id is the microsecond of arrival
and the inode of the queue file - so a row that lives ninety days can meet a
later message under its id. The window does not rule that out, it limits it
to the days the row's own message could still be in the queue; this is the
bound the roadmap item asked for. The search of the current file that the
page did before had the same exposure for a day. `enable_long_queue_ids`
would end it and is a change to the mail path, not part of this item.
### Retention
Lines live as long as a send-log row with their queue id does. One rule covers
both ends: a line older than an hour whose queue id has no send-log row is
deleted. That removes the lines of a row the retention sweep has just deleted,
and the lines of messages that never had a row. It runs at start and after
every send-log sweep, six-hourly. It asks the send log once per queue id; at a
million lines that is still a scan of the table, of the order of a second,
during which the one connection is taken.
There is no cap of its own: with port-25 mail left out, the table is bounded
by what authenticated applications send, which level 1 and level 2 limit.
### After an unclean stop
The tailer saves its place every few seconds, so after a crash it reads some
lines twice. While it is catching up at start a line is written only if the
same line is not there already - the rule the inbound journal follows.
### What is not kept
- Lines the tailer never reads are not stored: the tail of a file that was
rotated away while the panel was down, as with delivery statuses
(architecture.md § Log tailer).
- A message in the queue across the upgrade to 2.2.0 has only its later lines
stored, and the page then shows those alone: the search of the file runs
only for a row with nothing stored. The only installation is the owner's
test server; no merge of the two sources is built for one moment of it.
### The page
The page reads the table: the latest 200 lines of the row, oldest first. When
the table has none, it still searches the current `mail.log` as before - that
is what shows a message accepted before 2.2.0 while its lines are in today's
file. Lines in rotated files are not imported: the only installation is the
owner's test server.
Nothing in the page's layout changes; the `out-message` mockup already draws
the box as a `log_pane`. Two texts change, mockup first: the *Outbound log*
help topic (it says an old message may show no lines) and the note shown when
there are none.
### Size
Six to ten lines of about 200 bytes per message: 1-2 KB next to a row of a few
hundred bytes. The table becomes the larger part of what the send log costs in
`/data` and in a backup; the retention setting bounds both together.
## Schema (migration `0003_send_log_lines.sql`)
```sql
CREATE TABLE send_log_lines (
id INTEGER PRIMARY KEY AUTOINCREMENT,
queue_id TEXT NOT NULL,
logged_at TEXT NOT NULL,
line TEXT NOT NULL
);
CREATE INDEX idx_send_log_lines_queue_id ON send_log_lines (queue_id, logged_at);
```
No foreign key: a line is written before its row exists, and one queue id is
several rows.
## Scope
In: the table, the tailer writing and pruning it, the page reading it, the two
texts, docs.
Out: the inbound journal (it decides for itself, roadmap § delivery-log-storage);
importing rotated files; `enable_long_queue_ids`; any change to `mail.log`
rotation or to the Server › Mail log page.
## Security
- The lines carry the far end's text (replies, HELO names). They are shown
today from the file through the same escaping; storing them changes where
they are read from, not what is shown or to whom. A domain administrator
sees the lines of messages of their own domains, as now.
- They now enter the database and so a full backup, which `log/` does not:
submitting clients' addresses and SASL logins, message ids, receivers'
replies. [security.md](../security.md) says so.
- A line is matched from its head, so a reply cannot make itself a line of
another message; Postfix does not log raw line breaks, and `postlogd` is the
file's only writer.
- A stranger on port 25 does not write to the table (above).
## Done when
A message sent before the last rotation shows its lines on its page; the lines
disappear with the row; the page's wording no longer blames rotation for an
ordinary case.
## Model routing (this plan)
| Role | Model |
|------|-------|
| All implementation (code, tests, docs) | **Opus** |
| Technical review (reviewer ≠ author: a separate read-only agent) | **Opus** |
| Security review | **Fable** |
## Implementation checklist
Target version cut: **`2.2.0`** (MINOR). One commit per step;
[development.md](../development.md) § Plan checklists.
- [x] Migration `0003_send_log_lines.sql`, store methods (insert, insert once, lines of a row, prune) and their tests — **Opus**
- [x] Log tailer: store queue-id lines, write once while catching up, prune with the send-log sweep — **Opus** (hourly as first built; six-hourly after the reviews)
- [x] The two texts, mockup first: the *Outbound log* help topic and the empty note (`design:` commit) — **Opus** (the help topic is in the mockups; the empty note is not drawn there - the `out-message` mockup shows a message with lines - and changes with the page)
- [x] Message page reads the table, falls back to the current `mail.log`; help text in `help.go` — **Opus**
- [x] [architecture.md](../architecture.md), [guide.md](../guide.md), [schema-migrations.md](../schema-migrations.md) — **Opus** (schema-migrations.md went in with the migration)
- [x] `go vet`, `go test`, e2e: a message's page still lists its lines after a forced rotation — **Opus** (on the test server: vet ok, 22 packages ok, e2e ok with the new step `delivery_log_survives_rotation`)
- [x] Technical review — **Opus** (2026-10-12; no Critical or High. Fixed: one transaction per pass instead of a commit per line, the prune asks the send log once per queue id and runs six-hourly, the cut keeps characters whole, tests for what the comments claimed. Left as found and written down under *What is not kept*: lines the tailer never read, and a message in the queue across the upgrade)
- [x] Security review — **Fable** (2026-10-12; no Critical or High. Fixed: port-25 mail is not stored, so a stranger cannot grow the table; [security.md](../security.md) states what a backup now carries. The window against queue-id reuse is described as the limit it is)
+90 -21
View File
@@ -1,6 +1,6 @@
# Plan: inbound-antispam-panel
**Status:** agreed
**Status:** shipped in `2.1.0` (2026-10-12)
**Date:** 2026-08-19
**Version:** `2.1.0` MINOR (opt-in; no change to outbound-only or inbound-without-filter paths). Was `1.10.0`; on 2026-10-09 the owner put the panel redesign first, so this ships on the 2.0 baseline and its screens are built from the redesign's component kit ([panel-redesign.md](panel-redesign.md) § Component kit).
@@ -34,8 +34,8 @@ Lists let the operator correct false positives/negatives (e.g. always allow
## Quarantine
**Not in this plan.** Held-mail review/release is tracked separately as roadmap
candidate [inbound-quarantine](inbound-quarantine.md). The journal may still
**Not in this plan.** Held-mail review/release is an idea kept outside the roadmap,
[inbound-quarantine](inbound-quarantine.md). The journal may still
record a `quarantine` **decision label** when the external filter reports it,
without SelfPost storing the message.
@@ -57,8 +57,14 @@ smtpd_milters = { antispam, inbound-journal }
```
Antispam runs first; inbound-journal runs at end-of-message on the **accept**
path and reads rspamd-added headers (`X-Rspamd-Action`, `X-Rspamd-Score`, symbol
summary when present).
path and reads rspamd-added headers (`X-Rspamd-Action`, and `X-Spamd-Result`
for the score and symbols - rspamd has no `X-Rspamd-Score`; the sidecar adds
both through `deploy/antispam/rspamd/milter_headers.conf`). As built they are
read only next to rspamd's signature of the message, `X-SelfPost-Filter`
(owner, 2026-10-12: "А rspamd не может дописывать какой-то заголовок, по
которому мы будем знать, что это заголовки нашего спам фильтра?") - an HMAC
of the queue id, since a fixed mark could be written by a sender as easily as
the verdict itself.
**Antispam rejects** do not reach end-of-message on the journal milter. v1 also
tails `mail.log` for inbound `postfix/smtpd` **`milter-reject`** / **`reject`**
@@ -73,7 +79,9 @@ antispam hook). Without an external filter:
- **accept** rows — message relayed (engine `selfpost`, detail empty);
- **reject** / **tempfail** rows — from the mail.log tailer (Postfix policy,
size limit, unknown recipient, etc.).
unknown recipient, a sender domain that does not exist. As built, a refusal
whose log line names no recipient - the size limit answered at MAIL FROM -
and the per-client rate limit, which Postfix logs as a warning, make no row).
Panel copy explains that rspamd symbols appear only when the antispam milter is
configured.
@@ -83,6 +91,25 @@ configured.
**Global administrator only** for journal and lists — same as inbound relay. No
domain-admin access in v1.
*As built (2026-10-11):* the lists are the global role's; the **log follows
the delegation of Inbound** that 2.0 introduced after this was written - a
user with the `domain` role sees the rows of the inbound domains assigned to
them and nothing else, without the *Allow / Deny sender* links. That is what
the accepted `in-log` mockup draws (no "global role" in its head). The owner
was told and can turn it back into a global-only page.
**Inbound › Log accepted by the owner on 2026-10-12** ("Этот экран
утверждаем"), from screenshots of the page built from its fixture, after his
changes to it: the sender column is headed *From*; *Why* is not a column but
a sheet over the page, opened by the decision tag, holding a centred table of
who decided, the verdict and symbols or the reply, and the subject whole
(kept on the sheet at his word: the table clips it); a message Postfix itself
refused offers no *Deny sender*. Inbound › Filter lists was redrawn at his word
the same day - kind in a column of its own, the note on the entry's sheet,
matches counted (§ Filter list usage) - and **accepted by him on 2026-10-12**
("Да"), after a deny entry he added in the panel refused a test message with
the list's reply and the journal recorded it.
### Retention
Mirror send-log retention:
@@ -101,13 +128,53 @@ Mirror send-log retention:
| Entry types | sender address, sender domain (`@domain` suffix), client IP / CIDR |
| Precedence | **deny overrides allow** |
| Scope | instance-wide (not per inbound domain in v1) |
| Where applied | **rspamd map files** under `/data/antispam/` (allow.map, deny.map); documented compose fragment mounts them into the sidecar |
| Where applied | **rspamd map files** under `/data/antispam/`, one per list and entry type (`allow_ip.map`, `allow_from.map`, `allow_domain.map`, `deny_…`) - an IP map and a sender map are different map types to rspamd; documented compose fragment mounts them into the sidecar. Deny is a prefilter reject, allow a weight of -100 (as built: two prefilters do not give "deny overrides allow") |
| SelfPost pre-check | **out of v1** — lists live in rspamd only |
| Edit access | global administrator only |
| When inactive | lists editable in the panel but UI warns that sync applies only when `INBOUND_ANTISPAM_MILTER` is set; `postfix reload` / rspamd reload documented in guide |
List change → atomic map rewrite (same pattern as Postfix maps) → rspamd reload
via documented operator step or sidecar `SIGHUP` in the compose fragment.
List change → atomic map rewrite (same pattern as Postfix maps). No reload:
rspamd rereads a changed map file by itself (about ten seconds on 3.11).
### Filter list usage (owner, 2026-10-12)
Each entry shows how often it decided a message (*Matched*, or *never*) and,
on its sheet, when it last did - so that entries nothing matches any more can
be removed. **The count is read off the journal on every view of the page;
nothing is stored for it.** A journal row counts for an entry when the list
decided the message - a rejected row whose reply is the deny rule's message,
an accepted row whose verdict carries the allow rule's symbol
(`SELFPOST_ALLOW_IP` / `_FROM` / `_DOMAIN`, which the journal milter always
keeps) - and the entry fits the row's sender or client address the way rspamd
matches it (`inbound.Lists.Usage`).
What follows from counting this way, and is accepted:
- the count reaches back as far as the journal does: its retention (90 days by
default), which is what the sheet says;
- a row counts for every fitting entry of the deciding list, since the journal
does not say which of two the filter met first;
- a message refused at end-of-message is one row whatever its recipient count,
so a count is of messages, roughly.
**When to move to a counter kept with each entry** (a migration adding hits
and last-hit to `inbound_spam_list`, bumped where the journal row is written;
the owner asked for the condition to be written down). Any one of:
1. **The journal is cut by its 10 000-row cap and not by age.** Then it holds
fewer days than its retention, the sheet's "in the last 90 days" is no
longer true, and *never* stops being evidence that an entry is stale. The
sign: the log tailer's prune line keeps reporting rows removed while the
oldest row of Inbound › Log is younger than the retention.
2. **The retention is set short** (towards its minimum of 7 days) for reasons
of its own, privacy or disk. A week without a match does not make an entry
stale.
3. **Counts are wanted that outlive the journal** - since the entry was added.
4. **The page is slow.** A view reads every journal row a list decided and
tries it against every entry. With the cap that is at most 10 000 rows
times the number of entries; past a few hundred entries, measure it.
Until then no migration is spent on it.
### Activation
@@ -200,7 +267,9 @@ flowchart LR
## Security
- Same validation whitelists as inbound relay ([security.md](../security.md)).
- Journal rows contain PII — retention and global-admin-only access.
- Journal rows contain PII — retention, and access by reach: the global role
reads all of it, a `domain` user the rows of assigned inbound domains (as
built; see § RBAC).
- Inbound journal milter fail-open must not block mail (backup-MX role).
- List map writes injection-safe; deny wins over allow.
@@ -243,14 +312,14 @@ Target version cut: **`2.1.0`** (MINOR), after `2.0.0`. One commit per step;
[development.md](../development.md) § Plan checklists.
- [x] Agree journal fields, retention, and RBAC (this plan § Decisions) — **Composer**
- [ ] Migration `0002_inbound_spam_log.sql` on the 2.0 baseline — **Composer**
- [ ] Inbound journal-milter + Postfix wiring — **Composer**
- [ ] mail.log tailer: inbound reject rows — **Composer**
- [ ] List CRUD + validation + atomic rspamd map sync — **Composer**
- [ ] Panel: Inbound › Log and Inbound › Filter lists from the kit partials, mockups `in-log` / `in-filter-lists` (add client IP and engine columns mockup-first), evidence pairs — **Composer**
- [ ] Backup includes `/data/antispam/` — **Composer**
- [ ] Unit + handler tests — **Composer**
- [ ] [guide.md](../guide.md), [architecture.md](../architecture.md) — **Composer**
- [ ] `go vet`, `go test`, e2e if applicable — **Composer**
- [ ] Technical review of inbound antispam changes — **Opus**
- [ ] Security review inbound antispam path — **Fable**
- [x] Migration `0002_inbound_spam_log.sql` on the 2.0 baseline — **Composer**
- [x] Inbound journal-milter + Postfix wiring — **Composer**
- [x] mail.log tailer: inbound reject rows — **Composer**
- [x] List CRUD + validation + atomic rspamd map sync — **Composer**
- [x] Panel: Inbound › Log and Inbound › Filter lists from the kit partials, mockups `in-log` / `in-filter-lists` (add client IP and engine columns mockup-first), evidence pairs — **Composer**
- [x] Backup includes `/data/antispam/` — **Composer** (it always did: a backup walks all of `/data`; now held by a test, and the guide says the `antispam/` fragment directory is not in it)
- [x] Unit + handler tests — **Composer** (written with each step: store, milter, tailer, lists and their usage, view fixtures held to the outlines, handlers for both roles)
- [x] [guide.md](../guide.md), [architecture.md](../architecture.md) — **Composer** (kept up with each step; [security.md](../security.md) follows the security review)
- [x] `go vet`, `go test`, e2e if applicable — **Composer** (on the test server for every step; last on `7192c38`: vet ok, 22 packages ok, e2e ok, which relays, refuses, reads the log page and edits a list through the panel)
- [x] Technical review of inbound antispam changes — **Opus** (2026-10-12; findings fixed: client address of a connection's later messages, the lists' writer under a lock, the tailer's queue-id memory, hold rows written twice, the cap by time; and, outside the feature, the level-2 limit and client-address list skipped after a connection's first message)
- [x] Security review inbound antispam path — **Fable** (2026-10-12; no Critical or High; HELO-forged journal rows and the six-hourly row cap fixed; a sender's own verdict while the filter is down closed by the filter's signature, the owner's idea; accepted risks recorded in [security.md](../security.md))
+1 -1
View File
@@ -1,6 +1,6 @@
# Plan: inbound-quarantine
**Status:** candidate — design TBD; needs explicit agreement before any code
**Status:** an idea to think about, not on the roadmap (owner, 2026-10-12) — design TBD; needs explicit agreement before any code
**Version:** TBD (`1.x` MINOR if opt-in and backward-compatible; may warrant `2.x`
if it changes inbound storage or backup contract).
+119
View File
@@ -0,0 +1,119 @@
# Plan: panel-notifications
**Status:** agreed 2026-10-12 - the design questions were closed with the owner and he took it into work ("Делаем")
**Date:** 2026-10-12
**Version:** `2.3.0` MINOR (one migration; opt-out by a switch).
---
## Goal
The panel writes to its administrators when something needs them, instead of
waiting to be opened.
## Decisions (owner, 2026-10-12)
### Events of the first version
Four, each of them something the panel can already tell and a person has to
act on:
| Event | Who is told |
|---|---|
| A sending domain's DNS record stopped matching (SPF, DKIM, DMARC) | global users, and `domain` users the domain is assigned to |
| A level-2 rate limit refused mail | the same, for the domain of the limit |
| The TLS certificate is about to expire | global users |
| A message has been in the queue for more than **three days** | global users |
Three days, not one (owner: "one day is not that much"): Postfix keeps a
message five, so two are left to act before it bounces. The threshold is a
constant, not a setting.
Left for later: a process or milter socket down (a relay in that state often
cannot send), an inbound upstream unreachable, DMARC failures appearing
(noisy). Mail held in quarantine is not an event: quarantine is an idea
outside the roadmap.
### Sender and path
From `postmaster@<SELFPOST_HOSTNAME>`, submitted locally. Not from one of the
sending domains (owner): with several domains the choice is arbitrary, and a
message about the certificate has no business coming from a domain.
These messages have to arrive (owner: "we need the best deliverability for
them"), so the server's own name gets what a sending domain has:
- a **DKIM key for the hostname**, generated by the panel, with which
notifications are signed - no domain has to be chosen for it. An SPF record
alone would pass DMARC, but not after the user's mailbox forwards the
message on, and large receivers treat unsigned mail more strictly;
- an **SPF record** for the hostname, `v=spf1 a -all`.
The records themselves are under **Server › Settings** (owner: "the DNS
records go to Settings, Health has only the status"), in a box of their own
drawn with the record component of a domain's DNS page: host, value to
publish, copy buttons, state. **Server › Health** has a card of its own,
*Notification mail*, with the state of each record and a link to Settings.
A DMARC record for the hostname is not asked for: the name inherits its
parent domain's policy, and with SPF and DKIM aligned it passes any.
Without the records notifications are still sent, and the card warns.
Notifications are the panel's own mail: they are not rows of the Outbound log
and count against no level-2 limit.
A relay that cannot send cannot say so; none of the four events is "the relay
is down", and that is why.
### Who chooses what
- One switch for the instance under **Server › Settings**.
- One checkbox per user under **Account**, "send me notifications" - on when
the profile has an e-mail. No choice per event: four events, none of them
noise. Per-event choice comes when there are more and some are.
- A user without an e-mail gets nothing.
### When a message is sent
A background check once an hour for DNS, certificate and queue.
| Event | Sent | "Cleared" message |
|---|---|---|
| DNS mismatch | after two checks in a row found it (it has held for an hour). A resolver that did not answer is not a mismatch | after two good checks in a row |
| Certificate | 14 days before expiry, and again 3 days before if it is still the same certificate | none |
| Level-2 limit refused mail | at the first refusal for a domain or application, then at most once a day for the same limit | none |
| Queue older than three days | when such a message appears | one, when none is left |
Event state is kept in the database, so a restart announces nothing a second
time.
## To settle while building
- How the panel submits: `sendmail` as the unprivileged panel user, and what
that needs of the image.
- Whether a notification carries a link to the panel - there is no setting
for the panel's own address.
- Where the refusal of a level-2 limit is noticed: in the journal milter as
it refuses, or from the `rejected` rows of the send log on the hourly check.
- Schema: the user's checkbox, the instance switch, the event state table.
## Screens
Not drawn. The Notifications box was taken out of the `account` and
`settings` mockups on 2026-10-10 (owner: 2.0 ships no box for a feature that
is not built). Drawing them again - one checkbox on Account, one switch on
Server › Settings, the SPF check on Server › Health - is the first step, shown
to the owner before any code ([panel-redesign.md](panel-redesign.md)
§ Enforcement).
## Implementation checklist
- [x] Mockups: Account checkbox, Settings switch and records, Health status card, help topic; accepted by the owner 2026-10-12 (`design:` commit)
- [x] Migration and store: user flag, instance switch, event state
- [x] Sending: local submission from `postmaster@<hostname>`, message text (`sendmail` as the panel user; signed through `non_smtpd_milters` with a key of the server's own name; no link in the text - the panel does not know its own address)
- [x] Background checker: DNS, certificate, queue; the rules of *When a message is sent*
- [x] Level-2 refusal event (from the `rejected` rows of the send log, looked at every minute)
- [x] Panel: Account, Settings, Health card, help
- [x] guide.md, architecture.md, security.md, schema-migrations.md
- [x] `go vet`, `go test`, e2e: a level-2 refusal reaches a mailbox as a signed notification (on the test server: vet ok, 23 packages ok, e2e ok; "once" and "clears" are held by the unit tests of `internal/notify`)
- [ ] Technical review
- [ ] Security review
+4 -4
View File
@@ -235,9 +235,9 @@ the first step of that feature's plan — not invented during implementation.
| Roadmap item | Where it lands | Built from | Drawn |
|---|---|---|---|
| inbound-antispam-panel (agreed, `2.1.0`) | journal = **Inbound › Log** (add client IP and engine to the mockup's columns; `quarantine` joins the decisions); lists = new **Inbound › Filter lists**, global-only because the lists are instance-wide | list layout: add form + two tables, `status_tag`, filter form in `box_head`; the log links *Allow / Deny sender* to the lists | `in-log`, `in-filter-lists` |
| inbound-quarantine (candidate) | new **Inbound › Quarantine**, list + detail with *Release* / *Discard* | list layout; detail like `out-message` with `facts`, `log_pane` for headers only, Release beside a `sp-danger-zone` Discard | `in-quarantine`, `in-quarantine-message` — the open questions of its plan (where mail lives, what release means, retention, RBAC) stay open; the mockup assumes SelfPost-held mail, release to upstream, 14 days, delegated like the rest of Inbound |
| inbound-quarantine (an idea, off the roadmap since 2026-10-12) | new **Inbound › Quarantine**, list + detail with *Release* / *Discard* | list layout; detail like `out-message` with `facts`, `log_pane` for headers only, Release beside a `sp-danger-zone` Discard | `in-quarantine`, `in-quarantine-message` — the open questions of its plan (where mail lives, what release means, retention, RBAC) stay open; the mockup assumes SelfPost-held mail, release to upstream, 14 days, delegated like the rest of Inbound |
| preflight (candidate) | new **Server › Preflight**, next to Health: Health is what runs now, Preflight is the deeper on-demand installation check with a test e-mail form | `postmark` verdict, a table of checks with `status_tag` and a *What to do* column, one form box for the test e-mail | `preflight` |
| panel-notifications (candidate) | per-user event choice on **Account**; one switch under **Server › Settings** | checkboxes in a box, `help` | **no** — drawn once, taken out of `account` and `settings` on 2026-10-10 (owner): 2.0 ships no box for a feature that is not built; redrawn as the first step of its plan |
| panel-notifications (candidate) | one checkbox per user on **Account** (no choice per event, owner 2026-10-12); one switch under **Server › Settings** | a checkbox in a box, `help` | **no** — drawn once, taken out of `account` and `settings` on 2026-10-10 (owner): 2.0 ships no box for a feature that is not built; redrawn as the first step of its plan |
| password-reset (candidate) | a link on sign-in, two signed-out screens (request, set new password) | the signed-out layout | **no** |
| delivery-log-storage (candidate) | the *Delivery log* box of **Outbound › Log › message**, filled from a table instead of a grep of today's `mail.log` | `log_pane`, unchanged | `out-message` |
| csrf-tokens | no screen; every form gains a hidden field — forms are rendered through one helper so the token cannot be forgotten | — | n/a |
@@ -539,5 +539,5 @@ only when both are done. Every stage-2 step ends with the evidence of
order suits development, preflight, delivery-log-storage, panel-notifications,
password-reset (two screens to draw first), queue as a table with the inbound
queue, csrf-tokens
(one hidden field in the form partial once security.md is settled),
inbound-quarantine after its open questions.
(one hidden field in the form partial once security.md is settled).
inbound-quarantine is an idea outside the roadmap.
+90 -52
View File
@@ -30,24 +30,28 @@ in `git log` and [CHANGELOG.md](../CHANGELOG.md).
|---|---|---|---|---|
| preflight | Installation check page (`/preflight`) | candidate | — | [plans/preflight.md](plans/preflight.md) |
| contributing | `CONTRIBUTING.md` | candidate | — | — |
| inbound-antispam-panel | Inbound antispam journal + allow/deny lists | agreed | 1/12 | [plans/inbound-antispam-panel.md](plans/inbound-antispam-panel.md) |
| inbound-quarantine | Inbound spam quarantine (hold / review / release) | candidate | — | [plans/inbound-quarantine.md](plans/inbound-quarantine.md) |
| inbound-antispam-panel | Inbound antispam journal + allow/deny lists | shipped `2.1.0` | 12/12 | [plans/inbound-antispam-panel.md](plans/inbound-antispam-panel.md) |
| csrf-tokens | Per-session CSRF tokens on state-changing forms | candidate | — | — |
| template-data-typing | Typed structs for template data instead of `map[string]any` | candidate | — | — |
| structured-logging | `log/slog` with levels and fields | candidate | — | — |
| review-2026-08-followups | Remaining findings of the 2026-08-19 code review | candidate | — | — |
| panel-notifications | E-mail notifications about important events | candidate | — | — |
| delivery-log-storage | Keep a message's `mail.log` lines with its send-log row | candidate | — | — |
| panel-notifications | E-mail notifications about important events | in progress | 8/10 | [plans/panel-notifications.md](plans/panel-notifications.md) |
| delivery-log-storage | Keep a message's `mail.log` lines with its send-log row | shipped `2.2.0` | 8/8 | [plans/delivery-log-storage.md](plans/delivery-log-storage.md) |
| password-reset | Password reset by e-mail | candidate | — | — |
| phone-desktop-mode | A phone browser in "desktop site" mode gets the phone navigation | candidate | — | — |
| panel-redesign | Panel redesign (accepted mockups, design contract) | shipped `2.0.0` | 27/27 | [plans/panel-redesign.md](plans/panel-redesign.md) |
| schema-squash | Squash SQLite migrations into a 2.x baseline | built in panel-redesign (stage 1) | — | [plans/panel-redesign.md](plans/panel-redesign.md) |
**Recommended order** (not binding; owner, 2026-10-09: feature order follows
development convenience): **panel-redesign** shipped as `2.0.0` (2026-10-11); the next stage is
**inbound-antispam-panel** (`2.1.0`), built in the new component kit —
styling its two screens once instead of twice. panel-docs shipped in
development convenience): **panel-redesign** shipped as `2.0.0` (2026-10-11) and
**inbound-antispam-panel** as `2.1.0` (2026-10-12), built in the new component
kit — styling its two screens once instead of twice. **delivery-log-storage**
shipped as `2.2.0` the same day. Agreed after it: **panel-notifications**
(`2.3.0`); every other remaining
item is a candidate. panel-docs shipped in
[CHANGELOG.md](../CHANGELOG.md) `[1.8.0]`; dmarc-reports in `[1.7.0]`
(security review of the ingest path pending); domain-stats-auto-ratelimit in
(its ingest path first ran in `2.0.0` and was reviewed on 2026-10-11, see
[security.md](security.md)); domain-stats-auto-ratelimit in
`[1.6.0]`; send-log-retention in `[1.5.0]`; inbound-relay in `[1.4.0]`;
queue-retries in `[1.3.1]`; the 2026-08-13 full-tree review follow-ups are in
`[1.3.0]`. Candidates need explicit agreement before they join the queue.
@@ -121,8 +125,8 @@ action) plus editable allow/deny lists that tune the external filter — without
shipping rspamd or another engine inside the image.
**Boundary:** metadata journal and list sync only; SelfPost does not run the
filter ([product.md](product.md)). Quarantine:
[inbound-quarantine](#inbound-quarantine). Design closed in
filter ([product.md](product.md)). Quarantine is not on the roadmap
([Ideas](#ideas-not-on-the-roadmap)). Design closed in
[plans/inbound-antispam-panel.md](plans/inbound-antispam-panel.md) § Decisions.
**Done when:** with inbound relay enabled, the operator sees journal rows
@@ -133,28 +137,7 @@ hook is set.
**Dependencies / risks:** inbound relay `[1.4.0]`; rspamd map/header coupling;
journal PII; global-admin-only RBAC.
**Version:** `2.1.0` MINOR (was `1.10.0`; reordered after the redesign on 2026-10-09); **agreed**.
---
## inbound-quarantine
**Goal:** optional hold for suspicious inbound mail — review in the panel, then
release to upstream, discard, or tune lists. Design not settled.
**Boundary:** expands inbound from pure relay toward short-lived message
storage; may conflict with "no mailboxes" in [product.md](product.md) until
product explicitly agrees. Alternatives (rspamd-only quarantine vs SelfPost
store) are listed in [plans/inbound-quarantine.md](plans/inbound-quarantine.md).
**Done when:** TBD after open questions in the plan are answered.
**Dependencies / risks:** inbound relay `[1.4.0]`; storage/retention/backup
size; malware in held MIME; overlap with
[inbound-antispam-panel](#inbound-antispam-panel). No implementation checklist
yet.
**Version:** TBD; `candidate` until explicitly agreed.
**Version:** shipped `2.1.0` (2026-10-12), MINOR (was `1.10.0`; reordered after the redesign on 2026-10-09).
---
@@ -348,38 +331,43 @@ time to final status plus the queue lifetime), not to the id alone; the table
grows with the send log (small — a few hundred bytes per message) and is
covered by the same retention.
**Version:** `2.x` MINOR; `candidate`. Fits after the redesign, built against
the kit's `log_pane`.
**Version:** shipped `2.2.0` (2026-10-12), MINOR; option 1. Decisions and
checklist: [plans/delivery-log-storage.md](plans/delivery-log-storage.md).
---
## panel-notifications
**Goal:** the panel tells its administrators when something needs them instead
of waiting to be opened: a DNS record stops matching, a domain or application
hits its rate limit and mail is refused, an inbound upstream is unreachable,
DMARC failures appear, mail is held in quarantine; for the global
administrator also the TLS certificate nearing expiry, a process or milter
socket down, a queue older than a day.
of waiting to be opened. First version, four events: a sending domain's DNS
record stops matching, a level-2 rate limit refuses mail, the TLS certificate
is about to expire, a message has been in the queue for more than three days.
**Boundary:** e-mail only, sent through the relay itself from
`postmaster@<hostname>`. Each user picks events under **Account**; a domain
administrator hears only about assigned domains. One message when a problem
starts and one when it clears — no streams, no digests in v1. One instance
switch under **Server › Settings**. No webhooks, no chat integrations.
**Boundary:** e-mail only, from `postmaster@<hostname>`, submitted locally -
not from one of the sending domains. One checkbox per user under **Account**
(no choice per event), one instance switch under **Server › Settings**; a
domain administrator hears only about assigned domains. No streams, no
digests, no webhooks, no chat integrations. Not in the first version: a
process or milter down, an inbound upstream unreachable, DMARC failures.
**Done when:** an event from the list reaches the account e-mail of every user
who chose it and may see its subject, once on start and once on clear; a user
without an e-mail gets nothing; the switch turns it all off.
**Done when:** each of the four events reaches the account e-mail of every
user who has notifications on and may see its subject, by the rules of the
plan (a "cleared" message for DNS and the queue only); a user without an
e-mail gets nothing; the switch turns it all off.
**Dependencies / risks:** the account e-mail (2.0 baseline, see
[plans/panel-redesign.md](plans/panel-redesign.md) § Account e-mail); a relay
that cannot send cannot report that it cannot send — server events need the
direct-to-MX path or stay panel-only; event state must survive restarts so a
reboot does not re-announce everything. Screens are drawn: `account`,
`settings`.
that cannot send cannot report that it cannot send - none of the four events
is "the relay is down"; mail from the bare hostname needs an SPF record for
it to be believed, which Server › Health will check; event state must survive
restarts so a reboot does not re-announce everything. The screens are not
drawn: the Notifications box was taken out of `account` and `settings` on
2026-10-10 and is redrawn as the first step.
**Version:** `2.x` MINOR; `candidate`.
**Version:** `2.3.0` MINOR; agreed 2026-10-12 - the design questions were closed
with the owner on 2026-10-12
([plans/panel-notifications.md](plans/panel-notifications.md)) and he took it into
work the same day.
---
@@ -405,6 +393,41 @@ goes and today Account saves it without the password.
---
## phone-desktop-mode
**Problem (owner, 2026-10-11):** the panel opened on a phone with the
browser's "desktop site" switched on does not become the desktop panel.
Seen on Server › Health at `selfpost.mixfed.ru`: the page body is laid out in
columns (Machine and Processes side by side, three cards in the next row),
while the navigation is the phone one - not a bar but a stack with every
section open (Outbound, Inbound, Server, the user menu), filling the first
screen before the page starts. The content also sits flush against the left
edge of the screen.
**Goal:** find out why, then decide what the panel should be in that mode -
the desktop panel, or a phone panel that is at least consistent.
**Where to look first (a guess, not checked):** "desktop site" on Android
lays a page out about 980 px wide. Bulma turns the navigation into a bar at
1024 px and puts columns side by side from 769 px, so 980 px falls between
the two: desktop columns under a touch navigation. `panel.css` has its own
breakpoints at 768, 1023 and 1407 px. Check the width the phone really
reports, why every dropdown is open rather than folded behind the burger,
and where the left padding went.
**Boundary:** the navigation and page gutter between 769 and 1023 px, which
is also a tablet in portrait and a narrow desktop window - the same defect is
expected there. A design change is mockup-first
([plans/panel-redesign.md](plans/panel-redesign.md) § Enforcement).
**Done when:** the cause is written down here, and the panel at that width
has one coherent layout, checked on a phone in "desktop site" mode and in a
980 px window.
**Version:** `2.x` PATCH or MINOR; `candidate`.
---
## panel-redesign
**Goal:** a panel an operator opens a few times a year and still finds their
@@ -482,3 +505,18 @@ explicit major). Squashing the current short chain is not a reason to cut 2.x on
its own. A missed gate leaves a mid-chain 1.x database silently stuck.
**Version:** `2.x` major only; not a 1.x item.
---
## Ideas, not on the roadmap
Thoughts kept so they are not lost. They are not candidates: nothing here is
waiting for an OK, has a version or takes a place in the order above. An idea
joins the index only when the owner says so.
- **Inbound quarantine** (moved here by the owner on 2026-10-12: "an idea to
think about") - hold suspicious inbound mail, review it in the panel, then
release it to the upstream or discard it. It would take the relay from
passing mail on to storing it for a while, against "no mailboxes" in
[product.md](product.md). Notes and open questions:
[plans/inbound-quarantine.md](plans/inbound-quarantine.md); the screens
`in-quarantine` and `in-quarantine-message` are drawn and stay unbuilt.
+19 -10
View File
@@ -17,8 +17,8 @@ When you add or review a migration, update this file in the same change.
| Field | Value |
|---|---|
| Chain head | `user_version = 1` |
| Files | `0001_init.sql` (the 2.0 baseline) |
| Chain head | `user_version = 4` |
| Files | `0001_init.sql` (the 2.0 baseline), `0002_inbound_spam_log.sql`, `0003_send_log_lines.sql`, `0004_notifications.sql` |
| File mark | `PRAGMA application_id = 0x53503230` ("SP20") |
| Database file | `/data/selfpost.db` (bind mount) |
| Compatibility | 2.x MINOR releases must boot a `2.0.0` data directory. **Nothing is carried over from 1.x.** |
@@ -60,10 +60,11 @@ What that means in practice:
**Implication:** once `2.0.0` is released, `0001_init.sql` is never edited
again and no file is deleted, renamed or reordered — every 2.x database has
already run it. Until that release the baseline is still being written:
later steps of the redesign and the features queued behind it (the next one
is `0002_inbound_spam_log.sql` of inbound-antispam-panel, `2.1.0`) add files
after it.
already run it. `2.0.0` and `2.0.1` shipped with the baseline alone; the
features after them add files, the first being `0002_inbound_spam_log.sql` of
inbound-antispam-panel (`2.1.0`), then `0003_send_log_lines.sql` of
delivery-log-storage (`2.2.0`) and `0004_notifications.sql` of
panel-notifications (`2.3.0`).
---
@@ -72,18 +73,21 @@ after it.
| Ver | File | Shipped | Kind | Summary |
|-----|------|---------|------|---------|
| 1 | `0001_init.sql` | 2.0.0 | DDL | The whole 2.0 schema, below |
| 2 | `0002_inbound_spam_log.sql` | 2.1.0 | DDL | `inbound_spam_log`, `inbound_spam_list` and two indexes; nothing existing changes |
| 3 | `0003_send_log_lines.sql` | 2.2.0 | DDL | `send_log_lines` and one index; nothing existing changes |
| 4 | `0004_notifications.sql` | 2.3.0 | DDL | `users.notify` (default 1: every user starts with notifications on) and `notify_state` |
**Kind:** *DDL* — schema only; *data* — `INSERT`/`UPDATE` that must stay correct
for operators upgrading from older 2.x images.
---
## Schema at head (v1)
## Schema at head (v2)
| Table | Role |
|-------|------|
| `settings` | Key/value settings of the instance — what is true whoever is signed in (send-log retention). Nothing that belongs to a user |
| `users` | Panel logins. `role` is `global` or `domain`; `email` is the user's own address; `all_domains` / `all_inbound_domains` widen a `domain` user to every domain of that direction |
| `users` | Panel logins. `role` is `global` or `domain`; `email` is the user's own address; `notify` is the user's own switch for notification e-mails (v4); `all_domains` / `all_inbound_domains` widen a `domain` user to every domain of that direction |
| `sessions` | Panel login sessions (token hash, owning user by id and deleted with them, sliding idle expiry) |
| `domains` | Sending (outbound) domains. `dmarc_rua` is where aggregate reports go (`''` = no reports) |
| `user_domains` | Outbound domains assigned to a `domain` user |
@@ -98,9 +102,14 @@ for operators upgrading from older 2.x images.
| `user_inbound_domains` | Inbound domains assigned to a `domain` user — granted separately from the outbound ones |
| `dmarc_reports` | Parsed aggregate report summaries |
| `dmarc_report_records` | Per-source rows inside a report |
| `inbound_spam_log` | Inbound antispam journal: one row per recipient of a message offered on port 25, with the decision (`accept`, `reject`, `tempfail`, `quarantine`) and who made it (`rspamd`, `postfix`, `selfpost`). Metadata only (v2) |
| `inbound_spam_list` | Allow / deny entries for the external filter (`address`, `domain`, `ip`), instance-wide; unique per list (v2) |
| `notify_state` | What the panel has already written to its users about, one row per subject (`dns:<domain>`, `cert`, `queue`, `limit:<domain>:<login>`), so that each thing is said once and a restart says nothing again (v4) |
| `send_log_lines` | The `mail.log` lines Postfix wrote under a queue id, with the line's own time; what a message's page shows once the log has rotated. No reference to `send_log`: a line belongs to a row by queue id and time, and goes when no row carries its id (v3) |
`TestBaselineSchema` (`internal/store/schema_test.go`) pins every table, column
and index of this list.
and index of this list; `TestOpenMigratesA20Database` opens a database that
holds the baseline alone.
### What changed against the last 1.x schema
@@ -129,7 +138,7 @@ for the domain. Nothing refers to a user: deleting one changes no domain.
1. Add `internal/store/migrations/00NN_short_name.sql` (next number).
2. Update the table in [Migration chain](#migration-chain), the snapshot at the
top, and the affected rows of [Schema at head](#schema-at-head-v1).
top, and the affected rows of [Schema at head](#schema-at-head-v2).
3. Update `TestBaselineSchema` if tables, columns or indexes change, and add a
test that opens a database at the previous version and migrates it.
4. A *data* migration states in a comment which 2.x versions it is written for.
+140 -5
View File
@@ -26,7 +26,31 @@ the user form did not end that user's sessions, internal errors reached forms a
(see [Authentication and sessions](#authentication-and-sessions) and [Errors
shown to users](#errors-shown-to-users)); two deliberate behaviours are
recorded under Accepted risks (upstream of a delegated inbound domain, profile
saved without the password).
saved without the password). The 2026-10-11 review (Fable) covered the DMARC
report path from port 25 to the report pages, which ran for the first time in
2.0.0 - the 2026-08-19 pass read code that Postfix could not yet start: no
Critical or High finding; a report's size was bounded only as bytes (now 5 000
records, 255 bytes a field), the retention cap was one for the whole server
(now per domain), the delivery command could run a hundred at once (now one),
and the reason logged for a dropped message carried the sender's bytes as they
came (now cut and reduced to printable ASCII). The 2026-10-12 reviews of
inbound-antispam-panel (`2.1.0`: the inbound journal milter, refusals read from
`mail.log`, the filter lists and their pages) were two, by different models
over the same diff. The technical one (Opus) found, outside the feature and
there since `1.9.0`, that **the level-2 rate limit and an application's
client-address list held only for the first message of a connection** - see
[Mail path](#mail-path-security-relevant) - which was reproduced on the test
stand and fixed. The security one (Fable) found no Critical or High: a client's
HELO name could supply the sender, recipient and reply of a journal row and so
make a refusal look like the deny list's (the reply now ends at Postfix's own
`; from=<`); the journal's row cap was applied only every six hours, between
which refusals were written at whatever rate a stranger sent recipients (the
tailer now cuts to the cap every 500 rows); and two behaviours are recorded
under Accepted risks (the allow list keyed on an address anyone can claim, the
list files readable by anyone). A third it named - with the filter down, a
sender's own verdict headers were recorded as the filter's - was closed the
same day at the owner's suggestion: the filter signs each message and a
verdict is read only next to its signature.
(2) **Accepted risks** —
deliberate departures beyond the mandatory, recorded so the decision is not
lost.
@@ -113,6 +137,8 @@ assigned outbound domains and inbound domains **separately**, each list with
| Outbound: domain page, applications, Domain settings (report address, rate limit, export), log | every domain | assigned outbound domains |
| Outbound › DMARC reports | every report, with the server's ingest statistics | reports of assigned domains, without the ingest box |
| Inbound: domain page, upstream, recipients | every inbound domain | assigned inbound domains |
| Inbound › Log | every row, including those of a domain deleted since | rows of assigned inbound domains (by name), without the links into the filter lists |
| Inbound › Filter lists; the inbound log's retention | yes | no — 404 |
| Add or delete a domain, either direction; import a domain | yes | no |
| Account, Help | yes | yes |
@@ -171,6 +197,41 @@ operator can act on it.
refuse unknown addresses at RCPT (no backscatter). OpenDKIM is not attached
on inbound. An optional antispam milter is inbound-only; default action is
fail-open (`accept`).
- **The panel's milters judge every message of a connection.** Postfix
introduces a client to a milter once per connection, and the milter library
replaces the milter's session after any reply that is not "continue". Until
`2.1.0` the journal milter knew the client's address only from that
introduction, and both of its checks passed a message that came without one:
a second `MAIL FROM` after a refused one, or any message after the first on
one connection, got past the level-2 limit and past an application's
client-address list. Now the address comes with every `MAIL FROM`
(`milter_mail_macros` carries `{client_addr}`), the level-2 limit does not
ask for it at all, an application with a client-address list is refused when
no address is known, and neither milter replies anything but "continue" to a
message it lets through. `test/e2e` sends two `MAIL FROM` on one connection.
- **The inbound journal is written from what strangers send.** The inbound
journal milter is the last on port 25, read-only and attached fail-open; it
keeps of the filter's verdict only what its patterns recognise. **A verdict
is read only next to the filter's signature.** The filter is attached so
that mail flows when it is down, and then nothing removes verdict headers a
sender wrote himself. So rspamd adds a third header, `X-SelfPost-Filter`:
the HMAC-SHA256 of the message's queue id under a key SelfPost generates
(`data/antispam/filter.key`) and the sidecar reads from the directory it
mounts. The journal milter recomputes it at end-of-message; with no
signature, or one that does not fit, the row is that of mail nobody judged.
The signature goes on to the upstream with the mail and is of no use there:
it fits one queue id. Refusals are
parsed from `mail.log` from the head of the line, with the reply ending at
Postfix's own `; from=<` so that the HELO name after it - the client's text,
logged as written - cannot stand in for the sender, the recipient or the
reply; an address with a quote, a bracket or a space in it is not kept.
Only a recipient at an inbound domain makes a row. The journal is cut to
10 000 rows every 500 rows the tailer writes, and by age every six hours.
- **Filter lists** reach another container as files: values are whitelisted on
the way in (address, domain, canonical IP or network; no `/0`, no
IPv4-mapped network), the writer refuses a stored value with whitespace, a
newline or `#`, and a change of the table and the rewrite of the six files
are one step under a lock.
- TLS is mandatory before credentials are transmitted (465 wrapper / 587
`encrypt`).
- `TRUSTED_PROXY_CIDR` — only explicitly trusted proxies may supply
@@ -183,6 +244,11 @@ operator can act on it.
private key from `certs/` when present; a domain export carries the DKIM key
and **working** application passwords in the clear (otherwise a transfer
without recreating credentials would be impossible).
- Since 2.2.0 a full backup also carries, in the database, what Postfix logged
about each message still in the send log (`send_log_lines`): the submitting
client's address and SASL login, the message id, the receiving server and
its reply. `mail.log` itself stays out of a backup; these lines are state
the panel shows, kept for the send log's retention and no longer.
- Both downloads are encrypted with a password unless the box on the form is
unticked (it is ticked by default):
scrypt (N=2¹⁵, r=8, p=1) → AES-256-GCM, streamed in 64 KiB chunks, each
@@ -274,20 +340,60 @@ deferred item from the roadmap.
reporters are arbitrary third parties and the mailbox address is published in
DNS. The ingest already constrains the meaningful part — a report cannot be
filed under a domain this instance does not host, and the claimed domain is
cross-checked against the `+tag` of the delivery address — and the data is
advisory statistics only; nothing enforces off it. The trigger to revisit is
cross-checked against the `+tag` of the delivery address (effective since
2.0.0: before it the recipient never reached the command) — and the data is
advisory statistics only; nothing enforces off it. What a stranger can do is
limited to a domain that chose a SelfPost-hosted address, through that
address: file reports under it, replace one whose reporter and id they know,
and fill that domain's 500 — not another domain's. A message that is not a
report the relay keeps is dropped, not bounced (the envelope sender is the
author's to write); a genuine report the parser cannot read is lost the same
way, with a count and a line in the mail log. The trigger to revisit is
any feature that lets report contents drive an automatic action (policy
changes, rate limiting, alerting an operator into a config change).
- **The application auth-IP allow-list is enforced fail-open.** The check runs
in the journal milter (`internal/milter/authip.go`) with
`default_action=accept` and skips on store errors, so if the milter is down
the restriction is not applied; the SASL login itself always succeeds. This
the restriction is not applied; the SASL login itself always succeeds.
(While the milter is up the list holds for every message, and an unknown
client address is refused - see Mail path; what is accepted here is the
milter being down.) This
is documented in [guide.md](guide.md) as **not an authentication boundary** —
it is a blast-radius limiter for a leaked application password, on the same
fail-open path as level-2 rate limiting, and a milter outage already stops
the journal from recording sends at all. The trigger to revisit is a
requirement for the allow-list to hold as a security boundary — at which
point it must move to the SASL/smtpd layer and fail closed.
- **The key the spam filter signs with is readable by anyone on the host.**
The inbound journal reads the filter's verdict only next to the filter's
signature (see Mail path), and the key for it, `data/antispam/filter.key`,
is 0644 like the list files beside it: the filter is another user in another
container. Someone with a shell on the host could therefore sign a verdict
of his own - into a monitoring page, on a host where he could do far more.
Senders on the internet, whom the signature is for, cannot read it. Trigger
to revisit: the same as for the list files - a shared host.
- **An allow entry for a sender address or domain trusts what the sending
server says.** The filter matches the envelope sender, which is not
authenticated: anyone who knows or guesses an allowed address can send as it
and is given the entry's weight, which no spam score outweighs. The page's
Help and the guide say so and recommend an entry for the client's address or
network instead. Deny entries have no such problem - claiming to be a denied
sender only gets mail refused - and deny wins over allow. Trigger to
revisit: allow entries being used as the main way to receive mail from a
sender, at which point the rule should also require the sender's SPF or
DKIM to pass.
- **The filter lists are files readable by anyone on the host.**
`/data/antispam` is 0755 and its six files 0644, unlike the rest of `/data`:
the filter runs under another user in another container. They hold sender
addresses, domains and networks - who the operator blocks and whom he lets
through - but no message data. The inbound journal itself is in the
database, 0640. Trigger to revisit: a shared host, where the sidecar should
instead be given the files through a group it shares with the panel.
- **A row of the inbound log belongs to a domain by name.** A `domain` user
sees the rows whose inbound domain has the name of one assigned to them, so
an inbound domain that is deleted and added again shows its earlier rows to
whoever it is assigned to next. Accepted: the journal is of a mail domain,
and the rows age out with the retention.
- **Level-2 in-flight reservations can outlive a dropped SMTP session until TTL.**
The journal milter cannot observe connection-close directly through
go-milter, so if a client disconnects without an explicit `Abort`, its
@@ -303,7 +409,36 @@ deferred item from the roadmap.
the file is inaccessible to others. The log holds envelope addresses and
client IPs, but neither message bodies nor headers; it is excluded from
backups (`log/` is skipped) so that a dump stays state rather than
diagnostics.
diagnostics. Since 2.2.0 the tailer copies the lines of queued outbound
messages into the database for the message's page, and those are in a
backup (§ Backup and domain export); mail offered on port 25 is not copied.
- **The panel submits mail of its own.** Notifications leave through
`sendmail(1)` run by the unprivileged panel user: a fixed argument list, the
two addresses after `--`, no shell. The recipient is a panel user's account
e-mail, validated when it was saved and checked again before it reaches a
header or the command line; the subject and body are composed by the panel
from its own readings. Two of those readings can carry text from outside - a
DNS check's finding, and a queued message's sender address - and both are
reduced to printable ASCII and cut before they are printed, so a
notification's body can contain an address a stranger chose, as a few plain
characters and nothing else. The panel's
membership in `postdrop`, described in the image as read-side, is now what
lets it submit; any local user could already do that through the
world-executable `sendmail`. Notifications are signed with a key of the
server's own name and bypass the level-1 and level-2 limits: they are a few
messages a day at most, to addresses only a panel user can set.
- **A message's page can show another message's log lines after a queue-id
collision.** Postfix's short queue ids repeat: the id is the microsecond of
arrival and the queue file's inode, and inodes come round within seconds.
The page takes the stored lines of the row's queue id dated within the
row's own queue lifetime (about six days), so a second message has to get
the same microsecond on the same inode in that span - about one in a
million per reuse of the inode, and not something a sender can aim. When it
happens, a `domain` user could read the log lines of a message that is not
theirs. The search of the current `mail.log` that the page did before
2.2.0 had the same exposure over a day. Trigger to revisit: a report of
foreign lines on a page, or a reason to turn on `enable_long_queue_ids`,
which ends it.
- **A `domain` user may point an inbound domain's upstream at any host and
port.** The upstream form checks that the host is a well-formed name or IP
and the port is 1–65535; loopback, private and link-local addresses are
+84 -10
View File
@@ -1,16 +1,16 @@
// Package backup implements SelfPost's full-server backup and the restore
// version guard (architecture.md § Persistence). A full backup is a
// gzip-compressed tar of the consolidated persistent state under /data — the
// SQLite database (as a consistent snapshot), the per-domain DKIM keys, the
// SASL database, and the Postfix queue — plus docker-compose.yml, .env, and
// certs/ from the operator's deploy directory, and a manifest recording the
// SelfPost version that produced it. Postfix delivery logs under log/ are
// excluded (diagnostics, not state).
// SQLite database (as a consistent snapshot), the per-domain DKIM keys and the
// SASL database — plus docker-compose.yml, .env, and certs/ from the
// operator's deploy directory, and a manifest recording the SelfPost version
// that produced it. Postfix delivery logs under log/ are excluded
// (diagnostics, not state), and so is the Postfix queue (see QueueDir).
//
// Restore is not a separate code path in the panel: a backup is extracted into
// the operator's project directory (data/, docker-compose.yml, .env, certs/)
// before first start. The archive carries everything needed to bring the
// instance back — DKIM keys, sasldb2, sender map, queue, and deploy files —
// instance back — DKIM keys, sasldb2, sender map, and deploy files —
// so the operator only adjusts hostname or proxy settings on a new host. The
// restore-specific steps the panel runs are CheckRestore, which refuses to
// boot if the manifest's version does not match the running binary so
@@ -28,6 +28,7 @@ import (
"compress/gzip"
"database/sql"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
@@ -81,6 +82,18 @@ type Params struct {
OnWarn func(string)
}
// QueueDir is the Postfix queue, relative to the data directory. No backup
// holds it (owner, 2026-10-11). An archive is made while the instance runs,
// and Postfix moves messages between the queue's directories all the time: a
// copy taken then can hold a message twice or not at all, so what it would
// bring back is not the queue. Nor could the panel read it - Postfix keeps
// every queue file readable by its own user alone, which is why the panel's
// backup failed outright from 1.3.0 to 2.0.0. What is not carried over is mail
// still waiting for delivery; a start without a queue directory builds an
// empty one. Whoever needs the queue stops the instance and archives the
// project directory (guide § Taking a full backup).
const QueueDir = "postfix/queue"
// excludedFromArchive lists the data-directory entries a backup never carries.
// The live database files are replaced by a consistent VACUUM INTO snapshot
// written under the canonical name; the setup token is transient bootstrap
@@ -145,7 +158,7 @@ func Create(w io.Writer, p Params) error {
return err
}
if err := addTree(tw, p.DataDir, DataArchivePrefix, excludedFromArchive); err != nil {
if err := addTree(tw, p.DataDir, DataArchivePrefix, excludedFromArchive, skipQueue); err != nil {
return err
}
@@ -181,6 +194,52 @@ func validateDeployRoot(deployRoot string) error {
return nil
}
// skipFunc says whether the entry at path (rel to the root being walked) is
// left out of the archive. A directory left out is not descended into.
type skipFunc func(path, rel string, d fs.DirEntry) bool
// skipQueue leaves the Postfix queue out of every archive.
func skipQueue(_, rel string, _ fs.DirEntry) bool {
return filepath.ToSlash(rel) == QueueDir
}
// Check reads through everything Create would put into the archive, without
// writing one, and returns the first thing that cannot be read. The panel
// streams its archive to the browser, so once Create has begun a failure can
// only cut the download short; this is run first, while a refusal can still be
// shown as a page.
//
// The panel is not root, and nothing is left out on that account: its archive
// is the one selfpost-backup makes, or it is not made. What it most likely
// cannot read is the private key under certs/ - see UnreadableCert.
func Check(p Params) error {
if p.DataDir == "" || p.DBPath == "" {
return fmt.Errorf("backup: DataDir and DBPath are required")
}
if err := ValidateDeployRoot(p.DeployRoot); err != nil {
return err
}
if err := addTree(nil, p.DataDir, DataArchivePrefix, excludedFromArchive, skipQueue); err != nil {
return err
}
return addDeployFiles(nil, p.DeployRoot, nil)
}
// UnreadableCert reports whether err, from Check or Create, is a file under
// the deploy root's certs/ that the caller is not allowed to read, and names
// it relative to the deploy root ("certs/privkey.pem").
func UnreadableCert(err error, deployRoot string) (name string, ok bool) {
var pathErr *fs.PathError
if !errors.As(err, &pathErr) || !errors.Is(err, fs.ErrPermission) {
return "", false
}
rel, relErr := filepath.Rel(filepath.Join(deployRoot, CertsDirName), pathErr.Path)
if relErr != nil || rel == ".." || strings.HasPrefix(filepath.ToSlash(rel), "../") {
return "", false
}
return CertsDirName + "/" + filepath.ToSlash(rel), true
}
func addDeployFiles(tw *tar.Writer, deployRoot string, onWarn func(string)) error {
for _, name := range []string{ComposeFileName, EnvFileName} {
src := filepath.Join(deployRoot, name)
@@ -206,14 +265,16 @@ func addDeployFiles(tw *tar.Writer, deployRoot string, onWarn func(string)) erro
}
return fmt.Errorf("backup: stat %s: %w", CertsDirName, err)
}
return addTree(tw, certsDir, CertsDirName+"/", nil)
return addTree(tw, certsDir, CertsDirName+"/", nil, nil)
}
// addTree walks root and adds every regular file (and directory, to preserve
// empty ones and modes) to tw under archivePrefix + path relative to root.
// When exclude is non-nil, top-level names relative to root are skipped.
// When exclude is non-nil, top-level names relative to root are skipped; skip,
// when non-nil, leaves out whatever it says. With a nil tw nothing is written:
// every file is opened and closed, which is how Check finds what cannot be read.
// Non-regular, non-directory entries (symlinks, sockets) are skipped.
func addTree(tw *tar.Writer, root, archivePrefix string, exclude map[string]bool) error {
func addTree(tw *tar.Writer, root, archivePrefix string, exclude map[string]bool, skip skipFunc) error {
return filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
@@ -236,12 +297,22 @@ func addTree(tw *tar.Writer, root, archivePrefix string, exclude map[string]bool
}
}
if skip != nil && skip(path, rel, d) {
if d.IsDir() {
return fs.SkipDir
}
return nil
}
info, err := d.Info()
if err != nil {
return err
}
switch {
case d.IsDir():
if tw == nil {
return nil
}
hdr := &tar.Header{
Typeflag: tar.TypeDir,
Name: name + "/",
@@ -282,6 +353,9 @@ func writeTarFile(tw *tar.Writer, name string, mode fs.FileMode, srcPath string)
return fmt.Errorf("backup: open %s: %w", srcPath, err)
}
defer f.Close()
if tw == nil {
return nil
}
info, err := f.Stat()
if err != nil {
+75 -1
View File
@@ -5,7 +5,9 @@ import (
"bytes"
"compress/gzip"
"encoding/json"
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
"strings"
@@ -40,6 +42,7 @@ func seedDataDir(t *testing.T) (dataDir, dbPath, deployRoot string) {
writeFile(t, filepath.Join(dataDir, "opendkim", "keys", "example.com", "selfpost.private"), "PRIVATE KEY")
writeFile(t, filepath.Join(dataDir, "sasl", "sasldb2"), "SASLDB")
writeFile(t, filepath.Join(dataDir, "postfix", "sender_login_maps"), "@example.com login")
writeFile(t, filepath.Join(dataDir, "antispam", "deny_domain.map"), "evil.example\n")
writeFile(t, filepath.Join(dataDir, "postfix", "queue", "deferred", "sample"), "queue-file")
// Transient files that must NOT be archived.
writeFile(t, filepath.Join(dataDir, "setup-token"), "secret-token")
@@ -112,7 +115,10 @@ func TestCreateIncludesStateExcludesTransient(t *testing.T) {
DataArchivePrefix + "opendkim/keys/example.com/selfpost.private",
DataArchivePrefix + "sasl/sasldb2",
DataArchivePrefix + "postfix/sender_login_maps",
DataArchivePrefix + "postfix/queue/deferred/sample",
// The filter lists as the spam filter reads them. They are rewritten
// from the database on the first start after a restore; carrying them
// means the filter has its lists before the panel is up.
DataArchivePrefix + "antispam/deny_domain.map",
ComposeFileName,
EnvFileName,
CertsDirName + "/fullchain.pem",
@@ -129,6 +135,7 @@ func TestCreateIncludesStateExcludesTransient(t *testing.T) {
DataArchivePrefix + "selfpost.db-shm",
DataArchivePrefix + "log/mail.log",
DataArchivePrefix + "log/mail.log.1",
DataArchivePrefix + "postfix/queue/deferred/sample",
} {
if _, ok := files[name]; ok {
t.Errorf("archive should not contain %s", name)
@@ -235,3 +242,70 @@ func TestCheckRestoreWrongFormatRejected(t *testing.T) {
t.Error("CheckRestore wrong format reported a restore")
}
}
// No archive holds the Postfix queue; the rest is there, the private key
// included, and Check agrees that it can be made.
func TestCreateLeavesTheQueueOut(t *testing.T) {
dataDir, dbPath, deployRoot := seedDataDir(t)
p := Params{DataDir: dataDir, DBPath: dbPath, Version: "1.2.3", DeployRoot: deployRoot}
if err := Check(p); err != nil {
t.Fatalf("Check: %v", err)
}
var buf bytes.Buffer
if err := Create(&buf, p); err != nil {
t.Fatalf("Create: %v", err)
}
files := readArchive(t, buf.Bytes())
for name := range files {
if strings.HasPrefix(name, DataArchivePrefix+QueueDir) {
t.Errorf("the panel's archive holds %s", name)
}
}
for _, name := range []string{
DataArchivePrefix + "selfpost.db",
DataArchivePrefix + "opendkim/keys/example.com/selfpost.private",
DataArchivePrefix + "sasl/sasldb2",
DataArchivePrefix + "postfix/sender_login_maps",
ComposeFileName, EnvFileName,
CertsDirName + "/fullchain.pem",
CertsDirName + "/privkey.pem",
} {
if _, ok := files[name]; !ok {
t.Errorf("archive missing %s", name)
}
}
}
// A file under certs/ that may not be read is told apart from any other
// failure, so the panel can say what to do about it.
func TestUnreadableCert(t *testing.T) {
root := filepath.Join("srv", "deploy")
denied := func(path string) error {
return fmt.Errorf("backup: open %s: %w", path, &fs.PathError{Op: "open", Path: path, Err: fs.ErrPermission})
}
if name, ok := UnreadableCert(denied(filepath.Join(root, "certs", "privkey.pem")), root); !ok || name != "certs/privkey.pem" {
t.Errorf("certs/privkey.pem: got %q, %v", name, ok)
}
if name, ok := UnreadableCert(denied(filepath.Join(root, ".env")), root); ok {
t.Errorf(".env was taken for a certificate file: %q", name)
}
if name, ok := UnreadableCert(denied(filepath.Join("data", "postfix", "x")), root); ok {
t.Errorf("a data file was taken for a certificate file: %q", name)
}
missing := &fs.PathError{Op: "open", Path: filepath.Join(root, "certs", "privkey.pem"), Err: fs.ErrNotExist}
if _, ok := UnreadableCert(missing, root); ok {
t.Error("a missing file was taken for one that may not be read")
}
}
// Check finds what Create would stumble over, before anything is written.
func TestCheckReportsWhatCannotBeRead(t *testing.T) {
dataDir, dbPath, deployRoot := seedDataDir(t)
if err := os.Remove(filepath.Join(deployRoot, EnvFileName)); err != nil {
t.Fatal(err)
}
if err := Check(Params{DataDir: dataDir, DBPath: dbPath, Version: "1.2.3", DeployRoot: deployRoot}); err == nil {
t.Fatal("Check passed with .env missing from the deploy root")
}
}
+109
View File
@@ -0,0 +1,109 @@
package dmarc
import (
"errors"
"strings"
"testing"
"time"
"github.com/mixeme/selfpost/internal/store"
)
func reportXML(orgName, domain, records string) string {
return `<?xml version="1.0"?><feedback><report_metadata><org_name>` + orgName +
`</org_name><email>a@example.net</email><report_id>1</report_id>` +
`<date_range><begin>1723593600</begin><end>1723680000</end></date_range></report_metadata>` +
`<policy_published><domain>` + domain + `</domain><p>none</p></policy_published>` + records + `</feedback>`
}
const oneRecord = `<record><row><source_ip>203.0.113.10</source_ip><count>3</count>` +
`<policy_evaluated><disposition>none</disposition><dkim>pass</dkim><spf>pass</spf></policy_evaluated></row>` +
`<identifiers><header_from>example.com</header_from></identifiers></record>`
// The XML is a stranger's: what it may make the relay store is bounded.
func TestParseAggregateBoundsWhatAReportHolds(t *testing.T) {
long := strings.Repeat("é", 4000)
got, err := ParseAggregate([]byte(reportXML(long, "example.com", oneRecord)))
if err != nil {
t.Fatal(err)
}
if len(got.Reporter) > maxFieldLen || !strings.HasPrefix(long, got.Reporter) || got.Reporter == "" {
t.Fatalf("reporter name is %d bytes, want the start of the name in at most %d", len(got.Reporter), maxFieldLen)
}
if strings.ContainsRune(got.Reporter, '�') {
t.Fatal("the name was cut in the middle of a character")
}
huge := `<record><row><source_ip>203.0.113.10</source_ip><count>9223372036854775807</count>` +
`<policy_evaluated><dkim>pass</dkim></policy_evaluated></row></record>`
got, err = ParseAggregate([]byte(reportXML("r", "example.com", huge+huge)))
if err != nil {
t.Fatal(err)
}
if got.PassCount != 2*maxRecordCount {
t.Fatalf("pass count = %d, want two records clamped to %d each", got.PassCount, maxRecordCount)
}
if _, err := ParseAggregate([]byte(reportXML("r", "example.com", strings.Repeat(oneRecord, maxReportRecords)))); err != nil {
t.Fatalf("a report of %d records was refused: %v", maxReportRecords, err)
}
if _, err := ParseAggregate([]byte(reportXML("r", "example.com", strings.Repeat(oneRecord, maxReportRecords+1)))); err == nil {
t.Fatalf("a report of %d records was accepted", maxReportRecords+1)
}
}
// A report over the bound is one the relay does not keep: dropped, not retried.
func TestIngestMessageRejectsAReportWithTooManyRecords(t *testing.T) {
st, err := store.Open(t.TempDir() + "/test.db")
if err != nil {
t.Fatal(err)
}
defer st.Close()
if _, err := st.AddDomain("example.com", "selfpost"); err != nil {
t.Fatal(err)
}
xml := reportXML("r", "example.com", strings.Repeat(oneRecord, maxReportRecords+1))
err = IngestMessage(st, strings.NewReader(gzipMessage(t, xml)), "dmarc-reports+example.com@mail.example.com", time.Now().UTC())
if !errors.Is(err, ErrRejected) {
t.Fatalf("err = %v, want ErrRejected", err)
}
}
// Reports filed under one domain do not push another domain's reports out.
func TestIngestKeepsEachDomainsReportsApart(t *testing.T) {
st, err := store.Open(t.TempDir() + "/test.db")
if err != nil {
t.Fatal(err)
}
defer st.Close()
now := time.Now().UTC()
if _, err := st.InsertDMARCReport(store.DMARCReport{
Domain: "quiet.example", Reporter: "genuine", ReportID: "1", PeriodBegin: now, PeriodEnd: now, ReceivedAt: now.Add(-time.Hour),
}); err != nil {
t.Fatal(err)
}
for i := 0; i < store.DMARCReportsMaxKeep+20; i++ {
if _, err := st.InsertDMARCReport(store.DMARCReport{
Domain: "flooded.example", Reporter: "stranger", ReportID: strings.Repeat("x", 1+i), PeriodBegin: now, PeriodEnd: now, ReceivedAt: now,
}); err != nil {
t.Fatal(err)
}
}
if err := st.PruneDMARCReports("flooded.example"); err != nil {
t.Fatal(err)
}
quiet, err := st.ListDMARCReports(store.DMARCReportScope{Domains: []string{"quiet.example"}}, 10)
if err != nil {
t.Fatal(err)
}
if len(quiet) != 1 {
t.Fatalf("the other domain has %d reports after the flood, want its 1", len(quiet))
}
flooded, err := st.ListDMARCReports(store.DMARCReportScope{Domains: []string{"flooded.example"}}, 10000)
if err != nil {
t.Fatal(err)
}
if len(flooded) > store.DMARCReportsMaxKeep {
t.Fatalf("the flooded domain keeps %d reports, want at most %d", len(flooded), store.DMARCReportsMaxKeep)
}
}
+16 -6
View File
@@ -3,6 +3,7 @@ package dmarc
import (
"bytes"
"encoding/base64"
"errors"
"fmt"
"io"
"mime"
@@ -16,6 +17,15 @@ import (
"github.com/mixeme/selfpost/internal/store"
)
// ErrRejected marks a message that is not a report this relay keeps and never
// will be: it cannot be read as an aggregate report, or it reports on a domain
// that is not sent from here, or it came to another domain's address. Sending
// it again changes nothing, so the caller drops it. Any other error is the
// store's and worth another try.
var ErrRejected = errors.New("dmarc report rejected")
func rejected(err error) error { return fmt.Errorf("%w: %w", ErrRejected, err) }
// IngestMessage parses a raw RFC 5322 message from r and stores any DMARC
// aggregate attachment it finds.
//
@@ -34,11 +44,11 @@ func IngestMessage(st *store.Store, r io.Reader, recipient string, receivedAt ti
}
payload, err := extractAggregatePayload(raw)
if err != nil {
return err
return rejected(err)
}
parsed, err := ParseAggregate(payload)
if err != nil {
return err
return rejected(err)
}
recipient = strings.ToLower(strings.TrimSpace(recipient))
if err := checkReportDomain(st, parsed.Domain, recipient); err != nil {
@@ -74,7 +84,7 @@ func IngestMessage(st *store.Store, r io.Reader, recipient string, receivedAt ti
if _, err := st.InsertDMARCReport(rep); err != nil {
return err
}
return st.PruneDMARCReports()
return st.PruneDMARCReports(rep.Domain)
}
// checkReportDomain refuses a report whose policy domain is not one of this
@@ -82,17 +92,17 @@ func IngestMessage(st *store.Store, r io.Reader, recipient string, receivedAt ti
// delivered to.
func checkReportDomain(st *store.Store, domain, recipient string) error {
if domain == "" {
return fmt.Errorf("dmarc report: no policy domain")
return rejected(fmt.Errorf("no policy domain"))
}
if tagged := TaggedDomain(recipient); tagged != "" && tagged != domain {
return fmt.Errorf("dmarc report: domain %q does not match the address it was sent to (%s)", domain, recipient)
return rejected(fmt.Errorf("domain %q does not match the address it was sent to (%s)", domain, recipient))
}
ok, err := st.DomainExists(domain)
if err != nil {
return err
}
if !ok {
return fmt.Errorf("dmarc report: %q is not a domain configured on this relay", domain)
return rejected(fmt.Errorf("%q is not a domain configured on this relay", domain))
}
return nil
}
+50 -15
View File
@@ -9,6 +9,7 @@ import (
"io"
"strings"
"time"
"unicode/utf8"
)
// ParsedReport is the structured form of a DMARC aggregate XML payload.
@@ -74,6 +75,31 @@ type feedbackXML struct {
} `xml:"record"`
}
// What a report may hold. The XML comes from anyone who knows a report address,
// and eight megabytes of it (maxAggregateXML) fit some 160 000 empty records or
// one name of eight megabytes - every one of them a row or a table cell later.
// A real report has tens of records, a few thousand from the largest
// receivers, and no field longer than a line.
const (
maxReportRecords = 5000
maxFieldLen = 255
maxRecordCount = 1_000_000_000
)
// field trims a text value of the report and cuts it to maxFieldLen bytes, on
// a character boundary.
func field(s string) string {
s = strings.TrimSpace(s)
if len(s) <= maxFieldLen {
return s
}
s = s[:maxFieldLen]
for len(s) > 0 && !utf8.ValidString(s) {
s = s[:len(s)-1]
}
return s
}
// ParseAggregate decodes gzip- or zip-compressed, or raw, DMARC aggregate XML.
func ParseAggregate(raw []byte) (ParsedReport, error) {
data, err := decompress(raw)
@@ -87,34 +113,43 @@ func ParseAggregate(raw []byte) (ParsedReport, error) {
if doc.Metadata.ReportID == "" || doc.Policy.Domain == "" {
return ParsedReport{}, fmt.Errorf("dmarc xml: missing report_id or domain")
}
if len(doc.Records) > maxReportRecords {
return ParsedReport{}, fmt.Errorf("dmarc xml: %d records, more than the %d a report may hold", len(doc.Records), maxReportRecords)
}
// Cut, not refused: a domain name is at most 253 bytes, so a longer one
// names nothing this relay sends for and is turned away as such.
out := ParsedReport{
Reporter: strings.TrimSpace(doc.Metadata.OrgName),
ReportID: strings.TrimSpace(doc.Metadata.ReportID),
ContactEmail: strings.TrimSpace(doc.Metadata.Email),
Reporter: field(doc.Metadata.OrgName),
ReportID: field(doc.Metadata.ReportID),
ContactEmail: field(doc.Metadata.Email),
PeriodBegin: time.Unix(doc.Metadata.DateRange.Begin, 0).UTC(),
PeriodEnd: time.Unix(doc.Metadata.DateRange.End, 0).UTC(),
Domain: strings.ToLower(strings.TrimSpace(doc.Policy.Domain)),
PolicyP: strings.TrimSpace(doc.Policy.P),
PolicySP: strings.TrimSpace(doc.Policy.SP),
Domain: strings.ToLower(field(doc.Policy.Domain)),
PolicyP: field(doc.Policy.P),
PolicySP: field(doc.Policy.SP),
PolicyPct: doc.Policy.Pct,
PolicyADKIM: strings.TrimSpace(doc.Policy.ADKIM),
PolicyASPF: strings.TrimSpace(doc.Policy.ASPF),
PolicyADKIM: field(doc.Policy.ADKIM),
PolicyASPF: field(doc.Policy.ASPF),
}
if out.PolicyPct == 0 {
out.PolicyPct = 100
}
for _, rec := range doc.Records {
row := ParsedRecord{
SourceIP: strings.TrimSpace(rec.Row.SourceIP),
SourceIP: field(rec.Row.SourceIP),
Count: rec.Row.Count,
Disposition: strings.TrimSpace(rec.Row.Policy.Disposition),
SPFResult: strings.TrimSpace(rec.Row.Policy.SPF),
DKIMResult: strings.TrimSpace(rec.Row.Policy.DKIM),
HeaderFrom: strings.TrimSpace(rec.Identifiers.HeaderFrom),
Disposition: field(rec.Row.Policy.Disposition),
SPFResult: field(rec.Row.Policy.SPF),
DKIMResult: field(rec.Row.Policy.DKIM),
HeaderFrom: field(rec.Identifiers.HeaderFrom),
}
if row.Count <= 0 {
row.Count = 1
}
// Bounded so that the sums of a report cannot wrap around.
if row.Count > maxRecordCount {
row.Count = maxRecordCount
}
if dmarcRecordPasses(row) {
out.PassCount += row.Count
} else {
@@ -172,12 +207,12 @@ func unzip(raw []byte) ([]byte, error) {
}
rc, err := f.Open()
if err != nil {
return nil, fmt.Errorf("dmarc zip open %s: %w", f.Name, err)
return nil, fmt.Errorf("dmarc zip open %q: %w", field(f.Name), err)
}
data, err := io.ReadAll(io.LimitReader(rc, maxAggregateXML))
rc.Close()
if err != nil {
return nil, fmt.Errorf("dmarc zip read %s: %w", f.Name, err)
return nil, fmt.Errorf("dmarc zip read %q: %w", field(f.Name), err)
}
return data, nil
}
+45 -4
View File
@@ -4,6 +4,7 @@ import (
"archive/zip"
"bytes"
"compress/gzip"
"errors"
"testing"
"time"
@@ -126,8 +127,9 @@ func TestIngestMessageRejectsUnknownDomain(t *testing.T) {
}
defer st.Close()
if err := IngestMessage(st, bytes.NewBufferString(gzipMessage(t, sampleXML)), "dmarc-reports@mail.example.com", time.Now().UTC()); err == nil {
t.Fatal("report for an unconfigured domain was accepted")
err = IngestMessage(st, bytes.NewBufferString(gzipMessage(t, sampleXML)), "dmarc-reports@mail.example.com", time.Now().UTC())
if !errors.Is(err, ErrRejected) {
t.Fatalf("report for an unconfigured domain: err = %v, want ErrRejected", err)
}
list, err := st.ListDMARCReports(store.DMARCReportScope{AllDomains: true}, 10)
if err != nil {
@@ -155,8 +157,47 @@ func TestIngestMessageRejectsTagMismatch(t *testing.T) {
err = IngestMessage(st, bytes.NewBufferString(gzipMessage(t, sampleXML)),
"dmarc-reports+other.example@mail.example.com", time.Now().UTC())
if err == nil {
t.Fatal("report was accepted at an address tagged for another domain")
if !errors.Is(err, ErrRejected) {
t.Fatalf("report at an address tagged for another domain: err = %v, want ErrRejected", err)
}
}
// What cannot be read as a report is rejected, not put off: Postfix would
// offer the same bytes again for days.
func TestIngestMessageRejectsWhatIsNotAReport(t *testing.T) {
st, err := store.Open(t.TempDir() + "/test.db")
if err != nil {
t.Fatal(err)
}
defer st.Close()
if _, err := st.AddDomain("example.com", "selfpost"); err != nil {
t.Fatal(err)
}
for name, msg := range map[string]string{
"plain text": "From: someone@example.net\r\nSubject: hello\r\n\r\nnot a report\r\n",
"no headers": "garbage",
"broken gzip": "From: a@example.net\r\nContent-Type: application/gzip\r\n\r\nnot gzip at all",
} {
err := IngestMessage(st, bytes.NewBufferString(msg), "dmarc-reports+example.com@mail.example.com", time.Now().UTC())
if !errors.Is(err, ErrRejected) {
t.Errorf("%s: err = %v, want ErrRejected", name, err)
}
}
}
// A failing store is not the report's fault: the message is to be tried again.
func TestIngestMessageStoreFailureIsNotARejection(t *testing.T) {
st, err := store.Open(t.TempDir() + "/test.db")
if err != nil {
t.Fatal(err)
}
if _, err := st.AddDomain("example.com", "selfpost"); err != nil {
t.Fatal(err)
}
st.Close()
err = IngestMessage(st, bytes.NewBufferString(gzipMessage(t, sampleXML)), "dmarc-reports+example.com@mail.example.com", time.Now().UTC())
if err == nil || errors.Is(err, ErrRejected) {
t.Fatalf("closed store: err = %v, want an error that is not ErrRejected", err)
}
}
+52
View File
@@ -100,6 +100,7 @@ type Checker struct {
servers map[string]cached[Server]
domains map[string]cached[Domain]
inbound map[string]cached[Result]
mail map[string]cached[Mail]
}
type cached[T any] struct {
@@ -123,6 +124,7 @@ func newChecker(r resolver, timeout, srvTTL, domTTL time.Duration) *Checker {
servers: make(map[string]cached[Server]),
domains: make(map[string]cached[Domain]),
inbound: make(map[string]cached[Result]),
mail: make(map[string]cached[Mail]),
}
}
@@ -170,6 +172,56 @@ func (c *Checker) Domain(q Query, force bool) Domain {
return d
}
// Mail is the published-DNS state of the server's own name as a sender: the
// panel's notifications go out from postmaster@ the hostname, signed with a
// key of that name.
type Mail struct {
Name string
DKIM Result
SPF Result
Overall health.Status
CheckedAt time.Time
}
// ServerMailSPF is the SPF record to publish for the server's own name: its
// address record is the one sender.
const ServerMailSPF = "v=spf1 a -all"
// ServerMail checks the DKIM and SPF records of the server's own name. q is as
// for a domain, with Name the hostname. Cached like a domain; force skips the
// cache.
func (c *Checker) ServerMail(q Query, force bool) Mail {
key := "\x00mail\x00" + q.Name
if !force {
c.mu.Lock()
entry, ok := c.mail[key]
c.mu.Unlock()
if ok && time.Now().Before(entry.expires) {
return entry.value
}
}
ctx, cancel := context.WithTimeout(context.Background(), c.timeout)
defer cancel()
m := Mail{Name: q.Name, CheckedAt: time.Now()}
var wg sync.WaitGroup
wg.Add(2)
go func() { defer wg.Done(); m.DKIM = c.checkDKIM(ctx, q) }()
go func() { defer wg.Done(); m.SPF = c.checkSPF(ctx, q) }()
wg.Wait()
// checkSPF words a missing record for a sending domain; for the server's
// own name the record to publish is a different one.
if m.SPF.Status == health.StatusError && len(m.SPF.Records) == 0 {
m.SPF.Detail = fmt.Sprintf("No SPF record is published for %s. Publish the TXT record %q - without it a receiver has nothing authorising this server to send as its own name.", q.Name, ServerMailSPF)
}
m.Overall = health.Worst(m.DKIM.Status, m.SPF.Status)
c.mu.Lock()
c.mail[key] = cached[Mail]{value: m, expires: m.CheckedAt.Add(c.domainTTL)}
c.mu.Unlock()
return m
}
// Forget drops a domain's cached result, so the next page view re-checks it.
// Used when a domain is removed or re-imported.
func (c *Checker) Forget(domainName string) {
+82
View File
@@ -0,0 +1,82 @@
package dnscheck
import (
"net"
"strings"
"testing"
"github.com/mixeme/selfpost/internal/health"
)
const serverDKIM = "v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAserverkey"
func serverMailQuery() Query {
return Query{
Name: "mail.example.org",
Selector: "sp",
ExpectedDKIM: serverDKIM,
Hostname: "mail.example.org",
ServerIPs: []string{"203.0.113.25"},
}
}
// The server's own name passes with its key published and "v=spf1 a -all":
// the name's address record is this server.
func TestServerMailPublished(t *testing.T) {
f := &fakeResolver{
txt: map[string][]string{
"sp._domainkey.mail.example.org": {serverDKIM},
"mail.example.org": {ServerMailSPF},
},
addr: map[string][]net.IPAddr{"mail.example.org": ipAddrs("203.0.113.25")},
}
c := newTestChecker(f)
m := c.ServerMail(serverMailQuery(), false)
if m.DKIM.Status != health.StatusOK || m.SPF.Status != health.StatusOK || m.Overall != health.StatusOK {
t.Fatalf("DKIM %s (%s), SPF %s (%s), overall %s", m.DKIM.Status, m.DKIM.Detail, m.SPF.Status, m.SPF.Detail, m.Overall)
}
// Cached like a domain, and re-read when forced.
before := f.lookups
c.ServerMail(serverMailQuery(), false)
if f.lookups != before {
t.Errorf("a second reading looked DNS up again (%d lookups)", f.lookups-before)
}
c.ServerMail(serverMailQuery(), true)
if f.lookups == before {
t.Error("a forced reading used the cache")
}
}
// With nothing published both records are errors, and the SPF advice names the
// record for a server's own name, not the one for a sending domain.
func TestServerMailMissing(t *testing.T) {
c := newTestChecker(&fakeResolver{})
m := c.ServerMail(serverMailQuery(), false)
if m.DKIM.Status != health.StatusError || m.SPF.Status != health.StatusError || m.Overall != health.StatusError {
t.Fatalf("DKIM %s, SPF %s, overall %s; want errors", m.DKIM.Status, m.SPF.Status, m.Overall)
}
if !strings.Contains(m.SPF.Detail, ServerMailSPF) || strings.Contains(m.SPF.Detail, "a:mail.example.org") {
t.Errorf("SPF advice: %s", m.SPF.Detail)
}
}
// A sending domain of the same name does not share the cache entry.
func TestServerMailCacheIsItsOwn(t *testing.T) {
f := &fakeResolver{
txt: map[string][]string{
"sp._domainkey.mail.example.org": {serverDKIM},
"mail.example.org": {ServerMailSPF},
},
addr: map[string][]net.IPAddr{"mail.example.org": ipAddrs("203.0.113.25")},
}
c := newTestChecker(f)
c.Domain(serverMailQuery(), false)
before := f.lookups
c.ServerMail(serverMailQuery(), false)
if f.lookups == before {
t.Error("the server's mail check was answered from the domain's cache entry")
}
}
+109
View File
@@ -0,0 +1,109 @@
package domain
import (
"os"
"strings"
"testing"
)
func readTables(t *testing.T, odk *OpenDKIM) (keyTable, signingTable string) {
t.Helper()
kt, err := os.ReadFile(odk.keyTablePath)
if err != nil {
t.Fatalf("read KeyTable: %v", err)
}
st, err := os.ReadFile(odk.signingTablePath)
if err != nil {
t.Fatalf("read SigningTable: %v", err)
}
return string(kt), string(st)
}
// The server's own name is signed like a sending domain - a key, a line in
// each table - without being one: no row in the registry, so no application
// can send as it.
func TestServerNameIsSignedWithoutBeingADomain(t *testing.T) {
svc, odk := newTestService(t, &fakeApps{})
svc.SetServerName("mail.example.org")
if _, err := svc.Add("example.com"); err != nil {
t.Fatalf("add domain: %v", err)
}
kt, st := readTables(t, odk)
for _, want := range []string{"mail.example.org mail.example.org:selfpost:", "example.com example.com:selfpost:"} {
if !strings.Contains(kt, want) {
t.Errorf("KeyTable lacks %q:\n%s", want, kt)
}
}
if !strings.Contains(st, "*@mail.example.org mail.example.org\n") {
t.Errorf("SigningTable does not sign the server's own name:\n%s", st)
}
domains, err := svc.List()
if err != nil || len(domains) != 1 {
t.Fatalf("registry holds %d domains, %v; want the one that was added", len(domains), err)
}
mail, err := svc.ServerMail()
if err != nil {
t.Fatalf("ServerMail: %v", err)
}
if mail.Name != "mail.example.org" || mail.Selector != "selfpost" ||
mail.Record.Name != "selfpost._domainkey.mail.example.org" || !strings.HasPrefix(mail.Record.Value, "v=DKIM1;") {
t.Errorf("server mail = %+v", mail)
}
// Asked again, it is the same key: the record published in DNS stays valid.
again, err := svc.ServerMail()
if err != nil || again.Record.Value != mail.Record.Value {
t.Errorf("the server's key changed between two readings (%v)", err)
}
}
// When the hostname is itself a sending domain, there is one key and one line,
// the domain's; deleting the domain leaves the server's mail signed with it.
func TestServerNameThatIsAlsoADomain(t *testing.T) {
svc, odk := newTestService(t, &fakeApps{})
svc.SetServerName("example.com")
d, err := svc.Add("example.com")
if err != nil {
t.Fatalf("add domain: %v", err)
}
_, st := readTables(t, odk)
if strings.Count(st, "*@example.com ") != 1 {
t.Errorf("the name is in the SigningTable more than once:\n%s", st)
}
asDomain, err := svc.DKIMRecord(d)
if err != nil {
t.Fatal(err)
}
mail, err := svc.ServerMail()
if err != nil || mail.Record.Value != asDomain.Value {
t.Fatalf("the server's record is not the domain's (%v)", err)
}
if err := svc.Delete(d.ID); err != nil {
t.Fatalf("delete domain: %v", err)
}
_, st = readTables(t, odk)
if !strings.Contains(st, "*@example.com example.com\n") {
t.Errorf("deleting the domain stopped the server's own mail being signed:\n%s", st)
}
after, err := svc.ServerMail()
if err != nil || after.Record.Value != asDomain.Value {
t.Errorf("deleting the domain changed the server's key (%v)", err)
}
}
// Without a hostname nothing is added and nothing is signed for it.
func TestNoServerName(t *testing.T) {
svc, odk := newTestService(t, &fakeApps{})
if _, err := svc.Add("example.com"); err != nil {
t.Fatal(err)
}
if _, st := readTables(t, odk); strings.Count(st, "\n") != 1 {
t.Errorf("SigningTable has more than the one domain:\n%s", st)
}
if _, err := svc.ServerMail(); err == nil {
t.Error("ServerMail without a hostname returned no error")
}
}
+77 -1
View File
@@ -38,6 +38,64 @@ type Service struct {
odk *OpenDKIM
apps Applications
selector string
// serverName is the server's own hostname, which the panel's notifications
// are sent from and signed as (SetServerName). Empty: nothing is signed
// for it.
serverName string
}
// SetServerName gives the service the server's own hostname. The panel sends
// its notifications from postmaster@ that name, and for a receiver to believe
// them the name is signed like a sending domain: it gets a DKIM key and a line
// in OpenDKIM's tables, without being a row of the registry - no application
// can send as it. The name comes from SELFPOST_HOSTNAME, which the entrypoint
// has validated; it is not user input.
func (s *Service) SetServerName(hostname string) {
s.serverName = hostname
}
// ServerMail is how the server's own name is signed.
type ServerMail struct {
Name string
Selector string
Record DKIMRecord
}
// ServerMail returns the name notifications are sent as, its DKIM selector and
// the record to publish for it, creating the key the first time. When the
// hostname is itself a sending domain, it is that domain's key and selector:
// one name has one key.
func (s *Service) ServerMail() (ServerMail, error) {
if s.serverName == "" {
return ServerMail{}, fmt.Errorf("server mail: no hostname")
}
selector, err := s.serverSelector()
if err != nil {
return ServerMail{}, err
}
if _, err := s.odk.EnsureKey(s.serverName, selector); err != nil {
return ServerMail{}, err
}
rec, err := s.odk.Record(s.serverName, selector)
if err != nil {
return ServerMail{}, err
}
return ServerMail{Name: s.serverName, Selector: selector, Record: rec}, nil
}
// serverSelector is the selector the server's own name is signed with: that of
// the sending domain of the same name if there is one, the default otherwise.
func (s *Service) serverSelector() (string, error) {
domains, err := s.store.ListDomains()
if err != nil {
return "", err
}
for _, d := range domains {
if d.Name == s.serverName {
return d.DKIMSelector, nil
}
}
return s.selector, nil
}
// NewService builds the domain service. selectorDefault is the DKIM selector
@@ -125,6 +183,12 @@ func (s *Service) Delete(id int64) error {
if err := s.apps.Resync(); err != nil {
return fmt.Errorf("rebuild sender map after deleting %s: %w", d.Name, err)
}
// A sending domain named like the server shared its key with the server's
// own mail, which goes on being signed with it (resync above kept the name
// in the tables).
if d.Name == s.serverName {
return nil
}
if err := s.odk.RemoveKey(d.Name); err != nil {
// The domain is gone from the registry and tables; a leftover key
// directory is harmless. Surface it so it is not silently ignored.
@@ -172,9 +236,21 @@ func (s *Service) resync() error {
if err != nil {
return err
}
signing := make([]SigningDomain, 0, len(domains))
signing := make([]SigningDomain, 0, len(domains)+1)
registered := false
for _, d := range domains {
signing = append(signing, SigningDomain{Name: d.Name, Selector: d.DKIMSelector})
if d.Name == s.serverName {
registered = true
}
}
// The server's own name, for the panel's notifications - unless it is a
// sending domain already and so in the tables above.
if s.serverName != "" && !registered {
if _, err := s.odk.EnsureKey(s.serverName, s.selector); err != nil {
return fmt.Errorf("dkim key for %s: %w", s.serverName, err)
}
signing = append(signing, SigningDomain{Name: s.serverName, Selector: s.selector})
}
return s.odk.Rebuild(signing)
}
+447
View File
@@ -0,0 +1,447 @@
package inbound
import (
"crypto/rand"
"encoding/hex"
"fmt"
"net/netip"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"time"
"unicode/utf8"
"github.com/mixeme/selfpost/internal/atomicfile"
"github.com/mixeme/selfpost/internal/configsafe"
"github.com/mixeme/selfpost/internal/store"
)
// ListStore is the slice of the store the filter lists need. *store.Store
// satisfies it.
type ListStore interface {
AddSpamListEntry(list, kind, value, note string) (store.SpamListEntry, error)
ListSpamListEntries() ([]store.SpamListEntry, error)
DeleteSpamListEntry(id int64) error
ListSpamListHits(marks ...string) ([]store.SpamListHit, error)
}
// Lists owns the allow and deny lists of the external spam filter: the rows in
// SQLite and the map files the filter reads. SelfPost does not consult the
// lists itself - it writes them out, six files in one directory, one entry a
// line, and the filter (rspamd, deploy/antispam/rspamd/multimap.conf) rereads
// a file when it changes. Which list wins when both match is the filter's
// configuration: deny.
type Lists struct {
store ListStore
dir string
// mu makes a change of the table and the rewrite of the files one step.
// Two requests at once could otherwise write the files in the other order
// than they changed the table - the first to read the entries being the
// last to write them - and leave the filter with an entry the page no
// longer lists. One Lists serves the panel, so a mutex is enough.
mu sync.Mutex
}
// NewLists builds the filter lists over the store and the map directory
// (/data/antispam).
func NewLists(st ListStore, dir string) *Lists {
return &Lists{store: st, dir: dir}
}
// noteMaxRunes caps the operator's comment on an entry.
const noteMaxRunes = 200
// mapFiles names the file of every (list, kind). An IP map and a sender map
// are different things to the filter, and a domain entry is matched against
// another part of the sender than an address, so each has its own file.
var mapFiles = map[[2]string]string{
{store.SpamListAllow, store.SpamEntryIP}: "allow_ip.map",
{store.SpamListAllow, store.SpamEntryAddress}: "allow_from.map",
{store.SpamListAllow, store.SpamEntryDomain}: "allow_domain.map",
{store.SpamListDeny, store.SpamEntryIP}: "deny_ip.map",
{store.SpamListDeny, store.SpamEntryAddress}: "deny_from.map",
{store.SpamListDeny, store.SpamEntryDomain}: "deny_domain.map",
}
// Entries returns every entry of both lists.
func (l *Lists) Entries() ([]store.SpamListEntry, error) {
return l.store.ListSpamListEntries()
}
// Add validates an entry, stores it normalised and rewrites the maps. A
// refusal is a *ValidationError; an entry already on that list is
// store.ErrSpamListEntryExists.
func (l *Lists) Add(list, kind, rawValue, rawNote string) (store.SpamListEntry, error) {
if list != store.SpamListAllow && list != store.SpamListDeny {
return store.SpamListEntry{}, invalid("choose the allow or the deny list")
}
value, err := normalizeListValue(kind, rawValue)
if err != nil {
return store.SpamListEntry{}, err
}
note, err := normalizeNote(rawNote)
if err != nil {
return store.SpamListEntry{}, err
}
l.mu.Lock()
defer l.mu.Unlock()
e, err := l.store.AddSpamListEntry(list, kind, value, note)
if err != nil {
return store.SpamListEntry{}, err
}
if err := l.sync(); err != nil {
// An entry the filter never got is worse than no entry: the page
// would show a rule that does nothing. The files are written again
// after the row is taken back, since some of the six may already
// hold it; if that fails too, the next change or start puts it right.
_ = l.store.DeleteSpamListEntry(e.ID)
_ = l.sync()
return store.SpamListEntry{}, err
}
return e, nil
}
// Delete removes an entry and rewrites the maps. If the maps cannot be
// written the entry is put back: an entry that is gone from the page and
// still in the filter's file could not even be removed a second time.
func (l *Lists) Delete(id int64) error {
l.mu.Lock()
defer l.mu.Unlock()
entries, err := l.store.ListSpamListEntries()
if err != nil {
return err
}
var gone *store.SpamListEntry
for i := range entries {
if entries[i].ID == id {
gone = &entries[i]
}
}
if err := l.store.DeleteSpamListEntry(id); err != nil {
return err
}
if err := l.sync(); err != nil {
if gone != nil {
_, _ = l.store.AddSpamListEntry(gone.List, gone.Kind, gone.Value, gone.Note)
}
return err
}
return nil
}
// Sync writes all six map files from SQLite, each atomically, so the filter
// reads a whole old file or a whole new one. Every file is written even when
// empty: the filter's configuration names all six.
func (l *Lists) Sync() error {
l.mu.Lock()
defer l.mu.Unlock()
return l.sync()
}
// sync is Sync for a caller that holds mu.
func (l *Lists) sync() error {
entries, err := l.store.ListSpamListEntries()
if err != nil {
return err
}
lines := make(map[string][]string, len(mapFiles))
for _, name := range mapFiles {
lines[name] = nil
}
for _, e := range entries {
name, ok := mapFiles[[2]string{e.List, e.Kind}]
if !ok {
return fmt.Errorf("filter list entry %d: unknown list %q or kind %q", e.ID, e.List, e.Kind)
}
// Values were validated on the way in; this is the writer's own guard
// against a row that got into the table some other way. In a map file a
// newline ends the entry, whitespace starts its value and # a comment.
if err := configsafe.Token("filter list entry", e.Value, " \t\r\n#"); err != nil {
return err
}
lines[name] = append(lines[name], e.Value)
}
// The filter runs under another user in another container: the directory
// and the files are readable by anyone. They hold addresses, not secrets.
if err := os.MkdirAll(l.dir, 0o755); err != nil {
return fmt.Errorf("create %s: %w", l.dir, err)
}
for name, vals := range lines {
sort.Strings(vals)
var b strings.Builder
b.WriteString("# Written by SelfPost (Inbound > Filter lists). Do not edit: the next change in the panel replaces this file.\n")
for _, v := range vals {
b.WriteString(v)
b.WriteByte('\n')
}
if err := atomicfile.Write(filepath.Join(l.dir, name), []byte(b.String()), 0o644); err != nil {
return err
}
}
return nil
}
// LastWritten is when the maps were last written: the newest modification
// time of the six files, the zero time while none exists (the filter lists
// have never been touched, or the directory is gone).
func (l *Lists) LastWritten() time.Time {
var newest time.Time
for _, name := range mapFiles {
if fi, err := os.Stat(filepath.Join(l.dir, name)); err == nil && fi.ModTime().After(newest) {
newest = fi.ModTime()
}
}
return newest
}
// Listable reports whether value could be added to a list as an entry of kind.
// The Inbound log asks before it offers a row's sender to the lists: an
// envelope sender may be spelt in ways the lists do not take.
func Listable(kind, value string) bool {
_, err := normalizeListValue(kind, value)
return err == nil
}
// normalizeListValue validates one entry and returns the form it is stored and
// written in: lower case for an address or a domain, the canonical text of an
// address or a network for an IP.
func normalizeListValue(kind, raw string) (string, error) {
v := strings.ToLower(strings.TrimSpace(raw))
switch kind {
case store.SpamEntryAddress:
if v == "" {
return "", invalid("sender address is required")
}
if len(v) > 254 {
return "", invalid("sender address must be at most 254 characters")
}
at := strings.LastIndexByte(v, '@')
if at <= 0 || at >= len(v)-1 {
return "", invalid("%q is not a valid email address", v)
}
local, domain := v[:at], v[at+1:]
if local[0] == '.' || local[len(local)-1] == '.' || len(local) > 64 {
return "", invalid("%q: invalid local part", v)
}
for i := 0; i < len(local); i++ {
c := local[i]
lower := c >= 'a' && c <= 'z'
digit := c >= '0' && c <= '9'
// '=' for the rewritten senders of forwarders (SRS0=…).
if !lower && !digit && !strings.ContainsRune("._+-=", rune(c)) {
return "", invalid("%q: local part contains invalid characters", v)
}
}
if err := checkDomain(domain); err != nil {
return "", err
}
return v, nil
case store.SpamEntryDomain:
// "@example.com" is how a domain is often written in such a list.
v = strings.TrimPrefix(v, "@")
if err := checkDomain(v); err != nil {
return "", err
}
return v, nil
case store.SpamEntryIP:
if v == "" {
return "", invalid("address or network is required")
}
if strings.Contains(v, "/") {
p, err := netip.ParsePrefix(v)
if err != nil {
return "", invalid("%q is not a network in CIDR form (e.g. 192.0.2.0/24)", v)
}
if p.Addr().Is4In6() {
return "", invalid("%q: write an IPv4 network as IPv4 (e.g. 192.0.2.0/24)", v)
}
if p.Bits() == 0 {
return "", invalid("%q is every address there is", v)
}
// A network of one address is that address: 192.0.2.7/32 and
// 192.0.2.7 are one entry, not two that look different.
if p.Bits() == p.Addr().BitLen() {
return p.Addr().String(), nil
}
return p.Masked().String(), nil
}
a, err := netip.ParseAddr(v)
if err != nil || a.Zone() != "" {
return "", invalid("%q is not an IP address or a network in CIDR form", v)
}
return a.Unmap().String(), nil
default:
return "", invalid("choose what the entry is: a sender address, a sender domain or a client address")
}
}
// normalizeNote trims the operator's comment and refuses what a single line of
// text cannot be.
func normalizeNote(raw string) (string, error) {
note := strings.TrimSpace(raw)
if utf8.RuneCountInString(note) > noteMaxRunes {
return "", invalid("note must be at most %d characters", noteMaxRunes)
}
for _, r := range note {
if r < 0x20 || r == 0x7f || r == utf8.RuneError {
return "", invalid("note must be one line of plain text")
}
}
return note, nil
}
// What a list leaves in the journal when it decides a message - the two ends
// of deploy/antispam/rspamd/multimap.conf this package depends on. A deny rule
// refuses with a message, which the log tailer files as the reply of a
// rejected row. An allow rule adds a symbol named after the kind of entry,
// which the journal milter keeps in the verdict of an accepted row.
const (
denyReplyMark = "this server's deny list"
allowMark = "SELFPOST_ALLOW_"
)
var allowSymbol = map[string]string{
store.SpamEntryIP: "SELFPOST_ALLOW_IP(",
store.SpamEntryAddress: "SELFPOST_ALLOW_FROM(",
store.SpamEntryDomain: "SELFPOST_ALLOW_DOMAIN(",
}
// Usage is how often an entry decided a message, as far back as the journal
// reaches, and when it last did. The zero value is an entry that never did.
type Usage struct {
Hits int
Last time.Time
}
// Usage counts, for every entry, the journal rows it decided. Nothing is
// stored for this: the journal already says which list decided a message and
// who the message was from, and an entry is matched against that the way the
// filter matches it - an address whole, a domain as the sender's domain, an
// address or network against the client's. So the count reaches back as far
// as the journal does (its retention, or less when it is full), and a row
// counts for every entry of the deciding list that fits it: the journal does
// not say which of two fitting entries the filter met first.
//
// A counter kept with each entry would not depend on the journal; the plan
// (docs/plans/inbound-antispam-panel.md, Filter list usage) says when that is
// worth a migration.
func (l *Lists) Usage() (map[int64]Usage, error) {
entries, err := l.store.ListSpamListEntries()
if err != nil {
return nil, err
}
out := make(map[int64]Usage, len(entries))
if len(entries) == 0 {
return out, nil
}
hits, err := l.store.ListSpamListHits(denyReplyMark, allowMark)
if err != nil {
return nil, err
}
type matcher struct {
id int64
list string
kind string
value string
prefix netip.Prefix
}
matchers := make([]matcher, 0, len(entries))
for _, e := range entries {
m := matcher{id: e.ID, list: e.List, kind: e.Kind, value: e.Value}
if e.Kind == store.SpamEntryIP {
if p, err := netip.ParsePrefix(e.Value); err == nil {
m.prefix = p
} else if a, err := netip.ParseAddr(e.Value); err == nil {
m.prefix = netip.PrefixFrom(a, a.BitLen())
} else {
continue // not an entry this package wrote
}
}
matchers = append(matchers, m)
}
for _, h := range hits {
from := strings.ToLower(h.From)
domain := ""
if i := strings.LastIndexByte(from, '@'); i >= 0 {
domain = from[i+1:]
}
client, clientErr := netip.ParseAddr(h.ClientIP)
client = client.Unmap()
denied := h.Decision != store.SpamDecisionAccept && strings.Contains(h.Detail, denyReplyMark)
for _, m := range matchers {
switch m.list {
case store.SpamListDeny:
if !denied {
continue
}
case store.SpamListAllow:
if h.Decision != store.SpamDecisionAccept || !strings.Contains(h.Detail, allowSymbol[m.kind]) {
continue
}
}
fits := false
switch m.kind {
case store.SpamEntryAddress:
fits = from == m.value
case store.SpamEntryDomain:
fits = domain == m.value
case store.SpamEntryIP:
fits = clientErr == nil && m.prefix.Contains(client)
}
if !fits {
continue
}
u := out[m.id]
u.Hits++
if h.CreatedAt.After(u.Last) {
u.Last = h.CreatedAt
}
out[m.id] = u
}
}
return out, nil
}
// FilterKeyFile is the file, beside the list files, that holds the key the
// spam filter signs messages with.
const FilterKeyFile = "filter.key"
// FilterKey returns the key the spam filter signs each message with, creating
// it on first use: 32 random bytes as 64 hex characters on one line, in the
// directory the filter's container mounts.
//
// The filter is attached so that mail flows when it is down. Then nothing
// removes verdict headers a sender wrote himself, and the inbound journal
// would record his verdict as the filter's. So the filter signs each message
// with this key and the journal reads a verdict only next to a signature it
// can recompute (internal/milter, FilterSignature).
//
// The file is readable by anyone on the host, like the list files and for the
// same reason: the filter is another user in another container. What the key
// protects is the truth of a monitoring page, against senders on the internet;
// someone with a shell on the host is not who it is kept from.
func FilterKey(dir string) (string, error) {
path := filepath.Join(dir, FilterKeyFile)
if raw, err := os.ReadFile(path); err == nil {
if key := strings.TrimSpace(string(raw)); len(key) >= 32 && !strings.ContainsAny(key, " \t\r\n") {
return key, nil
}
// Empty or cut short: not a key anyone should sign with. Replaced below.
} else if !os.IsNotExist(err) {
return "", fmt.Errorf("read %s: %w", path, err)
}
if err := os.MkdirAll(dir, 0o755); err != nil {
return "", fmt.Errorf("create %s: %w", dir, err)
}
buf := make([]byte, 32)
if _, err := rand.Read(buf); err != nil {
return "", fmt.Errorf("generate the filter key: %w", err)
}
key := hex.EncodeToString(buf)
if err := atomicfile.Write(path, []byte(key+"\n"), 0o644); err != nil {
return "", err
}
return key, nil
}
+440
View File
@@ -0,0 +1,440 @@
package inbound
import (
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/mixeme/selfpost/internal/store"
)
func testLists(t *testing.T) (*Lists, *store.Store, string) {
t.Helper()
st, err := store.Open(filepath.Join(t.TempDir(), "test.db"))
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { st.Close() })
dir := filepath.Join(t.TempDir(), "antispam")
return NewLists(st, dir), st, dir
}
// mapLines reads a map file without its comment line.
func mapLines(t *testing.T, dir, name string) []string {
t.Helper()
raw, err := os.ReadFile(filepath.Join(dir, name))
if err != nil {
t.Fatalf("read %s: %v", name, err)
}
var out []string
for _, line := range strings.Split(strings.TrimRight(string(raw), "\n"), "\n") {
if !strings.HasPrefix(line, "#") {
out = append(out, line)
}
}
return out
}
func TestNormalizeListValue(t *testing.T) {
for _, c := range []struct{ kind, raw, want string }{
{store.SpamEntryAddress, " Billing@Vendor.Example ", "billing@vendor.example"},
{store.SpamEntryAddress, "srs0=ab12=cd=example.org=user@forwarder.example", "srs0=ab12=cd=example.org=user@forwarder.example"},
{store.SpamEntryDomain, "@Evil.Example", "evil.example"},
{store.SpamEntryDomain, "sub.evil.example", "sub.evil.example"},
{store.SpamEntryIP, "192.0.2.7", "192.0.2.7"},
{store.SpamEntryIP, "192.0.2.77/24", "192.0.2.0/24"},
{store.SpamEntryIP, "192.0.2.7/32", "192.0.2.7"},
{store.SpamEntryIP, "2001:db8::1/128", "2001:db8::1"},
{store.SpamEntryIP, "::ffff:192.0.2.7", "192.0.2.7"},
{store.SpamEntryIP, "2001:DB8:0:0::1", "2001:db8::1"},
{store.SpamEntryIP, "2001:db8:1::5/48", "2001:db8:1::/48"},
} {
got, err := normalizeListValue(c.kind, c.raw)
if err != nil || got != c.want {
t.Errorf("normalizeListValue(%s, %q) = %q, %v; want %q", c.kind, c.raw, got, err, c.want)
}
}
for _, c := range []struct{ kind, raw string }{
{store.SpamEntryAddress, ""},
{store.SpamEntryAddress, "no-at-sign"},
{store.SpamEntryAddress, "@example.com"},
{store.SpamEntryAddress, "a@localhost"},
{store.SpamEntryAddress, "a b@example.com"},
{store.SpamEntryAddress, "a#b@example.com"},
{store.SpamEntryAddress, "a\nb@example.com"},
{store.SpamEntryAddress, ".a@example.com"},
{store.SpamEntryDomain, ""},
{store.SpamEntryDomain, "example"},
{store.SpamEntryDomain, "exa mple.com"},
{store.SpamEntryDomain, "example.com # all of it"},
{store.SpamEntryDomain, "user@example.com"},
{store.SpamEntryIP, ""},
{store.SpamEntryIP, "example.com"},
{store.SpamEntryIP, "192.0.2.300"},
{store.SpamEntryIP, "192.0.2.0/33"},
{store.SpamEntryIP, "0.0.0.0/0"},
{store.SpamEntryIP, "::/0"},
{store.SpamEntryIP, "fe80::1%eth0"},
{store.SpamEntryIP, "::ffff:192.0.2.0/120"},
{store.SpamEntryIP, "192.0.2.7 192.0.2.8"},
{"subject", "pills"},
{"", "x"},
} {
got, err := normalizeListValue(c.kind, c.raw)
var ve *ValidationError
if err == nil || !errors.As(err, &ve) {
t.Errorf("normalizeListValue(%q, %q) = %q, %v; want a ValidationError", c.kind, c.raw, got, err)
}
}
}
func TestListsAddWritesTheMaps(t *testing.T) {
l, _, dir := testLists(t)
for _, e := range []struct{ list, kind, value string }{
{store.SpamListAllow, store.SpamEntryAddress, "Billing@Vendor.Example"},
{store.SpamListAllow, store.SpamEntryAddress, "accounts@vendor.example"},
{store.SpamListDeny, store.SpamEntryDomain, "@evil.example"},
{store.SpamListDeny, store.SpamEntryIP, "198.51.100.9/24"},
// The same value on both lists is two entries: deny wins in the filter.
{store.SpamListDeny, store.SpamEntryAddress, "billing@vendor.example"},
} {
if _, err := l.Add(e.list, e.kind, e.value, " why "); err != nil {
t.Fatalf("Add(%s, %s, %q): %v", e.list, e.kind, e.value, err)
}
}
want := map[string]string{
"allow_from.map": "accounts@vendor.example billing@vendor.example",
"allow_domain.map": "",
"allow_ip.map": "",
"deny_from.map": "billing@vendor.example",
"deny_domain.map": "evil.example",
"deny_ip.map": "198.51.100.0/24",
}
for name, lines := range want {
if got := strings.Join(mapLines(t, dir, name), " "); got != lines {
t.Errorf("%s = %q, want %q", name, got, lines)
}
// The filter is another user in another container.
if info, err := os.Stat(filepath.Join(dir, name)); err != nil {
t.Errorf("stat %s: %v", name, err)
} else if info.Mode().Perm()&0o004 == 0 && os.PathSeparator == '/' {
t.Errorf("%s is mode %o, not readable by the filter", name, info.Mode().Perm())
}
}
if files, _ := os.ReadDir(dir); len(files) != len(want) {
t.Errorf("the directory holds %d files, want %d (no temp file left behind)", len(files), len(want))
}
entries, err := l.Entries()
if err != nil || len(entries) != 5 {
t.Fatalf("Entries = %d, %v", len(entries), err)
}
if entries[0].Note != "why" {
t.Errorf("note = %q, want it trimmed", entries[0].Note)
}
}
func TestListsRefusals(t *testing.T) {
l, _, dir := testLists(t)
if _, err := l.Add(store.SpamListAllow, store.SpamEntryIP, "192.0.2.7", ""); err != nil {
t.Fatalf("Add: %v", err)
}
// Written differently, the same entry.
if _, err := l.Add(store.SpamListAllow, store.SpamEntryIP, "::ffff:192.0.2.7", ""); !errors.Is(err, store.ErrSpamListEntryExists) {
t.Errorf("the same address again: %v, want ErrSpamListEntryExists", err)
}
var ve *ValidationError
for name, add := range map[string]func() error{
"an unknown list": func() error { _, err := l.Add("block", store.SpamEntryIP, "192.0.2.8", ""); return err },
"a bad value": func() error { _, err := l.Add(store.SpamListDeny, store.SpamEntryIP, "nope", ""); return err },
"a long note": func() error {
_, err := l.Add(store.SpamListDeny, store.SpamEntryIP, "192.0.2.8", strings.Repeat("x", noteMaxRunes+1))
return err
},
"a note of two lines": func() error {
_, err := l.Add(store.SpamListDeny, store.SpamEntryIP, "192.0.2.8", "one\ntwo")
return err
},
} {
if err := add(); !errors.As(err, &ve) {
t.Errorf("%s: %v, want a ValidationError", name, err)
}
}
if got := strings.Join(mapLines(t, dir, "deny_ip.map"), " "); got != "" {
t.Errorf("deny_ip.map = %q after refusals only", got)
}
if entries, _ := l.Entries(); len(entries) != 1 {
t.Errorf("entries = %d, want the first one alone", len(entries))
}
}
func TestListsDelete(t *testing.T) {
l, _, dir := testLists(t)
e, err := l.Add(store.SpamListDeny, store.SpamEntryDomain, "evil.example", "")
if err != nil {
t.Fatalf("Add: %v", err)
}
if err := l.Delete(e.ID); err != nil {
t.Fatalf("Delete: %v", err)
}
if got := mapLines(t, dir, "deny_domain.map"); len(got) != 0 {
t.Errorf("deny_domain.map = %v after the delete", got)
}
if err := l.Delete(e.ID); !errors.Is(err, store.ErrSpamListEntryNotFound) {
t.Errorf("a second delete: %v", err)
}
}
// An entry the filter never got must not stay on the page.
func TestListsAddRollsBackWhenTheMapsCannotBeWritten(t *testing.T) {
st, err := store.Open(filepath.Join(t.TempDir(), "test.db"))
if err != nil {
t.Fatal(err)
}
defer st.Close()
// A file where the directory should be.
blocked := filepath.Join(t.TempDir(), "antispam")
if err := os.WriteFile(blocked, nil, 0o600); err != nil {
t.Fatal(err)
}
l := NewLists(st, blocked)
if _, err := l.Add(store.SpamListDeny, store.SpamEntryDomain, "evil.example", ""); err == nil {
t.Fatal("Add succeeded with nowhere to write the maps")
}
if entries, _ := l.Entries(); len(entries) != 0 {
t.Errorf("the entry stayed: %+v", entries)
}
}
// The writer's own guard: a row that got into the table some other way is not
// written into a map.
func TestListsSyncRefusesAnUnsafeRow(t *testing.T) {
l, st, dir := testLists(t)
if _, err := st.AddSpamListEntry(store.SpamListDeny, store.SpamEntryDomain, "evil.example\ngood.example", ""); err != nil {
t.Fatalf("AddSpamListEntry: %v", err)
}
if err := l.Sync(); err == nil {
t.Fatal("Sync wrote a value with a newline in it")
}
if _, err := os.Stat(filepath.Join(dir, "deny_domain.map")); err == nil {
t.Error("a map file was written")
}
}
// With nothing on the lists all six files are there, empty.
func TestListsSyncWritesEmptyMaps(t *testing.T) {
l, _, dir := testLists(t)
if err := l.Sync(); err != nil {
t.Fatalf("Sync: %v", err)
}
for _, name := range mapFiles {
if got := mapLines(t, dir, name); len(got) != 0 {
t.Errorf("%s = %v", name, got)
}
}
}
func TestListsLastWritten(t *testing.T) {
l, _, dir := testLists(t)
if got := l.LastWritten(); !got.IsZero() {
t.Fatalf("LastWritten before anything was written = %v", got)
}
if err := l.Sync(); err != nil {
t.Fatal(err)
}
if got := l.LastWritten(); got.IsZero() || time.Since(got) > time.Minute {
t.Errorf("LastWritten after a sync = %v", got)
}
// The newest of the six counts, whichever it is.
old := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
for _, name := range mapFiles {
if err := os.Chtimes(filepath.Join(dir, name), old, old); err != nil {
t.Fatal(err)
}
}
newest := old.Add(48 * time.Hour)
if err := os.Chtimes(filepath.Join(dir, "deny_domain.map"), newest, newest); err != nil {
t.Fatal(err)
}
if got := l.LastWritten(); !got.Equal(newest) {
t.Errorf("LastWritten = %v, want %v", got, newest)
}
// A file that is not one of the six does not count.
other := newest.Add(time.Hour)
if err := os.WriteFile(filepath.Join(dir, "notes.txt"), nil, 0o644); err != nil {
t.Fatal(err)
}
if err := os.Chtimes(filepath.Join(dir, "notes.txt"), other, other); err != nil {
t.Fatal(err)
}
if got := l.LastWritten(); !got.Equal(newest) {
t.Errorf("LastWritten counts a stray file: %v", got)
}
}
// How often an entry decided a message is read off the journal: the list that
// decided, and whether the entry fits who the message was from.
func TestListsUsage(t *testing.T) {
l, st, _ := testLists(t)
add := func(list, kind, value string) int64 {
t.Helper()
e, err := l.Add(list, kind, value, "")
if err != nil {
t.Fatalf("Add(%s, %s, %s): %v", list, kind, value, err)
}
return e.ID
}
denyDomain := add(store.SpamListDeny, store.SpamEntryDomain, "evil.example")
denyAddr := add(store.SpamListDeny, store.SpamEntryAddress, "spammer@evil.example")
denyNet := add(store.SpamListDeny, store.SpamEntryIP, "198.51.100.0/24")
denyV6 := add(store.SpamListDeny, store.SpamEntryIP, "2001:db8::/32")
allowAddr := add(store.SpamListAllow, store.SpamEntryAddress, "billing@vendor.example")
allowDomain := add(store.SpamListAllow, store.SpamEntryDomain, "vendor.example")
allowIP := add(store.SpamListAllow, store.SpamEntryIP, "203.0.113.5")
unused := add(store.SpamListAllow, store.SpamEntryDomain, "quiet.example")
const denied = "5.7.1 Refused by this server's deny list"
at := time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC)
hour := 0
row := func(from, ip, decision, engine, detail string) {
t.Helper()
hour++
err := st.InsertInboundSpam(store.InboundSpamEntry{
InboundDomain: "in.example.com", To: "a@in.example.com", From: from, ClientIP: ip,
Decision: decision, Engine: engine, Detail: detail, At: at.Add(time.Duration(hour) * time.Hour),
})
if err != nil {
t.Fatal(err)
}
}
// hour 1: the address and its domain are both on the deny list, and both fit.
row("Spammer@Evil.Example", "192.0.2.9", store.SpamDecisionReject, store.SpamEngineRspamd, denied)
// hour 2: another address of the domain.
row("other@evil.example", "192.0.2.9", store.SpamDecisionReject, store.SpamEngineRspamd, denied)
// hour 3, 4: the networks, v4 (as Postfix may log it, mapped) and v6.
row("x@elsewhere.example", "::ffff:198.51.100.77", store.SpamDecisionReject, store.SpamEngineRspamd, denied)
row("x@elsewhere.example", "2001:db8::1", store.SpamDecisionReject, store.SpamEngineRspamd, denied)
// hour 5: a subdomain is not the domain.
row("x@sub.evil.example", "192.0.2.9", store.SpamDecisionReject, store.SpamEngineRspamd, denied)
// hour 6: allowed by its address and its domain - two symbols, two entries.
row("billing@vendor.example", "192.0.2.50", store.SpamDecisionAccept, store.SpamEngineRspamd,
"no action, score -199.00 / 15.00; SELFPOST_ALLOW_DOMAIN(-100.00) SELFPOST_ALLOW_FROM(-100.00)")
// hour 7: allowed by its domain alone.
row("accounts@vendor.example", "192.0.2.50", store.SpamDecisionAccept, store.SpamEngineRspamd,
"no action, score -99.00 / 15.00; SELFPOST_ALLOW_DOMAIN(-100.00)")
// hour 8: allowed by the client address; the sender's domain is not why.
row("x@quiet.example", "203.0.113.5", store.SpamDecisionAccept, store.SpamEngineRspamd,
"no action, score -99.00 / 15.00; SELFPOST_ALLOW_IP(-100.00)")
// Nobody's: the filter's own refusal, an accept with no list symbol, a
// refusal by Postfix, and a sender who spells the mark in his own address.
row("spammer@evil.example", "192.0.2.9", store.SpamDecisionReject, store.SpamEngineRspamd, "5.7.1 Spam message rejected")
row("billing@vendor.example", "192.0.2.50", store.SpamDecisionAccept, store.SpamEngineRspamd, "no action, score 0.10 / 15.00")
row("spammer@evil.example", "198.51.100.1", store.SpamDecisionReject, store.SpamEnginePostfix, "550 5.1.1 <nobody@in.example.com>: this server's deny list")
row("billing@vendor.example", "203.0.113.5", store.SpamDecisionAccept, store.SpamEngineSelfPost, "")
usage, err := l.Usage()
if err != nil {
t.Fatalf("Usage: %v", err)
}
for name, c := range map[string]struct {
id int64
hits int
last int // hour
}{
"deny domain": {denyDomain, 2, 2}, "deny address": {denyAddr, 1, 1},
"deny network": {denyNet, 1, 3}, "deny v6 network": {denyV6, 1, 4},
"allow address": {allowAddr, 1, 6}, "allow domain": {allowDomain, 2, 7}, "allow ip": {allowIP, 1, 8},
"unused": {unused, 0, 0},
} {
got := usage[c.id]
wantLast := time.Time{}
if c.hits > 0 {
wantLast = at.Add(time.Duration(c.last) * time.Hour)
}
if got.Hits != c.hits || !got.Last.Equal(wantLast) {
t.Errorf("%s: %d hits, last %v; want %d, %v", name, got.Hits, got.Last, c.hits, wantLast)
}
}
}
func TestListsUsageWithNoEntries(t *testing.T) {
l, _, _ := testLists(t)
if usage, err := l.Usage(); err != nil || len(usage) != 0 {
t.Errorf("Usage = %v, %v", usage, err)
}
}
// The same address written as a network of one is the same entry.
func TestListsOneAddressNetworkIsTheAddress(t *testing.T) {
l, _, dir := testLists(t)
if _, err := l.Add(store.SpamListDeny, store.SpamEntryIP, "192.0.2.7", ""); err != nil {
t.Fatalf("Add: %v", err)
}
if _, err := l.Add(store.SpamListDeny, store.SpamEntryIP, "192.0.2.7/32", ""); !errors.Is(err, store.ErrSpamListEntryExists) {
t.Errorf("192.0.2.7/32 after 192.0.2.7: %v, want ErrSpamListEntryExists", err)
}
if got := strings.Join(mapLines(t, dir, "deny_ip.map"), " "); got != "192.0.2.7" {
t.Errorf("deny_ip.map = %q", got)
}
}
func TestListable(t *testing.T) {
for value, want := range map[string]bool{
"billing@vendor.example": true, "srs0=ab=cd=example.org=user@forwarder.example": true,
"o'brien@example.org": false, "a#b@example.org": false, "": false, "not-an-address": false,
} {
if got := Listable(store.SpamEntryAddress, value); got != want {
t.Errorf("Listable(address, %q) = %v, want %v", value, got, want)
}
}
}
// The key the filter signs with is made once and then read, and is readable by
// the filter, which is another user in another container.
func TestFilterKey(t *testing.T) {
dir := filepath.Join(t.TempDir(), "antispam")
key, err := FilterKey(dir)
if err != nil || len(key) != 64 || strings.Trim(key, "0123456789abcdef") != "" {
t.Fatalf("FilterKey = %q, %v", key, err)
}
again, err := FilterKey(dir)
if err != nil || again != key {
t.Errorf("a second call = %q, %v; want the same key", again, err)
}
raw, err := os.ReadFile(filepath.Join(dir, FilterKeyFile))
if err != nil || string(raw) != key+"\n" {
t.Errorf("file = %q, %v", raw, err)
}
if info, _ := os.Stat(filepath.Join(dir, FilterKeyFile)); os.PathSeparator == '/' && info.Mode().Perm() != 0o644 {
t.Errorf("mode = %o", info.Mode().Perm())
}
// A file that is not a key is replaced, not signed with.
if err := os.WriteFile(filepath.Join(dir, FilterKeyFile), []byte("short\n"), 0o644); err != nil {
t.Fatal(err)
}
fresh, err := FilterKey(dir)
if err != nil || len(fresh) != 64 || fresh == key {
t.Errorf("after a damaged file: %q, %v", fresh, err)
}
// The lists leave it alone.
st, err := store.Open(filepath.Join(t.TempDir(), "test.db"))
if err != nil {
t.Fatal(err)
}
defer st.Close()
if err := NewLists(st, dir).Sync(); err != nil {
t.Fatal(err)
}
if kept, _ := FilterKey(dir); kept != fresh {
t.Errorf("the key changed when the lists were written")
}
}
+300
View File
@@ -0,0 +1,300 @@
package logtail
import (
"context"
"log"
"regexp"
"strings"
"time"
"github.com/mixeme/selfpost/internal/store"
)
// InboundStore is what the tailer needs of the inbound antispam journal:
// writing the rows for refused mail, knowing which domains are inbound, and
// pruning. *store.Store satisfies it.
type InboundStore interface {
InsertInboundSpam(e store.InboundSpamEntry) error
InsertInboundSpamOnce(e store.InboundSpamEntry) (bool, error)
IsInboundDomain(name string) (bool, error)
PruneInboundSpam(cutoff time.Time, keep int) (int64, error)
}
// Inbound turns on the inbound half of the tailer: journal rows for mail that
// port 25 refused, and the journal's retention. Run takes nil when the inbound
// relay is off.
type Inbound struct {
Store InboundStore
Retention RetentionDays
}
// The inbound journal milter writes a row for mail that was accepted; mail
// that was refused never reaches it, and what is known about it is one line
// of mail.log:
//
// postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from mx.example.net[203.0.113.9]: 550 5.1.1 <nobody@in.example.com>: Recipient address rejected: User unknown in relay recipient table; from=<a@example.net> to=<nobody@in.example.com> proto=ESMTP helo=<mx.example.net>
// postfix/cleanup[52]: 4A1B2C3D: milter-reject: END-OF-MESSAGE from mx.example.net[203.0.113.9]: 5.7.1 Spam message rejected; from=<a@example.net> to=<b@in.example.com> proto=ESMTP helo=<mx.example.net>
//
// A message the filter holds (milter-hold) is not read here: it goes on to the
// inbound journal milter, which writes its row with the subject.
//
// The first is Postfix's own policy, written by the port-25 smtpd, which
// build/postfix-config.sh names postfix/inbound so that its lines are told
// from those of the submission ports. The second is the filter's verdict at
// end-of-message, which cleanup(8) writes under one name for every listener -
// so it counts only for a queue id the port-25 smtpd was seen to open
// (inboundClientRe).
//
// Both are matched from the head of the line, after the timestamp and host
// name: the tail of such a line is the sender's own words (HELO name,
// addresses), and a pattern that could start anywhere would let him write
// journal rows.
//
// For the same reason the reply is the shortest text before "; from=<", and
// the HELO name - the last thing on the line, which Postfix logs as the client
// wrote it, spaces and brackets included - is whatever is left. Taking the
// longest reply instead would let a HELO name that itself spells
// "; from=<…> to=<…> proto=…" supply the sender, the recipient and the reply
// of the row (security review, 2026-10-12).
var (
inboundRejectRe = regexp.MustCompile(`^\S+ postfix/(inbound/smtpd|cleanup)\[\d+\]: ([0-9A-Za-z]+): (reject|milter-reject|milter-discard): \S+ from [^\[\s]*\[([0-9A-Fa-f:.]+)\]: (.*?); from=<([^>]*)> to=<([^>]*)> proto=\S+(?: helo=<.*>)?$`)
inboundClientRe = regexp.MustCompile(`^\S+ postfix/inbound/smtpd\[\d+\]: ([0-9A-Za-z]+): client=`)
// A queue id stops being port 25's when the message is queued (qmgr takes
// it: the filter let it through) or when another listener opens a message
// under the same id - Postfix reuses ids.
inboundForgetRe = regexp.MustCompile(`^\S+ postfix/(?:qmgr\[\d+\]: ([0-9A-Za-z]+): from=|(?:[a-z0-9-]+/)?smtpd\[\d+\]: ([0-9A-Za-z]+): client=)`)
)
// A row holds one line of reason; Postfix's reply text carries the far end's
// bytes and is cut and reduced to printable ASCII like any of them.
const inboundDetailMax = 255
// inboundQueueIDs is how many port-25 queue ids are remembered while waiting
// for cleanup's verdict line on them. An id is forgotten when the verdict
// comes, when the message is queued, or when another listener reuses it; the
// cap bounds what is left - messages the client abandoned - and a flood.
const inboundQueueIDs = 4096
// inboundPruneEvery is how many rows the tailer writes before it has the
// journal cut back to its row cap, without waiting for the six-hourly prune.
// Refusals are written at the rate strangers send RCPT TO, which no limit of
// Postfix's counts, so between two prunes the journal would otherwise grow as
// fast as someone cared to make it.
const inboundPruneEvery = 500
// inboundJournal is the per-run state of the inbound half.
type inboundJournal struct {
st InboundStore
// port25 holds the queue ids the port-25 smtpd opened and nobody has
// accounted for yet, each with the number of its arrival; order is those
// arrivals, for dropping the oldest. An id that was forgotten and opened
// again is in order twice: the number tells the stale entry from the live
// one when the stale one comes up for dropping.
port25 map[string]uint64
order []queued
seq uint64
// written counts the rows since the journal was last cut to its cap.
written int
}
type queued struct {
id string
seq uint64
}
func newInboundJournal(st InboundStore) *inboundJournal {
return &inboundJournal{st: st, port25: make(map[string]uint64)}
}
// handle looks at one mail.log line and writes a journal row when it is a
// refusal of mail for an inbound domain. Errors are logged; the tailer goes on.
//
// catchingUp says the line is one of those the tailer reads when it starts,
// from where it last saved its place to the end of the file. It saves its
// place every few seconds, so after an unclean stop some of those lines were
// read before; a row is then written only if it is not there already. Lines
// read as they are written are always new, and two identical refusals in one
// second stay two rows.
func (j *inboundJournal) handle(line string, now time.Time, catchingUp bool) {
at, rest := lineTime(line, now)
if m := inboundClientRe.FindStringSubmatch(rest); m != nil {
j.remember(m[1])
return
}
if m := inboundForgetRe.FindStringSubmatch(rest); m != nil {
delete(j.port25, m[1]+m[2])
return
}
m := inboundRejectRe.FindStringSubmatch(rest)
if m == nil {
return
}
program, queueID, kind, clientIP, reason, from, to := m[1], m[2], m[3], m[4], m[5], m[6], m[7]
// The sender's address is his own text too, and with a quoted local part it
// may hold "> to=<…> proto=…" - which would end the sender early and supply
// the recipient. Such a line shows it: the sender is left with an unclosed
// quote. An address with a quote, a bracket or a space in it is not one the
// journal keeps.
if strings.ContainsAny(from, "\"<> ") || strings.ContainsAny(to, "\"<> ") {
return
}
if program == "cleanup" {
if _, ok := j.port25[queueID]; !ok {
return // a submission-port message, or one opened before this run
}
delete(j.port25, queueID)
}
// Only mail for an inbound domain is the relay's. "Relay access denied"
// for a stranger's domain is every scanner on the internet, and a line
// with no recipient (size limit at MAIL) belongs to no domain.
domain := ""
if i := strings.LastIndexByte(to, '@'); i >= 0 {
domain = strings.ToLower(to[i+1:])
}
if domain == "" {
return
}
ok, err := j.st.IsInboundDomain(domain)
if err != nil {
log.Printf("log-tailer: inbound journal: look up %s: %v", domain, err)
return
}
if !ok {
return
}
e := store.InboundSpamEntry{
InboundDomain: domain,
ClientIP: clientIP,
From: from,
To: to,
Decision: store.SpamDecisionReject,
Engine: store.SpamEngineRspamd,
Detail: printable(reason, inboundDetailMax),
At: at,
}
if kind == "reject" {
e.Engine = store.SpamEnginePostfix
}
// "450 4.7.1 …" from Postfix, "4.7.1 …" from a milter: either way a reply
// that starts with 4 asks the sender to come back.
if e.Decision == store.SpamDecisionReject && strings.HasPrefix(reason, "4") {
e.Decision = store.SpamDecisionTempfail
}
if catchingUp {
if wrote, err := j.st.InsertInboundSpamOnce(e); err != nil {
log.Printf("log-tailer: inbound journal: record %s -> %s: %v", from, to, err)
return
} else if !wrote {
return // read before the last stop
}
} else if err := j.st.InsertInboundSpam(e); err != nil {
log.Printf("log-tailer: inbound journal: record %s -> %s: %v", from, to, err)
return
}
if j.written++; j.written >= inboundPruneEvery {
j.written = 0
// The zero time removes nothing by age: this is the cap alone.
if _, err := j.st.PruneInboundSpam(time.Time{}, inboundSpamLogMaxRows); err != nil {
log.Printf("log-tailer: inbound journal: cut to %d rows: %v", inboundSpamLogMaxRows, err)
}
}
}
func (j *inboundJournal) remember(queueID string) {
if _, ok := j.port25[queueID]; ok {
return
}
for len(j.order) >= inboundQueueIDs {
oldest := j.order[0]
j.order = j.order[1:]
if j.port25[oldest.id] == oldest.seq {
delete(j.port25, oldest.id)
}
}
j.seq++
j.port25[queueID] = j.seq
j.order = append(j.order, queued{queueID, j.seq})
}
// printable cuts s to max bytes and replaces everything outside printable
// ASCII.
func printable(s string, max int) string {
if len(s) > max {
s = s[:max]
}
b := []byte(s)
for i, c := range b {
if c < 0x20 || c > 0x7e {
b[i] = '?'
}
}
return string(b)
}
// lineTime reads the time at the head of a mail.log line and returns it with
// the rest of the line. A refusal is dated by its line and not by when the
// tailer got to it: after the panel was down the two are hours apart. The
// syslog format carries neither year nor zone - the year is taken from now
// (a stamp that would then lie in the future is last year's), the zone is the
// container's, which is what postlogd wrote it in. A line with no readable
// stamp is dated now.
func lineTime(line string, now time.Time) (time.Time, string) {
if m := isoStampRe.FindStringSubmatch(line); m != nil {
rest := strings.TrimSpace(line[len(m[0]):])
if t, err := time.Parse(time.RFC3339Nano, strings.TrimSpace(m[0])); err == nil {
return t.UTC(), rest
}
return now, rest
}
if m := syslogStampRe.FindStringSubmatch(line); m != nil {
rest := strings.TrimSpace(line[len(m[0]):])
stamp := strings.Join(strings.Fields(m[1]), " ")
t, err := time.ParseInLocation("Jan 2 15:04:05", stamp, time.Local)
if err != nil {
return now, rest
}
t = t.AddDate(now.In(time.Local).Year(), 0, 0)
if t.After(now.Add(24 * time.Hour)) {
t = t.AddDate(-1, 0, 0)
}
return t.UTC(), rest
}
return now, line
}
// The journal keeps at most this many rows whatever their age: refusals are
// written at the rate strangers knock on port 25.
const inboundSpamLogMaxRows = 10000
// inboundRetentionLoop prunes the inbound journal at once and then on the
// cadence of the send log.
func inboundRetentionLoop(ctx context.Context, in *Inbound) {
prune := func() {
days := in.Retention()
if days <= 0 {
days = store.InboundSpamLogRetentionDaysDefault
}
n, err := in.Store.PruneInboundSpam(time.Now().UTC().AddDate(0, 0, -days), inboundSpamLogMaxRows)
if err != nil {
log.Printf("log-tailer: inbound journal prune: %v", err)
return
}
if n > 0 {
log.Printf("log-tailer: pruned %d inbound journal rows (older than %d days or beyond %d rows)", n, days, inboundSpamLogMaxRows)
}
}
prune()
t := time.NewTicker(retentionInterval)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
prune()
}
}
}
+333
View File
@@ -0,0 +1,333 @@
package logtail
import (
"context"
"errors"
"strings"
"testing"
"time"
"github.com/mixeme/selfpost/internal/store"
)
type fakeInboundStore struct {
domains map[string]bool
rows []store.InboundSpamEntry
lookupErr error
pruned []int
}
func (f *fakeInboundStore) InsertInboundSpam(e store.InboundSpamEntry) error {
f.rows = append(f.rows, e)
return nil
}
// InsertInboundSpamOnce writes unless an equal row is there, like the store's.
func (f *fakeInboundStore) InsertInboundSpamOnce(e store.InboundSpamEntry) (bool, error) {
for _, r := range f.rows {
if r == e {
return false, nil
}
}
f.rows = append(f.rows, e)
return true, nil
}
func (f *fakeInboundStore) IsInboundDomain(name string) (bool, error) {
if f.lookupErr != nil {
return false, f.lookupErr
}
return f.domains[name], nil
}
func (f *fakeInboundStore) PruneInboundSpam(_ time.Time, keep int) (int64, error) {
f.pruned = append(f.pruned, keep)
return 0, nil
}
func inboundFixture() (*inboundJournal, *fakeInboundStore) {
st := &fakeInboundStore{domains: map[string]bool{"in.example.com": true}}
return newInboundJournal(st), st
}
var tailNow = time.Date(2026, 10, 11, 18, 0, 0, 0, time.UTC)
const (
head = "2026-10-11T12:00:05.123456+00:00 mail "
unknownRcp = head + "postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from mx.example.net[203.0.113.9]: 550 5.1.1 <nobody@in.example.com>: Recipient address rejected: User unknown in relay recipient table; from=<a@example.net> to=<nobody@in.example.com> proto=ESMTP helo=<mx.example.net>"
)
// Postfix's own refusal of a recipient at an inbound domain is a row, dated by
// its line.
func TestInboundRejectByPostfix(t *testing.T) {
j, st := inboundFixture()
j.handle(unknownRcp, tailNow, false)
if len(st.rows) != 1 {
t.Fatalf("rows = %+v", st.rows)
}
want := store.InboundSpamEntry{
InboundDomain: "in.example.com", ClientIP: "203.0.113.9",
From: "a@example.net", To: "nobody@in.example.com",
Decision: store.SpamDecisionReject, Engine: store.SpamEnginePostfix,
Detail: "550 5.1.1 <nobody@in.example.com>: Recipient address rejected: User unknown in relay recipient table",
At: time.Date(2026, 10, 11, 12, 0, 5, 123456000, time.UTC),
}
if st.rows[0] != want {
t.Errorf("row = %+v\nwant %+v", st.rows[0], want)
}
}
func TestInboundTempfailByPostfix(t *testing.T) {
j, st := inboundFixture()
j.handle(head+"postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from unknown[2001:db8::9]: 450 4.1.8 <a@nx.example>: Sender address rejected: Domain not found; from=<a@nx.example> to=<B@IN.example.com> proto=ESMTP helo=<x>", tailNow, false)
if len(st.rows) != 1 || st.rows[0].Decision != store.SpamDecisionTempfail || st.rows[0].ClientIP != "2001:db8::9" || st.rows[0].InboundDomain != "in.example.com" {
t.Errorf("rows = %+v", st.rows)
}
}
// The filter's verdict is written by cleanup, under one name for every
// listener: it is the journal's only for a queue id port 25 opened.
func TestInboundMilterVerdictNeedsAPort25QueueID(t *testing.T) {
const verdict = "postfix/cleanup[52]: 4A1B2C3D: milter-reject: END-OF-MESSAGE from mx.example.net[203.0.113.9]: 5.7.1 Spam message rejected; from=<a@example.net> to=<b@in.example.com> proto=ESMTP helo=<mx.example.net>"
j, st := inboundFixture()
// Opened by a submission port: postfix/smtpd, not postfix/inbound/smtpd.
j.handle(head+"postfix/smtpd[20]: 4A1B2C3D: client=app.example.com[203.0.113.4], sasl_method=PLAIN, sasl_username=app", tailNow, false)
j.handle(head+verdict, tailNow, false)
if len(st.rows) != 0 {
t.Fatalf("a submission-port verdict was journalled: %+v", st.rows)
}
j.handle(head+"postfix/inbound/smtpd[41]: 4A1B2C3D: client=mx.example.net[203.0.113.9]", tailNow, false)
j.handle(head+verdict, tailNow, false)
if len(st.rows) != 1 {
t.Fatalf("rows = %+v", st.rows)
}
got := st.rows[0]
if got.Decision != store.SpamDecisionReject || got.Engine != store.SpamEngineRspamd || got.Detail != "5.7.1 Spam message rejected" {
t.Errorf("row = %+v", got)
}
// One verdict per message: the id is forgotten with it.
j.handle(head+verdict, tailNow, false)
if len(st.rows) != 1 {
t.Errorf("the same queue id was journalled twice")
}
}
func TestInboundMilterDecisions(t *testing.T) {
for kind, c := range map[string]struct{ reply, decision string }{
"milter-reject": {"4.7.1 Try again later", store.SpamDecisionTempfail},
"milter-discard": {"milter triggers DISCARD action", store.SpamDecisionReject},
} {
j, st := inboundFixture()
j.handle(head+"postfix/inbound/smtpd[41]: Q1: client=mx.example.net[203.0.113.9]", tailNow, false)
j.handle(head+"postfix/cleanup[52]: Q1: "+kind+": END-OF-MESSAGE from mx.example.net[203.0.113.9]: "+c.reply+"; from=<a@example.net> to=<b@in.example.com> proto=ESMTP helo=<mx.example.net>", tailNow, false)
if len(st.rows) != 1 || st.rows[0].Decision != c.decision || st.rows[0].Engine != store.SpamEngineRspamd {
t.Errorf("%s: rows = %+v", kind, st.rows)
}
}
}
// What is not the relay's mail, or is not a refusal at all, leaves no row.
func TestInboundIgnores(t *testing.T) {
for name, line := range map[string]string{
"another domain (relay access denied)": head + "postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from x[203.0.113.9]: 554 5.7.1 <c@elsewhere.example>: Relay access denied; from=<a@example.net> to=<c@elsewhere.example> proto=ESMTP helo=<x>",
"no recipient (size limit at MAIL)": head + "postfix/inbound/smtpd[41]: NOQUEUE: reject: MAIL from x[203.0.113.9]: 552 5.3.4 Message size exceeds fixed limit; proto=ESMTP helo=<x>",
"a submission port": head + "postfix/smtpd[20]: NOQUEUE: reject: RCPT from x[203.0.113.9]: 553 5.7.1 <b@in.example.com>: Sender address rejected: not owned by user app; from=<a@example.net> to=<b@in.example.com> proto=ESMTP helo=<x>",
"a delivery line": head + "postfix/smtp[26]: 41E862C00D9E: to=<b@in.example.com>, relay=mx[203.0.113.9]:25, dsn=2.0.0, status=sent (250 OK)",
"a connect line": head + "postfix/inbound/smtpd[41]: connect from mx.example.net[203.0.113.9]",
// The HELO name is the sender's; a line cannot be started from inside one.
"a refusal spelled inside another line": head + "postfix/inbound/smtpd[41]: warning: hostname x does not resolve: mail postfix/inbound/smtpd[1]: NOQUEUE: reject: RCPT from x[203.0.113.9]: 550 no; from=<a@example.net> to=<b@in.example.com> proto=ESMTP helo=<x>",
"an address with a bracket in it": head + "postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from x[203.0.113.9]: 554 5.7.1 <b@elsewhere.example>: Relay access denied; from=<\"a> to=<v@in.example.com> proto=X helo=<y\"@example.net> to=<b@elsewhere.example> proto=ESMTP helo=<x>",
} {
j, st := inboundFixture()
j.handle(line, tailNow, false)
if len(st.rows) != 0 {
t.Errorf("%s: journalled %+v", name, st.rows)
}
}
}
func TestInboundDetailIsCutAndPrintable(t *testing.T) {
j, st := inboundFixture()
j.handle(head+"postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from x[203.0.113.9]: 550 5.7.1 caf\xc3\xa9 \x1b[31m"+strings.Repeat("x", 400)+"; from=<a@example.net> to=<b@in.example.com> proto=ESMTP helo=<x>", tailNow, false)
if len(st.rows) != 1 {
t.Fatalf("rows = %+v", st.rows)
}
d := st.rows[0].Detail
if len(d) != inboundDetailMax || !strings.HasPrefix(d, "550 5.7.1 caf?? ?[31mxxx") {
t.Errorf("detail (%d bytes) = %q", len(d), d)
}
}
func TestInboundLookupFailureLeavesNoRow(t *testing.T) {
j, st := inboundFixture()
st.lookupErr = errors.New("boom")
j.handle(unknownRcp, tailNow, false)
if len(st.rows) != 0 {
t.Errorf("rows = %+v", st.rows)
}
}
// A message the filter holds goes on to the inbound journal milter, which
// writes its row; a second one from the log would be the same message twice.
func TestInboundHoldIsTheMiltersRow(t *testing.T) {
j, st := inboundFixture()
j.handle(head+"postfix/inbound/smtpd[41]: Q1: client=mx.example.net[203.0.113.9]", tailNow, false)
j.handle(head+"postfix/cleanup[52]: Q1: milter-hold: END-OF-MESSAGE from mx.example.net[203.0.113.9]: milter triggers HOLD action; from=<a@example.net> to=<b@in.example.com> proto=ESMTP helo=<mx.example.net>", tailNow, false)
if len(st.rows) != 0 {
t.Errorf("rows = %+v", st.rows)
}
}
// The HELO name is the last thing on the line and is logged as the client
// wrote it. It cannot supply the row: the reply ends at the first "; from=<",
// which is Postfix's own.
func TestInboundHeloCannotWriteTheRow(t *testing.T) {
j, st := inboundFixture()
j.handle(head+"postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from x[203.0.113.9]: 550 5.1.1 <nobody@in.example.com>: Recipient address rejected: User unknown in relay recipient table; from=<real@example.net> to=<nobody@in.example.com> proto=ESMTP helo=<x 5.7.1 Refused by this server's deny list; from=<victim@bank.example> to=<ceo@in.example.com> proto=ESMTP helo=<y>", tailNow, false)
if len(st.rows) != 1 {
t.Fatalf("rows = %+v", st.rows)
}
got := st.rows[0]
if got.From != "real@example.net" || got.To != "nobody@in.example.com" ||
got.Detail != "550 5.1.1 <nobody@in.example.com>: Recipient address rejected: User unknown in relay recipient table" {
t.Errorf("row = %+v", got)
}
// A HELO name with a closing bracket in it does not lose the line either.
j, st = inboundFixture()
j.handle(head+"postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from x[203.0.113.9]: 550 5.1.1 <nobody@in.example.com>: no; from=<real@example.net> to=<nobody@in.example.com> proto=ESMTP helo=<a>b>", tailNow, false)
if len(st.rows) != 1 || st.rows[0].From != "real@example.net" {
t.Errorf("rows = %+v", st.rows)
}
// The same on a verdict line: the filter's reply is not extended by the
// HELO name, so a stranger cannot make a refusal look like the deny list's.
j, st = inboundFixture()
j.handle(head+"postfix/inbound/smtpd[41]: Q1: client=x[203.0.113.9]", tailNow, false)
j.handle(head+"postfix/cleanup[52]: Q1: milter-reject: END-OF-MESSAGE from x[203.0.113.9]: 5.7.1 Spam message rejected; from=<real@example.net> to=<b@in.example.com> proto=ESMTP helo=<x this server's deny list; from=<spoofed@example.org> to=<b@in.example.com> proto=ESMTP helo=<y>", tailNow, false)
if len(st.rows) != 1 || st.rows[0].Detail != "5.7.1 Spam message rejected" || st.rows[0].From != "real@example.net" {
t.Errorf("rows = %+v", st.rows)
}
}
// A queue id is port 25's only until the message is queued or the id is
// opened again by another listener - Postfix reuses ids.
func TestInboundQueueIDIsForgotten(t *testing.T) {
const verdict = "postfix/cleanup[52]: Q1: milter-reject: END-OF-MESSAGE from app[203.0.113.4]: 4.7.1 Service unavailable - try again later; from=<a@example.net> to=<b@in.example.com> proto=ESMTP helo=<app>"
for name, forget := range map[string]string{
"queued": "postfix/qmgr[58]: Q1: from=<a@example.net>, size=600, nrcpt=1 (queue active)",
"reused by the submission port": "postfix/smtpd[20]: Q1: client=app.example.com[203.0.113.4], sasl_method=PLAIN, sasl_username=app",
} {
j, st := inboundFixture()
j.handle(head+"postfix/inbound/smtpd[41]: Q1: client=mx.example.net[203.0.113.9]", tailNow, false)
j.handle(head+forget, tailNow, false)
j.handle(head+verdict, tailNow, false)
if len(st.rows) != 0 || len(j.port25) != 0 {
t.Errorf("%s: rows %+v, remembered %v", name, st.rows, j.port25)
}
}
}
// An id that was forgotten and opened again is not dropped by its own stale
// place in the queue of arrivals.
func TestInboundStaleArrivalDoesNotDropALiveID(t *testing.T) {
j, _ := inboundFixture()
j.remember("Q1")
delete(j.port25, "Q1") // its verdict came
for i := 0; i < inboundQueueIDs-2; i++ {
j.remember("F" + time.Duration(i).String())
}
j.remember("Q1") // reused; order is now full, with the stale Q1 first
j.remember("LAST")
if _, ok := j.port25["Q1"]; !ok {
t.Error("the live Q1 was dropped by its stale arrival")
}
}
// Between the six-hourly prunes the tailer has the journal cut to its cap
// itself, every so many rows: refusals come at the rate strangers send them.
func TestInboundCutsTheJournalAsItWrites(t *testing.T) {
j, st := inboundFixture()
for i := 0; i < inboundPruneEvery*2+10; i++ {
j.handle(unknownRcp, tailNow, false)
}
if len(st.pruned) != 2 || st.pruned[0] != inboundSpamLogMaxRows {
t.Errorf("cuts = %v, want two, to %d rows", st.pruned, inboundSpamLogMaxRows)
}
}
// The queue ids waiting for a verdict are bounded.
func TestInboundQueueIDsAreBounded(t *testing.T) {
j, _ := inboundFixture()
for i := 0; i < inboundQueueIDs+10; i++ {
j.remember("Q" + strings.Repeat("A", i%7) + string(rune('0'+i%10)) + time.Duration(i).String())
}
if len(j.port25) != inboundQueueIDs || len(j.order) != inboundQueueIDs {
t.Errorf("remembered %d ids (%d in order), want %d", len(j.port25), len(j.order), inboundQueueIDs)
}
}
func TestLineTime(t *testing.T) {
now := time.Date(2026, 1, 2, 10, 0, 0, 0, time.UTC)
for _, c := range []struct {
line string
want time.Time
rest string
}{
{"2026-01-02T09:00:00.5+03:00 mail postfix/x: y", time.Date(2026, 1, 2, 6, 0, 0, 500000000, time.UTC), "mail postfix/x: y"},
{"Jan 2 09:30:00 mail postfix/x: y", time.Date(2026, 1, 2, 9, 30, 0, 0, time.Local).UTC(), "mail postfix/x: y"},
// December read in January is last year's.
{"Dec 31 23:59:00 mail postfix/x: y", time.Date(2025, 12, 31, 23, 59, 0, 0, time.Local).UTC(), "mail postfix/x: y"},
{"no stamp here", now, "no stamp here"},
} {
got, rest := lineTime(c.line, now)
if !got.Equal(c.want) || rest != c.rest {
t.Errorf("lineTime(%q) = %v, %q; want %v, %q", c.line, got, rest, c.want, c.rest)
}
}
}
func TestInboundRetentionPrunesAtStart(t *testing.T) {
old := retentionInterval
retentionInterval = time.Hour
defer func() { retentionInterval = old }()
st := &fakeInboundStore{}
ctx, cancel := contextWithCancel()
done := make(chan struct{})
go func() {
inboundRetentionLoop(ctx, &Inbound{Store: st, Retention: func() int { return 0 }})
close(done)
}()
deadline := time.Now().Add(2 * time.Second)
for len(st.pruned) == 0 && time.Now().Before(deadline) {
time.Sleep(10 * time.Millisecond)
}
cancel()
<-done
if len(st.pruned) != 1 || st.pruned[0] != inboundSpamLogMaxRows {
t.Errorf("prune calls = %v", st.pruned)
}
}
func contextWithCancel() (context.Context, context.CancelFunc) {
return context.WithCancel(context.Background())
}
// The lines the tailer reads when it starts may be ones it read before its
// last stop: a refusal among them is one row, not two. Lines read as they are
// written are always new.
func TestInboundCatchUpDoesNotWriteARowTwice(t *testing.T) {
j, st := inboundFixture()
j.handle(unknownRcp, tailNow, false) // before the stop
j.handle(unknownRcp, tailNow, true) // the same line, read again on start
if len(st.rows) != 1 {
t.Fatalf("rows = %d, want the one refusal", len(st.rows))
}
j.handle(unknownRcp, tailNow, false) // a second refusal in the same second
if len(st.rows) != 2 {
t.Errorf("rows = %d, want two: a line read as it is written is new", len(st.rows))
}
}
+160
View File
@@ -0,0 +1,160 @@
package logtail
import (
"log"
"regexp"
"time"
"github.com/mixeme/selfpost/internal/store"
)
// A message's page shows what Postfix wrote about it. mail.log is rotated
// daily and fourteen files are kept, while a send-log row lives for the
// retention window, so the lines are kept in the database as the tailer reads
// them (architecture.md § Log tailer).
//
// queueLineRe finds the queue id at the head of a line, after the timestamp
// (already taken off by lineTime), the host name and "program[pid]:":
//
// mail postfix/smtp[26]: 41E862C00D9E: to=<a@example.net>, relay=…, status=sent (250 OK)
//
// It is matched from the start of the line and nowhere else: the tail of a
// line is the far end's words - a reply, a HELO name - and a pattern that
// could start anywhere would let them file a line under a queue id of their
// choosing.
//
// What stands in that place is not always a queue id: "NOQUEUE:" and
// "warning:" stand there too. A queue id is hexadecimal in capitals, or, with
// enable_long_queue_ids, digits and letters without vowels; neither word is
// either.
var queueLineRe = regexp.MustCompile(`^\S+ ([^\s\[]+)\[\d+\]: ([0-9A-F]{6,24}|[0-9B-DF-HJ-NP-TV-Zb-df-hj-np-tv-z]{10,24}): (.*)`)
// smtpdProgramRe tells an smtpd from the other programs, whatever listener it
// serves: "postfix/smtpd", "postfix/submission/smtpd", "postfix/inbound/smtpd".
var smtpdProgramRe = regexp.MustCompile(`^postfix/(?:[a-z0-9-]+/)?smtpd$`)
// port25Program is the smtpd of the inbound relay's listener, which
// build/postfix-config.sh names so that its lines are told from those of the
// submission ports.
const port25Program = "postfix/inbound/smtpd"
const (
// lineGrace is how old a line must be before the prune asks whether a
// send-log row carries its queue id. The first lines of a message are
// logged when it arrives and its row is written at end-of-message; an hour
// is a message body sent slowly.
lineGrace = time.Hour
// lineBatch is how many lines are held before they are written without
// waiting for the end of the pass over the log. A pass is a second's worth
// of lines while following, and the whole backlog when the tailer starts.
lineBatch = 500
// port25IDs is how many queue ids opened on port 25 are remembered so that
// their lines are not kept. Beyond it the oldest knowledge is given up, and
// those messages' lines are kept until the prune finds them without a row.
port25IDs = 4096
)
// lineKeeper collects the lines headed by a queue id and writes them in
// batches. It is used from the follow loop only, so it needs no locking.
type lineKeeper struct {
st StatusStore
// buf holds the lines not yet written; once says they were read while
// catching up and may have been read before the last stop.
buf []store.SendLogLine
once bool
// port25 holds the queue ids the port-25 smtpd opened: mail offered by
// strangers, which never gets a send-log row. Its lines are not kept - what
// a stranger can make Postfix log is not something to store on his say-so
// and prune later. An id leaves when qmgr removes the message or when
// another listener opens a message under it (Postfix reuses ids).
port25 map[string]struct{}
}
func newLineKeeper(st StatusStore) *lineKeeper {
return &lineKeeper{st: st, port25: make(map[string]struct{})}
}
// add takes one mail.log line and keeps it if it is headed by the queue id of
// a message that did not come in on port 25. It does not ask whether a
// send-log row exists for the id: for a message's first lines none does yet,
// and the lines of messages that never get one - the bounces Postfix
// generates, port-25 mail met in mid-flight after a restart - are removed by
// pruneLines.
//
// catchingUp is as in inboundJournal.handle: the line may have been read
// before the last stop, and is then written only if it is not there already.
// The line is dated by its own stamp - after the panel was down, that and now
// are hours apart, and a line is matched to its row by time.
func (k *lineKeeper) add(line string, now time.Time, catchingUp bool) {
at, rest := lineTime(line, now)
m := queueLineRe.FindStringSubmatch(rest)
if m == nil {
return
}
program, queueID, text := m[1], m[2], m[3]
if smtpdProgramRe.MatchString(program) && len(text) >= 7 && text[:7] == "client=" {
if program == port25Program {
k.rememberPort25(queueID)
} else {
delete(k.port25, queueID)
}
}
if _, inbound := k.port25[queueID]; inbound {
if program == "postfix/qmgr" && text == "removed" {
delete(k.port25, queueID)
}
return
}
if catchingUp != k.once {
k.flush()
k.once = catchingUp
}
k.buf = append(k.buf, store.SendLogLine{QueueID: queueID, At: at, Line: line})
if len(k.buf) >= lineBatch {
k.flush()
}
}
func (k *lineKeeper) rememberPort25(queueID string) {
if len(k.port25) >= port25IDs {
// Which one goes does not matter: the cost of forgetting an id is its
// remaining lines kept until the prune.
for id := range k.port25 {
delete(k.port25, id)
break
}
}
k.port25[queueID] = struct{}{}
}
// flush writes the lines held. The follow loop calls it after every pass over
// the log and before it saves its place, so the place saved is never ahead of
// what was written. Lines that cannot be written are dropped and said so: the
// tailer goes on, and a page without them falls back to the log file.
func (k *lineKeeper) flush() {
if len(k.buf) == 0 {
return
}
if _, err := k.st.InsertSendLogLines(k.buf, k.once); err != nil {
log.Printf("log-tailer: keep %d log lines: %v", len(k.buf), err)
}
k.buf = k.buf[:0]
}
// pruneLines removes the stored lines no send-log row carries the queue id of:
// those of rows the retention sweep deleted, and those of messages that never
// had a row.
func pruneLines(st StatusStore) {
n, err := st.PruneSendLogLines(time.Now().UTC().Add(-lineGrace))
if err != nil {
log.Printf("log-tailer: prune stored log lines: %v", err)
return
}
if n > 0 {
log.Printf("log-tailer: pruned %d stored log lines with no send-log row", n)
}
}
+230
View File
@@ -0,0 +1,230 @@
package logtail
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// Which lines are kept for a message's page, and under which queue id: the id
// at the head of the line and nothing the far end wrote further along it.
func TestQueueLineHead(t *testing.T) {
cases := []struct {
name, rest, want string
}{
{"delivery", "mail postfix/smtp[26]: 41E862C00D9E: to=<a@example.net>, relay=mx.example.net[203.0.113.9]:25, dsn=2.0.0, status=sent (250 OK)", "41E862C00D9E"},
{"submission smtpd opens the id", "mail postfix/submission/smtpd[10]: 41E862C00D9E: client=unknown[203.0.113.7], sasl_method=PLAIN, sasl_username=app", "41E862C00D9E"},
{"port-25 smtpd", "mail postfix/inbound/smtpd[41]: 4A1B2C3D: client=mx.example.net[203.0.113.9]", "4A1B2C3D"},
{"cleanup", "mail postfix/cleanup[52]: 41E862C00D9E: message-id=<1@example.com>", "41E862C00D9E"},
{"qmgr removes it", "mail postfix/qmgr[10]: 41E862C00D9E: removed", "41E862C00D9E"},
{"a bounce is sent: the id at the head, not the new message's", "mail postfix/bounce[30]: 41E862C00D9E: sender non-delivery notification: 5F00AA12", "41E862C00D9E"},
{"long queue id", "mail postfix/qmgr[10]: 3sJ8Xh2Cgqz4Xy1: removed", "3sJ8Xh2Cgqz4Xy1"},
{"refused before queueing", "mail postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from mx.example.net[203.0.113.9]: 550 5.1.1 <n@in.example.com>: Recipient address rejected; from=<a@example.net> to=<n@in.example.com> proto=ESMTP helo=<mx.example.net>", ""},
{"a warning", "mail postfix/smtpd[10]: warning: hostname mx.example.net does not resolve to address 203.0.113.9", ""},
{"a connection", "mail postfix/smtpd[10]: connect from unknown[203.0.113.7]", ""},
{"no program", "preexisting line, ignored on start", ""},
{
// The reply is the receiving server's text. One that spells the head
// of a line is still the tail of this one: the line is the message's
// it was written about, and no other's.
"a reply that quotes a line head",
"mail postfix/smtp[26]: 9F1A2C00D9E: to=<a@example.net>, relay=mx.example.net[203.0.113.9]:25, dsn=5.1.1, status=bounced (host said: 550 mail postfix/smtp[1]: 41E862C00D9E: to=<b@example.net>, status=sent)",
"9F1A2C00D9E",
},
}
for _, c := range cases {
got := ""
if m := queueLineRe.FindStringSubmatch(c.rest); m != nil {
got = m[2]
}
if got != c.want {
t.Errorf("%s: queue id %q, want %q", c.name, got, c.want)
}
}
}
// A line is dated by its own stamp, not by when the tailer got to it, and is
// kept as the log has it. Nothing is written until the pass over the log ends.
func TestLineKeeperDatesByTheLine(t *testing.T) {
cs := &captureStore{}
k := newLineKeeper(cs)
now := time.Date(2026, 10, 12, 18, 0, 0, 0, time.UTC)
line := "Oct 12 09:00:00 mail postfix/qmgr[7]: 41E862C00D9E: removed"
k.add(line, now, false)
k.add("Oct 12 09:00:01 mail postfix/smtpd[7]: connect from unknown[203.0.113.7]", now, false)
if len(cs.lines) != 0 {
t.Fatalf("written before the pass ended: %v", cs.kept())
}
k.flush()
if len(cs.lines) != 1 {
t.Fatalf("kept %d lines, want 1: %v", len(cs.lines), cs.kept())
}
got := cs.lines[0]
if got.QueueID != "41E862C00D9E" || got.Line != line {
t.Errorf("kept %q under %q", got.Line, got.QueueID)
}
if want := time.Date(2026, 10, 12, 9, 0, 0, 0, time.Local).UTC(); !got.At.Equal(want) {
t.Errorf("dated %v, want %v", got.At, want)
}
}
// A backlog is written in batches of lineBatch, not held whole, and a pass
// that changes from catching up to following does not write the later lines
// under the earlier rule.
func TestLineKeeperWritesInBatches(t *testing.T) {
cs := &captureStore{}
k := newLineKeeper(cs)
now := time.Date(2026, 10, 12, 18, 0, 0, 0, time.UTC)
line := func(i int) string {
return "host postfix/qmgr[7]: 41E862C00D9E: from=<a@example.com>, size=" + itoa(i) + ", nrcpt=1 (queue active)"
}
for i := 0; i < lineBatch+10; i++ {
k.add(line(i), now, true)
}
if len(cs.lines) != lineBatch || cs.batches != 1 {
t.Fatalf("after %d lines: %d written in %d batches, want %d in 1", lineBatch+10, len(cs.lines), cs.batches, lineBatch)
}
// The same line again, now as one read while following: it is a new line,
// and the ten still held from the catch-up are written first, under theirs.
k.add(line(0), now, false)
k.flush()
if got, want := len(cs.lines), lineBatch+11; got != want {
t.Errorf("%d lines written, want %d", got, want)
}
if cs.batches != 3 {
t.Errorf("%d batches, want 3", cs.batches)
}
}
// Mail that comes in on port 25 is a stranger's and never gets a send-log row:
// none of its lines are kept, from the smtpd that opened the queue id to
// qmgr's "removed". The id is then anybody's again.
func TestLineKeeperLeavesOutPort25Mail(t *testing.T) {
cs := &captureStore{}
k := newLineKeeper(cs)
now := time.Date(2026, 10, 12, 18, 0, 0, 0, time.UTC)
const (
outClient = "host postfix/submission/smtpd[10]: 41E862C00D9E: client=app.example.com[203.0.113.7], sasl_method=PLAIN, sasl_username=app"
outSent = "host postfix/smtp[26]: 41E862C00D9E: to=<a@example.net>, relay=mx.example.net[203.0.113.9]:25, dsn=2.0.0, status=sent (250 OK)"
reused = "host postfix/submission/smtpd[10]: 4A1B2C3D: client=app.example.com[203.0.113.7], sasl_method=PLAIN, sasl_username=app"
bounce = "host postfix/cleanup[52]: 5F00AA12: message-id=<20261012180000.5F00AA12@mail.example.com>"
)
for _, line := range []string{
"host postfix/inbound/smtpd[41]: 4A1B2C3D: client=mx.example.net[198.51.100.9]",
outClient,
"host postfix/cleanup[52]: 4A1B2C3D: message-id=<whatever-the-stranger-wrote@example.net>",
"host postfix/qmgr[10]: 4A1B2C3D: from=<a@example.net>, size=900, nrcpt=1 (queue active)",
outSent,
"host postfix/smtp[27]: 4A1B2C3D: to=<b@in.example.com>, relay=upstream.example.com[192.0.2.1]:25, dsn=2.0.0, status=sent (250 OK)",
"host postfix/qmgr[10]: 4A1B2C3D: removed",
// The id comes round again for a submitted message, and a bounce has no
// smtpd at all: both are kept.
reused,
bounce,
} {
k.add(line, now, false)
}
k.flush()
want := []string{"41E862C00D9E|" + outClient, "41E862C00D9E|" + outSent, "4A1B2C3D|" + reused, "5F00AA12|" + bounce}
if got := cs.kept(); strings.Join(got, "\n") != strings.Join(want, "\n") {
t.Errorf("kept:\n%s\nwant:\n%s", strings.Join(got, "\n"), strings.Join(want, "\n"))
}
// A port-25 id that a submission listener opens before qmgr's "removed" was
// seen (the line was lost, the panel restarted) is the submitted message's.
k.add("host postfix/inbound/smtpd[41]: 6C00BB34: client=mx.example.net[198.51.100.9]", now, false)
again := "host postfix/smtps/smtpd[11]: 6C00BB34: client=app.example.com[203.0.113.7], sasl_method=PLAIN, sasl_username=app"
k.add(again, now, false)
k.flush()
if got := cs.kept(); got[len(got)-1] != "6C00BB34|"+again {
t.Errorf("a reused port-25 id was not kept for the submitted message: %v", got[len(got)-1])
}
// What is remembered of port 25 has a ceiling.
for i := 0; i < port25IDs+50; i++ {
k.rememberPort25("7D" + itoa(100000+i))
}
if len(k.port25) > port25IDs {
t.Errorf("%d port-25 ids remembered, ceiling is %d", len(k.port25), port25IDs)
}
}
// The tailer keeps queue-id lines as it follows the log, and after an unclean
// stop - its saved place lies before lines it had already read - it does not
// keep them a second time.
func TestRunKeepsQueueLinesOnce(t *testing.T) {
old := pollInterval
pollInterval = 10 * time.Millisecond
t.Cleanup(func() { pollInterval = old })
dir := t.TempDir()
path := filepath.Join(dir, "mail.log")
// Longer than a fingerprint, so the file is recognised across the restart.
seed := strings.Repeat("host postfix/smtpd[1]: connect from unknown[203.0.113.7]\n", 20)
if err := os.WriteFile(path, []byte(seed), 0o644); err != nil {
t.Fatalf("seed log: %v", err)
}
const (
client = "host postfix/submission/smtpd[1]: 41E862C00D9E: client=unknown[203.0.113.7]"
sent = "host postfix/smtp[1]: 41E862C00D9E: to=<a@example.net>, dsn=2.0.0, status=sent (ok)"
later = "host postfix/qmgr[1]: 41E862C00D9E: removed"
)
cs := &captureStore{}
stop := startRun(t, path, cs)
appendLine(t, path, client)
appendLine(t, path, "host postfix/smtpd[1]: disconnect from unknown[203.0.113.7]")
appendLine(t, path, sent)
waitFor(t, func() bool { return len(cs.kept()) == 2 })
stop()
// The process died before it saved its place: what it has is the place it
// was at before those lines.
cs.mu.Lock()
cs.state.Offset = int64(len(seed))
cs.mu.Unlock()
appendLine(t, path, later)
stop = startRun(t, path, cs)
waitFor(t, func() bool { return len(cs.kept()) >= 3 })
stop()
want := []string{"41E862C00D9E|" + client, "41E862C00D9E|" + sent, "41E862C00D9E|" + later}
if got := cs.kept(); strings.Join(got, "\n") != strings.Join(want, "\n") {
t.Errorf("kept:\n%s\nwant:\n%s", strings.Join(got, "\n"), strings.Join(want, "\n"))
}
}
// Lines without a send-log row are removed at the start and after every
// send-log sweep, and only once they are old enough for the row to have been
// written.
func TestRetentionLoopPrunesLines(t *testing.T) {
old := retentionInterval
retentionInterval = 20 * time.Millisecond
t.Cleanup(func() { retentionInterval = old })
cs := &captureStore{}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go retentionLoop(ctx, cs, func() int { return 90 })
waitFor(t, func() bool { return cs.prunes() >= 3 })
cs.mu.Lock()
defer cs.mu.Unlock()
for _, cutoff := range cs.pruned {
if age := time.Since(cutoff); age < lineGrace || age > lineGrace+time.Minute {
t.Errorf("prune cutoff %v ago, want %v", age, lineGrace)
}
}
}
+45 -8
View File
@@ -4,7 +4,8 @@
// Postfix only decides sent / deferred / bounced later, per recipient, and
// reports it in mail.log. This package parses those lines by queue-id +
// recipient and advances the matching rows, and prunes rows past the retention
// window.
// window. It also keeps every line Postfix wrote under a queue id, so that a
// message's page can show them after the log has rotated (lines.go).
package logtail
import (
@@ -25,12 +26,14 @@ import (
// StatusStore is the slice of the store the log-tailer needs: advancing
// delivery statuses, finding the rows still waiting for one, pruning the
// retention window, and remembering how far into mail.log it has read.
// *store.Store satisfies it.
// retention window, keeping the lines of queued messages, and remembering how
// far into mail.log it has read. *store.Store satisfies it.
type StatusStore interface {
UpdateStatus(queueID, recipient, status string) (int64, error)
ListQueuedOlderThan(cutoff time.Time) ([]store.QueuedDelivery, error)
DeleteSendLogBefore(cutoff time.Time) (int64, error)
InsertSendLogLines(lines []store.SendLogLine, once bool) (int, error)
PruneSendLogLines(cutoff time.Time) (int64, error)
LogtailState(path string) (store.LogtailState, bool, error)
SaveLogtailState(path string, st store.LogtailState) error
}
@@ -107,9 +110,18 @@ func parseDelivery(line string) (queueID, recipient, status string, ok bool) {
// Run follows path and updates send-log statuses until ctx is cancelled, while
// a background sweep prunes rows older than retention(). Reading resumes at
// the offset the previous run persisted, so a restart parses the delivery lines
// written while the panel was down. It returns nil on a clean shutdown.
func Run(ctx context.Context, path string, st StatusStore, retention RetentionDays) error {
// written while the panel was down. The lines headed by a queue id are kept
// for the message's page, except those of mail that came in on port 25
// (lines.go). With inbound set it also writes the
// inbound journal's rows for refused mail and prunes that journal
// (inbound.go). It returns nil on a clean shutdown.
func Run(ctx context.Context, path string, st StatusStore, retention RetentionDays, inbound *Inbound) error {
go retentionLoop(ctx, st, retention)
var journal *inboundJournal
if inbound != nil {
go inboundRetentionLoop(ctx, inbound)
journal = newInboundJournal(inbound.Store)
}
// The reconcile sweep must not run against a backlog the tailer has not
// read yet: on a restart the log holds the very lines that resolve the rows
@@ -118,7 +130,22 @@ func Run(ctx context.Context, path string, st StatusStore, retention RetentionDa
caughtUp := make(chan struct{})
go reconcileLoop(ctx, st, caughtUp)
return follow(ctx, path, &tracker{st: st, path: path}, caughtUp, func(line string) {
keeper := newLineKeeper(st)
return follow(ctx, path, &tracker{st: st, path: path}, caughtUp, keeper.flush, func(line string) {
// Until the first read reaches end-of-file the lines are those written
// while the panel was down - and, after an unclean stop, some it had
// read already.
catchingUp := true
select {
case <-caughtUp:
catchingUp = false
default:
}
now := time.Now().UTC()
if journal != nil {
journal.handle(line, now, catchingUp)
}
keeper.add(line, now, catchingUp)
queueID, recipient, status, ok := parseDelivery(line)
if !ok {
return
@@ -198,7 +225,10 @@ func reconcile(st StatusStore, cutoff time.Time) {
}
}
// retentionLoop prunes expired send-log rows immediately and then periodically.
// retentionLoop prunes expired send-log rows immediately and then periodically,
// and after each sweep the stored log lines no row carries the queue id of:
// those the deleted rows leave behind, and those that never had a row
// (lines.go).
func retentionLoop(ctx context.Context, st StatusStore, retention RetentionDays) {
prune := func() {
retentionDays := retention()
@@ -217,6 +247,7 @@ func retentionLoop(ctx context.Context, st StatusStore, retention RetentionDays)
}
prune()
pruneLines(st)
t := time.NewTicker(retentionInterval)
defer t.Stop()
for {
@@ -225,6 +256,7 @@ func retentionLoop(ctx context.Context, st StatusStore, retention RetentionDays)
return
case <-t.C:
prune()
pruneLines(st)
}
}
}
@@ -426,7 +458,11 @@ func SplitTimestamp(line string) (stamp, rest string) {
// caughtUp is closed after the first read that reaches end-of-file, which is
// the point where every line the panel missed while it was down has been
// handled.
func follow(ctx context.Context, path string, tr *tracker, caughtUp chan struct{}, handle func(string)) error {
//
// flush is called at the end of every pass over the file, before the read
// position is saved: a handler that gathers lines to write them together
// writes them there, so the saved position is never ahead of what was written.
func follow(ctx context.Context, path string, tr *tracker, caughtUp chan struct{}, flush func(), handle func(string)) error {
var (
f *os.File
r *bufio.Reader
@@ -476,6 +512,7 @@ func follow(ctx context.Context, path string, tr *tracker, caughtUp chan struct{
}()
drain := func() {
defer flush()
for {
line, err := r.ReadString('\n')
if err == io.EOF {
+54 -2
View File
@@ -147,6 +147,58 @@ type captureStore struct {
// queued is what ListQueuedOlderThan returns, for the reconcile sweep.
queued []store.QueuedDelivery
queuedErr error
// lines are the log lines kept for messages' pages; pruned is the cutoff of
// each PruneSendLogLines call.
lines []store.SendLogLine
batches int
pruned []time.Time
}
// InsertSendLogLines keeps what the store would: with once, a line is not
// kept a second time under its queue id. batches counts the calls.
func (c *captureStore) InsertSendLogLines(lines []store.SendLogLine, once bool) (int, error) {
c.mu.Lock()
defer c.mu.Unlock()
c.batches++
wrote := 0
next:
for _, l := range lines {
if once {
for _, have := range c.lines {
if have.QueueID == l.QueueID && have.Line == l.Line {
continue next
}
}
}
c.lines = append(c.lines, l)
wrote++
}
return wrote, nil
}
func (c *captureStore) PruneSendLogLines(cutoff time.Time) (int64, error) {
c.mu.Lock()
defer c.mu.Unlock()
c.pruned = append(c.pruned, cutoff)
return 0, nil
}
// kept returns the stored lines as "queue-id|line".
func (c *captureStore) kept() []string {
c.mu.Lock()
defer c.mu.Unlock()
out := make([]string, len(c.lines))
for i, l := range c.lines {
out[i] = l.QueueID + "|" + l.Line
}
return out
}
func (c *captureStore) prunes() int {
c.mu.Lock()
defer c.mu.Unlock()
return len(c.pruned)
}
func (c *captureStore) UpdateStatus(queueID, recipient, status string) (int64, error) {
@@ -212,7 +264,7 @@ func TestFollowTailsAndRotates(t *testing.T) {
cs := &captureStore{}
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { done <- Run(ctx, path, cs, func() int { return 90 }) }()
go func() { done <- Run(ctx, path, cs, func() int { return 90 }, nil) }()
// Give follow() time to open at EOF (it seeks to end immediately on start,
// so the seed line above is ignored), then append a delivery line.
@@ -295,7 +347,7 @@ func startRun(t *testing.T, path string, cs *captureStore) func() {
t.Helper()
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { done <- Run(ctx, path, cs, func() int { return 90 }) }()
go func() { done <- Run(ctx, path, cs, func() int { return 90 }, nil) }()
// follow() opens and seeks on start; give it a moment before the caller
// appends, so the append is not raced by the initial open.
time.Sleep(50 * time.Millisecond)
+7 -4
View File
@@ -20,8 +20,10 @@ const (
// which os.SameFile cannot answer across a restart.
fingerprintSize = 512
// persistInterval throttles the offset write. Losing up to this much
// progress on a crash only means re-parsing a few lines (UpdateStatus is
// idempotent), which is much cheaper than a database write per poll tick.
// progress on a crash means re-parsing a few lines, which is much cheaper
// than a database write per poll tick. For a delivery status that changes
// nothing (UpdateStatus is idempotent), and a refusal among them is not
// written to the inbound journal a second time (inboundJournal.handle).
persistInterval = 5 * time.Second
)
@@ -46,8 +48,9 @@ type tracker struct {
// state whose fingerprint still matches means continue from it, parsing the
// tail written while the panel was down; anything else means the file is not
// the one the offset referred to (rotated, recreated or truncated in the
// meantime), so read it from the start. Re-parsing lines already seen is
// harmless: UpdateStatus writes the same status onto the same row.
// meantime), so read it from the start. Re-parsing lines already seen writes
// the same status onto the same send-log row, and no second inbound journal
// row for a refusal among them.
func (t *tracker) resume(f *os.File) int64 {
size, err := fileSize(f)
if err != nil {
+6
View File
@@ -30,6 +30,12 @@ func (s *retentionProbeStore) DeleteSendLogBefore(cutoff time.Time) (int64, erro
return 0, nil
}
func (s *retentionProbeStore) InsertSendLogLines([]store.SendLogLine, bool) (int, error) {
return 0, nil
}
func (s *retentionProbeStore) PruneSendLogLines(time.Time) (int64, error) { return 0, nil }
func (s *retentionProbeStore) LogtailState(string) (store.LogtailState, bool, error) {
return store.LogtailState{}, false, nil
}
+9 -4
View File
@@ -8,11 +8,12 @@ import (
)
// authIPAllowed reports whether the authenticated application may submit from
// the connecting client IP. When the application has no IP restriction, or the
// client IP is not known, the check passes. Store errors are fail-open (see
// overLimit).
// the connecting client IP. When the application has no IP restriction the
// check passes. When it has one and the client's address is not known, the
// check fails: an address list that an unknown address satisfies is no list.
// Store errors are fail-open (see overLimit).
func (s *session) authIPAllowed() bool {
if s.login == "" || s.clientIP == "" {
if s.login == "" {
return true
}
a, err := s.rec.ApplicationByLogin(s.login)
@@ -26,6 +27,10 @@ func (s *session) authIPAllowed() bool {
if a.AllowsAuthFromIP(s.clientIP) {
return true
}
if s.clientIP == "" {
log.Printf("journal-milter: application %q refused — its client IPs are restricted and Postfix gave no client address", s.login)
return false
}
log.Printf("journal-milter: application %q refused from %s — client IP not allowed", s.login, s.clientIP)
return false
}
+290
View File
@@ -0,0 +1,290 @@
package milter
import (
"context"
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"log"
"net"
"net/textproto"
"regexp"
"strings"
"github.com/emersion/go-milter"
"github.com/mixeme/selfpost/internal/mailhdr"
"github.com/mixeme/selfpost/internal/store"
)
// InboundStore is what the inbound journal milter needs: the journal itself
// and the list of inbound domains, since port 25 also takes mail that is not
// the relay's. *store.Store satisfies it; tests substitute a fake.
type InboundStore interface {
InsertInboundSpam(e store.InboundSpamEntry) error
IsInboundDomain(name string) (bool, error)
}
// inboundSession is the inbound journal milter: the last milter on port 25,
// after the optional external filter. It writes one journal row per recipient
// of a message that reached end-of-message - that is, one the filter did not
// refuse; refusals never get this far and are read from mail.log instead.
//
// Like the send-log milter it only watches: every callback continues, and
// Postfix attaches the socket with default_action=accept, so
// neither an error here nor the panel being down holds up inbound mail.
type inboundSession struct {
milter.NoOpMilter
rec InboundStore
// key is what the filter signs each message with (FilterSignature), or
// empty when no filter is attached. A verdict is read only next to a
// signature that fits: the filter is attached so that mail flows when it is
// down, and then the verdict headers of a message are whatever its sender
// wrote.
key string
// clientIP is taken at Connect and again from {client_addr} with every
// MAIL FROM: a session the library made after a non-continue reply never
// sees Connect (see session.MailFrom in milter.go).
clientIP string
from string
rcpts []string
subject string
action string // X-Rspamd-Action
result string // X-Spamd-Result
seen bool // any verdict header of the filter
sig string // X-SelfPost-Filter
}
func (s *inboundSession) Connect(host, family string, port uint16, addr net.IP, m *milter.Modifier) (milter.Response, error) {
if addr != nil {
s.clientIP = addr.String()
}
return milter.RespContinue, nil
}
// MailFrom starts a message: a connection may carry several.
func (s *inboundSession) MailFrom(from string, m *milter.Modifier) (milter.Response, error) {
s.reset()
if ip := clientAddr(m); ip != "" {
s.clientIP = ip
}
s.from = cleanAddress(from)
return milter.RespContinue, nil
}
func (s *inboundSession) RcptTo(rcpt string, m *milter.Modifier) (milter.Response, error) {
s.rcpts = append(s.rcpts, cleanAddress(rcpt))
return milter.RespContinue, nil
}
// Header keeps the first Subject, the filter's verdict and its signature.
// Postfix hands the message to its milters one after another at
// end-of-message, each seeing what the one before added, which is how the
// filter's headers arrive here. The filter replaces headers of these names, so
// the last one read is its own - when it is up. Whether it was is decided at
// end-of-message, by the signature.
func (s *inboundSession) Header(name, value string, m *milter.Modifier) (milter.Response, error) {
switch textproto.CanonicalMIMEHeaderKey(name) {
case "Subject":
if s.subject == "" {
s.subject = mailhdr.DecodeSubject(value)
}
case "X-Rspamd-Action":
s.action, s.seen = value, true
case "X-Spamd-Result":
s.result, s.seen = value, true
case signatureHeader:
s.sig = strings.TrimSpace(value)
}
return milter.RespContinue, nil
}
// SignatureHeader is the header the filter signs a message in
// (deploy/antispam/rspamd/milter_headers.conf writes it).
const SignatureHeader = "X-SelfPost-Filter"
var signatureHeader = textproto.CanonicalMIMEHeaderKey(SignatureHeader)
// FilterSignature is what the filter writes into SignatureHeader for a message:
// the HMAC-SHA256 of its queue id under the key SelfPost shares with the
// filter, in hex. The key is used as the text it is stored as. A signature
// fits one queue id, so the header going on to the upstream with the mail
// gives nothing to whoever reads it there.
func FilterSignature(key, queueID string) string {
mac := hmac.New(sha256.New, []byte(key))
mac.Write([]byte(queueID))
return hex.EncodeToString(mac.Sum(nil))
}
// signed reports whether the message carries the filter's signature for its
// queue id. No key, no queue id or no signature is "no".
func (s *inboundSession) signed(queueID string) bool {
if s.key == "" || queueID == "" || s.sig == "" {
return false
}
return hmac.Equal([]byte(strings.ToLower(s.sig)), []byte(FilterSignature(s.key, queueID)))
}
// Body fires at end-of-message: the message is about to be queued for its
// upstream. "Continue" lets it through as "accept" would, and keeps this
// session for the connection's next message.
func (s *inboundSession) Body(m *milter.Modifier) (milter.Response, error) {
// The queue id is known only now, and with it whether the verdict headers
// read on the way here are the filter's.
if !s.signed(macro(m, "i")) {
s.seen = false
}
s.record()
s.reset()
return milter.RespContinue, nil
}
func (s *inboundSession) Abort(m *milter.Modifier) error {
s.reset()
return nil
}
func (s *inboundSession) reset() {
s.from, s.rcpts, s.subject = "", nil, ""
s.action, s.result, s.seen, s.sig = "", "", false, ""
}
// record writes one row per recipient at an inbound domain. Other recipients
// of port 25 (a DMARC report address at the server's own name) are not the
// relay's mail and get no row. Failures are logged, never propagated.
func (s *inboundSession) record() {
e := store.InboundSpamEntry{
ClientIP: s.clientIP,
From: s.from,
Subject: s.subject,
Decision: store.SpamDecisionAccept,
Engine: store.SpamEngineSelfPost,
}
if s.seen {
action, detail := rspamdVerdict(s.action, s.result)
e.Engine, e.Detail = store.SpamEngineRspamd, detail
if action == "quarantine" {
e.Decision = store.SpamDecisionQuarantine
}
}
for _, to := range s.rcpts {
domain := domainOf(to)
ok, err := s.rec.IsInboundDomain(domain)
if err != nil {
log.Printf("inbound-journal: look up %s: %v", domain, err)
continue
}
if !ok {
continue
}
e.InboundDomain, e.To = domain, to
if err := s.rec.InsertInboundSpam(e); err != nil {
log.Printf("inbound-journal: record %s -> %s: %v", s.from, to, err)
}
}
}
// What the journal keeps of a verdict: enough to see why, on one line.
const (
verdictMaxSymbols = 6
verdictMaxBytes = 255
)
// ListSymbolPrefix starts the names of the symbols the filter lists' rules add
// (deploy/antispam/rspamd/multimap.conf): SELFPOST_ALLOW_IP, _FROM, _DOMAIN.
const ListSymbolPrefix = "SELFPOST_"
var (
// "default: False [7.20 / 15.00]" - the score and the reject threshold.
rspamdScore = regexp.MustCompile(`\[(-?[0-9]+(?:\.[0-9]+)?) / (-?[0-9]+(?:\.[0-9]+)?)\]`)
// "BAYES_SPAM(5.10)[100.00%]" - a symbol and its weight.
rspamdSymbol = regexp.MustCompile(`([A-Z][A-Z0-9_]{1,63})\((-?[0-9]+(?:\.[0-9]+)?)\)`)
)
// rspamdVerdict reduces the filter's headers to an action and a one-line
// summary: the action, the score against the threshold, and the first few
// symbols that carried weight. It reads only what it recognises - the headers
// are read only next to the filter's signature, and even so nothing of them is
// kept verbatim except through these patterns.
func rspamdVerdict(actionHeader, resultHeader string) (action, detail string) {
action = strings.ToLower(strings.TrimSpace(actionHeader))
for _, r := range action {
if (r < 'a' || r > 'z') && r != ' ' {
action = ""
break
}
}
if len(action) > 32 {
action = ""
}
var parts []string
if action != "" {
parts = append(parts, action)
}
if m := rspamdScore.FindStringSubmatch(resultHeader); m != nil {
parts = append(parts, "score "+m[1]+" / "+m[2])
}
detail = strings.Join(parts, ", ")
// The filter lists' own symbols come first and are kept whatever else there
// is: Filter lists counts how often an entry decided by finding them here,
// and rspamd writes symbols heaviest first, which puts an allow entry's
// -100 last, behind any six others.
var symbols, rest []string
for _, m := range rspamdSymbol.FindAllStringSubmatch(resultHeader, -1) {
if strings.Trim(m[2], "-0.") == "" { // weight zero: informational
continue
}
if strings.HasPrefix(m[1], ListSymbolPrefix) {
if len(symbols) < verdictMaxSymbols {
symbols = append(symbols, m[1]+"("+m[2]+")")
}
continue
}
rest = append(rest, m[1]+"("+m[2]+")")
}
for _, sym := range rest {
if len(symbols) == verdictMaxSymbols {
break
}
symbols = append(symbols, sym)
}
if len(symbols) > 0 {
if detail != "" {
detail += "; "
}
detail += strings.Join(symbols, " ")
}
if len(detail) > verdictMaxBytes {
detail = detail[:verdictMaxBytes]
}
return action, detail
}
// ServeInbound runs the inbound journal milter on ln until ctx is cancelled.
// key is what the external filter in front of it signs messages with, or
// empty when none is attached (INBOUND_ANTISPAM_MILTER): no verdict is then
// read at all. It returns nil on a clean shutdown.
func ServeInbound(ctx context.Context, ln net.Listener, rec InboundStore, key string) error {
srv := &milter.Server{
NewMilter: func() milter.Milter { return &inboundSession{rec: rec, key: key} },
Actions: 0, // read-only
Protocol: milter.OptNoBody, // headers and end-of-message, not the body
}
go func() {
<-ctx.Done()
_ = srv.Close()
}()
if err := srv.Serve(ln); err != nil {
if ctx.Err() != nil {
return nil
}
return err
}
return nil
}
+350
View File
@@ -0,0 +1,350 @@
package milter
import (
"errors"
"net"
"strings"
"testing"
"github.com/emersion/go-milter"
"github.com/mixeme/selfpost/internal/store"
)
// fakeInbound is the journal and the inbound-domain list of one test.
type fakeInbound struct {
domains map[string]bool
rows []store.InboundSpamEntry
insertErr error
lookupErr error
}
func (f *fakeInbound) InsertInboundSpam(e store.InboundSpamEntry) error {
if f.insertErr != nil {
return f.insertErr
}
f.rows = append(f.rows, e)
return nil
}
func (f *fakeInbound) IsInboundDomain(name string) (bool, error) {
if f.lookupErr != nil {
return false, f.lookupErr
}
return f.domains[name], nil
}
// A header of the kind the documented rspamd sidecar adds.
const spamdResult = "default: False [7.20 / 15.00];\n\tBAYES_SPAM(5.10)[100.00%];\n\tMIME_GOOD(-0.10)[text/plain];\n\tARC_NA(0.00)[];\n\tRCVD_COUNT_ZERO(0.00)[0];\n\tR_DKIM_NA(2.20)[]"
// message replays one message through s: envelope, headers in order, end.
func message(t *testing.T, s *inboundSession, from string, rcpts []string, headers [][2]string) milter.Response {
t.Helper()
if _, err := s.MailFrom(from, mods(nil)); err != nil {
t.Fatalf("MailFrom: %v", err)
}
for _, r := range rcpts {
if _, err := s.RcptTo(r, mods(nil)); err != nil {
t.Fatalf("RcptTo: %v", err)
}
}
for _, h := range headers {
if _, err := s.Header(h[0], h[1], mods(nil)); err != nil {
t.Fatalf("Header: %v", err)
}
}
resp, err := s.Body(mods(map[string]string{"i": "ABC123"}))
if err != nil {
t.Fatalf("Body: %v", err)
}
return resp
}
// testKey is the key the tests' filter signs with; the messages they replay
// have the queue id ABC123.
const testKey = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
var testSig = [2]string{SignatureHeader, FilterSignature(testKey, "ABC123")}
func inboundSessionFor(t *testing.T, rec InboundStore, filter bool) *inboundSession {
t.Helper()
s := &inboundSession{rec: rec}
if filter {
s.key = testKey
}
if _, err := s.Connect("mx.example.net", "tcp4", 0, net.ParseIP("203.0.113.7"), mods(nil)); err != nil {
t.Fatalf("Connect: %v", err)
}
return s
}
// With no filter attached an accepted message is a plain row, one per
// recipient, decided by nobody.
func TestInboundRecordsAcceptedMessage(t *testing.T) {
rec := &fakeInbound{domains: map[string]bool{"in.example.com": true}}
s := inboundSessionFor(t, rec, false)
resp := message(t, s, "<sender@example.net> SIZE=1200",
[]string{"<a@in.example.com>", "b@IN.example.com"},
[][2]string{{"Subject", "=?utf-8?Q?Sch=C3=B6n?="}})
if resp != milter.RespContinue {
t.Errorf("end-of-message response = %v, want continue", resp)
}
if len(rec.rows) != 2 {
t.Fatalf("rows = %+v, want one per recipient", rec.rows)
}
want := store.InboundSpamEntry{
InboundDomain: "in.example.com", ClientIP: "203.0.113.7",
From: "sender@example.net", To: "a@in.example.com", Subject: "Schön",
Decision: store.SpamDecisionAccept, Engine: store.SpamEngineSelfPost,
}
if rec.rows[0] != want {
t.Errorf("row = %+v\nwant %+v", rec.rows[0], want)
}
if rec.rows[1].To != "b@IN.example.com" || rec.rows[1].InboundDomain != "in.example.com" {
t.Errorf("second row = %+v", rec.rows[1])
}
}
// Port 25 also takes DMARC reports for the server's own name. They are not
// the relay's mail and the journal says nothing about them.
func TestInboundSkipsRecipientsOutsideInboundDomains(t *testing.T) {
rec := &fakeInbound{domains: map[string]bool{"in.example.com": true}}
s := inboundSessionFor(t, rec, false)
message(t, s, "noreply-dmarc@google.com",
[]string{"dmarc-example.com@mail.example.org", "a@in.example.com"}, nil)
if len(rec.rows) != 1 || rec.rows[0].To != "a@in.example.com" {
t.Errorf("rows = %+v, want the inbound recipient alone", rec.rows)
}
}
// With the filter attached its verdict headers become the engine and detail.
func TestInboundReadsTheFiltersVerdict(t *testing.T) {
rec := &fakeInbound{domains: map[string]bool{"in.example.com": true}}
s := inboundSessionFor(t, rec, true)
message(t, s, "sender@example.net", []string{"a@in.example.com"}, [][2]string{
{"Subject", "Cheap pills"},
{"X-Rspamd-Action", "add header"},
{"X-Spamd-Result", spamdResult},
testSig,
})
if len(rec.rows) != 1 {
t.Fatalf("rows = %+v", rec.rows)
}
got := rec.rows[0]
if got.Engine != store.SpamEngineRspamd || got.Decision != store.SpamDecisionAccept {
t.Errorf("engine %q, decision %q", got.Engine, got.Decision)
}
if want := "add header, score 7.20 / 15.00; BAYES_SPAM(5.10) MIME_GOOD(-0.10) R_DKIM_NA(2.20)"; got.Detail != want {
t.Errorf("detail = %q\nwant %q", got.Detail, want)
}
}
func TestInboundQuarantineAction(t *testing.T) {
rec := &fakeInbound{domains: map[string]bool{"in.example.com": true}}
s := inboundSessionFor(t, rec, true)
message(t, s, "sender@example.net", []string{"a@in.example.com"},
[][2]string{{"x-rspamd-action", "quarantine"}, {"x-selfpost-filter", strings.ToUpper(testSig[1])}})
if len(rec.rows) != 1 || rec.rows[0].Decision != store.SpamDecisionQuarantine || rec.rows[0].Detail != "quarantine" {
t.Errorf("rows = %+v", rec.rows)
}
}
// Without a filter in front, a verdict header is the sender's own writing.
func TestInboundIgnoresVerdictHeadersWithoutAFilter(t *testing.T) {
rec := &fakeInbound{domains: map[string]bool{"in.example.com": true}}
s := inboundSessionFor(t, rec, false)
message(t, s, "sender@example.net", []string{"a@in.example.com"}, [][2]string{
{"X-Rspamd-Action", "no action"},
{"X-Spamd-Result", spamdResult},
})
if len(rec.rows) != 1 || rec.rows[0].Engine != store.SpamEngineSelfPost || rec.rows[0].Detail != "" {
t.Errorf("rows = %+v", rec.rows)
}
}
// The filter is attached fail-open, so with it down the headers are again the
// sender's: nothing of them reaches the journal except through the patterns.
func TestRspamdVerdictKeepsOnlyWhatItRecognises(t *testing.T) {
action, detail := rspamdVerdict("<script>alert(1)</script>", "default: False [1 / 15]; <b>HTML</b> lower_case(9.9) \x1b[31m")
if action != "" {
t.Errorf("action = %q, want it dropped", action)
}
if detail != "score 1 / 15" {
t.Errorf("detail = %q", detail)
}
var many strings.Builder
for i := 0; i < 400; i++ {
many.WriteString("SYMBOL_WITH_A_LONG_NAME_" + strings.Repeat("X", 30) + "(1.00)[];")
}
if _, detail := rspamdVerdict("reject", many.String()); len(detail) > verdictMaxBytes || strings.Count(detail, "(") > verdictMaxSymbols {
t.Errorf("detail of %d bytes with %d symbols", len(detail), strings.Count(detail, "("))
}
if action, _ := rspamdVerdict(strings.Repeat("a", 33), ""); action != "" {
t.Errorf("an action of 33 letters was kept")
}
}
// A second message on the same connection starts clean.
func TestInboundResetsBetweenMessages(t *testing.T) {
rec := &fakeInbound{domains: map[string]bool{"in.example.com": true}}
s := inboundSessionFor(t, rec, true)
message(t, s, "one@example.net", []string{"a@in.example.com"}, [][2]string{
{"Subject", "First"}, {"X-Rspamd-Action", "add header"}, testSig,
})
message(t, s, "two@example.net", []string{"b@in.example.com"}, nil)
if len(rec.rows) != 2 {
t.Fatalf("rows = %+v", rec.rows)
}
want := store.InboundSpamEntry{
InboundDomain: "in.example.com", ClientIP: "203.0.113.7",
From: "two@example.net", To: "b@in.example.com",
Decision: store.SpamDecisionAccept, Engine: store.SpamEngineSelfPost,
}
if rec.rows[1] != want {
t.Errorf("second row = %+v\nwant %+v", rec.rows[1], want)
}
// An aborted transaction leaves nothing behind for the next one either.
if _, err := s.MailFrom("three@example.net", mods(nil)); err != nil {
t.Fatal(err)
}
if _, err := s.RcptTo("c@in.example.com", mods(nil)); err != nil {
t.Fatal(err)
}
if err := s.Abort(mods(nil)); err != nil {
t.Fatal(err)
}
if len(rec.rows) != 2 || len(s.rcpts) != 0 {
t.Errorf("after an abort: rows %d, pending recipients %v", len(rec.rows), s.rcpts)
}
}
// The journal failing is never the mail's problem.
func TestInboundAcceptsWhenTheStoreFails(t *testing.T) {
for name, rec := range map[string]*fakeInbound{
"insert": {domains: map[string]bool{"in.example.com": true}, insertErr: errors.New("boom")},
"lookup": {lookupErr: errors.New("boom")},
} {
s := inboundSessionFor(t, rec, false)
if resp := message(t, s, "sender@example.net", []string{"a@in.example.com"}, nil); resp != milter.RespContinue {
t.Errorf("%s failure: response = %v, want continue", name, resp)
}
}
}
// rspamd writes symbols heaviest first, so an allow entry's -100 comes last.
// Filter lists counts an entry's matches by finding that symbol in the journal:
// it is kept whatever else there is.
func TestRspamdVerdictKeepsTheListSymbols(t *testing.T) {
result := "default: False [-91.00 / 15.00]; SYM_A(3.00)[]; SYM_B(2.00)[]; SYM_C(1.50)[]; SYM_D(1.00)[]; SYM_E(0.80)[]; SYM_F(0.50)[]; SYM_G(0.20)[]; ARC_NA(0.00)[]; SELFPOST_ALLOW_FROM(-100.00)[billing@vendor.example]"
_, detail := rspamdVerdict("no action", result)
want := "no action, score -91.00 / 15.00; SELFPOST_ALLOW_FROM(-100.00) SYM_A(3.00) SYM_B(2.00) SYM_C(1.50) SYM_D(1.00) SYM_E(0.80)"
if detail != want {
t.Errorf("detail = %q\nwant %q", detail, want)
}
}
// The client address of a connection's later messages comes from the MAIL
// macro: a session the library made after a non-continue reply saw no Connect.
func TestInboundClientAddressFromTheMailMacro(t *testing.T) {
rec := &fakeInbound{domains: map[string]bool{"in.example.com": true}}
s := &inboundSession{rec: rec} // no Connect
if _, err := s.MailFrom("sender@example.net", mods(map[string]string{"{client_addr}": "IPv6:2001:db8::7"})); err != nil {
t.Fatal(err)
}
if _, err := s.RcptTo("a@in.example.com", mods(nil)); err != nil {
t.Fatal(err)
}
if _, err := s.Body(mods(nil)); err != nil {
t.Fatal(err)
}
if len(rec.rows) != 1 || rec.rows[0].ClientIP != "2001:db8::7" {
t.Errorf("rows = %+v", rec.rows)
}
}
// No callback of the inbound milter answers anything but "continue": it must
// never be the reason a message is held up, and a non-continue reply would
// cost it its session.
func TestInboundOnlyEverContinues(t *testing.T) {
rec := &fakeInbound{domains: map[string]bool{"in.example.com": true}}
s := &inboundSession{rec: rec, key: testKey}
check := func(stage string, resp milter.Response, err error) {
t.Helper()
if err != nil || resp != milter.RespContinue {
t.Errorf("%s: %v, %v", stage, resp, err)
}
}
resp, err := s.Connect("mx", "tcp4", 0, net.ParseIP("203.0.113.7"), mods(nil))
check("connect", resp, err)
resp, err = s.MailFrom("a@example.net", mods(nil))
check("mail", resp, err)
resp, err = s.RcptTo("a@in.example.com", mods(nil))
check("rcpt", resp, err)
resp, err = s.Header("X-Rspamd-Action", "reject", mods(nil))
check("header", resp, err)
resp, err = s.Body(mods(nil))
check("end of message", resp, err)
}
// The filter is attached so that mail flows when it is down, and then the
// verdict headers of a message are its sender's own. A verdict is recorded only
// next to the filter's signature for this message's queue id.
func TestInboundReadsAVerdictOnlyWithTheFiltersSignature(t *testing.T) {
verdict := [][2]string{{"X-Rspamd-Action", "quarantine"}, {"X-Spamd-Result", spamdResult}}
for name, c := range map[string]struct {
extra [][2]string
signed bool
}{
"signed": {[][2]string{testSig}, true},
"no signature": {nil, false},
"a made-up signature": {[][2]string{{SignatureHeader, strings.Repeat("ab", 32)}}, false},
"another message's signature": {[][2]string{{SignatureHeader, FilterSignature(testKey, "OTHER1")}}, false},
"a signature under another key": {[][2]string{{SignatureHeader, FilterSignature(strings.Repeat("f", 64), "ABC123")}}, false},
"the sender's first, then the real": {[][2]string{{SignatureHeader, "forged"}, testSig}, true},
} {
rec := &fakeInbound{domains: map[string]bool{"in.example.com": true}}
s := inboundSessionFor(t, rec, true)
message(t, s, "sender@example.net", []string{"a@in.example.com"}, append(append([][2]string{}, verdict...), c.extra...))
if len(rec.rows) != 1 {
t.Fatalf("%s: rows = %+v", name, rec.rows)
}
got := rec.rows[0]
if c.signed && (got.Engine != store.SpamEngineRspamd || got.Decision != store.SpamDecisionQuarantine) {
t.Errorf("%s: a signed verdict was not read: %+v", name, got)
}
if !c.signed && (got.Engine != store.SpamEngineSelfPost || got.Decision != store.SpamDecisionAccept || got.Detail != "") {
t.Errorf("%s: an unsigned verdict was recorded: %+v", name, got)
}
}
// A signature is of no use to the next message on the connection.
rec := &fakeInbound{domains: map[string]bool{"in.example.com": true}}
s := inboundSessionFor(t, rec, true)
message(t, s, "one@example.net", []string{"a@in.example.com"}, append(append([][2]string{}, verdict...), testSig))
message(t, s, "two@example.net", []string{"a@in.example.com"}, verdict)
if len(rec.rows) != 2 || rec.rows[1].Engine != store.SpamEngineSelfPost {
t.Errorf("rows = %+v", rec.rows)
}
// With no queue id nothing can be checked, so nothing is trusted.
rec = &fakeInbound{domains: map[string]bool{"in.example.com": true}}
s = inboundSessionFor(t, rec, true)
_, _ = s.MailFrom("sender@example.net", mods(nil))
_, _ = s.RcptTo("a@in.example.com", mods(nil))
_, _ = s.Header("X-Rspamd-Action", "reject", mods(nil))
_, _ = s.Header(SignatureHeader, FilterSignature(testKey, ""), mods(nil))
_, _ = s.Body(mods(nil))
if len(rec.rows) != 1 || rec.rows[0].Engine != store.SpamEngineSelfPost {
t.Errorf("no queue id: rows = %+v", rec.rows)
}
}
// What rspamd 3.11 wrote for this key and queue id in a throwaway container
// (dev trial, 2026-10-12): the two ends must compute the same thing.
func TestFilterSignatureIsWhatTheFilterWrites(t *testing.T) {
if got := FilterSignature(testKey, "4A1B2C3D5E"); got != "2aa1ca6dd1545d3f16c8c0c29ba59bfc28c050224ac2c5c953bd2617f73b6de2" {
t.Errorf("FilterSignature = %s", got)
}
}
+39 -6
View File
@@ -53,7 +53,11 @@ type session struct {
// valid zero value (no in-flight accounting).
flight *inflight
clientIP string // captured once per connection
// clientIP is the client's address. Postfix gives it at Connect, once per
// connection, and again with every MAIL FROM as the {client_addr} macro
// (milter_mail_macros in build/postfix-config.sh). The second is the one
// that can be relied on: see MailFrom.
clientIP string
login string
from string
@@ -62,9 +66,8 @@ type session struct {
reserved []*reservation // level-2 slots held by the current message
}
// Connect captures the client IP, which comes from the addr parameter rather
// than a macro (the {client_addr} macro was empty in testing). It is the
// rate-limit key; here it is recorded for completeness.
// Connect captures the client IP of a connection's first message. It is not
// enough on its own - see MailFrom.
func (s *session) Connect(host, family string, port uint16, addr net.IP, m *milter.Modifier) (milter.Response, error) {
if addr != nil {
s.clientIP = addr.String()
@@ -79,8 +82,21 @@ func (s *session) Connect(host, family string, port uint16, addr net.IP, m *milt
// limit is enforced here: over the limit, the message is refused with a 4xx
// tempfail before recipients are even offered (guide § Rate limiting).
// Enforcement is fail-open — see overLimit.
//
// The client address is read again here, from the {client_addr} macro. The
// milter library replaces the session with a new one after every reply that is
// not "continue" - a refusal, or the accept that used to end a message - and
// Postfix does not send Connect again on the same connection. A session made
// that way never saw the client, so every message of a connection after the
// first, and every MAIL FROM after a refused one, used to be judged with no
// client address: the level-2 limit and an application's client-address list
// both let it through. (Found on 2026-10-12; test/e2e sends two MAIL FROM on
// one connection.)
func (s *session) MailFrom(from string, m *milter.Modifier) (milter.Response, error) {
s.releaseReservations() // a previous transaction that ended without EOM/ABORT
if ip := clientAddr(m); ip != "" {
s.clientIP = ip
}
s.from = cleanAddress(from)
s.login = macro(m, "auth_authen")
s.rcpts = nil
@@ -113,13 +129,15 @@ func (s *session) Header(name, value string, m *milter.Modifier) (milter.Respons
// Body fires at end-of-message, when the queue-id macro {i} is set and the
// message is about to be committed to the queue. This is where the "queued"
// rows are written. We accept (this milter is done) without ever rejecting.
// rows are written. The reply is "continue", which at end-of-message lets the
// message through just as "accept" does - and, unlike it, leaves this session
// in place for the connection's next message (see MailFrom).
func (s *session) Body(m *milter.Modifier) (milter.Response, error) {
s.record(macro(m, "i"))
// The rows are in the send log now, so the stored count sees this message
// and its level-2 slots are no longer needed.
s.releaseReservations()
return milter.RespAccept, nil
return milter.RespContinue, nil
}
// Abort ends the current transaction without an end-of-message (client RSET, or
@@ -143,6 +161,21 @@ func macro(m *milter.Modifier, name string) string {
return m.Macros["{"+name+"}"]
}
// clientAddr is the client's address from the {client_addr} macro, or "" when
// Postfix did not send one that reads as an address. An IPv6 address may come
// in Sendmail's form, with "IPv6:" in front.
func clientAddr(m *milter.Modifier) string {
raw := strings.TrimSpace(macro(m, "client_addr"))
if len(raw) > 5 && strings.EqualFold(raw[:5], "IPv6:") {
raw = raw[5:]
}
ip := net.ParseIP(raw)
if ip == nil {
return ""
}
return ip.String()
}
// record writes one send-log row per recipient. Failures are logged, never
// propagated: journalling must not affect mail acceptance (architecture.md §
// Mail path).
+64 -6
View File
@@ -166,8 +166,10 @@ func TestBodyAcceptsEvenWhenRecorderFails(t *testing.T) {
if err != nil {
t.Fatalf("Body returned error, must fail open: %v", err)
}
if resp != milter.RespAccept {
t.Fatalf("Body response = %v, want Accept", resp)
// "Continue" at end-of-message lets the message through, and keeps the
// session for the connection's next message.
if resp != milter.RespContinue {
t.Fatalf("Body response = %v, want Continue", resp)
}
}
@@ -389,21 +391,77 @@ func TestRateLimitFailsOpenOnCountError(t *testing.T) {
}
}
func TestRateLimitNoIPKeyDoesNotApply(t *testing.T) {
// A ceiling is on a domain or an application, not on an address: a message that
// arrives with no client address is counted like any other. It used to be let
// through, which is how the second MAIL FROM of a connection got past a limit.
func TestRateLimitAppliesWithoutAClientIP(t *testing.T) {
rec := &fakeRecorder{
limits: map[string]store.RateLimit{
store.RateLimitScopeDomain + "|example.com": domainLimit(),
},
counts: map[string]int64{store.RateLimitScopeDomain + "|example.com": 999},
}
// A session with no client IP (e.g. local submission) cannot be keyed.
s := &session{rec: rec}
resp, err := s.MailFrom("a@example.com", mods(map[string]string{"auth_authen": "app1"}))
if err != nil {
t.Fatalf("MailFrom: %v", err)
}
if resp != milter.RespContinue {
t.Fatalf("no-IP session = %v, want Continue", resp)
if resp != milter.RespTempFail {
t.Fatalf("over the limit with no client IP = %v, want TempFail", resp)
}
}
// The milter library hands the milter a fresh session after every reply that
// is not "continue", and Postfix sends Connect once per connection. Such a
// session knows the client only from the {client_addr} macro of MAIL FROM, and
// that has to be enough for an application's client-address list.
func TestAuthIPRestrictHoldsForASessionThatNeverSawConnect(t *testing.T) {
rec := &fakeRecorder{apps: map[string]store.Application{
"app1": {Login: "app1", AuthIPRestrict: true, AuthAllowedIPs: []string{"198.51.100.1"}},
}}
for name, c := range map[string]struct {
macros map[string]string
want milter.Response
}{
"an unlisted address in the macro": {map[string]string{"auth_authen": "app1", "{client_addr}": limitIP}, milter.RespTempFail},
"the listed address in the macro": {map[string]string{"auth_authen": "app1", "{client_addr}": "198.51.100.1"}, milter.RespContinue},
// No address at all is not a pass: a list an unknown address satisfies is no list.
"no client address": {map[string]string{"auth_authen": "app1"}, milter.RespTempFail},
"not an address": {map[string]string{"auth_authen": "app1", "{client_addr}": "unknown"}, milter.RespTempFail},
} {
s := &session{rec: rec} // no Connect
resp, err := s.MailFrom("a@example.com", mods(c.macros))
if err != nil || resp != c.want {
t.Errorf("%s: %v, %v; want %v", name, resp, err, c.want)
}
}
// An application with no list is not held up by a missing address.
open := &fakeRecorder{apps: map[string]store.Application{"app1": {Login: "app1"}}}
s := &session{rec: open}
if resp, _ := s.MailFrom("a@example.com", mods(map[string]string{"auth_authen": "app1"})); resp != milter.RespContinue {
t.Errorf("no list, no address: %v, want Continue", resp)
}
}
// The macro wins over what Connect said, and reads both spellings of IPv6.
func TestClientAddrFromTheMailMacro(t *testing.T) {
for raw, want := range map[string]string{
"203.0.113.7": "203.0.113.7", "2001:db8::1": "2001:db8::1", "IPv6:2001:DB8::1": "2001:db8::1",
"ipv6:::1": "::1", "": "", "unknown": "", "IPv6:": "",
} {
if got := clientAddr(mods(map[string]string{"{client_addr}": raw})); got != want {
t.Errorf("clientAddr(%q) = %q, want %q", raw, got, want)
}
}
s := &session{rec: &fakeRecorder{}}
_, _ = s.Connect("h", "tcp4", 0, net.ParseIP("192.0.2.1"), mods(nil))
_, _ = s.MailFrom("a@example.com", mods(map[string]string{"client_addr": "192.0.2.2"}))
if s.clientIP != "192.0.2.2" {
t.Errorf("clientIP = %q, want the macro's", s.clientIP)
}
_, _ = s.MailFrom("a@example.com", mods(nil))
if s.clientIP != "192.0.2.2" {
t.Errorf("clientIP = %q after a MAIL FROM with no macro, want it kept", s.clientIP)
}
}
+3 -4
View File
@@ -26,10 +26,9 @@ import (
// reaches the send log (or is abandoned) — see inflight for why the stored
// count alone is not enough.
func (s *session) overLimit() bool {
if s.clientIP == "" {
return false // no client IP to key on; level-2 does not apply
}
// A ceiling is on a domain or an application, whatever address the mail
// comes from, so the client address is not asked for here. It used to be,
// and a message that arrived without one was let through uncounted.
if s.login != "" {
rl, ok, err := s.rec.RateLimit(store.RateLimitScopeApp, s.login)
if err != nil {
+99
View File
@@ -0,0 +1,99 @@
package notify
import (
"bytes"
"context"
"crypto/rand"
"encoding/hex"
"fmt"
"mime"
"os/exec"
"strings"
"time"
)
const (
sendmailPath = "/usr/sbin/sendmail"
sendmailTimeout = 20 * time.Second
)
// Mailer hands notifications to Postfix by local submission, as
// postmaster@<hostname>. They are the panel's own mail: they pass no SASL
// login, are not rows of the send log and count against no rate limit.
// Postfix signs them through OpenDKIM with the key of the server's own name
// (non_smtpd_milters in build/postfix-config.sh, domain.Service.SetServerName).
type Mailer struct {
hostname string
// run submits a finished message; a field so tests can read it instead.
run func(from, to string, msg []byte) error
}
// NewMailer returns a mailer for the server named hostname.
func NewMailer(hostname string) *Mailer {
return &Mailer{hostname: hostname, run: sendmail}
}
// From is the address notifications are sent from.
func From(hostname string) string { return "postmaster@" + hostname }
// Send submits one message to one recipient.
func (m *Mailer) Send(to, subject, body string) error {
// The address is a panel user's, validated when it was saved. It goes into
// a header and onto a command line all the same, so what cannot be part of
// an address is refused here as well (security.md).
if to == "" || strings.ContainsAny(to, " \t\r\n<>,;\"") || strings.HasPrefix(to, "-") || !strings.Contains(to, "@") {
return fmt.Errorf("not an address a notification can go to: %q", to)
}
from := From(m.hostname)
id, err := messageID(m.hostname)
if err != nil {
return err
}
var b bytes.Buffer
fmt.Fprintf(&b, "From: SelfPost <%s>\r\n", from)
fmt.Fprintf(&b, "To: <%s>\r\n", to)
fmt.Fprintf(&b, "Subject: %s\r\n", mime.QEncoding.Encode("utf-8", oneLine(subject)))
fmt.Fprintf(&b, "Date: %s\r\n", time.Now().UTC().Format(time.RFC1123Z))
fmt.Fprintf(&b, "Message-ID: %s\r\n", id)
// Tells a vacation responder and a ticket system that nobody wrote this.
b.WriteString("Auto-Submitted: auto-generated\r\n")
b.WriteString("MIME-Version: 1.0\r\n")
b.WriteString("Content-Type: text/plain; charset=utf-8\r\n")
b.WriteString("Content-Transfer-Encoding: 8bit\r\n")
b.WriteString("\r\n")
b.WriteString(strings.ReplaceAll(strings.ReplaceAll(body, "\r\n", "\n"), "\n", "\r\n"))
return m.run(from, to, b.Bytes())
}
// oneLine keeps a subject on its line whatever went into it.
func oneLine(s string) string {
return strings.Join(strings.Fields(s), " ")
}
func messageID(hostname string) (string, error) {
var raw [12]byte
if _, err := rand.Read(raw[:]); err != nil {
return "", fmt.Errorf("message id: %w", err)
}
return fmt.Sprintf("<%s.%d@%s>", hex.EncodeToString(raw[:]), time.Now().Unix(), hostname), nil
}
// sendmail submits msg through Postfix's sendmail command. The argument list is
// fixed apart from the two addresses, which follow "--" so that neither can be
// read as an option; nothing goes through a shell (security.md). -i keeps a
// line holding a single dot from ending the message.
func sendmail(from, to string, msg []byte) error {
ctx, cancel := context.WithTimeout(context.Background(), sendmailTimeout)
defer cancel()
cmd := exec.CommandContext(ctx, sendmailPath, "-i", "-f", from, "--", to)
cmd.Stdin = bytes.NewReader(msg)
out, err := cmd.CombinedOutput()
if ctx.Err() == context.DeadlineExceeded {
return fmt.Errorf("sendmail timed out after %s", sendmailTimeout)
}
if err != nil {
return fmt.Errorf("sendmail: %w: %s", err, strings.TrimSpace(string(out)))
}
return nil
}
+416
View File
@@ -0,0 +1,416 @@
// Package notify writes to the panel's users when something needs them
// (architecture.md § Notifications): a sending domain's DNS record stops
// matching, a level-2 rate limit refuses mail, the TLS certificate is about to
// expire, a message has waited in the queue for more than three days.
//
// Each subject has a row of state in the database, so a thing is said once and
// a restart says nothing again. The rules of when a message goes out are the
// owner's (docs/plans/panel-notifications.md § When a message is sent) and
// live in this file; what the messages say is in text.go, how they leave in
// mailer.go.
package notify
import (
"context"
"fmt"
"log"
"sort"
"strings"
"time"
"github.com/mixeme/selfpost/internal/dnscheck"
"github.com/mixeme/selfpost/internal/health"
"github.com/mixeme/selfpost/internal/postfix"
"github.com/mixeme/selfpost/internal/store"
)
const (
// CheckInterval is how often DNS, the certificate and the queue are looked
// at. A var-free constant: the rules below count in checks.
CheckInterval = time.Hour
// LimitInterval is how often refused mail is looked for. A refusal is
// reported within the minute rather than on the hour: it is mail not being
// sent now. The look is one indexed query.
LimitInterval = time.Minute
// dnsChecks is how many checks in a row must agree before a DNS mismatch is
// reported, and before it is reported gone. One check is a resolver's bad
// minute; two, an hour apart, is a record that is wrong.
dnsChecks = 2
// QueueAge is how long a message waits before the queue is reported. Postfix
// keeps a message for five days by default, so two are left to act in.
QueueAge = 3 * 24 * time.Hour
// The certificate is reported this many days before it expires, and again at
// the second number if it is still the same certificate.
certFirstDays = 14
certLastDays = 3
// limitRepeat is the shortest time between two messages about the same
// level-2 limit.
limitRepeat = 24 * time.Hour
phasePending = "pending" // seen wrong, not yet said
phaseOpen = "open" // said
)
// Store is what the notifier needs of the database. *store.Store satisfies it.
type Store interface {
NotificationsEnabled() (bool, error)
NotifyRecipients(domainName string) ([]string, error)
GetNotifyState(key string) (store.NotifyState, bool, error)
SetNotifyState(st store.NotifyState) error
DeleteNotifyState(key string) error
ListNotifyStateKeys(prefix string) ([]string, error)
LimitRefusalsSince(since time.Time) ([]store.LimitRefusal, error)
ListDomains() ([]store.Domain, error)
}
// Sender hands one message to the mail system. *Mailer does; tests collect.
type Sender interface {
Send(to, subject, body string) error
}
// Checks are the readings the notifier acts on, as functions so that tests
// need neither DNS nor a certificate nor a queue.
type Checks struct {
// Domain is a fresh check of a sending domain's published records.
Domain func(d store.Domain) (dnscheck.Domain, error)
// Certificate is the state of the TLS certificate Postfix serves.
Certificate func() health.Certificate
// Queue lists the messages Postfix holds; an error says nothing about any.
Queue func() ([]postfix.QueuedMessage, error)
}
// Notifier applies the rules. Create one with New and call Run.
type Notifier struct {
st Store
checks Checks
sender Sender
hostname string
now func() time.Time
}
// New builds a notifier for the server named hostname.
func New(st Store, checks Checks, sender Sender, hostname string) *Notifier {
return &Notifier{st: st, checks: checks, sender: sender, hostname: hostname, now: func() time.Time { return time.Now().UTC() }}
}
// Run checks on both cadences until ctx is cancelled. The first hourly check
// runs a few minutes after the start, not at it: a container that has just
// come up is still starting Postfix and may have no network yet.
func (n *Notifier) Run(ctx context.Context) error {
first := time.NewTimer(3 * time.Minute)
defer first.Stop()
hourly := time.NewTicker(CheckInterval)
defer hourly.Stop()
limits := time.NewTicker(LimitInterval)
defer limits.Stop()
for {
select {
case <-ctx.Done():
return nil
case <-first.C:
n.Check()
case <-hourly.C:
n.Check()
case <-limits.C:
n.CheckLimits()
}
}
}
// enabled reports whether anything is to be done at all. With the instance
// switch off nothing is checked and no state moves, so that what is wrong
// when it is turned on again is said then.
func (n *Notifier) enabled() bool {
on, err := n.st.NotificationsEnabled()
if err != nil {
log.Printf("notify: read the switch: %v", err)
return false
}
return on
}
// Check is the hourly pass: DNS of every sending domain, the certificate, the
// queue.
func (n *Notifier) Check() {
if !n.enabled() {
return
}
n.checkDNS()
n.checkCertificate()
n.checkQueue()
}
// send writes one message to every recipient of a subject, each in a message
// of their own - users are not shown each other's addresses. It reports
// whether the subject counts as said: nobody to tell is said, a message that
// could not be handed over to anyone is not, and is tried at the next check.
func (n *Notifier) send(domainName string, m message) bool {
to, err := n.st.NotifyRecipients(domainName)
if err != nil {
log.Printf("notify: recipients: %v", err)
return false
}
if len(to) == 0 {
return true
}
sent := 0
for _, addr := range to {
if err := n.sender.Send(addr, m.subject, m.body); err != nil {
log.Printf("notify: send %q to %s: %v", m.subject, addr, err)
continue
}
sent++
}
if sent > 0 {
log.Printf("notify: %q sent to %d of %d", m.subject, sent, len(to))
}
return sent > 0
}
func (n *Notifier) save(st store.NotifyState) {
if err := n.st.SetNotifyState(st); err != nil {
log.Printf("notify: save state %s: %v", st.Key, err)
}
}
func (n *Notifier) forget(key string) {
if err := n.st.DeleteNotifyState(key); err != nil {
log.Printf("notify: forget state %s: %v", key, err)
}
}
// ---- DNS
// dnsFault is what is wrong with a domain's records: the checks that failed,
// in the order the page shows them.
type dnsFault struct {
Record string // "DKIM", "SPF", "DMARC"
Detail string
}
// dnsReading sorts a check into wrong, right, or nothing learned. A record
// that fails is wrong whatever the others say. A lookup that did not answer is
// nothing learned - unless another record is plainly wrong. A warning (no DMARC
// record, an SPF include this check does not follow) is advice the page gives
// and not a thing to write about.
func dnsReading(d dnscheck.Domain) (faults []dnsFault, known bool) {
known = true
for _, r := range []struct {
name string
res dnscheck.Result
}{{"DKIM", d.DKIM}, {"SPF", d.SPF}, {"DMARC", d.DMARC}} {
switch r.res.Status {
case health.StatusError:
faults = append(faults, dnsFault{Record: r.name, Detail: r.res.Detail})
case health.StatusUnknown:
known = false
}
}
if len(faults) > 0 {
known = true
}
return faults, known
}
func (n *Notifier) checkDNS() {
domains, err := n.st.ListDomains()
if err != nil {
log.Printf("notify: list domains: %v", err)
return
}
names := make(map[string]bool, len(domains))
for _, d := range domains {
names[d.Name] = true
res, err := n.checks.Domain(d)
if err != nil {
log.Printf("notify: check %s: %v", d.Name, err)
continue
}
n.dnsStep(d.Name, res)
}
// A deleted domain has no records to be right or wrong.
keys, err := n.st.ListNotifyStateKeys("dns:")
if err != nil {
log.Printf("notify: list dns state: %v", err)
return
}
for _, key := range keys {
if !names[strings.TrimPrefix(key, "dns:")] {
n.forget(key)
}
}
}
// dnsStep moves one domain's state by one check.
func (n *Notifier) dnsStep(name string, res dnscheck.Domain) {
faults, known := dnsReading(res)
if !known {
return
}
bad := len(faults) > 0
key := "dns:" + name
st, have, err := n.st.GetNotifyState(key)
if err != nil {
log.Printf("notify: %v", err)
return
}
now := n.now()
switch {
case !have && !bad:
// Nothing wrong and nothing said.
case !have && bad:
n.save(store.NotifyState{Key: key, Phase: phasePending, Count: 1, Since: now})
case st.Phase == phasePending && !bad:
n.forget(key) // it was the resolver's bad minute
case st.Phase == phasePending && bad:
st.Count++
if st.Count >= dnsChecks {
if !n.send(name, dnsProblem(n.hostname, name, faults, st.Since)) {
n.save(st)
return
}
st.Phase, st.Count, st.SentAt = phaseOpen, 0, now
}
n.save(st)
case st.Phase == phaseOpen && bad:
if st.Count != 0 {
st.Count = 0 // it went right for a check and wrong again: still the same trouble
n.save(st)
}
case st.Phase == phaseOpen && !bad:
st.Count++
if st.Count >= dnsChecks {
if n.send(name, dnsCleared(n.hostname, name, st.Since)) {
n.forget(key)
return
}
}
n.save(st)
}
}
// ---- certificate
func (n *Notifier) checkCertificate() {
c := n.checks.Certificate()
if c.NotAfter.IsZero() {
return // no certificate configured, or one that cannot be read: Health says so
}
const key = "cert"
mark := c.NotAfter.UTC().Format(time.RFC3339)
st, have, err := n.st.GetNotifyState(key)
if err != nil {
log.Printf("notify: %v", err)
return
}
if have && st.Mark != mark {
// A different certificate: what was said about the old one is over.
n.forget(key)
have, st = false, store.NotifyState{}
}
left := c.NotAfter.Sub(n.now())
level := ""
switch {
case left <= certLastDays*24*time.Hour:
level = "last"
case left <= certFirstDays*24*time.Hour:
level = "first"
}
if level == "" || (have && (st.Phase == level || st.Phase == "last")) {
return
}
if !n.send("", certExpiring(n.hostname, c.NotAfter, left)) {
return
}
n.save(store.NotifyState{Key: key, Phase: level, Mark: mark, Since: n.now(), SentAt: n.now()})
}
// ---- queue
func (n *Notifier) checkQueue() {
msgs, err := n.checks.Queue()
if err != nil {
log.Printf("notify: read the queue: %v", err)
return // no listing is no information
}
now := n.now()
var old []postfix.QueuedMessage
for _, m := range msgs {
if now.Sub(m.Arrival) > QueueAge {
old = append(old, m)
}
}
sort.Slice(old, func(i, j int) bool { return old[i].Arrival.Before(old[j].Arrival) })
const key = "queue"
st, have, err := n.st.GetNotifyState(key)
if err != nil {
log.Printf("notify: %v", err)
return
}
switch {
case len(old) > 0 && !have:
if n.send("", queueStuck(n.hostname, old, now)) {
n.save(store.NotifyState{Key: key, Phase: phaseOpen, Since: now, SentAt: now})
}
case len(old) == 0 && have:
if n.send("", queueCleared(n.hostname, st.Since)) {
n.forget(key)
}
}
}
// ---- level-2 limits
// CheckLimits looks for mail refused under a level-2 rate limit and reports
// each limit that refused, at most once in limitRepeat.
func (n *Notifier) CheckLimits() {
if !n.enabled() {
return
}
now := n.now()
refusals, err := n.st.LimitRefusalsSince(now.Add(-limitRepeat))
if err != nil {
log.Printf("notify: %v", err)
return
}
for _, r := range refusals {
if r.Domain == "" {
continue
}
key := fmt.Sprintf("limit:%s:%s", r.Domain, r.AppLogin)
st, have, err := n.st.GetNotifyState(key)
if err != nil {
log.Printf("notify: %v", err)
continue
}
// Told within the day: the refusals counted now are the ones it was
// told about, or more of the same. Told longer ago: everything in the
// window is new since then.
if have && now.Sub(st.SentAt) < limitRepeat {
continue
}
if n.send(r.Domain, limitRefused(n.hostname, r)) {
n.save(store.NotifyState{Key: key, Phase: phaseOpen, Since: r.First, SentAt: now})
}
}
// A limit nobody has hit for two days is not a subject any more.
keys, err := n.st.ListNotifyStateKeys("limit:")
if err != nil {
log.Printf("notify: list limit state: %v", err)
return
}
for _, key := range keys {
st, have, err := n.st.GetNotifyState(key)
if err == nil && have && now.Sub(st.SentAt) > 2*limitRepeat {
n.forget(key)
}
}
}
+380
View File
@@ -0,0 +1,380 @@
package notify
import (
"errors"
"path/filepath"
"strings"
"testing"
"time"
"github.com/mixeme/selfpost/internal/dnscheck"
"github.com/mixeme/selfpost/internal/health"
"github.com/mixeme/selfpost/internal/postfix"
"github.com/mixeme/selfpost/internal/store"
)
// outbox collects what would have been handed to Postfix.
type outbox struct {
sent []string // "to|subject"
body []string
fail bool
}
func (o *outbox) Send(to, subject, body string) error {
if o.fail {
return errors.New("sendmail: queue file write error")
}
o.sent = append(o.sent, to+"|"+subject)
o.body = append(o.body, body)
return nil
}
func (o *outbox) take() []string {
s := o.sent
o.sent, o.body = nil, nil
return s
}
// bench is a notifier over a real database with a clock and readings the test
// sets: one global user, one user of example.com, one user of another domain.
type bench struct {
n *Notifier
st *store.Store
out *outbox
now time.Time
dns map[string]dnscheck.Domain
cert health.Certificate
q []postfix.QueuedMessage
qErr error
}
func newBench(t *testing.T) *bench {
t.Helper()
st, err := store.Open(filepath.Join(t.TempDir(), "test.db"))
if err != nil {
t.Fatalf("open store: %v", err)
}
t.Cleanup(func() { _ = st.Close() })
b := &bench{st: st, out: &outbox{}, now: time.Date(2026, 10, 12, 12, 0, 0, 0, time.UTC), dns: map[string]dnscheck.Domain{}}
mine, err := st.AddDomain("example.com", "sp")
if err != nil {
t.Fatal(err)
}
other, err := st.AddDomain("other.example", "sp")
if err != nil {
t.Fatal(err)
}
for _, u := range []struct {
name string
role store.Role
reach store.Reach
}{
{"root", store.RoleGlobal, store.Reach{}},
{"mine", store.RoleDomain, store.Reach{DomainIDs: []int64{mine.ID}}},
{"theirs", store.RoleDomain, store.Reach{DomainIDs: []int64{other.ID}}},
} {
id, err := st.CreateUser(u.name, "hash", u.role, u.reach)
if err != nil {
t.Fatal(err)
}
if err := st.UpdateUser(id, u.name, "hash", u.name+"@example.org"); err != nil {
t.Fatal(err)
}
}
b.n = New(st, Checks{
Domain: func(d store.Domain) (dnscheck.Domain, error) {
res, ok := b.dns[d.Name]
if !ok {
return good(), nil
}
return res, nil
},
Certificate: func() health.Certificate { return b.cert },
Queue: func() ([]postfix.QueuedMessage, error) { return b.q, b.qErr },
}, b.out, "mail.example.org")
b.n.now = func() time.Time { return b.now }
return b
}
// hour runs the hourly check and moves the clock on by an hour.
func (b *bench) hour() []string {
b.n.Check()
b.now = b.now.Add(time.Hour)
return b.out.take()
}
func good() dnscheck.Domain {
ok := dnscheck.Result{Status: health.StatusOK}
return dnscheck.Domain{DKIM: ok, SPF: ok, DMARC: ok}
}
func badDKIM() dnscheck.Domain {
d := good()
d.DKIM = dnscheck.Result{Status: health.StatusError, Detail: "A TXT record exists but its public key is not the one this server signs with."}
return d
}
func want(t *testing.T, what string, got []string, wanted ...string) {
t.Helper()
if strings.Join(got, "\n") != strings.Join(wanted, "\n") {
t.Errorf("%s: sent\n %s\nwant\n %s", what, strings.Join(got, "\n "), strings.Join(wanted, "\n "))
}
}
// A DNS mismatch is said once, after two checks in a row found it, to the
// global users and the domain's own; it is said to be over once, after two
// checks in a row found the records right.
func TestDNSMismatchIsSaidOnceAndClearedOnce(t *testing.T) {
b := newBench(t)
b.dns["example.com"] = badDKIM()
want(t, "first bad check", b.hour())
problem := "[SelfPost] example.com: DKIM no longer matches"
want(t, "second bad check", b.hour(), "mine@example.org|"+problem, "root@example.org|"+problem)
want(t, "third bad check", b.hour())
b.dns["example.com"] = good()
want(t, "first good check", b.hour())
// One good check does not end it: wrong again is the same trouble, not a new one.
b.dns["example.com"] = badDKIM()
want(t, "wrong again", b.hour())
b.dns["example.com"] = good()
want(t, "good once more", b.hour())
cleared := "[SelfPost] example.com: DNS records match again"
want(t, "second good check in a row", b.hour(), "mine@example.org|"+cleared, "root@example.org|"+cleared)
want(t, "after it cleared", b.hour())
if _, have, _ := b.st.GetNotifyState("dns:example.com"); have {
t.Error("state left behind after the trouble was over")
}
}
// One bad check followed by a good one was the resolver, and nothing is said.
// A lookup that did not answer says nothing either way; a warning is advice
// the page gives, not a thing to write about.
func TestDNSNoiseIsNotSaid(t *testing.T) {
b := newBench(t)
b.dns["example.com"] = badDKIM()
want(t, "one bad check", b.hour())
b.dns["example.com"] = good()
want(t, "then good", b.hour())
b.dns["example.com"] = badDKIM()
want(t, "bad again, counted from one", b.hour())
unknown := good()
unknown.SPF = dnscheck.Result{Status: health.StatusUnknown, Detail: "Could not look up the SPF record."}
b.dns["example.com"] = unknown
want(t, "resolver did not answer", b.hour())
if st, have, _ := b.st.GetNotifyState("dns:example.com"); !have || st.Count != 1 {
t.Errorf("an unanswered lookup moved the state: %+v, %v", st, have)
}
warn := good()
warn.DMARC = dnscheck.Result{Status: health.StatusWarn, Detail: "No DMARC record."}
b.dns["other.example"] = warn
want(t, "a warning", b.hour())
want(t, "a warning, twice", b.hour())
}
// The instance switch stops everything, state included; a domain that is
// deleted takes its state with it; a message that could not be handed over is
// tried again at the next check.
func TestDNSSwitchDeletionAndRetry(t *testing.T) {
b := newBench(t)
b.dns["example.com"] = badDKIM()
if err := b.st.SetNotificationsEnabled(false); err != nil {
t.Fatal(err)
}
want(t, "switched off", b.hour())
want(t, "switched off", b.hour())
if _, have, _ := b.st.GetNotifyState("dns:example.com"); have {
t.Fatal("state moved while notifications were off")
}
if err := b.st.SetNotificationsEnabled(true); err != nil {
t.Fatal(err)
}
want(t, "on again, first check", b.hour())
b.out.fail = true
want(t, "second check, mail system down", b.hour())
b.out.fail = false
if got := b.hour(); len(got) != 2 {
t.Errorf("after the mail system came back: %v, want the two messages", got)
}
domains, _ := b.st.ListDomains()
for _, d := range domains {
if d.Name == "example.com" {
if err := b.st.DeleteDomain(d.ID); err != nil {
t.Fatal(err)
}
}
}
want(t, "domain deleted", b.hour())
if _, have, _ := b.st.GetNotifyState("dns:example.com"); have {
t.Error("a deleted domain kept its state")
}
}
// The certificate is said at 14 days and again at 3 if it is still the same
// one, to global users only; a new certificate starts over.
func TestCertificateIsSaidTwice(t *testing.T) {
b := newBench(t)
expiry := b.now.Add(20 * 24 * time.Hour)
b.cert = health.Certificate{NotAfter: expiry}
want(t, "20 days left", b.hour())
b.now = expiry.Add(-14 * 24 * time.Hour)
want(t, "14 days left", b.hour(), "root@example.org|[SelfPost] TLS certificate of mail.example.org expires in 14 days")
want(t, "an hour later", b.hour())
b.now = expiry.Add(-8 * 24 * time.Hour)
want(t, "8 days left", b.hour())
b.now = expiry.Add(-3 * 24 * time.Hour)
want(t, "3 days left", b.hour(), "root@example.org|[SelfPost] TLS certificate of mail.example.org expires in 3 days")
b.now = expiry.Add(time.Hour)
want(t, "expired, already said", b.hour())
// Renewed: nothing to say, and the next expiry is a new subject.
renewed := b.now.Add(90 * 24 * time.Hour)
b.cert = health.Certificate{NotAfter: renewed}
want(t, "renewed", b.hour())
b.now = renewed.Add(-2 * 24 * time.Hour)
want(t, "the new one, first seen with 2 days left", b.hour(), "root@example.org|[SelfPost] TLS certificate of mail.example.org expires in 2 days")
// No certificate configured: Health says so, mail does not.
b.cert = health.Certificate{Status: health.StatusUnknown}
want(t, "no certificate", b.hour())
}
// A message older than three days is said once, and its going once; a queue
// that cannot be listed says nothing.
func TestQueueIsSaidOnceAndClearedOnce(t *testing.T) {
b := newBench(t)
b.q = []postfix.QueuedMessage{
{ID: "FRESH001", Queue: "deferred", Sender: "app@example.com", Arrival: b.now.Add(-2 * 24 * time.Hour)},
}
want(t, "two days old", b.hour())
b.q = append(b.q, postfix.QueuedMessage{ID: "OLD00001", Queue: "deferred", Sender: "app@example.com", Arrival: b.now.Add(-4 * 24 * time.Hour)})
b.n.Check()
if len(b.out.sent) != 1 || b.out.sent[0] != "root@example.org|[SelfPost] A message has been in the queue of mail.example.org for more than 3 days" {
t.Fatalf("four days old: %v", b.out.sent)
}
if body := b.out.body[0]; !strings.Contains(body, "OLD00001") || strings.Contains(body, "FRESH001") {
t.Errorf("the message names the wrong mail:\n%s", body)
}
b.out.take()
want(t, "still there", b.hour())
b.qErr = errors.New("postqueue: Permission denied")
b.q = nil
want(t, "queue cannot be listed", b.hour())
b.qErr = nil
want(t, "gone", b.hour(), "root@example.org|[SelfPost] The queue of mail.example.org holds nothing older than 3 days")
want(t, "after", b.hour())
}
// A limit that refuses mail is said to the domain's users, at most once a day.
func TestLimitRefusalIsSaidOnceADay(t *testing.T) {
b := newBench(t)
b.now = time.Now().UTC() // refusals are dated by the store's clock
refuse := func() {
t.Helper()
if err := b.st.InsertRejected(store.SendLogEntry{Domain: "example.com", AppLogin: "app1", From: "a@example.com"}); err != nil {
t.Fatal(err)
}
}
b.n.CheckLimits()
want(t, "nothing refused", b.out.take())
refuse()
b.n.CheckLimits()
subject := "[SelfPost] example.com: a rate limit is refusing mail"
want(t, "first refusal", b.out.take(), "mine@example.org|"+subject, "root@example.org|"+subject)
refuse()
b.now = b.now.Add(2 * time.Hour)
b.n.CheckLimits()
want(t, "more of it two hours later", b.out.take())
// A day after it was said, mail is still being refused: said again. The
// refusals are dated by the store's clock, so it is the telling that is
// moved back, not the clock forward.
st, have, err := b.st.GetNotifyState("limit:example.com:app1")
if err != nil || !have {
t.Fatalf("no state for the limit: %v, %v", have, err)
}
b.now = time.Now().UTC()
st.SentAt = b.now.Add(-25 * time.Hour)
if err := b.st.SetNotifyState(st); err != nil {
t.Fatal(err)
}
b.n.CheckLimits()
want(t, "a day later, still refusing", b.out.take(), "mine@example.org|"+subject, "root@example.org|"+subject)
if body := limitRefused("mail.example.org", store.LimitRefusal{Domain: "example.com", AppLogin: "app1", Count: 3, First: b.now}).body; !strings.Contains(body, "3 messages from the application app1 of example.com") {
t.Errorf("the message does not say who was refused how much:\n%s", body)
}
// Nobody has hit the limit for two days: it is not a subject any more.
st.SentAt = b.now.Add(-49 * time.Hour)
if err := b.st.SetNotifyState(st); err != nil {
t.Fatal(err)
}
if _, err := b.st.LimitRefusalsSince(b.now); err != nil {
t.Fatal(err)
}
b.now = b.now.Add(25 * time.Hour) // past the window the refusals lie in
b.n.CheckLimits()
want(t, "long quiet", b.out.take())
if _, have, _ := b.st.GetNotifyState("limit:example.com:app1"); have {
t.Error("a limit nobody hits kept its state")
}
}
func TestMailerComposesAndRefuses(t *testing.T) {
var from, to string
var msg []byte
m := NewMailer("mail.example.org")
m.run = func(f, t string, b []byte) error { from, to, msg = f, t, b; return nil }
if err := m.Send("root@example.org", "[SelfPost] example.com:\r\nBcc: x@evil.example", "line one\n.\nline three\n"); err != nil {
t.Fatalf("Send: %v", err)
}
if from != "postmaster@mail.example.org" || to != "root@example.org" {
t.Errorf("envelope %s -> %s", from, to)
}
text := string(msg)
head, body, ok := strings.Cut(text, "\r\n\r\n")
if !ok {
t.Fatalf("no header/body break:\n%s", text)
}
for _, h := range []string{
"From: SelfPost <postmaster@mail.example.org>",
"To: <root@example.org>",
"Subject: [SelfPost] example.com: Bcc: x@evil.example",
"Auto-Submitted: auto-generated",
"Content-Type: text/plain; charset=utf-8",
} {
if !strings.Contains(head, h+"\r\n") && !strings.HasSuffix(head, h) {
t.Errorf("header %q missing from:\n%s", h, head)
}
}
if strings.Count(head, "\r\n")+1 != 9 {
t.Errorf("a subject broke out of its line; headers:\n%s", head)
}
if body != "line one\r\n.\r\nline three\r\n" {
t.Errorf("body = %q", body)
}
for _, bad := range []string{"", "no-at-sign", "-oQ/tmp@example.org", "a@example.org\nBcc: b@example.org", "a b@example.org", "<a@example.org>"} {
if err := m.Send(bad, "s", "b"); err == nil {
t.Errorf("address %q was accepted", bad)
}
}
}
+149
View File
@@ -0,0 +1,149 @@
package notify
import (
"fmt"
"strings"
"time"
"github.com/mixeme/selfpost/internal/postfix"
"github.com/mixeme/selfpost/internal/store"
)
// message is one notification: a subject line and a plain-text body.
//
// The texts name the page of the panel to open rather than link to it: the
// panel does not know its own address - it sits behind the operator's reverse
// proxy - and a link built from a guess would be a wrong link in a message
// people are asked to trust.
type message struct {
subject string
body string
}
// maxListed is how many messages of the queue a notification names. The rest
// are counted: the page shows them all.
const maxListed = 10
func compose(hostname, subject string, paragraphs ...string) message {
paragraphs = append(paragraphs,
fmt.Sprintf("Sent by the SelfPost panel on %s. You receive this because notifications are on in your Account there; turn them off on the same page.", hostname))
return message{
subject: "[SelfPost] " + subject,
body: strings.Join(paragraphs, "\n\n") + "\n",
}
}
func stamp(t time.Time) string { return t.UTC().Format("2006-01-02 15:04 UTC") }
// plain makes text that came from outside fit for a message body: a queued
// message's sender is whatever its sender wrote, and a DNS finding may quote a
// name. What is not printable ASCII becomes "?", and it is cut to max bytes -
// a notification is read by a person, and nothing in it is theirs to act on
// but the panel's own words.
func plain(s string, max int) string {
if len(s) > max {
s = s[:max]
}
b := []byte(s)
for i, c := range b {
if c < 0x20 || c > 0x7e {
b[i] = '?'
}
}
return string(b)
}
// days words a duration in whole days, the unit these messages count in.
func days(d time.Duration) string {
n := int(d.Hours() / 24)
if n == 1 {
return "1 day"
}
return fmt.Sprintf("%d days", n)
}
func dnsProblem(hostname, domain string, faults []dnsFault, since time.Time) message {
names := make([]string, len(faults))
lines := make([]string, len(faults))
for i, f := range faults {
names[i] = f.Record
lines[i] = f.Record + ": " + plain(f.Detail, 600)
}
return compose(hostname,
fmt.Sprintf("%s: %s no longer matches", domain, strings.Join(names, ", ")),
fmt.Sprintf("The DNS records of %s do not match what this server sends with. Two checks an hour apart found the same, the first at %s.", domain, stamp(since)),
strings.Join(lines, "\n\n"),
fmt.Sprintf("Until the records are right, receivers may refuse mail from %s or file it as spam. The records to publish are in the panel under Outbound > Domains > %s.", domain, domain),
"One more message follows when the records match again.",
)
}
func dnsCleared(hostname, domain string, since time.Time) message {
return compose(hostname,
fmt.Sprintf("%s: DNS records match again", domain),
fmt.Sprintf("The DNS records of %s match what this server sends with again; two checks an hour apart agree. They had not since %s.", domain, stamp(since)),
)
}
func certExpiring(hostname string, notAfter time.Time, left time.Duration) message {
subject := fmt.Sprintf("TLS certificate of %s expires in %s", hostname, days(left))
first := fmt.Sprintf("The TLS certificate this server presents on its submission ports expires on %s, in %s.", stamp(notAfter), days(left))
if left <= 0 {
subject = fmt.Sprintf("TLS certificate of %s has expired", hostname)
first = fmt.Sprintf("The TLS certificate this server presents on its submission ports expired on %s.", stamp(notAfter))
}
return compose(hostname, subject,
first,
"Once it has expired, applications that verify the certificate can no longer submit mail. Renewal normally happens on the host, at the reverse proxy the certificate comes from; check that it still runs. The panel shows the certificate under Server > Health.",
)
}
func queueStuck(hostname string, old []postfix.QueuedMessage, now time.Time) message {
var b strings.Builder
for i, m := range old {
if i == maxListed {
fmt.Fprintf(&b, "... and %d more\n", len(old)-maxListed)
break
}
sender := m.Sender
if sender == "" {
sender = "(bounce)"
}
fmt.Fprintf(&b, "%s from %s waiting %s, since %s\n", plain(m.ID, 24), plain(sender, 120), days(now.Sub(m.Arrival)), stamp(m.Arrival))
}
what := "A message has"
if len(old) > 1 {
what = fmt.Sprintf("%d messages have", len(old))
}
return compose(hostname,
fmt.Sprintf("%s been in the queue of %s for more than %s", what, hostname, days(QueueAge)),
fmt.Sprintf("%s waited in the mail queue for more than %s. Postfix goes on retrying and returns a message to its sender when the queue lifetime runs out.", what, days(QueueAge)),
strings.TrimRight(b.String(), "\n"),
"The reason the receiving server gives is in the panel: Outbound > Queue lists what is held, and the message's page in Outbound > Log shows what Postfix logged about it.",
"One more message follows when nothing that old is left in the queue.",
)
}
func queueCleared(hostname string, since time.Time) message {
return compose(hostname,
fmt.Sprintf("The queue of %s holds nothing older than %s", hostname, days(QueueAge)),
fmt.Sprintf("No message in the mail queue is older than %s any more: what was waiting since the notification of %s was delivered, or returned to its sender.", days(QueueAge), stamp(since)),
)
}
func limitRefused(hostname string, r store.LimitRefusal) message {
who := "the domain " + r.Domain
if r.AppLogin != "" {
who = fmt.Sprintf("the application %s of %s", r.AppLogin, r.Domain)
}
count := "1 message"
if r.Count != 1 {
count = fmt.Sprintf("%d messages", r.Count)
}
return compose(hostname,
fmt.Sprintf("%s: a rate limit is refusing mail", r.Domain),
fmt.Sprintf("In the last 24 hours this server refused %s from %s under a level-2 rate limit, the first at %s. Refused mail is not queued: the application has to send it again.", count, who, stamp(r.First)),
fmt.Sprintf("If the volume is expected, raise the limit in the panel: Outbound > Domains > %s, on the domain or on the application. If it is not, the application is sending more than it should.", r.Domain),
"This limit is reported at most once a day.",
)
}
+86
View File
@@ -0,0 +1,86 @@
package postfix
import (
"bufio"
"context"
"encoding/json"
"fmt"
"os/exec"
"strings"
"time"
)
// QueuedMessage is one message Postfix is holding, as `postqueue -j` reports
// it: where it is, and since when.
type QueuedMessage struct {
ID string
Queue string // "deferred", "active", "hold", "incoming", "maildrop"
Sender string
Arrival time.Time
}
// readQueueJSON runs `postqueue -j`. It is a var so tests can answer without a
// running Postfix, the way readRetryConf is.
var readQueueJSON = postqueueJSON
func postqueueJSON() ([]byte, error) {
ctx, cancel := context.WithTimeout(context.Background(), postqueueTimeout)
defer cancel()
// A fixed flag and no user input, never through a shell (security.md).
cmd := exec.CommandContext(ctx, "postqueue", "-j")
out := &limitWriter{remaining: maxQueueOutput}
errOut := &limitWriter{remaining: 4096}
cmd.Stdout = out
cmd.Stderr = errOut
err := cmd.Run()
if ctx.Err() == context.DeadlineExceeded {
return nil, fmt.Errorf("postqueue -j timed out after %s", postqueueTimeout)
}
if err != nil {
return nil, fmt.Errorf("postqueue -j: %w: %s", err, strings.TrimSpace(errOut.buf.String()))
}
// A listing cut at the size limit is still a list of whole lines up to the
// cut; the last, broken one is dropped by the parser.
return out.buf.Bytes(), nil
}
// QueueMessages returns the messages Postfix holds with the time each arrived,
// for the notification about mail that has waited too long. An error means the
// queue could not be listed and says nothing about any message.
func QueueMessages() ([]QueuedMessage, error) {
out, err := readQueueJSON()
if err != nil {
return nil, err
}
return parseQueueJSON(out), nil
}
// parseQueueJSON reads `postqueue -j` output: one JSON object per line. A line
// that is not one - the tail of a listing cut at its size limit - is skipped.
func parseQueueJSON(out []byte) []QueuedMessage {
var msgs []QueuedMessage
sc := bufio.NewScanner(strings.NewReader(string(out)))
sc.Buffer(make([]byte, 0, 64*1024), 1<<20)
for sc.Scan() {
line := strings.TrimSpace(sc.Text())
if line == "" {
continue
}
var e struct {
QueueName string `json:"queue_name"`
QueueID string `json:"queue_id"`
ArrivalTime int64 `json:"arrival_time"`
Sender string `json:"sender"`
}
if err := json.Unmarshal([]byte(line), &e); err != nil || e.QueueID == "" || e.ArrivalTime <= 0 {
continue
}
msgs = append(msgs, QueuedMessage{
ID: e.QueueID,
Queue: e.QueueName,
Sender: e.Sender,
Arrival: time.Unix(e.ArrivalTime, 0).UTC(),
})
}
return msgs
}
+43
View File
@@ -0,0 +1,43 @@
package postfix
import (
"errors"
"testing"
"time"
)
func TestParseQueueJSON(t *testing.T) {
out := []byte(`{"queue_name": "deferred", "queue_id": "41E862C00D9E", "arrival_time": 1791000000, "message_size": 446, "forced_expire": false, "sender": "app@example.com", "recipients": [{"address": "a@example.net", "delay_reason": "connect to mx.example.net[203.0.113.9]:25: Connection timed out"}]}
{"queue_name": "active", "queue_id": "5F00AA12", "arrival_time": 1791250000, "message_size": 900, "sender": "", "recipients": [{"address": "b@example.net"}]}
not json at all
{"queue_name": "deferred", "queue_id": "", "arrival_time": 1791250000}
{"queue_name": "deferred", "queue_id": "6C00BB34", "arrival_time": 17912`)
got := parseQueueJSON(out)
if len(got) != 2 {
t.Fatalf("parsed %d messages, want 2: %+v", len(got), got)
}
if got[0].ID != "41E862C00D9E" || got[0].Queue != "deferred" || got[0].Sender != "app@example.com" ||
!got[0].Arrival.Equal(time.Unix(1791000000, 0)) {
t.Errorf("first message = %+v", got[0])
}
if got[1].ID != "5F00AA12" || got[1].Queue != "active" {
t.Errorf("second message = %+v", got[1])
}
if len(parseQueueJSON(nil)) != 0 {
t.Error("an empty queue is not empty")
}
}
// A queue that cannot be listed is an error, not an empty queue: the caller
// must not conclude that nothing is waiting.
func TestQueueMessagesReportsAFailedListing(t *testing.T) {
old := readQueueJSON
t.Cleanup(func() { readQueueJSON = old })
readQueueJSON = func() ([]byte, error) { return nil, errors.New("postqueue: Permission denied") }
if msgs, err := QueueMessages(); err == nil {
t.Fatalf("no error for a failed listing (got %v)", msgs)
}
}
+11
View File
@@ -129,6 +129,17 @@ func (p RetryPolicy) BackoffCap() string {
return FormatDuration(p.MaximalBackoff)
}
// LongestLifetime is how long Postfix may go on working on a message: the
// larger of the queue lifetime for ordinary mail and for mail from the null
// sender, which an application may submit too. It bounds which stored log
// lines can be about a send-log row (store.SendLogLines).
func (p RetryPolicy) LongestLifetime() time.Duration {
if p.BounceQueueLifetime > p.MaximalQueueLifetime {
return p.BounceQueueLifetime
}
return p.MaximalQueueLifetime
}
// QueueLifetime is the human string for maximal_queue_lifetime.
func (p RetryPolicy) QueueLifetime() string {
return FormatDuration(p.MaximalQueueLifetime)
+9 -5
View File
@@ -131,14 +131,18 @@ func (s *Store) InsertDMARCReport(rep DMARCReport) (int64, error) {
return id, nil
}
// PruneDMARCReports enforces count and age caps.
func (s *Store) PruneDMARCReports() error {
// PruneDMARCReports enforces the age cap for every domain and the count cap
// for domain. The count is kept per domain on purpose: reports are accepted
// from anyone who knows a domain's report address, and with one cap for the
// whole instance a stranger filing reports under one domain would push every
// other domain's reports out.
func (s *Store) PruneDMARCReports(domain string) error {
cutoff := time.Now().UTC().AddDate(0, 0, -DMARCReportsMaxAgeDays).Format(time.RFC3339)
if _, err := s.db.Exec(`DELETE FROM dmarc_reports WHERE received_at < ?`, cutoff); err != nil {
return fmt.Errorf("prune dmarc by age: %w", err)
}
var count int
if err := s.db.QueryRow(`SELECT COUNT(*) FROM dmarc_reports`).Scan(&count); err != nil {
if err := s.db.QueryRow(`SELECT COUNT(*) FROM dmarc_reports WHERE domain = ?`, domain).Scan(&count); err != nil {
return fmt.Errorf("count dmarc reports: %w", err)
}
if count <= DMARCReportsMaxKeep {
@@ -147,8 +151,8 @@ func (s *Store) PruneDMARCReports() error {
excess := count - DMARCReportsMaxKeep
_, err := s.db.Exec(`
DELETE FROM dmarc_reports WHERE id IN (
SELECT id FROM dmarc_reports ORDER BY received_at ASC LIMIT ?
)`, excess)
SELECT id FROM dmarc_reports WHERE domain = ? ORDER BY received_at ASC LIMIT ?
)`, domain, excess)
if err != nil {
return fmt.Errorf("prune dmarc by count: %w", err)
}
+1 -1
View File
@@ -58,7 +58,7 @@ func TestPruneDMARCReports(t *testing.T) {
t.Fatal(err)
}
}
if err := st.PruneDMARCReports(); err != nil {
if err := st.PruneDMARCReports("example.com"); err != nil {
t.Fatal(err)
}
list, _ := st.ListDMARCReports(DMARCReportScope{AllDomains: true}, 100)
+379
View File
@@ -0,0 +1,379 @@
package store
import (
"database/sql"
"errors"
"fmt"
"strings"
"time"
)
// Decisions of the inbound antispam journal. Kept in sync with the CHECK
// constraint of inbound_spam_log.
const (
SpamDecisionAccept = "accept"
SpamDecisionReject = "reject"
SpamDecisionTempfail = "tempfail"
SpamDecisionQuarantine = "quarantine"
)
// Who made a decision of the inbound antispam journal. Kept in sync with the
// CHECK constraint of inbound_spam_log.
const (
SpamEngineRspamd = "rspamd" // the external filter
SpamEnginePostfix = "postfix" // Postfix's own policy: an unknown recipient, a sender domain that does not exist
SpamEngineSelfPost = "selfpost" // nobody: relayed with no filter attached
)
// InboundSpamEntry is one row of the inbound antispam journal: what happened
// to a message offered on port 25 for one recipient. Metadata only - the
// message itself is relayed, not stored. Subject and Detail may be empty.
type InboundSpamEntry struct {
InboundDomain string
ClientIP string
From string
To string
Subject string
Decision string
Engine string
Detail string
// At is when it happened; zero means now. The log tailer dates a refusal
// by its mail.log line, which it may read long after.
At time.Time
}
// InsertInboundSpam records one journal row. It is called from the inbound
// journal milter's hot path and from the log tailer, so it returns any error
// for the caller to log; neither of them may hold up mail over it.
func (s *Store) InsertInboundSpam(e InboundSpamEntry) error {
at := e.At
if at.IsZero() {
at = time.Now()
}
_, err := s.db.Exec(
`INSERT INTO inbound_spam_log
(inbound_domain, client_ip, from_addr, to_addr, subject, decision, engine, detail, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
e.InboundDomain, e.ClientIP, e.From, e.To, e.Subject, e.Decision, e.Engine, e.Detail,
at.UTC().Format(time.RFC3339),
)
if err != nil {
return fmt.Errorf("insert inbound_spam_log: %w", err)
}
return nil
}
// InsertInboundSpamOnce records a journal row unless the same row is there
// already - same moment, client, sender, recipient, decision, engine and
// detail - and says whether it wrote. The log tailer uses it for the lines it
// reads when it starts: after an unclean stop those are lines it may have read
// before, and a refusal read twice is otherwise two rows. e.At must be set; a
// row dated "now" is never the same as another.
func (s *Store) InsertInboundSpamOnce(e InboundSpamEntry) (bool, error) {
at := e.At.UTC().Format(time.RFC3339)
res, err := s.db.Exec(
`INSERT INTO inbound_spam_log
(inbound_domain, client_ip, from_addr, to_addr, subject, decision, engine, detail, created_at)
SELECT ?, ?, ?, ?, ?, ?, ?, ?, ?
WHERE NOT EXISTS (
SELECT 1 FROM inbound_spam_log
WHERE created_at = ? AND client_ip = ? AND from_addr = ? AND to_addr = ?
AND decision = ? AND engine = ? AND detail = ?)`,
e.InboundDomain, e.ClientIP, e.From, e.To, e.Subject, e.Decision, e.Engine, e.Detail, at,
at, e.ClientIP, e.From, e.To, e.Decision, e.Engine, e.Detail,
)
if err != nil {
return false, fmt.Errorf("insert inbound_spam_log once: %w", err)
}
n, _ := res.RowsAffected()
return n > 0, nil
}
// PruneInboundSpam deletes journal rows older than cutoff and then, if more
// than keep rows are left, the oldest of those. It returns how many went.
func (s *Store) PruneInboundSpam(cutoff time.Time, keep int) (int64, error) {
res, err := s.db.Exec("DELETE FROM inbound_spam_log WHERE created_at < ?", cutoff.UTC().Format(time.RFC3339))
if err != nil {
return 0, fmt.Errorf("prune inbound_spam_log by age: %w", err)
}
n, _ := res.RowsAffected()
// Oldest by when it happened, not by id: the tailer dates a refusal by its
// log line, so after the panel was down the newest ids are old rows.
res, err = s.db.Exec(
`DELETE FROM inbound_spam_log WHERE id IN
(SELECT id FROM inbound_spam_log ORDER BY created_at DESC, id DESC LIMIT -1 OFFSET ?)`, keep)
if err != nil {
return n, fmt.Errorf("prune inbound_spam_log by count: %w", err)
}
m, _ := res.RowsAffected()
return n + m, nil
}
// InboundSpamRow is one row of the inbound antispam journal as it is read
// back for the panel: the entry as it was written, with its id and when it
// happened.
type InboundSpamRow struct {
ID int64
InboundDomain string
ClientIP string
From string
To string
Subject string
Decision string
Engine string
Detail string
CreatedAt time.Time
}
// InboundSpamFilter narrows QueryInboundSpam and CountInboundSpam. Like
// SendLogFilter it carries two kinds of narrowing that behave in opposite ways
// on purpose.
//
// Domain, Decision, Since and Until are the operator's own filters, chosen in
// the UI: an empty field matches everything. Since and Until are days, both
// inclusive and read as UTC days whatever the time of day they carry.
//
// Domains is the authorization scope, which no query parameter may widen: the
// exhaustive set of inbound domain names the caller may read, applied as an IN
// constraint. nil lifts it (the global role reads the whole journal, including
// rows of a domain deleted since); an empty set is a caller who reaches
// nothing and gets nothing. Unlike SendLogFilter, forgetting the scope is
// therefore the permissive case - the caller states "all" by leaving it out -
// so every caller that serves someone other than the global role must set it.
type InboundSpamFilter struct {
Domain string
Decision string
Since time.Time
Until time.Time
Domains []string
}
// QueryInboundSpam returns journal rows matching filter, newest first, for
// the Inbound log's server-side pagination. A refusal is dated by the mail.log
// line the tailer read it from, which can be older than rows written since, so
// the order is by time and not by id.
func (s *Store) QueryInboundSpam(filter InboundSpamFilter, limit, offset int) ([]InboundSpamRow, error) {
where, args := inboundSpamWhere(filter)
args = append(args, limit, offset)
rows, err := s.db.Query(
`SELECT id, inbound_domain, client_ip, from_addr, to_addr, subject, decision, engine, detail, created_at
FROM inbound_spam_log`+where+`
ORDER BY created_at DESC, id DESC
LIMIT ? OFFSET ?`,
args...,
)
if err != nil {
return nil, fmt.Errorf("query inbound_spam_log: %w", err)
}
defer rows.Close()
var out []InboundSpamRow
for rows.Next() {
var (
row InboundSpamRow
created string
)
if err := rows.Scan(&row.ID, &row.InboundDomain, &row.ClientIP, &row.From, &row.To,
&row.Subject, &row.Decision, &row.Engine, &row.Detail, &created); err != nil {
return nil, fmt.Errorf("scan inbound_spam_log row: %w", err)
}
row.CreatedAt, _ = time.Parse(time.RFC3339, created)
out = append(out, row)
}
return out, rows.Err()
}
// CountInboundSpam returns how many journal rows match filter, for the page
// numbers under the Inbound log.
func (s *Store) CountInboundSpam(filter InboundSpamFilter) (int64, error) {
where, args := inboundSpamWhere(filter)
var n int64
if err := s.db.QueryRow(`SELECT COUNT(*) FROM inbound_spam_log`+where, args...).Scan(&n); err != nil {
return 0, fmt.Errorf("count inbound_spam_log: %w", err)
}
return n, nil
}
func inboundSpamWhere(f InboundSpamFilter) (string, []any) {
var clauses []string
var args []any
if f.Domains != nil {
if len(f.Domains) == 0 {
return " WHERE 1 = 0", nil
}
marks := make([]string, len(f.Domains))
for i, name := range f.Domains {
marks[i] = "?"
args = append(args, name)
}
clauses = append(clauses, "inbound_domain IN ("+strings.Join(marks, ", ")+")")
}
if f.Domain != "" {
clauses = append(clauses, "inbound_domain = ?")
args = append(args, f.Domain)
}
if f.Decision != "" {
clauses = append(clauses, "decision = ?")
args = append(args, f.Decision)
}
// created_at is RFC3339 UTC, so comparing it as text against the same
// format is comparing moments: from midnight of the first day up to, not
// including, midnight after the last.
if !f.Since.IsZero() {
y, m, d := f.Since.UTC().Date()
clauses = append(clauses, "created_at >= ?")
args = append(args, time.Date(y, m, d, 0, 0, 0, 0, time.UTC).Format(time.RFC3339))
}
if !f.Until.IsZero() {
y, m, d := f.Until.UTC().Date()
clauses = append(clauses, "created_at < ?")
args = append(args, time.Date(y, m, d+1, 0, 0, 0, 0, time.UTC).Format(time.RFC3339))
}
if len(clauses) == 0 {
return "", nil
}
return " WHERE " + strings.Join(clauses, " AND "), args
}
// IsInboundDomain reports whether name is a configured inbound domain. Port 25
// also takes mail that is not the relay's - DMARC reports for the server's own
// name - and the journal keeps rows for inbound domains only.
func (s *Store) IsInboundDomain(name string) (bool, error) {
var one int
err := s.db.QueryRow("SELECT 1 FROM inbound_domains WHERE name = ?", strings.ToLower(name)).Scan(&one)
if errors.Is(err, sql.ErrNoRows) {
return false, nil
}
if err != nil {
return false, fmt.Errorf("look up inbound domain: %w", err)
}
return true, nil
}
// Lists and entry kinds of the filter lists. Kept in sync with the CHECK
// constraints of inbound_spam_list.
const (
SpamListAllow = "allow"
SpamListDeny = "deny"
SpamEntryAddress = "address" // sender address
SpamEntryDomain = "domain" // sender domain
SpamEntryIP = "ip" // client address or network
)
// ErrSpamListEntryExists is returned when the entry is already on that list.
var ErrSpamListEntryExists = errors.New("filter list entry already exists")
// ErrSpamListEntryNotFound is returned when a filter list entry id does not exist.
var ErrSpamListEntryNotFound = errors.New("filter list entry not found")
// SpamListEntry is one allow or deny entry for the external filter. Value is
// stored normalised by the caller (internal/inbound), which is what makes the
// uniqueness of an entry on its list mean anything.
type SpamListEntry struct {
ID int64
List string
Kind string
Value string
Note string
CreatedAt time.Time
}
// AddSpamListEntry inserts an entry. The caller must have validated it.
func (s *Store) AddSpamListEntry(list, kind, value, note string) (SpamListEntry, error) {
now := time.Now().UTC()
res, err := s.db.Exec(
"INSERT INTO inbound_spam_list (list_type, entry_type, value, note, created_at) VALUES (?, ?, ?, ?, ?)",
list, kind, value, note, now.Format(time.RFC3339),
)
if err != nil {
if isUniqueViolation(err) {
return SpamListEntry{}, ErrSpamListEntryExists
}
return SpamListEntry{}, fmt.Errorf("insert inbound_spam_list: %w", err)
}
id, err := res.LastInsertId()
if err != nil {
return SpamListEntry{}, fmt.Errorf("filter list entry id: %w", err)
}
return SpamListEntry{ID: id, List: list, Kind: kind, Value: value, Note: note, CreatedAt: now.Truncate(time.Second)}, nil
}
// ListSpamListEntries returns every entry of both lists: by list, then kind,
// then value.
func (s *Store) ListSpamListEntries() ([]SpamListEntry, error) {
rows, err := s.db.Query(
"SELECT id, list_type, entry_type, value, note, created_at FROM inbound_spam_list ORDER BY list_type, entry_type, value")
if err != nil {
return nil, fmt.Errorf("list inbound_spam_list: %w", err)
}
defer rows.Close()
var out []SpamListEntry
for rows.Next() {
var e SpamListEntry
var created string
if err := rows.Scan(&e.ID, &e.List, &e.Kind, &e.Value, &e.Note, &created); err != nil {
return nil, fmt.Errorf("scan inbound_spam_list: %w", err)
}
e.CreatedAt, _ = time.Parse(time.RFC3339, created)
out = append(out, e)
}
return out, rows.Err()
}
// DeleteSpamListEntry removes an entry by id.
func (s *Store) DeleteSpamListEntry(id int64) error {
res, err := s.db.Exec("DELETE FROM inbound_spam_list WHERE id = ?", id)
if err != nil {
return fmt.Errorf("delete inbound_spam_list: %w", err)
}
if n, _ := res.RowsAffected(); n == 0 {
return ErrSpamListEntryNotFound
}
return nil
}
// SpamListHit is what the journal holds about one message a filter list
// decided: who it was from, which server offered it, when, and the detail that
// says which list it was.
type SpamListHit struct {
ClientIP string
From string
Decision string
Detail string
CreatedAt time.Time
}
// ListSpamListHits returns the journal rows of the filter in whose detail one
// of the marks occurs - the marks the filter lists leave (internal/inbound).
// Which entry a row belongs to is for the caller to work out; here the rows
// are only narrowed, so that a page view does not read the whole journal.
func (s *Store) ListSpamListHits(marks ...string) ([]SpamListHit, error) {
if len(marks) == 0 {
return nil, nil
}
likes := make([]string, len(marks))
args := []any{SpamEngineRspamd}
for i, m := range marks {
likes[i] = "instr(detail, ?) > 0"
args = append(args, m)
}
rows, err := s.db.Query(
`SELECT client_ip, from_addr, decision, detail, created_at FROM inbound_spam_log
WHERE engine = ? AND (`+strings.Join(likes, " OR ")+`)`, args...)
if err != nil {
return nil, fmt.Errorf("query filter list hits: %w", err)
}
defer rows.Close()
var out []SpamListHit
for rows.Next() {
var h SpamListHit
var created string
if err := rows.Scan(&h.ClientIP, &h.From, &h.Decision, &h.Detail, &created); err != nil {
return nil, fmt.Errorf("scan filter list hit: %w", err)
}
h.CreatedAt, _ = time.Parse(time.RFC3339, created)
out = append(out, h)
}
return out, rows.Err()
}
+338
View File
@@ -0,0 +1,338 @@
package store
import (
"testing"
"time"
)
func TestInsertInboundSpam(t *testing.T) {
st := openTestStore(t)
e := InboundSpamEntry{
InboundDomain: "in.example.com", ClientIP: "203.0.113.7",
From: "sender@example.net", To: "a@in.example.com", Subject: "Hello",
Decision: SpamDecisionAccept, Engine: SpamEngineRspamd, Detail: "no action, score 0.10 / 15.00",
}
if err := st.InsertInboundSpam(e); err != nil {
t.Fatalf("InsertInboundSpam: %v", err)
}
var got InboundSpamEntry
var created string
err := st.db.QueryRow(
"SELECT inbound_domain, client_ip, from_addr, to_addr, subject, decision, engine, detail, created_at FROM inbound_spam_log",
).Scan(&got.InboundDomain, &got.ClientIP, &got.From, &got.To, &got.Subject, &got.Decision, &got.Engine, &got.Detail, &created)
if err != nil {
t.Fatalf("read back: %v", err)
}
if got != e {
t.Errorf("row = %+v\nwant %+v", got, e)
}
if at, err := time.Parse(time.RFC3339, created); err != nil || time.Since(at) > time.Minute {
t.Errorf("created_at = %q (%v)", created, err)
}
// Every decision and engine the code names is one the table takes.
for _, d := range []string{SpamDecisionAccept, SpamDecisionReject, SpamDecisionTempfail, SpamDecisionQuarantine} {
for _, eng := range []string{SpamEngineRspamd, SpamEnginePostfix, SpamEngineSelfPost} {
e.Decision, e.Engine = d, eng
if err := st.InsertInboundSpam(e); err != nil {
t.Errorf("decision %s by %s: %v", d, eng, err)
}
}
}
e.Decision = "bounce"
if err := st.InsertInboundSpam(e); err == nil {
t.Error("an unknown decision was stored")
}
}
func TestIsInboundDomain(t *testing.T) {
st := openTestStore(t)
if _, err := st.AddInboundDomain("in.example.com"); err != nil {
t.Fatalf("AddInboundDomain: %v", err)
}
for name, want := range map[string]bool{
"in.example.com": true, "IN.Example.COM": true,
"example.com": false, "": false, "other.example.com": false,
} {
if got, err := st.IsInboundDomain(name); err != nil || got != want {
t.Errorf("IsInboundDomain(%q) = %v, %v; want %v", name, got, err, want)
}
}
}
func TestInsertInboundSpamKeepsItsOwnTime(t *testing.T) {
st := openTestStore(t)
at := time.Date(2026, 10, 1, 9, 30, 0, 0, time.FixedZone("msk", 3*3600))
err := st.InsertInboundSpam(InboundSpamEntry{
InboundDomain: "in.example.com", ClientIP: "203.0.113.7", To: "a@in.example.com",
Decision: SpamDecisionReject, Engine: SpamEnginePostfix, At: at,
})
if err != nil {
t.Fatalf("InsertInboundSpam: %v", err)
}
var created string
if err := st.db.QueryRow("SELECT created_at FROM inbound_spam_log").Scan(&created); err != nil || created != "2026-10-01T06:30:00Z" {
t.Errorf("created_at = %q, %v", created, err)
}
}
func TestPruneInboundSpam(t *testing.T) {
st := openTestStore(t)
now := time.Now().UTC()
add := func(to string, age time.Duration) {
t.Helper()
err := st.InsertInboundSpam(InboundSpamEntry{
InboundDomain: "in.example.com", ClientIP: "203.0.113.7", To: to,
Decision: SpamDecisionReject, Engine: SpamEnginePostfix, At: now.Add(-age),
})
if err != nil {
t.Fatalf("InsertInboundSpam: %v", err)
}
}
left := func() string {
t.Helper()
rows, err := st.db.Query("SELECT to_addr FROM inbound_spam_log ORDER BY id")
if err != nil {
t.Fatal(err)
}
defer rows.Close()
var out string
for rows.Next() {
var to string
if err := rows.Scan(&to); err != nil {
t.Fatal(err)
}
out += to + " "
}
return out
}
add("old", 100*24*time.Hour)
add("a", 3*time.Hour)
add("b", 2*time.Hour)
add("c", time.Hour)
// Age first: nothing else goes while the rest fits.
if n, err := st.PruneInboundSpam(now.AddDate(0, 0, -90), 10); err != nil || n != 1 {
t.Fatalf("prune by age = %d, %v", n, err)
}
if got := left(); got != "a b c " {
t.Fatalf("left = %q", got)
}
// Then the cap, oldest first.
if n, err := st.PruneInboundSpam(now.AddDate(0, 0, -90), 2); err != nil || n != 1 {
t.Fatalf("prune by count = %d, %v", n, err)
}
if got := left(); got != "b c " {
t.Fatalf("left = %q", got)
}
if n, err := st.PruneInboundSpam(now.AddDate(0, 0, -90), 2); err != nil || n != 0 {
t.Errorf("a second prune removed %d, %v", n, err)
}
}
func TestInboundSpamLogRetentionDays(t *testing.T) {
st := openTestStore(t)
if days, err := st.GetInboundSpamLogRetentionDays(30); err != nil || days != 30 {
t.Errorf("unset, env 30: %d, %v", days, err)
}
if days, err := st.GetInboundSpamLogRetentionDays(3); err != nil || days != InboundSpamLogRetentionDaysDefault {
t.Errorf("unset, env out of range: %d, %v", days, err)
}
if err := st.EnsureInboundSpamLogRetentionDays(30); err != nil {
t.Fatalf("Ensure: %v", err)
}
if err := st.EnsureInboundSpamLogRetentionDays(60); err != nil {
t.Fatalf("Ensure again: %v", err)
}
if days, _ := st.GetInboundSpamLogRetentionDays(90); days != 30 {
t.Errorf("after Ensure(30), Ensure(60): %d, want the first seed kept", days)
}
if err := st.SetInboundSpamLogRetentionDays(6); err == nil {
t.Error("6 days was accepted")
}
if err := st.SetInboundSpamLogRetentionDays(365); err != nil {
t.Errorf("365 days: %v", err)
}
if days, _ := st.GetInboundSpamLogRetentionDays(90); days != 365 {
t.Errorf("after Set(365): %d", days)
}
}
func TestQueryInboundSpamFilters(t *testing.T) {
st := openTestStore(t)
day := func(d, h int) time.Time { return time.Date(2026, 9, d, h, 0, 0, 0, time.UTC) }
add := func(domain, to, decision string, at time.Time) {
t.Helper()
err := st.InsertInboundSpam(InboundSpamEntry{
InboundDomain: domain, ClientIP: "203.0.113.7", From: "s@example.net", To: to,
Subject: "S " + to, Decision: decision, Engine: SpamEngineRspamd, Detail: "d", At: at,
})
if err != nil {
t.Fatalf("InsertInboundSpam: %v", err)
}
}
add("a.example", "1@a.example", SpamDecisionAccept, day(20, 23))
add("a.example", "2@a.example", SpamDecisionReject, day(21, 0))
add("b.example", "3@b.example", SpamDecisionAccept, day(21, 12))
add("b.example", "4@b.example", SpamDecisionTempfail, day(22, 0))
// Written last but dated earliest: the tailer reads a refusal after the fact.
add("a.example", "5@a.example", SpamDecisionReject, day(19, 8))
to := func(f InboundSpamFilter) string {
t.Helper()
rows, err := st.QueryInboundSpam(f, 50, 0)
if err != nil {
t.Fatalf("QueryInboundSpam(%+v): %v", f, err)
}
n, err := st.CountInboundSpam(f)
if err != nil || n != int64(len(rows)) {
t.Fatalf("CountInboundSpam(%+v) = %d, %v; the query returned %d", f, n, err, len(rows))
}
var out string
for _, r := range rows {
out += r.To[:1]
}
return out
}
// Newest first by time, not by the order of writing.
if got := to(InboundSpamFilter{}); got != "43215" {
t.Errorf("no filter = %q, want 43215", got)
}
if got := to(InboundSpamFilter{Domain: "a.example"}); got != "215" {
t.Errorf("domain = %q, want 215", got)
}
if got := to(InboundSpamFilter{Decision: SpamDecisionReject}); got != "25" {
t.Errorf("decision = %q, want 25", got)
}
// Days are whole UTC days and both ends are in: 21 st is 00:00 up to 23:59.
if got := to(InboundSpamFilter{Since: day(21, 15), Until: day(21, 3)}); got != "32" {
t.Errorf("one day = %q, want 32", got)
}
if got := to(InboundSpamFilter{Since: day(22, 0)}); got != "4" {
t.Errorf("since = %q, want 4", got)
}
if got := to(InboundSpamFilter{Until: day(20, 0)}); got != "15" {
t.Errorf("until = %q, want 15", got)
}
// A time in another zone is read as the UTC day it falls on.
msk := time.FixedZone("msk", 3*3600)
if got := to(InboundSpamFilter{Since: time.Date(2026, 9, 22, 2, 0, 0, 0, msk)}); got != "432" {
t.Errorf("since in another zone = %q, want 432 (it is the 21st in UTC)", got)
}
if got := to(InboundSpamFilter{Domain: "b.example", Decision: SpamDecisionAccept, Since: day(21, 0), Until: day(21, 0)}); got != "3" {
t.Errorf("all filters = %q, want 3", got)
}
// The scope: nil is everything, a set is its domains, an empty set nothing,
// and a chosen domain outside the set is nothing too.
if got := to(InboundSpamFilter{Domains: []string{"b.example"}}); got != "43" {
t.Errorf("scope = %q, want 43", got)
}
if got := to(InboundSpamFilter{Domains: []string{}}); got != "" {
t.Errorf("empty scope = %q, want nothing", got)
}
if got := to(InboundSpamFilter{Domains: []string{"b.example"}, Domain: "a.example"}); got != "" {
t.Errorf("a chosen domain outside the scope = %q, want nothing", got)
}
// Paging.
rows, err := st.QueryInboundSpam(InboundSpamFilter{}, 2, 2)
if err != nil || len(rows) != 2 || rows[0].To != "2@a.example" || rows[1].To != "1@a.example" {
t.Errorf("page 2 of 2 rows = %+v, %v", rows, err)
}
if rows[0].ID == 0 || !rows[0].CreatedAt.Equal(day(21, 0)) || rows[0].Subject != "S 2@a.example" ||
rows[0].Engine != SpamEngineRspamd || rows[0].Detail != "d" || rows[0].ClientIP != "203.0.113.7" || rows[0].From != "s@example.net" {
t.Errorf("a row read back = %+v", rows[0])
}
// A value is data, not SQL.
if got := to(InboundSpamFilter{Domain: "a.example' OR '1'='1"}); got != "" {
t.Errorf("injected domain = %q", got)
}
}
func TestListSpamListHits(t *testing.T) {
st := openTestStore(t)
add := func(engine, decision, detail string) {
t.Helper()
err := st.InsertInboundSpam(InboundSpamEntry{
InboundDomain: "in.example.com", ClientIP: "203.0.113.7", From: "a@example.net", To: "a@in.example.com",
Decision: decision, Engine: engine, Detail: detail, At: time.Date(2026, 10, 1, 9, 0, 0, 0, time.UTC),
})
if err != nil {
t.Fatal(err)
}
}
add(SpamEngineRspamd, SpamDecisionReject, "5.7.1 Refused by this server's deny list")
add(SpamEngineRspamd, SpamDecisionAccept, "no action, score -99.00 / 15.00; SELFPOST_ALLOW_IP(-100.00)")
add(SpamEngineRspamd, SpamDecisionReject, "5.7.1 Spam message rejected")
add(SpamEnginePostfix, SpamDecisionReject, "550 5.1.1 this server's deny list")
// a mark with the characters LIKE would treat as wildcards is text
add(SpamEngineRspamd, SpamDecisionAccept, "no action; SELFPOSTXALLOWX(1.00)")
hits, err := st.ListSpamListHits("this server's deny list", "SELFPOST_ALLOW_")
if err != nil || len(hits) != 2 {
t.Fatalf("hits = %+v, %v; want the two rows a list decided", hits, err)
}
h := hits[0]
if h.From != "a@example.net" || h.ClientIP != "203.0.113.7" || h.Decision != SpamDecisionReject || h.CreatedAt.Hour() != 9 {
t.Errorf("hit = %+v", h)
}
if hits, err := st.ListSpamListHits(); err != nil || hits != nil {
t.Errorf("no marks: %v, %v", hits, err)
}
}
// The cap removes the oldest by when it happened. After the panel was down the
// tailer writes old refusals with the newest ids, and they are not what stays.
func TestPruneInboundSpamCapIsByTime(t *testing.T) {
st := openTestStore(t)
now := time.Now().UTC()
for _, r := range []struct {
to string
age time.Duration
}{{"new-1", time.Minute}, {"new-2", 2 * time.Minute}, {"backdated-old", 5 * time.Hour}} {
err := st.InsertInboundSpam(InboundSpamEntry{
InboundDomain: "in.example.com", ClientIP: "203.0.113.7", To: r.to,
Decision: SpamDecisionReject, Engine: SpamEnginePostfix, At: now.Add(-r.age),
})
if err != nil {
t.Fatal(err)
}
}
if n, err := st.PruneInboundSpam(time.Time{}, 2); err != nil || n != 1 {
t.Fatalf("prune = %d, %v", n, err)
}
rows, err := st.QueryInboundSpam(InboundSpamFilter{}, 10, 0)
if err != nil || len(rows) != 2 || rows[0].To != "new-1" || rows[1].To != "new-2" {
t.Errorf("left = %+v, %v; want the two newest by time", rows, err)
}
}
func TestInsertInboundSpamOnce(t *testing.T) {
st := openTestStore(t)
e := InboundSpamEntry{
InboundDomain: "in.example.com", ClientIP: "203.0.113.7", From: "a@example.net", To: "nobody@in.example.com",
Decision: SpamDecisionReject, Engine: SpamEnginePostfix, Detail: "550 5.1.1 no",
At: time.Date(2026, 10, 12, 9, 0, 5, 0, time.UTC),
}
for i, want := range []bool{true, false, false} {
if wrote, err := st.InsertInboundSpamOnce(e); err != nil || wrote != want {
t.Fatalf("attempt %d: wrote %v, %v; want %v", i+1, wrote, err, want)
}
}
// Anything that tells two refusals apart makes a second row.
later, other := e, e
later.At = e.At.Add(time.Second)
other.To = "somebody@in.example.com"
for name, v := range map[string]InboundSpamEntry{"a second later": later, "another recipient": other} {
if wrote, err := st.InsertInboundSpamOnce(v); err != nil || !wrote {
t.Errorf("%s: wrote %v, %v", name, wrote, err)
}
}
if n, err := st.CountInboundSpam(InboundSpamFilter{}); err != nil || n != 3 {
t.Errorf("rows = %d, %v; want 3", n, err)
}
}
@@ -0,0 +1,40 @@
-- Inbound antispam journal and filter lists (inbound-antispam-panel, 2.1.0).
-- The first migration after the 2.0 baseline: DDL only, so a 2.0.x database
-- gains two empty tables and nothing in it changes.
-- What happened to a message offered on port 25, one row per recipient.
-- Metadata only - SelfPost relays inbound mail and stores none of it. Accepted
-- messages are written by the inbound journal milter, refused ones by the
-- mail.log tailer, which is why a row can lack a subject (refused before DATA)
-- and why engine says who decided: the external filter, Postfix's own policy,
-- or nobody (selfpost: relayed with no filter attached).
CREATE TABLE inbound_spam_log (
id INTEGER PRIMARY KEY AUTOINCREMENT,
inbound_domain TEXT NOT NULL,
client_ip TEXT NOT NULL,
from_addr TEXT NOT NULL,
to_addr TEXT NOT NULL,
subject TEXT NOT NULL,
decision TEXT NOT NULL CHECK (decision IN ('accept', 'reject', 'tempfail', 'quarantine')),
engine TEXT NOT NULL CHECK (engine IN ('rspamd', 'postfix', 'selfpost')),
detail TEXT NOT NULL,
created_at TEXT NOT NULL
);
-- Retention prunes by age; the Log page lists newest first, for the whole
-- server or for one domain.
CREATE INDEX idx_inbound_spam_log_created_at ON inbound_spam_log (created_at);
CREATE INDEX idx_inbound_spam_log_domain_created_at ON inbound_spam_log (inbound_domain, created_at);
-- Allow / deny entries for the external filter, instance-wide. The panel
-- writes them out as map files under /data/antispam/; SelfPost itself does not
-- consult them. value is stored normalised, so the same entry cannot be added
-- twice to one list.
CREATE TABLE inbound_spam_list (
id INTEGER PRIMARY KEY AUTOINCREMENT,
list_type TEXT NOT NULL CHECK (list_type IN ('allow', 'deny')),
entry_type TEXT NOT NULL CHECK (entry_type IN ('address', 'domain', 'ip')),
value TEXT NOT NULL,
note TEXT NOT NULL DEFAULT '',
created_at TEXT NOT NULL,
UNIQUE (list_type, entry_type, value)
);
@@ -0,0 +1,22 @@
-- What Postfix wrote about a message, kept with its send-log rows
-- (delivery-log-storage, 2.2.0). DDL only: a 2.1.0 database gains one empty
-- table and nothing in it changes.
-- One row per mail.log line headed by a queue id, written by the log tailer as
-- it reads the log. mail.log is rotated daily and fourteen files are kept; a
-- send-log row lives for the retention window, ninety days by default, and
-- its page shows these lines for as long.
--
-- There is no reference to send_log: the first lines of a message are logged
-- before the journal milter writes its row, and a message with several
-- recipients is several rows under one queue id. A line belongs to a row by
-- queue id and by time (Postfix reuses queue ids), and a line whose queue id
-- has no send-log row is deleted by the tailer's prune.
CREATE TABLE send_log_lines (
id INTEGER PRIMARY KEY AUTOINCREMENT,
queue_id TEXT NOT NULL,
logged_at TEXT NOT NULL,
line TEXT NOT NULL
);
-- A message's page reads the lines of one queue id within a span of time.
CREATE INDEX idx_send_log_lines_queue_id ON send_log_lines (queue_id, logged_at);
@@ -0,0 +1,28 @@
-- Notifications: the panel writes to its users when something needs them
-- (panel-notifications, 2.3.0). DDL only; nothing existing changes, and every
-- user starts with notifications on.
-- A user's own switch, on Account. It decides nothing by itself: a user is
-- written to when it is on, the profile has an e-mail and the instance switch
-- (settings key "notifications") is on.
ALTER TABLE users ADD COLUMN notify INTEGER NOT NULL DEFAULT 1 CHECK (notify IN (0, 1));
-- What the panel has already said, so that it says each thing once and a
-- restart announces nothing again. One row per subject:
-- dns:<domain> a sending domain's records
-- cert the TLS certificate
-- queue a message older than three days
-- limit:<domain>:<login> a level-2 limit that refused mail
-- phase is where the subject stands ("pending": seen wrong, not yet said;
-- "open": said; for the certificate, the threshold last said). count is the
-- checks in a row that agree with the phase's direction, mark what the phase
-- refers to (the certificate's expiry), since when it began, sent_at when the
-- last message about it went out.
CREATE TABLE notify_state (
key TEXT PRIMARY KEY,
phase TEXT NOT NULL,
count INTEGER NOT NULL DEFAULT 0,
mark TEXT NOT NULL DEFAULT '',
since TEXT NOT NULL,
sent_at TEXT NOT NULL DEFAULT ''
);
+194
View File
@@ -0,0 +1,194 @@
package store
import (
"database/sql"
"errors"
"fmt"
"time"
)
// settingNotifications is the instance switch for notification e-mails. A
// missing value means on: the switch exists to turn them off.
const settingNotifications = "notifications"
// NotificationsEnabled reports whether the instance sends notification e-mails
// at all (Server › Settings).
func (s *Store) NotificationsEnabled() (bool, error) {
v, err := s.GetSetting(settingNotifications)
if err != nil {
return false, err
}
return v != "off", nil
}
// SetNotificationsEnabled turns notification e-mails on or off for everyone.
func (s *Store) SetNotificationsEnabled(on bool) error {
v := "on"
if !on {
v = "off"
}
return s.SetSetting(settingNotifications, v)
}
// SetUserNotify sets a user's own switch (Account).
func (s *Store) SetUserNotify(id int64, on bool) error {
res, err := s.db.Exec("UPDATE users SET notify = ? WHERE id = ?", on, id)
if err != nil {
return fmt.Errorf("set user notify: %w", err)
}
if n, _ := res.RowsAffected(); n == 0 {
return ErrUserNotFound
}
return nil
}
// NotifyRecipients returns the addresses a notification goes to: every global
// user, and for a subject that is a sending domain's (domainName not empty)
// also the domain users it is assigned to - by name on their list, or by
// "all domains". Only users with an e-mail and their own switch on; an address
// two users share is returned once.
func (s *Store) NotifyRecipients(domainName string) ([]string, error) {
query := `SELECT DISTINCT email FROM users
WHERE notify = 1 AND email <> '' AND role = ?`
args := []any{RoleGlobal}
if domainName != "" {
query = `SELECT DISTINCT email FROM users
WHERE notify = 1 AND email <> '' AND (
role = ?
OR all_domains = 1
OR EXISTS (SELECT 1 FROM user_domains ud JOIN domains d ON d.id = ud.domain_id
WHERE ud.user_id = users.id AND d.name = ?))`
args = append(args, domainName)
}
rows, err := s.db.Query(query+" ORDER BY email", args...)
if err != nil {
return nil, fmt.Errorf("notify recipients: %w", err)
}
defer rows.Close()
var out []string
for rows.Next() {
var email string
if err := rows.Scan(&email); err != nil {
return nil, fmt.Errorf("notify recipients scan: %w", err)
}
out = append(out, email)
}
return out, rows.Err()
}
// NotifyState is what the panel remembers about one subject of notification
// (migration 0004 describes the keys and the fields).
type NotifyState struct {
Key string
Phase string
Count int
Mark string
Since time.Time
SentAt time.Time // zero when nothing was sent yet
}
// GetNotifyState returns the state kept under key; ok is false when there is
// none, which is the state of a subject nothing is wrong with.
func (s *Store) GetNotifyState(key string) (NotifyState, bool, error) {
st := NotifyState{Key: key}
var since, sentAt string
err := s.db.QueryRow(
"SELECT phase, count, mark, since, sent_at FROM notify_state WHERE key = ?", key,
).Scan(&st.Phase, &st.Count, &st.Mark, &since, &sentAt)
if errors.Is(err, sql.ErrNoRows) {
return NotifyState{}, false, nil
}
if err != nil {
return NotifyState{}, false, fmt.Errorf("get notify state %s: %w", key, err)
}
st.Since, _ = time.Parse(time.RFC3339, since)
if sentAt != "" {
st.SentAt, _ = time.Parse(time.RFC3339, sentAt)
}
return st, true, nil
}
// SetNotifyState writes the state of one subject, replacing what was there.
func (s *Store) SetNotifyState(st NotifyState) error {
sentAt := ""
if !st.SentAt.IsZero() {
sentAt = st.SentAt.UTC().Format(time.RFC3339)
}
_, err := s.db.Exec(
`INSERT INTO notify_state (key, phase, count, mark, since, sent_at) VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT(key) DO UPDATE SET phase = excluded.phase, count = excluded.count,
mark = excluded.mark, since = excluded.since, sent_at = excluded.sent_at`,
st.Key, st.Phase, st.Count, st.Mark, st.Since.UTC().Format(time.RFC3339), sentAt,
)
if err != nil {
return fmt.Errorf("set notify state %s: %w", st.Key, err)
}
return nil
}
// DeleteNotifyState forgets a subject: nothing is wrong with it any more, or
// it no longer exists.
func (s *Store) DeleteNotifyState(key string) error {
if _, err := s.db.Exec("DELETE FROM notify_state WHERE key = ?", key); err != nil {
return fmt.Errorf("delete notify state %s: %w", key, err)
}
return nil
}
// ListNotifyStateKeys returns the keys that begin with prefix, so that the
// states of subjects that are gone - a deleted domain - can be found.
func (s *Store) ListNotifyStateKeys(prefix string) ([]string, error) {
rows, err := s.db.Query(
"SELECT key FROM notify_state WHERE substr(key, 1, ?) = ? ORDER BY key", len(prefix), prefix)
if err != nil {
return nil, fmt.Errorf("list notify state: %w", err)
}
defer rows.Close()
var out []string
for rows.Next() {
var key string
if err := rows.Scan(&key); err != nil {
return nil, fmt.Errorf("list notify state scan: %w", err)
}
out = append(out, key)
}
return out, rows.Err()
}
// LimitRefusal is the mail one application was refused under a level-2 rate
// limit in a span of time: how many messages and when the first.
type LimitRefusal struct {
Domain string
AppLogin string
Count int
First time.Time
}
// LimitRefusalsSince returns, per domain and application, the messages the
// journal milter refused under a level-2 limit at or after since (send-log
// rows with status "rejected").
func (s *Store) LimitRefusalsSince(since time.Time) ([]LimitRefusal, error) {
rows, err := s.db.Query(
`SELECT COALESCE(domain, ''), COALESCE(app_login, ''), COUNT(*), MIN(created_at) FROM send_log
WHERE status = ? AND created_at >= ?
GROUP BY domain, app_login ORDER BY domain, app_login`,
StatusRejected, since.UTC().Format(time.RFC3339),
)
if err != nil {
return nil, fmt.Errorf("limit refusals: %w", err)
}
defer rows.Close()
var out []LimitRefusal
for rows.Next() {
var (
r LimitRefusal
first string
)
if err := rows.Scan(&r.Domain, &r.AppLogin, &r.Count, &first); err != nil {
return nil, fmt.Errorf("limit refusals scan: %w", err)
}
r.First, _ = time.Parse(time.RFC3339, first)
out = append(out, r)
}
return out, rows.Err()
}
+166
View File
@@ -0,0 +1,166 @@
package store
import (
"strings"
"testing"
"time"
)
// The instance switch is on until someone turns it off: a database that never
// saw the setting sends notifications.
func TestNotificationsEnabledByDefault(t *testing.T) {
s := openTestStore(t)
if on, err := s.NotificationsEnabled(); err != nil || !on {
t.Fatalf("a fresh database: enabled = %v, %v; want on", on, err)
}
if err := s.SetNotificationsEnabled(false); err != nil {
t.Fatal(err)
}
if on, err := s.NotificationsEnabled(); err != nil || on {
t.Errorf("after turning off: enabled = %v, %v", on, err)
}
if err := s.SetNotificationsEnabled(true); err != nil {
t.Fatal(err)
}
if on, err := s.NotificationsEnabled(); err != nil || !on {
t.Errorf("after turning on again: enabled = %v, %v", on, err)
}
}
// Who is written to: every global user about everything, a domain user about
// the domains on their list or all of them - and nobody without an e-mail or
// with their own switch off.
func TestNotifyRecipients(t *testing.T) {
s := openTestStore(t)
mine, err := s.AddDomain("mine.example.com", "sp")
if err != nil {
t.Fatal(err)
}
other, err := s.AddDomain("other.example.com", "sp")
if err != nil {
t.Fatal(err)
}
user := func(name string, role Role, reach Reach, email string, notify bool) {
t.Helper()
id, err := s.CreateUser(name, "hash", role, reach)
if err != nil {
t.Fatalf("create %s: %v", name, err)
}
if err := s.UpdateUser(id, name, "hash", email); err != nil {
t.Fatalf("update %s: %v", name, err)
}
if u, err := s.GetUser(id); err != nil || !u.Notify {
t.Fatalf("%s: a new user's switch is %v, %v; want on", name, u.Notify, err)
}
if !notify {
if err := s.SetUserNotify(id, false); err != nil {
t.Fatalf("switch off %s: %v", name, err)
}
}
}
user("root", RoleGlobal, Reach{}, "root@example.org", true)
user("root2", RoleGlobal, Reach{}, "root@example.org", true) // the same mailbox
user("quiet", RoleGlobal, Reach{}, "quiet@example.org", false)
user("nomail", RoleGlobal, Reach{}, "", true)
user("mine", RoleDomain, Reach{DomainIDs: []int64{mine.ID}}, "mine@example.org", true)
user("theirs", RoleDomain, Reach{DomainIDs: []int64{other.ID}}, "theirs@example.org", true)
user("every", RoleDomain, Reach{AllDomains: true}, "every@example.org", true)
user("mineoff", RoleDomain, Reach{DomainIDs: []int64{mine.ID}}, "mineoff@example.org", false)
for _, c := range []struct{ domain, want string }{
{"", "root@example.org"},
{"mine.example.com", "every@example.org mine@example.org root@example.org"},
{"other.example.com", "every@example.org root@example.org theirs@example.org"},
{"gone.example.com", "every@example.org root@example.org"},
} {
got, err := s.NotifyRecipients(c.domain)
if err != nil {
t.Fatalf("recipients for %q: %v", c.domain, err)
}
if strings.Join(got, " ") != c.want {
t.Errorf("recipients for %q = %q, want %q", c.domain, strings.Join(got, " "), c.want)
}
}
if err := s.SetUserNotify(99999, true); err != ErrUserNotFound {
t.Errorf("switch of a user that does not exist: %v", err)
}
}
// What has been said is remembered by key, replaced whole, and forgotten.
func TestNotifyStateRoundTrip(t *testing.T) {
s := openTestStore(t)
if _, ok, err := s.GetNotifyState("dns:example.com"); err != nil || ok {
t.Fatalf("no state yet: ok = %v, %v", ok, err)
}
since := time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC)
st := NotifyState{Key: "dns:example.com", Phase: "pending", Count: 1, Since: since}
if err := s.SetNotifyState(st); err != nil {
t.Fatal(err)
}
got, ok, err := s.GetNotifyState("dns:example.com")
if err != nil || !ok || got.Phase != "pending" || got.Count != 1 || !got.Since.Equal(since) || !got.SentAt.IsZero() {
t.Fatalf("state = %+v, %v, %v", got, ok, err)
}
st.Phase, st.Count, st.Mark, st.SentAt = "open", 0, "x", since.Add(time.Hour)
if err := s.SetNotifyState(st); err != nil {
t.Fatal(err)
}
got, _, _ = s.GetNotifyState("dns:example.com")
if got.Phase != "open" || got.Mark != "x" || !got.SentAt.Equal(since.Add(time.Hour)) {
t.Errorf("replaced state = %+v", got)
}
for _, key := range []string{"dns:other.example.com", "queue", "limit:example.com:app"} {
if err := s.SetNotifyState(NotifyState{Key: key, Phase: "open", Since: since}); err != nil {
t.Fatal(err)
}
}
keys, err := s.ListNotifyStateKeys("dns:")
if err != nil || strings.Join(keys, " ") != "dns:example.com dns:other.example.com" {
t.Errorf("dns keys = %v, %v", keys, err)
}
if err := s.DeleteNotifyState("dns:example.com"); err != nil {
t.Fatal(err)
}
if _, ok, _ := s.GetNotifyState("dns:example.com"); ok {
t.Error("state still there after delete")
}
}
// Refused mail is counted per domain and application from a point in time;
// delivered mail and older refusals are not.
func TestLimitRefusalsSince(t *testing.T) {
s := openTestStore(t)
for _, e := range []SendLogEntry{
{Domain: "example.com", AppLogin: "app1", From: "a@example.com"},
{Domain: "example.com", AppLogin: "app1", From: "a@example.com"},
{Domain: "example.com", AppLogin: "app2", From: "b@example.com"},
} {
if err := s.InsertRejected(e); err != nil {
t.Fatal(err)
}
}
if err := s.InsertQueued(SendLogEntry{QueueID: "ABC123", Domain: "example.com", AppLogin: "app3", To: "x@example.net"}); err != nil {
t.Fatal(err)
}
if err := s.InsertRejected(SendLogEntry{Domain: "old.example.com", AppLogin: "old"}); err != nil {
t.Fatal(err)
}
if _, err := s.db.Exec(`UPDATE send_log SET created_at = '2026-01-01T00:00:00Z' WHERE app_login = 'old'`); err != nil {
t.Fatal(err)
}
got, err := s.LimitRefusalsSince(time.Now().UTC().Add(-time.Hour))
if err != nil {
t.Fatal(err)
}
if len(got) != 2 || got[0].AppLogin != "app1" || got[0].Count != 2 || got[1].AppLogin != "app2" || got[1].Count != 1 {
t.Fatalf("refusals = %+v", got)
}
if got[0].Domain != "example.com" || got[0].First.IsZero() {
t.Errorf("first refusal = %+v", got[0])
}
}
+99 -4
View File
@@ -40,7 +40,7 @@ func TestBaselineSchema(t *testing.T) {
st := openTestStore(t)
want := map[string]string{
"settings": "key value",
"users": "id username password_hash role email all_domains all_inbound_domains created_at",
"users": "id username password_hash role email all_domains all_inbound_domains created_at notify",
"sessions": "token_hash user_id created_at expires_at",
"domains": "id name dkim_selector dmarc_rua created_at",
"user_domains": "user_id domain_id",
@@ -55,6 +55,10 @@ func TestBaselineSchema(t *testing.T) {
"user_inbound_domains": "user_id inbound_domain_id",
"dmarc_reports": "id domain reporter report_id period_begin period_end received_at contact_email policy_p policy_sp policy_pct policy_adkim policy_aspf pass_count fail_count recipient",
"dmarc_report_records": "id report_row_id source_ip count disposition spf_result dkim_result header_from",
"inbound_spam_log": "id inbound_domain client_ip from_addr to_addr subject decision engine detail created_at",
"inbound_spam_list": "id list_type entry_type value note created_at",
"send_log_lines": "id queue_id logged_at line",
"notify_state": "key phase count mark since sent_at",
}
rows, err := st.db.Query("SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%' ORDER BY name")
if err != nil {
@@ -99,7 +103,8 @@ func TestBaselineSchema(t *testing.T) {
rows.Close()
wantIdx := []string{
"idx_dmarc_report_records_report", "idx_dmarc_reports_domain", "idx_dmarc_reports_received",
"idx_send_log_app_login_created_at", "idx_send_log_created_at", "idx_send_log_domain", "idx_send_log_queue_id",
"idx_inbound_spam_log_created_at", "idx_inbound_spam_log_domain_created_at",
"idx_send_log_app_login_created_at", "idx_send_log_created_at", "idx_send_log_domain", "idx_send_log_lines_queue_id", "idx_send_log_queue_id",
"idx_sessions_expires_at",
}
sort.Strings(wantIdx)
@@ -108,8 +113,8 @@ func TestBaselineSchema(t *testing.T) {
}
var version, appID int64
if err := st.db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 1 {
t.Errorf("user_version = %d, %v; want 1", version, err)
if err := st.db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 4 {
t.Errorf("user_version = %d, %v; want 4", version, err)
}
if err := st.db.QueryRow("PRAGMA application_id").Scan(&appID); err != nil || appID != applicationID {
t.Errorf("application_id = %#x, %v; want %#x", appID, err, applicationID)
@@ -156,6 +161,96 @@ func TestReopenKeepsTheDatabase(t *testing.T) {
}
}
// A 2.0.x data directory - the baseline and nothing after it - is brought to
// the head by opening it, with what it holds left alone.
func TestOpenMigratesA20Database(t *testing.T) {
path := filepath.Join(t.TempDir(), "test.db")
baseline, err := migrationsFS.ReadFile("migrations/0001_init.sql")
if err != nil {
t.Fatalf("read the baseline: %v", err)
}
db, err := sql.Open("sqlite", "file:"+path)
if err != nil {
t.Fatalf("create a 2.0 db: %v", err)
}
for _, stmt := range []string{
string(baseline),
"PRAGMA user_version = 1",
"INSERT INTO users (username, password_hash, role, created_at) VALUES ('admin', 'hash', 'global', '2026-10-11T00:00:00Z')",
} {
if _, err := db.Exec(stmt); err != nil {
t.Fatalf("prepare a 2.0 db: %v", err)
}
}
db.Close()
st, err := Open(path)
if err != nil {
t.Fatalf("open a 2.0 database: %v", err)
}
defer st.Close()
var version int
if err := st.db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 4 {
t.Errorf("user_version = %d, %v; want 4", version, err)
}
if ok, err := st.UserExists(); err != nil || !ok {
t.Errorf("the user of the 2.0 database: exists = %v, %v", ok, err)
}
for _, table := range []string{"inbound_spam_log", "inbound_spam_list", "send_log_lines", "notify_state"} {
if len(tableColumns(t, st, table)) == 0 {
t.Errorf("table %s was not created", table)
}
}
}
// What the antispam tables refuse: a decision, engine or list the panel does
// not know, and the same entry twice in one list.
func TestInboundSpamConstraints(t *testing.T) {
st := openTestStore(t)
logRow := func(decision, engine string) error {
_, err := st.db.Exec(
"INSERT INTO inbound_spam_log (inbound_domain, client_ip, from_addr, to_addr, subject, decision, engine, detail, created_at)"+
" VALUES ('in.example.com', '192.0.2.1', 'a@example.net', 'b@in.example.com', '', ?, ?, '', '2026-10-11T00:00:00Z')",
decision, engine)
return err
}
if err := logRow("reject", "postfix"); err != nil {
t.Fatalf("a reject by postfix: %v", err)
}
if err := logRow("bounce", "postfix"); err == nil {
t.Error("decision bounce was accepted")
}
if err := logRow("accept", "spamassassin"); err == nil {
t.Error("engine spamassassin was accepted")
}
entry := func(list, kind, value string) error {
_, err := st.db.Exec(
"INSERT INTO inbound_spam_list (list_type, entry_type, value, created_at) VALUES (?, ?, ?, '2026-10-11T00:00:00Z')",
list, kind, value)
return err
}
if err := entry("allow", "address", "billing@vendor.example"); err != nil {
t.Fatalf("an allow entry: %v", err)
}
if err := entry("allow", "address", "billing@vendor.example"); err == nil {
t.Error("the same entry was added to one list twice")
}
if err := entry("deny", "address", "billing@vendor.example"); err != nil {
t.Errorf("the same value on the other list: %v", err)
}
if err := entry("block", "address", "x@example.net"); err == nil {
t.Error("list type block was accepted")
}
if err := entry("deny", "subject", "pills"); err == nil {
t.Error("entry type subject was accepted")
}
var note string
if err := st.db.QueryRow("SELECT note FROM inbound_spam_list WHERE list_type = 'allow'").Scan(&note); err != nil || note != "" {
t.Errorf("note of an entry added without one = %q, %v", note, err)
}
}
// A 1.x data directory must be refused, loudly. Its user_version (10 at the
// last 1.x release) would otherwise read as "the baseline is already applied",
// and the panel would start on tables it does not know.
+152
View File
@@ -0,0 +1,152 @@
package store
import (
"fmt"
"time"
"unicode/utf8"
)
// SendLogLine is one mail.log line Postfix wrote under a queue id, kept so a
// message's page can show it after mail.log has rotated (architecture.md § Log
// tailer). At is the line's own time, not when the tailer read it.
type SendLogLine struct {
QueueID string
At time.Time
Line string
}
// SendLogLineMax is how many bytes of a line are kept. Postfix quotes the far
// end's reply in a delivery line, so a line is as long as someone else cared
// to make it; what a page needs of it is the beginning.
const SendLogLineMax = 4096
// sendLogLineBefore is how long before a row was written a line may be dated
// and still be the row's. The journal milter writes the row at end-of-message;
// the smtpd line that opens the queue id is written when the first recipient
// is accepted, and the message body can take its time after that.
const sendLogLineBefore = time.Hour
// cutSendLogLine cuts a line to SendLogLineMax bytes, and not through the
// middle of a character.
func cutSendLogLine(line string) string {
if len(line) <= SendLogLineMax {
return line
}
end := SendLogLineMax
for end > 0 && !utf8.RuneStart(line[end]) {
end--
}
return line[:end]
}
// InsertSendLogLines stores lines in one transaction and returns how many it
// wrote. The tailer calls it with what it read in one pass over the log,
// whether or not send-log rows exist for the queue ids yet: a commit per line
// would be a disk sync per line, on the one connection the journal milter
// writes through.
//
// With once set, a line is written only if the same line is not there already
// under its queue id. That is for the lines the tailer reads again after an
// unclean stop; a line carries its time and its process id, so two that are
// equal are one line read twice.
func (s *Store) InsertSendLogLines(lines []SendLogLine, once bool) (int, error) {
if len(lines) == 0 {
return 0, nil
}
query := `INSERT INTO send_log_lines (queue_id, logged_at, line) VALUES (?1, ?2, ?3)`
if once {
query = `INSERT INTO send_log_lines (queue_id, logged_at, line)
SELECT ?1, ?2, ?3
WHERE NOT EXISTS (SELECT 1 FROM send_log_lines WHERE queue_id = ?1 AND line = ?3)`
}
tx, err := s.db.Begin()
if err != nil {
return 0, fmt.Errorf("insert send_log_lines: %w", err)
}
defer tx.Rollback()
stmt, err := tx.Prepare(query)
if err != nil {
return 0, fmt.Errorf("insert send_log_lines: %w", err)
}
defer stmt.Close()
wrote := 0
for _, l := range lines {
res, err := stmt.Exec(l.QueueID, l.At.UTC().Format(time.RFC3339), cutSendLogLine(l.Line))
if err != nil {
return 0, fmt.Errorf("insert send_log_lines: %w", err)
}
n, _ := res.RowsAffected()
wrote += int(n)
}
if err := tx.Commit(); err != nil {
return 0, fmt.Errorf("insert send_log_lines: %w", err)
}
return wrote, nil
}
// SendLogLines returns the stored lines of the message a send-log row is
// about, oldest first and at most n of them - the latest n when there are
// more, since what a message did last is what its page is opened for.
//
// Postfix reuses a queue id once the message has left the queue, and a row
// lives far longer than that, so the id alone does not say whose a line is.
// A line is the row's when it is dated from shortly before the row was written
// to lifetime after it; lifetime is how long Postfix may keep working on a
// message, with a margin, and the caller's to state (postfix.RetryPolicy).
func (s *Store) SendLogLines(row SendLogRow, lifetime time.Duration, n int) ([]SendLogLine, error) {
if row.QueueID == "" || n <= 0 {
return nil, nil
}
rows, err := s.db.Query(
`SELECT logged_at, line FROM (
SELECT id, logged_at, line FROM send_log_lines
WHERE queue_id = ? AND logged_at >= ? AND logged_at <= ?
ORDER BY id DESC LIMIT ?)
ORDER BY id`,
row.QueueID,
row.CreatedAt.Add(-sendLogLineBefore).UTC().Format(time.RFC3339),
row.CreatedAt.Add(lifetime).UTC().Format(time.RFC3339),
n,
)
if err != nil {
return nil, fmt.Errorf("query send_log_lines: %w", err)
}
defer rows.Close()
var out []SendLogLine
for rows.Next() {
var at string
l := SendLogLine{QueueID: row.QueueID}
if err := rows.Scan(&at, &l.Line); err != nil {
return nil, fmt.Errorf("scan send_log_lines row: %w", err)
}
l.At, _ = time.Parse(time.RFC3339, at)
out = append(out, l)
}
return out, rows.Err()
}
// PruneSendLogLines deletes the lines dated before cutoff whose queue id has
// no send-log row, and returns how many went. That one rule is the whole of
// the table's retention: it removes the lines of rows the send-log sweep has
// deleted, and the lines of messages that never had a row - the bounces
// Postfix generates. cutoff leaves alone the lines of a message still being
// received, whose row the journal milter has yet to write.
//
// The send log is asked once per queue id, not once per line: the ids are read
// off the index, and a message is several lines.
func (s *Store) PruneSendLogLines(cutoff time.Time) (int64, error) {
res, err := s.db.Exec(
`DELETE FROM send_log_lines
WHERE logged_at < ?
AND queue_id IN (
SELECT q.queue_id FROM (SELECT DISTINCT queue_id FROM send_log_lines) AS q
WHERE NOT EXISTS (SELECT 1 FROM send_log WHERE send_log.queue_id = q.queue_id))`,
cutoff.UTC().Format(time.RFC3339),
)
if err != nil {
return 0, fmt.Errorf("prune send_log_lines: %w", err)
}
n, _ := res.RowsAffected()
return n, nil
}
+203
View File
@@ -0,0 +1,203 @@
package store
import (
"strings"
"testing"
"time"
"unicode/utf8"
)
func sendLogLineTexts(lines []SendLogLine) []string {
out := make([]string, len(lines))
for i, l := range lines {
out[i] = l.Line
}
return out
}
func countSendLogLines(t *testing.T, s *Store) int {
t.Helper()
var n int
if err := s.db.QueryRow(`SELECT COUNT(*) FROM send_log_lines`).Scan(&n); err != nil {
t.Fatalf("count send_log_lines: %v", err)
}
return n
}
// A row's page gets the lines of its queue id, oldest first, and not the lines
// of another message.
func TestSendLogLinesOfOneMessage(t *testing.T) {
s := openTestStore(t)
at := time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC)
if n, err := s.InsertSendLogLines([]SendLogLine{
{QueueID: "41E862C00D9E", At: at, Line: "client=app"},
{QueueID: "5F00AA", At: at, Line: "someone else's"},
{QueueID: "41E862C00D9E", At: at.Add(time.Second), Line: "status=sent"},
{QueueID: "41E862C00D9E", At: at.Add(2 * time.Second), Line: "removed"},
}, false); err != nil || n != 4 {
t.Fatalf("insert: wrote %d, %v", n, err)
}
row := SendLogRow{QueueID: "41E862C00D9E", CreatedAt: at}
lines, err := s.SendLogLines(row, 6*24*time.Hour, 200)
if err != nil {
t.Fatalf("SendLogLines: %v", err)
}
if got := strings.Join(sendLogLineTexts(lines), "|"); got != "client=app|status=sent|removed" {
t.Errorf("lines = %q", got)
}
if len(lines) > 0 && !lines[0].At.Equal(at) {
t.Errorf("first line dated %v, want %v", lines[0].At, at)
}
// More than the page shows: the latest are kept, still oldest first.
lines, err = s.SendLogLines(row, 6*24*time.Hour, 2)
if err != nil {
t.Fatalf("SendLogLines: %v", err)
}
if got := strings.Join(sendLogLineTexts(lines), "|"); got != "status=sent|removed" {
t.Errorf("latest two = %q", got)
}
// A row the milter refused has no queue id and so no lines - not every line.
lines, err = s.SendLogLines(SendLogRow{CreatedAt: at}, 6*24*time.Hour, 200)
if err != nil || len(lines) != 0 {
t.Errorf("lines of a row without a queue id = %v, %v", lines, err)
}
}
// Postfix reuses a queue id once the message has left the queue. The lines of
// a later message under the same id are not the earlier row's, nor the other
// way round.
func TestSendLogLinesBoundedByTheRowsLifetime(t *testing.T) {
s := openTestStore(t)
first := time.Date(2026, 9, 1, 9, 0, 0, 0, time.UTC)
second := first.AddDate(0, 0, 30)
lifetime := 6 * 24 * time.Hour
if _, err := s.InsertSendLogLines([]SendLogLine{
{QueueID: "ABC123", At: first.Add(-30 * time.Minute), Line: "first: client"},
{QueueID: "ABC123", At: first.Add(5 * 24 * time.Hour), Line: "first: expired"},
{QueueID: "ABC123", At: second.Add(-2 * time.Hour), Line: "between the two"},
{QueueID: "ABC123", At: second, Line: "second: sent"},
}, false); err != nil {
t.Fatalf("insert: %v", err)
}
lines, err := s.SendLogLines(SendLogRow{QueueID: "ABC123", CreatedAt: first}, lifetime, 200)
if err != nil {
t.Fatalf("SendLogLines: %v", err)
}
if got := strings.Join(sendLogLineTexts(lines), "|"); got != "first: client|first: expired" {
t.Errorf("the first message's lines = %q", got)
}
lines, err = s.SendLogLines(SendLogRow{QueueID: "ABC123", CreatedAt: second}, lifetime, 200)
if err != nil {
t.Fatalf("SendLogLines: %v", err)
}
if got := strings.Join(sendLogLineTexts(lines), "|"); got != "second: sent" {
t.Errorf("the second message's lines = %q", got)
}
}
// After an unclean stop the tailer reads some lines twice; the second reading
// writes nothing, also when the line comes twice in one batch. The same text
// under another queue id is another line. Without once, what is given is
// written.
func TestInsertSendLogLinesOnce(t *testing.T) {
s := openTestStore(t)
l := SendLogLine{QueueID: "ABC123", At: time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC), Line: "Oct 12 09:00:00 mail postfix/qmgr[7]: ABC123: removed"}
other := l
other.QueueID = "DEF456"
if n, err := s.InsertSendLogLines([]SendLogLine{l, l, other}, true); err != nil || n != 2 {
t.Errorf("first reading wrote %d, %v; want 2", n, err)
}
if n, err := s.InsertSendLogLines([]SendLogLine{l, other}, true); err != nil || n != 0 {
t.Errorf("second reading wrote %d, %v; want 0", n, err)
}
if n := countSendLogLines(t, s); n != 2 {
t.Errorf("%d lines stored, want 2", n)
}
if n, err := s.InsertSendLogLines([]SendLogLine{l}, false); err != nil || n != 1 {
t.Errorf("a line as it is written: wrote %d, %v; want 1", n, err)
}
if n, err := s.InsertSendLogLines(nil, true); err != nil || n != 0 {
t.Errorf("nothing to write: %d, %v", n, err)
}
}
// The far end writes the reply a delivery line quotes; what is kept of a line
// has a ceiling, not set through the middle of a character, and a line read
// twice is still recognised past it.
func TestSendLogLineIsCut(t *testing.T) {
s := openTestStore(t)
at := time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC)
// Two bytes a character, starting at an odd offset: the ceiling falls inside one.
l := SendLogLine{QueueID: "ABC123", At: at, Line: "x" + strings.Repeat("é", SendLogLineMax)}
if _, err := s.InsertSendLogLines([]SendLogLine{l}, false); err != nil {
t.Fatalf("insert: %v", err)
}
if n, err := s.InsertSendLogLines([]SendLogLine{l}, true); err != nil || n != 0 {
t.Errorf("the same long line again: wrote %d, %v", n, err)
}
lines, err := s.SendLogLines(SendLogRow{QueueID: "ABC123", CreatedAt: at}, time.Hour, 10)
if err != nil || len(lines) != 1 {
t.Fatalf("lines = %d, %v", len(lines), err)
}
if got := lines[0].Line; len(got) != SendLogLineMax-1 || !utf8.ValidString(got) {
t.Errorf("kept %d bytes (valid UTF-8: %v), want %d whole characters", len(got), utf8.ValidString(got), SendLogLineMax-1)
}
}
// The prune is the table's retention: lines go when no send-log row carries
// their queue id - the row was pruned, or there never was one - and a message
// still being received is left alone.
func TestPruneSendLogLines(t *testing.T) {
s := openTestStore(t)
now := time.Date(2026, 10, 12, 12, 0, 0, 0, time.UTC)
old := now.Add(-3 * time.Hour)
if err := s.InsertQueued(SendLogEntry{QueueID: "KEPT01", Domain: "example.com", To: "a@example.net"}); err != nil {
t.Fatalf("insert row: %v", err)
}
// A row the milter refused has no queue id; it keeps no line alive.
if err := s.InsertRejected(SendLogEntry{Domain: "example.com", From: "a@example.com"}); err != nil {
t.Fatalf("insert rejected row: %v", err)
}
if _, err := s.InsertSendLogLines([]SendLogLine{
{QueueID: "KEPT01", At: old, Line: "has a row"},
{QueueID: "INBOUND1", At: old, Line: "never had a row"},
{QueueID: "ARRIVING", At: now.Add(-time.Minute), Line: "its row is not written yet"},
}, false); err != nil {
t.Fatalf("insert: %v", err)
}
n, err := s.PruneSendLogLines(now.Add(-time.Hour))
if err != nil || n != 1 {
t.Fatalf("prune removed %d, %v; want 1", n, err)
}
var left []string
rows, err := s.db.Query(`SELECT queue_id FROM send_log_lines ORDER BY id`)
if err != nil {
t.Fatal(err)
}
for rows.Next() {
var id string
if err := rows.Scan(&id); err != nil {
t.Fatal(err)
}
left = append(left, id)
}
rows.Close()
if got := strings.Join(left, " "); got != "KEPT01 ARRIVING" {
t.Errorf("left after the prune: %q", got)
}
// The send-log sweep deletes the row; the next prune takes its lines.
if _, err := s.DeleteSendLogBefore(time.Now().UTC().Add(time.Hour)); err != nil {
t.Fatalf("delete rows: %v", err)
}
if n, err := s.PruneSendLogLines(now.Add(-time.Hour)); err != nil || n != 1 {
t.Errorf("prune after the row went removed %d, %v; want 1", n, err)
}
}
+66
View File
@@ -77,6 +77,72 @@ func (s *Store) EnsureSendLogRetentionDays(envDefault int) error {
return s.SetSendLogRetentionDays(sendLogRetentionFallback(envDefault))
}
const (
// InboundSpamLogRetentionDaysKey is the settings table key for the
// inbound antispam journal's retention. Same range as the send log's.
InboundSpamLogRetentionDaysKey = "inbound_spam_log_retention_days"
InboundSpamLogRetentionDaysMin = 7
InboundSpamLogRetentionDaysMax = 365
// InboundSpamLogRetentionDaysDefault matches
// INBOUND_SPAM_LOG_RETENTION_DAYS when unset.
InboundSpamLogRetentionDaysDefault = 90
)
// ErrInboundSpamLogRetentionDaysOutOfRange is returned when the inbound
// journal's retention is outside the allowed panel range.
var ErrInboundSpamLogRetentionDaysOutOfRange = errors.New("inbound journal retention days out of range")
// ValidateInboundSpamLogRetentionDays checks the panel-allowed window.
func ValidateInboundSpamLogRetentionDays(days int) error {
if days < InboundSpamLogRetentionDaysMin || days > InboundSpamLogRetentionDaysMax {
return fmt.Errorf("%w: must be between %d and %d days", ErrInboundSpamLogRetentionDaysOutOfRange, InboundSpamLogRetentionDaysMin, InboundSpamLogRetentionDaysMax)
}
return nil
}
func inboundSpamLogRetentionFallback(envDefault int) int {
if ValidateInboundSpamLogRetentionDays(envDefault) == nil {
return envDefault
}
return InboundSpamLogRetentionDaysDefault
}
// GetInboundSpamLogRetentionDays returns the effective retention window of
// the inbound journal. When the setting is missing or invalid, envDefault is
// used (INBOUND_SPAM_LOG_RETENTION_DAYS).
func (s *Store) GetInboundSpamLogRetentionDays(envDefault int) (int, error) {
raw, err := s.GetSetting(InboundSpamLogRetentionDaysKey)
if err != nil {
return 0, err
}
days, err := strconv.Atoi(raw)
if err != nil || ValidateInboundSpamLogRetentionDays(days) != nil {
return inboundSpamLogRetentionFallback(envDefault), nil
}
return days, nil
}
// SetInboundSpamLogRetentionDays persists the inbound journal's retention.
func (s *Store) SetInboundSpamLogRetentionDays(days int) error {
if err := ValidateInboundSpamLogRetentionDays(days); err != nil {
return err
}
return s.SetSetting(InboundSpamLogRetentionDaysKey, strconv.Itoa(days))
}
// EnsureInboundSpamLogRetentionDays seeds the setting from envDefault when it
// has never been written.
func (s *Store) EnsureInboundSpamLogRetentionDays(envDefault int) error {
raw, err := s.GetSetting(InboundSpamLogRetentionDaysKey)
if err != nil {
return err
}
if raw != "" {
return nil
}
return s.SetInboundSpamLogRetentionDays(inboundSpamLogRetentionFallback(envDefault))
}
// GetSetting returns a settings value or empty string when missing.
func (s *Store) GetSetting(key string) (string, error) {
var value string
+8 -5
View File
@@ -43,12 +43,15 @@ type User struct {
// are then ignored.
AllDomains bool
AllInboundDomains bool
CreatedAt time.Time
DomainIDs []int64
InboundDomainIDs []int64
// Notify is the user's own switch for notification e-mails (Account). It
// is on for a new user and does nothing without an e-mail.
Notify bool
CreatedAt time.Time
DomainIDs []int64
InboundDomainIDs []int64
}
const userColumns = "id, username, password_hash, role, email, all_domains, all_inbound_domains, created_at"
const userColumns = "id, username, password_hash, role, email, all_domains, all_inbound_domains, notify, created_at"
func scanUser(r scanRow) (User, error) {
var (
@@ -56,7 +59,7 @@ func scanUser(r scanRow) (User, error) {
createdAt string
)
err := r.Scan(&u.ID, &u.Username, &u.PasswordHash, &u.Role, &u.Email,
&u.AllDomains, &u.AllInboundDomains, &createdAt)
&u.AllDomains, &u.AllInboundDomains, &u.Notify, &createdAt)
if err != nil {
return User{}, err
}
+3
View File
@@ -56,6 +56,9 @@ var routes = struct{ always, inbound, dmarc []string }{
"POST /inbound/domains/{id}/upstream",
"POST /inbound/domains/{id}/recipients",
"GET /inbound/domains/{id}/delete", "POST /inbound/domains/{id}/delete",
"GET /inbound/log",
"GET /inbound/filter-lists", "POST /inbound/filter-lists",
"POST /inbound/filter-lists/{id}/delete",
},
dmarc: []string{
"GET /outbound/dmarc", "GET /outbound/dmarc/domains/{id}", "GET /outbound/dmarc/reports/{id}",
+134
View File
@@ -0,0 +1,134 @@
package handlers
import (
"encoding/binary"
"net"
"strings"
"sync"
"testing"
)
// fakeZone is a DNS server on the loopback that answers from a table, so that a
// page can be rendered against records that are published, wrong or absent
// without leaving the machine. Names are lower case without the trailing dot;
// a name the zone does not hold is NXDOMAIN, which is how a record that is not
// published looks to the checker.
type fakeZone struct {
mu sync.Mutex
txt map[string][]string
addr map[string]string // name -> IPv4 address
}
// newFakeZone starts the server for the length of the test and returns the
// resolver address to give dnscheck.New.
func newFakeZone(t *testing.T) (*fakeZone, string) {
t.Helper()
conn, err := net.ListenPacket("udp", "127.0.0.1:0")
if err != nil {
t.Fatalf("fake DNS: listen: %v", err)
}
z := &fakeZone{txt: map[string][]string{}, addr: map[string]string{}}
t.Cleanup(func() { _ = conn.Close() })
go func() {
buf := make([]byte, 1500)
for {
n, from, err := conn.ReadFrom(buf)
if err != nil {
return
}
if resp := z.answer(buf[:n]); resp != nil {
_, _ = conn.WriteTo(resp, from)
}
}
}()
return z, conn.LocalAddr().String()
}
// publishTXT sets the TXT record of a name; publishA its IPv4 address.
func (z *fakeZone) publishTXT(name, value string) {
z.mu.Lock()
defer z.mu.Unlock()
z.txt[strings.ToLower(name)] = append(z.txt[strings.ToLower(name)], value)
}
func (z *fakeZone) publishA(name, ip string) {
z.mu.Lock()
defer z.mu.Unlock()
z.addr[strings.ToLower(name)] = ip
}
// answer builds the reply to one query: the question is copied back, then the
// records of the asked type if the zone has any. A name with nothing of the
// asked type but known under another type is an empty answer (NOERROR), an
// unknown name is NXDOMAIN.
func (z *fakeZone) answer(q []byte) []byte {
if len(q) < 12 || binary.BigEndian.Uint16(q[4:6]) != 1 {
return nil
}
i, labels := 12, []string{}
for {
if i >= len(q) {
return nil
}
l := int(q[i])
if l == 0 {
i++
break
}
if i+1+l > len(q) {
return nil
}
labels = append(labels, string(q[i+1:i+1+l]))
i += 1 + l
}
if i+4 > len(q) {
return nil
}
name, qtype := strings.ToLower(strings.Join(labels, ".")), binary.BigEndian.Uint16(q[i:i+2])
question := q[12 : i+4]
z.mu.Lock()
txt, ip := z.txt[name], z.addr[name]
z.mu.Unlock()
var answers [][]byte
switch {
case qtype == 16:
for _, v := range txt {
var rdata []byte
for len(v) > 255 {
rdata = append(rdata, 255)
rdata = append(rdata, v[:255]...)
v = v[255:]
}
rdata = append(rdata, byte(len(v)))
rdata = append(rdata, v...)
answers = append(answers, record(16, rdata))
}
case qtype == 1 && ip != "":
answers = append(answers, record(1, net.ParseIP(ip).To4()))
}
rcode := uint16(0)
if len(txt) == 0 && ip == "" {
rcode = 3
}
resp := make([]byte, 12, 512)
copy(resp, q[:2])
binary.BigEndian.PutUint16(resp[2:], 0x8180|rcode)
binary.BigEndian.PutUint16(resp[4:], 1)
binary.BigEndian.PutUint16(resp[6:], uint16(len(answers)))
resp = append(resp, question...)
for _, a := range answers {
resp = append(resp, a...)
}
return resp
}
// record is one answer record named by the question (a pointer to offset 12),
// class IN, with a TTL of a minute.
func record(rtype uint16, rdata []byte) []byte {
r := []byte{0xC0, 0x0C, 0, 0, 0, 1, 0, 0, 0, 60, 0, 0}
binary.BigEndian.PutUint16(r[2:], rtype)
binary.BigEndian.PutUint16(r[10:], uint16(len(rdata)))
return append(r, rdata...)
}
+6
View File
@@ -50,6 +50,9 @@ type Config struct {
// SendLogRetentionEnvDefault is SEND_LOG_RETENTION_DAYS at panel start; used
// as bootstrap and fallback when the settings row is missing or invalid.
SendLogRetentionEnvDefault int
// InboundSpamLogRetentionEnvDefault is INBOUND_SPAM_LOG_RETENTION_DAYS at
// panel start, the same bootstrap and fallback for the inbound journal.
InboundSpamLogRetentionEnvDefault int
}
// Handlers holds dependencies for authenticated panel routes.
@@ -58,6 +61,7 @@ type Handlers struct {
domains *domain.Service
apps *app.Service
inbound *inbound.Service
lists *inbound.Lists
dmarc *dmarc.Service
cfg Config
view *view.Engine
@@ -72,6 +76,7 @@ func New(
domains *domain.Service,
apps *app.Service,
inboundSvc *inbound.Service,
lists *inbound.Lists,
dmarcSvc *dmarc.Service,
cfg Config,
v *view.Engine,
@@ -84,6 +89,7 @@ func New(
domains: domains,
apps: apps,
inbound: inboundSvc,
lists: lists,
dmarc: dmarcSvc,
cfg: cfg,
view: v,
+13 -3
View File
@@ -26,6 +26,7 @@ type accountForm struct {
Err string
Username string
Email string
Notify bool
}
func (h *Handlers) accountUser(w http.ResponseWriter, r *http.Request) (store.User, bool) {
@@ -44,7 +45,7 @@ func (h *Handlers) accountUser(w http.ResponseWriter, r *http.Request) (store.Us
}
func formOf(u store.User) accountForm {
return accountForm{Username: u.Username, Email: u.Email}
return accountForm{Username: u.Username, Email: u.Email, Notify: u.Notify}
}
// HandleAccount shows the Account page.
@@ -58,7 +59,7 @@ func (h *Handlers) HandleAccount(w http.ResponseWriter, r *http.Request) {
func (h *Handlers) renderAccount(w http.ResponseWriter, r *http.Request, status int, u store.User, f accountForm) {
page := view.NewAccount(h.shellMeta(r), string(u.Role))
page.Username, page.Email = f.Username, f.Email
page.Username, page.Email, page.Notify = f.Username, f.Email, f.Notify
page.WithResult(accountFlash(r), f.Err)
h.view.Render(w, status, "account", page)
}
@@ -90,6 +91,8 @@ func (h *Handlers) HandleAccountProfile(w http.ResponseWriter, r *http.Request)
}
f.Username = strings.TrimSpace(r.PostFormValue("username"))
f.Email = strings.TrimSpace(r.PostFormValue("email"))
// An unticked checkbox is not sent at all: absent is off.
f.Notify = r.PostFormValue("notify") != ""
if f.Username == "" {
f.Username = u.Username
}
@@ -112,7 +115,14 @@ func (h *Handlers) HandleAccountProfile(w http.ResponseWriter, r *http.Request)
fail(http.StatusInternalServerError, "Could not save the profile. Please check the logs and try again.")
return
}
logf("panel: user %d profile updated (username: %t, e-mail: %t)", u.ID, f.Username != u.Username, f.Email != u.Email)
if f.Notify != u.Notify {
if err := h.store.SetUserNotify(u.ID, f.Notify); err != nil {
logf("panel: account: set notifications of user %d: %v", u.ID, err)
fail(http.StatusInternalServerError, "Could not save the profile. Please check the logs and try again.")
return
}
}
logf("panel: user %d profile updated (username: %t, e-mail: %t, notifications: %t)", u.ID, f.Username != u.Username, f.Email != u.Email, f.Notify)
http.Redirect(w, r, "/account?done=profile", http.StatusSeeOther)
}

Some files were not shown because too many files have changed in this diff Show More