Compare commits
55
Commits
v2.0.0
...
b905f3e560
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b905f3e560 | ||
|
|
c9027381e5 | ||
|
|
50fd6b859f | ||
|
|
06ad174fb9 | ||
|
|
87fa0a4381 | ||
|
|
eff8010d39 | ||
|
|
394eb67c4c | ||
|
|
972a9f825c | ||
|
|
a990e7cc23 | ||
|
|
815236a607 | ||
|
|
a6845ae948 | ||
|
|
1e6836b3c1 | ||
|
|
fc184ecc01 | ||
|
|
81df131217 | ||
|
|
3d17206063 | ||
|
|
4468aa8dd7 | ||
|
|
977d63a2ea | ||
|
|
ae06d8b47e | ||
|
|
1f44e40382 | ||
|
|
d398f488e2 | ||
|
|
89774ab847 | ||
|
|
c731a98a0f | ||
|
|
5f86457848 | ||
|
|
00feb7c1c7 | ||
|
|
d2a778edc0 | ||
|
|
7192c3861e | ||
|
|
e0ccff257d | ||
|
|
c1da784193 | ||
|
|
9d330bed77 | ||
|
|
970c8ca8da | ||
|
|
19156814b8 | ||
|
|
d2b97b3963 | ||
|
|
d4efdb483d | ||
|
|
56c5dfee40 | ||
|
|
82d4a209be | ||
|
|
737611a07e | ||
|
|
eea40bcd49 | ||
|
|
27fd355e53 | ||
|
|
94e94490af | ||
|
|
28ec784ae2 | ||
|
|
a3cc52d6d4 | ||
|
|
d65d632505 | ||
|
|
dd0fefdcb7 | ||
|
|
75ebbf6a57 | ||
|
|
ede3783734 | ||
|
|
d70371a50c | ||
|
|
21e961cb29 | ||
|
|
82f212ab39 | ||
|
|
b55b33ae92 | ||
|
|
8c9a71e6a0 | ||
|
|
646ec1e149 | ||
|
|
8496b289ec | ||
|
|
4dd46ea355 | ||
|
|
505260e227 | ||
|
|
09d9625f32 |
+242
@@ -5,6 +5,248 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); version
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
- **Schema for notifications** (migration `0004_notifications.sql`, DDL only):
|
||||
a user's own switch, `users.notify`, on for everyone; `notify_state`, what
|
||||
the panel has already written to its users about. The instance switch is a
|
||||
settings key, on unless turned off
|
||||
([plans/panel-notifications.md](docs/plans/panel-notifications.md)).
|
||||
- **The panel writes to its users when something needs them.** Four things,
|
||||
each said once: a sending domain's DKIM, SPF or DMARC record stops matching
|
||||
(after two hourly checks in a row, and once more when it matches again); a
|
||||
level-2 rate limit refuses mail (at most once a day per limit); the TLS
|
||||
certificate is 14 days from expiry, and 3; a message has waited in the
|
||||
queue for more than three days (and once more when none has). Global users
|
||||
are told all of it, domain users about their own domains. Sent from
|
||||
`postmaster@<SELFPOST_HOSTNAME>` by local submission, one message per
|
||||
recipient; not rows of the Outbound log and counted against no limit.
|
||||
- **The server's own name is signed.** The panel generates a DKIM key for
|
||||
`SELFPOST_HOSTNAME` and OpenDKIM signs mail submitted locally as that name
|
||||
(`non_smtpd_milters`), so that notifications arrive. The name is not a
|
||||
sending domain: no application can send as it.
|
||||
- **Notifications in the panel.** Account has *Send me notifications* beside
|
||||
the e-mail; Server › Settings has the switch for everyone and the two DNS
|
||||
records of the server's own name, DKIM and SPF, with the values to publish;
|
||||
Server › Health has a *Notification mail* card with the state of each. Help
|
||||
has a Notifications topic. Every user starts with notifications on, and so
|
||||
does the instance.
|
||||
|
||||
## [2.2.0] - 2026-10-12
|
||||
|
||||
### Added
|
||||
|
||||
- **Schema: `send_log_lines`** (migration `0003_send_log_lines.sql`, DDL only).
|
||||
The table that will hold what Postfix wrote about a message for as long as
|
||||
its send-log row lives, with the store methods to write a line, to write it
|
||||
once, to read the lines of a row and to prune
|
||||
([plans/delivery-log-storage.md](docs/plans/delivery-log-storage.md)).
|
||||
- **The log tailer keeps what Postfix writes about queued messages.** A
|
||||
`mail.log` line headed by a queue id is stored with the line's own time, one
|
||||
transaction per pass over the log; after an unclean stop a line read twice
|
||||
is stored once. Mail that came in on port 25 is left out: a stranger's
|
||||
message never has a send-log row, and what he can make Postfix log is not
|
||||
written to the database. Lines whose queue id has no send-log row - the row
|
||||
was pruned, or the message never had one (bounces) - are deleted once they
|
||||
are an hour old, at start and with every six-hourly send-log sweep.
|
||||
|
||||
### Changed
|
||||
|
||||
- **A message's page keeps its delivery log after `mail.log` rotates.** The
|
||||
*Delivery log* box on Outbound › Log › a message was filled by searching
|
||||
the current `mail.log`, which is rotated daily, so it was empty for anything
|
||||
not sent today while the row itself lives for ninety days. The page now
|
||||
reads the lines the tailer stored: they are there for as long as the row
|
||||
is. Postfix reuses queue ids, so a row is given only the lines dated within
|
||||
its own queue lifetime. A message accepted before this version shows its
|
||||
lines only while they are still in the current file, as before; the note
|
||||
shown when there are none and the *Outbound log* help topic say so.
|
||||
|
||||
### Security
|
||||
|
||||
- **A full backup now carries the log lines of messages in the send log** -
|
||||
client addresses, SASL logins, message ids, receivers' replies - where
|
||||
`mail.log` itself is not in a backup. Recorded in
|
||||
[security.md](docs/security.md) with the accepted risk of a queue-id
|
||||
collision showing one message's lines on another's page. Technical and
|
||||
security reviews of 2026-10-12: no Critical or High finding.
|
||||
|
||||
## [2.1.0] - 2026-10-12
|
||||
|
||||
### Security
|
||||
|
||||
- **The level-2 rate limit and an application's client IP list held only for
|
||||
the first message of a connection.** After a refused `MAIL FROM`, a second
|
||||
one on the same connection was accepted, and so was every message after the
|
||||
first that a connection carried: an application over its limit could go on
|
||||
sending, and one restricted to certain client addresses could send from any
|
||||
other with a retry. In the code since `1.9.0` (the client IP list) and
|
||||
earlier (the limit); found by the technical review of 2026-10-12 and
|
||||
reproduced on the test stand. Postfix now gives the panel's milters the
|
||||
client address with every message, the limit no longer depends on it, and an
|
||||
application with a client IP list is refused when no address is known.
|
||||
- Reviews of the inbound antispam feature before its release (technical and
|
||||
security, 2026-10-12; nothing Critical or High in the feature itself). Fixed:
|
||||
a client's HELO name could supply the sender, recipient and reply of an
|
||||
inbound log row; the log's row cap was applied only every six hours; rows of
|
||||
a connection's later messages had no client address; two changes of the
|
||||
filter lists at once could leave the filter's files behind the page; a held
|
||||
message was logged twice; after an unclean stop of the panel the refusals of
|
||||
its last seconds were logged twice; with the spam filter down, a sender's
|
||||
own verdict headers were recorded as the filter's. Recorded as accepted in
|
||||
[docs/security.md](docs/security.md): an allow entry for a sender address
|
||||
trusts an address anyone can claim; the list files are readable by anyone on
|
||||
the host.
|
||||
- `INBOUND_ANTISPAM_MILTER` and `INBOUND_JOURNAL_MILTER_SOCKET` are checked
|
||||
character by character at start; the check looked at their first characters
|
||||
only.
|
||||
|
||||
### Changed
|
||||
|
||||
- **Port 25 logs as `postfix/inbound`.** Its lines in the mail log (Server ›
|
||||
System log) now read `postfix/inbound/smtpd[…]` where they read
|
||||
`postfix/smtpd[…]`; the submission ports keep `postfix/smtpd`. This is how a
|
||||
refusal on port 25 is told from one on 465 / 587.
|
||||
|
||||
### Added
|
||||
|
||||
- Schema: migration `0002_inbound_spam_log.sql`, the first after the 2.0
|
||||
baseline - two empty tables for the inbound antispam journal and the filter
|
||||
lists (inbound-antispam-panel). A 2.0.x database is migrated when the panel
|
||||
starts.
|
||||
- **Inbound mail is recorded.** With the inbound relay on, the panel runs a
|
||||
second milter, the last one on port 25, and writes a row for every recipient
|
||||
of a message that was accepted for an inbound domain: client address,
|
||||
envelope, subject and - with a spam filter attached - the filter's action,
|
||||
score and heaviest symbols. It only watches: it refuses nothing and Postfix
|
||||
accepts mail when it is down.
|
||||
- **Refused inbound mail is recorded too**, from the mail log: a recipient
|
||||
Postfix refused at an inbound domain (unknown user, sender domain not
|
||||
found), and a message the spam filter rejected, deferred, held or
|
||||
discarded. The row is dated by its log line and carries the reply text. A
|
||||
refusal to relay for a domain that is not an inbound one is not recorded.
|
||||
- The inbound journal is pruned every six hours: rows older than 90 days
|
||||
(`INBOUND_SPAM_LOG_RETENTION_DAYS`, 7-365) and anything beyond 10 000 rows.
|
||||
- The antispam compose fragment mounts `antispam/rspamd/milter_headers.conf`
|
||||
into the rspamd sidecar, which makes it write its verdict into
|
||||
`X-Rspamd-Action` and `X-Spamd-Result`. **Copy `deploy/antispam/rspamd/`
|
||||
next to the fragment before starting it** - Docker creates a directory in
|
||||
place of a missing file and the sidecar then fails to read its
|
||||
configuration.
|
||||
- **Allow and deny lists for the spam filter**, kept by SelfPost and applied
|
||||
by rspamd: sender addresses, sender domains, client addresses and networks.
|
||||
They are written as six files under `data/antispam/`, which the fragment
|
||||
mounts into the sidecar together with `antispam/rspamd/multimap.conf`;
|
||||
rspamd rereads a changed file by itself. Deny wins over allow. The files
|
||||
are there from the first start with the inbound relay on.
|
||||
- **Inbound › Log**: the journal as a page - time, sender and client address,
|
||||
recipient, subject and decision, filtered by domain, decision and day. The
|
||||
decision opens a sheet over the page with who decided and why: the filter's
|
||||
verdict and symbols, or the reply a refusal was given. A user with the
|
||||
*domain* role sees the rows of the inbound domains assigned to them. For the
|
||||
global role a row links to the filter lists with the sender filled in:
|
||||
*Deny sender* on accepted mail, *Allow sender* on mail the filter refused -
|
||||
unless the sender is spelt in a way the lists do not take.
|
||||
- **Inbound › Filter lists** (global role): add and remove allow and deny
|
||||
entries; the page says where the lists go and whether the filter is on to
|
||||
read them. Each entry shows how often it decided a message - *Matched*, or
|
||||
*never* - and its kind opens a sheet with the note, the day it was added and
|
||||
the last match, so that entries nothing matches any more can be found and
|
||||
removed. The count is read from the inbound log and reaches back as far as
|
||||
the log does. A network of one address (`192.0.2.7/32`) is stored as the
|
||||
address, so it cannot be on a list twice.
|
||||
- **Server › Settings**: *Keep inbound log rows, days*, shown with the inbound
|
||||
relay on.
|
||||
- Help: *Inbound log* and *Filter lists*.
|
||||
- New setting `ANTISPAM_DIR` (default `/data/antispam`).
|
||||
- **The spam filter signs what it judged.** rspamd adds `X-SelfPost-Filter` to
|
||||
every message it lets through - an HMAC of the message's queue id under a
|
||||
key SelfPost creates in `data/antispam/filter.key` - and the inbound log
|
||||
shows a verdict only next to a signature that fits. Nothing to configure,
|
||||
but the sidecar needs the new `antispam/rspamd/milter_headers.conf` and a
|
||||
restart to read it; until then accepted mail is logged as *selfpost*, with
|
||||
no verdict.
|
||||
- A full backup carries the inbound log, the filter lists and their files
|
||||
under `data/antispam/`. It does not carry the `antispam/` directory with the
|
||||
compose fragment and the rspamd configuration - copy that from
|
||||
`deploy/antispam/` when restoring.
|
||||
- New setting `INBOUND_JOURNAL_MILTER_SOCKET` (default
|
||||
`/run/selfpost/inbound-journal.sock`).
|
||||
|
||||
## [2.0.1] - 2026-10-11
|
||||
|
||||
### Changed
|
||||
|
||||
- **A message that is not a report this relay keeps is dropped at once.**
|
||||
One that cannot be read as an aggregate report, reports on a domain not
|
||||
sent from here, or came to another domain's report address used to be put
|
||||
off and offered again by Postfix for days, counted as a parse failure each
|
||||
time. It is now accepted and discarded - no retry, and no bounce, since the
|
||||
envelope sender of such a message is whatever its author wrote. It counts
|
||||
as one parse failure and the reason is in the mail log, on the message's
|
||||
own delivery line (Server › System log, `dropped: …`). Only a failure of
|
||||
the database is still retried.
|
||||
- **A full backup no longer holds the Postfix queue**, from the panel or from
|
||||
`selfpost-backup`. It is taken while the instance runs, and a copy of a
|
||||
queue Postfix is working on can hold a message twice or not at all. Mail
|
||||
still waiting for delivery is not carried over; a restored instance starts
|
||||
with an empty queue. To keep the queue, stop the instance and `tar` the
|
||||
project directory.
|
||||
- DMARC reports are kept 500 **for each domain** (and 90 days), not 500 for
|
||||
the whole server - see Security.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Full backup from the panel works.** Since the Postfix queue moved under
|
||||
`/data` in 1.3.0 the button answered with a 200 and ten bytes: the archive
|
||||
includes the queue, Postfix keeps the queue readable by its own user alone,
|
||||
and the panel is not that user ("open /data/postfix/queue/active:
|
||||
permission denied" in the log). `selfpost-backup` run with `docker exec`
|
||||
was not affected - it runs as root. The queue is no longer part of a backup
|
||||
(see Changed), so the two archives are the same again. Everything is read
|
||||
through before the download starts, so a failure is a message on the page
|
||||
and not a file cut short. One thing the panel may still be unable to read
|
||||
is the private key in `certs/`: it then refuses and says what to do -
|
||||
`chgrp 998 certs/privkey.pem && chmod 640 certs/privkey.pem` on the host,
|
||||
998 being the container's `selfpost` group, whose id is now fixed in the
|
||||
image.
|
||||
|
||||
### Security
|
||||
|
||||
Review of the DMARC report path (Fable, 2026-10-11), which started to run for
|
||||
the first time in 2.0.0. Nothing Critical or High.
|
||||
|
||||
- **A report is bounded before it is stored.** The XML comes from anyone who
|
||||
knows a report address, and its 8 MiB limit left room for some 160 000
|
||||
records or a single name of megabytes - rows in the database and cells of
|
||||
the report page. A report holds at most 5 000 records (a larger one is
|
||||
dropped), every text field is cut to 255 bytes, and a record's count is
|
||||
capped so the sums cannot wrap.
|
||||
- **One domain's reports cannot push out another's.** The cap of 500 was for
|
||||
the whole server, so a stranger filing reports under one domain's address
|
||||
could evict every other domain's history. It is counted per domain now.
|
||||
- The delivery command runs one at a time, not up to a hundred in parallel.
|
||||
- The reason written to the mail log for a dropped message is cut to 200
|
||||
bytes and anything outside printable ASCII is replaced: parts of it are the
|
||||
sender's own bytes (a header line, a file name inside an archive).
|
||||
- `SELFPOST_DATA_DIR` / `SELFPOST_DB_PATH` with whitespace or `$` are refused
|
||||
at start when DMARC ingest is on, instead of handing the delivery command
|
||||
another path and so an empty database of its own.
|
||||
|
||||
### Added
|
||||
|
||||
- e2e: the run now ends on port 25 with both optional features on. A DMARC
|
||||
report is mailed to a domain's hosted address and has to appear under
|
||||
Outbound › DMARC reports; a report on another domain and a message with no
|
||||
report are accepted and dropped; mail for an inbound domain reaches its
|
||||
upstream and other recipients are refused. Until now the suite only read
|
||||
back the generated configuration of these features.
|
||||
- e2e: full backup and restore. The run takes a backup from the panel, throws
|
||||
the installation away, unpacks the archive into the empty project directory
|
||||
and starts it: the administrator signs in, the domain has the same DKIM key,
|
||||
and mail sent by the application verifies against the record published
|
||||
before the backup.
|
||||
|
||||
## [2.0.0] - 2026-10-11
|
||||
|
||||
The panel redesign. A breaking release: every panel URL changed and the
|
||||
|
||||
@@ -112,7 +112,7 @@ docker run --rm -d --name selfpost-try \
|
||||
-e SELFPOST_HOSTNAME=mail.local.test \
|
||||
-e PANEL_COOKIE_SECURE=false \
|
||||
-v selfpost-try-data:/data \
|
||||
ghcr.io/mixeme/selfpost:2.0.0
|
||||
ghcr.io/mixeme/selfpost:2.2.0
|
||||
```
|
||||
|
||||
**Get the setup URL** (pick one):
|
||||
|
||||
+7
-2
@@ -62,7 +62,10 @@ RUN useradd --system --no-create-home --shell /usr/sbin/nologin panel
|
||||
# entrypoint.sh) — lets OpenDKIM read the panel-owned keys and lets Postfix read
|
||||
# the sasldb2/sender map, and lets the panel reach the supervisor control socket
|
||||
# to signal OpenDKIM/Postfix reloads without any process running as root.
|
||||
RUN groupadd --system selfpost \
|
||||
# The id is fixed, and it is the one the group has always had: an operator
|
||||
# gives the TLS private key on the host to this group by number, so that the
|
||||
# panel can put it into a full backup (guide § Taking a full backup).
|
||||
RUN groupadd --system --gid 998 selfpost \
|
||||
&& usermod -aG selfpost panel \
|
||||
&& usermod -aG selfpost opendkim \
|
||||
&& usermod -aG selfpost postfix
|
||||
@@ -73,7 +76,9 @@ RUN groupadd --system selfpost \
|
||||
# with `no-new-privileges`, which disables setgid transitions, so the panel
|
||||
# would always see "Permission denied". Making `panel` a real member of postdrop
|
||||
# grants the same access without relying on a setgid escalation the hardening
|
||||
# deliberately forbids. postdrop membership is read-side only: it does not let
|
||||
# deliberately forbids. The same membership is what lets the panel submit its
|
||||
# own notifications with sendmail(1) - postdrop, too, is setgid and would fail
|
||||
# to write the maildrop directory (internal/notify/mailer.go). It does not let
|
||||
# the panel bypass any Postfix restriction that a local user does not already
|
||||
# have through the world-executable sendmail.
|
||||
RUN usermod -aG postdrop panel
|
||||
|
||||
@@ -112,6 +112,15 @@ for f in relay_domains transport relay_recipients tls_policy; do
|
||||
done
|
||||
chown panel:selfpost /data/postfix
|
||||
chmod 2750 /data/postfix
|
||||
|
||||
# Filter lists for the external spam filter (written by the panel when
|
||||
# INBOUND_RELAY_ENABLE=true, read by the antispam sidecar, which mounts this
|
||||
# directory). The sidecar runs under a user of its own in another container,
|
||||
# so the directory and its files are readable by anyone: they hold sender
|
||||
# addresses and networks, no secrets.
|
||||
mkdir -p /data/antispam
|
||||
chown panel:selfpost /data/antispam
|
||||
chmod 0755 /data/antispam
|
||||
chown panel:selfpost /data/postfix/sender_login_maps
|
||||
chmod 0640 /data/postfix/sender_login_maps
|
||||
|
||||
|
||||
+64
-3
@@ -117,6 +117,9 @@ SUBMISSION_ENABLE="${SUBMISSION_ENABLE:-false}"
|
||||
INBOUND_RELAY_ENABLE="${INBOUND_RELAY_ENABLE:-false}"
|
||||
INBOUND_ANTISPAM_MILTER="${INBOUND_ANTISPAM_MILTER:-}"
|
||||
INBOUND_ANTISPAM_MILTER_ACTION="${INBOUND_ANTISPAM_MILTER_ACTION:-accept}"
|
||||
# The panel's inbound journal milter (monitoring, fail-open), on port 25 only
|
||||
# and only while the inbound relay is on. Matched by postfix-wrapper.sh.
|
||||
INBOUND_JOURNAL_SOCK="${INBOUND_JOURNAL_MILTER_SOCKET:-/run/selfpost/inbound-journal.sock}"
|
||||
INBOUND_RATE_LIMIT_MESSAGES_PER_IP="${INBOUND_RATE_LIMIT_MESSAGES_PER_IP:-20}"
|
||||
INBOUND_MESSAGE_SIZE_LIMIT="${INBOUND_MESSAGE_SIZE_LIMIT:-26214400}"
|
||||
RELAY_DOMAINS_MAP="${POSTFIX_RELAY_DOMAINS:-/data/postfix/relay_domains}"
|
||||
@@ -221,11 +224,25 @@ postconf -e \
|
||||
# if it is unreachable, defer rather than send unsigned). The journal-milter is
|
||||
# monitoring only and is fail-open (default_action=accept): its failure must
|
||||
# never block the relay. Per-milter settings use Postfix 3.0+ brace syntax.
|
||||
#
|
||||
# non_smtpd_milters is the chain for mail submitted locally with sendmail(1) -
|
||||
# the panel's own notifications, sent as postmaster@ the hostname. OpenDKIM
|
||||
# signs them with the key of the server's own name; the journal-milter is not
|
||||
# on this chain, since they are not an application's mail. Bounces Postfix
|
||||
# generates itself do not pass it (internal_mail_filter_classes is empty).
|
||||
postconf -e \
|
||||
"milter_protocol=6" \
|
||||
"milter_default_action=tempfail" \
|
||||
"smtpd_milters={ unix:${OPENDKIM_SOCK}, default_action=tempfail }, { unix:${JOURNAL_SOCK}, default_action=accept }" \
|
||||
"non_smtpd_milters="
|
||||
"non_smtpd_milters={ unix:${OPENDKIM_SOCK}, default_action=tempfail }"
|
||||
|
||||
# What Postfix tells a milter with MAIL FROM: its defaults plus the client's
|
||||
# address. The panel's milters judge every message by that address (an
|
||||
# application's client-address list) and record it (the inbound journal), and
|
||||
# the address given once at connect is not there for a connection's later
|
||||
# messages - see MailFrom in internal/milter/milter.go.
|
||||
postconf -e \
|
||||
"milter_mail_macros=i {auth_type} {auth_authen} {auth_author} {mail_addr} {mail_host} {mail_mailer} {client_addr}"
|
||||
|
||||
# Bounded milter timeouts: a *hung* milter (socket accepts but never replies)
|
||||
# must fail open just like a crash, not stall mail acceptance until the Postfix
|
||||
@@ -291,11 +308,23 @@ if [ "${INBOUND_RELAY_ENABLE}" = "true" ] || [ "${DMARC_REPORTS_ENABLE}" = "true
|
||||
# attribute" on every report. They go into the service line itself.
|
||||
# ${recipient} is expanded by pipe(8), not by the shell: the command
|
||||
# learns which report address the message came to from its argument.
|
||||
postconf -M 'dmarc-ingest/unix=dmarc-ingest unix - n n - - pipe flags=Rq user=panel argv=/usr/local/bin/dmarc-ingest ${recipient}'
|
||||
# One at a time (the 1 before "pipe"): reports are in no hurry, every run
|
||||
# writes to the database the panel writes to, and a hundred of them in
|
||||
# parallel - Postfix's default - is what a flood of reports would start.
|
||||
postconf -M 'dmarc-ingest/unix=dmarc-ingest unix - n n - 1 pipe flags=Rq user=panel argv=/usr/local/bin/dmarc-ingest ${recipient}'
|
||||
# One recipient per run, so ${recipient} is one argument.
|
||||
postconf -e "dmarc-ingest_destination_recipient_limit=1"
|
||||
# pipe(8) starts the command with export_environment only, so what the
|
||||
# command reads from the container's environment is named there.
|
||||
# export_environment is a list split at whitespace in which Postfix
|
||||
# expands "$": a path with either would reach the command as another
|
||||
# path, and it would quietly open a database of its own.
|
||||
case "${SELFPOST_DATA_DIR:-}${SELFPOST_DB_PATH:-}" in
|
||||
*[[:space:]]* | *'$'*)
|
||||
echo "postfix-config: SELFPOST_DATA_DIR and SELFPOST_DB_PATH must not contain whitespace or \$ when DMARC_REPORTS_ENABLE=true" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
DMARC_INGEST_ENV="DMARC_REPORTS_ENABLE=true"
|
||||
[ -z "${SELFPOST_DATA_DIR:-}" ] || DMARC_INGEST_ENV="${DMARC_INGEST_ENV} SELFPOST_DATA_DIR=${SELFPOST_DATA_DIR}"
|
||||
[ -z "${SELFPOST_DB_PATH:-}" ] || DMARC_INGEST_ENV="${DMARC_INGEST_ENV} SELFPOST_DB_PATH=${SELFPOST_DB_PATH}"
|
||||
@@ -345,10 +374,18 @@ if [ "${INBOUND_RELAY_ENABLE}" = "true" ] || [ "${DMARC_REPORTS_ENABLE}" = "true
|
||||
# would file inbound mail into the outbound send log under the *sender's*
|
||||
# domain — polluting Deliveries and letting a forged From: consume that
|
||||
# domain's level-2 rate-limit budget. So smtpd_milters is always overridden
|
||||
# here: either the optional antispam milter, or nothing at all.
|
||||
# here: the optional antispam milter, then - with the inbound relay on - the
|
||||
# panel's inbound journal milter, or nothing at all.
|
||||
INBOUND_MILTERS=""
|
||||
if [ "${INBOUND_RELAY_ENABLE}" = "true" ] && [ -n "${INBOUND_ANTISPAM_MILTER}" ]; then
|
||||
# A glob's "*" matches anything, so the shape alone would pass
|
||||
# "inet:a:1},{unix:/elsewhere": characters outside the set are refused
|
||||
# first, then the shape is checked.
|
||||
case "${INBOUND_ANTISPAM_MILTER}" in
|
||||
*[!A-Za-z0-9._:/-]*)
|
||||
echo "FATAL: INBOUND_ANTISPAM_MILTER must be inet:host:port or unix:/path, got: ${INBOUND_ANTISPAM_MILTER}" >&2
|
||||
exit 1
|
||||
;;
|
||||
inet:[A-Za-z0-9._-]*:[0-9]* | unix:/[A-Za-z0-9._/-]* ) ;;
|
||||
*)
|
||||
echo "FATAL: INBOUND_ANTISPAM_MILTER must be inet:host:port or unix:/path, got: ${INBOUND_ANTISPAM_MILTER}" >&2
|
||||
@@ -364,6 +401,26 @@ if [ "${INBOUND_RELAY_ENABLE}" = "true" ] || [ "${DMARC_REPORTS_ENABLE}" = "true
|
||||
esac
|
||||
INBOUND_MILTERS="{${INBOUND_ANTISPAM_MILTER},default_action=${INBOUND_ANTISPAM_MILTER_ACTION}}"
|
||||
fi
|
||||
if [ "${INBOUND_RELAY_ENABLE}" = "true" ]; then
|
||||
# Last in the chain: Postfix hands a message to its milters one after
|
||||
# another at end-of-message, so this one reads the verdict headers the
|
||||
# filter added and sees only what the filter let through. It watches and
|
||||
# never refuses, and must never hold up someone else's mail: accept when
|
||||
# it is down. A comma, not a space, between the two - see
|
||||
# master_cf_set_smtp_inet_o.
|
||||
case "${INBOUND_JOURNAL_SOCK}" in
|
||||
*[!A-Za-z0-9._/-]*)
|
||||
echo "FATAL: INBOUND_JOURNAL_MILTER_SOCKET must be an absolute path, got: ${INBOUND_JOURNAL_SOCK}" >&2
|
||||
exit 1
|
||||
;;
|
||||
/[A-Za-z0-9._/-]*) ;;
|
||||
*)
|
||||
echo "FATAL: INBOUND_JOURNAL_MILTER_SOCKET must be an absolute path, got: ${INBOUND_JOURNAL_SOCK}" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
INBOUND_MILTERS="${INBOUND_MILTERS:+${INBOUND_MILTERS},}{unix:${INBOUND_JOURNAL_SOCK},default_action=accept}"
|
||||
fi
|
||||
|
||||
PORT25_RATE="${INBOUND_RATE_LIMIT_MESSAGES_PER_IP}"
|
||||
PORT25_SIZE="${INBOUND_MESSAGE_SIZE_LIMIT}"
|
||||
@@ -380,8 +437,12 @@ if [ "${INBOUND_RELAY_ENABLE}" = "true" ] || [ "${DMARC_REPORTS_ENABLE}" = "true
|
||||
fi
|
||||
fi
|
||||
|
||||
# syslog_name: this listener's lines read postfix/inbound/smtpd, the
|
||||
# submission ports' postfix/smtpd. The log tailer files a refusal on port 25
|
||||
# into the inbound journal and has to tell the two apart.
|
||||
postconf -M "smtp/inet=smtp inet n - n - - smtpd"
|
||||
postconf -P \
|
||||
"smtp/inet/syslog_name=postfix/inbound" \
|
||||
"smtp/inet/smtpd_sasl_auth_enable=no" \
|
||||
"smtp/inet/smtpd_tls_auth_only=no" \
|
||||
"smtp/inet/smtpd_sender_login_maps=" \
|
||||
|
||||
@@ -2,8 +2,9 @@
|
||||
# Postfix start wrapper: solves the cold-start race where Postfix would try to
|
||||
# reach the milter sockets before they are listening.
|
||||
#
|
||||
# It blocks until BOTH milter sockets — OpenDKIM and the panel's journal-milter
|
||||
# — are present, then execs `postfix start-fg`. If they are not ready within the
|
||||
# It blocks until the milter sockets — OpenDKIM, the panel's journal-milter
|
||||
# and, with the inbound relay on, its inbound journal milter — are present,
|
||||
# then execs `postfix start-fg`. If they are not ready within the
|
||||
# timeout it exits non-zero WITHOUT starting Postfix, so supervisord/Docker see
|
||||
# an explicit startup failure instead of a relay running blind.
|
||||
#
|
||||
@@ -16,8 +17,13 @@ JOURNAL_SOCK="${JOURNAL_MILTER_SOCKET:-/run/selfpost/journal.sock}"
|
||||
TIMEOUT="${MILTER_WAIT_TIMEOUT:-30}"
|
||||
INTERVAL=1
|
||||
|
||||
SOCKS="$OPENDKIM_SOCK $JOURNAL_SOCK"
|
||||
if [ "${INBOUND_RELAY_ENABLE:-false}" = "true" ]; then
|
||||
SOCKS="$SOCKS ${INBOUND_JOURNAL_MILTER_SOCKET:-/run/selfpost/inbound-journal.sock}"
|
||||
fi
|
||||
|
||||
elapsed=0
|
||||
for sock in "$OPENDKIM_SOCK" "$JOURNAL_SOCK"; do
|
||||
for sock in $SOCKS; do
|
||||
while [ ! -S "$sock" ]; do
|
||||
if [ "$elapsed" -ge "$TIMEOUT" ]; then
|
||||
echo "postfix-wrapper: timed out after ${TIMEOUT}s waiting for milter socket $sock" >&2
|
||||
@@ -29,5 +35,5 @@ for sock in "$OPENDKIM_SOCK" "$JOURNAL_SOCK"; do
|
||||
echo "postfix-wrapper: milter socket ready: $sock"
|
||||
done
|
||||
|
||||
echo "postfix-wrapper: both milter sockets ready, starting postfix"
|
||||
echo "postfix-wrapper: milter sockets ready, starting postfix"
|
||||
exec postfix start-fg
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
@@ -28,14 +29,47 @@ func runDMARCIngestMode() error {
|
||||
|
||||
recipient := ingestRecipient(os.Args[1:])
|
||||
if err := dmarc.IngestMessage(st, io.LimitReader(os.Stdin, maxIngestStdin), recipient, time.Now().UTC()); err != nil {
|
||||
if !errors.Is(err, dmarc.ErrRejected) {
|
||||
return err
|
||||
}
|
||||
// Not a report this relay keeps. It is dropped: deferring would have
|
||||
// Postfix try it again for days, and a bounce would go to an envelope
|
||||
// sender anyone can write - the report addresses are public.
|
||||
if incrErr := st.IncrDMARCParseFailures(); incrErr != nil {
|
||||
log.Printf("dmarc ingest: record failure: %v", incrErr)
|
||||
}
|
||||
return err
|
||||
logRejected(err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// logRejected leaves the reason in the mail log. Postfix puts the start of
|
||||
// what a delivery command wrote - about a hundred bytes - into its own line
|
||||
// for the message ("status=sent (delivered via dmarc-ingest service (...))"),
|
||||
// so the reason goes out first and bare: no prefix, no date, no recipient,
|
||||
// all of which that line already has. postlog is no use here: pipe(8) starts
|
||||
// the command without the group that may write to Postfix's log socket.
|
||||
func logRejected(reason error) {
|
||||
fmt.Fprintln(os.Stderr, "dropped: "+logSafe(strings.TrimPrefix(reason.Error(), dmarc.ErrRejected.Error()+": ")))
|
||||
}
|
||||
|
||||
// logSafe makes a reason fit for a log line. Parts of it are the sender's own
|
||||
// bytes - a header line, a file name inside an archive, a piece of XML - so it
|
||||
// is cut short and anything that is not printable ASCII becomes "?", whatever
|
||||
// Postfix would have done with it.
|
||||
func logSafe(s string) string {
|
||||
const max = 200
|
||||
if len(s) > max {
|
||||
s = s[:max]
|
||||
}
|
||||
return strings.Map(func(r rune) rune {
|
||||
if r < 0x20 || r > 0x7e {
|
||||
return '?'
|
||||
}
|
||||
return r
|
||||
}, s)
|
||||
}
|
||||
|
||||
// ingestRecipient is the report address the message was delivered to. Postfix's
|
||||
// pipe(8) hands it over as an argument (${recipient} in the service's argv,
|
||||
// build/postfix-config.sh) - it sets no RECIPIENT variable, that is local(8).
|
||||
|
||||
@@ -18,3 +18,17 @@ func TestIngestRecipient(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestLogSafe(t *testing.T) {
|
||||
got := logSafe("zip entry \"a\nstatus=bounced\tx\" \x1b[31m é")
|
||||
if want := "zip entry \"a?status=bounced?x\" ?[31m ?"; got != want {
|
||||
t.Errorf("logSafe = %q, want %q", got, want)
|
||||
}
|
||||
long := make([]byte, 1000)
|
||||
for i := range long {
|
||||
long[i] = 'a'
|
||||
}
|
||||
if got := logSafe(string(long)); len(got) != 200 {
|
||||
t.Errorf("logSafe kept %d bytes of 1000, want 200", len(got))
|
||||
}
|
||||
}
|
||||
|
||||
+14
-2
@@ -24,6 +24,7 @@ type mailStack struct {
|
||||
Domains *domain.Service
|
||||
Apps *app.Service
|
||||
Inbound *inbound.Service
|
||||
Lists *inbound.Lists
|
||||
DMARC *dmarc.Service
|
||||
pf *postfix.Postfix
|
||||
odk *domain.OpenDKIM
|
||||
@@ -36,11 +37,15 @@ func newMailStack(cfg config, st *store.Store) *mailStack {
|
||||
odk := domain.NewOpenDKIM(cfg.opendkimDir)
|
||||
apps := app.NewService(st, app.NewSASLDB(cfg.saslDBPath, cfg.saslRealm), pf)
|
||||
domains := domain.NewService(st, odk, apps, cfg.dkimSelectorDef)
|
||||
// The panel's notifications are sent as postmaster@ the server's own name,
|
||||
// and signed as it.
|
||||
domains.SetServerName(cfg.hostname)
|
||||
inb := inbound.NewService(st, pf)
|
||||
dmarcSvc := dmarc.NewService(st, pf, cfg.hostname, cfg.dmarcEnabled)
|
||||
return &mailStack{
|
||||
Domains: domains, Apps: apps, Inbound: inb, DMARC: dmarcSvc,
|
||||
pf: pf, odk: odk, inboundEnabled: cfg.inboundEnabled, dmarcEnabled: cfg.dmarcEnabled,
|
||||
Lists: inbound.NewLists(st, cfg.antispamDir),
|
||||
pf: pf, odk: odk, inboundEnabled: cfg.inboundEnabled, dmarcEnabled: cfg.dmarcEnabled,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -57,6 +62,11 @@ func (m *mailStack) Resync() error {
|
||||
if err := m.Inbound.Resync(); err != nil {
|
||||
return fmt.Errorf("inbound maps resync: %w", err)
|
||||
}
|
||||
// The filter's configuration names all six list files; they exist from
|
||||
// the first start, empty until an entry is added.
|
||||
if err := m.Lists.Sync(); err != nil {
|
||||
return fmt.Errorf("filter lists resync: %w", err)
|
||||
}
|
||||
}
|
||||
if m.dmarcEnabled {
|
||||
if err := m.DMARC.Resync(); err != nil {
|
||||
@@ -91,7 +101,7 @@ func newPanel(cfg config, st *store.Store) (*web.Server, error) {
|
||||
// effective config, including a manual override; the panel keeps this
|
||||
// snapshot for the process lifetime (architecture.md).
|
||||
retryPolicy := postfix.LoadRetryPolicy()
|
||||
return web.New(st, ms.Domains, ms.Apps, ms.Inbound, ms.DMARC, web.Config{
|
||||
return web.New(st, ms.Domains, ms.Apps, ms.Inbound, ms.Lists, ms.DMARC, web.Config{
|
||||
Hostname: cfg.hostname,
|
||||
CookieSecure: cfg.cookieSecure,
|
||||
SubmissionEnabled: cfg.submissionEnabled,
|
||||
@@ -114,6 +124,8 @@ func newPanel(cfg config, st *store.Store) (*web.Server, error) {
|
||||
InboundAntispamAction: cfg.inboundAntispamAction,
|
||||
DMARCEnabled: cfg.dmarcEnabled,
|
||||
SendLogRetentionEnvDefault: cfg.retentionDays,
|
||||
|
||||
InboundSpamLogRetentionEnvDefault: cfg.inboundSpamLogRetentionDays,
|
||||
}, cfg.setupTokenPath)
|
||||
}
|
||||
|
||||
|
||||
+39
-9
@@ -8,6 +8,7 @@ import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
"github.com/mixeme/selfpost/internal/inbound"
|
||||
"github.com/mixeme/selfpost/internal/milter"
|
||||
"github.com/mixeme/selfpost/internal/store"
|
||||
)
|
||||
@@ -15,21 +16,52 @@ import (
|
||||
// serveJournal opens the journal-milter Unix socket and runs the real milter
|
||||
// (architecture.md § Mail path), recording accepted messages into the send
|
||||
// log. Socket lifecycle (creation, stale cleanup, group permissions) lives
|
||||
// here; the protocol handler lives in internal/milter.
|
||||
// in listenMilter; the protocol handler lives in internal/milter.
|
||||
func serveJournal(ctx context.Context, cfg config, st *store.Store) error {
|
||||
socketPath := cfg.journalSocket
|
||||
if err := os.MkdirAll(filepath.Dir(socketPath), 0o755); err != nil {
|
||||
ln, err := listenMilter(cfg.journalSocket)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
log.Printf("journal-milter listening on %s", cfg.journalSocket)
|
||||
return milter.Serve(ctx, ln, st)
|
||||
}
|
||||
|
||||
// serveInboundJournal runs the inbound journal milter, the last milter on
|
||||
// port 25 (internal/milter/inbound.go). It exists only while the inbound relay
|
||||
// is on: build/postfix-config.sh attaches the socket under the same condition.
|
||||
func serveInboundJournal(ctx context.Context, cfg config, st *store.Store) error {
|
||||
ln, err := listenMilter(cfg.inboundJournalSocket)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// With a filter attached, the key it signs with. Without the key no
|
||||
// verdict can be told from a sender's own, so none is read: the journal
|
||||
// still records the mail, and the log says why it shows no verdicts.
|
||||
key := ""
|
||||
if cfg.inboundAntispamMilter != "" {
|
||||
if key, err = inbound.FilterKey(cfg.antispamDir); err != nil {
|
||||
log.Printf("inbound-journal-milter: no key for the filter's signature, verdicts will not be recorded: %v", err)
|
||||
key = ""
|
||||
}
|
||||
}
|
||||
log.Printf("inbound-journal-milter listening on %s", cfg.inboundJournalSocket)
|
||||
return milter.ServeInbound(ctx, ln, st, key)
|
||||
}
|
||||
|
||||
// listenMilter opens a milter's Unix socket where Postfix can reach it.
|
||||
func listenMilter(socketPath string) (net.Listener, error) {
|
||||
if err := os.MkdirAll(filepath.Dir(socketPath), 0o755); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// Clear a stale socket left behind by an unclean shutdown, otherwise the
|
||||
// listen below fails with "address already in use".
|
||||
if err := os.Remove(socketPath); err != nil && !errors.Is(err, os.ErrNotExist) {
|
||||
return err
|
||||
return nil, err
|
||||
}
|
||||
|
||||
ln, err := net.Listen("unix", socketPath)
|
||||
if err != nil {
|
||||
return err
|
||||
return nil, err
|
||||
}
|
||||
// Postfix (user `postfix`) connects to this socket as a milter and needs
|
||||
// group access. The socket inherits group `selfpost` from the setgid parent
|
||||
@@ -37,9 +69,7 @@ func serveJournal(ctx context.Context, cfg config, st *store.Store) error {
|
||||
// it (connecting to a Unix socket needs write permission on the node).
|
||||
if err := os.Chmod(socketPath, 0o660); err != nil {
|
||||
ln.Close()
|
||||
return err
|
||||
return nil, err
|
||||
}
|
||||
|
||||
log.Printf("journal-milter listening on %s", socketPath)
|
||||
return milter.Serve(ctx, ln, st)
|
||||
return ln, nil
|
||||
}
|
||||
|
||||
+60
-4
@@ -23,6 +23,7 @@ import (
|
||||
"github.com/mixeme/selfpost/internal/backup"
|
||||
"github.com/mixeme/selfpost/internal/buildinfo"
|
||||
"github.com/mixeme/selfpost/internal/dnscheck"
|
||||
"github.com/mixeme/selfpost/internal/inbound"
|
||||
"github.com/mixeme/selfpost/internal/logtail"
|
||||
"github.com/mixeme/selfpost/internal/store"
|
||||
)
|
||||
@@ -89,6 +90,9 @@ type config struct {
|
||||
saslRealm string
|
||||
postfixDir string
|
||||
deployRoot string
|
||||
// Where the filter lists are written as map files for the external spam
|
||||
// filter; the antispam compose fragment mounts it into the sidecar.
|
||||
antispamDir string
|
||||
|
||||
inboundEnabled bool
|
||||
// The optional milter on the inbound listener (INBOUND_ANTISPAM_MILTER) and
|
||||
@@ -96,7 +100,12 @@ type config struct {
|
||||
// what attaches it; the panel only says so. See antispamAction.
|
||||
inboundAntispamMilter string
|
||||
inboundAntispamAction string
|
||||
dmarcEnabled bool
|
||||
// The inbound journal milter's socket; the default mirrors
|
||||
// build/postfix-config.sh. Served only while the inbound relay is on.
|
||||
inboundJournalSocket string
|
||||
// Retention of the inbound journal until Server > Settings says otherwise.
|
||||
inboundSpamLogRetentionDays int
|
||||
dmarcEnabled bool
|
||||
}
|
||||
|
||||
func loadConfig() config {
|
||||
@@ -157,11 +166,16 @@ func loadConfig() config {
|
||||
saslDBPath: envDefault("SASL_DB_PATH", filepath.Join(dataDir, "sasl", "sasldb2")),
|
||||
saslRealm: saslRealm(),
|
||||
postfixDir: envDefault("POSTFIX_DIR", filepath.Join(dataDir, "postfix")),
|
||||
deployRoot: envDefault("SELFPOST_DEPLOY_ROOT", "/selfpost-deploy"),
|
||||
|
||||
antispamDir: envDefault("ANTISPAM_DIR", filepath.Join(dataDir, "antispam")),
|
||||
deployRoot: envDefault("SELFPOST_DEPLOY_ROOT", "/selfpost-deploy"),
|
||||
// Optional inbound relay (backup-MX / forwarder). Off unless exactly "true".
|
||||
inboundEnabled: os.Getenv("INBOUND_RELAY_ENABLE") == "true",
|
||||
inboundAntispamMilter: os.Getenv("INBOUND_ANTISPAM_MILTER"),
|
||||
inboundAntispamAction: antispamAction(os.Getenv("INBOUND_ANTISPAM_MILTER_ACTION")),
|
||||
inboundJournalSocket: envDefault("INBOUND_JOURNAL_MILTER_SOCKET", "/run/selfpost/inbound-journal.sock"),
|
||||
|
||||
inboundSpamLogRetentionDays: envInt("INBOUND_SPAM_LOG_RETENTION_DAYS", store.InboundSpamLogRetentionDaysDefault),
|
||||
// Optional DMARC aggregate ingest on port 25. Off unless exactly "true".
|
||||
dmarcEnabled: os.Getenv("DMARC_REPORTS_ENABLE") == "true",
|
||||
}
|
||||
@@ -289,12 +303,46 @@ func run() error {
|
||||
}
|
||||
}
|
||||
|
||||
// The filter lists as files, on every start: the spam filter's
|
||||
// configuration names all six, and a data directory from before the lists
|
||||
// existed has none. Not fatal - the lists are the filter's business, and
|
||||
// the panel has mail to relay either way.
|
||||
if cfg.inboundEnabled && !restored {
|
||||
if err := inbound.NewLists(st, cfg.antispamDir).Sync(); err != nil {
|
||||
log.Printf("filter lists bootstrap: %v", err)
|
||||
}
|
||||
}
|
||||
// The key the filter signs with has to be in the shared directory before
|
||||
// the filter reads it, whether or not a filter is attached yet.
|
||||
if cfg.inboundEnabled {
|
||||
if _, err := inbound.FilterKey(cfg.antispamDir); err != nil {
|
||||
log.Printf("filter key bootstrap: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
if err := st.EnsureSendLogRetentionDays(cfg.retentionDays); err != nil {
|
||||
return fmt.Errorf("bootstrap send-log retention: %w", err)
|
||||
}
|
||||
// With the inbound relay on, the tailer also writes the inbound journal's
|
||||
// rows for refused mail and prunes that journal.
|
||||
var inboundTail *logtail.Inbound
|
||||
if cfg.inboundEnabled {
|
||||
if err := st.EnsureInboundSpamLogRetentionDays(cfg.inboundSpamLogRetentionDays); err != nil {
|
||||
return fmt.Errorf("bootstrap inbound journal retention: %w", err)
|
||||
}
|
||||
inboundTail = &logtail.Inbound{Store: st, Retention: func() int {
|
||||
days, err := st.GetInboundSpamLogRetentionDays(cfg.inboundSpamLogRetentionDays)
|
||||
if err != nil {
|
||||
log.Printf("inbound journal retention: %v", err)
|
||||
return cfg.inboundSpamLogRetentionDays
|
||||
}
|
||||
return days
|
||||
}}
|
||||
}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
errc := make(chan error, 4)
|
||||
// Room for every role to report a failure without blocking: six at most.
|
||||
errc := make(chan error, 6)
|
||||
|
||||
roles := []struct {
|
||||
name string
|
||||
@@ -310,12 +358,20 @@ func run() error {
|
||||
return cfg.retentionDays
|
||||
}
|
||||
return days
|
||||
})
|
||||
}, inboundTail)
|
||||
}},
|
||||
{"rate-limit-recalc", func(ctx context.Context) error {
|
||||
runAutoRateLimitRecalc(ctx, cfg, st)
|
||||
return nil
|
||||
}},
|
||||
{"notifier", func(ctx context.Context) error { return runNotifier(ctx, cfg, st) }},
|
||||
}
|
||||
|
||||
if cfg.inboundEnabled {
|
||||
roles = append(roles, struct {
|
||||
name string
|
||||
fn func(context.Context) error
|
||||
}{"inbound-journal-milter", func(ctx context.Context) error { return serveInboundJournal(ctx, cfg, st) }})
|
||||
}
|
||||
|
||||
for _, r := range roles {
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/mixeme/selfpost/internal/dnscheck"
|
||||
"github.com/mixeme/selfpost/internal/health"
|
||||
"github.com/mixeme/selfpost/internal/notify"
|
||||
"github.com/mixeme/selfpost/internal/postfix"
|
||||
"github.com/mixeme/selfpost/internal/store"
|
||||
)
|
||||
|
||||
// runNotifier writes to the panel's users when something needs them
|
||||
// (architecture.md § Notifications), until ctx is cancelled. It reads what the
|
||||
// pages read - a domain's published records against the key this server signs
|
||||
// with, the certificate file, Postfix's queue - with a DNS checker of its own,
|
||||
// so that its hourly look is never a page's cached one.
|
||||
func runNotifier(ctx context.Context, cfg config, st *store.Store) error {
|
||||
ms := newMailStack(cfg, st)
|
||||
dns := dnscheck.New(cfg.dnsResolvers)
|
||||
|
||||
n := notify.New(st, notify.Checks{
|
||||
Domain: func(d store.Domain) (dnscheck.Domain, error) {
|
||||
rec, err := ms.Domains.DKIMRecord(d)
|
||||
if err != nil {
|
||||
return dnscheck.Domain{}, err
|
||||
}
|
||||
srv := dns.Server(cfg.hostname, true)
|
||||
return dns.Domain(dnscheck.Query{
|
||||
Name: d.Name,
|
||||
Selector: d.DKIMSelector,
|
||||
ExpectedDKIM: rec.Value,
|
||||
Hostname: srv.Hostname,
|
||||
ServerIPs: srv.IPs,
|
||||
DMARCReportEmail: d.DMARCRua,
|
||||
}, true), nil
|
||||
},
|
||||
Certificate: func() health.Certificate { return health.CheckCertificate(cfg.tlsCertFile) },
|
||||
Queue: postfix.QueueMessages,
|
||||
}, notify.NewMailer(cfg.hostname), cfg.hostname)
|
||||
return n.Run(ctx)
|
||||
}
|
||||
@@ -30,6 +30,10 @@ DMARC_REPORTS_ENABLE=false
|
||||
# INBOUND_ANTISPAM_MILTER=inet:antispam:11332
|
||||
# INBOUND_ANTISPAM_MILTER_ACTION=accept
|
||||
|
||||
# Days the inbound journal keeps a row (7-365) until the panel says otherwise;
|
||||
# it never holds more than 10 000 rows.
|
||||
# INBOUND_SPAM_LOG_RETENTION_DAYS=90
|
||||
|
||||
# Coarse per-client-IP limit and message size on inbound smtpd (port 25).
|
||||
# INBOUND_RATE_LIMIT_MESSAGES_PER_IP=20
|
||||
# INBOUND_MESSAGE_SIZE_LIMIT=26214400
|
||||
|
||||
@@ -27,3 +27,10 @@ services:
|
||||
image: rspamd/rspamd:3.11
|
||||
restart: unless-stopped
|
||||
# No host ports: only SelfPost talks to the milter on the compose network.
|
||||
volumes:
|
||||
# The verdict headers SelfPost's inbound journal reads.
|
||||
- ./antispam/rspamd/milter_headers.conf:/etc/rspamd/local.d/milter_headers.conf:ro
|
||||
# The allow and deny lists kept in the panel: the rules, and the files
|
||||
# SelfPost writes them to.
|
||||
- ./antispam/rspamd/multimap.conf:/etc/rspamd/local.d/multimap.conf:ro
|
||||
- ./data/antispam:/etc/rspamd/selfpost:ro
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# Mounted into the sidecar as /etc/rspamd/local.d/milter_headers.conf.
|
||||
#
|
||||
# rspamd writes its verdict into two headers of every message it lets through:
|
||||
# the action it chose and the score with the symbols behind it. SelfPost's
|
||||
# inbound journal reads them (Inbound > Log shows them next to the message),
|
||||
# and the upstream mailbox server receives them with the mail.
|
||||
#
|
||||
# rspamd removes headers of these names that a sender wrote himself before it
|
||||
# adds its own. Without this file the journal still lists accepted mail, with
|
||||
# no verdict next to it.
|
||||
#
|
||||
# The third header is the proof that the other two are rspamd's. The filter is
|
||||
# attached so that mail still flows when it is down, and then nothing removes
|
||||
# what a sender wrote: he could send his own verdict. So rspamd signs each
|
||||
# message - an HMAC-SHA256 of the message's queue id under a key SelfPost
|
||||
# generates and leaves in the directory it shares with this container - and
|
||||
# SelfPost reads a verdict only next to a signature it can recompute. The
|
||||
# signature travels on with the mail and is of no use to whoever sees it: it
|
||||
# fits that one queue id.
|
||||
use = ["x-spamd-result", "x-rspamd-action", "selfpost-signature"];
|
||||
|
||||
custom {
|
||||
selfpost-signature = <<EOD
|
||||
return function(task, common_meta)
|
||||
local remove = { ['X-SelfPost-Filter'] = 0 }
|
||||
-- The key is read for every message and kept nowhere: SelfPost may make a
|
||||
-- new one (the file was lost, a backup was restored), and a key remembered
|
||||
-- here would go on signing what SelfPost no longer accepts until someone
|
||||
-- restarted this container. The file is one short line.
|
||||
local key
|
||||
local f = io.open('/etc/rspamd/selfpost/filter.key', 'r')
|
||||
if f then
|
||||
key = f:read('*l')
|
||||
f:close()
|
||||
end
|
||||
local qid = task:get_queue_id()
|
||||
if not key or key == '' or not qid or qid == '' then
|
||||
return nil, {}, remove, {}
|
||||
end
|
||||
local hash = require "rspamd_cryptobox_hash"
|
||||
local sig = hash.create_specific_keyed(key, 'sha256', qid):hex()
|
||||
return nil, { ['X-SelfPost-Filter'] = sig }, remove, {}
|
||||
end
|
||||
EOD;
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
# Mounted into the sidecar as /etc/rspamd/local.d/multimap.conf.
|
||||
#
|
||||
# The allow and deny lists kept in the panel (Inbound > Filter lists). SelfPost
|
||||
# writes them as six files under data/antispam/, which the fragment mounts at
|
||||
# /etc/rspamd/selfpost; rspamd rereads a file when it changes, no restart.
|
||||
#
|
||||
# Deny wins over allow, whichever entries match. A denied sender is refused
|
||||
# before anything else is looked at (a prefilter). An allowed one is not
|
||||
# accepted outright - rspamd would then let whichever of the two rules ran
|
||||
# first decide - but given a weight no spam score outweighs, which a refusal
|
||||
# by the deny list still overrides.
|
||||
#
|
||||
# A domain entry is that domain, not its subdomains.
|
||||
#
|
||||
# SelfPost counts how often an entry decided a message by what these rules
|
||||
# leave behind: the deny message below and the names of the allow symbols.
|
||||
# Change either and Inbound > Filter lists shows "never" for every entry.
|
||||
|
||||
SELFPOST_DENY_IP {
|
||||
type = "ip";
|
||||
map = "/etc/rspamd/selfpost/deny_ip.map";
|
||||
prefilter = true;
|
||||
action = "reject";
|
||||
message = "Refused by this server's deny list";
|
||||
description = "Client address on the deny list";
|
||||
}
|
||||
SELFPOST_DENY_FROM {
|
||||
type = "from";
|
||||
map = "/etc/rspamd/selfpost/deny_from.map";
|
||||
prefilter = true;
|
||||
action = "reject";
|
||||
message = "Refused by this server's deny list";
|
||||
description = "Sender address on the deny list";
|
||||
}
|
||||
SELFPOST_DENY_DOMAIN {
|
||||
type = "from";
|
||||
filter = "email:domain";
|
||||
map = "/etc/rspamd/selfpost/deny_domain.map";
|
||||
prefilter = true;
|
||||
action = "reject";
|
||||
message = "Refused by this server's deny list";
|
||||
description = "Sender domain on the deny list";
|
||||
}
|
||||
|
||||
SELFPOST_ALLOW_IP {
|
||||
type = "ip";
|
||||
map = "/etc/rspamd/selfpost/allow_ip.map";
|
||||
score = -100.0;
|
||||
description = "Client address on the allow list";
|
||||
}
|
||||
SELFPOST_ALLOW_FROM {
|
||||
type = "from";
|
||||
map = "/etc/rspamd/selfpost/allow_from.map";
|
||||
score = -100.0;
|
||||
description = "Sender address on the allow list";
|
||||
}
|
||||
SELFPOST_ALLOW_DOMAIN {
|
||||
type = "from";
|
||||
filter = "email:domain";
|
||||
map = "/etc/rspamd/selfpost/allow_domain.map";
|
||||
score = -100.0;
|
||||
description = "Sender domain on the allow list";
|
||||
}
|
||||
@@ -21,7 +21,7 @@
|
||||
|
||||
services:
|
||||
selfpost:
|
||||
image: ghcr.io/mixeme/selfpost:2.0.0
|
||||
image: ghcr.io/mixeme/selfpost:2.2.0
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
SELFPOST_HOSTNAME: "${SELFPOST_HOSTNAME:?set the mail/panel hostname, e.g. mail.example.com}"
|
||||
@@ -47,6 +47,7 @@ services:
|
||||
# How long the send log keeps rows before the background sweep deletes
|
||||
# them — the main driver of /data growth over time.
|
||||
SEND_LOG_RETENTION_DAYS: "${SEND_LOG_RETENTION_DAYS:-90}"
|
||||
INBOUND_SPAM_LOG_RETENTION_DAYS: "${INBOUND_SPAM_LOG_RETENTION_DAYS:-90}"
|
||||
# Sliding idle timeout for the panel login session, in days.
|
||||
PANEL_SESSION_IDLE_DAYS: "${PANEL_SESSION_IDLE_DAYS:-7}"
|
||||
# Resolvers the panel's PTR/SPF/DKIM/DMARC checks query directly, so they
|
||||
|
||||
+175
-17
@@ -27,8 +27,8 @@ Managed programs ([build/supervisord.conf](../build/supervisord.conf)):
|
||||
| Program | User | Priority | Role |
|
||||
|---|---|---|---|
|
||||
| `opendkim` | root → `opendkim` (`UserID` in opendkim.conf) | 100 | DKIM signing milter |
|
||||
| `panel` | panel | 200 | HTTP UI + journal-milter + log-tailer + rate-limit recalc |
|
||||
| `postfix` | root (wrapper) | 300 | MTA — started only after both milter sockets exist |
|
||||
| `panel` | panel | 200 | HTTP UI + journal-milter + log-tailer + rate-limit recalc; with the inbound relay on also the inbound journal milter |
|
||||
| `postfix` | root (wrapper) | 300 | MTA — started only after the milter sockets exist (OpenDKIM's, the journal milter's and, with the inbound relay on, the inbound journal milter's) |
|
||||
| `postfix-reload` | root | — | On-demand `postfix reload` (autostart off) |
|
||||
| `cert-reload` | root | 400 | Daily `postfix reload` for renewed TLS certs |
|
||||
| `logrotate` | root | 400 | Periodic `mail.log` rotation |
|
||||
@@ -74,10 +74,61 @@ smtp/inet` removes Debian's stock smtpd). Outbound delivery still uses the
|
||||
`smtpd_tls_security_level=encrypt`.
|
||||
- **25/smtp inet** — when `INBOUND_RELAY_ENABLE=true` and/or
|
||||
`DMARC_REPORTS_ENABLE=true`. No SASL, no OpenDKIM, no journal-milter —
|
||||
`smtpd_milters` is always overridden on this service (to the antispam milter
|
||||
or to nothing), because main.cf's submission chain would otherwise let a
|
||||
signing outage defer relayed mail and would file inbound mail into the
|
||||
outbound send log under the sender's domain.
|
||||
`smtpd_milters` is always overridden on this service, because main.cf's
|
||||
submission chain would otherwise let a signing outage defer relayed mail and
|
||||
would file inbound mail into the outbound send log under the sender's
|
||||
domain. With the inbound relay on the chain is the optional antispam milter,
|
||||
then the panel's **inbound journal milter** (unix socket
|
||||
`INBOUND_JOURNAL_MILTER_SOCKET`, default
|
||||
`/run/selfpost/inbound-journal.sock`, `default_action=accept`); with DMARC
|
||||
ingest alone it is empty. Postfix hands a message to its milters one after
|
||||
another at end-of-message, each seeing what the one before added, so the
|
||||
journal milter sees only mail the filter let through and reads the filter's
|
||||
verdict headers (`X-Rspamd-Action`, `X-Spamd-Result`) off it. It writes one
|
||||
`inbound_spam_log` row per recipient at an inbound domain - a DMARC report
|
||||
address is not one - and never refuses anything
|
||||
([internal/milter/inbound.go](../internal/milter/inbound.go)). The headers
|
||||
are read only while a filter is attached, and only next to the filter's
|
||||
signature: a third header, `X-SelfPost-Filter`, the HMAC-SHA256 of the
|
||||
message's queue id under the key in `/data/antispam/filter.key`
|
||||
(`inbound.FilterKey` makes it on first start; rspamd reads it through the
|
||||
mounted directory and signs in a custom routine of `milter_headers.conf`).
|
||||
With the filter down - it is attached fail-open - a sender's own verdict
|
||||
headers carry no signature and the row is that of mail nobody judged.
|
||||
Refused mail never reaches this milter.
|
||||
Refusals come from `mail.log`: this listener logs as `postfix/inbound`
|
||||
(`syslog_name`), and the log tailer turns a `reject:` line of
|
||||
`postfix/inbound/smtpd` (Postfix's own policy) and a `milter-reject` /
|
||||
`milter-hold` / `milter-discard` line of `cleanup` (the filter's verdict,
|
||||
counted only for a queue id that listener opened) into a row, dated by the
|
||||
line, when the recipient is at an inbound domain
|
||||
([internal/logtail/inbound.go](../internal/logtail/inbound.go)). A line
|
||||
names one recipient, so a message refused at end-of-message is one row
|
||||
whatever its recipient count. The tailer saves its place every five seconds,
|
||||
so after an unclean stop it reads some lines again; the lines it reads on
|
||||
start are written only if the same row is not there already. The
|
||||
journal is pruned every six hours: rows
|
||||
older than the `inbound_spam_log_retention_days` setting, then down to
|
||||
10 000.
|
||||
**Filter lists** (`inbound_spam_list`): allow and deny entries for the
|
||||
external filter - sender address, sender domain, client address or network -
|
||||
validated and normalised by `internal/inbound` (`lists.go`) and written as
|
||||
six map files under `/data/antispam/` (`ANTISPAM_DIR`), each atomically, on
|
||||
every change and on every start with the relay on. SelfPost never consults
|
||||
them. The sidecar mounts the directory and rspamd's multimap rules
|
||||
([deploy/antispam/rspamd/multimap.conf](../deploy/antispam/rspamd/multimap.conf))
|
||||
read them: a deny entry is a prefilter `reject`, an allow entry a symbol
|
||||
weighing -100. Allow is deliberately not a prefilter `accept`: with two
|
||||
prefilters rspamd lets the one that ran first decide, and deny then won or
|
||||
lost by rule order (seen on rspamd 3.11). A refusal by the deny list
|
||||
overrides any score, so deny wins whichever entries match.
|
||||
How often an entry decided a message is not stored: `Lists.Usage` reads it
|
||||
off the journal on each view of Filter lists - the rows a list decided (the
|
||||
deny rule's reply, the allow rule's symbol, which the journal milter keeps
|
||||
ahead of its six-symbol cap) matched against the entries as rspamd matches
|
||||
them. When that stops being good enough is written in
|
||||
[plans/inbound-antispam-panel.md](plans/inbound-antispam-panel.md) § Filter
|
||||
list usage.
|
||||
**Inbound relay** (`INBOUND_RELAY_ENABLE=true`): accepts only
|
||||
`relay_domains` + `relay_recipient_maps` (`reject_unauth_destination`,
|
||||
`reject_unlisted_recipient`). Maps under `/data/postfix/` (`relay_domains`,
|
||||
@@ -105,9 +156,17 @@ smtp/inet` removes Debian's stock smtpd). Outbound delivery still uses the
|
||||
Per-domain keys under `/data/opendkim/keys`; `KeyTable` / `SigningTable` maintained
|
||||
by the panel. Socket `/run/opendkim/opendkim.sock`.
|
||||
|
||||
The server's own name (`SELFPOST_HOSTNAME`) has a key and a line in both
|
||||
tables too, without being a row of the registry
|
||||
(`domain.Service.SetServerName`): the panel's notifications are sent as
|
||||
`postmaster@` that name and signed with it. No application can send as it -
|
||||
the sender map is built from the registry. A sending domain of the same name
|
||||
shares its key and selector with the server, and deleting that domain leaves
|
||||
the key in place.
|
||||
|
||||
### Panel binary ([cmd/panel](../cmd/panel))
|
||||
|
||||
One process, four roles:
|
||||
One process, five roles (six with the inbound relay on):
|
||||
|
||||
1. **HTTP server** — `:8080` (`PANEL_HTTP_ADDR`); HTTPS terminated by reverse
|
||||
proxy only. On start it runs `postconf -h` once for the deferred-mail retry
|
||||
@@ -133,10 +192,55 @@ One process, four roles:
|
||||
3. **log-tailer** — follows `MAIL_LOG`, updates send-log delivery status by
|
||||
queue-id. Send-log `queued → sent` transitions depend on this goroutine alone
|
||||
(`UpdateStatus` is only called from [internal/logtail](../internal/logtail/logtail.go)).
|
||||
It also keeps the lines Postfix writes under a queue id, for the message's
|
||||
page (§ Log tailer, *Stored lines*).
|
||||
4. **rate-limit recalc** — every six hours (and on demand from Domain settings and
|
||||
the application form), recomputes level-2 **Auto** rate limits from send-log statistics.
|
||||
5. **notifier** — writes to the panel's users when something needs them
|
||||
(§ Notifications below).
|
||||
|
||||
Milter chain in Postfix: OpenDKIM (tempfail) then journal (accept on failure).
|
||||
Mail submitted locally with `sendmail(1)` — the notifier's — passes OpenDKIM
|
||||
alone (`non_smtpd_milters`): it is signed, and it is not an application's
|
||||
mail to be journalled or limited. Bounces Postfix generates pass no milter.
|
||||
|
||||
### Notifications
|
||||
|
||||
[internal/notify](../internal/notify/notify.go) reports four things to the
|
||||
users of the panel: a sending domain's DNS record that stopped matching, a
|
||||
level-2 rate limit that refused mail, the TLS certificate about to expire, a
|
||||
message more than three days in the queue. The rules of when a message goes
|
||||
out are the owner's and are written in
|
||||
[plans/panel-notifications.md](plans/panel-notifications.md) § When a message
|
||||
is sent; the operator's view is guide § Notifications.
|
||||
|
||||
- **Readings.** Once an hour, starting three minutes after the process does:
|
||||
every sending domain's records through a DNS checker of the notifier's own,
|
||||
forced past the cache (`cmd/panel/notifier.go` builds the same query a
|
||||
domain's page does); the certificate file; `postqueue -j` for the arrival
|
||||
time of each queued message. Once a minute: the send log's `rejected` rows
|
||||
of the last day, per domain and application.
|
||||
- **State** is a row per subject in `notify_state` (`dns:<domain>`, `cert`,
|
||||
`queue`, `limit:<domain>:<login>`), which is what makes each thing be said
|
||||
once and a restart say nothing again. A DNS subject goes *pending* on the
|
||||
first bad check and *open*, with a message, on the second; back the same
|
||||
way. The certificate's row remembers which threshold was said for which
|
||||
expiry. A deleted domain's row goes with it.
|
||||
- **No information is not a finding.** A lookup that did not answer, a queue
|
||||
that cannot be listed, a certificate that cannot be read move no state.
|
||||
- **Recipients** come from the store at the moment of sending: global users
|
||||
for everything, domain users for subjects that are a domain's; only users
|
||||
with an e-mail and their own switch on. One message per recipient, so users
|
||||
do not see each other's addresses.
|
||||
- **The instance switch** (settings key `notifications`, on unless turned
|
||||
off) stops the checks themselves, not only the mail: no state moves while
|
||||
it is off.
|
||||
- **Sending** is `sendmail -i -f postmaster@<hostname> -- <address>` as the
|
||||
unprivileged panel user, which is a member of `postdrop` and so can write
|
||||
the maildrop directory without the setgid bit `no-new-privileges` forbids.
|
||||
A message that could not be handed over leaves the state where it was and
|
||||
is tried at the next check. The text names the page to open and has no
|
||||
link: the panel does not know its own address.
|
||||
|
||||
### Log tailer and `mail.log` rotation
|
||||
|
||||
@@ -183,15 +287,66 @@ leaves every row untouched rather than closing them all. Now that the log
|
||||
survives the container, reaching this path means the lines are gone for good —
|
||||
rotated past fourteen files while the panel was down, or deleted.
|
||||
|
||||
**Stored lines** (`send_log_lines`, since 2.2.0;
|
||||
[internal/logtail/lines.go](../internal/logtail/lines.go)). A send-log row
|
||||
lives for the retention window, ninety days by default, and `mail.log` for a
|
||||
day before it is rotated; a message's page (`/outbound/log/{id}`) shows what
|
||||
Postfix wrote about it, so the tailer keeps those lines in the database as it
|
||||
reads them:
|
||||
|
||||
- **What is kept:** a line headed by a queue id — after the timestamp, the
|
||||
host name and `program[pid]:` — with that id, the line's own time and the
|
||||
line as written, cut to 4096 bytes. The id is read from the head of the line
|
||||
only: the tail is the far end's text. `NOQUEUE` and `warning` stand in the
|
||||
same place and are not ids; the pattern accepts Postfix's short form
|
||||
(capital hexadecimal) and its long form (no vowels).
|
||||
- **Port-25 mail is left out.** A queue id opened by `postfix/inbound/smtpd`
|
||||
is a stranger's message and never gets a send-log row; none of its lines
|
||||
are stored, up to qmgr's `removed` or until another listener opens a
|
||||
message under the id (4096 such ids are remembered). What a stranger can
|
||||
make Postfix log is therefore not written to the database at his rate.
|
||||
- **No check for a row at write time.** A message's first lines (`client=`,
|
||||
`message-id=`) are logged before the journal milter writes its row at
|
||||
end-of-message, so the tailer cannot ask whether the id is in the send log.
|
||||
Lines that never get a row — bounces Postfix generates, port-25 mail met in
|
||||
mid-flight after a restart — are kept for a while.
|
||||
- **Written per pass, not per line:** the lines of one pass over the log go
|
||||
in one transaction (`InsertSendLogLines`), 500 at most, before the read
|
||||
offset is saved. A commit per line would be a disk sync per line on the
|
||||
connection the journal milter shares.
|
||||
- **Retention is one rule:** a line older than an hour whose queue id no
|
||||
send-log row carries is deleted (`PruneSendLogLines`), at start and after
|
||||
every six-hourly send-log sweep. That removes the lines of rows the sweep
|
||||
deleted and the lines that never had a row. The send log is asked once per
|
||||
queue id; the statement still walks the table, about a second per million
|
||||
lines, with the connection taken meanwhile.
|
||||
- **Which lines are a row's.** Postfix reuses a queue id once the message has
|
||||
left the queue, and a row outlives that by months. `SendLogLines` gives a row
|
||||
the lines of its queue id dated from one hour before the row was written to
|
||||
the queue lifetime (the larger of `maximal_queue_lifetime` and
|
||||
`bounce_queue_lifetime`) plus a day after it. That limits a chance reuse of
|
||||
the id to those days; it does not exclude it. A message with several
|
||||
recipients is several rows under one id, and each row's page shows all of
|
||||
the message's lines.
|
||||
- **Read twice, stored once.** The read offset is saved every few seconds, so
|
||||
after an unclean stop some lines are read again; while the tailer is
|
||||
catching up at start it writes a line only if the same line is not there.
|
||||
- **Not stored:** what the tailer never read — the tail of a file rotated away
|
||||
while the panel was down.
|
||||
- **Size:** six to ten lines of about 200 bytes per message — more than the row
|
||||
itself. The table is in the database and therefore in a full backup, which
|
||||
`log/` is not ([security.md](security.md) § Backup and domain export).
|
||||
|
||||
**Two one-shot reads** sit beside the follow loop and are unrelated to it, both
|
||||
serving panel pages on request: `TailLines` (the last *n* lines, for
|
||||
`/server/log`) and `QueueLines` (the lines carrying one queue-id, for
|
||||
`/outbound/log/{id}`). `QueueLines` scans a bounded tail of the current file —
|
||||
`/server/log`) and `QueueLines` (the lines carrying one queue-id). `QueueLines`
|
||||
is what a message's page falls back to when no lines are stored for its row —
|
||||
a message accepted before 2.2.0. It scans a bounded tail of the current file —
|
||||
finding a message's lines means reading rather than seeking — and matches the id
|
||||
anchored on the character before it, since queue ids are hexadecimal runs and a
|
||||
shorter one is regularly the tail of a longer one. Send-log rows outlive the log
|
||||
(retention 90 days, rotation 14 files), so an empty result is the expected end
|
||||
state for an older message and the page reports it as such, not as a failure.
|
||||
shorter one is regularly the tail of a longer one. For such a message an empty
|
||||
result is the expected end state once the log has rotated, and the page reports
|
||||
it as such, not as a failure.
|
||||
|
||||
---
|
||||
|
||||
@@ -531,16 +686,17 @@ full backups also archive the operator's `./certs` PEM files when present.
|
||||
|
||||
### Database
|
||||
|
||||
The schema is one embedded baseline, `internal/store/migrations/0001_init.sql`,
|
||||
applied by `store.Open` into an empty file; `PRAGMA user_version` is `1` and
|
||||
The schema is an embedded baseline, `internal/store/migrations/0001_init.sql`,
|
||||
and the migrations after it (`0002_inbound_spam_log.sql`), applied by
|
||||
`store.Open` in file order; `PRAGMA user_version` counts the files applied and
|
||||
`PRAGMA application_id` is `0x53503230` ("SP20"). There is no 1.x migration
|
||||
chain. **2.0 starts from an empty data directory:** a database with a schema
|
||||
version but without the 2.x `application_id` — what every 1.x file looks like —
|
||||
is refused at start with `ErrForeignSchema` and left untouched, as is one whose
|
||||
version is newer than the build knows. A restored 1.x backup fails the same
|
||||
way. Domains are carried across with a domain export, whose format is not the
|
||||
database. Later schema changes are new files (`0002_*.sql`, …); once `2.0.0` is
|
||||
released the baseline is never edited. History and the per-table list:
|
||||
database. A schema change is a new file; the baseline has not been edited
|
||||
since `2.0.0`. History and the per-table list:
|
||||
[schema-migrations.md](schema-migrations.md).
|
||||
|
||||
The tables that carry the panel's model:
|
||||
@@ -565,6 +721,7 @@ The tables that carry the panel's model:
|
||||
(send-log retention). Nothing that belongs to a user.
|
||||
- `applications` (with `auth_ip_restrict` / `auth_allowed_ips`),
|
||||
`application_addresses`, `rate_limits` (no `allowed_ips` column), `send_log`,
|
||||
`send_log_lines` (what Postfix logged under a queue id — § Log tailer),
|
||||
`logtail_state`, `inbound_domains`, `inbound_transports`,
|
||||
`inbound_recipients`, `dmarc_reports`, `dmarc_report_records`.
|
||||
|
||||
@@ -572,7 +729,8 @@ The panel is the only writer; SQLite runs in WAL mode on one connection (see
|
||||
[Panel binary](#panel-binary-cmdpanel)), which is why a full backup snapshots
|
||||
with `VACUUM INTO` instead of copying the file.
|
||||
|
||||
**Rotation:** send-log retention `SEND_LOG_RETENTION_DAYS` (default 90);
|
||||
**Rotation:** send-log retention `SEND_LOG_RETENTION_DAYS` (default 90), which
|
||||
the stored log lines follow;
|
||||
`mail.log` via logrotate (14 rotated files, check every 6h, rename +
|
||||
`postfix reload` in `postrotate` — see § Log tailer above).
|
||||
|
||||
|
||||
@@ -82,6 +82,8 @@
|
||||
<div class="field"><label class="label" for="username">Username</label><div class="control has-icons-left"><input class="input" id="username" name="username" value="admin" autocomplete="username" spellcheck="false" required><span class="icon is-left"><i class="ti ti-user"></i></span></div></div>
|
||||
<div class="field"><label class="label" for="email">E-mail</label><div class="control has-icons-left"><input class="input" id="email" name="email" type="email" value="mix@example.org" autocomplete="email"><span class="icon is-left"><i class="ti ti-mail"></i></span></div>
|
||||
<p class="help">Your own address. A domain's report address can be filled with it from Domain settings.</p></div>
|
||||
<div class="field"><label class="checkbox"><input type="checkbox" name="notify" checked> Send me notifications</label>
|
||||
<p class="help">To this address, when something needs you: a DNS record of a domain stops matching, a rate limit refuses mail, the TLS certificate is about to expire, a message has waited in the queue for three days. <a href="help.html#notifications">What is sent and when</a></p></div>
|
||||
<button type="submit" class="button is-primary">Save profile</button>
|
||||
</form>
|
||||
</div>
|
||||
|
||||
@@ -78,7 +78,7 @@
|
||||
<div class="box">
|
||||
<div class="sp-box-head"><span class="sp-no">01</span><h2>Full backup</h2><a class="sp-help" href="help.html#backup" title="Backups, encryption and restore"><i class="ti ti-help-circle"></i></a></div>
|
||||
<div class="sp-box-body">
|
||||
<p class="mb-3"><code>data/</code> (database, DKIM keys, credentials, the queue), <code>docker-compose.yml</code>, <code>.env</code> and <code>certs/</code> in one archive. Restore into an empty project directory and start the <strong>same SelfPost version</strong>. The reverse-proxy vhost is not included.</p>
|
||||
<p class="mb-3"><code>data/</code> (database, DKIM keys, credentials), <code>docker-compose.yml</code>, <code>.env</code> and <code>certs/</code> in one archive. The Postfix queue is left out: it cannot be copied reliably while the server is running. Restore into an empty project directory and start the <strong>same SelfPost version</strong>. The reverse-proxy vhost is not included.</p>
|
||||
<form class="sp-form" method="post" action="#">
|
||||
<div class="field"><label class="checkbox"><input type="checkbox" name="encrypt" checked> Encrypt with a password (<code>.spbk</code>)</label></div>
|
||||
<div class="sp-pair">
|
||||
|
||||
@@ -37,7 +37,7 @@ SIGNED_OUT = {"sp-signin", "sp-signin-brand", "sp-signin-form", "sp-foot"}
|
||||
# Selectors allowed to set a width. Everything else gets its width from the
|
||||
# layout it sits in — that is the whole point of the redesign.
|
||||
WIDTH_OK = (".navbar-brand img", ".sp-head .sp-lead", ".sp-head .sp-grow", ".sp-postmark", ".table td.sp-clip",
|
||||
".sp-h-icon", ".sp-signin", ".sp-timeline li::before", ".sp-perf")
|
||||
".sp-h-icon", ".sp-signin", ".sp-timeline li::before", ".sp-perf", ".sp-sheet")
|
||||
|
||||
|
||||
class Node:
|
||||
|
||||
@@ -114,6 +114,7 @@
|
||||
<tr><td>error, fail, mismatch, bounced</td><td><span class="tag is-danger is-light">fail</span></td><td class="sp-mono sp-small">tag is-danger is-light</td></tr>
|
||||
<tr><td>unknown, off, neutral counts</td><td><span class="tag is-light">off</span></td><td class="sp-mono sp-small">tag is-light</td></tr>
|
||||
<tr><td>held: quarantine · role: global</td><td><span class="tag is-primary is-light">quarantine</span></td><td class="sp-mono sp-small">tag is-primary is-light</td></tr>
|
||||
<tr><td>a tag that opens a <code>sheet</code></td><td><button type="button" class="tag is-danger is-light" popovertarget="kit-sheet">reject</button><div class="sp-sheet" id="kit-sheet" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-danger is-light">reject</span><h2>sheet</h2><button type="button" popovertarget="kit-sheet" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>What it holds</th><td>What does not fit the row, as name and value</td></tr><tr><th>How it closes</th><td class="sp-mono sp-small">Esc<br>a click outside<br>the cross</td></tr></tbody></table></div></div></div></div></td><td class="sp-mono sp-small">button.tag + sp-sheet (popover)</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</div>
|
||||
|
||||
@@ -80,7 +80,7 @@
|
||||
<div><dt>Last report</dt><dd>2026-09-21 06:12 UTC</dd></div>
|
||||
<div><dt>This week</dt><dd>14 kept · 0 parse failures</dd></div>
|
||||
<div><dt>Hosted addresses</dt><dd class="sp-mono sp-small">dmarc-reports+<domain>@mail.example.org</dd></div>
|
||||
<div><dt>Retention</dt><dd>500 reports or 90 days</dd></div>
|
||||
<div><dt>Retention</dt><dd>500 reports a domain or 90 days</dd></div>
|
||||
</dl>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -145,11 +145,30 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="box" id="configuration">
|
||||
<div class="sp-box-head"><span class="sp-no">06</span><h2>Configuration</h2><div class="sp-end">Safe to run at any time</div></div>
|
||||
<div class="sp-box-body">
|
||||
<p class="mb-3">Regenerates the OpenDKIM and Postfix configuration from the database and reloads both daemons. Use it after editing the files by hand, after restoring a backup, or when the running configuration looks out of step with the domain and application lists. It does not touch the queue or the TLS certificate.</p>
|
||||
<form method="post" action="#"><button type="submit" class="button"><i class="ti ti-reload"></i>Reload configuration</button></form>
|
||||
<div class="columns">
|
||||
<div class="column is-6">
|
||||
<div class="box" id="notification-mail">
|
||||
<div class="sp-box-head"><span class="sp-no">06</span><h2>Notification mail</h2><div class="sp-end"><span class="tag is-warning is-light">warn</span></div><a class="sp-help" href="help.html#notifications" title="What the panel sends and what makes it arrive"><i class="ti ti-help-circle"></i></a></div>
|
||||
<div class="table-container"><table class="table is-fullwidth">
|
||||
<thead><tr><th>Record</th><th>State</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>DKIM<br><span class="sp-muted sp-mono sp-small">sp2026._domainkey.mail.example.org</span></td><td><span class="tag is-success is-light">published</span></td></tr>
|
||||
<tr><td>SPF<br><span class="sp-muted sp-mono sp-small">mail.example.org</span></td><td><span class="tag is-warning is-light">missing</span></td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
<div class="sp-box-body">
|
||||
<p class="sp-muted">Sent from <span class="sp-mono">postmaster@mail.example.org</span>. What to publish is under <a href="settings.html#notification-mail">Settings</a>.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="column is-6">
|
||||
<div class="box" id="configuration">
|
||||
<div class="sp-box-head"><span class="sp-no">07</span><h2>Configuration</h2><div class="sp-end">Safe to run at any time</div></div>
|
||||
<div class="sp-box-body">
|
||||
<p class="mb-3">Regenerates the OpenDKIM and Postfix configuration from the database and reloads both daemons. Use it after editing the files by hand, after restoring a backup, or when the running configuration looks out of step with the domain and application lists. It does not touch the queue or the TLS certificate.</p>
|
||||
<form method="post" action="#"><button type="submit" class="button"><i class="ti ti-reload"></i>Reload configuration</button></form>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -97,12 +97,15 @@
|
||||
<li><a href="#upstream"><i class="ti ti-arrow-forward-up"></i>Upstream</a></li>
|
||||
<li><a href="#recipients"><i class="ti ti-address-book"></i>Valid recipients</a></li>
|
||||
<li><a href="#filter"><i class="ti ti-shield-check"></i>Spam filter</a></li>
|
||||
<li><a href="#inbound-log"><i class="ti ti-list-details"></i>Inbound log</a></li>
|
||||
<li><a href="#filter-lists"><i class="ti ti-filter"></i>Filter lists</a></li>
|
||||
</ul>
|
||||
<p class="menu-label">Server</p>
|
||||
<ul class="menu-list">
|
||||
<li><a href="#backup"><i class="ti ti-archive"></i>Backup and moving a domain</a></li>
|
||||
<li><a href="#users"><i class="ti ti-users"></i>Users and roles</a></li>
|
||||
<li><a href="#settings"><i class="ti ti-settings"></i>Settings</a></li>
|
||||
<li><a href="#notifications"><i class="ti ti-mail"></i>Notifications</a></li>
|
||||
</ul>
|
||||
<p class="menu-label">Account</p>
|
||||
<ul class="menu-list">
|
||||
@@ -122,7 +125,7 @@
|
||||
<div class="sp-help-topic" id="dns"><h3>DNS records</h3><p>DKIM proves the message was signed by this relay, SPF lists the addresses allowed to send for the domain, DMARC tells receivers what to do when both fail and where to report. Publish all three exactly as shown; merge SPF into an existing record rather than adding a second one. A domain's status is the worst of its three records, and results are cached for a few minutes, so use Re-check DNS after publishing. The SPF check is shallow: it looks for this server's address in the record itself and does not follow <code>include:</code> or <code>redirect=</code>, so a record that relies on them is reported as a warning, not a failure. <code>p=none</code> does not affect delivery; tighten to <code>quarantine</code>, then <code>reject</code>, once the reports look clean. A domain's report address is set in its Domain settings.</p></div>
|
||||
<div class="sp-help-topic" id="apps"><h3>Applications</h3><p>An application is one SASL login, unique across all domains. It may send as any address of its domain or only as listed addresses, and, with a list of client IPs set, only from those addresses. The password is shown once at creation or regeneration and is not stored in readable form.</p></div>
|
||||
<div class="sp-help-topic" id="limits"><h3>Rate limits</h3><p>Level 1 is a per-IP ceiling set in <code>.env</code>. Level 2 is optional, per domain and per application, entered by hand or computed from the last 30 days of sending, and can never exceed level 1. An application's own limit replaces its domain's.</p></div>
|
||||
<div class="sp-help-topic" id="log"><h3>Outbound log</h3><p>One row per message and recipient, with the status Postfix last reported: queued (accepted, nothing reported yet), sent (the receiving server took it; what the mailbox does next is not reported back), deferred (not taken yet, Postfix tries again), bounced (failed for good, or the queue lifetime ran out) and rejected (refused before queueing under a level-2 rate limit, so it never reached Postfix). A message's page lists these steps and the lines Postfix wrote about it in <code>mail.log</code>, found by queue id; that file is rotated daily and about two weeks are kept, so an old message may show none. Rows are deleted after the retention set in Server › Settings, 90 days by default. A domain administrator sees only the rows of their own domains.</p></div>
|
||||
<div class="sp-help-topic" id="log"><h3>Outbound log</h3><p>One row per message and recipient, with the status Postfix last reported: queued (accepted, nothing reported yet), sent (the receiving server took it; what the mailbox does next is not reported back), deferred (not taken yet, Postfix tries again), bounced (failed for good, or the queue lifetime ran out) and rejected (refused before queueing under a level-2 rate limit, so it never reached Postfix). A message's page lists these steps and the lines Postfix wrote about it in <code>mail.log</code>, found by queue id. The lines are kept with the row: they stay on the page after that file has rotated, for as long as the row does. A message accepted before version 2.2.0 shows them only while they are still in the current file. Rows are deleted after the retention set in Server › Settings, 90 days by default, and their lines with them. A domain administrator sees only the rows of their own domains.</p></div>
|
||||
<div class="sp-help-topic" id="queue"><h3>Queue and retries</h3><p>The page lists, as <code>postqueue -p</code> prints it, the messages Postfix has accepted and not yet delivered: mail being sent right now and mail deferred for a later attempt. Only the global role can open it, and it only shows; nothing here deletes or forces a message. A deferred message is tried again after the first retry delay, then at gaps that double up to a cap, until the receiving server takes it or the queue lifetime runs out and it bounces; there is no attempt counter, and the box on the page shows this server's values. A queue that keeps growing means a receiver is refusing or not answering: open the message from the Outbound log to read the reason Postfix logged.</p></div>
|
||||
<div class="sp-help-topic" id="dmarc"><h3>DMARC reports</h3><p>An aggregate report is the summary a receiving provider sends, usually daily, to the <code>rua=</code> address of your DMARC record: which servers sent mail as your domain and whether SPF or DKIM passed for it. A domain's report address is one of three choices, set in its Domain settings: no reports (the default for a new domain), the SelfPost hosted address (an address on this server whose reports are parsed and shown under DMARC reports; offered only while hosted reports are enabled on the server), or an address typed for the domain (an ordinary mailbox that receives the raw XML; a button fills in your own profile e-mail). When the typed address is on another domain than the one sending, that domain must publish a report-authorization record, or providers will not send; the domain's page shows it with the other DNS records and checks it. In the tables pass means SPF or DKIM passed and aligned, fail means neither did, and your own server shows as <em>this relay</em>. A fail from any other source is someone else sending as your domain — a forwarder, an application you forgot, or abuse — so do not tighten <code>p=</code> until that sender is fixed or removed. A domain administrator reads the reports of their own domains.</p></div>
|
||||
</div>
|
||||
@@ -132,16 +135,19 @@
|
||||
<div class="sp-help-topic" id="upstream"><h3>Upstream</h3><p>The upstream is the server that actually holds the mailboxes. Give its host and port — used exactly as written, with no MX lookup — and choose the TLS for the hop to it: Opportunistic uses TLS when the upstream offers it and otherwise sends in the clear, Required holds the mail until a TLS session is possible, Off never uses TLS. While the upstream does not answer, Postfix keeps the mail and retries as for any other mail.</p></div>
|
||||
<div class="sp-help-topic" id="recipients"><h3>Valid recipients</h3><p>Choose whom this domain accepts. Listed addresses only: the relay accepts exactly the addresses you list, one per line or comma-separated, and refuses any other while the sender is still connected, so it never has to send a bounce for them. Any recipient: everything at the domain is accepted and passed on, and an address the upstream then rejects comes back as a bounce from this server to a sender who is often forged — backscatter. Prefer a list, and keep it in step with the mailboxes upstream.</p></div>
|
||||
<div class="sp-help-topic" id="filter"><h3>Spam filter</h3><p>The spam filter belongs to the server, not to a domain. The server's administrator attaches it in the server's <code>.env</code>, and from then on every inbound domain is screened by it before mail is handed on; the box shows on or off and has nothing to switch. Off means mail goes to the upstream unfiltered. While the filter is down, mail is either accepted unfiltered (the default) or deferred, so the sending server tries again later, as the server's setting says; the box states which.</p></div>
|
||||
<div class="sp-help-topic" id="inbound-log"><h3>Inbound log</h3><p>One row per recipient of a message offered on port 25 for an inbound domain, with what was decided: <span class="tag is-success is-light">accept</span> — handed on to the upstream; <span class="tag is-danger is-light">reject</span> — refused for good; <span class="tag is-warning is-light">tempfail</span> — the sender was asked to come back later; <span class="tag is-primary is-light">quarantine</span> — held by the filter. <em>Why</em> names who decided. <code>rspamd</code> is the spam filter, with its action, the score against the reject threshold and the heaviest symbols; <code>postfix</code> is the relay's own rule — an unknown recipient, a sender domain that does not exist — with its reply; <code>selfpost</code> means nobody judged the message: it was relayed with no filter attached. Accept says what this server did, not what the upstream did with the message afterwards.<br><br>Accepted mail is recorded as it passes; refused mail is read from <code>mail.log</code>, so its row has no subject, and a message refused after its text was sent is one row however many recipients it had. A refusal to relay for a domain that is not an inbound one is every scanner on the internet and is not recorded. Rows are kept for the number of days set in Settings, and never more than 10 000.</p></div>
|
||||
<div class="sp-help-topic" id="filter-lists"><h3>Filter lists</h3><p>Two lists for the whole server that correct the spam filter by hand. An entry is a sender address, a sender domain — that domain, not its subdomains — or a client address or network in CIDR form. A sender on the Deny list is refused outright. A sender on the Allow list is given a weight no spam score outweighs, so the filter lets it through. The sender's address is whatever the sending server says it is - anyone can claim to be a sender you allow - so allow a client address or network where you can, and a sender address only for mail you would rather receive forged than lose. When entries of both lists match one message, <strong>Deny wins</strong>.<br><br>SelfPost does not apply the lists itself: it writes them as files under <code>/data/antispam/</code>, and rspamd rereads a file within seconds of a change. With the spam filter off the lists are kept and can be edited, but nothing reads them. The sender matched is the envelope sender — the one in the log — not the From line a mail program shows. Matched counts the messages in the Inbound log that the entry decided, so it reaches back as far as the log does; never means not once in that time, and such an entry is one to consider removing.</p></div>
|
||||
</div>
|
||||
<div class="box">
|
||||
<div class="sp-box-head"><span class="sp-no">04</span><h2>Server</h2></div>
|
||||
<div class="sp-help-topic" id="backup"><h3>Backup and moving a domain</h3><p>Full backup downloads one archive of the whole instance: <code>data/</code> (the database, DKIM keys, application credentials and the queue), <code>docker-compose.yml</code>, <code>.env</code> and <code>certs/</code>. It holds secrets, so leave Encrypt with a password ticked: the file is sealed as <code>.spbk</code>, the password is not stored, and without it nothing can be recovered. To restore, unpack it into an empty project directory (decrypt a <code>.spbk</code> first with <code>selfpost-backup -decrypt</code>) and start the same SelfPost version that made it; the panel refuses data from another version. The reverse-proxy configuration and <code>mail.log</code> are not included. One domain moves on its own: Export domain on its settings page writes a <code>.json</code> (or encrypted <code>.spde</code>) with its DKIM key and its applications' passwords, and Import a domain here reads it into another instance, so the published DNS needs no change.</p></div>
|
||||
<div class="sp-help-topic" id="backup"><h3>Backup and moving a domain</h3><p>Full backup downloads one archive of the whole instance: <code>data/</code> (the database, DKIM keys and application credentials), <code>docker-compose.yml</code>, <code>.env</code> and <code>certs/</code>. The Postfix queue is not in it. The backup is taken while the server is running, and Postfix keeps moving messages through the queue, so a copy could hold a message twice or miss it; mail still waiting for delivery is therefore not carried over, and a restored server starts with an empty queue. To keep the queue, stop the container and archive the project directory instead. The private key in <code>certs/</code> has to be readable by the panel: give it to group 998 on the host (<code>chgrp 998</code>, <code>chmod 640</code>). Until then the panel refuses to make a backup and names the file, rather than hand over an archive without the key. It holds secrets, so leave Encrypt with a password ticked: the file is sealed as <code>.spbk</code>, the password is not stored, and without it nothing can be recovered. To restore, unpack it into an empty project directory (decrypt a <code>.spbk</code> first with <code>selfpost-backup -decrypt</code>) and start the same SelfPost version that made it; the panel refuses data from another version. The reverse-proxy configuration and <code>mail.log</code> are not included. One domain moves on its own: Export domain on its settings page writes a <code>.json</code> (or encrypted <code>.spde</code>) with its DKIM key and its applications' passwords, and Import a domain here reads it into another instance, so the published DNS needs no change.</p></div>
|
||||
<div class="sp-help-topic" id="users"><h3>Users and roles</h3><p>The two roles are a reach, not a rank: every user is an administrator and signs in with a username and password. Global reaches the whole server — Overview, Server, the queue and every domain — and is the only role that adds or deletes domains and users. Domain reaches only the domains assigned to it, outbound and inbound separately (the same name on both lists is two assignments); <em>All</em> includes domains added later, and a domain user needs at least one domain or <em>All</em> on either list. The only global user cannot be demoted or deleted, and you cannot delete the account you are signed in with. A new password set here signs that user out of their other sessions.</p></div>
|
||||
<div class="sp-help-topic" id="settings"><h3>Settings</h3><p>Settings holds what is true of the whole instance. Log retention is how many days the Outbound log keeps its rows, from 7 to 365 and 90 by default; a sweep every six hours deletes older ones, and it is the main driver of how fast <code>/data</code> grows. It does not change the rotation of <code>mail.log</code>, which is daily with about two weeks kept. The sending rate limit shown under it is level 1, the per-client-IP ceiling set in <code>.env</code>: changing it needs a restart, and no level-2 limit can exceed it.</p></div>
|
||||
<div class="sp-help-topic" id="settings"><h3>Settings</h3><p>Settings holds what is true of the whole instance. Log retention is how many days the Outbound log keeps its rows, from 7 to 365 and 90 by default; a sweep every six hours deletes older ones, and it is the main driver of how fast <code>/data</code> grows. With the inbound relay on there is a second number, for the Inbound log, with the same range. It does not change the rotation of <code>mail.log</code>, which is daily with about two weeks kept. The sending rate limit shown under it is level 1, the per-client-IP ceiling set in <code>.env</code>: changing it needs a restart, and no level-2 limit can exceed it. Notifications are turned on or off here for everyone.</p></div>
|
||||
<div class="sp-help-topic" id="notifications"><h3>Notifications</h3><p>The panel writes to its users when something needs them, from <code>postmaster@</code> the server's hostname. Four things are reported. A sending domain's SPF, DKIM or DMARC record stops matching: sent once two hourly checks in a row have found it so, and once more when the record matches again. A level-2 rate limit refuses mail: sent at the first refusal, then at most once a day for the same limit. The TLS certificate is 14 days from expiry, and again at 3 days if it is still the same certificate. A message has waited in the queue for more than three days, with one more message when none is left. A global user is told all of it; a domain administrator is told about the DNS records and limits of their own domains.<br><br>Each user turns notifications on or off under Account and needs an e-mail there; Server › Settings turns them off for everyone. They are the panel's own mail: not rows of the Outbound log, and not counted against any limit.<br><br>For that mail to arrive, the server's own name needs two DNS records of its own, as each of your domains does: a DKIM key, which the panel generates and signs notifications with, and an SPF record, <code>v=spf1 a -all</code>. Server › Settings gives both with the values to publish, and Server › Health shows whether they are in place. Without them notifications are still sent, and a receiver may file them as spam. Mail of your domains does not depend on either.</p></div>
|
||||
</div>
|
||||
<div class="box">
|
||||
<div class="sp-box-head"><span class="sp-no">05</span><h2>Account</h2></div>
|
||||
<div class="sp-help-topic" id="account"><h3>Your account</h3><p>Account is your own, whatever your role: your username and e-mail, and your password. Changing the password signs out every other session of this account and keeps the one you are using; applications have their own logins and are not affected.</p></div>
|
||||
<div class="sp-help-topic" id="account"><h3>Your account</h3><p>Account is your own, whatever your role: your username and e-mail, whether the panel sends you notifications, and your password. Changing the password signs out every other session of this account and keeps the one you are using; applications have their own logins and are not affected.</p></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -74,7 +74,7 @@
|
||||
</div>
|
||||
|
||||
<div class="box">
|
||||
<div class="sp-box-head"><span class="sp-no">01</span><h2>Add an entry</h2><a class="sp-help" href="help.html#inbound" title="How the lists are applied"><i class="ti ti-help-circle"></i></a></div>
|
||||
<div class="sp-box-head"><span class="sp-no">01</span><h2>Add an entry</h2><a class="sp-help" href="help.html#filter-lists" title="How the lists are applied"><i class="ti ti-help-circle"></i></a></div>
|
||||
<div class="sp-box-body">
|
||||
<form class="sp-form" method="post" action="#">
|
||||
<div class="columns">
|
||||
@@ -84,7 +84,7 @@
|
||||
</div>
|
||||
<div class="column is-3">
|
||||
<div class="field"><label class="label" for="etype">Entry is</label>
|
||||
<div class="control"><div class="select is-fullwidth"><select id="etype" name="entry_type"><option>a sender address</option><option>a sender domain</option><option>a client IP or CIDR</option></select></div></div></div>
|
||||
<div class="control"><div class="select is-fullwidth"><select id="etype" name="entry_type"><option>a sender address</option><option>a sender domain</option><option>a client address or network</option></select></div></div></div>
|
||||
</div>
|
||||
<div class="column is-6">
|
||||
<div class="field"><label class="label" for="value">Value</label>
|
||||
@@ -103,11 +103,11 @@
|
||||
<div class="box">
|
||||
<div class="sp-box-head"><span class="sp-no">02</span><h2>Deny</h2><div class="sp-end">3 entries · wins over Allow</div></div>
|
||||
<div class="table-container"><table class="table is-fullwidth is-hoverable">
|
||||
<thead><tr><th>Value</th><th>Note</th><th>Added</th><th></th></tr></thead>
|
||||
<thead><tr><th>Value</th><th>Kind</th><th>Matched</th><th></th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td class="sp-mono sp-small">@bulk.example <span class="tag is-light">domain</span></td><td class="sp-muted">newsletter nobody subscribed to</td><td class="sp-muted sp-nowrap sp-small">2026-09-02</td><td class="sp-actions"><a class="has-text-danger" href="#">Remove</a></td></tr>
|
||||
<tr><td class="sp-mono sp-small">198.51.100.0/24 <span class="tag is-light">ip</span></td><td class="sp-muted">dictionary attack on lists.example.org</td><td class="sp-muted sp-nowrap sp-small">2026-08-19</td><td class="sp-actions"><a class="has-text-danger" href="#">Remove</a></td></tr>
|
||||
<tr><td class="sp-mono sp-small">win@lottery.example <span class="tag is-light">address</span></td><td class="sp-muted">—</td><td class="sp-muted sp-nowrap sp-small">2026-07-30</td><td class="sp-actions"><a class="has-text-danger" href="#">Remove</a></td></tr>
|
||||
<tr><td class="sp-mono sp-small">bulk.example</td><td><button type="button" class="tag is-light" popovertarget="deny-1">domain</button><div class="sp-sheet" id="deny-1" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-light">domain</span><h2>bulk.example</h2><button type="button" popovertarget="deny-1" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Note</th><td>newsletter nobody subscribed to</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-09-02</td></tr><tr><th>Matched</th><td>41 in the last 90 days</td></tr><tr><th>Last match</th><td class="sp-mono sp-small">2026-09-21 13:57</td></tr></tbody></table></div></div></div></div></td><td class="sp-nowrap">41</td><td class="sp-actions"><form method="post" action="#"><button type="submit" class="has-text-danger">Remove</button></form></td></tr>
|
||||
<tr><td class="sp-mono sp-small">198.51.100.0/24</td><td><button type="button" class="tag is-light" popovertarget="deny-2">ip</button><div class="sp-sheet" id="deny-2" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-light">ip</span><h2>198.51.100.0/24</h2><button type="button" popovertarget="deny-2" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Note</th><td>dictionary attack on lists.example.org</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-08-19</td></tr><tr><th>Matched</th><td>1203 in the last 90 days</td></tr><tr><th>Last match</th><td class="sp-mono sp-small">2026-09-21 09:12</td></tr></tbody></table></div></div></div></div></td><td class="sp-nowrap">1203</td><td class="sp-actions"><form method="post" action="#"><button type="submit" class="has-text-danger">Remove</button></form></td></tr>
|
||||
<tr><td class="sp-mono sp-small">win@lottery.example</td><td><button type="button" class="tag is-light" popovertarget="deny-3">address</button><div class="sp-sheet" id="deny-3" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-light">address</span><h2>win@lottery.example</h2><button type="button" popovertarget="deny-3" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Note</th><td>—</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-07-30</td></tr><tr><th>Matched</th><td>never in the last 90 days</td></tr></tbody></table></div></div></div></div></td><td class="sp-muted sp-nowrap">never</td><td class="sp-actions"><form method="post" action="#"><button type="submit" class="has-text-danger">Remove</button></form></td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</div>
|
||||
@@ -116,10 +116,10 @@
|
||||
<div class="box">
|
||||
<div class="sp-box-head"><span class="sp-no">03</span><h2>Allow</h2><div class="sp-end">2 entries</div></div>
|
||||
<div class="table-container"><table class="table is-fullwidth is-hoverable">
|
||||
<thead><tr><th>Value</th><th>Note</th><th>Added</th><th></th></tr></thead>
|
||||
<thead><tr><th>Value</th><th>Kind</th><th>Matched</th><th></th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td class="sp-mono sp-small">@partner.example <span class="tag is-light">domain</span></td><td class="sp-muted">invoices were scored as spam</td><td class="sp-muted sp-nowrap sp-small">2026-09-11</td><td class="sp-actions"><a class="has-text-danger" href="#">Remove</a></td></tr>
|
||||
<tr><td class="sp-mono sp-small">noreply@github.example <span class="tag is-light">address</span></td><td class="sp-muted">security advisories</td><td class="sp-muted sp-nowrap sp-small">2026-06-04</td><td class="sp-actions"><a class="has-text-danger" href="#">Remove</a></td></tr>
|
||||
<tr><td class="sp-mono sp-small">partner.example</td><td><button type="button" class="tag is-light" popovertarget="allow-1">domain</button><div class="sp-sheet" id="allow-1" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-light">domain</span><h2>partner.example</h2><button type="button" popovertarget="allow-1" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Note</th><td>invoices were scored as spam</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-09-11</td></tr><tr><th>Matched</th><td>7 in the last 90 days</td></tr><tr><th>Last match</th><td class="sp-mono sp-small">2026-09-19 16:40</td></tr></tbody></table></div></div></div></div></td><td class="sp-nowrap">7</td><td class="sp-actions"><form method="post" action="#"><button type="submit" class="has-text-danger">Remove</button></form></td></tr>
|
||||
<tr><td class="sp-mono sp-small">noreply@github.example</td><td><button type="button" class="tag is-light" popovertarget="allow-2">address</button><div class="sp-sheet" id="allow-2" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-light">address</span><h2>noreply@github.example</h2><button type="button" popovertarget="allow-2" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Note</th><td>security advisories</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-06-04</td></tr><tr><th>Matched</th><td>18 in the last 90 days</td></tr><tr><th>Last match</th><td class="sp-mono sp-small">2026-09-21 13:30</td></tr></tbody></table></div></div></div></div></td><td class="sp-nowrap">18</td><td class="sp-actions"><form method="post" action="#"><button type="submit" class="has-text-danger">Remove</button></form></td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</div>
|
||||
@@ -130,10 +130,10 @@
|
||||
<div class="sp-box-head"><span class="sp-no">04</span><h2>Where the lists go</h2><div class="sp-end"><span class="tag is-success is-light">applied</span></div></div>
|
||||
<div class="sp-box-body">
|
||||
<dl class="sp-facts">
|
||||
<div><dt>Applied by</dt><dd>rspamd, from map files</dd></div>
|
||||
<div><dt>Files</dt><dd class="sp-mono sp-small">/data/antispam/allow.map<br>/data/antispam/deny.map</dd></div>
|
||||
<div><dt>Applied by</dt><dd>rspamd, from map files it rereads within seconds of a change</dd></div>
|
||||
<div><dt>Files</dt><dd><span class="sp-mono">/data/antispam/</span> — six, one for each list and kind of entry</dd></div>
|
||||
<div><dt>Last written</dt><dd>2026-09-11 08:40 UTC</dd></div>
|
||||
<div><dt>When the filter is off</dt><dd class="sp-muted">The lists are kept and editable, but nothing reads them until <code>INBOUND_ANTISPAM_MILTER</code> is set.</dd></div>
|
||||
<div><dt>When both lists match</dt><dd>Deny wins, whichever entries they are.</dd></div>
|
||||
</dl>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -69,7 +69,7 @@
|
||||
<div class="sp-grow">
|
||||
<p class="sp-kicker">Inbound</p>
|
||||
<h1 class="title is-3">Inbound log</h1>
|
||||
<p class="sp-lead">What port 25 decided about each message, one row per recipient. Accepts come from the inbound journal, rejects from <code>mail.log</code>; the filter's reasons appear when the spam filter is on.</p>
|
||||
<p class="sp-lead">What port 25 decided about each message, one row per recipient. Accepted mail is recorded as it passes, refused mail is read from <code>mail.log</code>; the filter's verdict appears when the spam filter is on. Rows older than 90 days are deleted — change that in <a href="settings.html">Settings</a>.</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -83,19 +83,20 @@
|
||||
<input class="input" type="date" name="to" value="2026-09-21" aria-label="To date">
|
||||
<button type="submit" class="button"><i class="ti ti-filter"></i>Filter</button>
|
||||
</form>
|
||||
<a class="sp-help" href="help.html#inbound-log" title="What the decisions mean"><i class="ti ti-help-circle"></i></a>
|
||||
</div>
|
||||
<div class="table-container"><table class="table is-fullwidth is-hoverable">
|
||||
<thead><tr><th>Time, UTC</th><th>From · client</th><th>To</th><th>Subject</th><th>Decision</th><th>Why</th><th></th></tr></thead>
|
||||
<thead><tr><th>Time, UTC</th><th>From</th><th>To</th><th>Subject</th><th>Decision</th><th></th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 14:00:18</td><td>t.berg@<wbr>uni.example<br><span class="sp-mono sp-muted sp-small">192.0.2.77</span></td><td>dev@<wbr>lists.example.org</td><td class="sp-clip">Re: [dev] patch series v3</td><td><span class="tag is-success is-light">accept</span></td><td class="sp-muted sp-small">rspamd · score 0.4</td><td class="sp-actions"><a href="in-filter-lists.html">Deny sender</a></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:59:02</td><td>invoice@<wbr>partner-billing.example<br><span class="sp-mono sp-muted sp-small">203.0.113.140</span></td><td>owner@<wbr>lists.example.org</td><td class="sp-clip">Outstanding invoice — action required</td><td><span class="tag is-primary is-light">quarantine</span></td><td class="sp-muted sp-small">rspamd · score 9.8 · PHISHING, FORGED_SENDER</td><td class="sp-actions"><a href="in-quarantine-message.html">Review</a></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:57:41</td><td>promo@<wbr>bulk.example<br><span class="sp-mono sp-muted sp-small">198.51.100.90</span></td><td>sales@<wbr>lists.example.org</td><td class="sp-clip sp-muted">—</td><td><span class="tag is-danger is-light">reject</span></td><td class="sp-muted sp-small">postfix · 550 5.1.1 unknown recipient</td><td class="sp-actions"><a href="in-filter-lists.html">Deny sender</a></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:51:30</td><td>billing@<wbr>partner.example<br><span class="sp-mono sp-muted sp-small">198.51.100.12</span></td><td>owner@<wbr>lists.example.org</td><td class="sp-clip">Invoice 2026-0917</td><td><span class="tag is-danger is-light">reject</span></td><td class="sp-muted sp-small">rspamd · score 16.2 · BAYES_SPAM</td><td class="sp-actions"><a href="in-filter-lists.html">Allow sender</a></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:44:02</td><td>hr@<wbr>partner.example<br><span class="sp-mono sp-muted sp-small">192.0.2.31</span></td><td>jobs@<wbr>acme.io</td><td class="sp-clip">Application: backend engineer</td><td><span class="tag is-warning is-light">tempfail</span></td><td class="sp-muted sp-small">selfpost · upstream 10.0.4.12 timed out</td><td class="sp-actions"></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:30:55</td><td>noreply@<wbr>github.example<br><span class="sp-mono sp-muted sp-small">192.0.2.200</span></td><td>security@<wbr>lists.example.org</td><td class="sp-clip">[advisory] new report</td><td><span class="tag is-success is-light">accept</span></td><td class="sp-muted sp-small">rspamd · allow list</td><td class="sp-actions"></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 14:00:18</td><td>t.berg@<wbr>uni.example<br><span class="sp-mono sp-muted sp-small">192.0.2.77</span></td><td>dev@<wbr>lists.example.org</td><td class="sp-clip">Re: [dev] patch series v3</td><td><button type="button" class="tag is-success is-light" popovertarget="why-1">accept</button><div class="sp-sheet" id="why-1" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-success is-light">accept</span><h2>Why</h2><button type="button" popovertarget="why-1" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>rspamd</td></tr><tr><th>Verdict</th><td>no action, score 0.40 / 15.00</td></tr><tr><th>Subject</th><td>Re: [dev] patch series v3</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"><a href="in-filter-lists.html">Deny sender</a></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:59:02</td><td>invoice@<wbr>partner-billing.example<br><span class="sp-mono sp-muted sp-small">203.0.113.140</span></td><td>owner@<wbr>lists.example.org</td><td class="sp-clip">Outstanding invoice — action required</td><td><button type="button" class="tag is-success is-light" popovertarget="why-2">accept</button><div class="sp-sheet" id="why-2" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-success is-light">accept</span><h2>Why</h2><button type="button" popovertarget="why-2" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>rspamd</td></tr><tr><th>Verdict</th><td>add header, score 9.80 / 15.00</td></tr><tr><th>Symbols</th><td class="sp-mono sp-small">PHISHING(5.00)<br>FORGED_SENDER(3.00)<br>R_SPF_FAIL(1.00)</td></tr><tr><th>Subject</th><td>Outstanding invoice — action required</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"><a href="in-filter-lists.html">Deny sender</a></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:57:41</td><td>promo@<wbr>bulk.example<br><span class="sp-mono sp-muted sp-small">198.51.100.90</span></td><td>sales@<wbr>lists.example.org</td><td class="sp-clip sp-muted">—</td><td><button type="button" class="tag is-danger is-light" popovertarget="why-3">reject</button><div class="sp-sheet" id="why-3" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-danger is-light">reject</span><h2>Why</h2><button type="button" popovertarget="why-3" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>postfix</td></tr><tr><th>Reply</th><td class="sp-mono sp-small">550 5.1.1 <sales@lists.example.org>: Recipient address rejected: User unknown in relay recipient table</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:51:30</td><td>billing@<wbr>partner.example<br><span class="sp-mono sp-muted sp-small">198.51.100.12</span></td><td>owner@<wbr>lists.example.org</td><td class="sp-clip sp-muted">—</td><td><button type="button" class="tag is-danger is-light" popovertarget="why-4">reject</button><div class="sp-sheet" id="why-4" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-danger is-light">reject</span><h2>Why</h2><button type="button" popovertarget="why-4" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>rspamd</td></tr><tr><th>Reply</th><td class="sp-mono sp-small">5.7.1 Spam message rejected</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"><a href="in-filter-lists.html">Allow sender</a></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:44:02</td><td>hr@<wbr>partner.example<br><span class="sp-mono sp-muted sp-small">192.0.2.31</span></td><td>jobs@<wbr>acme.io</td><td class="sp-clip sp-muted">—</td><td><button type="button" class="tag is-warning is-light" popovertarget="why-5">tempfail</button><div class="sp-sheet" id="why-5" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-warning is-light">tempfail</span><h2>Why</h2><button type="button" popovertarget="why-5" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>rspamd</td></tr><tr><th>Reply</th><td class="sp-mono sp-small">4.7.1 Try again later</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"><a href="in-filter-lists.html">Allow sender</a></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:30:55</td><td>noreply@<wbr>github.example<br><span class="sp-mono sp-muted sp-small">192.0.2.200</span></td><td>security@<wbr>lists.example.org</td><td class="sp-clip">[advisory] new report</td><td><button type="button" class="tag is-success is-light" popovertarget="why-6">accept</button><div class="sp-sheet" id="why-6" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-success is-light">accept</span><h2>Why</h2><button type="button" popovertarget="why-6" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>rspamd</td></tr><tr><th>Verdict</th><td>no action, score -98.10 / 15.00</td></tr><tr><th>Symbols</th><td class="sp-mono sp-small">SELFPOST_ALLOW_FROM(-100.00)<br>R_DKIM_ALLOW(-0.20)</td></tr><tr><th>Subject</th><td>[advisory] new report</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"><a href="in-filter-lists.html">Deny sender</a></td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
<div class="sp-box-foot"><span>Page 1 of 38</span><a href="#">Older →</a><span class="sp-end">Allow / Deny sender opens <a href="in-filter-lists.html">Filter lists</a> with the entry filled in · global administrator only</span></div>
|
||||
<div class="sp-box-foot"><span>Page 1 of 38</span><a href="#">Older →</a><span class="sp-end">A decision opens its reason · Allow / Deny sender opens <a href="in-filter-lists.html">Filter lists</a> with the entry filled in</span></div>
|
||||
</div>
|
||||
|
||||
<footer class="sp-foot">SelfPost 1.8.0 · © 2026 Mix · <a href="#">License (AGPL-3.0)</a> · <a href="#">Source</a> · No warranty</footer>
|
||||
|
||||
@@ -108,7 +108,20 @@
|
||||
}
|
||||
]
|
||||
},
|
||||
"box[Configuration]:body"
|
||||
{
|
||||
"columns": [
|
||||
{
|
||||
"is-6": [
|
||||
"box[Notification mail]:table+body"
|
||||
]
|
||||
},
|
||||
{
|
||||
"is-6": [
|
||||
"box[Configuration]:body"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"help": [
|
||||
"head",
|
||||
@@ -123,8 +136,8 @@
|
||||
"is-9": [
|
||||
"box[Overview]:help×3",
|
||||
"box[Outbound]:help×6",
|
||||
"box[Inbound]:help×4",
|
||||
"box[Server]:help×3",
|
||||
"box[Inbound]:help×6",
|
||||
"box[Server]:help×4",
|
||||
"box[Account]:help"
|
||||
]
|
||||
}
|
||||
@@ -406,12 +419,14 @@
|
||||
"columns": [
|
||||
{
|
||||
"is-6": [
|
||||
"box[Log retention]:body"
|
||||
"box[Log retention]:body",
|
||||
"box[Sending rate limits]:body"
|
||||
]
|
||||
},
|
||||
{
|
||||
"is-6": [
|
||||
"box[Sending rate limits]:body"
|
||||
"box[Notifications]:body",
|
||||
"box[Notification mail]:record×2"
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
@@ -82,11 +82,12 @@
|
||||
<div class="field"><label class="label" for="ret">Keep outbound log rows, days</label>
|
||||
<div class="control"><input class="input" id="ret" name="send_log_retention_days" type="number" min="7" max="365" value="30" required></div>
|
||||
<p class="help">The main driver of <code>/data</code> growth. Does not affect <code>mail.log</code> rotation (fourteen daily files).</p></div>
|
||||
<div class="field"><label class="label" for="inret">Keep inbound log rows, days</label>
|
||||
<div class="control"><input class="input" id="inret" name="inbound_spam_log_retention_days" type="number" min="7" max="365" value="90" required></div>
|
||||
<p class="help">The inbound log also never holds more than 10 000 rows: refused mail is logged at the rate strangers knock on port 25.</p></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="column is-6">
|
||||
<div class="box" id="rate-limits">
|
||||
<div class="sp-box-head"><span class="sp-no">02</span><h2>Sending rate limits</h2><div class="sp-end">Read-only</div><a class="sp-help" href="help.html#limits" title="Levels 1 and 2"><i class="ti ti-help-circle"></i></a></div>
|
||||
<div class="sp-box-body">
|
||||
@@ -97,6 +98,43 @@
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="column is-6">
|
||||
<div class="box" id="notifications">
|
||||
<div class="sp-box-head"><span class="sp-no">03</span><h2>Notifications</h2><a class="sp-help" href="help.html#notifications" title="What is sent and when"><i class="ti ti-help-circle"></i></a></div>
|
||||
<div class="sp-box-body">
|
||||
<div class="field"><label class="checkbox"><input type="checkbox" name="notify" checked> Send notification e-mails</label>
|
||||
<p class="help">From <span class="sp-mono">postmaster@mail.example.org</span>, to every user who has an e-mail and has not turned them off under <a href="account.html">Account</a>.</p></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="box" id="notification-mail">
|
||||
<div class="sp-box-head"><span class="sp-no">04</span><h2>Notification mail</h2><div class="sp-end">DNS records of the server's own name</div><a class="sp-help" href="help.html#notifications" title="What the panel sends and what makes it arrive"><i class="ti ti-help-circle"></i></a></div>
|
||||
|
||||
<div class="sp-record">
|
||||
<div class="sp-record-head"><h3>DKIM</h3><span class="tag is-success is-light">published</span><small>Not a secret · selector <code>sp2026</code></small></div>
|
||||
<div class="sp-row">
|
||||
<div class="field"><label class="label">Host / name</label>
|
||||
<div class="field has-addons"><div class="control is-expanded"><input class="input" readonly value="sp2026._domainkey.mail.example.org"></div><div class="control"><button type="button" class="button"><i class="ti ti-copy"></i>Copy</button></div></div></div>
|
||||
<div class="field"><label class="label">Type</label><div class="control"><input class="input" readonly value="TXT"></div></div>
|
||||
</div>
|
||||
<div class="field"><label class="label">Value to publish</label>
|
||||
<div class="field has-addons"><div class="control is-expanded"><textarea class="textarea" rows="5" readonly>v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAq7n2Lw0c5vXeR1dYt8uJb3mKs6pHf9aZoWg4iNy2TqCx0lVjE7rUh5sBk1dMa8fPz3oGc6yXw9tIe4nQv2uL…IDAQAB</textarea></div><div class="control"><button type="button" class="button"><i class="ti ti-copy"></i>Copy</button></div></div></div>
|
||||
</div>
|
||||
|
||||
<div class="sp-record">
|
||||
<div class="sp-record-head"><h3>SPF</h3><span class="tag is-warning is-light">missing</span><small>For the server's own name, not for your domains</small></div>
|
||||
<div class="sp-row">
|
||||
<div class="field"><label class="label">Host / name</label>
|
||||
<div class="field has-addons"><div class="control is-expanded"><input class="input" readonly value="mail.example.org"></div><div class="control"><button type="button" class="button"><i class="ti ti-copy"></i>Copy</button></div></div></div>
|
||||
<div class="field"><label class="label">Type</label><div class="control"><input class="input" readonly value="TXT"></div></div>
|
||||
</div>
|
||||
<div class="field"><label class="label">Value to publish</label>
|
||||
<div class="field has-addons"><div class="control is-expanded"><input class="input" readonly value="v=spf1 a -all"></div><div class="control"><button type="button" class="button"><i class="ti ti-copy"></i>Copy</button></div></div></div>
|
||||
<p class="help">Notifications are sent without it, and a receiver may file them as spam.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="buttons"><button type="submit" class="button is-primary">Save settings</button></div>
|
||||
</form>
|
||||
|
||||
@@ -16,6 +16,8 @@
|
||||
<div class="field"><label class="label" for="username">Username</label><div class="control has-icons-left"><input class="input" id="username" name="username" value="admin" autocomplete="username" spellcheck="false" required><span class="icon is-left"><i class="ti ti-user"></i></span></div></div>
|
||||
<div class="field"><label class="label" for="email">E-mail</label><div class="control has-icons-left"><input class="input" id="email" name="email" type="email" value="mix@example.org" autocomplete="email"><span class="icon is-left"><i class="ti ti-mail"></i></span></div>
|
||||
<p class="help">Your own address. A domain's report address can be filled with it from Domain settings.</p></div>
|
||||
<div class="field"><label class="checkbox"><input type="checkbox" name="notify" checked> Send me notifications</label>
|
||||
<p class="help">To this address, when something needs you: a DNS record of a domain stops matching, a rate limit refuses mail, the TLS certificate is about to expire, a message has waited in the queue for three days. <a href="help.html#notifications">What is sent and when</a></p></div>
|
||||
<button type="submit" class="button is-primary">Save profile</button>
|
||||
</form>
|
||||
</div>
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
<div class="box">
|
||||
<div class="sp-box-head"><span class="sp-no">01</span><h2>Full backup</h2><a class="sp-help" href="help.html#backup" title="Backups, encryption and restore"><i class="ti ti-help-circle"></i></a></div>
|
||||
<div class="sp-box-body">
|
||||
<p class="mb-3"><code>data/</code> (database, DKIM keys, credentials, the queue), <code>docker-compose.yml</code>, <code>.env</code> and <code>certs/</code> in one archive. Restore into an empty project directory and start the <strong>same SelfPost version</strong>. The reverse-proxy vhost is not included.</p>
|
||||
<p class="mb-3"><code>data/</code> (database, DKIM keys, credentials), <code>docker-compose.yml</code>, <code>.env</code> and <code>certs/</code> in one archive. The Postfix queue is left out: it cannot be copied reliably while the server is running. Restore into an empty project directory and start the <strong>same SelfPost version</strong>. The reverse-proxy vhost is not included.</p>
|
||||
<form class="sp-form" method="post" action="#">
|
||||
<div class="field"><label class="checkbox"><input type="checkbox" name="encrypt" checked> Encrypt with a password (<code>.spbk</code>)</label></div>
|
||||
<div class="sp-pair">
|
||||
|
||||
@@ -48,6 +48,7 @@
|
||||
<tr><td>error, fail, mismatch, bounced</td><td><span class="tag is-danger is-light">fail</span></td><td class="sp-mono sp-small">tag is-danger is-light</td></tr>
|
||||
<tr><td>unknown, off, neutral counts</td><td><span class="tag is-light">off</span></td><td class="sp-mono sp-small">tag is-light</td></tr>
|
||||
<tr><td>held: quarantine · role: global</td><td><span class="tag is-primary is-light">quarantine</span></td><td class="sp-mono sp-small">tag is-primary is-light</td></tr>
|
||||
<tr><td>a tag that opens a <code>sheet</code></td><td><button type="button" class="tag is-danger is-light" popovertarget="kit-sheet">reject</button><div class="sp-sheet" id="kit-sheet" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-danger is-light">reject</span><h2>sheet</h2><button type="button" popovertarget="kit-sheet" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>What it holds</th><td>What does not fit the row, as name and value</td></tr><tr><th>How it closes</th><td class="sp-mono sp-small">Esc<br>a click outside<br>the cross</td></tr></tbody></table></div></div></div></div></td><td class="sp-mono sp-small">button.tag + sp-sheet (popover)</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</div>
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
<div><dt>Last report</dt><dd>2026-09-21 06:12 UTC</dd></div>
|
||||
<div><dt>This week</dt><dd>14 kept · 0 parse failures</dd></div>
|
||||
<div><dt>Hosted addresses</dt><dd class="sp-mono sp-small">dmarc-reports+<domain>@mail.example.org</dd></div>
|
||||
<div><dt>Retention</dt><dd>500 reports or 90 days</dd></div>
|
||||
<div><dt>Retention</dt><dd>500 reports a domain or 90 days</dd></div>
|
||||
</dl>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -79,10 +79,29 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="box" id="configuration">
|
||||
<div class="sp-box-head"><span class="sp-no">06</span><h2>Configuration</h2><div class="sp-end">Safe to run at any time</div></div>
|
||||
<div class="sp-box-body">
|
||||
<p class="mb-3">Regenerates the OpenDKIM and Postfix configuration from the database and reloads both daemons. Use it after editing the files by hand, after restoring a backup, or when the running configuration looks out of step with the domain and application lists. It does not touch the queue or the TLS certificate.</p>
|
||||
<form method="post" action="#"><button type="submit" class="button"><i class="ti ti-reload"></i>Reload configuration</button></form>
|
||||
<div class="columns">
|
||||
<div class="column is-6">
|
||||
<div class="box" id="notification-mail">
|
||||
<div class="sp-box-head"><span class="sp-no">06</span><h2>Notification mail</h2><div class="sp-end"><span class="tag is-warning is-light">warn</span></div><a class="sp-help" href="help.html#notifications" title="What the panel sends and what makes it arrive"><i class="ti ti-help-circle"></i></a></div>
|
||||
<div class="table-container"><table class="table is-fullwidth">
|
||||
<thead><tr><th>Record</th><th>State</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>DKIM<br><span class="sp-muted sp-mono sp-small">sp2026._domainkey.mail.example.org</span></td><td><span class="tag is-success is-light">published</span></td></tr>
|
||||
<tr><td>SPF<br><span class="sp-muted sp-mono sp-small">mail.example.org</span></td><td><span class="tag is-warning is-light">missing</span></td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
<div class="sp-box-body">
|
||||
<p class="sp-muted">Sent from <span class="sp-mono">postmaster@mail.example.org</span>. What to publish is under <a href="settings.html#notification-mail">Settings</a>.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="column is-6">
|
||||
<div class="box" id="configuration">
|
||||
<div class="sp-box-head"><span class="sp-no">07</span><h2>Configuration</h2><div class="sp-end">Safe to run at any time</div></div>
|
||||
<div class="sp-box-body">
|
||||
<p class="mb-3">Regenerates the OpenDKIM and Postfix configuration from the database and reloads both daemons. Use it after editing the files by hand, after restoring a backup, or when the running configuration looks out of step with the domain and application lists. It does not touch the queue or the TLS certificate.</p>
|
||||
<form method="post" action="#"><button type="submit" class="button"><i class="ti ti-reload"></i>Reload configuration</button></form>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -31,12 +31,15 @@
|
||||
<li><a href="#upstream"><i class="ti ti-arrow-forward-up"></i>Upstream</a></li>
|
||||
<li><a href="#recipients"><i class="ti ti-address-book"></i>Valid recipients</a></li>
|
||||
<li><a href="#filter"><i class="ti ti-shield-check"></i>Spam filter</a></li>
|
||||
<li><a href="#inbound-log"><i class="ti ti-list-details"></i>Inbound log</a></li>
|
||||
<li><a href="#filter-lists"><i class="ti ti-filter"></i>Filter lists</a></li>
|
||||
</ul>
|
||||
<p class="menu-label">Server</p>
|
||||
<ul class="menu-list">
|
||||
<li><a href="#backup"><i class="ti ti-archive"></i>Backup and moving a domain</a></li>
|
||||
<li><a href="#users"><i class="ti ti-users"></i>Users and roles</a></li>
|
||||
<li><a href="#settings"><i class="ti ti-settings"></i>Settings</a></li>
|
||||
<li><a href="#notifications"><i class="ti ti-mail"></i>Notifications</a></li>
|
||||
</ul>
|
||||
<p class="menu-label">Account</p>
|
||||
<ul class="menu-list">
|
||||
@@ -56,7 +59,7 @@
|
||||
<div class="sp-help-topic" id="dns"><h3>DNS records</h3><p>DKIM proves the message was signed by this relay, SPF lists the addresses allowed to send for the domain, DMARC tells receivers what to do when both fail and where to report. Publish all three exactly as shown; merge SPF into an existing record rather than adding a second one. A domain's status is the worst of its three records, and results are cached for a few minutes, so use Re-check DNS after publishing. The SPF check is shallow: it looks for this server's address in the record itself and does not follow <code>include:</code> or <code>redirect=</code>, so a record that relies on them is reported as a warning, not a failure. <code>p=none</code> does not affect delivery; tighten to <code>quarantine</code>, then <code>reject</code>, once the reports look clean. A domain's report address is set in its Domain settings.</p></div>
|
||||
<div class="sp-help-topic" id="apps"><h3>Applications</h3><p>An application is one SASL login, unique across all domains. It may send as any address of its domain or only as listed addresses, and, with a list of client IPs set, only from those addresses. The password is shown once at creation or regeneration and is not stored in readable form.</p></div>
|
||||
<div class="sp-help-topic" id="limits"><h3>Rate limits</h3><p>Level 1 is a per-IP ceiling set in <code>.env</code>. Level 2 is optional, per domain and per application, entered by hand or computed from the last 30 days of sending, and can never exceed level 1. An application's own limit replaces its domain's.</p></div>
|
||||
<div class="sp-help-topic" id="log"><h3>Outbound log</h3><p>One row per message and recipient, with the status Postfix last reported: queued (accepted, nothing reported yet), sent (the receiving server took it; what the mailbox does next is not reported back), deferred (not taken yet, Postfix tries again), bounced (failed for good, or the queue lifetime ran out) and rejected (refused before queueing under a level-2 rate limit, so it never reached Postfix). A message's page lists these steps and the lines Postfix wrote about it in <code>mail.log</code>, found by queue id; that file is rotated daily and about two weeks are kept, so an old message may show none. Rows are deleted after the retention set in Server › Settings, 90 days by default. A domain administrator sees only the rows of their own domains.</p></div>
|
||||
<div class="sp-help-topic" id="log"><h3>Outbound log</h3><p>One row per message and recipient, with the status Postfix last reported: queued (accepted, nothing reported yet), sent (the receiving server took it; what the mailbox does next is not reported back), deferred (not taken yet, Postfix tries again), bounced (failed for good, or the queue lifetime ran out) and rejected (refused before queueing under a level-2 rate limit, so it never reached Postfix). A message's page lists these steps and the lines Postfix wrote about it in <code>mail.log</code>, found by queue id. The lines are kept with the row: they stay on the page after that file has rotated, for as long as the row does. A message accepted before version 2.2.0 shows them only while they are still in the current file. Rows are deleted after the retention set in Server › Settings, 90 days by default, and their lines with them. A domain administrator sees only the rows of their own domains.</p></div>
|
||||
<div class="sp-help-topic" id="queue"><h3>Queue and retries</h3><p>The page lists, as <code>postqueue -p</code> prints it, the messages Postfix has accepted and not yet delivered: mail being sent right now and mail deferred for a later attempt. Only the global role can open it, and it only shows; nothing here deletes or forces a message. A deferred message is tried again after the first retry delay, then at gaps that double up to a cap, until the receiving server takes it or the queue lifetime runs out and it bounces; there is no attempt counter, and the box on the page shows this server's values. A queue that keeps growing means a receiver is refusing or not answering: open the message from the Outbound log to read the reason Postfix logged.</p></div>
|
||||
<div class="sp-help-topic" id="dmarc"><h3>DMARC reports</h3><p>An aggregate report is the summary a receiving provider sends, usually daily, to the <code>rua=</code> address of your DMARC record: which servers sent mail as your domain and whether SPF or DKIM passed for it. A domain's report address is one of three choices, set in its Domain settings: no reports (the default for a new domain), the SelfPost hosted address (an address on this server whose reports are parsed and shown under DMARC reports; offered only while hosted reports are enabled on the server), or an address typed for the domain (an ordinary mailbox that receives the raw XML; a button fills in your own profile e-mail). When the typed address is on another domain than the one sending, that domain must publish a report-authorization record, or providers will not send; the domain's page shows it with the other DNS records and checks it. In the tables pass means SPF or DKIM passed and aligned, fail means neither did, and your own server shows as <em>this relay</em>. A fail from any other source is someone else sending as your domain — a forwarder, an application you forgot, or abuse — so do not tighten <code>p=</code> until that sender is fixed or removed. A domain administrator reads the reports of their own domains.</p></div>
|
||||
</div>
|
||||
@@ -66,16 +69,19 @@
|
||||
<div class="sp-help-topic" id="upstream"><h3>Upstream</h3><p>The upstream is the server that actually holds the mailboxes. Give its host and port — used exactly as written, with no MX lookup — and choose the TLS for the hop to it: Opportunistic uses TLS when the upstream offers it and otherwise sends in the clear, Required holds the mail until a TLS session is possible, Off never uses TLS. While the upstream does not answer, Postfix keeps the mail and retries as for any other mail.</p></div>
|
||||
<div class="sp-help-topic" id="recipients"><h3>Valid recipients</h3><p>Choose whom this domain accepts. Listed addresses only: the relay accepts exactly the addresses you list, one per line or comma-separated, and refuses any other while the sender is still connected, so it never has to send a bounce for them. Any recipient: everything at the domain is accepted and passed on, and an address the upstream then rejects comes back as a bounce from this server to a sender who is often forged — backscatter. Prefer a list, and keep it in step with the mailboxes upstream.</p></div>
|
||||
<div class="sp-help-topic" id="filter"><h3>Spam filter</h3><p>The spam filter belongs to the server, not to a domain. The server's administrator attaches it in the server's <code>.env</code>, and from then on every inbound domain is screened by it before mail is handed on; the box shows on or off and has nothing to switch. Off means mail goes to the upstream unfiltered. While the filter is down, mail is either accepted unfiltered (the default) or deferred, so the sending server tries again later, as the server's setting says; the box states which.</p></div>
|
||||
<div class="sp-help-topic" id="inbound-log"><h3>Inbound log</h3><p>One row per recipient of a message offered on port 25 for an inbound domain, with what was decided: <span class="tag is-success is-light">accept</span> — handed on to the upstream; <span class="tag is-danger is-light">reject</span> — refused for good; <span class="tag is-warning is-light">tempfail</span> — the sender was asked to come back later; <span class="tag is-primary is-light">quarantine</span> — held by the filter. <em>Why</em> names who decided. <code>rspamd</code> is the spam filter, with its action, the score against the reject threshold and the heaviest symbols; <code>postfix</code> is the relay's own rule — an unknown recipient, a sender domain that does not exist — with its reply; <code>selfpost</code> means nobody judged the message: it was relayed with no filter attached. Accept says what this server did, not what the upstream did with the message afterwards.<br><br>Accepted mail is recorded as it passes; refused mail is read from <code>mail.log</code>, so its row has no subject, and a message refused after its text was sent is one row however many recipients it had. A refusal to relay for a domain that is not an inbound one is every scanner on the internet and is not recorded. Rows are kept for the number of days set in Settings, and never more than 10 000.</p></div>
|
||||
<div class="sp-help-topic" id="filter-lists"><h3>Filter lists</h3><p>Two lists for the whole server that correct the spam filter by hand. An entry is a sender address, a sender domain — that domain, not its subdomains — or a client address or network in CIDR form. A sender on the Deny list is refused outright. A sender on the Allow list is given a weight no spam score outweighs, so the filter lets it through. The sender's address is whatever the sending server says it is - anyone can claim to be a sender you allow - so allow a client address or network where you can, and a sender address only for mail you would rather receive forged than lose. When entries of both lists match one message, <strong>Deny wins</strong>.<br><br>SelfPost does not apply the lists itself: it writes them as files under <code>/data/antispam/</code>, and rspamd rereads a file within seconds of a change. With the spam filter off the lists are kept and can be edited, but nothing reads them. The sender matched is the envelope sender — the one in the log — not the From line a mail program shows. Matched counts the messages in the Inbound log that the entry decided, so it reaches back as far as the log does; never means not once in that time, and such an entry is one to consider removing.</p></div>
|
||||
</div>
|
||||
<div class="box">
|
||||
<div class="sp-box-head"><span class="sp-no">04</span><h2>Server</h2></div>
|
||||
<div class="sp-help-topic" id="backup"><h3>Backup and moving a domain</h3><p>Full backup downloads one archive of the whole instance: <code>data/</code> (the database, DKIM keys, application credentials and the queue), <code>docker-compose.yml</code>, <code>.env</code> and <code>certs/</code>. It holds secrets, so leave Encrypt with a password ticked: the file is sealed as <code>.spbk</code>, the password is not stored, and without it nothing can be recovered. To restore, unpack it into an empty project directory (decrypt a <code>.spbk</code> first with <code>selfpost-backup -decrypt</code>) and start the same SelfPost version that made it; the panel refuses data from another version. The reverse-proxy configuration and <code>mail.log</code> are not included. One domain moves on its own: Export domain on its settings page writes a <code>.json</code> (or encrypted <code>.spde</code>) with its DKIM key and its applications' passwords, and Import a domain here reads it into another instance, so the published DNS needs no change.</p></div>
|
||||
<div class="sp-help-topic" id="backup"><h3>Backup and moving a domain</h3><p>Full backup downloads one archive of the whole instance: <code>data/</code> (the database, DKIM keys and application credentials), <code>docker-compose.yml</code>, <code>.env</code> and <code>certs/</code>. The Postfix queue is not in it. The backup is taken while the server is running, and Postfix keeps moving messages through the queue, so a copy could hold a message twice or miss it; mail still waiting for delivery is therefore not carried over, and a restored server starts with an empty queue. To keep the queue, stop the container and archive the project directory instead. The private key in <code>certs/</code> has to be readable by the panel: give it to group 998 on the host (<code>chgrp 998</code>, <code>chmod 640</code>). Until then the panel refuses to make a backup and names the file, rather than hand over an archive without the key. It holds secrets, so leave Encrypt with a password ticked: the file is sealed as <code>.spbk</code>, the password is not stored, and without it nothing can be recovered. To restore, unpack it into an empty project directory (decrypt a <code>.spbk</code> first with <code>selfpost-backup -decrypt</code>) and start the same SelfPost version that made it; the panel refuses data from another version. The reverse-proxy configuration and <code>mail.log</code> are not included. One domain moves on its own: Export domain on its settings page writes a <code>.json</code> (or encrypted <code>.spde</code>) with its DKIM key and its applications' passwords, and Import a domain here reads it into another instance, so the published DNS needs no change.</p></div>
|
||||
<div class="sp-help-topic" id="users"><h3>Users and roles</h3><p>The two roles are a reach, not a rank: every user is an administrator and signs in with a username and password. Global reaches the whole server — Overview, Server, the queue and every domain — and is the only role that adds or deletes domains and users. Domain reaches only the domains assigned to it, outbound and inbound separately (the same name on both lists is two assignments); <em>All</em> includes domains added later, and a domain user needs at least one domain or <em>All</em> on either list. The only global user cannot be demoted or deleted, and you cannot delete the account you are signed in with. A new password set here signs that user out of their other sessions.</p></div>
|
||||
<div class="sp-help-topic" id="settings"><h3>Settings</h3><p>Settings holds what is true of the whole instance. Log retention is how many days the Outbound log keeps its rows, from 7 to 365 and 90 by default; a sweep every six hours deletes older ones, and it is the main driver of how fast <code>/data</code> grows. It does not change the rotation of <code>mail.log</code>, which is daily with about two weeks kept. The sending rate limit shown under it is level 1, the per-client-IP ceiling set in <code>.env</code>: changing it needs a restart, and no level-2 limit can exceed it.</p></div>
|
||||
<div class="sp-help-topic" id="settings"><h3>Settings</h3><p>Settings holds what is true of the whole instance. Log retention is how many days the Outbound log keeps its rows, from 7 to 365 and 90 by default; a sweep every six hours deletes older ones, and it is the main driver of how fast <code>/data</code> grows. With the inbound relay on there is a second number, for the Inbound log, with the same range. It does not change the rotation of <code>mail.log</code>, which is daily with about two weeks kept. The sending rate limit shown under it is level 1, the per-client-IP ceiling set in <code>.env</code>: changing it needs a restart, and no level-2 limit can exceed it. Notifications are turned on or off here for everyone.</p></div>
|
||||
<div class="sp-help-topic" id="notifications"><h3>Notifications</h3><p>The panel writes to its users when something needs them, from <code>postmaster@</code> the server's hostname. Four things are reported. A sending domain's SPF, DKIM or DMARC record stops matching: sent once two hourly checks in a row have found it so, and once more when the record matches again. A level-2 rate limit refuses mail: sent at the first refusal, then at most once a day for the same limit. The TLS certificate is 14 days from expiry, and again at 3 days if it is still the same certificate. A message has waited in the queue for more than three days, with one more message when none is left. A global user is told all of it; a domain administrator is told about the DNS records and limits of their own domains.<br><br>Each user turns notifications on or off under Account and needs an e-mail there; Server › Settings turns them off for everyone. They are the panel's own mail: not rows of the Outbound log, and not counted against any limit.<br><br>For that mail to arrive, the server's own name needs two DNS records of its own, as each of your domains does: a DKIM key, which the panel generates and signs notifications with, and an SPF record, <code>v=spf1 a -all</code>. Server › Settings gives both with the values to publish, and Server › Health shows whether they are in place. Without them notifications are still sent, and a receiver may file them as spam. Mail of your domains does not depend on either.</p></div>
|
||||
</div>
|
||||
<div class="box">
|
||||
<div class="sp-box-head"><span class="sp-no">05</span><h2>Account</h2></div>
|
||||
<div class="sp-help-topic" id="account"><h3>Your account</h3><p>Account is your own, whatever your role: your username and e-mail, and your password. Changing the password signs out every other session of this account and keeps the one you are using; applications have their own logins and are not affected.</p></div>
|
||||
<div class="sp-help-topic" id="account"><h3>Your account</h3><p>Account is your own, whatever your role: your username and e-mail, whether the panel sends you notifications, and your password. Changing the password signs out every other session of this account and keeps the one you are using; applications have their own logins and are not affected.</p></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
</div>
|
||||
|
||||
<div class="box">
|
||||
<div class="sp-box-head"><span class="sp-no">01</span><h2>Add an entry</h2><a class="sp-help" href="help.html#inbound" title="How the lists are applied"><i class="ti ti-help-circle"></i></a></div>
|
||||
<div class="sp-box-head"><span class="sp-no">01</span><h2>Add an entry</h2><a class="sp-help" href="help.html#filter-lists" title="How the lists are applied"><i class="ti ti-help-circle"></i></a></div>
|
||||
<div class="sp-box-body">
|
||||
<form class="sp-form" method="post" action="#">
|
||||
<div class="columns">
|
||||
@@ -18,7 +18,7 @@
|
||||
</div>
|
||||
<div class="column is-3">
|
||||
<div class="field"><label class="label" for="etype">Entry is</label>
|
||||
<div class="control"><div class="select is-fullwidth"><select id="etype" name="entry_type"><option>a sender address</option><option>a sender domain</option><option>a client IP or CIDR</option></select></div></div></div>
|
||||
<div class="control"><div class="select is-fullwidth"><select id="etype" name="entry_type"><option>a sender address</option><option>a sender domain</option><option>a client address or network</option></select></div></div></div>
|
||||
</div>
|
||||
<div class="column is-6">
|
||||
<div class="field"><label class="label" for="value">Value</label>
|
||||
@@ -37,11 +37,11 @@
|
||||
<div class="box">
|
||||
<div class="sp-box-head"><span class="sp-no">02</span><h2>Deny</h2><div class="sp-end">3 entries · wins over Allow</div></div>
|
||||
<div class="table-container"><table class="table is-fullwidth is-hoverable">
|
||||
<thead><tr><th>Value</th><th>Note</th><th>Added</th><th></th></tr></thead>
|
||||
<thead><tr><th>Value</th><th>Kind</th><th>Matched</th><th></th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td class="sp-mono sp-small">@bulk.example <span class="tag is-light">domain</span></td><td class="sp-muted">newsletter nobody subscribed to</td><td class="sp-muted sp-nowrap sp-small">2026-09-02</td><td class="sp-actions"><a class="has-text-danger" href="#">Remove</a></td></tr>
|
||||
<tr><td class="sp-mono sp-small">198.51.100.0/24 <span class="tag is-light">ip</span></td><td class="sp-muted">dictionary attack on lists.example.org</td><td class="sp-muted sp-nowrap sp-small">2026-08-19</td><td class="sp-actions"><a class="has-text-danger" href="#">Remove</a></td></tr>
|
||||
<tr><td class="sp-mono sp-small">win@lottery.example <span class="tag is-light">address</span></td><td class="sp-muted">—</td><td class="sp-muted sp-nowrap sp-small">2026-07-30</td><td class="sp-actions"><a class="has-text-danger" href="#">Remove</a></td></tr>
|
||||
<tr><td class="sp-mono sp-small">bulk.example</td><td><button type="button" class="tag is-light" popovertarget="deny-1">domain</button><div class="sp-sheet" id="deny-1" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-light">domain</span><h2>bulk.example</h2><button type="button" popovertarget="deny-1" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Note</th><td>newsletter nobody subscribed to</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-09-02</td></tr><tr><th>Matched</th><td>41 in the last 90 days</td></tr><tr><th>Last match</th><td class="sp-mono sp-small">2026-09-21 13:57</td></tr></tbody></table></div></div></div></div></td><td class="sp-nowrap">41</td><td class="sp-actions"><form method="post" action="#"><button type="submit" class="has-text-danger">Remove</button></form></td></tr>
|
||||
<tr><td class="sp-mono sp-small">198.51.100.0/24</td><td><button type="button" class="tag is-light" popovertarget="deny-2">ip</button><div class="sp-sheet" id="deny-2" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-light">ip</span><h2>198.51.100.0/24</h2><button type="button" popovertarget="deny-2" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Note</th><td>dictionary attack on lists.example.org</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-08-19</td></tr><tr><th>Matched</th><td>1203 in the last 90 days</td></tr><tr><th>Last match</th><td class="sp-mono sp-small">2026-09-21 09:12</td></tr></tbody></table></div></div></div></div></td><td class="sp-nowrap">1203</td><td class="sp-actions"><form method="post" action="#"><button type="submit" class="has-text-danger">Remove</button></form></td></tr>
|
||||
<tr><td class="sp-mono sp-small">win@lottery.example</td><td><button type="button" class="tag is-light" popovertarget="deny-3">address</button><div class="sp-sheet" id="deny-3" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-light">address</span><h2>win@lottery.example</h2><button type="button" popovertarget="deny-3" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Note</th><td>—</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-07-30</td></tr><tr><th>Matched</th><td>never in the last 90 days</td></tr></tbody></table></div></div></div></div></td><td class="sp-muted sp-nowrap">never</td><td class="sp-actions"><form method="post" action="#"><button type="submit" class="has-text-danger">Remove</button></form></td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</div>
|
||||
@@ -50,10 +50,10 @@
|
||||
<div class="box">
|
||||
<div class="sp-box-head"><span class="sp-no">03</span><h2>Allow</h2><div class="sp-end">2 entries</div></div>
|
||||
<div class="table-container"><table class="table is-fullwidth is-hoverable">
|
||||
<thead><tr><th>Value</th><th>Note</th><th>Added</th><th></th></tr></thead>
|
||||
<thead><tr><th>Value</th><th>Kind</th><th>Matched</th><th></th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td class="sp-mono sp-small">@partner.example <span class="tag is-light">domain</span></td><td class="sp-muted">invoices were scored as spam</td><td class="sp-muted sp-nowrap sp-small">2026-09-11</td><td class="sp-actions"><a class="has-text-danger" href="#">Remove</a></td></tr>
|
||||
<tr><td class="sp-mono sp-small">noreply@github.example <span class="tag is-light">address</span></td><td class="sp-muted">security advisories</td><td class="sp-muted sp-nowrap sp-small">2026-06-04</td><td class="sp-actions"><a class="has-text-danger" href="#">Remove</a></td></tr>
|
||||
<tr><td class="sp-mono sp-small">partner.example</td><td><button type="button" class="tag is-light" popovertarget="allow-1">domain</button><div class="sp-sheet" id="allow-1" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-light">domain</span><h2>partner.example</h2><button type="button" popovertarget="allow-1" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Note</th><td>invoices were scored as spam</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-09-11</td></tr><tr><th>Matched</th><td>7 in the last 90 days</td></tr><tr><th>Last match</th><td class="sp-mono sp-small">2026-09-19 16:40</td></tr></tbody></table></div></div></div></div></td><td class="sp-nowrap">7</td><td class="sp-actions"><form method="post" action="#"><button type="submit" class="has-text-danger">Remove</button></form></td></tr>
|
||||
<tr><td class="sp-mono sp-small">noreply@github.example</td><td><button type="button" class="tag is-light" popovertarget="allow-2">address</button><div class="sp-sheet" id="allow-2" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-light">address</span><h2>noreply@github.example</h2><button type="button" popovertarget="allow-2" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Note</th><td>security advisories</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-06-04</td></tr><tr><th>Matched</th><td>18 in the last 90 days</td></tr><tr><th>Last match</th><td class="sp-mono sp-small">2026-09-21 13:30</td></tr></tbody></table></div></div></div></div></td><td class="sp-nowrap">18</td><td class="sp-actions"><form method="post" action="#"><button type="submit" class="has-text-danger">Remove</button></form></td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</div>
|
||||
@@ -64,10 +64,10 @@
|
||||
<div class="sp-box-head"><span class="sp-no">04</span><h2>Where the lists go</h2><div class="sp-end"><span class="tag is-success is-light">applied</span></div></div>
|
||||
<div class="sp-box-body">
|
||||
<dl class="sp-facts">
|
||||
<div><dt>Applied by</dt><dd>rspamd, from map files</dd></div>
|
||||
<div><dt>Files</dt><dd class="sp-mono sp-small">/data/antispam/allow.map<br>/data/antispam/deny.map</dd></div>
|
||||
<div><dt>Applied by</dt><dd>rspamd, from map files it rereads within seconds of a change</dd></div>
|
||||
<div><dt>Files</dt><dd><span class="sp-mono">/data/antispam/</span> — six, one for each list and kind of entry</dd></div>
|
||||
<div><dt>Last written</dt><dd>2026-09-11 08:40 UTC</dd></div>
|
||||
<div><dt>When the filter is off</dt><dd class="sp-muted">The lists are kept and editable, but nothing reads them until <code>INBOUND_ANTISPAM_MILTER</code> is set.</dd></div>
|
||||
<div><dt>When both lists match</dt><dd>Deny wins, whichever entries they are.</dd></div>
|
||||
</dl>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
<div class="sp-grow">
|
||||
<p class="sp-kicker">Inbound</p>
|
||||
<h1 class="title is-3">Inbound log</h1>
|
||||
<p class="sp-lead">What port 25 decided about each message, one row per recipient. Accepts come from the inbound journal, rejects from <code>mail.log</code>; the filter's reasons appear when the spam filter is on.</p>
|
||||
<p class="sp-lead">What port 25 decided about each message, one row per recipient. Accepted mail is recorded as it passes, refused mail is read from <code>mail.log</code>; the filter's verdict appears when the spam filter is on. Rows older than 90 days are deleted — change that in <a href="settings.html">Settings</a>.</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -17,17 +17,18 @@
|
||||
<input class="input" type="date" name="to" value="2026-09-21" aria-label="To date">
|
||||
<button type="submit" class="button"><i class="ti ti-filter"></i>Filter</button>
|
||||
</form>
|
||||
<a class="sp-help" href="help.html#inbound-log" title="What the decisions mean"><i class="ti ti-help-circle"></i></a>
|
||||
</div>
|
||||
<div class="table-container"><table class="table is-fullwidth is-hoverable">
|
||||
<thead><tr><th>Time, UTC</th><th>From · client</th><th>To</th><th>Subject</th><th>Decision</th><th>Why</th><th></th></tr></thead>
|
||||
<thead><tr><th>Time, UTC</th><th>From</th><th>To</th><th>Subject</th><th>Decision</th><th></th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 14:00:18</td><td>t.berg@<wbr>uni.example<br><span class="sp-mono sp-muted sp-small">192.0.2.77</span></td><td>dev@<wbr>lists.example.org</td><td class="sp-clip">Re: [dev] patch series v3</td><td><span class="tag is-success is-light">accept</span></td><td class="sp-muted sp-small">rspamd · score 0.4</td><td class="sp-actions"><a href="in-filter-lists.html">Deny sender</a></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:59:02</td><td>invoice@<wbr>partner-billing.example<br><span class="sp-mono sp-muted sp-small">203.0.113.140</span></td><td>owner@<wbr>lists.example.org</td><td class="sp-clip">Outstanding invoice — action required</td><td><span class="tag is-primary is-light">quarantine</span></td><td class="sp-muted sp-small">rspamd · score 9.8 · PHISHING, FORGED_SENDER</td><td class="sp-actions"><a href="in-quarantine-message.html">Review</a></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:57:41</td><td>promo@<wbr>bulk.example<br><span class="sp-mono sp-muted sp-small">198.51.100.90</span></td><td>sales@<wbr>lists.example.org</td><td class="sp-clip sp-muted">—</td><td><span class="tag is-danger is-light">reject</span></td><td class="sp-muted sp-small">postfix · 550 5.1.1 unknown recipient</td><td class="sp-actions"><a href="in-filter-lists.html">Deny sender</a></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:51:30</td><td>billing@<wbr>partner.example<br><span class="sp-mono sp-muted sp-small">198.51.100.12</span></td><td>owner@<wbr>lists.example.org</td><td class="sp-clip">Invoice 2026-0917</td><td><span class="tag is-danger is-light">reject</span></td><td class="sp-muted sp-small">rspamd · score 16.2 · BAYES_SPAM</td><td class="sp-actions"><a href="in-filter-lists.html">Allow sender</a></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:44:02</td><td>hr@<wbr>partner.example<br><span class="sp-mono sp-muted sp-small">192.0.2.31</span></td><td>jobs@<wbr>acme.io</td><td class="sp-clip">Application: backend engineer</td><td><span class="tag is-warning is-light">tempfail</span></td><td class="sp-muted sp-small">selfpost · upstream 10.0.4.12 timed out</td><td class="sp-actions"></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:30:55</td><td>noreply@<wbr>github.example<br><span class="sp-mono sp-muted sp-small">192.0.2.200</span></td><td>security@<wbr>lists.example.org</td><td class="sp-clip">[advisory] new report</td><td><span class="tag is-success is-light">accept</span></td><td class="sp-muted sp-small">rspamd · allow list</td><td class="sp-actions"></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 14:00:18</td><td>t.berg@<wbr>uni.example<br><span class="sp-mono sp-muted sp-small">192.0.2.77</span></td><td>dev@<wbr>lists.example.org</td><td class="sp-clip">Re: [dev] patch series v3</td><td><button type="button" class="tag is-success is-light" popovertarget="why-1">accept</button><div class="sp-sheet" id="why-1" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-success is-light">accept</span><h2>Why</h2><button type="button" popovertarget="why-1" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>rspamd</td></tr><tr><th>Verdict</th><td>no action, score 0.40 / 15.00</td></tr><tr><th>Subject</th><td>Re: [dev] patch series v3</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"><a href="in-filter-lists.html">Deny sender</a></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:59:02</td><td>invoice@<wbr>partner-billing.example<br><span class="sp-mono sp-muted sp-small">203.0.113.140</span></td><td>owner@<wbr>lists.example.org</td><td class="sp-clip">Outstanding invoice — action required</td><td><button type="button" class="tag is-success is-light" popovertarget="why-2">accept</button><div class="sp-sheet" id="why-2" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-success is-light">accept</span><h2>Why</h2><button type="button" popovertarget="why-2" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>rspamd</td></tr><tr><th>Verdict</th><td>add header, score 9.80 / 15.00</td></tr><tr><th>Symbols</th><td class="sp-mono sp-small">PHISHING(5.00)<br>FORGED_SENDER(3.00)<br>R_SPF_FAIL(1.00)</td></tr><tr><th>Subject</th><td>Outstanding invoice — action required</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"><a href="in-filter-lists.html">Deny sender</a></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:57:41</td><td>promo@<wbr>bulk.example<br><span class="sp-mono sp-muted sp-small">198.51.100.90</span></td><td>sales@<wbr>lists.example.org</td><td class="sp-clip sp-muted">—</td><td><button type="button" class="tag is-danger is-light" popovertarget="why-3">reject</button><div class="sp-sheet" id="why-3" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-danger is-light">reject</span><h2>Why</h2><button type="button" popovertarget="why-3" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>postfix</td></tr><tr><th>Reply</th><td class="sp-mono sp-small">550 5.1.1 <sales@lists.example.org>: Recipient address rejected: User unknown in relay recipient table</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:51:30</td><td>billing@<wbr>partner.example<br><span class="sp-mono sp-muted sp-small">198.51.100.12</span></td><td>owner@<wbr>lists.example.org</td><td class="sp-clip sp-muted">—</td><td><button type="button" class="tag is-danger is-light" popovertarget="why-4">reject</button><div class="sp-sheet" id="why-4" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-danger is-light">reject</span><h2>Why</h2><button type="button" popovertarget="why-4" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>rspamd</td></tr><tr><th>Reply</th><td class="sp-mono sp-small">5.7.1 Spam message rejected</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"><a href="in-filter-lists.html">Allow sender</a></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:44:02</td><td>hr@<wbr>partner.example<br><span class="sp-mono sp-muted sp-small">192.0.2.31</span></td><td>jobs@<wbr>acme.io</td><td class="sp-clip sp-muted">—</td><td><button type="button" class="tag is-warning is-light" popovertarget="why-5">tempfail</button><div class="sp-sheet" id="why-5" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-warning is-light">tempfail</span><h2>Why</h2><button type="button" popovertarget="why-5" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>rspamd</td></tr><tr><th>Reply</th><td class="sp-mono sp-small">4.7.1 Try again later</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"><a href="in-filter-lists.html">Allow sender</a></td></tr>
|
||||
<tr><td class="sp-mono sp-muted sp-nowrap sp-small">09-21 13:30:55</td><td>noreply@<wbr>github.example<br><span class="sp-mono sp-muted sp-small">192.0.2.200</span></td><td>security@<wbr>lists.example.org</td><td class="sp-clip">[advisory] new report</td><td><button type="button" class="tag is-success is-light" popovertarget="why-6">accept</button><div class="sp-sheet" id="why-6" popover><div class="sp-sheet-in"><div class="sp-sheet-head"><span class="tag is-success is-light">accept</span><h2>Why</h2><button type="button" popovertarget="why-6" popovertargetaction="hide" aria-label="Close"><i class="ti ti-x"></i></button></div><div class="sp-sheet-body"><div class="table-container"><table class="table is-fullwidth"><tbody><tr><th>Decided by</th><td>rspamd</td></tr><tr><th>Verdict</th><td>no action, score -98.10 / 15.00</td></tr><tr><th>Symbols</th><td class="sp-mono sp-small">SELFPOST_ALLOW_FROM(-100.00)<br>R_DKIM_ALLOW(-0.20)</td></tr><tr><th>Subject</th><td>[advisory] new report</td></tr></tbody></table></div></div></div></div></td><td class="sp-actions"><a href="in-filter-lists.html">Deny sender</a></td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
<div class="sp-box-foot"><span>Page 1 of 38</span><a href="#">Older →</a><span class="sp-end">Allow / Deny sender opens <a href="in-filter-lists.html">Filter lists</a> with the entry filled in · global administrator only</span></div>
|
||||
<div class="sp-box-foot"><span>Page 1 of 38</span><a href="#">Older →</a><span class="sp-end">A decision opens its reason · Allow / Deny sender opens <a href="in-filter-lists.html">Filter lists</a> with the entry filled in</span></div>
|
||||
</div>
|
||||
|
||||
@@ -16,11 +16,12 @@
|
||||
<div class="field"><label class="label" for="ret">Keep outbound log rows, days</label>
|
||||
<div class="control"><input class="input" id="ret" name="send_log_retention_days" type="number" min="7" max="365" value="30" required></div>
|
||||
<p class="help">The main driver of <code>/data</code> growth. Does not affect <code>mail.log</code> rotation (fourteen daily files).</p></div>
|
||||
<div class="field"><label class="label" for="inret">Keep inbound log rows, days</label>
|
||||
<div class="control"><input class="input" id="inret" name="inbound_spam_log_retention_days" type="number" min="7" max="365" value="90" required></div>
|
||||
<p class="help">The inbound log also never holds more than 10 000 rows: refused mail is logged at the rate strangers knock on port 25.</p></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="column is-6">
|
||||
<div class="box" id="rate-limits">
|
||||
<div class="sp-box-head"><span class="sp-no">02</span><h2>Sending rate limits</h2><div class="sp-end">Read-only</div><a class="sp-help" href="help.html#limits" title="Levels 1 and 2"><i class="ti ti-help-circle"></i></a></div>
|
||||
<div class="sp-box-body">
|
||||
@@ -31,6 +32,43 @@
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="column is-6">
|
||||
<div class="box" id="notifications">
|
||||
<div class="sp-box-head"><span class="sp-no">03</span><h2>Notifications</h2><a class="sp-help" href="help.html#notifications" title="What is sent and when"><i class="ti ti-help-circle"></i></a></div>
|
||||
<div class="sp-box-body">
|
||||
<div class="field"><label class="checkbox"><input type="checkbox" name="notify" checked> Send notification e-mails</label>
|
||||
<p class="help">From <span class="sp-mono">postmaster@mail.example.org</span>, to every user who has an e-mail and has not turned them off under <a href="account.html">Account</a>.</p></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="box" id="notification-mail">
|
||||
<div class="sp-box-head"><span class="sp-no">04</span><h2>Notification mail</h2><div class="sp-end">DNS records of the server's own name</div><a class="sp-help" href="help.html#notifications" title="What the panel sends and what makes it arrive"><i class="ti ti-help-circle"></i></a></div>
|
||||
|
||||
<div class="sp-record">
|
||||
<div class="sp-record-head"><h3>DKIM</h3><span class="tag is-success is-light">published</span><small>Not a secret · selector <code>sp2026</code></small></div>
|
||||
<div class="sp-row">
|
||||
<div class="field"><label class="label">Host / name</label>
|
||||
<div class="field has-addons"><div class="control is-expanded"><input class="input" readonly value="sp2026._domainkey.mail.example.org"></div><div class="control"><button type="button" class="button"><i class="ti ti-copy"></i>Copy</button></div></div></div>
|
||||
<div class="field"><label class="label">Type</label><div class="control"><input class="input" readonly value="TXT"></div></div>
|
||||
</div>
|
||||
<div class="field"><label class="label">Value to publish</label>
|
||||
<div class="field has-addons"><div class="control is-expanded"><textarea class="textarea" rows="5" readonly>v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAq7n2Lw0c5vXeR1dYt8uJb3mKs6pHf9aZoWg4iNy2TqCx0lVjE7rUh5sBk1dMa8fPz3oGc6yXw9tIe4nQv2uL…IDAQAB</textarea></div><div class="control"><button type="button" class="button"><i class="ti ti-copy"></i>Copy</button></div></div></div>
|
||||
</div>
|
||||
|
||||
<div class="sp-record">
|
||||
<div class="sp-record-head"><h3>SPF</h3><span class="tag is-warning is-light">missing</span><small>For the server's own name, not for your domains</small></div>
|
||||
<div class="sp-row">
|
||||
<div class="field"><label class="label">Host / name</label>
|
||||
<div class="field has-addons"><div class="control is-expanded"><input class="input" readonly value="mail.example.org"></div><div class="control"><button type="button" class="button"><i class="ti ti-copy"></i>Copy</button></div></div></div>
|
||||
<div class="field"><label class="label">Type</label><div class="control"><input class="input" readonly value="TXT"></div></div>
|
||||
</div>
|
||||
<div class="field"><label class="label">Value to publish</label>
|
||||
<div class="field has-addons"><div class="control is-expanded"><input class="input" readonly value="v=spf1 a -all"></div><div class="control"><button type="button" class="button"><i class="ti ti-copy"></i>Copy</button></div></div></div>
|
||||
<p class="help">Notifications are sent without it, and a receiver may file them as spam.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="buttons"><button type="submit" class="button is-primary">Save settings</button></div>
|
||||
</form>
|
||||
|
||||
@@ -270,3 +270,27 @@ code { background: var(--sp-well); color: var(--sp-ink); border-radius: 3px; pad
|
||||
.sp-record-head small { margin-left: 0; text-align: left; flex-basis: 100%; }
|
||||
.sp-postmark { display: none; }
|
||||
}
|
||||
|
||||
/* sheet: what does not fit a table row, opened over the page from the row's
|
||||
tag. It is the browser's own popover (the tag is a button with
|
||||
popovertarget), so it needs no script: Esc, a click outside and the close
|
||||
button all shut it, and focus returns to the tag. It rises from the bottom
|
||||
edge across the whole window - the one component that is not laid out by the
|
||||
page, hence the widths - with its content, a table of name and value, in a
|
||||
column of reading width in the middle. */
|
||||
.sp-sheet {
|
||||
inset: auto 0 0 0; width: auto; margin: 0; padding: 0; border: 0; border-top: 3px solid var(--sp-brick);
|
||||
max-height: 75vh; overflow: auto; background: #fff; color: inherit; box-shadow: 0 -12px 40px rgba(40, 20, 15, .22);
|
||||
}
|
||||
.sp-sheet-in { max-width: 44rem; margin: 0 auto; padding: 0 1.25rem; }
|
||||
.sp-sheet::backdrop { background: rgba(40, 20, 15, .38); }
|
||||
.sp-sheet-head { display: flex; align-items: center; gap: .7rem; padding: 1rem 0 .8rem; border-bottom: 1px solid var(--sp-rule); }
|
||||
.sp-sheet-head h2 { font-size: 1.15rem; font-weight: 600; margin-right: auto; }
|
||||
.sp-sheet-head button { border: 0; background: none; cursor: pointer; font-size: 1.3rem; color: var(--sp-muted); }
|
||||
.sp-sheet-body { padding: .4rem 0 1.4rem; }
|
||||
/* The sheet's content is a table of name and value: the name is a label, set like the dt of facts. */
|
||||
.sp-sheet .table th { font-family: var(--sp-mono); font-size: var(--sp-fs-xs); font-weight: 500; letter-spacing: .08em; text-transform: uppercase; color: var(--sp-muted); white-space: nowrap; vertical-align: top; width: 1%; padding-right: 2rem; }
|
||||
.sp-sheet .table td { overflow-wrap: anywhere; }
|
||||
/* A tag that opens a sheet is a button: it keeps the tag's look and says it can
|
||||
be pressed with the dotted underline links to more detail carry. */
|
||||
button.tag { border: 0; cursor: pointer; text-decoration: underline dotted; text-underline-offset: 3px; }
|
||||
|
||||
+5
-2
@@ -107,7 +107,7 @@ than the default «meaningful step» rule in [§ Commits and release build](#com
|
||||
2. Git tag `vX.Y.Z` and publish the GitHub Release only on explicit request (see
|
||||
[§ Release image](#release-image)).
|
||||
|
||||
**Planned version cuts** (from pin `2.0.0`; adjust if semver changes mid-track):
|
||||
**Planned version cuts** (from pin `2.2.0`; adjust if semver changes mid-track):
|
||||
|
||||
| Stage | ID | Cut |
|
||||
|---|---|---|
|
||||
@@ -124,7 +124,10 @@ than the default «meaningful step» rule in [§ Commits and release build](#com
|
||||
| 11 | panel-redesign plan: component kit, stages, reorder (docs only, owner-requested cut) | `1.9.4` (PATCH) |
|
||||
| 12 | GHCR release fix: keep per-arch images, sweep orphaned versions (CI only) | `1.9.5` (PATCH) |
|
||||
| 13 | panel-redesign | `2.0.0` (MAJOR, no compatibility) |
|
||||
| 14 | inbound-antispam-panel | `2.1.0` (planned; was `1.10.0`, reordered 2026-10-09) |
|
||||
| 14 | DMARC ingest drops and bounds, full backup from the panel, no queue in a backup | `2.0.1` (PATCH) |
|
||||
| 15 | inbound-antispam-panel | `2.1.0` (was `1.10.0`, reordered 2026-10-09) |
|
||||
| 16 | delivery-log-storage | `2.2.0` |
|
||||
| 17 | panel-notifications | `2.3.0` |
|
||||
|
||||
Docs-only prep (checklists in plans, no product code) uses the same per-step
|
||||
commit rule but **no** version cut until the next product stage ships.
|
||||
|
||||
+154
-16
@@ -33,6 +33,7 @@ changed (see the CHANGELOG, *Removed*).
|
||||
- [Queue and System log](#queue-and-system-log)
|
||||
- [Account](#account)
|
||||
- [Settings](#settings)
|
||||
- [Notifications](#notifications)
|
||||
- [Users](#users)
|
||||
- [Sessions](#sessions)
|
||||
- [Upgrading](#upgrading)
|
||||
@@ -183,7 +184,7 @@ cat ./data/setup-token
|
||||
#### Fixed image tag
|
||||
|
||||
`deploy/docker-compose.yml` pins an explicit version (`ghcr.io/mixeme/selfpost:X.Y.Z`),
|
||||
deliberately never `:latest`. The current pin is `2.0.0`. Intermediate
|
||||
deliberately never `:latest`. The current pin is `2.2.0`. Intermediate
|
||||
CHANGELOG sections (`0.2.0`…`0.6.0`) record development cuts from before that
|
||||
image was published. Pinning matters because of the backup version check (see
|
||||
[Full backup and restore](#full-backup-and-restore)): the panel binary's
|
||||
@@ -209,11 +210,13 @@ expected to set; defaults match the code exactly.
|
||||
| `DMARC_MESSAGE_SIZE_LIMIT` | Maximum report message size in bytes when DMARC ingest is on. | `5242880` (5 MiB) | `.env` |
|
||||
| `INBOUND_ANTISPAM_MILTER` | Optional milter on the inbound relay listener only (requires `INBOUND_RELAY_ENABLE=true`; not 465/587 or DMARC-only port 25). Empty = off. Format `inet:host:port` or `unix:/path`. Example with [deploy/antispam/docker-compose.antispam.yml](../deploy/antispam/docker-compose.antispam.yml): `inet:antispam:11332`. The panel reads it to report on the *Spam filter* box of the inbound pages and to check that the filter answers (Server › Health). | *(empty)* | `.env` |
|
||||
| `INBOUND_ANTISPAM_MILTER_ACTION` | What Postfix does if that milter is down: `accept` (fail-open) or `tempfail` (defer). The *Spam filter* box states which. | `accept` | `.env` |
|
||||
| `INBOUND_SPAM_LOG_RETENTION_DAYS` | How long the inbound journal keeps a row, 7-365 days; it also never holds more than 10 000 rows. Seeds the setting on the first start with the inbound relay on. | `90` | `.env` |
|
||||
| `INBOUND_JOURNAL_MILTER_SOCKET` | Unix socket of the panel's inbound journal milter, the last milter on port 25 while the inbound relay is on. It records accepted inbound mail and never refuses any; there is no reason to change it. | `/run/selfpost/inbound-journal.sock` | compose |
|
||||
| `INBOUND_RATE_LIMIT_MESSAGES_PER_IP` | Coarse per-client-IP cap on inbound smtpd (`smtpd_client_message_rate_limit`). Uses the same window as `RATE_LIMIT_WINDOW_SECONDS`. | `20` | `.env` |
|
||||
| `INBOUND_MESSAGE_SIZE_LIMIT` | Maximum message size in bytes on inbound smtpd (`message_size_limit`). When both inbound relay and DMARC ingest are on, the shared listener uses the **greater** of this and `DMARC_MESSAGE_SIZE_LIMIT`. | `26214400` (25 MiB) | `.env` |
|
||||
| `RATE_LIMIT_MESSAGES_PER_IP` | Level-1 backstop: maximum messages one client IP may submit per window (Postfix `smtpd_client_message_rate_limit`). See [Rate limiting — level 1](#rate-limiting--level-1-ip-backstop). | `100` | `.env` |
|
||||
| `RATE_LIMIT_WINDOW_SECONDS` | Level-1 window length in seconds (Postfix `anvil_rate_time_unit`). | `3600` | `.env` |
|
||||
| `SEND_LOG_RETENTION_DAYS` | Initial default for how many days of send-log history are kept before the background sweep deletes rows — the main driver of `/data` growth over time. After the first panel start, change retention under **Server › Settings** (global role); the env value is only used to seed SQLite when the setting has never been saved. | `90` | `.env` |
|
||||
| `SEND_LOG_RETENTION_DAYS` | Initial default for how many days of send-log history are kept before the background sweep deletes rows, and with them the `mail.log` lines kept for each message (1–2 KB a message) — the main driver of `/data` growth over time and of the size of a full backup. After the first panel start, change retention under **Server › Settings** (global role); the env value is only used to seed SQLite when the setting has never been saved. | `90` | `.env` |
|
||||
| `PANEL_SESSION_IDLE_DAYS` | Sliding idle timeout for the panel login session, in days. There is no absolute cap: a user who keeps coming back stays signed in indefinitely. | `7` | `.env` |
|
||||
| `SELFPOST_DNS_RESOLVERS` | Comma-separated recursive resolvers the panel's PTR/SPF/DKIM/DMARC checks query directly (so they report what the internet sees, not what this host's stub resolver synthesises). | `1.1.1.1:53`, `8.8.8.8:53`, `9.9.9.9:53` when unset | `.env` |
|
||||
| `TRUSTED_PROXY_CIDR` | Comma-separated CIDRs (bare IPs allowed) of reverse proxies allowed to supply `X-Forwarded-For` for login, setup, and account-change rate-limiting. **Leave unset unless you know the exact address of your reverse proxy.** A wrong value lets a client spoof its rate-limit key by sending a forged `X-Forwarded-For` header — the panel trusts the last hop only when the TCP peer matches one of these CIDRs. Behind the default Apache host-network setup this is typically the Docker bridge gateway, e.g. `172.18.0.1`. | *(empty — XFF ignored)* | `.env` |
|
||||
@@ -417,10 +420,11 @@ readings:
|
||||
**Account** (`/account`, every role) is the signed-in user's own: **Profile**
|
||||
(username and e-mail) and **Password**. Saving the profile does not ask for the
|
||||
password; changing the password asks for the current one and is rate-limited
|
||||
like a sign-in. The e-mail is the user's own address. Nothing is sent to it
|
||||
yet; it is what a domain's *Use my e-mail* button fills in (see [Domain-level
|
||||
DNS](#domain-level-dns-spf-dkim-dmarc)). Application SASL logins are separate
|
||||
and are not changed here.
|
||||
like a sign-in. The e-mail is the user's own address: it is where the panel
|
||||
sends [notifications](#notifications) while *Send me notifications* is ticked
|
||||
beside it, and what a domain's *Use my e-mail* button fills in (see
|
||||
[Domain-level DNS](#domain-level-dns-spf-dkim-dmarc)). Application SASL logins
|
||||
are separate and are not changed here.
|
||||
|
||||
### Settings
|
||||
|
||||
@@ -429,7 +433,53 @@ the whole instance: how many days the **Outbound log** keeps its rows before the
|
||||
background sweep deletes them (7–365 days; takes effect on the next six-hour
|
||||
prune cycle without a container restart), and, read-only, the level-1 rate
|
||||
limit that comes from `.env` ([Rate limiting — level 1](#rate-limiting--level-1-ip-backstop)).
|
||||
There is no panel-wide DMARC report address: each domain has its own.
|
||||
There is no panel-wide DMARC report address: each domain has its own. The page
|
||||
also has the switch for [notifications](#notifications) and the two DNS records
|
||||
that make them arrive.
|
||||
|
||||
### Notifications
|
||||
|
||||
The panel writes to its users when something needs them. Four things are
|
||||
reported, each once:
|
||||
|
||||
| What | When a message is sent | A second message |
|
||||
|---|---|---|
|
||||
| A sending domain's DKIM, SPF or DMARC record stops matching | after two hourly checks in a row found it wrong. A resolver that did not answer is not a mismatch; a missing DMARC record is advice on the domain's page, not a notification | when two checks in a row find the records right again |
|
||||
| A level-2 rate limit refuses mail | within a minute of the first refusal | at most once a day for the same domain and application, while it goes on |
|
||||
| The TLS certificate is about to expire | 14 days before | 3 days before, if it is still the same certificate |
|
||||
| A message has waited in the queue for more than three days | at the next hourly check | when no message that old is left |
|
||||
|
||||
A user with the **global** role is told all of it; a user with the **domain**
|
||||
role is told about the DNS records and rate limits of the outbound domains
|
||||
assigned to them. A message goes to the e-mail on the user's
|
||||
[Account](#account) page and only while *Send me notifications* is ticked
|
||||
there; a user without an e-mail gets none. **Server › Settings** has one switch
|
||||
that turns notifications off for everyone. With it off nothing is checked, so
|
||||
what is wrong when it is turned on again is reported then.
|
||||
|
||||
Notifications are the panel's own mail. They are sent from
|
||||
`postmaster@<SELFPOST_HOSTNAME>`, one message per recipient, are not rows of
|
||||
the Outbound log and count against no rate limit. They name the page of the
|
||||
panel to open and carry no link: the panel does not know the address its
|
||||
reverse proxy serves it under.
|
||||
|
||||
**Two DNS records make them arrive.** The server's own name is not one of your
|
||||
sending domains, so it needs records of its own, shown with the values to
|
||||
publish under **Server › Settings › Notification mail**:
|
||||
|
||||
- a **DKIM** key for the hostname, which the panel generates and signs
|
||||
notifications with (`<selector>._domainkey.<hostname>`, TXT);
|
||||
- an **SPF** record for the hostname: `v=spf1 a -all` (TXT).
|
||||
|
||||
**Server › Health** shows whether each is in place in the *Notification mail*
|
||||
card. Without them notifications are still sent, and a receiver may file them
|
||||
as spam or, if the parent domain publishes a strict DMARC policy, refuse them.
|
||||
Mail of your sending domains depends on neither. If the hostname is itself one
|
||||
of your sending domains, the records are that domain's and are already
|
||||
published.
|
||||
|
||||
A relay that cannot send cannot say so: none of the four is "the relay is
|
||||
down". Watch that from outside ([Container health](#container-health)).
|
||||
|
||||
### Users
|
||||
|
||||
@@ -542,10 +592,14 @@ journal-milter (level 2) is down. There is no per-IP bypass.
|
||||
### Full backup and restore
|
||||
|
||||
**Full backup** is a self-contained project archive: `data/` (SQLite, all
|
||||
domains' DKIM keys, all applications' SASL credentials, the Postfix queue,
|
||||
`manifest.json` with the version that created it), plus `docker-compose.yml`,
|
||||
`.env`, and `certs/` from the operator directory next to `./data`. Delivery
|
||||
logs under `data/log/` are excluded. The base compose file mounts the project
|
||||
domains' DKIM keys, all applications' SASL credentials, `manifest.json` with
|
||||
the version that created it), plus `docker-compose.yml`, `.env`, and `certs/`
|
||||
from the operator directory next to `./data`. Delivery logs under `data/log/`
|
||||
are excluded, and so is the Postfix queue: the archive is taken while the
|
||||
instance runs, and a copy of a queue Postfix is working on can hold a message
|
||||
twice or not at all. Mail still waiting for delivery is therefore not carried
|
||||
over, and a restored instance starts with an empty queue; to keep the queue,
|
||||
stop the instance and use `tar` (methods 3 and 4 below). The base compose file mounts the project
|
||||
directory read-only at `/selfpost-deploy` so the panel and CLI can read those
|
||||
deploy files — without that mount, *Full backup* refuses with an error.
|
||||
|
||||
@@ -556,12 +610,25 @@ instance keeps running, what goes into the archive, and what happens on restore.
|
||||
|
||||
**1. Panel — while the container is running (no downtime).** *Server › Backup*
|
||||
→ *Full backup*. *Encrypt with a password* is ticked; untick it for a plain
|
||||
archive. Same archive as the CLI below.
|
||||
archive. Same archive as the CLI below. The panel runs as its own user, not as
|
||||
root, so **the private key in `certs/` has to be readable by it**: give the
|
||||
file to the container's `selfpost` group, which has the fixed id 998 and holds
|
||||
both the panel and Postfix —
|
||||
|
||||
```sh
|
||||
chgrp 998 certs/privkey.pem && chmod 640 certs/privkey.pem
|
||||
```
|
||||
|
||||
— and have whatever renews the certificate do the same (a certbot deploy hook,
|
||||
for one). A key readable by root alone is enough for Postfix, but then the
|
||||
panel refuses to make a backup and names the file: it does not hand over an
|
||||
archive without the key.
|
||||
|
||||
**2. `selfpost-backup` CLI — while the container is running (no downtime).**
|
||||
The panel button and this command call the same code path: a consistent SQLite
|
||||
snapshot via `VACUUM INTO`, then a gzip tar of `data/` (minus diagnostics) plus
|
||||
deploy files.
|
||||
deploy files. `docker exec` runs the command as root, so it reads the private
|
||||
key whatever its permissions.
|
||||
|
||||
```sh
|
||||
docker exec <container> selfpost-backup > selfpost-backup.tar.gz
|
||||
@@ -616,6 +683,8 @@ methods avoid it by stopping first.
|
||||
| Self-contained archive | Yes | Yes | No — deploy files separate |
|
||||
| SQLite consistency | `VACUUM INTO` snapshot | Safe when stopped | Safe when stopped |
|
||||
| `data/log/` | Excluded | Included (optional exclude) | Included (optional exclude) |
|
||||
| Postfix queue | No | Yes | Yes |
|
||||
| `certs/` private key | Yes (the panel needs it readable by group 998) | Yes | No — deploy files separate |
|
||||
| `data/manifest.json` | Written (version stamp) | Absent | Absent |
|
||||
| Restore version check | Yes — must match image tag | No | No |
|
||||
| Post-restore *Resync* | Yes, on first boot | No | No |
|
||||
@@ -691,6 +760,15 @@ needed) for the new hostname or proxy, then `docker compose up -d`. The archive
|
||||
carries `certs/` from the old host — re-issue certificates when the hostname or
|
||||
IP changes. Set up the reverse-proxy vhost separately (not in the backup).
|
||||
|
||||
The inbound log and the filter lists are in the database and so in the
|
||||
backup, and so are the list files under `data/antispam/`; the panel writes
|
||||
those files again from the database on the first start after a restore. What
|
||||
the backup does not hold is the spam filter's own side: the `antispam/`
|
||||
directory with the compose fragment and the rspamd configuration. Like the
|
||||
reverse-proxy fragments it is part of the installation, not of its state -
|
||||
copy it from [deploy/antispam/](../deploy/antispam/) again, or the restored
|
||||
instance runs with no filter.
|
||||
|
||||
**Restoring an encrypted (`.spbk`) backup** needs a running container to
|
||||
decrypt it first — any container with the `selfpost-backup` CLI works; decryption
|
||||
does not read `/data` and performs no version check. Start one normally
|
||||
@@ -822,6 +900,58 @@ only sees SelfPost. Default action is fail-open (`accept`) so a down sidecar
|
||||
does not block backup-MX; set `INBOUND_ANTISPAM_MILTER_ACTION=tempfail` to
|
||||
defer instead.
|
||||
|
||||
The fragment mounts one file into the sidecar,
|
||||
[deploy/antispam/rspamd/milter_headers.conf](../deploy/antispam/rspamd/milter_headers.conf):
|
||||
it makes rspamd write its verdict - the action and the score with its symbols
|
||||
- into two headers of every message it lets through. SelfPost records them
|
||||
with the message, and the upstream receives them with the mail. Copy the
|
||||
`antispam/rspamd/` directory next to the fragment; without the file accepted
|
||||
mail is still recorded, with no verdict.
|
||||
|
||||
The same file makes rspamd **sign** each message, in a third header,
|
||||
`X-SelfPost-Filter`. The filter is attached so that mail still flows when it
|
||||
is down, and then a sender could write a verdict of his own into those
|
||||
headers; SelfPost therefore records a verdict only next to a signature it can
|
||||
recompute. The key is `data/antispam/filter.key`: SelfPost creates it on its
|
||||
first start with the inbound relay on, and the sidecar reads it from the
|
||||
directory it already mounts - there is nothing to configure. If Inbound › Log
|
||||
shows *selfpost* for mail the filter did judge, the sidecar is running without
|
||||
this file: copy `antispam/rspamd/` again and restart the sidecar. Nothing of
|
||||
rspamd's needs clearing, then or after a change of the lists or of the key: it
|
||||
rereads the list files by itself and reads the key anew for every message. A
|
||||
restart is needed only for a change of its own configuration - the files of
|
||||
`antispam/rspamd/`.
|
||||
|
||||
The same directory holds
|
||||
[multimap.conf](../deploy/antispam/rspamd/multimap.conf), the rules for the
|
||||
**allow and deny lists** kept in the panel. SelfPost does not apply the lists
|
||||
itself: it writes them as six files under `data/antispam/` (`allow_ip.map`,
|
||||
`allow_from.map`, `allow_domain.map` and the three `deny_` ones), the fragment
|
||||
mounts that directory into the sidecar, and rspamd rereads a file within
|
||||
seconds of a change - no restart. An entry is a sender address, a sender
|
||||
domain (that domain, not its subdomains) or a client address or network. A
|
||||
denied sender is refused outright. An allowed one is given a weight that no
|
||||
spam score outweighs, so it is accepted - unless a deny entry matches the same
|
||||
message: **deny wins**, whichever entries they are. **A sender address or
|
||||
domain on the allow list is not proof of anything**: it is the envelope
|
||||
sender, which the sending server states and nobody checks, so whoever knows an
|
||||
allowed address can send as it. Allow a client address or network where you
|
||||
can. The files are readable by
|
||||
anyone on the host; they hold addresses, not secrets.
|
||||
|
||||
In the panel the lists are **Inbound › Filter lists** (global role), and what
|
||||
port 25 decided about each message is **Inbound › Log**: accepted mail with the
|
||||
filter's verdict, refused mail with the reply it was given. A row of the log
|
||||
leads to the lists with its sender filled in. Beside each entry of a list is
|
||||
how often it decided a message, counted from the log - so over the days the
|
||||
log keeps - and *never* marks an entry that may have outlived its use. The
|
||||
count depends on two things in
|
||||
[multimap.conf](../deploy/antispam/rspamd/multimap.conf): the message of the
|
||||
deny rules and the names of the allow symbols; change them and every entry
|
||||
reads *never*. A user with the *domain* role
|
||||
sees the log of the inbound domains assigned to them, and not the lists. How
|
||||
long the log keeps a row is set under **Server › Settings**.
|
||||
|
||||
The panel only reports this setting; it does not change it. Every inbound
|
||||
domain page has a **Spam filter** box — on or off, what happens to mail while
|
||||
the filter is down, and that the filter belongs to the server, not to the
|
||||
@@ -891,7 +1021,13 @@ sees the *Ingest* box — whether ingest works, the last report received, report
|
||||
kept this week and parse failures. Open a domain's roll-up from the list or
|
||||
from *DMARC reports* in the domain page's menu. A user with the *domain* role
|
||||
sees the reports of the domains assigned to them, without the *Ingest* box.
|
||||
Summaries are pruned (500 kept, 90 days max).
|
||||
Summaries are pruned (500 kept for each domain, 90 days max).
|
||||
|
||||
A message that is not a report the relay keeps — unreadable as an aggregate
|
||||
report, about a domain not sent from here, or delivered to another domain's
|
||||
report address — is dropped without a retry and without a bounce. It adds one
|
||||
to *parse failures*, and the reason is on the message's delivery line in the mail log
|
||||
(*Server › System log*, `dropped: …`).
|
||||
|
||||
**Not an open relay.** The inbound smtpd offers no SASL. With DMARC ingest
|
||||
alone, `check_recipient_access` permits only configured report addresses;
|
||||
@@ -1056,8 +1192,10 @@ backoff cap, queue lifetime), the same numbers the Outbound queue shows; users
|
||||
with the *domain* role see them here because they cannot open the queue. Under
|
||||
both sit the `mail.log` lines for its queue id: the connection to the
|
||||
receiving server, the server's reply, and the status that reply was filed
|
||||
as. Rows outlive `mail.log`, so an older message's lines may have rotated
|
||||
away; the page says so. Retention is set under **Server › Settings** (global
|
||||
as. The lines are kept in the database with the row, so they stay on the
|
||||
page after `mail.log` has rotated and are deleted together with the row; a
|
||||
message accepted before `2.2.0` shows them only while they are still in the
|
||||
current file, and the page says so when there are none. Retention is set under **Server › Settings** (global
|
||||
role); `SEND_LOG_RETENTION_DAYS` in `.env` is only the initial default until it
|
||||
is changed there.
|
||||
|
||||
|
||||
@@ -0,0 +1,184 @@
|
||||
# Plan: delivery-log-storage
|
||||
|
||||
**Status:** shipped in `2.2.0` (2026-10-12); agreed the same day (owner: option 1 of the roadmap item, "store the lines")
|
||||
**Date:** 2026-10-12
|
||||
**Version:** `2.2.0` MINOR (one DDL migration; a 2.1.0 data directory boots unchanged).
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
A message's page (`/outbound/log/{id}`, mockup `out-message`) shows what
|
||||
Postfix wrote about the message for as long as its row is in the send log.
|
||||
Today the *Delivery log* box is filled by a search of the current `mail.log`
|
||||
on every page view, so it is empty for anything sent before the last daily
|
||||
rotation, while the row itself lives for ninety days.
|
||||
|
||||
## Context (as-built)
|
||||
|
||||
- The log tailer ([internal/logtail](../../internal/logtail/logtail.go))
|
||||
already reads every `mail.log` line once, from a persisted offset, to
|
||||
advance delivery statuses.
|
||||
- `logtail.QueueLines` searches the last 4 MB of the current file for one
|
||||
queue id; `Handlers.deliveryLog` calls it per page view.
|
||||
- A message with several recipients is several send-log rows under one queue
|
||||
id; the page of each shows every line of the message, as the search does
|
||||
today.
|
||||
|
||||
## Decisions
|
||||
|
||||
### What is stored
|
||||
|
||||
A `mail.log` line whose head is a queue id - after the timestamp, the host
|
||||
name and `program[pid]:` - goes into a new table with that id, the line's own
|
||||
time (`lineTime`, the same reading the inbound journal uses) and the line as
|
||||
written, cut to 4096 bytes on a character boundary. The head is matched from
|
||||
the start of the line: the tail is the far end's words. `NOQUEUE` and
|
||||
`warning` are not queue ids - the pattern takes Postfix's short form
|
||||
(hexadecimal) and its long form (no vowels) and nothing else.
|
||||
|
||||
The tailer does not ask whether the id belongs to a send-log row before it
|
||||
writes. The first lines of a message (`client=`, `message-id=`) are logged
|
||||
before the journal milter writes the row at end-of-message, so such a check
|
||||
would drop exactly them.
|
||||
|
||||
**Mail that came in on port 25 is left out** (security review, 2026-10-12). A
|
||||
queue id that `postfix/inbound/smtpd` opened is a stranger's message; it never
|
||||
gets a send-log row, and what a stranger can make Postfix log - his own
|
||||
`message-id`, up to the cut - is not stored to be pruned later. The tailer
|
||||
remembers such ids (4096 at most) until qmgr's `removed` or until another
|
||||
listener opens a message under the id. What is still stored without a row is
|
||||
little: the bounces Postfix generates, and port-25 mail met in mid-flight
|
||||
after a restart.
|
||||
|
||||
Lines are written in one transaction per pass over the log, 500 at most
|
||||
(technical review): a commit per line is a disk sync per line on the one
|
||||
connection the journal milter writes through.
|
||||
|
||||
### Which lines are a row's
|
||||
|
||||
A line is shown on a row's page when the queue id is the same **and** the
|
||||
line's time lies between one hour before the row was written and the queue
|
||||
lifetime (the larger of `maximal_queue_lifetime` and `bounce_queue_lifetime`,
|
||||
as `postconf` reports them) plus a day after it. Postfix reuses a short queue
|
||||
id once the message has left the queue - the id is the microsecond of arrival
|
||||
and the inode of the queue file - so a row that lives ninety days can meet a
|
||||
later message under its id. The window does not rule that out, it limits it
|
||||
to the days the row's own message could still be in the queue; this is the
|
||||
bound the roadmap item asked for. The search of the current file that the
|
||||
page did before had the same exposure for a day. `enable_long_queue_ids`
|
||||
would end it and is a change to the mail path, not part of this item.
|
||||
|
||||
### Retention
|
||||
|
||||
Lines live as long as a send-log row with their queue id does. One rule covers
|
||||
both ends: a line older than an hour whose queue id has no send-log row is
|
||||
deleted. That removes the lines of a row the retention sweep has just deleted,
|
||||
and the lines of messages that never had a row. It runs at start and after
|
||||
every send-log sweep, six-hourly. It asks the send log once per queue id; at a
|
||||
million lines that is still a scan of the table, of the order of a second,
|
||||
during which the one connection is taken.
|
||||
|
||||
There is no cap of its own: with port-25 mail left out, the table is bounded
|
||||
by what authenticated applications send, which level 1 and level 2 limit.
|
||||
|
||||
### After an unclean stop
|
||||
|
||||
The tailer saves its place every few seconds, so after a crash it reads some
|
||||
lines twice. While it is catching up at start a line is written only if the
|
||||
same line is not there already - the rule the inbound journal follows.
|
||||
|
||||
### What is not kept
|
||||
|
||||
- Lines the tailer never reads are not stored: the tail of a file that was
|
||||
rotated away while the panel was down, as with delivery statuses
|
||||
(architecture.md § Log tailer).
|
||||
- A message in the queue across the upgrade to 2.2.0 has only its later lines
|
||||
stored, and the page then shows those alone: the search of the file runs
|
||||
only for a row with nothing stored. The only installation is the owner's
|
||||
test server; no merge of the two sources is built for one moment of it.
|
||||
|
||||
### The page
|
||||
|
||||
The page reads the table: the latest 200 lines of the row, oldest first. When
|
||||
the table has none, it still searches the current `mail.log` as before - that
|
||||
is what shows a message accepted before 2.2.0 while its lines are in today's
|
||||
file. Lines in rotated files are not imported: the only installation is the
|
||||
owner's test server.
|
||||
|
||||
Nothing in the page's layout changes; the `out-message` mockup already draws
|
||||
the box as a `log_pane`. Two texts change, mockup first: the *Outbound log*
|
||||
help topic (it says an old message may show no lines) and the note shown when
|
||||
there are none.
|
||||
|
||||
### Size
|
||||
|
||||
Six to ten lines of about 200 bytes per message: 1-2 KB next to a row of a few
|
||||
hundred bytes. The table becomes the larger part of what the send log costs in
|
||||
`/data` and in a backup; the retention setting bounds both together.
|
||||
|
||||
## Schema (migration `0003_send_log_lines.sql`)
|
||||
|
||||
```sql
|
||||
CREATE TABLE send_log_lines (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
queue_id TEXT NOT NULL,
|
||||
logged_at TEXT NOT NULL,
|
||||
line TEXT NOT NULL
|
||||
);
|
||||
CREATE INDEX idx_send_log_lines_queue_id ON send_log_lines (queue_id, logged_at);
|
||||
```
|
||||
|
||||
No foreign key: a line is written before its row exists, and one queue id is
|
||||
several rows.
|
||||
|
||||
## Scope
|
||||
|
||||
In: the table, the tailer writing and pruning it, the page reading it, the two
|
||||
texts, docs.
|
||||
|
||||
Out: the inbound journal (it decides for itself, roadmap § delivery-log-storage);
|
||||
importing rotated files; `enable_long_queue_ids`; any change to `mail.log`
|
||||
rotation or to the Server › Mail log page.
|
||||
|
||||
## Security
|
||||
|
||||
- The lines carry the far end's text (replies, HELO names). They are shown
|
||||
today from the file through the same escaping; storing them changes where
|
||||
they are read from, not what is shown or to whom. A domain administrator
|
||||
sees the lines of messages of their own domains, as now.
|
||||
- They now enter the database and so a full backup, which `log/` does not:
|
||||
submitting clients' addresses and SASL logins, message ids, receivers'
|
||||
replies. [security.md](../security.md) says so.
|
||||
- A line is matched from its head, so a reply cannot make itself a line of
|
||||
another message; Postfix does not log raw line breaks, and `postlogd` is the
|
||||
file's only writer.
|
||||
- A stranger on port 25 does not write to the table (above).
|
||||
|
||||
## Done when
|
||||
|
||||
A message sent before the last rotation shows its lines on its page; the lines
|
||||
disappear with the row; the page's wording no longer blames rotation for an
|
||||
ordinary case.
|
||||
|
||||
## Model routing (this plan)
|
||||
|
||||
| Role | Model |
|
||||
|------|-------|
|
||||
| All implementation (code, tests, docs) | **Opus** |
|
||||
| Technical review (reviewer ≠ author: a separate read-only agent) | **Opus** |
|
||||
| Security review | **Fable** |
|
||||
|
||||
## Implementation checklist
|
||||
|
||||
Target version cut: **`2.2.0`** (MINOR). One commit per step;
|
||||
[development.md](../development.md) § Plan checklists.
|
||||
|
||||
- [x] Migration `0003_send_log_lines.sql`, store methods (insert, insert once, lines of a row, prune) and their tests — **Opus**
|
||||
- [x] Log tailer: store queue-id lines, write once while catching up, prune with the send-log sweep — **Opus** (hourly as first built; six-hourly after the reviews)
|
||||
- [x] The two texts, mockup first: the *Outbound log* help topic and the empty note (`design:` commit) — **Opus** (the help topic is in the mockups; the empty note is not drawn there - the `out-message` mockup shows a message with lines - and changes with the page)
|
||||
- [x] Message page reads the table, falls back to the current `mail.log`; help text in `help.go` — **Opus**
|
||||
- [x] [architecture.md](../architecture.md), [guide.md](../guide.md), [schema-migrations.md](../schema-migrations.md) — **Opus** (schema-migrations.md went in with the migration)
|
||||
- [x] `go vet`, `go test`, e2e: a message's page still lists its lines after a forced rotation — **Opus** (on the test server: vet ok, 22 packages ok, e2e ok with the new step `delivery_log_survives_rotation`)
|
||||
- [x] Technical review — **Opus** (2026-10-12; no Critical or High. Fixed: one transaction per pass instead of a commit per line, the prune asks the send log once per queue id and runs six-hourly, the cut keeps characters whole, tests for what the comments claimed. Left as found and written down under *What is not kept*: lines the tailer never read, and a message in the queue across the upgrade)
|
||||
- [x] Security review — **Fable** (2026-10-12; no Critical or High. Fixed: port-25 mail is not stored, so a stranger cannot grow the table; [security.md](../security.md) states what a backup now carries. The window against queue-id reuse is described as the limit it is)
|
||||
@@ -1,6 +1,6 @@
|
||||
# Plan: inbound-antispam-panel
|
||||
|
||||
**Status:** agreed
|
||||
**Status:** shipped in `2.1.0` (2026-10-12)
|
||||
**Date:** 2026-08-19
|
||||
**Version:** `2.1.0` MINOR (opt-in; no change to outbound-only or inbound-without-filter paths). Was `1.10.0`; on 2026-10-09 the owner put the panel redesign first, so this ships on the 2.0 baseline and its screens are built from the redesign's component kit ([panel-redesign.md](panel-redesign.md) § Component kit).
|
||||
|
||||
@@ -34,8 +34,8 @@ Lists let the operator correct false positives/negatives (e.g. always allow
|
||||
|
||||
## Quarantine
|
||||
|
||||
**Not in this plan.** Held-mail review/release is tracked separately as roadmap
|
||||
candidate [inbound-quarantine](inbound-quarantine.md). The journal may still
|
||||
**Not in this plan.** Held-mail review/release is an idea kept outside the roadmap,
|
||||
[inbound-quarantine](inbound-quarantine.md). The journal may still
|
||||
record a `quarantine` **decision label** when the external filter reports it,
|
||||
without SelfPost storing the message.
|
||||
|
||||
@@ -57,8 +57,14 @@ smtpd_milters = { antispam, inbound-journal }
|
||||
```
|
||||
|
||||
Antispam runs first; inbound-journal runs at end-of-message on the **accept**
|
||||
path and reads rspamd-added headers (`X-Rspamd-Action`, `X-Rspamd-Score`, symbol
|
||||
summary when present).
|
||||
path and reads rspamd-added headers (`X-Rspamd-Action`, and `X-Spamd-Result`
|
||||
for the score and symbols - rspamd has no `X-Rspamd-Score`; the sidecar adds
|
||||
both through `deploy/antispam/rspamd/milter_headers.conf`). As built they are
|
||||
read only next to rspamd's signature of the message, `X-SelfPost-Filter`
|
||||
(owner, 2026-10-12: "А rspamd не может дописывать какой-то заголовок, по
|
||||
которому мы будем знать, что это заголовки нашего спам фильтра?") - an HMAC
|
||||
of the queue id, since a fixed mark could be written by a sender as easily as
|
||||
the verdict itself.
|
||||
|
||||
**Antispam rejects** do not reach end-of-message on the journal milter. v1 also
|
||||
tails `mail.log` for inbound `postfix/smtpd` **`milter-reject`** / **`reject`**
|
||||
@@ -73,7 +79,9 @@ antispam hook). Without an external filter:
|
||||
|
||||
- **accept** rows — message relayed (engine `selfpost`, detail empty);
|
||||
- **reject** / **tempfail** rows — from the mail.log tailer (Postfix policy,
|
||||
size limit, unknown recipient, etc.).
|
||||
unknown recipient, a sender domain that does not exist. As built, a refusal
|
||||
whose log line names no recipient - the size limit answered at MAIL FROM -
|
||||
and the per-client rate limit, which Postfix logs as a warning, make no row).
|
||||
|
||||
Panel copy explains that rspamd symbols appear only when the antispam milter is
|
||||
configured.
|
||||
@@ -83,6 +91,25 @@ configured.
|
||||
**Global administrator only** for journal and lists — same as inbound relay. No
|
||||
domain-admin access in v1.
|
||||
|
||||
*As built (2026-10-11):* the lists are the global role's; the **log follows
|
||||
the delegation of Inbound** that 2.0 introduced after this was written - a
|
||||
user with the `domain` role sees the rows of the inbound domains assigned to
|
||||
them and nothing else, without the *Allow / Deny sender* links. That is what
|
||||
the accepted `in-log` mockup draws (no "global role" in its head). The owner
|
||||
was told and can turn it back into a global-only page.
|
||||
|
||||
**Inbound › Log accepted by the owner on 2026-10-12** ("Этот экран
|
||||
утверждаем"), from screenshots of the page built from its fixture, after his
|
||||
changes to it: the sender column is headed *From*; *Why* is not a column but
|
||||
a sheet over the page, opened by the decision tag, holding a centred table of
|
||||
who decided, the verdict and symbols or the reply, and the subject whole
|
||||
(kept on the sheet at his word: the table clips it); a message Postfix itself
|
||||
refused offers no *Deny sender*. Inbound › Filter lists was redrawn at his word
|
||||
the same day - kind in a column of its own, the note on the entry's sheet,
|
||||
matches counted (§ Filter list usage) - and **accepted by him on 2026-10-12**
|
||||
("Да"), after a deny entry he added in the panel refused a test message with
|
||||
the list's reply and the journal recorded it.
|
||||
|
||||
### Retention
|
||||
|
||||
Mirror send-log retention:
|
||||
@@ -101,13 +128,53 @@ Mirror send-log retention:
|
||||
| Entry types | sender address, sender domain (`@domain` suffix), client IP / CIDR |
|
||||
| Precedence | **deny overrides allow** |
|
||||
| Scope | instance-wide (not per inbound domain in v1) |
|
||||
| Where applied | **rspamd map files** under `/data/antispam/` (allow.map, deny.map); documented compose fragment mounts them into the sidecar |
|
||||
| Where applied | **rspamd map files** under `/data/antispam/`, one per list and entry type (`allow_ip.map`, `allow_from.map`, `allow_domain.map`, `deny_…`) - an IP map and a sender map are different map types to rspamd; documented compose fragment mounts them into the sidecar. Deny is a prefilter reject, allow a weight of -100 (as built: two prefilters do not give "deny overrides allow") |
|
||||
| SelfPost pre-check | **out of v1** — lists live in rspamd only |
|
||||
| Edit access | global administrator only |
|
||||
| When inactive | lists editable in the panel but UI warns that sync applies only when `INBOUND_ANTISPAM_MILTER` is set; `postfix reload` / rspamd reload documented in guide |
|
||||
|
||||
List change → atomic map rewrite (same pattern as Postfix maps) → rspamd reload
|
||||
via documented operator step or sidecar `SIGHUP` in the compose fragment.
|
||||
List change → atomic map rewrite (same pattern as Postfix maps). No reload:
|
||||
rspamd rereads a changed map file by itself (about ten seconds on 3.11).
|
||||
|
||||
### Filter list usage (owner, 2026-10-12)
|
||||
|
||||
Each entry shows how often it decided a message (*Matched*, or *never*) and,
|
||||
on its sheet, when it last did - so that entries nothing matches any more can
|
||||
be removed. **The count is read off the journal on every view of the page;
|
||||
nothing is stored for it.** A journal row counts for an entry when the list
|
||||
decided the message - a rejected row whose reply is the deny rule's message,
|
||||
an accepted row whose verdict carries the allow rule's symbol
|
||||
(`SELFPOST_ALLOW_IP` / `_FROM` / `_DOMAIN`, which the journal milter always
|
||||
keeps) - and the entry fits the row's sender or client address the way rspamd
|
||||
matches it (`inbound.Lists.Usage`).
|
||||
|
||||
What follows from counting this way, and is accepted:
|
||||
|
||||
- the count reaches back as far as the journal does: its retention (90 days by
|
||||
default), which is what the sheet says;
|
||||
- a row counts for every fitting entry of the deciding list, since the journal
|
||||
does not say which of two the filter met first;
|
||||
- a message refused at end-of-message is one row whatever its recipient count,
|
||||
so a count is of messages, roughly.
|
||||
|
||||
**When to move to a counter kept with each entry** (a migration adding hits
|
||||
and last-hit to `inbound_spam_list`, bumped where the journal row is written;
|
||||
the owner asked for the condition to be written down). Any one of:
|
||||
|
||||
1. **The journal is cut by its 10 000-row cap and not by age.** Then it holds
|
||||
fewer days than its retention, the sheet's "in the last 90 days" is no
|
||||
longer true, and *never* stops being evidence that an entry is stale. The
|
||||
sign: the log tailer's prune line keeps reporting rows removed while the
|
||||
oldest row of Inbound › Log is younger than the retention.
|
||||
2. **The retention is set short** (towards its minimum of 7 days) for reasons
|
||||
of its own, privacy or disk. A week without a match does not make an entry
|
||||
stale.
|
||||
3. **Counts are wanted that outlive the journal** - since the entry was added.
|
||||
4. **The page is slow.** A view reads every journal row a list decided and
|
||||
tries it against every entry. With the cap that is at most 10 000 rows
|
||||
times the number of entries; past a few hundred entries, measure it.
|
||||
|
||||
Until then no migration is spent on it.
|
||||
|
||||
### Activation
|
||||
|
||||
@@ -200,7 +267,9 @@ flowchart LR
|
||||
## Security
|
||||
|
||||
- Same validation whitelists as inbound relay ([security.md](../security.md)).
|
||||
- Journal rows contain PII — retention and global-admin-only access.
|
||||
- Journal rows contain PII — retention, and access by reach: the global role
|
||||
reads all of it, a `domain` user the rows of assigned inbound domains (as
|
||||
built; see § RBAC).
|
||||
- Inbound journal milter fail-open must not block mail (backup-MX role).
|
||||
- List map writes injection-safe; deny wins over allow.
|
||||
|
||||
@@ -243,14 +312,14 @@ Target version cut: **`2.1.0`** (MINOR), after `2.0.0`. One commit per step;
|
||||
[development.md](../development.md) § Plan checklists.
|
||||
|
||||
- [x] Agree journal fields, retention, and RBAC (this plan § Decisions) — **Composer**
|
||||
- [ ] Migration `0002_inbound_spam_log.sql` on the 2.0 baseline — **Composer**
|
||||
- [ ] Inbound journal-milter + Postfix wiring — **Composer**
|
||||
- [ ] mail.log tailer: inbound reject rows — **Composer**
|
||||
- [ ] List CRUD + validation + atomic rspamd map sync — **Composer**
|
||||
- [ ] Panel: Inbound › Log and Inbound › Filter lists from the kit partials, mockups `in-log` / `in-filter-lists` (add client IP and engine columns mockup-first), evidence pairs — **Composer**
|
||||
- [ ] Backup includes `/data/antispam/` — **Composer**
|
||||
- [ ] Unit + handler tests — **Composer**
|
||||
- [ ] [guide.md](../guide.md), [architecture.md](../architecture.md) — **Composer**
|
||||
- [ ] `go vet`, `go test`, e2e if applicable — **Composer**
|
||||
- [ ] Technical review of inbound antispam changes — **Opus**
|
||||
- [ ] Security review inbound antispam path — **Fable**
|
||||
- [x] Migration `0002_inbound_spam_log.sql` on the 2.0 baseline — **Composer**
|
||||
- [x] Inbound journal-milter + Postfix wiring — **Composer**
|
||||
- [x] mail.log tailer: inbound reject rows — **Composer**
|
||||
- [x] List CRUD + validation + atomic rspamd map sync — **Composer**
|
||||
- [x] Panel: Inbound › Log and Inbound › Filter lists from the kit partials, mockups `in-log` / `in-filter-lists` (add client IP and engine columns mockup-first), evidence pairs — **Composer**
|
||||
- [x] Backup includes `/data/antispam/` — **Composer** (it always did: a backup walks all of `/data`; now held by a test, and the guide says the `antispam/` fragment directory is not in it)
|
||||
- [x] Unit + handler tests — **Composer** (written with each step: store, milter, tailer, lists and their usage, view fixtures held to the outlines, handlers for both roles)
|
||||
- [x] [guide.md](../guide.md), [architecture.md](../architecture.md) — **Composer** (kept up with each step; [security.md](../security.md) follows the security review)
|
||||
- [x] `go vet`, `go test`, e2e if applicable — **Composer** (on the test server for every step; last on `7192c38`: vet ok, 22 packages ok, e2e ok, which relays, refuses, reads the log page and edits a list through the panel)
|
||||
- [x] Technical review of inbound antispam changes — **Opus** (2026-10-12; findings fixed: client address of a connection's later messages, the lists' writer under a lock, the tailer's queue-id memory, hold rows written twice, the cap by time; and, outside the feature, the level-2 limit and client-address list skipped after a connection's first message)
|
||||
- [x] Security review inbound antispam path — **Fable** (2026-10-12; no Critical or High; HELO-forged journal rows and the six-hourly row cap fixed; a sender's own verdict while the filter is down closed by the filter's signature, the owner's idea; accepted risks recorded in [security.md](../security.md))
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Plan: inbound-quarantine
|
||||
|
||||
**Status:** candidate — design TBD; needs explicit agreement before any code
|
||||
**Status:** an idea to think about, not on the roadmap (owner, 2026-10-12) — design TBD; needs explicit agreement before any code
|
||||
**Version:** TBD (`1.x` MINOR if opt-in and backward-compatible; may warrant `2.x`
|
||||
if it changes inbound storage or backup contract).
|
||||
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
# Plan: panel-notifications
|
||||
|
||||
**Status:** agreed 2026-10-12 - the design questions were closed with the owner and he took it into work ("Делаем")
|
||||
**Date:** 2026-10-12
|
||||
**Version:** `2.3.0` MINOR (one migration; opt-out by a switch).
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
The panel writes to its administrators when something needs them, instead of
|
||||
waiting to be opened.
|
||||
|
||||
## Decisions (owner, 2026-10-12)
|
||||
|
||||
### Events of the first version
|
||||
|
||||
Four, each of them something the panel can already tell and a person has to
|
||||
act on:
|
||||
|
||||
| Event | Who is told |
|
||||
|---|---|
|
||||
| A sending domain's DNS record stopped matching (SPF, DKIM, DMARC) | global users, and `domain` users the domain is assigned to |
|
||||
| A level-2 rate limit refused mail | the same, for the domain of the limit |
|
||||
| The TLS certificate is about to expire | global users |
|
||||
| A message has been in the queue for more than **three days** | global users |
|
||||
|
||||
Three days, not one (owner: "one day is not that much"): Postfix keeps a
|
||||
message five, so two are left to act before it bounces. The threshold is a
|
||||
constant, not a setting.
|
||||
|
||||
Left for later: a process or milter socket down (a relay in that state often
|
||||
cannot send), an inbound upstream unreachable, DMARC failures appearing
|
||||
(noisy). Mail held in quarantine is not an event: quarantine is an idea
|
||||
outside the roadmap.
|
||||
|
||||
### Sender and path
|
||||
|
||||
From `postmaster@<SELFPOST_HOSTNAME>`, submitted locally. Not from one of the
|
||||
sending domains (owner): with several domains the choice is arbitrary, and a
|
||||
message about the certificate has no business coming from a domain.
|
||||
|
||||
These messages have to arrive (owner: "we need the best deliverability for
|
||||
them"), so the server's own name gets what a sending domain has:
|
||||
|
||||
- a **DKIM key for the hostname**, generated by the panel, with which
|
||||
notifications are signed - no domain has to be chosen for it. An SPF record
|
||||
alone would pass DMARC, but not after the user's mailbox forwards the
|
||||
message on, and large receivers treat unsigned mail more strictly;
|
||||
- an **SPF record** for the hostname, `v=spf1 a -all`.
|
||||
|
||||
The records themselves are under **Server › Settings** (owner: "the DNS
|
||||
records go to Settings, Health has only the status"), in a box of their own
|
||||
drawn with the record component of a domain's DNS page: host, value to
|
||||
publish, copy buttons, state. **Server › Health** has a card of its own,
|
||||
*Notification mail*, with the state of each record and a link to Settings.
|
||||
A DMARC record for the hostname is not asked for: the name inherits its
|
||||
parent domain's policy, and with SPF and DKIM aligned it passes any.
|
||||
Without the records notifications are still sent, and the card warns.
|
||||
|
||||
Notifications are the panel's own mail: they are not rows of the Outbound log
|
||||
and count against no level-2 limit.
|
||||
|
||||
A relay that cannot send cannot say so; none of the four events is "the relay
|
||||
is down", and that is why.
|
||||
|
||||
### Who chooses what
|
||||
|
||||
- One switch for the instance under **Server › Settings**.
|
||||
- One checkbox per user under **Account**, "send me notifications" - on when
|
||||
the profile has an e-mail. No choice per event: four events, none of them
|
||||
noise. Per-event choice comes when there are more and some are.
|
||||
- A user without an e-mail gets nothing.
|
||||
|
||||
### When a message is sent
|
||||
|
||||
A background check once an hour for DNS, certificate and queue.
|
||||
|
||||
| Event | Sent | "Cleared" message |
|
||||
|---|---|---|
|
||||
| DNS mismatch | after two checks in a row found it (it has held for an hour). A resolver that did not answer is not a mismatch | after two good checks in a row |
|
||||
| Certificate | 14 days before expiry, and again 3 days before if it is still the same certificate | none |
|
||||
| Level-2 limit refused mail | at the first refusal for a domain or application, then at most once a day for the same limit | none |
|
||||
| Queue older than three days | when such a message appears | one, when none is left |
|
||||
|
||||
Event state is kept in the database, so a restart announces nothing a second
|
||||
time.
|
||||
|
||||
## To settle while building
|
||||
|
||||
- How the panel submits: `sendmail` as the unprivileged panel user, and what
|
||||
that needs of the image.
|
||||
- Whether a notification carries a link to the panel - there is no setting
|
||||
for the panel's own address.
|
||||
- Where the refusal of a level-2 limit is noticed: in the journal milter as
|
||||
it refuses, or from the `rejected` rows of the send log on the hourly check.
|
||||
- Schema: the user's checkbox, the instance switch, the event state table.
|
||||
|
||||
## Screens
|
||||
|
||||
Not drawn. The Notifications box was taken out of the `account` and
|
||||
`settings` mockups on 2026-10-10 (owner: 2.0 ships no box for a feature that
|
||||
is not built). Drawing them again - one checkbox on Account, one switch on
|
||||
Server › Settings, the SPF check on Server › Health - is the first step, shown
|
||||
to the owner before any code ([panel-redesign.md](panel-redesign.md)
|
||||
§ Enforcement).
|
||||
|
||||
## Implementation checklist
|
||||
|
||||
- [x] Mockups: Account checkbox, Settings switch and records, Health status card, help topic; accepted by the owner 2026-10-12 (`design:` commit)
|
||||
- [x] Migration and store: user flag, instance switch, event state
|
||||
- [x] Sending: local submission from `postmaster@<hostname>`, message text (`sendmail` as the panel user; signed through `non_smtpd_milters` with a key of the server's own name; no link in the text - the panel does not know its own address)
|
||||
- [x] Background checker: DNS, certificate, queue; the rules of *When a message is sent*
|
||||
- [x] Level-2 refusal event (from the `rejected` rows of the send log, looked at every minute)
|
||||
- [x] Panel: Account, Settings, Health card, help
|
||||
- [x] guide.md, architecture.md, security.md, schema-migrations.md
|
||||
- [x] `go vet`, `go test`, e2e: a level-2 refusal reaches a mailbox as a signed notification (on the test server: vet ok, 23 packages ok, e2e ok; "once" and "clears" are held by the unit tests of `internal/notify`)
|
||||
- [ ] Technical review
|
||||
- [ ] Security review
|
||||
@@ -235,9 +235,9 @@ the first step of that feature's plan — not invented during implementation.
|
||||
| Roadmap item | Where it lands | Built from | Drawn |
|
||||
|---|---|---|---|
|
||||
| inbound-antispam-panel (agreed, `2.1.0`) | journal = **Inbound › Log** (add client IP and engine to the mockup's columns; `quarantine` joins the decisions); lists = new **Inbound › Filter lists**, global-only because the lists are instance-wide | list layout: add form + two tables, `status_tag`, filter form in `box_head`; the log links *Allow / Deny sender* to the lists | `in-log`, `in-filter-lists` |
|
||||
| inbound-quarantine (candidate) | new **Inbound › Quarantine**, list + detail with *Release* / *Discard* | list layout; detail like `out-message` with `facts`, `log_pane` for headers only, Release beside a `sp-danger-zone` Discard | `in-quarantine`, `in-quarantine-message` — the open questions of its plan (where mail lives, what release means, retention, RBAC) stay open; the mockup assumes SelfPost-held mail, release to upstream, 14 days, delegated like the rest of Inbound |
|
||||
| inbound-quarantine (an idea, off the roadmap since 2026-10-12) | new **Inbound › Quarantine**, list + detail with *Release* / *Discard* | list layout; detail like `out-message` with `facts`, `log_pane` for headers only, Release beside a `sp-danger-zone` Discard | `in-quarantine`, `in-quarantine-message` — the open questions of its plan (where mail lives, what release means, retention, RBAC) stay open; the mockup assumes SelfPost-held mail, release to upstream, 14 days, delegated like the rest of Inbound |
|
||||
| preflight (candidate) | new **Server › Preflight**, next to Health: Health is what runs now, Preflight is the deeper on-demand installation check with a test e-mail form | `postmark` verdict, a table of checks with `status_tag` and a *What to do* column, one form box for the test e-mail | `preflight` |
|
||||
| panel-notifications (candidate) | per-user event choice on **Account**; one switch under **Server › Settings** | checkboxes in a box, `help` | **no** — drawn once, taken out of `account` and `settings` on 2026-10-10 (owner): 2.0 ships no box for a feature that is not built; redrawn as the first step of its plan |
|
||||
| panel-notifications (candidate) | one checkbox per user on **Account** (no choice per event, owner 2026-10-12); one switch under **Server › Settings** | a checkbox in a box, `help` | **no** — drawn once, taken out of `account` and `settings` on 2026-10-10 (owner): 2.0 ships no box for a feature that is not built; redrawn as the first step of its plan |
|
||||
| password-reset (candidate) | a link on sign-in, two signed-out screens (request, set new password) | the signed-out layout | **no** |
|
||||
| delivery-log-storage (candidate) | the *Delivery log* box of **Outbound › Log › message**, filled from a table instead of a grep of today's `mail.log` | `log_pane`, unchanged | `out-message` |
|
||||
| csrf-tokens | no screen; every form gains a hidden field — forms are rendered through one helper so the token cannot be forgotten | — | n/a |
|
||||
@@ -539,5 +539,5 @@ only when both are done. Every stage-2 step ends with the evidence of
|
||||
order suits development, preflight, delivery-log-storage, panel-notifications,
|
||||
password-reset (two screens to draw first), queue as a table with the inbound
|
||||
queue, csrf-tokens
|
||||
(one hidden field in the form partial once security.md is settled),
|
||||
inbound-quarantine after its open questions.
|
||||
(one hidden field in the form partial once security.md is settled).
|
||||
inbound-quarantine is an idea outside the roadmap.
|
||||
|
||||
+90
-52
@@ -30,24 +30,28 @@ in `git log` and [CHANGELOG.md](../CHANGELOG.md).
|
||||
|---|---|---|---|---|
|
||||
| preflight | Installation check page (`/preflight`) | candidate | — | [plans/preflight.md](plans/preflight.md) |
|
||||
| contributing | `CONTRIBUTING.md` | candidate | — | — |
|
||||
| inbound-antispam-panel | Inbound antispam journal + allow/deny lists | agreed | 1/12 | [plans/inbound-antispam-panel.md](plans/inbound-antispam-panel.md) |
|
||||
| inbound-quarantine | Inbound spam quarantine (hold / review / release) | candidate | — | [plans/inbound-quarantine.md](plans/inbound-quarantine.md) |
|
||||
| inbound-antispam-panel | Inbound antispam journal + allow/deny lists | shipped `2.1.0` | 12/12 | [plans/inbound-antispam-panel.md](plans/inbound-antispam-panel.md) |
|
||||
| csrf-tokens | Per-session CSRF tokens on state-changing forms | candidate | — | — |
|
||||
| template-data-typing | Typed structs for template data instead of `map[string]any` | candidate | — | — |
|
||||
| structured-logging | `log/slog` with levels and fields | candidate | — | — |
|
||||
| review-2026-08-followups | Remaining findings of the 2026-08-19 code review | candidate | — | — |
|
||||
| panel-notifications | E-mail notifications about important events | candidate | — | — |
|
||||
| delivery-log-storage | Keep a message's `mail.log` lines with its send-log row | candidate | — | — |
|
||||
| panel-notifications | E-mail notifications about important events | in progress | 8/10 | [plans/panel-notifications.md](plans/panel-notifications.md) |
|
||||
| delivery-log-storage | Keep a message's `mail.log` lines with its send-log row | shipped `2.2.0` | 8/8 | [plans/delivery-log-storage.md](plans/delivery-log-storage.md) |
|
||||
| password-reset | Password reset by e-mail | candidate | — | — |
|
||||
| phone-desktop-mode | A phone browser in "desktop site" mode gets the phone navigation | candidate | — | — |
|
||||
| panel-redesign | Panel redesign (accepted mockups, design contract) | shipped `2.0.0` | 27/27 | [plans/panel-redesign.md](plans/panel-redesign.md) |
|
||||
| schema-squash | Squash SQLite migrations into a 2.x baseline | built in panel-redesign (stage 1) | — | [plans/panel-redesign.md](plans/panel-redesign.md) |
|
||||
|
||||
**Recommended order** (not binding; owner, 2026-10-09: feature order follows
|
||||
development convenience): **panel-redesign** shipped as `2.0.0` (2026-10-11); the next stage is
|
||||
**inbound-antispam-panel** (`2.1.0`), built in the new component kit —
|
||||
styling its two screens once instead of twice. panel-docs shipped in
|
||||
development convenience): **panel-redesign** shipped as `2.0.0` (2026-10-11) and
|
||||
**inbound-antispam-panel** as `2.1.0` (2026-10-12), built in the new component
|
||||
kit — styling its two screens once instead of twice. **delivery-log-storage**
|
||||
shipped as `2.2.0` the same day. Agreed after it: **panel-notifications**
|
||||
(`2.3.0`); every other remaining
|
||||
item is a candidate. panel-docs shipped in
|
||||
[CHANGELOG.md](../CHANGELOG.md) `[1.8.0]`; dmarc-reports in `[1.7.0]`
|
||||
(security review of the ingest path pending); domain-stats-auto-ratelimit in
|
||||
(its ingest path first ran in `2.0.0` and was reviewed on 2026-10-11, see
|
||||
[security.md](security.md)); domain-stats-auto-ratelimit in
|
||||
`[1.6.0]`; send-log-retention in `[1.5.0]`; inbound-relay in `[1.4.0]`;
|
||||
queue-retries in `[1.3.1]`; the 2026-08-13 full-tree review follow-ups are in
|
||||
`[1.3.0]`. Candidates need explicit agreement before they join the queue.
|
||||
@@ -121,8 +125,8 @@ action) plus editable allow/deny lists that tune the external filter — without
|
||||
shipping rspamd or another engine inside the image.
|
||||
|
||||
**Boundary:** metadata journal and list sync only; SelfPost does not run the
|
||||
filter ([product.md](product.md)). Quarantine:
|
||||
[inbound-quarantine](#inbound-quarantine). Design closed in
|
||||
filter ([product.md](product.md)). Quarantine is not on the roadmap
|
||||
([Ideas](#ideas-not-on-the-roadmap)). Design closed in
|
||||
[plans/inbound-antispam-panel.md](plans/inbound-antispam-panel.md) § Decisions.
|
||||
|
||||
**Done when:** with inbound relay enabled, the operator sees journal rows
|
||||
@@ -133,28 +137,7 @@ hook is set.
|
||||
**Dependencies / risks:** inbound relay `[1.4.0]`; rspamd map/header coupling;
|
||||
journal PII; global-admin-only RBAC.
|
||||
|
||||
**Version:** `2.1.0` MINOR (was `1.10.0`; reordered after the redesign on 2026-10-09); **agreed**.
|
||||
|
||||
---
|
||||
|
||||
## inbound-quarantine
|
||||
|
||||
**Goal:** optional hold for suspicious inbound mail — review in the panel, then
|
||||
release to upstream, discard, or tune lists. Design not settled.
|
||||
|
||||
**Boundary:** expands inbound from pure relay toward short-lived message
|
||||
storage; may conflict with "no mailboxes" in [product.md](product.md) until
|
||||
product explicitly agrees. Alternatives (rspamd-only quarantine vs SelfPost
|
||||
store) are listed in [plans/inbound-quarantine.md](plans/inbound-quarantine.md).
|
||||
|
||||
**Done when:** TBD after open questions in the plan are answered.
|
||||
|
||||
**Dependencies / risks:** inbound relay `[1.4.0]`; storage/retention/backup
|
||||
size; malware in held MIME; overlap with
|
||||
[inbound-antispam-panel](#inbound-antispam-panel). No implementation checklist
|
||||
yet.
|
||||
|
||||
**Version:** TBD; `candidate` until explicitly agreed.
|
||||
**Version:** shipped `2.1.0` (2026-10-12), MINOR (was `1.10.0`; reordered after the redesign on 2026-10-09).
|
||||
|
||||
---
|
||||
|
||||
@@ -348,38 +331,43 @@ time to final status plus the queue lifetime), not to the id alone; the table
|
||||
grows with the send log (small — a few hundred bytes per message) and is
|
||||
covered by the same retention.
|
||||
|
||||
**Version:** `2.x` MINOR; `candidate`. Fits after the redesign, built against
|
||||
the kit's `log_pane`.
|
||||
**Version:** shipped `2.2.0` (2026-10-12), MINOR; option 1. Decisions and
|
||||
checklist: [plans/delivery-log-storage.md](plans/delivery-log-storage.md).
|
||||
|
||||
---
|
||||
|
||||
## panel-notifications
|
||||
|
||||
**Goal:** the panel tells its administrators when something needs them instead
|
||||
of waiting to be opened: a DNS record stops matching, a domain or application
|
||||
hits its rate limit and mail is refused, an inbound upstream is unreachable,
|
||||
DMARC failures appear, mail is held in quarantine; for the global
|
||||
administrator also the TLS certificate nearing expiry, a process or milter
|
||||
socket down, a queue older than a day.
|
||||
of waiting to be opened. First version, four events: a sending domain's DNS
|
||||
record stops matching, a level-2 rate limit refuses mail, the TLS certificate
|
||||
is about to expire, a message has been in the queue for more than three days.
|
||||
|
||||
**Boundary:** e-mail only, sent through the relay itself from
|
||||
`postmaster@<hostname>`. Each user picks events under **Account**; a domain
|
||||
administrator hears only about assigned domains. One message when a problem
|
||||
starts and one when it clears — no streams, no digests in v1. One instance
|
||||
switch under **Server › Settings**. No webhooks, no chat integrations.
|
||||
**Boundary:** e-mail only, from `postmaster@<hostname>`, submitted locally -
|
||||
not from one of the sending domains. One checkbox per user under **Account**
|
||||
(no choice per event), one instance switch under **Server › Settings**; a
|
||||
domain administrator hears only about assigned domains. No streams, no
|
||||
digests, no webhooks, no chat integrations. Not in the first version: a
|
||||
process or milter down, an inbound upstream unreachable, DMARC failures.
|
||||
|
||||
**Done when:** an event from the list reaches the account e-mail of every user
|
||||
who chose it and may see its subject, once on start and once on clear; a user
|
||||
without an e-mail gets nothing; the switch turns it all off.
|
||||
**Done when:** each of the four events reaches the account e-mail of every
|
||||
user who has notifications on and may see its subject, by the rules of the
|
||||
plan (a "cleared" message for DNS and the queue only); a user without an
|
||||
e-mail gets nothing; the switch turns it all off.
|
||||
|
||||
**Dependencies / risks:** the account e-mail (2.0 baseline, see
|
||||
[plans/panel-redesign.md](plans/panel-redesign.md) § Account e-mail); a relay
|
||||
that cannot send cannot report that it cannot send — server events need the
|
||||
direct-to-MX path or stay panel-only; event state must survive restarts so a
|
||||
reboot does not re-announce everything. Screens are drawn: `account`,
|
||||
`settings`.
|
||||
that cannot send cannot report that it cannot send - none of the four events
|
||||
is "the relay is down"; mail from the bare hostname needs an SPF record for
|
||||
it to be believed, which Server › Health will check; event state must survive
|
||||
restarts so a reboot does not re-announce everything. The screens are not
|
||||
drawn: the Notifications box was taken out of `account` and `settings` on
|
||||
2026-10-10 and is redrawn as the first step.
|
||||
|
||||
**Version:** `2.x` MINOR; `candidate`.
|
||||
**Version:** `2.3.0` MINOR; agreed 2026-10-12 - the design questions were closed
|
||||
with the owner on 2026-10-12
|
||||
([plans/panel-notifications.md](plans/panel-notifications.md)) and he took it into
|
||||
work the same day.
|
||||
|
||||
---
|
||||
|
||||
@@ -405,6 +393,41 @@ goes and today Account saves it without the password.
|
||||
|
||||
---
|
||||
|
||||
## phone-desktop-mode
|
||||
|
||||
**Problem (owner, 2026-10-11):** the panel opened on a phone with the
|
||||
browser's "desktop site" switched on does not become the desktop panel.
|
||||
Seen on Server › Health at `selfpost.mixfed.ru`: the page body is laid out in
|
||||
columns (Machine and Processes side by side, three cards in the next row),
|
||||
while the navigation is the phone one - not a bar but a stack with every
|
||||
section open (Outbound, Inbound, Server, the user menu), filling the first
|
||||
screen before the page starts. The content also sits flush against the left
|
||||
edge of the screen.
|
||||
|
||||
**Goal:** find out why, then decide what the panel should be in that mode -
|
||||
the desktop panel, or a phone panel that is at least consistent.
|
||||
|
||||
**Where to look first (a guess, not checked):** "desktop site" on Android
|
||||
lays a page out about 980 px wide. Bulma turns the navigation into a bar at
|
||||
1024 px and puts columns side by side from 769 px, so 980 px falls between
|
||||
the two: desktop columns under a touch navigation. `panel.css` has its own
|
||||
breakpoints at 768, 1023 and 1407 px. Check the width the phone really
|
||||
reports, why every dropdown is open rather than folded behind the burger,
|
||||
and where the left padding went.
|
||||
|
||||
**Boundary:** the navigation and page gutter between 769 and 1023 px, which
|
||||
is also a tablet in portrait and a narrow desktop window - the same defect is
|
||||
expected there. A design change is mockup-first
|
||||
([plans/panel-redesign.md](plans/panel-redesign.md) § Enforcement).
|
||||
|
||||
**Done when:** the cause is written down here, and the panel at that width
|
||||
has one coherent layout, checked on a phone in "desktop site" mode and in a
|
||||
980 px window.
|
||||
|
||||
**Version:** `2.x` PATCH or MINOR; `candidate`.
|
||||
|
||||
---
|
||||
|
||||
## panel-redesign
|
||||
|
||||
**Goal:** a panel an operator opens a few times a year and still finds their
|
||||
@@ -482,3 +505,18 @@ explicit major). Squashing the current short chain is not a reason to cut 2.x on
|
||||
its own. A missed gate leaves a mid-chain 1.x database silently stuck.
|
||||
**Version:** `2.x` major only; not a 1.x item.
|
||||
|
||||
---
|
||||
|
||||
## Ideas, not on the roadmap
|
||||
|
||||
Thoughts kept so they are not lost. They are not candidates: nothing here is
|
||||
waiting for an OK, has a version or takes a place in the order above. An idea
|
||||
joins the index only when the owner says so.
|
||||
|
||||
- **Inbound quarantine** (moved here by the owner on 2026-10-12: "an idea to
|
||||
think about") - hold suspicious inbound mail, review it in the panel, then
|
||||
release it to the upstream or discard it. It would take the relay from
|
||||
passing mail on to storing it for a while, against "no mailboxes" in
|
||||
[product.md](product.md). Notes and open questions:
|
||||
[plans/inbound-quarantine.md](plans/inbound-quarantine.md); the screens
|
||||
`in-quarantine` and `in-quarantine-message` are drawn and stay unbuilt.
|
||||
|
||||
+19
-10
@@ -17,8 +17,8 @@ When you add or review a migration, update this file in the same change.
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Chain head | `user_version = 1` |
|
||||
| Files | `0001_init.sql` (the 2.0 baseline) |
|
||||
| Chain head | `user_version = 4` |
|
||||
| Files | `0001_init.sql` (the 2.0 baseline), `0002_inbound_spam_log.sql`, `0003_send_log_lines.sql`, `0004_notifications.sql` |
|
||||
| File mark | `PRAGMA application_id = 0x53503230` ("SP20") |
|
||||
| Database file | `/data/selfpost.db` (bind mount) |
|
||||
| Compatibility | 2.x MINOR releases must boot a `2.0.0` data directory. **Nothing is carried over from 1.x.** |
|
||||
@@ -60,10 +60,11 @@ What that means in practice:
|
||||
|
||||
**Implication:** once `2.0.0` is released, `0001_init.sql` is never edited
|
||||
again and no file is deleted, renamed or reordered — every 2.x database has
|
||||
already run it. Until that release the baseline is still being written:
|
||||
later steps of the redesign and the features queued behind it (the next one
|
||||
is `0002_inbound_spam_log.sql` of inbound-antispam-panel, `2.1.0`) add files
|
||||
after it.
|
||||
already run it. `2.0.0` and `2.0.1` shipped with the baseline alone; the
|
||||
features after them add files, the first being `0002_inbound_spam_log.sql` of
|
||||
inbound-antispam-panel (`2.1.0`), then `0003_send_log_lines.sql` of
|
||||
delivery-log-storage (`2.2.0`) and `0004_notifications.sql` of
|
||||
panel-notifications (`2.3.0`).
|
||||
|
||||
---
|
||||
|
||||
@@ -72,18 +73,21 @@ after it.
|
||||
| Ver | File | Shipped | Kind | Summary |
|
||||
|-----|------|---------|------|---------|
|
||||
| 1 | `0001_init.sql` | 2.0.0 | DDL | The whole 2.0 schema, below |
|
||||
| 2 | `0002_inbound_spam_log.sql` | 2.1.0 | DDL | `inbound_spam_log`, `inbound_spam_list` and two indexes; nothing existing changes |
|
||||
| 3 | `0003_send_log_lines.sql` | 2.2.0 | DDL | `send_log_lines` and one index; nothing existing changes |
|
||||
| 4 | `0004_notifications.sql` | 2.3.0 | DDL | `users.notify` (default 1: every user starts with notifications on) and `notify_state` |
|
||||
|
||||
**Kind:** *DDL* — schema only; *data* — `INSERT`/`UPDATE` that must stay correct
|
||||
for operators upgrading from older 2.x images.
|
||||
|
||||
---
|
||||
|
||||
## Schema at head (v1)
|
||||
## Schema at head (v2)
|
||||
|
||||
| Table | Role |
|
||||
|-------|------|
|
||||
| `settings` | Key/value settings of the instance — what is true whoever is signed in (send-log retention). Nothing that belongs to a user |
|
||||
| `users` | Panel logins. `role` is `global` or `domain`; `email` is the user's own address; `all_domains` / `all_inbound_domains` widen a `domain` user to every domain of that direction |
|
||||
| `users` | Panel logins. `role` is `global` or `domain`; `email` is the user's own address; `notify` is the user's own switch for notification e-mails (v4); `all_domains` / `all_inbound_domains` widen a `domain` user to every domain of that direction |
|
||||
| `sessions` | Panel login sessions (token hash, owning user by id and deleted with them, sliding idle expiry) |
|
||||
| `domains` | Sending (outbound) domains. `dmarc_rua` is where aggregate reports go (`''` = no reports) |
|
||||
| `user_domains` | Outbound domains assigned to a `domain` user |
|
||||
@@ -98,9 +102,14 @@ for operators upgrading from older 2.x images.
|
||||
| `user_inbound_domains` | Inbound domains assigned to a `domain` user — granted separately from the outbound ones |
|
||||
| `dmarc_reports` | Parsed aggregate report summaries |
|
||||
| `dmarc_report_records` | Per-source rows inside a report |
|
||||
| `inbound_spam_log` | Inbound antispam journal: one row per recipient of a message offered on port 25, with the decision (`accept`, `reject`, `tempfail`, `quarantine`) and who made it (`rspamd`, `postfix`, `selfpost`). Metadata only (v2) |
|
||||
| `inbound_spam_list` | Allow / deny entries for the external filter (`address`, `domain`, `ip`), instance-wide; unique per list (v2) |
|
||||
| `notify_state` | What the panel has already written to its users about, one row per subject (`dns:<domain>`, `cert`, `queue`, `limit:<domain>:<login>`), so that each thing is said once and a restart says nothing again (v4) |
|
||||
| `send_log_lines` | The `mail.log` lines Postfix wrote under a queue id, with the line's own time; what a message's page shows once the log has rotated. No reference to `send_log`: a line belongs to a row by queue id and time, and goes when no row carries its id (v3) |
|
||||
|
||||
`TestBaselineSchema` (`internal/store/schema_test.go`) pins every table, column
|
||||
and index of this list.
|
||||
and index of this list; `TestOpenMigratesA20Database` opens a database that
|
||||
holds the baseline alone.
|
||||
|
||||
### What changed against the last 1.x schema
|
||||
|
||||
@@ -129,7 +138,7 @@ for the domain. Nothing refers to a user: deleting one changes no domain.
|
||||
|
||||
1. Add `internal/store/migrations/00NN_short_name.sql` (next number).
|
||||
2. Update the table in [Migration chain](#migration-chain), the snapshot at the
|
||||
top, and the affected rows of [Schema at head](#schema-at-head-v1).
|
||||
top, and the affected rows of [Schema at head](#schema-at-head-v2).
|
||||
3. Update `TestBaselineSchema` if tables, columns or indexes change, and add a
|
||||
test that opens a database at the previous version and migrates it.
|
||||
4. A *data* migration states in a comment which 2.x versions it is written for.
|
||||
|
||||
+140
-5
@@ -26,7 +26,31 @@ the user form did not end that user's sessions, internal errors reached forms a
|
||||
(see [Authentication and sessions](#authentication-and-sessions) and [Errors
|
||||
shown to users](#errors-shown-to-users)); two deliberate behaviours are
|
||||
recorded under Accepted risks (upstream of a delegated inbound domain, profile
|
||||
saved without the password).
|
||||
saved without the password). The 2026-10-11 review (Fable) covered the DMARC
|
||||
report path from port 25 to the report pages, which ran for the first time in
|
||||
2.0.0 - the 2026-08-19 pass read code that Postfix could not yet start: no
|
||||
Critical or High finding; a report's size was bounded only as bytes (now 5 000
|
||||
records, 255 bytes a field), the retention cap was one for the whole server
|
||||
(now per domain), the delivery command could run a hundred at once (now one),
|
||||
and the reason logged for a dropped message carried the sender's bytes as they
|
||||
came (now cut and reduced to printable ASCII). The 2026-10-12 reviews of
|
||||
inbound-antispam-panel (`2.1.0`: the inbound journal milter, refusals read from
|
||||
`mail.log`, the filter lists and their pages) were two, by different models
|
||||
over the same diff. The technical one (Opus) found, outside the feature and
|
||||
there since `1.9.0`, that **the level-2 rate limit and an application's
|
||||
client-address list held only for the first message of a connection** - see
|
||||
[Mail path](#mail-path-security-relevant) - which was reproduced on the test
|
||||
stand and fixed. The security one (Fable) found no Critical or High: a client's
|
||||
HELO name could supply the sender, recipient and reply of a journal row and so
|
||||
make a refusal look like the deny list's (the reply now ends at Postfix's own
|
||||
`; from=<`); the journal's row cap was applied only every six hours, between
|
||||
which refusals were written at whatever rate a stranger sent recipients (the
|
||||
tailer now cuts to the cap every 500 rows); and two behaviours are recorded
|
||||
under Accepted risks (the allow list keyed on an address anyone can claim, the
|
||||
list files readable by anyone). A third it named - with the filter down, a
|
||||
sender's own verdict headers were recorded as the filter's - was closed the
|
||||
same day at the owner's suggestion: the filter signs each message and a
|
||||
verdict is read only next to its signature.
|
||||
(2) **Accepted risks** —
|
||||
deliberate departures beyond the mandatory, recorded so the decision is not
|
||||
lost.
|
||||
@@ -113,6 +137,8 @@ assigned outbound domains and inbound domains **separately**, each list with
|
||||
| Outbound: domain page, applications, Domain settings (report address, rate limit, export), log | every domain | assigned outbound domains |
|
||||
| Outbound › DMARC reports | every report, with the server's ingest statistics | reports of assigned domains, without the ingest box |
|
||||
| Inbound: domain page, upstream, recipients | every inbound domain | assigned inbound domains |
|
||||
| Inbound › Log | every row, including those of a domain deleted since | rows of assigned inbound domains (by name), without the links into the filter lists |
|
||||
| Inbound › Filter lists; the inbound log's retention | yes | no — 404 |
|
||||
| Add or delete a domain, either direction; import a domain | yes | no |
|
||||
| Account, Help | yes | yes |
|
||||
|
||||
@@ -171,6 +197,41 @@ operator can act on it.
|
||||
refuse unknown addresses at RCPT (no backscatter). OpenDKIM is not attached
|
||||
on inbound. An optional antispam milter is inbound-only; default action is
|
||||
fail-open (`accept`).
|
||||
- **The panel's milters judge every message of a connection.** Postfix
|
||||
introduces a client to a milter once per connection, and the milter library
|
||||
replaces the milter's session after any reply that is not "continue". Until
|
||||
`2.1.0` the journal milter knew the client's address only from that
|
||||
introduction, and both of its checks passed a message that came without one:
|
||||
a second `MAIL FROM` after a refused one, or any message after the first on
|
||||
one connection, got past the level-2 limit and past an application's
|
||||
client-address list. Now the address comes with every `MAIL FROM`
|
||||
(`milter_mail_macros` carries `{client_addr}`), the level-2 limit does not
|
||||
ask for it at all, an application with a client-address list is refused when
|
||||
no address is known, and neither milter replies anything but "continue" to a
|
||||
message it lets through. `test/e2e` sends two `MAIL FROM` on one connection.
|
||||
- **The inbound journal is written from what strangers send.** The inbound
|
||||
journal milter is the last on port 25, read-only and attached fail-open; it
|
||||
keeps of the filter's verdict only what its patterns recognise. **A verdict
|
||||
is read only next to the filter's signature.** The filter is attached so
|
||||
that mail flows when it is down, and then nothing removes verdict headers a
|
||||
sender wrote himself. So rspamd adds a third header, `X-SelfPost-Filter`:
|
||||
the HMAC-SHA256 of the message's queue id under a key SelfPost generates
|
||||
(`data/antispam/filter.key`) and the sidecar reads from the directory it
|
||||
mounts. The journal milter recomputes it at end-of-message; with no
|
||||
signature, or one that does not fit, the row is that of mail nobody judged.
|
||||
The signature goes on to the upstream with the mail and is of no use there:
|
||||
it fits one queue id. Refusals are
|
||||
parsed from `mail.log` from the head of the line, with the reply ending at
|
||||
Postfix's own `; from=<` so that the HELO name after it - the client's text,
|
||||
logged as written - cannot stand in for the sender, the recipient or the
|
||||
reply; an address with a quote, a bracket or a space in it is not kept.
|
||||
Only a recipient at an inbound domain makes a row. The journal is cut to
|
||||
10 000 rows every 500 rows the tailer writes, and by age every six hours.
|
||||
- **Filter lists** reach another container as files: values are whitelisted on
|
||||
the way in (address, domain, canonical IP or network; no `/0`, no
|
||||
IPv4-mapped network), the writer refuses a stored value with whitespace, a
|
||||
newline or `#`, and a change of the table and the rewrite of the six files
|
||||
are one step under a lock.
|
||||
- TLS is mandatory before credentials are transmitted (465 wrapper / 587
|
||||
`encrypt`).
|
||||
- `TRUSTED_PROXY_CIDR` — only explicitly trusted proxies may supply
|
||||
@@ -183,6 +244,11 @@ operator can act on it.
|
||||
private key from `certs/` when present; a domain export carries the DKIM key
|
||||
and **working** application passwords in the clear (otherwise a transfer
|
||||
without recreating credentials would be impossible).
|
||||
- Since 2.2.0 a full backup also carries, in the database, what Postfix logged
|
||||
about each message still in the send log (`send_log_lines`): the submitting
|
||||
client's address and SASL login, the message id, the receiving server and
|
||||
its reply. `mail.log` itself stays out of a backup; these lines are state
|
||||
the panel shows, kept for the send log's retention and no longer.
|
||||
- Both downloads are encrypted with a password unless the box on the form is
|
||||
unticked (it is ticked by default):
|
||||
scrypt (N=2¹⁵, r=8, p=1) → AES-256-GCM, streamed in 64 KiB chunks, each
|
||||
@@ -274,20 +340,60 @@ deferred item from the roadmap.
|
||||
reporters are arbitrary third parties and the mailbox address is published in
|
||||
DNS. The ingest already constrains the meaningful part — a report cannot be
|
||||
filed under a domain this instance does not host, and the claimed domain is
|
||||
cross-checked against the `+tag` of the delivery address — and the data is
|
||||
advisory statistics only; nothing enforces off it. The trigger to revisit is
|
||||
cross-checked against the `+tag` of the delivery address (effective since
|
||||
2.0.0: before it the recipient never reached the command) — and the data is
|
||||
advisory statistics only; nothing enforces off it. What a stranger can do is
|
||||
limited to a domain that chose a SelfPost-hosted address, through that
|
||||
address: file reports under it, replace one whose reporter and id they know,
|
||||
and fill that domain's 500 — not another domain's. A message that is not a
|
||||
report the relay keeps is dropped, not bounced (the envelope sender is the
|
||||
author's to write); a genuine report the parser cannot read is lost the same
|
||||
way, with a count and a line in the mail log. The trigger to revisit is
|
||||
any feature that lets report contents drive an automatic action (policy
|
||||
changes, rate limiting, alerting an operator into a config change).
|
||||
- **The application auth-IP allow-list is enforced fail-open.** The check runs
|
||||
in the journal milter (`internal/milter/authip.go`) with
|
||||
`default_action=accept` and skips on store errors, so if the milter is down
|
||||
the restriction is not applied; the SASL login itself always succeeds. This
|
||||
the restriction is not applied; the SASL login itself always succeeds.
|
||||
(While the milter is up the list holds for every message, and an unknown
|
||||
client address is refused - see Mail path; what is accepted here is the
|
||||
milter being down.) This
|
||||
is documented in [guide.md](guide.md) as **not an authentication boundary** —
|
||||
it is a blast-radius limiter for a leaked application password, on the same
|
||||
fail-open path as level-2 rate limiting, and a milter outage already stops
|
||||
the journal from recording sends at all. The trigger to revisit is a
|
||||
requirement for the allow-list to hold as a security boundary — at which
|
||||
point it must move to the SASL/smtpd layer and fail closed.
|
||||
- **The key the spam filter signs with is readable by anyone on the host.**
|
||||
The inbound journal reads the filter's verdict only next to the filter's
|
||||
signature (see Mail path), and the key for it, `data/antispam/filter.key`,
|
||||
is 0644 like the list files beside it: the filter is another user in another
|
||||
container. Someone with a shell on the host could therefore sign a verdict
|
||||
of his own - into a monitoring page, on a host where he could do far more.
|
||||
Senders on the internet, whom the signature is for, cannot read it. Trigger
|
||||
to revisit: the same as for the list files - a shared host.
|
||||
- **An allow entry for a sender address or domain trusts what the sending
|
||||
server says.** The filter matches the envelope sender, which is not
|
||||
authenticated: anyone who knows or guesses an allowed address can send as it
|
||||
and is given the entry's weight, which no spam score outweighs. The page's
|
||||
Help and the guide say so and recommend an entry for the client's address or
|
||||
network instead. Deny entries have no such problem - claiming to be a denied
|
||||
sender only gets mail refused - and deny wins over allow. Trigger to
|
||||
revisit: allow entries being used as the main way to receive mail from a
|
||||
sender, at which point the rule should also require the sender's SPF or
|
||||
DKIM to pass.
|
||||
- **The filter lists are files readable by anyone on the host.**
|
||||
`/data/antispam` is 0755 and its six files 0644, unlike the rest of `/data`:
|
||||
the filter runs under another user in another container. They hold sender
|
||||
addresses, domains and networks - who the operator blocks and whom he lets
|
||||
through - but no message data. The inbound journal itself is in the
|
||||
database, 0640. Trigger to revisit: a shared host, where the sidecar should
|
||||
instead be given the files through a group it shares with the panel.
|
||||
- **A row of the inbound log belongs to a domain by name.** A `domain` user
|
||||
sees the rows whose inbound domain has the name of one assigned to them, so
|
||||
an inbound domain that is deleted and added again shows its earlier rows to
|
||||
whoever it is assigned to next. Accepted: the journal is of a mail domain,
|
||||
and the rows age out with the retention.
|
||||
- **Level-2 in-flight reservations can outlive a dropped SMTP session until TTL.**
|
||||
The journal milter cannot observe connection-close directly through
|
||||
go-milter, so if a client disconnects without an explicit `Abort`, its
|
||||
@@ -303,7 +409,36 @@ deferred item from the roadmap.
|
||||
the file is inaccessible to others. The log holds envelope addresses and
|
||||
client IPs, but neither message bodies nor headers; it is excluded from
|
||||
backups (`log/` is skipped) so that a dump stays state rather than
|
||||
diagnostics.
|
||||
diagnostics. Since 2.2.0 the tailer copies the lines of queued outbound
|
||||
messages into the database for the message's page, and those are in a
|
||||
backup (§ Backup and domain export); mail offered on port 25 is not copied.
|
||||
- **The panel submits mail of its own.** Notifications leave through
|
||||
`sendmail(1)` run by the unprivileged panel user: a fixed argument list, the
|
||||
two addresses after `--`, no shell. The recipient is a panel user's account
|
||||
e-mail, validated when it was saved and checked again before it reaches a
|
||||
header or the command line; the subject and body are composed by the panel
|
||||
from its own readings. Two of those readings can carry text from outside - a
|
||||
DNS check's finding, and a queued message's sender address - and both are
|
||||
reduced to printable ASCII and cut before they are printed, so a
|
||||
notification's body can contain an address a stranger chose, as a few plain
|
||||
characters and nothing else. The panel's
|
||||
membership in `postdrop`, described in the image as read-side, is now what
|
||||
lets it submit; any local user could already do that through the
|
||||
world-executable `sendmail`. Notifications are signed with a key of the
|
||||
server's own name and bypass the level-1 and level-2 limits: they are a few
|
||||
messages a day at most, to addresses only a panel user can set.
|
||||
- **A message's page can show another message's log lines after a queue-id
|
||||
collision.** Postfix's short queue ids repeat: the id is the microsecond of
|
||||
arrival and the queue file's inode, and inodes come round within seconds.
|
||||
The page takes the stored lines of the row's queue id dated within the
|
||||
row's own queue lifetime (about six days), so a second message has to get
|
||||
the same microsecond on the same inode in that span - about one in a
|
||||
million per reuse of the inode, and not something a sender can aim. When it
|
||||
happens, a `domain` user could read the log lines of a message that is not
|
||||
theirs. The search of the current `mail.log` that the page did before
|
||||
2.2.0 had the same exposure over a day. Trigger to revisit: a report of
|
||||
foreign lines on a page, or a reason to turn on `enable_long_queue_ids`,
|
||||
which ends it.
|
||||
- **A `domain` user may point an inbound domain's upstream at any host and
|
||||
port.** The upstream form checks that the host is a well-formed name or IP
|
||||
and the port is 1–65535; loopback, private and link-local addresses are
|
||||
|
||||
+84
-10
@@ -1,16 +1,16 @@
|
||||
// Package backup implements SelfPost's full-server backup and the restore
|
||||
// version guard (architecture.md § Persistence). A full backup is a
|
||||
// gzip-compressed tar of the consolidated persistent state under /data — the
|
||||
// SQLite database (as a consistent snapshot), the per-domain DKIM keys, the
|
||||
// SASL database, and the Postfix queue — plus docker-compose.yml, .env, and
|
||||
// certs/ from the operator's deploy directory, and a manifest recording the
|
||||
// SelfPost version that produced it. Postfix delivery logs under log/ are
|
||||
// excluded (diagnostics, not state).
|
||||
// SQLite database (as a consistent snapshot), the per-domain DKIM keys and the
|
||||
// SASL database — plus docker-compose.yml, .env, and certs/ from the
|
||||
// operator's deploy directory, and a manifest recording the SelfPost version
|
||||
// that produced it. Postfix delivery logs under log/ are excluded
|
||||
// (diagnostics, not state), and so is the Postfix queue (see QueueDir).
|
||||
//
|
||||
// Restore is not a separate code path in the panel: a backup is extracted into
|
||||
// the operator's project directory (data/, docker-compose.yml, .env, certs/)
|
||||
// before first start. The archive carries everything needed to bring the
|
||||
// instance back — DKIM keys, sasldb2, sender map, queue, and deploy files —
|
||||
// instance back — DKIM keys, sasldb2, sender map, and deploy files —
|
||||
// so the operator only adjusts hostname or proxy settings on a new host. The
|
||||
// restore-specific steps the panel runs are CheckRestore, which refuses to
|
||||
// boot if the manifest's version does not match the running binary so
|
||||
@@ -28,6 +28,7 @@ import (
|
||||
"compress/gzip"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
@@ -81,6 +82,18 @@ type Params struct {
|
||||
OnWarn func(string)
|
||||
}
|
||||
|
||||
// QueueDir is the Postfix queue, relative to the data directory. No backup
|
||||
// holds it (owner, 2026-10-11). An archive is made while the instance runs,
|
||||
// and Postfix moves messages between the queue's directories all the time: a
|
||||
// copy taken then can hold a message twice or not at all, so what it would
|
||||
// bring back is not the queue. Nor could the panel read it - Postfix keeps
|
||||
// every queue file readable by its own user alone, which is why the panel's
|
||||
// backup failed outright from 1.3.0 to 2.0.0. What is not carried over is mail
|
||||
// still waiting for delivery; a start without a queue directory builds an
|
||||
// empty one. Whoever needs the queue stops the instance and archives the
|
||||
// project directory (guide § Taking a full backup).
|
||||
const QueueDir = "postfix/queue"
|
||||
|
||||
// excludedFromArchive lists the data-directory entries a backup never carries.
|
||||
// The live database files are replaced by a consistent VACUUM INTO snapshot
|
||||
// written under the canonical name; the setup token is transient bootstrap
|
||||
@@ -145,7 +158,7 @@ func Create(w io.Writer, p Params) error {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := addTree(tw, p.DataDir, DataArchivePrefix, excludedFromArchive); err != nil {
|
||||
if err := addTree(tw, p.DataDir, DataArchivePrefix, excludedFromArchive, skipQueue); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -181,6 +194,52 @@ func validateDeployRoot(deployRoot string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// skipFunc says whether the entry at path (rel to the root being walked) is
|
||||
// left out of the archive. A directory left out is not descended into.
|
||||
type skipFunc func(path, rel string, d fs.DirEntry) bool
|
||||
|
||||
// skipQueue leaves the Postfix queue out of every archive.
|
||||
func skipQueue(_, rel string, _ fs.DirEntry) bool {
|
||||
return filepath.ToSlash(rel) == QueueDir
|
||||
}
|
||||
|
||||
// Check reads through everything Create would put into the archive, without
|
||||
// writing one, and returns the first thing that cannot be read. The panel
|
||||
// streams its archive to the browser, so once Create has begun a failure can
|
||||
// only cut the download short; this is run first, while a refusal can still be
|
||||
// shown as a page.
|
||||
//
|
||||
// The panel is not root, and nothing is left out on that account: its archive
|
||||
// is the one selfpost-backup makes, or it is not made. What it most likely
|
||||
// cannot read is the private key under certs/ - see UnreadableCert.
|
||||
func Check(p Params) error {
|
||||
if p.DataDir == "" || p.DBPath == "" {
|
||||
return fmt.Errorf("backup: DataDir and DBPath are required")
|
||||
}
|
||||
if err := ValidateDeployRoot(p.DeployRoot); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := addTree(nil, p.DataDir, DataArchivePrefix, excludedFromArchive, skipQueue); err != nil {
|
||||
return err
|
||||
}
|
||||
return addDeployFiles(nil, p.DeployRoot, nil)
|
||||
}
|
||||
|
||||
// UnreadableCert reports whether err, from Check or Create, is a file under
|
||||
// the deploy root's certs/ that the caller is not allowed to read, and names
|
||||
// it relative to the deploy root ("certs/privkey.pem").
|
||||
func UnreadableCert(err error, deployRoot string) (name string, ok bool) {
|
||||
var pathErr *fs.PathError
|
||||
if !errors.As(err, &pathErr) || !errors.Is(err, fs.ErrPermission) {
|
||||
return "", false
|
||||
}
|
||||
rel, relErr := filepath.Rel(filepath.Join(deployRoot, CertsDirName), pathErr.Path)
|
||||
if relErr != nil || rel == ".." || strings.HasPrefix(filepath.ToSlash(rel), "../") {
|
||||
return "", false
|
||||
}
|
||||
return CertsDirName + "/" + filepath.ToSlash(rel), true
|
||||
}
|
||||
|
||||
func addDeployFiles(tw *tar.Writer, deployRoot string, onWarn func(string)) error {
|
||||
for _, name := range []string{ComposeFileName, EnvFileName} {
|
||||
src := filepath.Join(deployRoot, name)
|
||||
@@ -206,14 +265,16 @@ func addDeployFiles(tw *tar.Writer, deployRoot string, onWarn func(string)) erro
|
||||
}
|
||||
return fmt.Errorf("backup: stat %s: %w", CertsDirName, err)
|
||||
}
|
||||
return addTree(tw, certsDir, CertsDirName+"/", nil)
|
||||
return addTree(tw, certsDir, CertsDirName+"/", nil, nil)
|
||||
}
|
||||
|
||||
// addTree walks root and adds every regular file (and directory, to preserve
|
||||
// empty ones and modes) to tw under archivePrefix + path relative to root.
|
||||
// When exclude is non-nil, top-level names relative to root are skipped.
|
||||
// When exclude is non-nil, top-level names relative to root are skipped; skip,
|
||||
// when non-nil, leaves out whatever it says. With a nil tw nothing is written:
|
||||
// every file is opened and closed, which is how Check finds what cannot be read.
|
||||
// Non-regular, non-directory entries (symlinks, sockets) are skipped.
|
||||
func addTree(tw *tar.Writer, root, archivePrefix string, exclude map[string]bool) error {
|
||||
func addTree(tw *tar.Writer, root, archivePrefix string, exclude map[string]bool, skip skipFunc) error {
|
||||
return filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -236,12 +297,22 @@ func addTree(tw *tar.Writer, root, archivePrefix string, exclude map[string]bool
|
||||
}
|
||||
}
|
||||
|
||||
if skip != nil && skip(path, rel, d) {
|
||||
if d.IsDir() {
|
||||
return fs.SkipDir
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
info, err := d.Info()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch {
|
||||
case d.IsDir():
|
||||
if tw == nil {
|
||||
return nil
|
||||
}
|
||||
hdr := &tar.Header{
|
||||
Typeflag: tar.TypeDir,
|
||||
Name: name + "/",
|
||||
@@ -282,6 +353,9 @@ func writeTarFile(tw *tar.Writer, name string, mode fs.FileMode, srcPath string)
|
||||
return fmt.Errorf("backup: open %s: %w", srcPath, err)
|
||||
}
|
||||
defer f.Close()
|
||||
if tw == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
info, err := f.Stat()
|
||||
if err != nil {
|
||||
|
||||
@@ -5,7 +5,9 @@ import (
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
@@ -40,6 +42,7 @@ func seedDataDir(t *testing.T) (dataDir, dbPath, deployRoot string) {
|
||||
writeFile(t, filepath.Join(dataDir, "opendkim", "keys", "example.com", "selfpost.private"), "PRIVATE KEY")
|
||||
writeFile(t, filepath.Join(dataDir, "sasl", "sasldb2"), "SASLDB")
|
||||
writeFile(t, filepath.Join(dataDir, "postfix", "sender_login_maps"), "@example.com login")
|
||||
writeFile(t, filepath.Join(dataDir, "antispam", "deny_domain.map"), "evil.example\n")
|
||||
writeFile(t, filepath.Join(dataDir, "postfix", "queue", "deferred", "sample"), "queue-file")
|
||||
// Transient files that must NOT be archived.
|
||||
writeFile(t, filepath.Join(dataDir, "setup-token"), "secret-token")
|
||||
@@ -112,7 +115,10 @@ func TestCreateIncludesStateExcludesTransient(t *testing.T) {
|
||||
DataArchivePrefix + "opendkim/keys/example.com/selfpost.private",
|
||||
DataArchivePrefix + "sasl/sasldb2",
|
||||
DataArchivePrefix + "postfix/sender_login_maps",
|
||||
DataArchivePrefix + "postfix/queue/deferred/sample",
|
||||
// The filter lists as the spam filter reads them. They are rewritten
|
||||
// from the database on the first start after a restore; carrying them
|
||||
// means the filter has its lists before the panel is up.
|
||||
DataArchivePrefix + "antispam/deny_domain.map",
|
||||
ComposeFileName,
|
||||
EnvFileName,
|
||||
CertsDirName + "/fullchain.pem",
|
||||
@@ -129,6 +135,7 @@ func TestCreateIncludesStateExcludesTransient(t *testing.T) {
|
||||
DataArchivePrefix + "selfpost.db-shm",
|
||||
DataArchivePrefix + "log/mail.log",
|
||||
DataArchivePrefix + "log/mail.log.1",
|
||||
DataArchivePrefix + "postfix/queue/deferred/sample",
|
||||
} {
|
||||
if _, ok := files[name]; ok {
|
||||
t.Errorf("archive should not contain %s", name)
|
||||
@@ -235,3 +242,70 @@ func TestCheckRestoreWrongFormatRejected(t *testing.T) {
|
||||
t.Error("CheckRestore wrong format reported a restore")
|
||||
}
|
||||
}
|
||||
|
||||
// No archive holds the Postfix queue; the rest is there, the private key
|
||||
// included, and Check agrees that it can be made.
|
||||
func TestCreateLeavesTheQueueOut(t *testing.T) {
|
||||
dataDir, dbPath, deployRoot := seedDataDir(t)
|
||||
p := Params{DataDir: dataDir, DBPath: dbPath, Version: "1.2.3", DeployRoot: deployRoot}
|
||||
|
||||
if err := Check(p); err != nil {
|
||||
t.Fatalf("Check: %v", err)
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
if err := Create(&buf, p); err != nil {
|
||||
t.Fatalf("Create: %v", err)
|
||||
}
|
||||
files := readArchive(t, buf.Bytes())
|
||||
for name := range files {
|
||||
if strings.HasPrefix(name, DataArchivePrefix+QueueDir) {
|
||||
t.Errorf("the panel's archive holds %s", name)
|
||||
}
|
||||
}
|
||||
for _, name := range []string{
|
||||
DataArchivePrefix + "selfpost.db",
|
||||
DataArchivePrefix + "opendkim/keys/example.com/selfpost.private",
|
||||
DataArchivePrefix + "sasl/sasldb2",
|
||||
DataArchivePrefix + "postfix/sender_login_maps",
|
||||
ComposeFileName, EnvFileName,
|
||||
CertsDirName + "/fullchain.pem",
|
||||
CertsDirName + "/privkey.pem",
|
||||
} {
|
||||
if _, ok := files[name]; !ok {
|
||||
t.Errorf("archive missing %s", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A file under certs/ that may not be read is told apart from any other
|
||||
// failure, so the panel can say what to do about it.
|
||||
func TestUnreadableCert(t *testing.T) {
|
||||
root := filepath.Join("srv", "deploy")
|
||||
denied := func(path string) error {
|
||||
return fmt.Errorf("backup: open %s: %w", path, &fs.PathError{Op: "open", Path: path, Err: fs.ErrPermission})
|
||||
}
|
||||
if name, ok := UnreadableCert(denied(filepath.Join(root, "certs", "privkey.pem")), root); !ok || name != "certs/privkey.pem" {
|
||||
t.Errorf("certs/privkey.pem: got %q, %v", name, ok)
|
||||
}
|
||||
if name, ok := UnreadableCert(denied(filepath.Join(root, ".env")), root); ok {
|
||||
t.Errorf(".env was taken for a certificate file: %q", name)
|
||||
}
|
||||
if name, ok := UnreadableCert(denied(filepath.Join("data", "postfix", "x")), root); ok {
|
||||
t.Errorf("a data file was taken for a certificate file: %q", name)
|
||||
}
|
||||
missing := &fs.PathError{Op: "open", Path: filepath.Join(root, "certs", "privkey.pem"), Err: fs.ErrNotExist}
|
||||
if _, ok := UnreadableCert(missing, root); ok {
|
||||
t.Error("a missing file was taken for one that may not be read")
|
||||
}
|
||||
}
|
||||
|
||||
// Check finds what Create would stumble over, before anything is written.
|
||||
func TestCheckReportsWhatCannotBeRead(t *testing.T) {
|
||||
dataDir, dbPath, deployRoot := seedDataDir(t)
|
||||
if err := os.Remove(filepath.Join(deployRoot, EnvFileName)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := Check(Params{DataDir: dataDir, DBPath: dbPath, Version: "1.2.3", DeployRoot: deployRoot}); err == nil {
|
||||
t.Fatal("Check passed with .env missing from the deploy root")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
package dmarc
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mixeme/selfpost/internal/store"
|
||||
)
|
||||
|
||||
func reportXML(orgName, domain, records string) string {
|
||||
return `<?xml version="1.0"?><feedback><report_metadata><org_name>` + orgName +
|
||||
`</org_name><email>a@example.net</email><report_id>1</report_id>` +
|
||||
`<date_range><begin>1723593600</begin><end>1723680000</end></date_range></report_metadata>` +
|
||||
`<policy_published><domain>` + domain + `</domain><p>none</p></policy_published>` + records + `</feedback>`
|
||||
}
|
||||
|
||||
const oneRecord = `<record><row><source_ip>203.0.113.10</source_ip><count>3</count>` +
|
||||
`<policy_evaluated><disposition>none</disposition><dkim>pass</dkim><spf>pass</spf></policy_evaluated></row>` +
|
||||
`<identifiers><header_from>example.com</header_from></identifiers></record>`
|
||||
|
||||
// The XML is a stranger's: what it may make the relay store is bounded.
|
||||
func TestParseAggregateBoundsWhatAReportHolds(t *testing.T) {
|
||||
long := strings.Repeat("é", 4000)
|
||||
got, err := ParseAggregate([]byte(reportXML(long, "example.com", oneRecord)))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got.Reporter) > maxFieldLen || !strings.HasPrefix(long, got.Reporter) || got.Reporter == "" {
|
||||
t.Fatalf("reporter name is %d bytes, want the start of the name in at most %d", len(got.Reporter), maxFieldLen)
|
||||
}
|
||||
if strings.ContainsRune(got.Reporter, '�') {
|
||||
t.Fatal("the name was cut in the middle of a character")
|
||||
}
|
||||
|
||||
huge := `<record><row><source_ip>203.0.113.10</source_ip><count>9223372036854775807</count>` +
|
||||
`<policy_evaluated><dkim>pass</dkim></policy_evaluated></row></record>`
|
||||
got, err = ParseAggregate([]byte(reportXML("r", "example.com", huge+huge)))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.PassCount != 2*maxRecordCount {
|
||||
t.Fatalf("pass count = %d, want two records clamped to %d each", got.PassCount, maxRecordCount)
|
||||
}
|
||||
|
||||
if _, err := ParseAggregate([]byte(reportXML("r", "example.com", strings.Repeat(oneRecord, maxReportRecords)))); err != nil {
|
||||
t.Fatalf("a report of %d records was refused: %v", maxReportRecords, err)
|
||||
}
|
||||
if _, err := ParseAggregate([]byte(reportXML("r", "example.com", strings.Repeat(oneRecord, maxReportRecords+1)))); err == nil {
|
||||
t.Fatalf("a report of %d records was accepted", maxReportRecords+1)
|
||||
}
|
||||
}
|
||||
|
||||
// A report over the bound is one the relay does not keep: dropped, not retried.
|
||||
func TestIngestMessageRejectsAReportWithTooManyRecords(t *testing.T) {
|
||||
st, err := store.Open(t.TempDir() + "/test.db")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
if _, err := st.AddDomain("example.com", "selfpost"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
xml := reportXML("r", "example.com", strings.Repeat(oneRecord, maxReportRecords+1))
|
||||
err = IngestMessage(st, strings.NewReader(gzipMessage(t, xml)), "dmarc-reports+example.com@mail.example.com", time.Now().UTC())
|
||||
if !errors.Is(err, ErrRejected) {
|
||||
t.Fatalf("err = %v, want ErrRejected", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Reports filed under one domain do not push another domain's reports out.
|
||||
func TestIngestKeepsEachDomainsReportsApart(t *testing.T) {
|
||||
st, err := store.Open(t.TempDir() + "/test.db")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
now := time.Now().UTC()
|
||||
if _, err := st.InsertDMARCReport(store.DMARCReport{
|
||||
Domain: "quiet.example", Reporter: "genuine", ReportID: "1", PeriodBegin: now, PeriodEnd: now, ReceivedAt: now.Add(-time.Hour),
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i := 0; i < store.DMARCReportsMaxKeep+20; i++ {
|
||||
if _, err := st.InsertDMARCReport(store.DMARCReport{
|
||||
Domain: "flooded.example", Reporter: "stranger", ReportID: strings.Repeat("x", 1+i), PeriodBegin: now, PeriodEnd: now, ReceivedAt: now,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := st.PruneDMARCReports("flooded.example"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
quiet, err := st.ListDMARCReports(store.DMARCReportScope{Domains: []string{"quiet.example"}}, 10)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(quiet) != 1 {
|
||||
t.Fatalf("the other domain has %d reports after the flood, want its 1", len(quiet))
|
||||
}
|
||||
flooded, err := st.ListDMARCReports(store.DMARCReportScope{Domains: []string{"flooded.example"}}, 10000)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(flooded) > store.DMARCReportsMaxKeep {
|
||||
t.Fatalf("the flooded domain keeps %d reports, want at most %d", len(flooded), store.DMARCReportsMaxKeep)
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@ package dmarc
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/base64"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"mime"
|
||||
@@ -16,6 +17,15 @@ import (
|
||||
"github.com/mixeme/selfpost/internal/store"
|
||||
)
|
||||
|
||||
// ErrRejected marks a message that is not a report this relay keeps and never
|
||||
// will be: it cannot be read as an aggregate report, or it reports on a domain
|
||||
// that is not sent from here, or it came to another domain's address. Sending
|
||||
// it again changes nothing, so the caller drops it. Any other error is the
|
||||
// store's and worth another try.
|
||||
var ErrRejected = errors.New("dmarc report rejected")
|
||||
|
||||
func rejected(err error) error { return fmt.Errorf("%w: %w", ErrRejected, err) }
|
||||
|
||||
// IngestMessage parses a raw RFC 5322 message from r and stores any DMARC
|
||||
// aggregate attachment it finds.
|
||||
//
|
||||
@@ -34,11 +44,11 @@ func IngestMessage(st *store.Store, r io.Reader, recipient string, receivedAt ti
|
||||
}
|
||||
payload, err := extractAggregatePayload(raw)
|
||||
if err != nil {
|
||||
return err
|
||||
return rejected(err)
|
||||
}
|
||||
parsed, err := ParseAggregate(payload)
|
||||
if err != nil {
|
||||
return err
|
||||
return rejected(err)
|
||||
}
|
||||
recipient = strings.ToLower(strings.TrimSpace(recipient))
|
||||
if err := checkReportDomain(st, parsed.Domain, recipient); err != nil {
|
||||
@@ -74,7 +84,7 @@ func IngestMessage(st *store.Store, r io.Reader, recipient string, receivedAt ti
|
||||
if _, err := st.InsertDMARCReport(rep); err != nil {
|
||||
return err
|
||||
}
|
||||
return st.PruneDMARCReports()
|
||||
return st.PruneDMARCReports(rep.Domain)
|
||||
}
|
||||
|
||||
// checkReportDomain refuses a report whose policy domain is not one of this
|
||||
@@ -82,17 +92,17 @@ func IngestMessage(st *store.Store, r io.Reader, recipient string, receivedAt ti
|
||||
// delivered to.
|
||||
func checkReportDomain(st *store.Store, domain, recipient string) error {
|
||||
if domain == "" {
|
||||
return fmt.Errorf("dmarc report: no policy domain")
|
||||
return rejected(fmt.Errorf("no policy domain"))
|
||||
}
|
||||
if tagged := TaggedDomain(recipient); tagged != "" && tagged != domain {
|
||||
return fmt.Errorf("dmarc report: domain %q does not match the address it was sent to (%s)", domain, recipient)
|
||||
return rejected(fmt.Errorf("domain %q does not match the address it was sent to (%s)", domain, recipient))
|
||||
}
|
||||
ok, err := st.DomainExists(domain)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !ok {
|
||||
return fmt.Errorf("dmarc report: %q is not a domain configured on this relay", domain)
|
||||
return rejected(fmt.Errorf("%q is not a domain configured on this relay", domain))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
+50
-15
@@ -9,6 +9,7 @@ import (
|
||||
"io"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
// ParsedReport is the structured form of a DMARC aggregate XML payload.
|
||||
@@ -74,6 +75,31 @@ type feedbackXML struct {
|
||||
} `xml:"record"`
|
||||
}
|
||||
|
||||
// What a report may hold. The XML comes from anyone who knows a report address,
|
||||
// and eight megabytes of it (maxAggregateXML) fit some 160 000 empty records or
|
||||
// one name of eight megabytes - every one of them a row or a table cell later.
|
||||
// A real report has tens of records, a few thousand from the largest
|
||||
// receivers, and no field longer than a line.
|
||||
const (
|
||||
maxReportRecords = 5000
|
||||
maxFieldLen = 255
|
||||
maxRecordCount = 1_000_000_000
|
||||
)
|
||||
|
||||
// field trims a text value of the report and cuts it to maxFieldLen bytes, on
|
||||
// a character boundary.
|
||||
func field(s string) string {
|
||||
s = strings.TrimSpace(s)
|
||||
if len(s) <= maxFieldLen {
|
||||
return s
|
||||
}
|
||||
s = s[:maxFieldLen]
|
||||
for len(s) > 0 && !utf8.ValidString(s) {
|
||||
s = s[:len(s)-1]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// ParseAggregate decodes gzip- or zip-compressed, or raw, DMARC aggregate XML.
|
||||
func ParseAggregate(raw []byte) (ParsedReport, error) {
|
||||
data, err := decompress(raw)
|
||||
@@ -87,34 +113,43 @@ func ParseAggregate(raw []byte) (ParsedReport, error) {
|
||||
if doc.Metadata.ReportID == "" || doc.Policy.Domain == "" {
|
||||
return ParsedReport{}, fmt.Errorf("dmarc xml: missing report_id or domain")
|
||||
}
|
||||
if len(doc.Records) > maxReportRecords {
|
||||
return ParsedReport{}, fmt.Errorf("dmarc xml: %d records, more than the %d a report may hold", len(doc.Records), maxReportRecords)
|
||||
}
|
||||
// Cut, not refused: a domain name is at most 253 bytes, so a longer one
|
||||
// names nothing this relay sends for and is turned away as such.
|
||||
out := ParsedReport{
|
||||
Reporter: strings.TrimSpace(doc.Metadata.OrgName),
|
||||
ReportID: strings.TrimSpace(doc.Metadata.ReportID),
|
||||
ContactEmail: strings.TrimSpace(doc.Metadata.Email),
|
||||
Reporter: field(doc.Metadata.OrgName),
|
||||
ReportID: field(doc.Metadata.ReportID),
|
||||
ContactEmail: field(doc.Metadata.Email),
|
||||
PeriodBegin: time.Unix(doc.Metadata.DateRange.Begin, 0).UTC(),
|
||||
PeriodEnd: time.Unix(doc.Metadata.DateRange.End, 0).UTC(),
|
||||
Domain: strings.ToLower(strings.TrimSpace(doc.Policy.Domain)),
|
||||
PolicyP: strings.TrimSpace(doc.Policy.P),
|
||||
PolicySP: strings.TrimSpace(doc.Policy.SP),
|
||||
Domain: strings.ToLower(field(doc.Policy.Domain)),
|
||||
PolicyP: field(doc.Policy.P),
|
||||
PolicySP: field(doc.Policy.SP),
|
||||
PolicyPct: doc.Policy.Pct,
|
||||
PolicyADKIM: strings.TrimSpace(doc.Policy.ADKIM),
|
||||
PolicyASPF: strings.TrimSpace(doc.Policy.ASPF),
|
||||
PolicyADKIM: field(doc.Policy.ADKIM),
|
||||
PolicyASPF: field(doc.Policy.ASPF),
|
||||
}
|
||||
if out.PolicyPct == 0 {
|
||||
out.PolicyPct = 100
|
||||
}
|
||||
for _, rec := range doc.Records {
|
||||
row := ParsedRecord{
|
||||
SourceIP: strings.TrimSpace(rec.Row.SourceIP),
|
||||
SourceIP: field(rec.Row.SourceIP),
|
||||
Count: rec.Row.Count,
|
||||
Disposition: strings.TrimSpace(rec.Row.Policy.Disposition),
|
||||
SPFResult: strings.TrimSpace(rec.Row.Policy.SPF),
|
||||
DKIMResult: strings.TrimSpace(rec.Row.Policy.DKIM),
|
||||
HeaderFrom: strings.TrimSpace(rec.Identifiers.HeaderFrom),
|
||||
Disposition: field(rec.Row.Policy.Disposition),
|
||||
SPFResult: field(rec.Row.Policy.SPF),
|
||||
DKIMResult: field(rec.Row.Policy.DKIM),
|
||||
HeaderFrom: field(rec.Identifiers.HeaderFrom),
|
||||
}
|
||||
if row.Count <= 0 {
|
||||
row.Count = 1
|
||||
}
|
||||
// Bounded so that the sums of a report cannot wrap around.
|
||||
if row.Count > maxRecordCount {
|
||||
row.Count = maxRecordCount
|
||||
}
|
||||
if dmarcRecordPasses(row) {
|
||||
out.PassCount += row.Count
|
||||
} else {
|
||||
@@ -172,12 +207,12 @@ func unzip(raw []byte) ([]byte, error) {
|
||||
}
|
||||
rc, err := f.Open()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("dmarc zip open %s: %w", f.Name, err)
|
||||
return nil, fmt.Errorf("dmarc zip open %q: %w", field(f.Name), err)
|
||||
}
|
||||
data, err := io.ReadAll(io.LimitReader(rc, maxAggregateXML))
|
||||
rc.Close()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("dmarc zip read %s: %w", f.Name, err)
|
||||
return nil, fmt.Errorf("dmarc zip read %q: %w", field(f.Name), err)
|
||||
}
|
||||
return data, nil
|
||||
}
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"archive/zip"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -126,8 +127,9 @@ func TestIngestMessageRejectsUnknownDomain(t *testing.T) {
|
||||
}
|
||||
defer st.Close()
|
||||
|
||||
if err := IngestMessage(st, bytes.NewBufferString(gzipMessage(t, sampleXML)), "dmarc-reports@mail.example.com", time.Now().UTC()); err == nil {
|
||||
t.Fatal("report for an unconfigured domain was accepted")
|
||||
err = IngestMessage(st, bytes.NewBufferString(gzipMessage(t, sampleXML)), "dmarc-reports@mail.example.com", time.Now().UTC())
|
||||
if !errors.Is(err, ErrRejected) {
|
||||
t.Fatalf("report for an unconfigured domain: err = %v, want ErrRejected", err)
|
||||
}
|
||||
list, err := st.ListDMARCReports(store.DMARCReportScope{AllDomains: true}, 10)
|
||||
if err != nil {
|
||||
@@ -155,8 +157,47 @@ func TestIngestMessageRejectsTagMismatch(t *testing.T) {
|
||||
|
||||
err = IngestMessage(st, bytes.NewBufferString(gzipMessage(t, sampleXML)),
|
||||
"dmarc-reports+other.example@mail.example.com", time.Now().UTC())
|
||||
if err == nil {
|
||||
t.Fatal("report was accepted at an address tagged for another domain")
|
||||
if !errors.Is(err, ErrRejected) {
|
||||
t.Fatalf("report at an address tagged for another domain: err = %v, want ErrRejected", err)
|
||||
}
|
||||
}
|
||||
|
||||
// What cannot be read as a report is rejected, not put off: Postfix would
|
||||
// offer the same bytes again for days.
|
||||
func TestIngestMessageRejectsWhatIsNotAReport(t *testing.T) {
|
||||
st, err := store.Open(t.TempDir() + "/test.db")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
if _, err := st.AddDomain("example.com", "selfpost"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for name, msg := range map[string]string{
|
||||
"plain text": "From: someone@example.net\r\nSubject: hello\r\n\r\nnot a report\r\n",
|
||||
"no headers": "garbage",
|
||||
"broken gzip": "From: a@example.net\r\nContent-Type: application/gzip\r\n\r\nnot gzip at all",
|
||||
} {
|
||||
err := IngestMessage(st, bytes.NewBufferString(msg), "dmarc-reports+example.com@mail.example.com", time.Now().UTC())
|
||||
if !errors.Is(err, ErrRejected) {
|
||||
t.Errorf("%s: err = %v, want ErrRejected", name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A failing store is not the report's fault: the message is to be tried again.
|
||||
func TestIngestMessageStoreFailureIsNotARejection(t *testing.T) {
|
||||
st, err := store.Open(t.TempDir() + "/test.db")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := st.AddDomain("example.com", "selfpost"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st.Close()
|
||||
err = IngestMessage(st, bytes.NewBufferString(gzipMessage(t, sampleXML)), "dmarc-reports+example.com@mail.example.com", time.Now().UTC())
|
||||
if err == nil || errors.Is(err, ErrRejected) {
|
||||
t.Fatalf("closed store: err = %v, want an error that is not ErrRejected", err)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -100,6 +100,7 @@ type Checker struct {
|
||||
servers map[string]cached[Server]
|
||||
domains map[string]cached[Domain]
|
||||
inbound map[string]cached[Result]
|
||||
mail map[string]cached[Mail]
|
||||
}
|
||||
|
||||
type cached[T any] struct {
|
||||
@@ -123,6 +124,7 @@ func newChecker(r resolver, timeout, srvTTL, domTTL time.Duration) *Checker {
|
||||
servers: make(map[string]cached[Server]),
|
||||
domains: make(map[string]cached[Domain]),
|
||||
inbound: make(map[string]cached[Result]),
|
||||
mail: make(map[string]cached[Mail]),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -170,6 +172,56 @@ func (c *Checker) Domain(q Query, force bool) Domain {
|
||||
return d
|
||||
}
|
||||
|
||||
// Mail is the published-DNS state of the server's own name as a sender: the
|
||||
// panel's notifications go out from postmaster@ the hostname, signed with a
|
||||
// key of that name.
|
||||
type Mail struct {
|
||||
Name string
|
||||
DKIM Result
|
||||
SPF Result
|
||||
Overall health.Status
|
||||
CheckedAt time.Time
|
||||
}
|
||||
|
||||
// ServerMailSPF is the SPF record to publish for the server's own name: its
|
||||
// address record is the one sender.
|
||||
const ServerMailSPF = "v=spf1 a -all"
|
||||
|
||||
// ServerMail checks the DKIM and SPF records of the server's own name. q is as
|
||||
// for a domain, with Name the hostname. Cached like a domain; force skips the
|
||||
// cache.
|
||||
func (c *Checker) ServerMail(q Query, force bool) Mail {
|
||||
key := "\x00mail\x00" + q.Name
|
||||
if !force {
|
||||
c.mu.Lock()
|
||||
entry, ok := c.mail[key]
|
||||
c.mu.Unlock()
|
||||
if ok && time.Now().Before(entry.expires) {
|
||||
return entry.value
|
||||
}
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), c.timeout)
|
||||
defer cancel()
|
||||
m := Mail{Name: q.Name, CheckedAt: time.Now()}
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(2)
|
||||
go func() { defer wg.Done(); m.DKIM = c.checkDKIM(ctx, q) }()
|
||||
go func() { defer wg.Done(); m.SPF = c.checkSPF(ctx, q) }()
|
||||
wg.Wait()
|
||||
// checkSPF words a missing record for a sending domain; for the server's
|
||||
// own name the record to publish is a different one.
|
||||
if m.SPF.Status == health.StatusError && len(m.SPF.Records) == 0 {
|
||||
m.SPF.Detail = fmt.Sprintf("No SPF record is published for %s. Publish the TXT record %q - without it a receiver has nothing authorising this server to send as its own name.", q.Name, ServerMailSPF)
|
||||
}
|
||||
m.Overall = health.Worst(m.DKIM.Status, m.SPF.Status)
|
||||
|
||||
c.mu.Lock()
|
||||
c.mail[key] = cached[Mail]{value: m, expires: m.CheckedAt.Add(c.domainTTL)}
|
||||
c.mu.Unlock()
|
||||
return m
|
||||
}
|
||||
|
||||
// Forget drops a domain's cached result, so the next page view re-checks it.
|
||||
// Used when a domain is removed or re-imported.
|
||||
func (c *Checker) Forget(domainName string) {
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
package dnscheck
|
||||
|
||||
import (
|
||||
"net"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/mixeme/selfpost/internal/health"
|
||||
)
|
||||
|
||||
const serverDKIM = "v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAserverkey"
|
||||
|
||||
func serverMailQuery() Query {
|
||||
return Query{
|
||||
Name: "mail.example.org",
|
||||
Selector: "sp",
|
||||
ExpectedDKIM: serverDKIM,
|
||||
Hostname: "mail.example.org",
|
||||
ServerIPs: []string{"203.0.113.25"},
|
||||
}
|
||||
}
|
||||
|
||||
// The server's own name passes with its key published and "v=spf1 a -all":
|
||||
// the name's address record is this server.
|
||||
func TestServerMailPublished(t *testing.T) {
|
||||
f := &fakeResolver{
|
||||
txt: map[string][]string{
|
||||
"sp._domainkey.mail.example.org": {serverDKIM},
|
||||
"mail.example.org": {ServerMailSPF},
|
||||
},
|
||||
addr: map[string][]net.IPAddr{"mail.example.org": ipAddrs("203.0.113.25")},
|
||||
}
|
||||
c := newTestChecker(f)
|
||||
|
||||
m := c.ServerMail(serverMailQuery(), false)
|
||||
if m.DKIM.Status != health.StatusOK || m.SPF.Status != health.StatusOK || m.Overall != health.StatusOK {
|
||||
t.Fatalf("DKIM %s (%s), SPF %s (%s), overall %s", m.DKIM.Status, m.DKIM.Detail, m.SPF.Status, m.SPF.Detail, m.Overall)
|
||||
}
|
||||
|
||||
// Cached like a domain, and re-read when forced.
|
||||
before := f.lookups
|
||||
c.ServerMail(serverMailQuery(), false)
|
||||
if f.lookups != before {
|
||||
t.Errorf("a second reading looked DNS up again (%d lookups)", f.lookups-before)
|
||||
}
|
||||
c.ServerMail(serverMailQuery(), true)
|
||||
if f.lookups == before {
|
||||
t.Error("a forced reading used the cache")
|
||||
}
|
||||
}
|
||||
|
||||
// With nothing published both records are errors, and the SPF advice names the
|
||||
// record for a server's own name, not the one for a sending domain.
|
||||
func TestServerMailMissing(t *testing.T) {
|
||||
c := newTestChecker(&fakeResolver{})
|
||||
|
||||
m := c.ServerMail(serverMailQuery(), false)
|
||||
if m.DKIM.Status != health.StatusError || m.SPF.Status != health.StatusError || m.Overall != health.StatusError {
|
||||
t.Fatalf("DKIM %s, SPF %s, overall %s; want errors", m.DKIM.Status, m.SPF.Status, m.Overall)
|
||||
}
|
||||
if !strings.Contains(m.SPF.Detail, ServerMailSPF) || strings.Contains(m.SPF.Detail, "a:mail.example.org") {
|
||||
t.Errorf("SPF advice: %s", m.SPF.Detail)
|
||||
}
|
||||
}
|
||||
|
||||
// A sending domain of the same name does not share the cache entry.
|
||||
func TestServerMailCacheIsItsOwn(t *testing.T) {
|
||||
f := &fakeResolver{
|
||||
txt: map[string][]string{
|
||||
"sp._domainkey.mail.example.org": {serverDKIM},
|
||||
"mail.example.org": {ServerMailSPF},
|
||||
},
|
||||
addr: map[string][]net.IPAddr{"mail.example.org": ipAddrs("203.0.113.25")},
|
||||
}
|
||||
c := newTestChecker(f)
|
||||
c.Domain(serverMailQuery(), false)
|
||||
before := f.lookups
|
||||
c.ServerMail(serverMailQuery(), false)
|
||||
if f.lookups == before {
|
||||
t.Error("the server's mail check was answered from the domain's cache entry")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
package domain
|
||||
|
||||
import (
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func readTables(t *testing.T, odk *OpenDKIM) (keyTable, signingTable string) {
|
||||
t.Helper()
|
||||
kt, err := os.ReadFile(odk.keyTablePath)
|
||||
if err != nil {
|
||||
t.Fatalf("read KeyTable: %v", err)
|
||||
}
|
||||
st, err := os.ReadFile(odk.signingTablePath)
|
||||
if err != nil {
|
||||
t.Fatalf("read SigningTable: %v", err)
|
||||
}
|
||||
return string(kt), string(st)
|
||||
}
|
||||
|
||||
// The server's own name is signed like a sending domain - a key, a line in
|
||||
// each table - without being one: no row in the registry, so no application
|
||||
// can send as it.
|
||||
func TestServerNameIsSignedWithoutBeingADomain(t *testing.T) {
|
||||
svc, odk := newTestService(t, &fakeApps{})
|
||||
svc.SetServerName("mail.example.org")
|
||||
|
||||
if _, err := svc.Add("example.com"); err != nil {
|
||||
t.Fatalf("add domain: %v", err)
|
||||
}
|
||||
kt, st := readTables(t, odk)
|
||||
for _, want := range []string{"mail.example.org mail.example.org:selfpost:", "example.com example.com:selfpost:"} {
|
||||
if !strings.Contains(kt, want) {
|
||||
t.Errorf("KeyTable lacks %q:\n%s", want, kt)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(st, "*@mail.example.org mail.example.org\n") {
|
||||
t.Errorf("SigningTable does not sign the server's own name:\n%s", st)
|
||||
}
|
||||
domains, err := svc.List()
|
||||
if err != nil || len(domains) != 1 {
|
||||
t.Fatalf("registry holds %d domains, %v; want the one that was added", len(domains), err)
|
||||
}
|
||||
|
||||
mail, err := svc.ServerMail()
|
||||
if err != nil {
|
||||
t.Fatalf("ServerMail: %v", err)
|
||||
}
|
||||
if mail.Name != "mail.example.org" || mail.Selector != "selfpost" ||
|
||||
mail.Record.Name != "selfpost._domainkey.mail.example.org" || !strings.HasPrefix(mail.Record.Value, "v=DKIM1;") {
|
||||
t.Errorf("server mail = %+v", mail)
|
||||
}
|
||||
// Asked again, it is the same key: the record published in DNS stays valid.
|
||||
again, err := svc.ServerMail()
|
||||
if err != nil || again.Record.Value != mail.Record.Value {
|
||||
t.Errorf("the server's key changed between two readings (%v)", err)
|
||||
}
|
||||
}
|
||||
|
||||
// When the hostname is itself a sending domain, there is one key and one line,
|
||||
// the domain's; deleting the domain leaves the server's mail signed with it.
|
||||
func TestServerNameThatIsAlsoADomain(t *testing.T) {
|
||||
svc, odk := newTestService(t, &fakeApps{})
|
||||
svc.SetServerName("example.com")
|
||||
|
||||
d, err := svc.Add("example.com")
|
||||
if err != nil {
|
||||
t.Fatalf("add domain: %v", err)
|
||||
}
|
||||
_, st := readTables(t, odk)
|
||||
if strings.Count(st, "*@example.com ") != 1 {
|
||||
t.Errorf("the name is in the SigningTable more than once:\n%s", st)
|
||||
}
|
||||
asDomain, err := svc.DKIMRecord(d)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mail, err := svc.ServerMail()
|
||||
if err != nil || mail.Record.Value != asDomain.Value {
|
||||
t.Fatalf("the server's record is not the domain's (%v)", err)
|
||||
}
|
||||
|
||||
if err := svc.Delete(d.ID); err != nil {
|
||||
t.Fatalf("delete domain: %v", err)
|
||||
}
|
||||
_, st = readTables(t, odk)
|
||||
if !strings.Contains(st, "*@example.com example.com\n") {
|
||||
t.Errorf("deleting the domain stopped the server's own mail being signed:\n%s", st)
|
||||
}
|
||||
after, err := svc.ServerMail()
|
||||
if err != nil || after.Record.Value != asDomain.Value {
|
||||
t.Errorf("deleting the domain changed the server's key (%v)", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Without a hostname nothing is added and nothing is signed for it.
|
||||
func TestNoServerName(t *testing.T) {
|
||||
svc, odk := newTestService(t, &fakeApps{})
|
||||
if _, err := svc.Add("example.com"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, st := readTables(t, odk); strings.Count(st, "\n") != 1 {
|
||||
t.Errorf("SigningTable has more than the one domain:\n%s", st)
|
||||
}
|
||||
if _, err := svc.ServerMail(); err == nil {
|
||||
t.Error("ServerMail without a hostname returned no error")
|
||||
}
|
||||
}
|
||||
@@ -38,6 +38,64 @@ type Service struct {
|
||||
odk *OpenDKIM
|
||||
apps Applications
|
||||
selector string
|
||||
// serverName is the server's own hostname, which the panel's notifications
|
||||
// are sent from and signed as (SetServerName). Empty: nothing is signed
|
||||
// for it.
|
||||
serverName string
|
||||
}
|
||||
|
||||
// SetServerName gives the service the server's own hostname. The panel sends
|
||||
// its notifications from postmaster@ that name, and for a receiver to believe
|
||||
// them the name is signed like a sending domain: it gets a DKIM key and a line
|
||||
// in OpenDKIM's tables, without being a row of the registry - no application
|
||||
// can send as it. The name comes from SELFPOST_HOSTNAME, which the entrypoint
|
||||
// has validated; it is not user input.
|
||||
func (s *Service) SetServerName(hostname string) {
|
||||
s.serverName = hostname
|
||||
}
|
||||
|
||||
// ServerMail is how the server's own name is signed.
|
||||
type ServerMail struct {
|
||||
Name string
|
||||
Selector string
|
||||
Record DKIMRecord
|
||||
}
|
||||
|
||||
// ServerMail returns the name notifications are sent as, its DKIM selector and
|
||||
// the record to publish for it, creating the key the first time. When the
|
||||
// hostname is itself a sending domain, it is that domain's key and selector:
|
||||
// one name has one key.
|
||||
func (s *Service) ServerMail() (ServerMail, error) {
|
||||
if s.serverName == "" {
|
||||
return ServerMail{}, fmt.Errorf("server mail: no hostname")
|
||||
}
|
||||
selector, err := s.serverSelector()
|
||||
if err != nil {
|
||||
return ServerMail{}, err
|
||||
}
|
||||
if _, err := s.odk.EnsureKey(s.serverName, selector); err != nil {
|
||||
return ServerMail{}, err
|
||||
}
|
||||
rec, err := s.odk.Record(s.serverName, selector)
|
||||
if err != nil {
|
||||
return ServerMail{}, err
|
||||
}
|
||||
return ServerMail{Name: s.serverName, Selector: selector, Record: rec}, nil
|
||||
}
|
||||
|
||||
// serverSelector is the selector the server's own name is signed with: that of
|
||||
// the sending domain of the same name if there is one, the default otherwise.
|
||||
func (s *Service) serverSelector() (string, error) {
|
||||
domains, err := s.store.ListDomains()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
for _, d := range domains {
|
||||
if d.Name == s.serverName {
|
||||
return d.DKIMSelector, nil
|
||||
}
|
||||
}
|
||||
return s.selector, nil
|
||||
}
|
||||
|
||||
// NewService builds the domain service. selectorDefault is the DKIM selector
|
||||
@@ -125,6 +183,12 @@ func (s *Service) Delete(id int64) error {
|
||||
if err := s.apps.Resync(); err != nil {
|
||||
return fmt.Errorf("rebuild sender map after deleting %s: %w", d.Name, err)
|
||||
}
|
||||
// A sending domain named like the server shared its key with the server's
|
||||
// own mail, which goes on being signed with it (resync above kept the name
|
||||
// in the tables).
|
||||
if d.Name == s.serverName {
|
||||
return nil
|
||||
}
|
||||
if err := s.odk.RemoveKey(d.Name); err != nil {
|
||||
// The domain is gone from the registry and tables; a leftover key
|
||||
// directory is harmless. Surface it so it is not silently ignored.
|
||||
@@ -172,9 +236,21 @@ func (s *Service) resync() error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
signing := make([]SigningDomain, 0, len(domains))
|
||||
signing := make([]SigningDomain, 0, len(domains)+1)
|
||||
registered := false
|
||||
for _, d := range domains {
|
||||
signing = append(signing, SigningDomain{Name: d.Name, Selector: d.DKIMSelector})
|
||||
if d.Name == s.serverName {
|
||||
registered = true
|
||||
}
|
||||
}
|
||||
// The server's own name, for the panel's notifications - unless it is a
|
||||
// sending domain already and so in the tables above.
|
||||
if s.serverName != "" && !registered {
|
||||
if _, err := s.odk.EnsureKey(s.serverName, s.selector); err != nil {
|
||||
return fmt.Errorf("dkim key for %s: %w", s.serverName, err)
|
||||
}
|
||||
signing = append(signing, SigningDomain{Name: s.serverName, Selector: s.selector})
|
||||
}
|
||||
return s.odk.Rebuild(signing)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,447 @@
|
||||
package inbound
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"net/netip"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/mixeme/selfpost/internal/atomicfile"
|
||||
"github.com/mixeme/selfpost/internal/configsafe"
|
||||
"github.com/mixeme/selfpost/internal/store"
|
||||
)
|
||||
|
||||
// ListStore is the slice of the store the filter lists need. *store.Store
|
||||
// satisfies it.
|
||||
type ListStore interface {
|
||||
AddSpamListEntry(list, kind, value, note string) (store.SpamListEntry, error)
|
||||
ListSpamListEntries() ([]store.SpamListEntry, error)
|
||||
DeleteSpamListEntry(id int64) error
|
||||
ListSpamListHits(marks ...string) ([]store.SpamListHit, error)
|
||||
}
|
||||
|
||||
// Lists owns the allow and deny lists of the external spam filter: the rows in
|
||||
// SQLite and the map files the filter reads. SelfPost does not consult the
|
||||
// lists itself - it writes them out, six files in one directory, one entry a
|
||||
// line, and the filter (rspamd, deploy/antispam/rspamd/multimap.conf) rereads
|
||||
// a file when it changes. Which list wins when both match is the filter's
|
||||
// configuration: deny.
|
||||
type Lists struct {
|
||||
store ListStore
|
||||
dir string
|
||||
// mu makes a change of the table and the rewrite of the files one step.
|
||||
// Two requests at once could otherwise write the files in the other order
|
||||
// than they changed the table - the first to read the entries being the
|
||||
// last to write them - and leave the filter with an entry the page no
|
||||
// longer lists. One Lists serves the panel, so a mutex is enough.
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
// NewLists builds the filter lists over the store and the map directory
|
||||
// (/data/antispam).
|
||||
func NewLists(st ListStore, dir string) *Lists {
|
||||
return &Lists{store: st, dir: dir}
|
||||
}
|
||||
|
||||
// noteMaxRunes caps the operator's comment on an entry.
|
||||
const noteMaxRunes = 200
|
||||
|
||||
// mapFiles names the file of every (list, kind). An IP map and a sender map
|
||||
// are different things to the filter, and a domain entry is matched against
|
||||
// another part of the sender than an address, so each has its own file.
|
||||
var mapFiles = map[[2]string]string{
|
||||
{store.SpamListAllow, store.SpamEntryIP}: "allow_ip.map",
|
||||
{store.SpamListAllow, store.SpamEntryAddress}: "allow_from.map",
|
||||
{store.SpamListAllow, store.SpamEntryDomain}: "allow_domain.map",
|
||||
{store.SpamListDeny, store.SpamEntryIP}: "deny_ip.map",
|
||||
{store.SpamListDeny, store.SpamEntryAddress}: "deny_from.map",
|
||||
{store.SpamListDeny, store.SpamEntryDomain}: "deny_domain.map",
|
||||
}
|
||||
|
||||
// Entries returns every entry of both lists.
|
||||
func (l *Lists) Entries() ([]store.SpamListEntry, error) {
|
||||
return l.store.ListSpamListEntries()
|
||||
}
|
||||
|
||||
// Add validates an entry, stores it normalised and rewrites the maps. A
|
||||
// refusal is a *ValidationError; an entry already on that list is
|
||||
// store.ErrSpamListEntryExists.
|
||||
func (l *Lists) Add(list, kind, rawValue, rawNote string) (store.SpamListEntry, error) {
|
||||
if list != store.SpamListAllow && list != store.SpamListDeny {
|
||||
return store.SpamListEntry{}, invalid("choose the allow or the deny list")
|
||||
}
|
||||
value, err := normalizeListValue(kind, rawValue)
|
||||
if err != nil {
|
||||
return store.SpamListEntry{}, err
|
||||
}
|
||||
note, err := normalizeNote(rawNote)
|
||||
if err != nil {
|
||||
return store.SpamListEntry{}, err
|
||||
}
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
e, err := l.store.AddSpamListEntry(list, kind, value, note)
|
||||
if err != nil {
|
||||
return store.SpamListEntry{}, err
|
||||
}
|
||||
if err := l.sync(); err != nil {
|
||||
// An entry the filter never got is worse than no entry: the page
|
||||
// would show a rule that does nothing. The files are written again
|
||||
// after the row is taken back, since some of the six may already
|
||||
// hold it; if that fails too, the next change or start puts it right.
|
||||
_ = l.store.DeleteSpamListEntry(e.ID)
|
||||
_ = l.sync()
|
||||
return store.SpamListEntry{}, err
|
||||
}
|
||||
return e, nil
|
||||
}
|
||||
|
||||
// Delete removes an entry and rewrites the maps. If the maps cannot be
|
||||
// written the entry is put back: an entry that is gone from the page and
|
||||
// still in the filter's file could not even be removed a second time.
|
||||
func (l *Lists) Delete(id int64) error {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
entries, err := l.store.ListSpamListEntries()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var gone *store.SpamListEntry
|
||||
for i := range entries {
|
||||
if entries[i].ID == id {
|
||||
gone = &entries[i]
|
||||
}
|
||||
}
|
||||
if err := l.store.DeleteSpamListEntry(id); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := l.sync(); err != nil {
|
||||
if gone != nil {
|
||||
_, _ = l.store.AddSpamListEntry(gone.List, gone.Kind, gone.Value, gone.Note)
|
||||
}
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Sync writes all six map files from SQLite, each atomically, so the filter
|
||||
// reads a whole old file or a whole new one. Every file is written even when
|
||||
// empty: the filter's configuration names all six.
|
||||
func (l *Lists) Sync() error {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
return l.sync()
|
||||
}
|
||||
|
||||
// sync is Sync for a caller that holds mu.
|
||||
func (l *Lists) sync() error {
|
||||
entries, err := l.store.ListSpamListEntries()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
lines := make(map[string][]string, len(mapFiles))
|
||||
for _, name := range mapFiles {
|
||||
lines[name] = nil
|
||||
}
|
||||
for _, e := range entries {
|
||||
name, ok := mapFiles[[2]string{e.List, e.Kind}]
|
||||
if !ok {
|
||||
return fmt.Errorf("filter list entry %d: unknown list %q or kind %q", e.ID, e.List, e.Kind)
|
||||
}
|
||||
// Values were validated on the way in; this is the writer's own guard
|
||||
// against a row that got into the table some other way. In a map file a
|
||||
// newline ends the entry, whitespace starts its value and # a comment.
|
||||
if err := configsafe.Token("filter list entry", e.Value, " \t\r\n#"); err != nil {
|
||||
return err
|
||||
}
|
||||
lines[name] = append(lines[name], e.Value)
|
||||
}
|
||||
// The filter runs under another user in another container: the directory
|
||||
// and the files are readable by anyone. They hold addresses, not secrets.
|
||||
if err := os.MkdirAll(l.dir, 0o755); err != nil {
|
||||
return fmt.Errorf("create %s: %w", l.dir, err)
|
||||
}
|
||||
for name, vals := range lines {
|
||||
sort.Strings(vals)
|
||||
var b strings.Builder
|
||||
b.WriteString("# Written by SelfPost (Inbound > Filter lists). Do not edit: the next change in the panel replaces this file.\n")
|
||||
for _, v := range vals {
|
||||
b.WriteString(v)
|
||||
b.WriteByte('\n')
|
||||
}
|
||||
if err := atomicfile.Write(filepath.Join(l.dir, name), []byte(b.String()), 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// LastWritten is when the maps were last written: the newest modification
|
||||
// time of the six files, the zero time while none exists (the filter lists
|
||||
// have never been touched, or the directory is gone).
|
||||
func (l *Lists) LastWritten() time.Time {
|
||||
var newest time.Time
|
||||
for _, name := range mapFiles {
|
||||
if fi, err := os.Stat(filepath.Join(l.dir, name)); err == nil && fi.ModTime().After(newest) {
|
||||
newest = fi.ModTime()
|
||||
}
|
||||
}
|
||||
return newest
|
||||
}
|
||||
|
||||
// Listable reports whether value could be added to a list as an entry of kind.
|
||||
// The Inbound log asks before it offers a row's sender to the lists: an
|
||||
// envelope sender may be spelt in ways the lists do not take.
|
||||
func Listable(kind, value string) bool {
|
||||
_, err := normalizeListValue(kind, value)
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// normalizeListValue validates one entry and returns the form it is stored and
|
||||
// written in: lower case for an address or a domain, the canonical text of an
|
||||
// address or a network for an IP.
|
||||
func normalizeListValue(kind, raw string) (string, error) {
|
||||
v := strings.ToLower(strings.TrimSpace(raw))
|
||||
switch kind {
|
||||
case store.SpamEntryAddress:
|
||||
if v == "" {
|
||||
return "", invalid("sender address is required")
|
||||
}
|
||||
if len(v) > 254 {
|
||||
return "", invalid("sender address must be at most 254 characters")
|
||||
}
|
||||
at := strings.LastIndexByte(v, '@')
|
||||
if at <= 0 || at >= len(v)-1 {
|
||||
return "", invalid("%q is not a valid email address", v)
|
||||
}
|
||||
local, domain := v[:at], v[at+1:]
|
||||
if local[0] == '.' || local[len(local)-1] == '.' || len(local) > 64 {
|
||||
return "", invalid("%q: invalid local part", v)
|
||||
}
|
||||
for i := 0; i < len(local); i++ {
|
||||
c := local[i]
|
||||
lower := c >= 'a' && c <= 'z'
|
||||
digit := c >= '0' && c <= '9'
|
||||
// '=' for the rewritten senders of forwarders (SRS0=…).
|
||||
if !lower && !digit && !strings.ContainsRune("._+-=", rune(c)) {
|
||||
return "", invalid("%q: local part contains invalid characters", v)
|
||||
}
|
||||
}
|
||||
if err := checkDomain(domain); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return v, nil
|
||||
case store.SpamEntryDomain:
|
||||
// "@example.com" is how a domain is often written in such a list.
|
||||
v = strings.TrimPrefix(v, "@")
|
||||
if err := checkDomain(v); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return v, nil
|
||||
case store.SpamEntryIP:
|
||||
if v == "" {
|
||||
return "", invalid("address or network is required")
|
||||
}
|
||||
if strings.Contains(v, "/") {
|
||||
p, err := netip.ParsePrefix(v)
|
||||
if err != nil {
|
||||
return "", invalid("%q is not a network in CIDR form (e.g. 192.0.2.0/24)", v)
|
||||
}
|
||||
if p.Addr().Is4In6() {
|
||||
return "", invalid("%q: write an IPv4 network as IPv4 (e.g. 192.0.2.0/24)", v)
|
||||
}
|
||||
if p.Bits() == 0 {
|
||||
return "", invalid("%q is every address there is", v)
|
||||
}
|
||||
// A network of one address is that address: 192.0.2.7/32 and
|
||||
// 192.0.2.7 are one entry, not two that look different.
|
||||
if p.Bits() == p.Addr().BitLen() {
|
||||
return p.Addr().String(), nil
|
||||
}
|
||||
return p.Masked().String(), nil
|
||||
}
|
||||
a, err := netip.ParseAddr(v)
|
||||
if err != nil || a.Zone() != "" {
|
||||
return "", invalid("%q is not an IP address or a network in CIDR form", v)
|
||||
}
|
||||
return a.Unmap().String(), nil
|
||||
default:
|
||||
return "", invalid("choose what the entry is: a sender address, a sender domain or a client address")
|
||||
}
|
||||
}
|
||||
|
||||
// normalizeNote trims the operator's comment and refuses what a single line of
|
||||
// text cannot be.
|
||||
func normalizeNote(raw string) (string, error) {
|
||||
note := strings.TrimSpace(raw)
|
||||
if utf8.RuneCountInString(note) > noteMaxRunes {
|
||||
return "", invalid("note must be at most %d characters", noteMaxRunes)
|
||||
}
|
||||
for _, r := range note {
|
||||
if r < 0x20 || r == 0x7f || r == utf8.RuneError {
|
||||
return "", invalid("note must be one line of plain text")
|
||||
}
|
||||
}
|
||||
return note, nil
|
||||
}
|
||||
|
||||
// What a list leaves in the journal when it decides a message - the two ends
|
||||
// of deploy/antispam/rspamd/multimap.conf this package depends on. A deny rule
|
||||
// refuses with a message, which the log tailer files as the reply of a
|
||||
// rejected row. An allow rule adds a symbol named after the kind of entry,
|
||||
// which the journal milter keeps in the verdict of an accepted row.
|
||||
const (
|
||||
denyReplyMark = "this server's deny list"
|
||||
allowMark = "SELFPOST_ALLOW_"
|
||||
)
|
||||
|
||||
var allowSymbol = map[string]string{
|
||||
store.SpamEntryIP: "SELFPOST_ALLOW_IP(",
|
||||
store.SpamEntryAddress: "SELFPOST_ALLOW_FROM(",
|
||||
store.SpamEntryDomain: "SELFPOST_ALLOW_DOMAIN(",
|
||||
}
|
||||
|
||||
// Usage is how often an entry decided a message, as far back as the journal
|
||||
// reaches, and when it last did. The zero value is an entry that never did.
|
||||
type Usage struct {
|
||||
Hits int
|
||||
Last time.Time
|
||||
}
|
||||
|
||||
// Usage counts, for every entry, the journal rows it decided. Nothing is
|
||||
// stored for this: the journal already says which list decided a message and
|
||||
// who the message was from, and an entry is matched against that the way the
|
||||
// filter matches it - an address whole, a domain as the sender's domain, an
|
||||
// address or network against the client's. So the count reaches back as far
|
||||
// as the journal does (its retention, or less when it is full), and a row
|
||||
// counts for every entry of the deciding list that fits it: the journal does
|
||||
// not say which of two fitting entries the filter met first.
|
||||
//
|
||||
// A counter kept with each entry would not depend on the journal; the plan
|
||||
// (docs/plans/inbound-antispam-panel.md, Filter list usage) says when that is
|
||||
// worth a migration.
|
||||
func (l *Lists) Usage() (map[int64]Usage, error) {
|
||||
entries, err := l.store.ListSpamListEntries()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make(map[int64]Usage, len(entries))
|
||||
if len(entries) == 0 {
|
||||
return out, nil
|
||||
}
|
||||
hits, err := l.store.ListSpamListHits(denyReplyMark, allowMark)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
type matcher struct {
|
||||
id int64
|
||||
list string
|
||||
kind string
|
||||
value string
|
||||
prefix netip.Prefix
|
||||
}
|
||||
matchers := make([]matcher, 0, len(entries))
|
||||
for _, e := range entries {
|
||||
m := matcher{id: e.ID, list: e.List, kind: e.Kind, value: e.Value}
|
||||
if e.Kind == store.SpamEntryIP {
|
||||
if p, err := netip.ParsePrefix(e.Value); err == nil {
|
||||
m.prefix = p
|
||||
} else if a, err := netip.ParseAddr(e.Value); err == nil {
|
||||
m.prefix = netip.PrefixFrom(a, a.BitLen())
|
||||
} else {
|
||||
continue // not an entry this package wrote
|
||||
}
|
||||
}
|
||||
matchers = append(matchers, m)
|
||||
}
|
||||
for _, h := range hits {
|
||||
from := strings.ToLower(h.From)
|
||||
domain := ""
|
||||
if i := strings.LastIndexByte(from, '@'); i >= 0 {
|
||||
domain = from[i+1:]
|
||||
}
|
||||
client, clientErr := netip.ParseAddr(h.ClientIP)
|
||||
client = client.Unmap()
|
||||
denied := h.Decision != store.SpamDecisionAccept && strings.Contains(h.Detail, denyReplyMark)
|
||||
for _, m := range matchers {
|
||||
switch m.list {
|
||||
case store.SpamListDeny:
|
||||
if !denied {
|
||||
continue
|
||||
}
|
||||
case store.SpamListAllow:
|
||||
if h.Decision != store.SpamDecisionAccept || !strings.Contains(h.Detail, allowSymbol[m.kind]) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
fits := false
|
||||
switch m.kind {
|
||||
case store.SpamEntryAddress:
|
||||
fits = from == m.value
|
||||
case store.SpamEntryDomain:
|
||||
fits = domain == m.value
|
||||
case store.SpamEntryIP:
|
||||
fits = clientErr == nil && m.prefix.Contains(client)
|
||||
}
|
||||
if !fits {
|
||||
continue
|
||||
}
|
||||
u := out[m.id]
|
||||
u.Hits++
|
||||
if h.CreatedAt.After(u.Last) {
|
||||
u.Last = h.CreatedAt
|
||||
}
|
||||
out[m.id] = u
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// FilterKeyFile is the file, beside the list files, that holds the key the
|
||||
// spam filter signs messages with.
|
||||
const FilterKeyFile = "filter.key"
|
||||
|
||||
// FilterKey returns the key the spam filter signs each message with, creating
|
||||
// it on first use: 32 random bytes as 64 hex characters on one line, in the
|
||||
// directory the filter's container mounts.
|
||||
//
|
||||
// The filter is attached so that mail flows when it is down. Then nothing
|
||||
// removes verdict headers a sender wrote himself, and the inbound journal
|
||||
// would record his verdict as the filter's. So the filter signs each message
|
||||
// with this key and the journal reads a verdict only next to a signature it
|
||||
// can recompute (internal/milter, FilterSignature).
|
||||
//
|
||||
// The file is readable by anyone on the host, like the list files and for the
|
||||
// same reason: the filter is another user in another container. What the key
|
||||
// protects is the truth of a monitoring page, against senders on the internet;
|
||||
// someone with a shell on the host is not who it is kept from.
|
||||
func FilterKey(dir string) (string, error) {
|
||||
path := filepath.Join(dir, FilterKeyFile)
|
||||
if raw, err := os.ReadFile(path); err == nil {
|
||||
if key := strings.TrimSpace(string(raw)); len(key) >= 32 && !strings.ContainsAny(key, " \t\r\n") {
|
||||
return key, nil
|
||||
}
|
||||
// Empty or cut short: not a key anyone should sign with. Replaced below.
|
||||
} else if !os.IsNotExist(err) {
|
||||
return "", fmt.Errorf("read %s: %w", path, err)
|
||||
}
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return "", fmt.Errorf("create %s: %w", dir, err)
|
||||
}
|
||||
buf := make([]byte, 32)
|
||||
if _, err := rand.Read(buf); err != nil {
|
||||
return "", fmt.Errorf("generate the filter key: %w", err)
|
||||
}
|
||||
key := hex.EncodeToString(buf)
|
||||
if err := atomicfile.Write(path, []byte(key+"\n"), 0o644); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return key, nil
|
||||
}
|
||||
@@ -0,0 +1,440 @@
|
||||
package inbound
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mixeme/selfpost/internal/store"
|
||||
)
|
||||
|
||||
func testLists(t *testing.T) (*Lists, *store.Store, string) {
|
||||
t.Helper()
|
||||
st, err := store.Open(filepath.Join(t.TempDir(), "test.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { st.Close() })
|
||||
dir := filepath.Join(t.TempDir(), "antispam")
|
||||
return NewLists(st, dir), st, dir
|
||||
}
|
||||
|
||||
// mapLines reads a map file without its comment line.
|
||||
func mapLines(t *testing.T, dir, name string) []string {
|
||||
t.Helper()
|
||||
raw, err := os.ReadFile(filepath.Join(dir, name))
|
||||
if err != nil {
|
||||
t.Fatalf("read %s: %v", name, err)
|
||||
}
|
||||
var out []string
|
||||
for _, line := range strings.Split(strings.TrimRight(string(raw), "\n"), "\n") {
|
||||
if !strings.HasPrefix(line, "#") {
|
||||
out = append(out, line)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func TestNormalizeListValue(t *testing.T) {
|
||||
for _, c := range []struct{ kind, raw, want string }{
|
||||
{store.SpamEntryAddress, " Billing@Vendor.Example ", "billing@vendor.example"},
|
||||
{store.SpamEntryAddress, "srs0=ab12=cd=example.org=user@forwarder.example", "srs0=ab12=cd=example.org=user@forwarder.example"},
|
||||
{store.SpamEntryDomain, "@Evil.Example", "evil.example"},
|
||||
{store.SpamEntryDomain, "sub.evil.example", "sub.evil.example"},
|
||||
{store.SpamEntryIP, "192.0.2.7", "192.0.2.7"},
|
||||
{store.SpamEntryIP, "192.0.2.77/24", "192.0.2.0/24"},
|
||||
{store.SpamEntryIP, "192.0.2.7/32", "192.0.2.7"},
|
||||
{store.SpamEntryIP, "2001:db8::1/128", "2001:db8::1"},
|
||||
{store.SpamEntryIP, "::ffff:192.0.2.7", "192.0.2.7"},
|
||||
{store.SpamEntryIP, "2001:DB8:0:0::1", "2001:db8::1"},
|
||||
{store.SpamEntryIP, "2001:db8:1::5/48", "2001:db8:1::/48"},
|
||||
} {
|
||||
got, err := normalizeListValue(c.kind, c.raw)
|
||||
if err != nil || got != c.want {
|
||||
t.Errorf("normalizeListValue(%s, %q) = %q, %v; want %q", c.kind, c.raw, got, err, c.want)
|
||||
}
|
||||
}
|
||||
|
||||
for _, c := range []struct{ kind, raw string }{
|
||||
{store.SpamEntryAddress, ""},
|
||||
{store.SpamEntryAddress, "no-at-sign"},
|
||||
{store.SpamEntryAddress, "@example.com"},
|
||||
{store.SpamEntryAddress, "a@localhost"},
|
||||
{store.SpamEntryAddress, "a b@example.com"},
|
||||
{store.SpamEntryAddress, "a#b@example.com"},
|
||||
{store.SpamEntryAddress, "a\nb@example.com"},
|
||||
{store.SpamEntryAddress, ".a@example.com"},
|
||||
{store.SpamEntryDomain, ""},
|
||||
{store.SpamEntryDomain, "example"},
|
||||
{store.SpamEntryDomain, "exa mple.com"},
|
||||
{store.SpamEntryDomain, "example.com # all of it"},
|
||||
{store.SpamEntryDomain, "user@example.com"},
|
||||
{store.SpamEntryIP, ""},
|
||||
{store.SpamEntryIP, "example.com"},
|
||||
{store.SpamEntryIP, "192.0.2.300"},
|
||||
{store.SpamEntryIP, "192.0.2.0/33"},
|
||||
{store.SpamEntryIP, "0.0.0.0/0"},
|
||||
{store.SpamEntryIP, "::/0"},
|
||||
{store.SpamEntryIP, "fe80::1%eth0"},
|
||||
{store.SpamEntryIP, "::ffff:192.0.2.0/120"},
|
||||
{store.SpamEntryIP, "192.0.2.7 192.0.2.8"},
|
||||
{"subject", "pills"},
|
||||
{"", "x"},
|
||||
} {
|
||||
got, err := normalizeListValue(c.kind, c.raw)
|
||||
var ve *ValidationError
|
||||
if err == nil || !errors.As(err, &ve) {
|
||||
t.Errorf("normalizeListValue(%q, %q) = %q, %v; want a ValidationError", c.kind, c.raw, got, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestListsAddWritesTheMaps(t *testing.T) {
|
||||
l, _, dir := testLists(t)
|
||||
for _, e := range []struct{ list, kind, value string }{
|
||||
{store.SpamListAllow, store.SpamEntryAddress, "Billing@Vendor.Example"},
|
||||
{store.SpamListAllow, store.SpamEntryAddress, "accounts@vendor.example"},
|
||||
{store.SpamListDeny, store.SpamEntryDomain, "@evil.example"},
|
||||
{store.SpamListDeny, store.SpamEntryIP, "198.51.100.9/24"},
|
||||
// The same value on both lists is two entries: deny wins in the filter.
|
||||
{store.SpamListDeny, store.SpamEntryAddress, "billing@vendor.example"},
|
||||
} {
|
||||
if _, err := l.Add(e.list, e.kind, e.value, " why "); err != nil {
|
||||
t.Fatalf("Add(%s, %s, %q): %v", e.list, e.kind, e.value, err)
|
||||
}
|
||||
}
|
||||
|
||||
want := map[string]string{
|
||||
"allow_from.map": "accounts@vendor.example billing@vendor.example",
|
||||
"allow_domain.map": "",
|
||||
"allow_ip.map": "",
|
||||
"deny_from.map": "billing@vendor.example",
|
||||
"deny_domain.map": "evil.example",
|
||||
"deny_ip.map": "198.51.100.0/24",
|
||||
}
|
||||
for name, lines := range want {
|
||||
if got := strings.Join(mapLines(t, dir, name), " "); got != lines {
|
||||
t.Errorf("%s = %q, want %q", name, got, lines)
|
||||
}
|
||||
// The filter is another user in another container.
|
||||
if info, err := os.Stat(filepath.Join(dir, name)); err != nil {
|
||||
t.Errorf("stat %s: %v", name, err)
|
||||
} else if info.Mode().Perm()&0o004 == 0 && os.PathSeparator == '/' {
|
||||
t.Errorf("%s is mode %o, not readable by the filter", name, info.Mode().Perm())
|
||||
}
|
||||
}
|
||||
if files, _ := os.ReadDir(dir); len(files) != len(want) {
|
||||
t.Errorf("the directory holds %d files, want %d (no temp file left behind)", len(files), len(want))
|
||||
}
|
||||
|
||||
entries, err := l.Entries()
|
||||
if err != nil || len(entries) != 5 {
|
||||
t.Fatalf("Entries = %d, %v", len(entries), err)
|
||||
}
|
||||
if entries[0].Note != "why" {
|
||||
t.Errorf("note = %q, want it trimmed", entries[0].Note)
|
||||
}
|
||||
}
|
||||
|
||||
func TestListsRefusals(t *testing.T) {
|
||||
l, _, dir := testLists(t)
|
||||
if _, err := l.Add(store.SpamListAllow, store.SpamEntryIP, "192.0.2.7", ""); err != nil {
|
||||
t.Fatalf("Add: %v", err)
|
||||
}
|
||||
// Written differently, the same entry.
|
||||
if _, err := l.Add(store.SpamListAllow, store.SpamEntryIP, "::ffff:192.0.2.7", ""); !errors.Is(err, store.ErrSpamListEntryExists) {
|
||||
t.Errorf("the same address again: %v, want ErrSpamListEntryExists", err)
|
||||
}
|
||||
var ve *ValidationError
|
||||
for name, add := range map[string]func() error{
|
||||
"an unknown list": func() error { _, err := l.Add("block", store.SpamEntryIP, "192.0.2.8", ""); return err },
|
||||
"a bad value": func() error { _, err := l.Add(store.SpamListDeny, store.SpamEntryIP, "nope", ""); return err },
|
||||
"a long note": func() error {
|
||||
_, err := l.Add(store.SpamListDeny, store.SpamEntryIP, "192.0.2.8", strings.Repeat("x", noteMaxRunes+1))
|
||||
return err
|
||||
},
|
||||
"a note of two lines": func() error {
|
||||
_, err := l.Add(store.SpamListDeny, store.SpamEntryIP, "192.0.2.8", "one\ntwo")
|
||||
return err
|
||||
},
|
||||
} {
|
||||
if err := add(); !errors.As(err, &ve) {
|
||||
t.Errorf("%s: %v, want a ValidationError", name, err)
|
||||
}
|
||||
}
|
||||
if got := strings.Join(mapLines(t, dir, "deny_ip.map"), " "); got != "" {
|
||||
t.Errorf("deny_ip.map = %q after refusals only", got)
|
||||
}
|
||||
if entries, _ := l.Entries(); len(entries) != 1 {
|
||||
t.Errorf("entries = %d, want the first one alone", len(entries))
|
||||
}
|
||||
}
|
||||
|
||||
func TestListsDelete(t *testing.T) {
|
||||
l, _, dir := testLists(t)
|
||||
e, err := l.Add(store.SpamListDeny, store.SpamEntryDomain, "evil.example", "")
|
||||
if err != nil {
|
||||
t.Fatalf("Add: %v", err)
|
||||
}
|
||||
if err := l.Delete(e.ID); err != nil {
|
||||
t.Fatalf("Delete: %v", err)
|
||||
}
|
||||
if got := mapLines(t, dir, "deny_domain.map"); len(got) != 0 {
|
||||
t.Errorf("deny_domain.map = %v after the delete", got)
|
||||
}
|
||||
if err := l.Delete(e.ID); !errors.Is(err, store.ErrSpamListEntryNotFound) {
|
||||
t.Errorf("a second delete: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// An entry the filter never got must not stay on the page.
|
||||
func TestListsAddRollsBackWhenTheMapsCannotBeWritten(t *testing.T) {
|
||||
st, err := store.Open(filepath.Join(t.TempDir(), "test.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
// A file where the directory should be.
|
||||
blocked := filepath.Join(t.TempDir(), "antispam")
|
||||
if err := os.WriteFile(blocked, nil, 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
l := NewLists(st, blocked)
|
||||
if _, err := l.Add(store.SpamListDeny, store.SpamEntryDomain, "evil.example", ""); err == nil {
|
||||
t.Fatal("Add succeeded with nowhere to write the maps")
|
||||
}
|
||||
if entries, _ := l.Entries(); len(entries) != 0 {
|
||||
t.Errorf("the entry stayed: %+v", entries)
|
||||
}
|
||||
}
|
||||
|
||||
// The writer's own guard: a row that got into the table some other way is not
|
||||
// written into a map.
|
||||
func TestListsSyncRefusesAnUnsafeRow(t *testing.T) {
|
||||
l, st, dir := testLists(t)
|
||||
if _, err := st.AddSpamListEntry(store.SpamListDeny, store.SpamEntryDomain, "evil.example\ngood.example", ""); err != nil {
|
||||
t.Fatalf("AddSpamListEntry: %v", err)
|
||||
}
|
||||
if err := l.Sync(); err == nil {
|
||||
t.Fatal("Sync wrote a value with a newline in it")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, "deny_domain.map")); err == nil {
|
||||
t.Error("a map file was written")
|
||||
}
|
||||
}
|
||||
|
||||
// With nothing on the lists all six files are there, empty.
|
||||
func TestListsSyncWritesEmptyMaps(t *testing.T) {
|
||||
l, _, dir := testLists(t)
|
||||
if err := l.Sync(); err != nil {
|
||||
t.Fatalf("Sync: %v", err)
|
||||
}
|
||||
for _, name := range mapFiles {
|
||||
if got := mapLines(t, dir, name); len(got) != 0 {
|
||||
t.Errorf("%s = %v", name, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestListsLastWritten(t *testing.T) {
|
||||
l, _, dir := testLists(t)
|
||||
if got := l.LastWritten(); !got.IsZero() {
|
||||
t.Fatalf("LastWritten before anything was written = %v", got)
|
||||
}
|
||||
if err := l.Sync(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := l.LastWritten(); got.IsZero() || time.Since(got) > time.Minute {
|
||||
t.Errorf("LastWritten after a sync = %v", got)
|
||||
}
|
||||
|
||||
// The newest of the six counts, whichever it is.
|
||||
old := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||
for _, name := range mapFiles {
|
||||
if err := os.Chtimes(filepath.Join(dir, name), old, old); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
newest := old.Add(48 * time.Hour)
|
||||
if err := os.Chtimes(filepath.Join(dir, "deny_domain.map"), newest, newest); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := l.LastWritten(); !got.Equal(newest) {
|
||||
t.Errorf("LastWritten = %v, want %v", got, newest)
|
||||
}
|
||||
|
||||
// A file that is not one of the six does not count.
|
||||
other := newest.Add(time.Hour)
|
||||
if err := os.WriteFile(filepath.Join(dir, "notes.txt"), nil, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Chtimes(filepath.Join(dir, "notes.txt"), other, other); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := l.LastWritten(); !got.Equal(newest) {
|
||||
t.Errorf("LastWritten counts a stray file: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// How often an entry decided a message is read off the journal: the list that
|
||||
// decided, and whether the entry fits who the message was from.
|
||||
func TestListsUsage(t *testing.T) {
|
||||
l, st, _ := testLists(t)
|
||||
add := func(list, kind, value string) int64 {
|
||||
t.Helper()
|
||||
e, err := l.Add(list, kind, value, "")
|
||||
if err != nil {
|
||||
t.Fatalf("Add(%s, %s, %s): %v", list, kind, value, err)
|
||||
}
|
||||
return e.ID
|
||||
}
|
||||
denyDomain := add(store.SpamListDeny, store.SpamEntryDomain, "evil.example")
|
||||
denyAddr := add(store.SpamListDeny, store.SpamEntryAddress, "spammer@evil.example")
|
||||
denyNet := add(store.SpamListDeny, store.SpamEntryIP, "198.51.100.0/24")
|
||||
denyV6 := add(store.SpamListDeny, store.SpamEntryIP, "2001:db8::/32")
|
||||
allowAddr := add(store.SpamListAllow, store.SpamEntryAddress, "billing@vendor.example")
|
||||
allowDomain := add(store.SpamListAllow, store.SpamEntryDomain, "vendor.example")
|
||||
allowIP := add(store.SpamListAllow, store.SpamEntryIP, "203.0.113.5")
|
||||
unused := add(store.SpamListAllow, store.SpamEntryDomain, "quiet.example")
|
||||
|
||||
const denied = "5.7.1 Refused by this server's deny list"
|
||||
at := time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC)
|
||||
hour := 0
|
||||
row := func(from, ip, decision, engine, detail string) {
|
||||
t.Helper()
|
||||
hour++
|
||||
err := st.InsertInboundSpam(store.InboundSpamEntry{
|
||||
InboundDomain: "in.example.com", To: "a@in.example.com", From: from, ClientIP: ip,
|
||||
Decision: decision, Engine: engine, Detail: detail, At: at.Add(time.Duration(hour) * time.Hour),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
// hour 1: the address and its domain are both on the deny list, and both fit.
|
||||
row("Spammer@Evil.Example", "192.0.2.9", store.SpamDecisionReject, store.SpamEngineRspamd, denied)
|
||||
// hour 2: another address of the domain.
|
||||
row("other@evil.example", "192.0.2.9", store.SpamDecisionReject, store.SpamEngineRspamd, denied)
|
||||
// hour 3, 4: the networks, v4 (as Postfix may log it, mapped) and v6.
|
||||
row("x@elsewhere.example", "::ffff:198.51.100.77", store.SpamDecisionReject, store.SpamEngineRspamd, denied)
|
||||
row("x@elsewhere.example", "2001:db8::1", store.SpamDecisionReject, store.SpamEngineRspamd, denied)
|
||||
// hour 5: a subdomain is not the domain.
|
||||
row("x@sub.evil.example", "192.0.2.9", store.SpamDecisionReject, store.SpamEngineRspamd, denied)
|
||||
// hour 6: allowed by its address and its domain - two symbols, two entries.
|
||||
row("billing@vendor.example", "192.0.2.50", store.SpamDecisionAccept, store.SpamEngineRspamd,
|
||||
"no action, score -199.00 / 15.00; SELFPOST_ALLOW_DOMAIN(-100.00) SELFPOST_ALLOW_FROM(-100.00)")
|
||||
// hour 7: allowed by its domain alone.
|
||||
row("accounts@vendor.example", "192.0.2.50", store.SpamDecisionAccept, store.SpamEngineRspamd,
|
||||
"no action, score -99.00 / 15.00; SELFPOST_ALLOW_DOMAIN(-100.00)")
|
||||
// hour 8: allowed by the client address; the sender's domain is not why.
|
||||
row("x@quiet.example", "203.0.113.5", store.SpamDecisionAccept, store.SpamEngineRspamd,
|
||||
"no action, score -99.00 / 15.00; SELFPOST_ALLOW_IP(-100.00)")
|
||||
// Nobody's: the filter's own refusal, an accept with no list symbol, a
|
||||
// refusal by Postfix, and a sender who spells the mark in his own address.
|
||||
row("spammer@evil.example", "192.0.2.9", store.SpamDecisionReject, store.SpamEngineRspamd, "5.7.1 Spam message rejected")
|
||||
row("billing@vendor.example", "192.0.2.50", store.SpamDecisionAccept, store.SpamEngineRspamd, "no action, score 0.10 / 15.00")
|
||||
row("spammer@evil.example", "198.51.100.1", store.SpamDecisionReject, store.SpamEnginePostfix, "550 5.1.1 <nobody@in.example.com>: this server's deny list")
|
||||
row("billing@vendor.example", "203.0.113.5", store.SpamDecisionAccept, store.SpamEngineSelfPost, "")
|
||||
|
||||
usage, err := l.Usage()
|
||||
if err != nil {
|
||||
t.Fatalf("Usage: %v", err)
|
||||
}
|
||||
for name, c := range map[string]struct {
|
||||
id int64
|
||||
hits int
|
||||
last int // hour
|
||||
}{
|
||||
"deny domain": {denyDomain, 2, 2}, "deny address": {denyAddr, 1, 1},
|
||||
"deny network": {denyNet, 1, 3}, "deny v6 network": {denyV6, 1, 4},
|
||||
"allow address": {allowAddr, 1, 6}, "allow domain": {allowDomain, 2, 7}, "allow ip": {allowIP, 1, 8},
|
||||
"unused": {unused, 0, 0},
|
||||
} {
|
||||
got := usage[c.id]
|
||||
wantLast := time.Time{}
|
||||
if c.hits > 0 {
|
||||
wantLast = at.Add(time.Duration(c.last) * time.Hour)
|
||||
}
|
||||
if got.Hits != c.hits || !got.Last.Equal(wantLast) {
|
||||
t.Errorf("%s: %d hits, last %v; want %d, %v", name, got.Hits, got.Last, c.hits, wantLast)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestListsUsageWithNoEntries(t *testing.T) {
|
||||
l, _, _ := testLists(t)
|
||||
if usage, err := l.Usage(); err != nil || len(usage) != 0 {
|
||||
t.Errorf("Usage = %v, %v", usage, err)
|
||||
}
|
||||
}
|
||||
|
||||
// The same address written as a network of one is the same entry.
|
||||
func TestListsOneAddressNetworkIsTheAddress(t *testing.T) {
|
||||
l, _, dir := testLists(t)
|
||||
if _, err := l.Add(store.SpamListDeny, store.SpamEntryIP, "192.0.2.7", ""); err != nil {
|
||||
t.Fatalf("Add: %v", err)
|
||||
}
|
||||
if _, err := l.Add(store.SpamListDeny, store.SpamEntryIP, "192.0.2.7/32", ""); !errors.Is(err, store.ErrSpamListEntryExists) {
|
||||
t.Errorf("192.0.2.7/32 after 192.0.2.7: %v, want ErrSpamListEntryExists", err)
|
||||
}
|
||||
if got := strings.Join(mapLines(t, dir, "deny_ip.map"), " "); got != "192.0.2.7" {
|
||||
t.Errorf("deny_ip.map = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestListable(t *testing.T) {
|
||||
for value, want := range map[string]bool{
|
||||
"billing@vendor.example": true, "srs0=ab=cd=example.org=user@forwarder.example": true,
|
||||
"o'brien@example.org": false, "a#b@example.org": false, "": false, "not-an-address": false,
|
||||
} {
|
||||
if got := Listable(store.SpamEntryAddress, value); got != want {
|
||||
t.Errorf("Listable(address, %q) = %v, want %v", value, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The key the filter signs with is made once and then read, and is readable by
|
||||
// the filter, which is another user in another container.
|
||||
func TestFilterKey(t *testing.T) {
|
||||
dir := filepath.Join(t.TempDir(), "antispam")
|
||||
key, err := FilterKey(dir)
|
||||
if err != nil || len(key) != 64 || strings.Trim(key, "0123456789abcdef") != "" {
|
||||
t.Fatalf("FilterKey = %q, %v", key, err)
|
||||
}
|
||||
again, err := FilterKey(dir)
|
||||
if err != nil || again != key {
|
||||
t.Errorf("a second call = %q, %v; want the same key", again, err)
|
||||
}
|
||||
raw, err := os.ReadFile(filepath.Join(dir, FilterKeyFile))
|
||||
if err != nil || string(raw) != key+"\n" {
|
||||
t.Errorf("file = %q, %v", raw, err)
|
||||
}
|
||||
if info, _ := os.Stat(filepath.Join(dir, FilterKeyFile)); os.PathSeparator == '/' && info.Mode().Perm() != 0o644 {
|
||||
t.Errorf("mode = %o", info.Mode().Perm())
|
||||
}
|
||||
|
||||
// A file that is not a key is replaced, not signed with.
|
||||
if err := os.WriteFile(filepath.Join(dir, FilterKeyFile), []byte("short\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
fresh, err := FilterKey(dir)
|
||||
if err != nil || len(fresh) != 64 || fresh == key {
|
||||
t.Errorf("after a damaged file: %q, %v", fresh, err)
|
||||
}
|
||||
|
||||
// The lists leave it alone.
|
||||
st, err := store.Open(filepath.Join(t.TempDir(), "test.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
if err := NewLists(st, dir).Sync(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if kept, _ := FilterKey(dir); kept != fresh {
|
||||
t.Errorf("the key changed when the lists were written")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,300 @@
|
||||
package logtail
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/mixeme/selfpost/internal/store"
|
||||
)
|
||||
|
||||
// InboundStore is what the tailer needs of the inbound antispam journal:
|
||||
// writing the rows for refused mail, knowing which domains are inbound, and
|
||||
// pruning. *store.Store satisfies it.
|
||||
type InboundStore interface {
|
||||
InsertInboundSpam(e store.InboundSpamEntry) error
|
||||
InsertInboundSpamOnce(e store.InboundSpamEntry) (bool, error)
|
||||
IsInboundDomain(name string) (bool, error)
|
||||
PruneInboundSpam(cutoff time.Time, keep int) (int64, error)
|
||||
}
|
||||
|
||||
// Inbound turns on the inbound half of the tailer: journal rows for mail that
|
||||
// port 25 refused, and the journal's retention. Run takes nil when the inbound
|
||||
// relay is off.
|
||||
type Inbound struct {
|
||||
Store InboundStore
|
||||
Retention RetentionDays
|
||||
}
|
||||
|
||||
// The inbound journal milter writes a row for mail that was accepted; mail
|
||||
// that was refused never reaches it, and what is known about it is one line
|
||||
// of mail.log:
|
||||
//
|
||||
// postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from mx.example.net[203.0.113.9]: 550 5.1.1 <nobody@in.example.com>: Recipient address rejected: User unknown in relay recipient table; from=<a@example.net> to=<nobody@in.example.com> proto=ESMTP helo=<mx.example.net>
|
||||
// postfix/cleanup[52]: 4A1B2C3D: milter-reject: END-OF-MESSAGE from mx.example.net[203.0.113.9]: 5.7.1 Spam message rejected; from=<a@example.net> to=<b@in.example.com> proto=ESMTP helo=<mx.example.net>
|
||||
//
|
||||
// A message the filter holds (milter-hold) is not read here: it goes on to the
|
||||
// inbound journal milter, which writes its row with the subject.
|
||||
//
|
||||
// The first is Postfix's own policy, written by the port-25 smtpd, which
|
||||
// build/postfix-config.sh names postfix/inbound so that its lines are told
|
||||
// from those of the submission ports. The second is the filter's verdict at
|
||||
// end-of-message, which cleanup(8) writes under one name for every listener -
|
||||
// so it counts only for a queue id the port-25 smtpd was seen to open
|
||||
// (inboundClientRe).
|
||||
//
|
||||
// Both are matched from the head of the line, after the timestamp and host
|
||||
// name: the tail of such a line is the sender's own words (HELO name,
|
||||
// addresses), and a pattern that could start anywhere would let him write
|
||||
// journal rows.
|
||||
//
|
||||
// For the same reason the reply is the shortest text before "; from=<", and
|
||||
// the HELO name - the last thing on the line, which Postfix logs as the client
|
||||
// wrote it, spaces and brackets included - is whatever is left. Taking the
|
||||
// longest reply instead would let a HELO name that itself spells
|
||||
// "; from=<…> to=<…> proto=…" supply the sender, the recipient and the reply
|
||||
// of the row (security review, 2026-10-12).
|
||||
var (
|
||||
inboundRejectRe = regexp.MustCompile(`^\S+ postfix/(inbound/smtpd|cleanup)\[\d+\]: ([0-9A-Za-z]+): (reject|milter-reject|milter-discard): \S+ from [^\[\s]*\[([0-9A-Fa-f:.]+)\]: (.*?); from=<([^>]*)> to=<([^>]*)> proto=\S+(?: helo=<.*>)?$`)
|
||||
inboundClientRe = regexp.MustCompile(`^\S+ postfix/inbound/smtpd\[\d+\]: ([0-9A-Za-z]+): client=`)
|
||||
// A queue id stops being port 25's when the message is queued (qmgr takes
|
||||
// it: the filter let it through) or when another listener opens a message
|
||||
// under the same id - Postfix reuses ids.
|
||||
inboundForgetRe = regexp.MustCompile(`^\S+ postfix/(?:qmgr\[\d+\]: ([0-9A-Za-z]+): from=|(?:[a-z0-9-]+/)?smtpd\[\d+\]: ([0-9A-Za-z]+): client=)`)
|
||||
)
|
||||
|
||||
// A row holds one line of reason; Postfix's reply text carries the far end's
|
||||
// bytes and is cut and reduced to printable ASCII like any of them.
|
||||
const inboundDetailMax = 255
|
||||
|
||||
// inboundQueueIDs is how many port-25 queue ids are remembered while waiting
|
||||
// for cleanup's verdict line on them. An id is forgotten when the verdict
|
||||
// comes, when the message is queued, or when another listener reuses it; the
|
||||
// cap bounds what is left - messages the client abandoned - and a flood.
|
||||
const inboundQueueIDs = 4096
|
||||
|
||||
// inboundPruneEvery is how many rows the tailer writes before it has the
|
||||
// journal cut back to its row cap, without waiting for the six-hourly prune.
|
||||
// Refusals are written at the rate strangers send RCPT TO, which no limit of
|
||||
// Postfix's counts, so between two prunes the journal would otherwise grow as
|
||||
// fast as someone cared to make it.
|
||||
const inboundPruneEvery = 500
|
||||
|
||||
// inboundJournal is the per-run state of the inbound half.
|
||||
type inboundJournal struct {
|
||||
st InboundStore
|
||||
// port25 holds the queue ids the port-25 smtpd opened and nobody has
|
||||
// accounted for yet, each with the number of its arrival; order is those
|
||||
// arrivals, for dropping the oldest. An id that was forgotten and opened
|
||||
// again is in order twice: the number tells the stale entry from the live
|
||||
// one when the stale one comes up for dropping.
|
||||
port25 map[string]uint64
|
||||
order []queued
|
||||
seq uint64
|
||||
// written counts the rows since the journal was last cut to its cap.
|
||||
written int
|
||||
}
|
||||
|
||||
type queued struct {
|
||||
id string
|
||||
seq uint64
|
||||
}
|
||||
|
||||
func newInboundJournal(st InboundStore) *inboundJournal {
|
||||
return &inboundJournal{st: st, port25: make(map[string]uint64)}
|
||||
}
|
||||
|
||||
// handle looks at one mail.log line and writes a journal row when it is a
|
||||
// refusal of mail for an inbound domain. Errors are logged; the tailer goes on.
|
||||
//
|
||||
// catchingUp says the line is one of those the tailer reads when it starts,
|
||||
// from where it last saved its place to the end of the file. It saves its
|
||||
// place every few seconds, so after an unclean stop some of those lines were
|
||||
// read before; a row is then written only if it is not there already. Lines
|
||||
// read as they are written are always new, and two identical refusals in one
|
||||
// second stay two rows.
|
||||
func (j *inboundJournal) handle(line string, now time.Time, catchingUp bool) {
|
||||
at, rest := lineTime(line, now)
|
||||
if m := inboundClientRe.FindStringSubmatch(rest); m != nil {
|
||||
j.remember(m[1])
|
||||
return
|
||||
}
|
||||
if m := inboundForgetRe.FindStringSubmatch(rest); m != nil {
|
||||
delete(j.port25, m[1]+m[2])
|
||||
return
|
||||
}
|
||||
m := inboundRejectRe.FindStringSubmatch(rest)
|
||||
if m == nil {
|
||||
return
|
||||
}
|
||||
program, queueID, kind, clientIP, reason, from, to := m[1], m[2], m[3], m[4], m[5], m[6], m[7]
|
||||
// The sender's address is his own text too, and with a quoted local part it
|
||||
// may hold "> to=<…> proto=…" - which would end the sender early and supply
|
||||
// the recipient. Such a line shows it: the sender is left with an unclosed
|
||||
// quote. An address with a quote, a bracket or a space in it is not one the
|
||||
// journal keeps.
|
||||
if strings.ContainsAny(from, "\"<> ") || strings.ContainsAny(to, "\"<> ") {
|
||||
return
|
||||
}
|
||||
if program == "cleanup" {
|
||||
if _, ok := j.port25[queueID]; !ok {
|
||||
return // a submission-port message, or one opened before this run
|
||||
}
|
||||
delete(j.port25, queueID)
|
||||
}
|
||||
|
||||
// Only mail for an inbound domain is the relay's. "Relay access denied"
|
||||
// for a stranger's domain is every scanner on the internet, and a line
|
||||
// with no recipient (size limit at MAIL) belongs to no domain.
|
||||
domain := ""
|
||||
if i := strings.LastIndexByte(to, '@'); i >= 0 {
|
||||
domain = strings.ToLower(to[i+1:])
|
||||
}
|
||||
if domain == "" {
|
||||
return
|
||||
}
|
||||
ok, err := j.st.IsInboundDomain(domain)
|
||||
if err != nil {
|
||||
log.Printf("log-tailer: inbound journal: look up %s: %v", domain, err)
|
||||
return
|
||||
}
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
e := store.InboundSpamEntry{
|
||||
InboundDomain: domain,
|
||||
ClientIP: clientIP,
|
||||
From: from,
|
||||
To: to,
|
||||
Decision: store.SpamDecisionReject,
|
||||
Engine: store.SpamEngineRspamd,
|
||||
Detail: printable(reason, inboundDetailMax),
|
||||
At: at,
|
||||
}
|
||||
if kind == "reject" {
|
||||
e.Engine = store.SpamEnginePostfix
|
||||
}
|
||||
// "450 4.7.1 …" from Postfix, "4.7.1 …" from a milter: either way a reply
|
||||
// that starts with 4 asks the sender to come back.
|
||||
if e.Decision == store.SpamDecisionReject && strings.HasPrefix(reason, "4") {
|
||||
e.Decision = store.SpamDecisionTempfail
|
||||
}
|
||||
if catchingUp {
|
||||
if wrote, err := j.st.InsertInboundSpamOnce(e); err != nil {
|
||||
log.Printf("log-tailer: inbound journal: record %s -> %s: %v", from, to, err)
|
||||
return
|
||||
} else if !wrote {
|
||||
return // read before the last stop
|
||||
}
|
||||
} else if err := j.st.InsertInboundSpam(e); err != nil {
|
||||
log.Printf("log-tailer: inbound journal: record %s -> %s: %v", from, to, err)
|
||||
return
|
||||
}
|
||||
if j.written++; j.written >= inboundPruneEvery {
|
||||
j.written = 0
|
||||
// The zero time removes nothing by age: this is the cap alone.
|
||||
if _, err := j.st.PruneInboundSpam(time.Time{}, inboundSpamLogMaxRows); err != nil {
|
||||
log.Printf("log-tailer: inbound journal: cut to %d rows: %v", inboundSpamLogMaxRows, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (j *inboundJournal) remember(queueID string) {
|
||||
if _, ok := j.port25[queueID]; ok {
|
||||
return
|
||||
}
|
||||
for len(j.order) >= inboundQueueIDs {
|
||||
oldest := j.order[0]
|
||||
j.order = j.order[1:]
|
||||
if j.port25[oldest.id] == oldest.seq {
|
||||
delete(j.port25, oldest.id)
|
||||
}
|
||||
}
|
||||
j.seq++
|
||||
j.port25[queueID] = j.seq
|
||||
j.order = append(j.order, queued{queueID, j.seq})
|
||||
}
|
||||
|
||||
// printable cuts s to max bytes and replaces everything outside printable
|
||||
// ASCII.
|
||||
func printable(s string, max int) string {
|
||||
if len(s) > max {
|
||||
s = s[:max]
|
||||
}
|
||||
b := []byte(s)
|
||||
for i, c := range b {
|
||||
if c < 0x20 || c > 0x7e {
|
||||
b[i] = '?'
|
||||
}
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// lineTime reads the time at the head of a mail.log line and returns it with
|
||||
// the rest of the line. A refusal is dated by its line and not by when the
|
||||
// tailer got to it: after the panel was down the two are hours apart. The
|
||||
// syslog format carries neither year nor zone - the year is taken from now
|
||||
// (a stamp that would then lie in the future is last year's), the zone is the
|
||||
// container's, which is what postlogd wrote it in. A line with no readable
|
||||
// stamp is dated now.
|
||||
func lineTime(line string, now time.Time) (time.Time, string) {
|
||||
if m := isoStampRe.FindStringSubmatch(line); m != nil {
|
||||
rest := strings.TrimSpace(line[len(m[0]):])
|
||||
if t, err := time.Parse(time.RFC3339Nano, strings.TrimSpace(m[0])); err == nil {
|
||||
return t.UTC(), rest
|
||||
}
|
||||
return now, rest
|
||||
}
|
||||
if m := syslogStampRe.FindStringSubmatch(line); m != nil {
|
||||
rest := strings.TrimSpace(line[len(m[0]):])
|
||||
stamp := strings.Join(strings.Fields(m[1]), " ")
|
||||
t, err := time.ParseInLocation("Jan 2 15:04:05", stamp, time.Local)
|
||||
if err != nil {
|
||||
return now, rest
|
||||
}
|
||||
t = t.AddDate(now.In(time.Local).Year(), 0, 0)
|
||||
if t.After(now.Add(24 * time.Hour)) {
|
||||
t = t.AddDate(-1, 0, 0)
|
||||
}
|
||||
return t.UTC(), rest
|
||||
}
|
||||
return now, line
|
||||
}
|
||||
|
||||
// The journal keeps at most this many rows whatever their age: refusals are
|
||||
// written at the rate strangers knock on port 25.
|
||||
const inboundSpamLogMaxRows = 10000
|
||||
|
||||
// inboundRetentionLoop prunes the inbound journal at once and then on the
|
||||
// cadence of the send log.
|
||||
func inboundRetentionLoop(ctx context.Context, in *Inbound) {
|
||||
prune := func() {
|
||||
days := in.Retention()
|
||||
if days <= 0 {
|
||||
days = store.InboundSpamLogRetentionDaysDefault
|
||||
}
|
||||
n, err := in.Store.PruneInboundSpam(time.Now().UTC().AddDate(0, 0, -days), inboundSpamLogMaxRows)
|
||||
if err != nil {
|
||||
log.Printf("log-tailer: inbound journal prune: %v", err)
|
||||
return
|
||||
}
|
||||
if n > 0 {
|
||||
log.Printf("log-tailer: pruned %d inbound journal rows (older than %d days or beyond %d rows)", n, days, inboundSpamLogMaxRows)
|
||||
}
|
||||
}
|
||||
|
||||
prune()
|
||||
t := time.NewTicker(retentionInterval)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
prune()
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,333 @@
|
||||
package logtail
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mixeme/selfpost/internal/store"
|
||||
)
|
||||
|
||||
type fakeInboundStore struct {
|
||||
domains map[string]bool
|
||||
rows []store.InboundSpamEntry
|
||||
lookupErr error
|
||||
pruned []int
|
||||
}
|
||||
|
||||
func (f *fakeInboundStore) InsertInboundSpam(e store.InboundSpamEntry) error {
|
||||
f.rows = append(f.rows, e)
|
||||
return nil
|
||||
}
|
||||
|
||||
// InsertInboundSpamOnce writes unless an equal row is there, like the store's.
|
||||
func (f *fakeInboundStore) InsertInboundSpamOnce(e store.InboundSpamEntry) (bool, error) {
|
||||
for _, r := range f.rows {
|
||||
if r == e {
|
||||
return false, nil
|
||||
}
|
||||
}
|
||||
f.rows = append(f.rows, e)
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func (f *fakeInboundStore) IsInboundDomain(name string) (bool, error) {
|
||||
if f.lookupErr != nil {
|
||||
return false, f.lookupErr
|
||||
}
|
||||
return f.domains[name], nil
|
||||
}
|
||||
|
||||
func (f *fakeInboundStore) PruneInboundSpam(_ time.Time, keep int) (int64, error) {
|
||||
f.pruned = append(f.pruned, keep)
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
func inboundFixture() (*inboundJournal, *fakeInboundStore) {
|
||||
st := &fakeInboundStore{domains: map[string]bool{"in.example.com": true}}
|
||||
return newInboundJournal(st), st
|
||||
}
|
||||
|
||||
var tailNow = time.Date(2026, 10, 11, 18, 0, 0, 0, time.UTC)
|
||||
|
||||
const (
|
||||
head = "2026-10-11T12:00:05.123456+00:00 mail "
|
||||
unknownRcp = head + "postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from mx.example.net[203.0.113.9]: 550 5.1.1 <nobody@in.example.com>: Recipient address rejected: User unknown in relay recipient table; from=<a@example.net> to=<nobody@in.example.com> proto=ESMTP helo=<mx.example.net>"
|
||||
)
|
||||
|
||||
// Postfix's own refusal of a recipient at an inbound domain is a row, dated by
|
||||
// its line.
|
||||
func TestInboundRejectByPostfix(t *testing.T) {
|
||||
j, st := inboundFixture()
|
||||
j.handle(unknownRcp, tailNow, false)
|
||||
if len(st.rows) != 1 {
|
||||
t.Fatalf("rows = %+v", st.rows)
|
||||
}
|
||||
want := store.InboundSpamEntry{
|
||||
InboundDomain: "in.example.com", ClientIP: "203.0.113.9",
|
||||
From: "a@example.net", To: "nobody@in.example.com",
|
||||
Decision: store.SpamDecisionReject, Engine: store.SpamEnginePostfix,
|
||||
Detail: "550 5.1.1 <nobody@in.example.com>: Recipient address rejected: User unknown in relay recipient table",
|
||||
At: time.Date(2026, 10, 11, 12, 0, 5, 123456000, time.UTC),
|
||||
}
|
||||
if st.rows[0] != want {
|
||||
t.Errorf("row = %+v\nwant %+v", st.rows[0], want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInboundTempfailByPostfix(t *testing.T) {
|
||||
j, st := inboundFixture()
|
||||
j.handle(head+"postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from unknown[2001:db8::9]: 450 4.1.8 <a@nx.example>: Sender address rejected: Domain not found; from=<a@nx.example> to=<B@IN.example.com> proto=ESMTP helo=<x>", tailNow, false)
|
||||
if len(st.rows) != 1 || st.rows[0].Decision != store.SpamDecisionTempfail || st.rows[0].ClientIP != "2001:db8::9" || st.rows[0].InboundDomain != "in.example.com" {
|
||||
t.Errorf("rows = %+v", st.rows)
|
||||
}
|
||||
}
|
||||
|
||||
// The filter's verdict is written by cleanup, under one name for every
|
||||
// listener: it is the journal's only for a queue id port 25 opened.
|
||||
func TestInboundMilterVerdictNeedsAPort25QueueID(t *testing.T) {
|
||||
const verdict = "postfix/cleanup[52]: 4A1B2C3D: milter-reject: END-OF-MESSAGE from mx.example.net[203.0.113.9]: 5.7.1 Spam message rejected; from=<a@example.net> to=<b@in.example.com> proto=ESMTP helo=<mx.example.net>"
|
||||
|
||||
j, st := inboundFixture()
|
||||
// Opened by a submission port: postfix/smtpd, not postfix/inbound/smtpd.
|
||||
j.handle(head+"postfix/smtpd[20]: 4A1B2C3D: client=app.example.com[203.0.113.4], sasl_method=PLAIN, sasl_username=app", tailNow, false)
|
||||
j.handle(head+verdict, tailNow, false)
|
||||
if len(st.rows) != 0 {
|
||||
t.Fatalf("a submission-port verdict was journalled: %+v", st.rows)
|
||||
}
|
||||
|
||||
j.handle(head+"postfix/inbound/smtpd[41]: 4A1B2C3D: client=mx.example.net[203.0.113.9]", tailNow, false)
|
||||
j.handle(head+verdict, tailNow, false)
|
||||
if len(st.rows) != 1 {
|
||||
t.Fatalf("rows = %+v", st.rows)
|
||||
}
|
||||
got := st.rows[0]
|
||||
if got.Decision != store.SpamDecisionReject || got.Engine != store.SpamEngineRspamd || got.Detail != "5.7.1 Spam message rejected" {
|
||||
t.Errorf("row = %+v", got)
|
||||
}
|
||||
// One verdict per message: the id is forgotten with it.
|
||||
j.handle(head+verdict, tailNow, false)
|
||||
if len(st.rows) != 1 {
|
||||
t.Errorf("the same queue id was journalled twice")
|
||||
}
|
||||
}
|
||||
|
||||
func TestInboundMilterDecisions(t *testing.T) {
|
||||
for kind, c := range map[string]struct{ reply, decision string }{
|
||||
"milter-reject": {"4.7.1 Try again later", store.SpamDecisionTempfail},
|
||||
"milter-discard": {"milter triggers DISCARD action", store.SpamDecisionReject},
|
||||
} {
|
||||
j, st := inboundFixture()
|
||||
j.handle(head+"postfix/inbound/smtpd[41]: Q1: client=mx.example.net[203.0.113.9]", tailNow, false)
|
||||
j.handle(head+"postfix/cleanup[52]: Q1: "+kind+": END-OF-MESSAGE from mx.example.net[203.0.113.9]: "+c.reply+"; from=<a@example.net> to=<b@in.example.com> proto=ESMTP helo=<mx.example.net>", tailNow, false)
|
||||
if len(st.rows) != 1 || st.rows[0].Decision != c.decision || st.rows[0].Engine != store.SpamEngineRspamd {
|
||||
t.Errorf("%s: rows = %+v", kind, st.rows)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// What is not the relay's mail, or is not a refusal at all, leaves no row.
|
||||
func TestInboundIgnores(t *testing.T) {
|
||||
for name, line := range map[string]string{
|
||||
"another domain (relay access denied)": head + "postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from x[203.0.113.9]: 554 5.7.1 <c@elsewhere.example>: Relay access denied; from=<a@example.net> to=<c@elsewhere.example> proto=ESMTP helo=<x>",
|
||||
"no recipient (size limit at MAIL)": head + "postfix/inbound/smtpd[41]: NOQUEUE: reject: MAIL from x[203.0.113.9]: 552 5.3.4 Message size exceeds fixed limit; proto=ESMTP helo=<x>",
|
||||
"a submission port": head + "postfix/smtpd[20]: NOQUEUE: reject: RCPT from x[203.0.113.9]: 553 5.7.1 <b@in.example.com>: Sender address rejected: not owned by user app; from=<a@example.net> to=<b@in.example.com> proto=ESMTP helo=<x>",
|
||||
"a delivery line": head + "postfix/smtp[26]: 41E862C00D9E: to=<b@in.example.com>, relay=mx[203.0.113.9]:25, dsn=2.0.0, status=sent (250 OK)",
|
||||
"a connect line": head + "postfix/inbound/smtpd[41]: connect from mx.example.net[203.0.113.9]",
|
||||
// The HELO name is the sender's; a line cannot be started from inside one.
|
||||
"a refusal spelled inside another line": head + "postfix/inbound/smtpd[41]: warning: hostname x does not resolve: mail postfix/inbound/smtpd[1]: NOQUEUE: reject: RCPT from x[203.0.113.9]: 550 no; from=<a@example.net> to=<b@in.example.com> proto=ESMTP helo=<x>",
|
||||
"an address with a bracket in it": head + "postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from x[203.0.113.9]: 554 5.7.1 <b@elsewhere.example>: Relay access denied; from=<\"a> to=<v@in.example.com> proto=X helo=<y\"@example.net> to=<b@elsewhere.example> proto=ESMTP helo=<x>",
|
||||
} {
|
||||
j, st := inboundFixture()
|
||||
j.handle(line, tailNow, false)
|
||||
if len(st.rows) != 0 {
|
||||
t.Errorf("%s: journalled %+v", name, st.rows)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestInboundDetailIsCutAndPrintable(t *testing.T) {
|
||||
j, st := inboundFixture()
|
||||
j.handle(head+"postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from x[203.0.113.9]: 550 5.7.1 caf\xc3\xa9 \x1b[31m"+strings.Repeat("x", 400)+"; from=<a@example.net> to=<b@in.example.com> proto=ESMTP helo=<x>", tailNow, false)
|
||||
if len(st.rows) != 1 {
|
||||
t.Fatalf("rows = %+v", st.rows)
|
||||
}
|
||||
d := st.rows[0].Detail
|
||||
if len(d) != inboundDetailMax || !strings.HasPrefix(d, "550 5.7.1 caf?? ?[31mxxx") {
|
||||
t.Errorf("detail (%d bytes) = %q", len(d), d)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInboundLookupFailureLeavesNoRow(t *testing.T) {
|
||||
j, st := inboundFixture()
|
||||
st.lookupErr = errors.New("boom")
|
||||
j.handle(unknownRcp, tailNow, false)
|
||||
if len(st.rows) != 0 {
|
||||
t.Errorf("rows = %+v", st.rows)
|
||||
}
|
||||
}
|
||||
|
||||
// A message the filter holds goes on to the inbound journal milter, which
|
||||
// writes its row; a second one from the log would be the same message twice.
|
||||
func TestInboundHoldIsTheMiltersRow(t *testing.T) {
|
||||
j, st := inboundFixture()
|
||||
j.handle(head+"postfix/inbound/smtpd[41]: Q1: client=mx.example.net[203.0.113.9]", tailNow, false)
|
||||
j.handle(head+"postfix/cleanup[52]: Q1: milter-hold: END-OF-MESSAGE from mx.example.net[203.0.113.9]: milter triggers HOLD action; from=<a@example.net> to=<b@in.example.com> proto=ESMTP helo=<mx.example.net>", tailNow, false)
|
||||
if len(st.rows) != 0 {
|
||||
t.Errorf("rows = %+v", st.rows)
|
||||
}
|
||||
}
|
||||
|
||||
// The HELO name is the last thing on the line and is logged as the client
|
||||
// wrote it. It cannot supply the row: the reply ends at the first "; from=<",
|
||||
// which is Postfix's own.
|
||||
func TestInboundHeloCannotWriteTheRow(t *testing.T) {
|
||||
j, st := inboundFixture()
|
||||
j.handle(head+"postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from x[203.0.113.9]: 550 5.1.1 <nobody@in.example.com>: Recipient address rejected: User unknown in relay recipient table; from=<real@example.net> to=<nobody@in.example.com> proto=ESMTP helo=<x 5.7.1 Refused by this server's deny list; from=<victim@bank.example> to=<ceo@in.example.com> proto=ESMTP helo=<y>", tailNow, false)
|
||||
if len(st.rows) != 1 {
|
||||
t.Fatalf("rows = %+v", st.rows)
|
||||
}
|
||||
got := st.rows[0]
|
||||
if got.From != "real@example.net" || got.To != "nobody@in.example.com" ||
|
||||
got.Detail != "550 5.1.1 <nobody@in.example.com>: Recipient address rejected: User unknown in relay recipient table" {
|
||||
t.Errorf("row = %+v", got)
|
||||
}
|
||||
|
||||
// A HELO name with a closing bracket in it does not lose the line either.
|
||||
j, st = inboundFixture()
|
||||
j.handle(head+"postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from x[203.0.113.9]: 550 5.1.1 <nobody@in.example.com>: no; from=<real@example.net> to=<nobody@in.example.com> proto=ESMTP helo=<a>b>", tailNow, false)
|
||||
if len(st.rows) != 1 || st.rows[0].From != "real@example.net" {
|
||||
t.Errorf("rows = %+v", st.rows)
|
||||
}
|
||||
|
||||
// The same on a verdict line: the filter's reply is not extended by the
|
||||
// HELO name, so a stranger cannot make a refusal look like the deny list's.
|
||||
j, st = inboundFixture()
|
||||
j.handle(head+"postfix/inbound/smtpd[41]: Q1: client=x[203.0.113.9]", tailNow, false)
|
||||
j.handle(head+"postfix/cleanup[52]: Q1: milter-reject: END-OF-MESSAGE from x[203.0.113.9]: 5.7.1 Spam message rejected; from=<real@example.net> to=<b@in.example.com> proto=ESMTP helo=<x this server's deny list; from=<spoofed@example.org> to=<b@in.example.com> proto=ESMTP helo=<y>", tailNow, false)
|
||||
if len(st.rows) != 1 || st.rows[0].Detail != "5.7.1 Spam message rejected" || st.rows[0].From != "real@example.net" {
|
||||
t.Errorf("rows = %+v", st.rows)
|
||||
}
|
||||
}
|
||||
|
||||
// A queue id is port 25's only until the message is queued or the id is
|
||||
// opened again by another listener - Postfix reuses ids.
|
||||
func TestInboundQueueIDIsForgotten(t *testing.T) {
|
||||
const verdict = "postfix/cleanup[52]: Q1: milter-reject: END-OF-MESSAGE from app[203.0.113.4]: 4.7.1 Service unavailable - try again later; from=<a@example.net> to=<b@in.example.com> proto=ESMTP helo=<app>"
|
||||
for name, forget := range map[string]string{
|
||||
"queued": "postfix/qmgr[58]: Q1: from=<a@example.net>, size=600, nrcpt=1 (queue active)",
|
||||
"reused by the submission port": "postfix/smtpd[20]: Q1: client=app.example.com[203.0.113.4], sasl_method=PLAIN, sasl_username=app",
|
||||
} {
|
||||
j, st := inboundFixture()
|
||||
j.handle(head+"postfix/inbound/smtpd[41]: Q1: client=mx.example.net[203.0.113.9]", tailNow, false)
|
||||
j.handle(head+forget, tailNow, false)
|
||||
j.handle(head+verdict, tailNow, false)
|
||||
if len(st.rows) != 0 || len(j.port25) != 0 {
|
||||
t.Errorf("%s: rows %+v, remembered %v", name, st.rows, j.port25)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// An id that was forgotten and opened again is not dropped by its own stale
|
||||
// place in the queue of arrivals.
|
||||
func TestInboundStaleArrivalDoesNotDropALiveID(t *testing.T) {
|
||||
j, _ := inboundFixture()
|
||||
j.remember("Q1")
|
||||
delete(j.port25, "Q1") // its verdict came
|
||||
for i := 0; i < inboundQueueIDs-2; i++ {
|
||||
j.remember("F" + time.Duration(i).String())
|
||||
}
|
||||
j.remember("Q1") // reused; order is now full, with the stale Q1 first
|
||||
j.remember("LAST")
|
||||
if _, ok := j.port25["Q1"]; !ok {
|
||||
t.Error("the live Q1 was dropped by its stale arrival")
|
||||
}
|
||||
}
|
||||
|
||||
// Between the six-hourly prunes the tailer has the journal cut to its cap
|
||||
// itself, every so many rows: refusals come at the rate strangers send them.
|
||||
func TestInboundCutsTheJournalAsItWrites(t *testing.T) {
|
||||
j, st := inboundFixture()
|
||||
for i := 0; i < inboundPruneEvery*2+10; i++ {
|
||||
j.handle(unknownRcp, tailNow, false)
|
||||
}
|
||||
if len(st.pruned) != 2 || st.pruned[0] != inboundSpamLogMaxRows {
|
||||
t.Errorf("cuts = %v, want two, to %d rows", st.pruned, inboundSpamLogMaxRows)
|
||||
}
|
||||
}
|
||||
|
||||
// The queue ids waiting for a verdict are bounded.
|
||||
func TestInboundQueueIDsAreBounded(t *testing.T) {
|
||||
j, _ := inboundFixture()
|
||||
for i := 0; i < inboundQueueIDs+10; i++ {
|
||||
j.remember("Q" + strings.Repeat("A", i%7) + string(rune('0'+i%10)) + time.Duration(i).String())
|
||||
}
|
||||
if len(j.port25) != inboundQueueIDs || len(j.order) != inboundQueueIDs {
|
||||
t.Errorf("remembered %d ids (%d in order), want %d", len(j.port25), len(j.order), inboundQueueIDs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLineTime(t *testing.T) {
|
||||
now := time.Date(2026, 1, 2, 10, 0, 0, 0, time.UTC)
|
||||
for _, c := range []struct {
|
||||
line string
|
||||
want time.Time
|
||||
rest string
|
||||
}{
|
||||
{"2026-01-02T09:00:00.5+03:00 mail postfix/x: y", time.Date(2026, 1, 2, 6, 0, 0, 500000000, time.UTC), "mail postfix/x: y"},
|
||||
{"Jan 2 09:30:00 mail postfix/x: y", time.Date(2026, 1, 2, 9, 30, 0, 0, time.Local).UTC(), "mail postfix/x: y"},
|
||||
// December read in January is last year's.
|
||||
{"Dec 31 23:59:00 mail postfix/x: y", time.Date(2025, 12, 31, 23, 59, 0, 0, time.Local).UTC(), "mail postfix/x: y"},
|
||||
{"no stamp here", now, "no stamp here"},
|
||||
} {
|
||||
got, rest := lineTime(c.line, now)
|
||||
if !got.Equal(c.want) || rest != c.rest {
|
||||
t.Errorf("lineTime(%q) = %v, %q; want %v, %q", c.line, got, rest, c.want, c.rest)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestInboundRetentionPrunesAtStart(t *testing.T) {
|
||||
old := retentionInterval
|
||||
retentionInterval = time.Hour
|
||||
defer func() { retentionInterval = old }()
|
||||
|
||||
st := &fakeInboundStore{}
|
||||
ctx, cancel := contextWithCancel()
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
inboundRetentionLoop(ctx, &Inbound{Store: st, Retention: func() int { return 0 }})
|
||||
close(done)
|
||||
}()
|
||||
deadline := time.Now().Add(2 * time.Second)
|
||||
for len(st.pruned) == 0 && time.Now().Before(deadline) {
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
cancel()
|
||||
<-done
|
||||
if len(st.pruned) != 1 || st.pruned[0] != inboundSpamLogMaxRows {
|
||||
t.Errorf("prune calls = %v", st.pruned)
|
||||
}
|
||||
}
|
||||
|
||||
func contextWithCancel() (context.Context, context.CancelFunc) {
|
||||
return context.WithCancel(context.Background())
|
||||
}
|
||||
|
||||
// The lines the tailer reads when it starts may be ones it read before its
|
||||
// last stop: a refusal among them is one row, not two. Lines read as they are
|
||||
// written are always new.
|
||||
func TestInboundCatchUpDoesNotWriteARowTwice(t *testing.T) {
|
||||
j, st := inboundFixture()
|
||||
j.handle(unknownRcp, tailNow, false) // before the stop
|
||||
j.handle(unknownRcp, tailNow, true) // the same line, read again on start
|
||||
if len(st.rows) != 1 {
|
||||
t.Fatalf("rows = %d, want the one refusal", len(st.rows))
|
||||
}
|
||||
j.handle(unknownRcp, tailNow, false) // a second refusal in the same second
|
||||
if len(st.rows) != 2 {
|
||||
t.Errorf("rows = %d, want two: a line read as it is written is new", len(st.rows))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
package logtail
|
||||
|
||||
import (
|
||||
"log"
|
||||
"regexp"
|
||||
"time"
|
||||
|
||||
"github.com/mixeme/selfpost/internal/store"
|
||||
)
|
||||
|
||||
// A message's page shows what Postfix wrote about it. mail.log is rotated
|
||||
// daily and fourteen files are kept, while a send-log row lives for the
|
||||
// retention window, so the lines are kept in the database as the tailer reads
|
||||
// them (architecture.md § Log tailer).
|
||||
//
|
||||
// queueLineRe finds the queue id at the head of a line, after the timestamp
|
||||
// (already taken off by lineTime), the host name and "program[pid]:":
|
||||
//
|
||||
// mail postfix/smtp[26]: 41E862C00D9E: to=<a@example.net>, relay=…, status=sent (250 OK)
|
||||
//
|
||||
// It is matched from the start of the line and nowhere else: the tail of a
|
||||
// line is the far end's words - a reply, a HELO name - and a pattern that
|
||||
// could start anywhere would let them file a line under a queue id of their
|
||||
// choosing.
|
||||
//
|
||||
// What stands in that place is not always a queue id: "NOQUEUE:" and
|
||||
// "warning:" stand there too. A queue id is hexadecimal in capitals, or, with
|
||||
// enable_long_queue_ids, digits and letters without vowels; neither word is
|
||||
// either.
|
||||
var queueLineRe = regexp.MustCompile(`^\S+ ([^\s\[]+)\[\d+\]: ([0-9A-F]{6,24}|[0-9B-DF-HJ-NP-TV-Zb-df-hj-np-tv-z]{10,24}): (.*)`)
|
||||
|
||||
// smtpdProgramRe tells an smtpd from the other programs, whatever listener it
|
||||
// serves: "postfix/smtpd", "postfix/submission/smtpd", "postfix/inbound/smtpd".
|
||||
var smtpdProgramRe = regexp.MustCompile(`^postfix/(?:[a-z0-9-]+/)?smtpd$`)
|
||||
|
||||
// port25Program is the smtpd of the inbound relay's listener, which
|
||||
// build/postfix-config.sh names so that its lines are told from those of the
|
||||
// submission ports.
|
||||
const port25Program = "postfix/inbound/smtpd"
|
||||
|
||||
const (
|
||||
// lineGrace is how old a line must be before the prune asks whether a
|
||||
// send-log row carries its queue id. The first lines of a message are
|
||||
// logged when it arrives and its row is written at end-of-message; an hour
|
||||
// is a message body sent slowly.
|
||||
lineGrace = time.Hour
|
||||
|
||||
// lineBatch is how many lines are held before they are written without
|
||||
// waiting for the end of the pass over the log. A pass is a second's worth
|
||||
// of lines while following, and the whole backlog when the tailer starts.
|
||||
lineBatch = 500
|
||||
|
||||
// port25IDs is how many queue ids opened on port 25 are remembered so that
|
||||
// their lines are not kept. Beyond it the oldest knowledge is given up, and
|
||||
// those messages' lines are kept until the prune finds them without a row.
|
||||
port25IDs = 4096
|
||||
)
|
||||
|
||||
// lineKeeper collects the lines headed by a queue id and writes them in
|
||||
// batches. It is used from the follow loop only, so it needs no locking.
|
||||
type lineKeeper struct {
|
||||
st StatusStore
|
||||
// buf holds the lines not yet written; once says they were read while
|
||||
// catching up and may have been read before the last stop.
|
||||
buf []store.SendLogLine
|
||||
once bool
|
||||
// port25 holds the queue ids the port-25 smtpd opened: mail offered by
|
||||
// strangers, which never gets a send-log row. Its lines are not kept - what
|
||||
// a stranger can make Postfix log is not something to store on his say-so
|
||||
// and prune later. An id leaves when qmgr removes the message or when
|
||||
// another listener opens a message under it (Postfix reuses ids).
|
||||
port25 map[string]struct{}
|
||||
}
|
||||
|
||||
func newLineKeeper(st StatusStore) *lineKeeper {
|
||||
return &lineKeeper{st: st, port25: make(map[string]struct{})}
|
||||
}
|
||||
|
||||
// add takes one mail.log line and keeps it if it is headed by the queue id of
|
||||
// a message that did not come in on port 25. It does not ask whether a
|
||||
// send-log row exists for the id: for a message's first lines none does yet,
|
||||
// and the lines of messages that never get one - the bounces Postfix
|
||||
// generates, port-25 mail met in mid-flight after a restart - are removed by
|
||||
// pruneLines.
|
||||
//
|
||||
// catchingUp is as in inboundJournal.handle: the line may have been read
|
||||
// before the last stop, and is then written only if it is not there already.
|
||||
// The line is dated by its own stamp - after the panel was down, that and now
|
||||
// are hours apart, and a line is matched to its row by time.
|
||||
func (k *lineKeeper) add(line string, now time.Time, catchingUp bool) {
|
||||
at, rest := lineTime(line, now)
|
||||
m := queueLineRe.FindStringSubmatch(rest)
|
||||
if m == nil {
|
||||
return
|
||||
}
|
||||
program, queueID, text := m[1], m[2], m[3]
|
||||
|
||||
if smtpdProgramRe.MatchString(program) && len(text) >= 7 && text[:7] == "client=" {
|
||||
if program == port25Program {
|
||||
k.rememberPort25(queueID)
|
||||
} else {
|
||||
delete(k.port25, queueID)
|
||||
}
|
||||
}
|
||||
if _, inbound := k.port25[queueID]; inbound {
|
||||
if program == "postfix/qmgr" && text == "removed" {
|
||||
delete(k.port25, queueID)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
if catchingUp != k.once {
|
||||
k.flush()
|
||||
k.once = catchingUp
|
||||
}
|
||||
k.buf = append(k.buf, store.SendLogLine{QueueID: queueID, At: at, Line: line})
|
||||
if len(k.buf) >= lineBatch {
|
||||
k.flush()
|
||||
}
|
||||
}
|
||||
|
||||
func (k *lineKeeper) rememberPort25(queueID string) {
|
||||
if len(k.port25) >= port25IDs {
|
||||
// Which one goes does not matter: the cost of forgetting an id is its
|
||||
// remaining lines kept until the prune.
|
||||
for id := range k.port25 {
|
||||
delete(k.port25, id)
|
||||
break
|
||||
}
|
||||
}
|
||||
k.port25[queueID] = struct{}{}
|
||||
}
|
||||
|
||||
// flush writes the lines held. The follow loop calls it after every pass over
|
||||
// the log and before it saves its place, so the place saved is never ahead of
|
||||
// what was written. Lines that cannot be written are dropped and said so: the
|
||||
// tailer goes on, and a page without them falls back to the log file.
|
||||
func (k *lineKeeper) flush() {
|
||||
if len(k.buf) == 0 {
|
||||
return
|
||||
}
|
||||
if _, err := k.st.InsertSendLogLines(k.buf, k.once); err != nil {
|
||||
log.Printf("log-tailer: keep %d log lines: %v", len(k.buf), err)
|
||||
}
|
||||
k.buf = k.buf[:0]
|
||||
}
|
||||
|
||||
// pruneLines removes the stored lines no send-log row carries the queue id of:
|
||||
// those of rows the retention sweep deleted, and those of messages that never
|
||||
// had a row.
|
||||
func pruneLines(st StatusStore) {
|
||||
n, err := st.PruneSendLogLines(time.Now().UTC().Add(-lineGrace))
|
||||
if err != nil {
|
||||
log.Printf("log-tailer: prune stored log lines: %v", err)
|
||||
return
|
||||
}
|
||||
if n > 0 {
|
||||
log.Printf("log-tailer: pruned %d stored log lines with no send-log row", n)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,230 @@
|
||||
package logtail
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Which lines are kept for a message's page, and under which queue id: the id
|
||||
// at the head of the line and nothing the far end wrote further along it.
|
||||
func TestQueueLineHead(t *testing.T) {
|
||||
cases := []struct {
|
||||
name, rest, want string
|
||||
}{
|
||||
{"delivery", "mail postfix/smtp[26]: 41E862C00D9E: to=<a@example.net>, relay=mx.example.net[203.0.113.9]:25, dsn=2.0.0, status=sent (250 OK)", "41E862C00D9E"},
|
||||
{"submission smtpd opens the id", "mail postfix/submission/smtpd[10]: 41E862C00D9E: client=unknown[203.0.113.7], sasl_method=PLAIN, sasl_username=app", "41E862C00D9E"},
|
||||
{"port-25 smtpd", "mail postfix/inbound/smtpd[41]: 4A1B2C3D: client=mx.example.net[203.0.113.9]", "4A1B2C3D"},
|
||||
{"cleanup", "mail postfix/cleanup[52]: 41E862C00D9E: message-id=<1@example.com>", "41E862C00D9E"},
|
||||
{"qmgr removes it", "mail postfix/qmgr[10]: 41E862C00D9E: removed", "41E862C00D9E"},
|
||||
{"a bounce is sent: the id at the head, not the new message's", "mail postfix/bounce[30]: 41E862C00D9E: sender non-delivery notification: 5F00AA12", "41E862C00D9E"},
|
||||
{"long queue id", "mail postfix/qmgr[10]: 3sJ8Xh2Cgqz4Xy1: removed", "3sJ8Xh2Cgqz4Xy1"},
|
||||
{"refused before queueing", "mail postfix/inbound/smtpd[41]: NOQUEUE: reject: RCPT from mx.example.net[203.0.113.9]: 550 5.1.1 <n@in.example.com>: Recipient address rejected; from=<a@example.net> to=<n@in.example.com> proto=ESMTP helo=<mx.example.net>", ""},
|
||||
{"a warning", "mail postfix/smtpd[10]: warning: hostname mx.example.net does not resolve to address 203.0.113.9", ""},
|
||||
{"a connection", "mail postfix/smtpd[10]: connect from unknown[203.0.113.7]", ""},
|
||||
{"no program", "preexisting line, ignored on start", ""},
|
||||
{
|
||||
// The reply is the receiving server's text. One that spells the head
|
||||
// of a line is still the tail of this one: the line is the message's
|
||||
// it was written about, and no other's.
|
||||
"a reply that quotes a line head",
|
||||
"mail postfix/smtp[26]: 9F1A2C00D9E: to=<a@example.net>, relay=mx.example.net[203.0.113.9]:25, dsn=5.1.1, status=bounced (host said: 550 mail postfix/smtp[1]: 41E862C00D9E: to=<b@example.net>, status=sent)",
|
||||
"9F1A2C00D9E",
|
||||
},
|
||||
}
|
||||
for _, c := range cases {
|
||||
got := ""
|
||||
if m := queueLineRe.FindStringSubmatch(c.rest); m != nil {
|
||||
got = m[2]
|
||||
}
|
||||
if got != c.want {
|
||||
t.Errorf("%s: queue id %q, want %q", c.name, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A line is dated by its own stamp, not by when the tailer got to it, and is
|
||||
// kept as the log has it. Nothing is written until the pass over the log ends.
|
||||
func TestLineKeeperDatesByTheLine(t *testing.T) {
|
||||
cs := &captureStore{}
|
||||
k := newLineKeeper(cs)
|
||||
now := time.Date(2026, 10, 12, 18, 0, 0, 0, time.UTC)
|
||||
line := "Oct 12 09:00:00 mail postfix/qmgr[7]: 41E862C00D9E: removed"
|
||||
|
||||
k.add(line, now, false)
|
||||
k.add("Oct 12 09:00:01 mail postfix/smtpd[7]: connect from unknown[203.0.113.7]", now, false)
|
||||
if len(cs.lines) != 0 {
|
||||
t.Fatalf("written before the pass ended: %v", cs.kept())
|
||||
}
|
||||
k.flush()
|
||||
|
||||
if len(cs.lines) != 1 {
|
||||
t.Fatalf("kept %d lines, want 1: %v", len(cs.lines), cs.kept())
|
||||
}
|
||||
got := cs.lines[0]
|
||||
if got.QueueID != "41E862C00D9E" || got.Line != line {
|
||||
t.Errorf("kept %q under %q", got.Line, got.QueueID)
|
||||
}
|
||||
if want := time.Date(2026, 10, 12, 9, 0, 0, 0, time.Local).UTC(); !got.At.Equal(want) {
|
||||
t.Errorf("dated %v, want %v", got.At, want)
|
||||
}
|
||||
}
|
||||
|
||||
// A backlog is written in batches of lineBatch, not held whole, and a pass
|
||||
// that changes from catching up to following does not write the later lines
|
||||
// under the earlier rule.
|
||||
func TestLineKeeperWritesInBatches(t *testing.T) {
|
||||
cs := &captureStore{}
|
||||
k := newLineKeeper(cs)
|
||||
now := time.Date(2026, 10, 12, 18, 0, 0, 0, time.UTC)
|
||||
line := func(i int) string {
|
||||
return "host postfix/qmgr[7]: 41E862C00D9E: from=<a@example.com>, size=" + itoa(i) + ", nrcpt=1 (queue active)"
|
||||
}
|
||||
|
||||
for i := 0; i < lineBatch+10; i++ {
|
||||
k.add(line(i), now, true)
|
||||
}
|
||||
if len(cs.lines) != lineBatch || cs.batches != 1 {
|
||||
t.Fatalf("after %d lines: %d written in %d batches, want %d in 1", lineBatch+10, len(cs.lines), cs.batches, lineBatch)
|
||||
}
|
||||
|
||||
// The same line again, now as one read while following: it is a new line,
|
||||
// and the ten still held from the catch-up are written first, under theirs.
|
||||
k.add(line(0), now, false)
|
||||
k.flush()
|
||||
if got, want := len(cs.lines), lineBatch+11; got != want {
|
||||
t.Errorf("%d lines written, want %d", got, want)
|
||||
}
|
||||
if cs.batches != 3 {
|
||||
t.Errorf("%d batches, want 3", cs.batches)
|
||||
}
|
||||
}
|
||||
|
||||
// Mail that comes in on port 25 is a stranger's and never gets a send-log row:
|
||||
// none of its lines are kept, from the smtpd that opened the queue id to
|
||||
// qmgr's "removed". The id is then anybody's again.
|
||||
func TestLineKeeperLeavesOutPort25Mail(t *testing.T) {
|
||||
cs := &captureStore{}
|
||||
k := newLineKeeper(cs)
|
||||
now := time.Date(2026, 10, 12, 18, 0, 0, 0, time.UTC)
|
||||
|
||||
const (
|
||||
outClient = "host postfix/submission/smtpd[10]: 41E862C00D9E: client=app.example.com[203.0.113.7], sasl_method=PLAIN, sasl_username=app"
|
||||
outSent = "host postfix/smtp[26]: 41E862C00D9E: to=<a@example.net>, relay=mx.example.net[203.0.113.9]:25, dsn=2.0.0, status=sent (250 OK)"
|
||||
reused = "host postfix/submission/smtpd[10]: 4A1B2C3D: client=app.example.com[203.0.113.7], sasl_method=PLAIN, sasl_username=app"
|
||||
bounce = "host postfix/cleanup[52]: 5F00AA12: message-id=<20261012180000.5F00AA12@mail.example.com>"
|
||||
)
|
||||
for _, line := range []string{
|
||||
"host postfix/inbound/smtpd[41]: 4A1B2C3D: client=mx.example.net[198.51.100.9]",
|
||||
outClient,
|
||||
"host postfix/cleanup[52]: 4A1B2C3D: message-id=<whatever-the-stranger-wrote@example.net>",
|
||||
"host postfix/qmgr[10]: 4A1B2C3D: from=<a@example.net>, size=900, nrcpt=1 (queue active)",
|
||||
outSent,
|
||||
"host postfix/smtp[27]: 4A1B2C3D: to=<b@in.example.com>, relay=upstream.example.com[192.0.2.1]:25, dsn=2.0.0, status=sent (250 OK)",
|
||||
"host postfix/qmgr[10]: 4A1B2C3D: removed",
|
||||
// The id comes round again for a submitted message, and a bounce has no
|
||||
// smtpd at all: both are kept.
|
||||
reused,
|
||||
bounce,
|
||||
} {
|
||||
k.add(line, now, false)
|
||||
}
|
||||
k.flush()
|
||||
|
||||
want := []string{"41E862C00D9E|" + outClient, "41E862C00D9E|" + outSent, "4A1B2C3D|" + reused, "5F00AA12|" + bounce}
|
||||
if got := cs.kept(); strings.Join(got, "\n") != strings.Join(want, "\n") {
|
||||
t.Errorf("kept:\n%s\nwant:\n%s", strings.Join(got, "\n"), strings.Join(want, "\n"))
|
||||
}
|
||||
|
||||
// A port-25 id that a submission listener opens before qmgr's "removed" was
|
||||
// seen (the line was lost, the panel restarted) is the submitted message's.
|
||||
k.add("host postfix/inbound/smtpd[41]: 6C00BB34: client=mx.example.net[198.51.100.9]", now, false)
|
||||
again := "host postfix/smtps/smtpd[11]: 6C00BB34: client=app.example.com[203.0.113.7], sasl_method=PLAIN, sasl_username=app"
|
||||
k.add(again, now, false)
|
||||
k.flush()
|
||||
if got := cs.kept(); got[len(got)-1] != "6C00BB34|"+again {
|
||||
t.Errorf("a reused port-25 id was not kept for the submitted message: %v", got[len(got)-1])
|
||||
}
|
||||
|
||||
// What is remembered of port 25 has a ceiling.
|
||||
for i := 0; i < port25IDs+50; i++ {
|
||||
k.rememberPort25("7D" + itoa(100000+i))
|
||||
}
|
||||
if len(k.port25) > port25IDs {
|
||||
t.Errorf("%d port-25 ids remembered, ceiling is %d", len(k.port25), port25IDs)
|
||||
}
|
||||
}
|
||||
|
||||
// The tailer keeps queue-id lines as it follows the log, and after an unclean
|
||||
// stop - its saved place lies before lines it had already read - it does not
|
||||
// keep them a second time.
|
||||
func TestRunKeepsQueueLinesOnce(t *testing.T) {
|
||||
old := pollInterval
|
||||
pollInterval = 10 * time.Millisecond
|
||||
t.Cleanup(func() { pollInterval = old })
|
||||
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "mail.log")
|
||||
// Longer than a fingerprint, so the file is recognised across the restart.
|
||||
seed := strings.Repeat("host postfix/smtpd[1]: connect from unknown[203.0.113.7]\n", 20)
|
||||
if err := os.WriteFile(path, []byte(seed), 0o644); err != nil {
|
||||
t.Fatalf("seed log: %v", err)
|
||||
}
|
||||
|
||||
const (
|
||||
client = "host postfix/submission/smtpd[1]: 41E862C00D9E: client=unknown[203.0.113.7]"
|
||||
sent = "host postfix/smtp[1]: 41E862C00D9E: to=<a@example.net>, dsn=2.0.0, status=sent (ok)"
|
||||
later = "host postfix/qmgr[1]: 41E862C00D9E: removed"
|
||||
)
|
||||
|
||||
cs := &captureStore{}
|
||||
stop := startRun(t, path, cs)
|
||||
appendLine(t, path, client)
|
||||
appendLine(t, path, "host postfix/smtpd[1]: disconnect from unknown[203.0.113.7]")
|
||||
appendLine(t, path, sent)
|
||||
waitFor(t, func() bool { return len(cs.kept()) == 2 })
|
||||
stop()
|
||||
|
||||
// The process died before it saved its place: what it has is the place it
|
||||
// was at before those lines.
|
||||
cs.mu.Lock()
|
||||
cs.state.Offset = int64(len(seed))
|
||||
cs.mu.Unlock()
|
||||
appendLine(t, path, later)
|
||||
|
||||
stop = startRun(t, path, cs)
|
||||
waitFor(t, func() bool { return len(cs.kept()) >= 3 })
|
||||
stop()
|
||||
|
||||
want := []string{"41E862C00D9E|" + client, "41E862C00D9E|" + sent, "41E862C00D9E|" + later}
|
||||
if got := cs.kept(); strings.Join(got, "\n") != strings.Join(want, "\n") {
|
||||
t.Errorf("kept:\n%s\nwant:\n%s", strings.Join(got, "\n"), strings.Join(want, "\n"))
|
||||
}
|
||||
}
|
||||
|
||||
// Lines without a send-log row are removed at the start and after every
|
||||
// send-log sweep, and only once they are old enough for the row to have been
|
||||
// written.
|
||||
func TestRetentionLoopPrunesLines(t *testing.T) {
|
||||
old := retentionInterval
|
||||
retentionInterval = 20 * time.Millisecond
|
||||
t.Cleanup(func() { retentionInterval = old })
|
||||
|
||||
cs := &captureStore{}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
go retentionLoop(ctx, cs, func() int { return 90 })
|
||||
|
||||
waitFor(t, func() bool { return cs.prunes() >= 3 })
|
||||
|
||||
cs.mu.Lock()
|
||||
defer cs.mu.Unlock()
|
||||
for _, cutoff := range cs.pruned {
|
||||
if age := time.Since(cutoff); age < lineGrace || age > lineGrace+time.Minute {
|
||||
t.Errorf("prune cutoff %v ago, want %v", age, lineGrace)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -4,7 +4,8 @@
|
||||
// Postfix only decides sent / deferred / bounced later, per recipient, and
|
||||
// reports it in mail.log. This package parses those lines by queue-id +
|
||||
// recipient and advances the matching rows, and prunes rows past the retention
|
||||
// window.
|
||||
// window. It also keeps every line Postfix wrote under a queue id, so that a
|
||||
// message's page can show them after the log has rotated (lines.go).
|
||||
package logtail
|
||||
|
||||
import (
|
||||
@@ -25,12 +26,14 @@ import (
|
||||
|
||||
// StatusStore is the slice of the store the log-tailer needs: advancing
|
||||
// delivery statuses, finding the rows still waiting for one, pruning the
|
||||
// retention window, and remembering how far into mail.log it has read.
|
||||
// *store.Store satisfies it.
|
||||
// retention window, keeping the lines of queued messages, and remembering how
|
||||
// far into mail.log it has read. *store.Store satisfies it.
|
||||
type StatusStore interface {
|
||||
UpdateStatus(queueID, recipient, status string) (int64, error)
|
||||
ListQueuedOlderThan(cutoff time.Time) ([]store.QueuedDelivery, error)
|
||||
DeleteSendLogBefore(cutoff time.Time) (int64, error)
|
||||
InsertSendLogLines(lines []store.SendLogLine, once bool) (int, error)
|
||||
PruneSendLogLines(cutoff time.Time) (int64, error)
|
||||
LogtailState(path string) (store.LogtailState, bool, error)
|
||||
SaveLogtailState(path string, st store.LogtailState) error
|
||||
}
|
||||
@@ -107,9 +110,18 @@ func parseDelivery(line string) (queueID, recipient, status string, ok bool) {
|
||||
// Run follows path and updates send-log statuses until ctx is cancelled, while
|
||||
// a background sweep prunes rows older than retention(). Reading resumes at
|
||||
// the offset the previous run persisted, so a restart parses the delivery lines
|
||||
// written while the panel was down. It returns nil on a clean shutdown.
|
||||
func Run(ctx context.Context, path string, st StatusStore, retention RetentionDays) error {
|
||||
// written while the panel was down. The lines headed by a queue id are kept
|
||||
// for the message's page, except those of mail that came in on port 25
|
||||
// (lines.go). With inbound set it also writes the
|
||||
// inbound journal's rows for refused mail and prunes that journal
|
||||
// (inbound.go). It returns nil on a clean shutdown.
|
||||
func Run(ctx context.Context, path string, st StatusStore, retention RetentionDays, inbound *Inbound) error {
|
||||
go retentionLoop(ctx, st, retention)
|
||||
var journal *inboundJournal
|
||||
if inbound != nil {
|
||||
go inboundRetentionLoop(ctx, inbound)
|
||||
journal = newInboundJournal(inbound.Store)
|
||||
}
|
||||
|
||||
// The reconcile sweep must not run against a backlog the tailer has not
|
||||
// read yet: on a restart the log holds the very lines that resolve the rows
|
||||
@@ -118,7 +130,22 @@ func Run(ctx context.Context, path string, st StatusStore, retention RetentionDa
|
||||
caughtUp := make(chan struct{})
|
||||
go reconcileLoop(ctx, st, caughtUp)
|
||||
|
||||
return follow(ctx, path, &tracker{st: st, path: path}, caughtUp, func(line string) {
|
||||
keeper := newLineKeeper(st)
|
||||
return follow(ctx, path, &tracker{st: st, path: path}, caughtUp, keeper.flush, func(line string) {
|
||||
// Until the first read reaches end-of-file the lines are those written
|
||||
// while the panel was down - and, after an unclean stop, some it had
|
||||
// read already.
|
||||
catchingUp := true
|
||||
select {
|
||||
case <-caughtUp:
|
||||
catchingUp = false
|
||||
default:
|
||||
}
|
||||
now := time.Now().UTC()
|
||||
if journal != nil {
|
||||
journal.handle(line, now, catchingUp)
|
||||
}
|
||||
keeper.add(line, now, catchingUp)
|
||||
queueID, recipient, status, ok := parseDelivery(line)
|
||||
if !ok {
|
||||
return
|
||||
@@ -198,7 +225,10 @@ func reconcile(st StatusStore, cutoff time.Time) {
|
||||
}
|
||||
}
|
||||
|
||||
// retentionLoop prunes expired send-log rows immediately and then periodically.
|
||||
// retentionLoop prunes expired send-log rows immediately and then periodically,
|
||||
// and after each sweep the stored log lines no row carries the queue id of:
|
||||
// those the deleted rows leave behind, and those that never had a row
|
||||
// (lines.go).
|
||||
func retentionLoop(ctx context.Context, st StatusStore, retention RetentionDays) {
|
||||
prune := func() {
|
||||
retentionDays := retention()
|
||||
@@ -217,6 +247,7 @@ func retentionLoop(ctx context.Context, st StatusStore, retention RetentionDays)
|
||||
}
|
||||
|
||||
prune()
|
||||
pruneLines(st)
|
||||
t := time.NewTicker(retentionInterval)
|
||||
defer t.Stop()
|
||||
for {
|
||||
@@ -225,6 +256,7 @@ func retentionLoop(ctx context.Context, st StatusStore, retention RetentionDays)
|
||||
return
|
||||
case <-t.C:
|
||||
prune()
|
||||
pruneLines(st)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -426,7 +458,11 @@ func SplitTimestamp(line string) (stamp, rest string) {
|
||||
// caughtUp is closed after the first read that reaches end-of-file, which is
|
||||
// the point where every line the panel missed while it was down has been
|
||||
// handled.
|
||||
func follow(ctx context.Context, path string, tr *tracker, caughtUp chan struct{}, handle func(string)) error {
|
||||
//
|
||||
// flush is called at the end of every pass over the file, before the read
|
||||
// position is saved: a handler that gathers lines to write them together
|
||||
// writes them there, so the saved position is never ahead of what was written.
|
||||
func follow(ctx context.Context, path string, tr *tracker, caughtUp chan struct{}, flush func(), handle func(string)) error {
|
||||
var (
|
||||
f *os.File
|
||||
r *bufio.Reader
|
||||
@@ -476,6 +512,7 @@ func follow(ctx context.Context, path string, tr *tracker, caughtUp chan struct{
|
||||
}()
|
||||
|
||||
drain := func() {
|
||||
defer flush()
|
||||
for {
|
||||
line, err := r.ReadString('\n')
|
||||
if err == io.EOF {
|
||||
|
||||
@@ -147,6 +147,58 @@ type captureStore struct {
|
||||
// queued is what ListQueuedOlderThan returns, for the reconcile sweep.
|
||||
queued []store.QueuedDelivery
|
||||
queuedErr error
|
||||
|
||||
// lines are the log lines kept for messages' pages; pruned is the cutoff of
|
||||
// each PruneSendLogLines call.
|
||||
lines []store.SendLogLine
|
||||
batches int
|
||||
pruned []time.Time
|
||||
}
|
||||
|
||||
// InsertSendLogLines keeps what the store would: with once, a line is not
|
||||
// kept a second time under its queue id. batches counts the calls.
|
||||
func (c *captureStore) InsertSendLogLines(lines []store.SendLogLine, once bool) (int, error) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
c.batches++
|
||||
wrote := 0
|
||||
next:
|
||||
for _, l := range lines {
|
||||
if once {
|
||||
for _, have := range c.lines {
|
||||
if have.QueueID == l.QueueID && have.Line == l.Line {
|
||||
continue next
|
||||
}
|
||||
}
|
||||
}
|
||||
c.lines = append(c.lines, l)
|
||||
wrote++
|
||||
}
|
||||
return wrote, nil
|
||||
}
|
||||
|
||||
func (c *captureStore) PruneSendLogLines(cutoff time.Time) (int64, error) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
c.pruned = append(c.pruned, cutoff)
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
// kept returns the stored lines as "queue-id|line".
|
||||
func (c *captureStore) kept() []string {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
out := make([]string, len(c.lines))
|
||||
for i, l := range c.lines {
|
||||
out[i] = l.QueueID + "|" + l.Line
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (c *captureStore) prunes() int {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return len(c.pruned)
|
||||
}
|
||||
|
||||
func (c *captureStore) UpdateStatus(queueID, recipient, status string) (int64, error) {
|
||||
@@ -212,7 +264,7 @@ func TestFollowTailsAndRotates(t *testing.T) {
|
||||
cs := &captureStore{}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- Run(ctx, path, cs, func() int { return 90 }) }()
|
||||
go func() { done <- Run(ctx, path, cs, func() int { return 90 }, nil) }()
|
||||
|
||||
// Give follow() time to open at EOF (it seeks to end immediately on start,
|
||||
// so the seed line above is ignored), then append a delivery line.
|
||||
@@ -295,7 +347,7 @@ func startRun(t *testing.T, path string, cs *captureStore) func() {
|
||||
t.Helper()
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- Run(ctx, path, cs, func() int { return 90 }) }()
|
||||
go func() { done <- Run(ctx, path, cs, func() int { return 90 }, nil) }()
|
||||
// follow() opens and seeks on start; give it a moment before the caller
|
||||
// appends, so the append is not raced by the initial open.
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
|
||||
@@ -20,8 +20,10 @@ const (
|
||||
// which os.SameFile cannot answer across a restart.
|
||||
fingerprintSize = 512
|
||||
// persistInterval throttles the offset write. Losing up to this much
|
||||
// progress on a crash only means re-parsing a few lines (UpdateStatus is
|
||||
// idempotent), which is much cheaper than a database write per poll tick.
|
||||
// progress on a crash means re-parsing a few lines, which is much cheaper
|
||||
// than a database write per poll tick. For a delivery status that changes
|
||||
// nothing (UpdateStatus is idempotent), and a refusal among them is not
|
||||
// written to the inbound journal a second time (inboundJournal.handle).
|
||||
persistInterval = 5 * time.Second
|
||||
)
|
||||
|
||||
@@ -46,8 +48,9 @@ type tracker struct {
|
||||
// state whose fingerprint still matches means continue from it, parsing the
|
||||
// tail written while the panel was down; anything else means the file is not
|
||||
// the one the offset referred to (rotated, recreated or truncated in the
|
||||
// meantime), so read it from the start. Re-parsing lines already seen is
|
||||
// harmless: UpdateStatus writes the same status onto the same row.
|
||||
// meantime), so read it from the start. Re-parsing lines already seen writes
|
||||
// the same status onto the same send-log row, and no second inbound journal
|
||||
// row for a refusal among them.
|
||||
func (t *tracker) resume(f *os.File) int64 {
|
||||
size, err := fileSize(f)
|
||||
if err != nil {
|
||||
|
||||
@@ -30,6 +30,12 @@ func (s *retentionProbeStore) DeleteSendLogBefore(cutoff time.Time) (int64, erro
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
func (s *retentionProbeStore) InsertSendLogLines([]store.SendLogLine, bool) (int, error) {
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
func (s *retentionProbeStore) PruneSendLogLines(time.Time) (int64, error) { return 0, nil }
|
||||
|
||||
func (s *retentionProbeStore) LogtailState(string) (store.LogtailState, bool, error) {
|
||||
return store.LogtailState{}, false, nil
|
||||
}
|
||||
|
||||
@@ -8,11 +8,12 @@ import (
|
||||
)
|
||||
|
||||
// authIPAllowed reports whether the authenticated application may submit from
|
||||
// the connecting client IP. When the application has no IP restriction, or the
|
||||
// client IP is not known, the check passes. Store errors are fail-open (see
|
||||
// overLimit).
|
||||
// the connecting client IP. When the application has no IP restriction the
|
||||
// check passes. When it has one and the client's address is not known, the
|
||||
// check fails: an address list that an unknown address satisfies is no list.
|
||||
// Store errors are fail-open (see overLimit).
|
||||
func (s *session) authIPAllowed() bool {
|
||||
if s.login == "" || s.clientIP == "" {
|
||||
if s.login == "" {
|
||||
return true
|
||||
}
|
||||
a, err := s.rec.ApplicationByLogin(s.login)
|
||||
@@ -26,6 +27,10 @@ func (s *session) authIPAllowed() bool {
|
||||
if a.AllowsAuthFromIP(s.clientIP) {
|
||||
return true
|
||||
}
|
||||
if s.clientIP == "" {
|
||||
log.Printf("journal-milter: application %q refused — its client IPs are restricted and Postfix gave no client address", s.login)
|
||||
return false
|
||||
}
|
||||
log.Printf("journal-milter: application %q refused from %s — client IP not allowed", s.login, s.clientIP)
|
||||
return false
|
||||
}
|
||||
|
||||
@@ -0,0 +1,290 @@
|
||||
package milter
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/hmac"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"log"
|
||||
"net"
|
||||
"net/textproto"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"github.com/emersion/go-milter"
|
||||
|
||||
"github.com/mixeme/selfpost/internal/mailhdr"
|
||||
"github.com/mixeme/selfpost/internal/store"
|
||||
)
|
||||
|
||||
// InboundStore is what the inbound journal milter needs: the journal itself
|
||||
// and the list of inbound domains, since port 25 also takes mail that is not
|
||||
// the relay's. *store.Store satisfies it; tests substitute a fake.
|
||||
type InboundStore interface {
|
||||
InsertInboundSpam(e store.InboundSpamEntry) error
|
||||
IsInboundDomain(name string) (bool, error)
|
||||
}
|
||||
|
||||
// inboundSession is the inbound journal milter: the last milter on port 25,
|
||||
// after the optional external filter. It writes one journal row per recipient
|
||||
// of a message that reached end-of-message - that is, one the filter did not
|
||||
// refuse; refusals never get this far and are read from mail.log instead.
|
||||
//
|
||||
// Like the send-log milter it only watches: every callback continues, and
|
||||
// Postfix attaches the socket with default_action=accept, so
|
||||
// neither an error here nor the panel being down holds up inbound mail.
|
||||
type inboundSession struct {
|
||||
milter.NoOpMilter
|
||||
rec InboundStore
|
||||
// key is what the filter signs each message with (FilterSignature), or
|
||||
// empty when no filter is attached. A verdict is read only next to a
|
||||
// signature that fits: the filter is attached so that mail flows when it is
|
||||
// down, and then the verdict headers of a message are whatever its sender
|
||||
// wrote.
|
||||
key string
|
||||
|
||||
// clientIP is taken at Connect and again from {client_addr} with every
|
||||
// MAIL FROM: a session the library made after a non-continue reply never
|
||||
// sees Connect (see session.MailFrom in milter.go).
|
||||
clientIP string
|
||||
|
||||
from string
|
||||
rcpts []string
|
||||
subject string
|
||||
action string // X-Rspamd-Action
|
||||
result string // X-Spamd-Result
|
||||
seen bool // any verdict header of the filter
|
||||
sig string // X-SelfPost-Filter
|
||||
}
|
||||
|
||||
func (s *inboundSession) Connect(host, family string, port uint16, addr net.IP, m *milter.Modifier) (milter.Response, error) {
|
||||
if addr != nil {
|
||||
s.clientIP = addr.String()
|
||||
}
|
||||
return milter.RespContinue, nil
|
||||
}
|
||||
|
||||
// MailFrom starts a message: a connection may carry several.
|
||||
func (s *inboundSession) MailFrom(from string, m *milter.Modifier) (milter.Response, error) {
|
||||
s.reset()
|
||||
if ip := clientAddr(m); ip != "" {
|
||||
s.clientIP = ip
|
||||
}
|
||||
s.from = cleanAddress(from)
|
||||
return milter.RespContinue, nil
|
||||
}
|
||||
|
||||
func (s *inboundSession) RcptTo(rcpt string, m *milter.Modifier) (milter.Response, error) {
|
||||
s.rcpts = append(s.rcpts, cleanAddress(rcpt))
|
||||
return milter.RespContinue, nil
|
||||
}
|
||||
|
||||
// Header keeps the first Subject, the filter's verdict and its signature.
|
||||
// Postfix hands the message to its milters one after another at
|
||||
// end-of-message, each seeing what the one before added, which is how the
|
||||
// filter's headers arrive here. The filter replaces headers of these names, so
|
||||
// the last one read is its own - when it is up. Whether it was is decided at
|
||||
// end-of-message, by the signature.
|
||||
func (s *inboundSession) Header(name, value string, m *milter.Modifier) (milter.Response, error) {
|
||||
switch textproto.CanonicalMIMEHeaderKey(name) {
|
||||
case "Subject":
|
||||
if s.subject == "" {
|
||||
s.subject = mailhdr.DecodeSubject(value)
|
||||
}
|
||||
case "X-Rspamd-Action":
|
||||
s.action, s.seen = value, true
|
||||
case "X-Spamd-Result":
|
||||
s.result, s.seen = value, true
|
||||
case signatureHeader:
|
||||
s.sig = strings.TrimSpace(value)
|
||||
}
|
||||
return milter.RespContinue, nil
|
||||
}
|
||||
|
||||
// SignatureHeader is the header the filter signs a message in
|
||||
// (deploy/antispam/rspamd/milter_headers.conf writes it).
|
||||
const SignatureHeader = "X-SelfPost-Filter"
|
||||
|
||||
var signatureHeader = textproto.CanonicalMIMEHeaderKey(SignatureHeader)
|
||||
|
||||
// FilterSignature is what the filter writes into SignatureHeader for a message:
|
||||
// the HMAC-SHA256 of its queue id under the key SelfPost shares with the
|
||||
// filter, in hex. The key is used as the text it is stored as. A signature
|
||||
// fits one queue id, so the header going on to the upstream with the mail
|
||||
// gives nothing to whoever reads it there.
|
||||
func FilterSignature(key, queueID string) string {
|
||||
mac := hmac.New(sha256.New, []byte(key))
|
||||
mac.Write([]byte(queueID))
|
||||
return hex.EncodeToString(mac.Sum(nil))
|
||||
}
|
||||
|
||||
// signed reports whether the message carries the filter's signature for its
|
||||
// queue id. No key, no queue id or no signature is "no".
|
||||
func (s *inboundSession) signed(queueID string) bool {
|
||||
if s.key == "" || queueID == "" || s.sig == "" {
|
||||
return false
|
||||
}
|
||||
return hmac.Equal([]byte(strings.ToLower(s.sig)), []byte(FilterSignature(s.key, queueID)))
|
||||
}
|
||||
|
||||
// Body fires at end-of-message: the message is about to be queued for its
|
||||
// upstream. "Continue" lets it through as "accept" would, and keeps this
|
||||
// session for the connection's next message.
|
||||
func (s *inboundSession) Body(m *milter.Modifier) (milter.Response, error) {
|
||||
// The queue id is known only now, and with it whether the verdict headers
|
||||
// read on the way here are the filter's.
|
||||
if !s.signed(macro(m, "i")) {
|
||||
s.seen = false
|
||||
}
|
||||
s.record()
|
||||
s.reset()
|
||||
return milter.RespContinue, nil
|
||||
}
|
||||
|
||||
func (s *inboundSession) Abort(m *milter.Modifier) error {
|
||||
s.reset()
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *inboundSession) reset() {
|
||||
s.from, s.rcpts, s.subject = "", nil, ""
|
||||
s.action, s.result, s.seen, s.sig = "", "", false, ""
|
||||
}
|
||||
|
||||
// record writes one row per recipient at an inbound domain. Other recipients
|
||||
// of port 25 (a DMARC report address at the server's own name) are not the
|
||||
// relay's mail and get no row. Failures are logged, never propagated.
|
||||
func (s *inboundSession) record() {
|
||||
e := store.InboundSpamEntry{
|
||||
ClientIP: s.clientIP,
|
||||
From: s.from,
|
||||
Subject: s.subject,
|
||||
Decision: store.SpamDecisionAccept,
|
||||
Engine: store.SpamEngineSelfPost,
|
||||
}
|
||||
if s.seen {
|
||||
action, detail := rspamdVerdict(s.action, s.result)
|
||||
e.Engine, e.Detail = store.SpamEngineRspamd, detail
|
||||
if action == "quarantine" {
|
||||
e.Decision = store.SpamDecisionQuarantine
|
||||
}
|
||||
}
|
||||
for _, to := range s.rcpts {
|
||||
domain := domainOf(to)
|
||||
ok, err := s.rec.IsInboundDomain(domain)
|
||||
if err != nil {
|
||||
log.Printf("inbound-journal: look up %s: %v", domain, err)
|
||||
continue
|
||||
}
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
e.InboundDomain, e.To = domain, to
|
||||
if err := s.rec.InsertInboundSpam(e); err != nil {
|
||||
log.Printf("inbound-journal: record %s -> %s: %v", s.from, to, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// What the journal keeps of a verdict: enough to see why, on one line.
|
||||
const (
|
||||
verdictMaxSymbols = 6
|
||||
verdictMaxBytes = 255
|
||||
)
|
||||
|
||||
// ListSymbolPrefix starts the names of the symbols the filter lists' rules add
|
||||
// (deploy/antispam/rspamd/multimap.conf): SELFPOST_ALLOW_IP, _FROM, _DOMAIN.
|
||||
const ListSymbolPrefix = "SELFPOST_"
|
||||
|
||||
var (
|
||||
// "default: False [7.20 / 15.00]" - the score and the reject threshold.
|
||||
rspamdScore = regexp.MustCompile(`\[(-?[0-9]+(?:\.[0-9]+)?) / (-?[0-9]+(?:\.[0-9]+)?)\]`)
|
||||
// "BAYES_SPAM(5.10)[100.00%]" - a symbol and its weight.
|
||||
rspamdSymbol = regexp.MustCompile(`([A-Z][A-Z0-9_]{1,63})\((-?[0-9]+(?:\.[0-9]+)?)\)`)
|
||||
)
|
||||
|
||||
// rspamdVerdict reduces the filter's headers to an action and a one-line
|
||||
// summary: the action, the score against the threshold, and the first few
|
||||
// symbols that carried weight. It reads only what it recognises - the headers
|
||||
// are read only next to the filter's signature, and even so nothing of them is
|
||||
// kept verbatim except through these patterns.
|
||||
func rspamdVerdict(actionHeader, resultHeader string) (action, detail string) {
|
||||
action = strings.ToLower(strings.TrimSpace(actionHeader))
|
||||
for _, r := range action {
|
||||
if (r < 'a' || r > 'z') && r != ' ' {
|
||||
action = ""
|
||||
break
|
||||
}
|
||||
}
|
||||
if len(action) > 32 {
|
||||
action = ""
|
||||
}
|
||||
|
||||
var parts []string
|
||||
if action != "" {
|
||||
parts = append(parts, action)
|
||||
}
|
||||
if m := rspamdScore.FindStringSubmatch(resultHeader); m != nil {
|
||||
parts = append(parts, "score "+m[1]+" / "+m[2])
|
||||
}
|
||||
detail = strings.Join(parts, ", ")
|
||||
|
||||
// The filter lists' own symbols come first and are kept whatever else there
|
||||
// is: Filter lists counts how often an entry decided by finding them here,
|
||||
// and rspamd writes symbols heaviest first, which puts an allow entry's
|
||||
// -100 last, behind any six others.
|
||||
var symbols, rest []string
|
||||
for _, m := range rspamdSymbol.FindAllStringSubmatch(resultHeader, -1) {
|
||||
if strings.Trim(m[2], "-0.") == "" { // weight zero: informational
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(m[1], ListSymbolPrefix) {
|
||||
if len(symbols) < verdictMaxSymbols {
|
||||
symbols = append(symbols, m[1]+"("+m[2]+")")
|
||||
}
|
||||
continue
|
||||
}
|
||||
rest = append(rest, m[1]+"("+m[2]+")")
|
||||
}
|
||||
for _, sym := range rest {
|
||||
if len(symbols) == verdictMaxSymbols {
|
||||
break
|
||||
}
|
||||
symbols = append(symbols, sym)
|
||||
}
|
||||
if len(symbols) > 0 {
|
||||
if detail != "" {
|
||||
detail += "; "
|
||||
}
|
||||
detail += strings.Join(symbols, " ")
|
||||
}
|
||||
if len(detail) > verdictMaxBytes {
|
||||
detail = detail[:verdictMaxBytes]
|
||||
}
|
||||
return action, detail
|
||||
}
|
||||
|
||||
// ServeInbound runs the inbound journal milter on ln until ctx is cancelled.
|
||||
// key is what the external filter in front of it signs messages with, or
|
||||
// empty when none is attached (INBOUND_ANTISPAM_MILTER): no verdict is then
|
||||
// read at all. It returns nil on a clean shutdown.
|
||||
func ServeInbound(ctx context.Context, ln net.Listener, rec InboundStore, key string) error {
|
||||
srv := &milter.Server{
|
||||
NewMilter: func() milter.Milter { return &inboundSession{rec: rec, key: key} },
|
||||
Actions: 0, // read-only
|
||||
Protocol: milter.OptNoBody, // headers and end-of-message, not the body
|
||||
}
|
||||
|
||||
go func() {
|
||||
<-ctx.Done()
|
||||
_ = srv.Close()
|
||||
}()
|
||||
|
||||
if err := srv.Serve(ln); err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,350 @@
|
||||
package milter
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/emersion/go-milter"
|
||||
|
||||
"github.com/mixeme/selfpost/internal/store"
|
||||
)
|
||||
|
||||
// fakeInbound is the journal and the inbound-domain list of one test.
|
||||
type fakeInbound struct {
|
||||
domains map[string]bool
|
||||
rows []store.InboundSpamEntry
|
||||
insertErr error
|
||||
lookupErr error
|
||||
}
|
||||
|
||||
func (f *fakeInbound) InsertInboundSpam(e store.InboundSpamEntry) error {
|
||||
if f.insertErr != nil {
|
||||
return f.insertErr
|
||||
}
|
||||
f.rows = append(f.rows, e)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeInbound) IsInboundDomain(name string) (bool, error) {
|
||||
if f.lookupErr != nil {
|
||||
return false, f.lookupErr
|
||||
}
|
||||
return f.domains[name], nil
|
||||
}
|
||||
|
||||
// A header of the kind the documented rspamd sidecar adds.
|
||||
const spamdResult = "default: False [7.20 / 15.00];\n\tBAYES_SPAM(5.10)[100.00%];\n\tMIME_GOOD(-0.10)[text/plain];\n\tARC_NA(0.00)[];\n\tRCVD_COUNT_ZERO(0.00)[0];\n\tR_DKIM_NA(2.20)[]"
|
||||
|
||||
// message replays one message through s: envelope, headers in order, end.
|
||||
func message(t *testing.T, s *inboundSession, from string, rcpts []string, headers [][2]string) milter.Response {
|
||||
t.Helper()
|
||||
if _, err := s.MailFrom(from, mods(nil)); err != nil {
|
||||
t.Fatalf("MailFrom: %v", err)
|
||||
}
|
||||
for _, r := range rcpts {
|
||||
if _, err := s.RcptTo(r, mods(nil)); err != nil {
|
||||
t.Fatalf("RcptTo: %v", err)
|
||||
}
|
||||
}
|
||||
for _, h := range headers {
|
||||
if _, err := s.Header(h[0], h[1], mods(nil)); err != nil {
|
||||
t.Fatalf("Header: %v", err)
|
||||
}
|
||||
}
|
||||
resp, err := s.Body(mods(map[string]string{"i": "ABC123"}))
|
||||
if err != nil {
|
||||
t.Fatalf("Body: %v", err)
|
||||
}
|
||||
return resp
|
||||
}
|
||||
|
||||
// testKey is the key the tests' filter signs with; the messages they replay
|
||||
// have the queue id ABC123.
|
||||
const testKey = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
|
||||
|
||||
var testSig = [2]string{SignatureHeader, FilterSignature(testKey, "ABC123")}
|
||||
|
||||
func inboundSessionFor(t *testing.T, rec InboundStore, filter bool) *inboundSession {
|
||||
t.Helper()
|
||||
s := &inboundSession{rec: rec}
|
||||
if filter {
|
||||
s.key = testKey
|
||||
}
|
||||
if _, err := s.Connect("mx.example.net", "tcp4", 0, net.ParseIP("203.0.113.7"), mods(nil)); err != nil {
|
||||
t.Fatalf("Connect: %v", err)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// With no filter attached an accepted message is a plain row, one per
|
||||
// recipient, decided by nobody.
|
||||
func TestInboundRecordsAcceptedMessage(t *testing.T) {
|
||||
rec := &fakeInbound{domains: map[string]bool{"in.example.com": true}}
|
||||
s := inboundSessionFor(t, rec, false)
|
||||
resp := message(t, s, "<sender@example.net> SIZE=1200",
|
||||
[]string{"<a@in.example.com>", "b@IN.example.com"},
|
||||
[][2]string{{"Subject", "=?utf-8?Q?Sch=C3=B6n?="}})
|
||||
if resp != milter.RespContinue {
|
||||
t.Errorf("end-of-message response = %v, want continue", resp)
|
||||
}
|
||||
if len(rec.rows) != 2 {
|
||||
t.Fatalf("rows = %+v, want one per recipient", rec.rows)
|
||||
}
|
||||
want := store.InboundSpamEntry{
|
||||
InboundDomain: "in.example.com", ClientIP: "203.0.113.7",
|
||||
From: "sender@example.net", To: "a@in.example.com", Subject: "Schön",
|
||||
Decision: store.SpamDecisionAccept, Engine: store.SpamEngineSelfPost,
|
||||
}
|
||||
if rec.rows[0] != want {
|
||||
t.Errorf("row = %+v\nwant %+v", rec.rows[0], want)
|
||||
}
|
||||
if rec.rows[1].To != "b@IN.example.com" || rec.rows[1].InboundDomain != "in.example.com" {
|
||||
t.Errorf("second row = %+v", rec.rows[1])
|
||||
}
|
||||
}
|
||||
|
||||
// Port 25 also takes DMARC reports for the server's own name. They are not
|
||||
// the relay's mail and the journal says nothing about them.
|
||||
func TestInboundSkipsRecipientsOutsideInboundDomains(t *testing.T) {
|
||||
rec := &fakeInbound{domains: map[string]bool{"in.example.com": true}}
|
||||
s := inboundSessionFor(t, rec, false)
|
||||
message(t, s, "noreply-dmarc@google.com",
|
||||
[]string{"dmarc-example.com@mail.example.org", "a@in.example.com"}, nil)
|
||||
if len(rec.rows) != 1 || rec.rows[0].To != "a@in.example.com" {
|
||||
t.Errorf("rows = %+v, want the inbound recipient alone", rec.rows)
|
||||
}
|
||||
}
|
||||
|
||||
// With the filter attached its verdict headers become the engine and detail.
|
||||
func TestInboundReadsTheFiltersVerdict(t *testing.T) {
|
||||
rec := &fakeInbound{domains: map[string]bool{"in.example.com": true}}
|
||||
s := inboundSessionFor(t, rec, true)
|
||||
message(t, s, "sender@example.net", []string{"a@in.example.com"}, [][2]string{
|
||||
{"Subject", "Cheap pills"},
|
||||
{"X-Rspamd-Action", "add header"},
|
||||
{"X-Spamd-Result", spamdResult},
|
||||
testSig,
|
||||
})
|
||||
if len(rec.rows) != 1 {
|
||||
t.Fatalf("rows = %+v", rec.rows)
|
||||
}
|
||||
got := rec.rows[0]
|
||||
if got.Engine != store.SpamEngineRspamd || got.Decision != store.SpamDecisionAccept {
|
||||
t.Errorf("engine %q, decision %q", got.Engine, got.Decision)
|
||||
}
|
||||
if want := "add header, score 7.20 / 15.00; BAYES_SPAM(5.10) MIME_GOOD(-0.10) R_DKIM_NA(2.20)"; got.Detail != want {
|
||||
t.Errorf("detail = %q\nwant %q", got.Detail, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInboundQuarantineAction(t *testing.T) {
|
||||
rec := &fakeInbound{domains: map[string]bool{"in.example.com": true}}
|
||||
s := inboundSessionFor(t, rec, true)
|
||||
message(t, s, "sender@example.net", []string{"a@in.example.com"},
|
||||
[][2]string{{"x-rspamd-action", "quarantine"}, {"x-selfpost-filter", strings.ToUpper(testSig[1])}})
|
||||
if len(rec.rows) != 1 || rec.rows[0].Decision != store.SpamDecisionQuarantine || rec.rows[0].Detail != "quarantine" {
|
||||
t.Errorf("rows = %+v", rec.rows)
|
||||
}
|
||||
}
|
||||
|
||||
// Without a filter in front, a verdict header is the sender's own writing.
|
||||
func TestInboundIgnoresVerdictHeadersWithoutAFilter(t *testing.T) {
|
||||
rec := &fakeInbound{domains: map[string]bool{"in.example.com": true}}
|
||||
s := inboundSessionFor(t, rec, false)
|
||||
message(t, s, "sender@example.net", []string{"a@in.example.com"}, [][2]string{
|
||||
{"X-Rspamd-Action", "no action"},
|
||||
{"X-Spamd-Result", spamdResult},
|
||||
})
|
||||
if len(rec.rows) != 1 || rec.rows[0].Engine != store.SpamEngineSelfPost || rec.rows[0].Detail != "" {
|
||||
t.Errorf("rows = %+v", rec.rows)
|
||||
}
|
||||
}
|
||||
|
||||
// The filter is attached fail-open, so with it down the headers are again the
|
||||
// sender's: nothing of them reaches the journal except through the patterns.
|
||||
func TestRspamdVerdictKeepsOnlyWhatItRecognises(t *testing.T) {
|
||||
action, detail := rspamdVerdict("<script>alert(1)</script>", "default: False [1 / 15]; <b>HTML</b> lower_case(9.9) \x1b[31m")
|
||||
if action != "" {
|
||||
t.Errorf("action = %q, want it dropped", action)
|
||||
}
|
||||
if detail != "score 1 / 15" {
|
||||
t.Errorf("detail = %q", detail)
|
||||
}
|
||||
|
||||
var many strings.Builder
|
||||
for i := 0; i < 400; i++ {
|
||||
many.WriteString("SYMBOL_WITH_A_LONG_NAME_" + strings.Repeat("X", 30) + "(1.00)[];")
|
||||
}
|
||||
if _, detail := rspamdVerdict("reject", many.String()); len(detail) > verdictMaxBytes || strings.Count(detail, "(") > verdictMaxSymbols {
|
||||
t.Errorf("detail of %d bytes with %d symbols", len(detail), strings.Count(detail, "("))
|
||||
}
|
||||
if action, _ := rspamdVerdict(strings.Repeat("a", 33), ""); action != "" {
|
||||
t.Errorf("an action of 33 letters was kept")
|
||||
}
|
||||
}
|
||||
|
||||
// A second message on the same connection starts clean.
|
||||
func TestInboundResetsBetweenMessages(t *testing.T) {
|
||||
rec := &fakeInbound{domains: map[string]bool{"in.example.com": true}}
|
||||
s := inboundSessionFor(t, rec, true)
|
||||
message(t, s, "one@example.net", []string{"a@in.example.com"}, [][2]string{
|
||||
{"Subject", "First"}, {"X-Rspamd-Action", "add header"}, testSig,
|
||||
})
|
||||
message(t, s, "two@example.net", []string{"b@in.example.com"}, nil)
|
||||
if len(rec.rows) != 2 {
|
||||
t.Fatalf("rows = %+v", rec.rows)
|
||||
}
|
||||
want := store.InboundSpamEntry{
|
||||
InboundDomain: "in.example.com", ClientIP: "203.0.113.7",
|
||||
From: "two@example.net", To: "b@in.example.com",
|
||||
Decision: store.SpamDecisionAccept, Engine: store.SpamEngineSelfPost,
|
||||
}
|
||||
if rec.rows[1] != want {
|
||||
t.Errorf("second row = %+v\nwant %+v", rec.rows[1], want)
|
||||
}
|
||||
|
||||
// An aborted transaction leaves nothing behind for the next one either.
|
||||
if _, err := s.MailFrom("three@example.net", mods(nil)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.RcptTo("c@in.example.com", mods(nil)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.Abort(mods(nil)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(rec.rows) != 2 || len(s.rcpts) != 0 {
|
||||
t.Errorf("after an abort: rows %d, pending recipients %v", len(rec.rows), s.rcpts)
|
||||
}
|
||||
}
|
||||
|
||||
// The journal failing is never the mail's problem.
|
||||
func TestInboundAcceptsWhenTheStoreFails(t *testing.T) {
|
||||
for name, rec := range map[string]*fakeInbound{
|
||||
"insert": {domains: map[string]bool{"in.example.com": true}, insertErr: errors.New("boom")},
|
||||
"lookup": {lookupErr: errors.New("boom")},
|
||||
} {
|
||||
s := inboundSessionFor(t, rec, false)
|
||||
if resp := message(t, s, "sender@example.net", []string{"a@in.example.com"}, nil); resp != milter.RespContinue {
|
||||
t.Errorf("%s failure: response = %v, want continue", name, resp)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// rspamd writes symbols heaviest first, so an allow entry's -100 comes last.
|
||||
// Filter lists counts an entry's matches by finding that symbol in the journal:
|
||||
// it is kept whatever else there is.
|
||||
func TestRspamdVerdictKeepsTheListSymbols(t *testing.T) {
|
||||
result := "default: False [-91.00 / 15.00]; SYM_A(3.00)[]; SYM_B(2.00)[]; SYM_C(1.50)[]; SYM_D(1.00)[]; SYM_E(0.80)[]; SYM_F(0.50)[]; SYM_G(0.20)[]; ARC_NA(0.00)[]; SELFPOST_ALLOW_FROM(-100.00)[billing@vendor.example]"
|
||||
_, detail := rspamdVerdict("no action", result)
|
||||
want := "no action, score -91.00 / 15.00; SELFPOST_ALLOW_FROM(-100.00) SYM_A(3.00) SYM_B(2.00) SYM_C(1.50) SYM_D(1.00) SYM_E(0.80)"
|
||||
if detail != want {
|
||||
t.Errorf("detail = %q\nwant %q", detail, want)
|
||||
}
|
||||
}
|
||||
|
||||
// The client address of a connection's later messages comes from the MAIL
|
||||
// macro: a session the library made after a non-continue reply saw no Connect.
|
||||
func TestInboundClientAddressFromTheMailMacro(t *testing.T) {
|
||||
rec := &fakeInbound{domains: map[string]bool{"in.example.com": true}}
|
||||
s := &inboundSession{rec: rec} // no Connect
|
||||
if _, err := s.MailFrom("sender@example.net", mods(map[string]string{"{client_addr}": "IPv6:2001:db8::7"})); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.RcptTo("a@in.example.com", mods(nil)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.Body(mods(nil)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(rec.rows) != 1 || rec.rows[0].ClientIP != "2001:db8::7" {
|
||||
t.Errorf("rows = %+v", rec.rows)
|
||||
}
|
||||
}
|
||||
|
||||
// No callback of the inbound milter answers anything but "continue": it must
|
||||
// never be the reason a message is held up, and a non-continue reply would
|
||||
// cost it its session.
|
||||
func TestInboundOnlyEverContinues(t *testing.T) {
|
||||
rec := &fakeInbound{domains: map[string]bool{"in.example.com": true}}
|
||||
s := &inboundSession{rec: rec, key: testKey}
|
||||
check := func(stage string, resp milter.Response, err error) {
|
||||
t.Helper()
|
||||
if err != nil || resp != milter.RespContinue {
|
||||
t.Errorf("%s: %v, %v", stage, resp, err)
|
||||
}
|
||||
}
|
||||
resp, err := s.Connect("mx", "tcp4", 0, net.ParseIP("203.0.113.7"), mods(nil))
|
||||
check("connect", resp, err)
|
||||
resp, err = s.MailFrom("a@example.net", mods(nil))
|
||||
check("mail", resp, err)
|
||||
resp, err = s.RcptTo("a@in.example.com", mods(nil))
|
||||
check("rcpt", resp, err)
|
||||
resp, err = s.Header("X-Rspamd-Action", "reject", mods(nil))
|
||||
check("header", resp, err)
|
||||
resp, err = s.Body(mods(nil))
|
||||
check("end of message", resp, err)
|
||||
}
|
||||
|
||||
// The filter is attached so that mail flows when it is down, and then the
|
||||
// verdict headers of a message are its sender's own. A verdict is recorded only
|
||||
// next to the filter's signature for this message's queue id.
|
||||
func TestInboundReadsAVerdictOnlyWithTheFiltersSignature(t *testing.T) {
|
||||
verdict := [][2]string{{"X-Rspamd-Action", "quarantine"}, {"X-Spamd-Result", spamdResult}}
|
||||
for name, c := range map[string]struct {
|
||||
extra [][2]string
|
||||
signed bool
|
||||
}{
|
||||
"signed": {[][2]string{testSig}, true},
|
||||
"no signature": {nil, false},
|
||||
"a made-up signature": {[][2]string{{SignatureHeader, strings.Repeat("ab", 32)}}, false},
|
||||
"another message's signature": {[][2]string{{SignatureHeader, FilterSignature(testKey, "OTHER1")}}, false},
|
||||
"a signature under another key": {[][2]string{{SignatureHeader, FilterSignature(strings.Repeat("f", 64), "ABC123")}}, false},
|
||||
"the sender's first, then the real": {[][2]string{{SignatureHeader, "forged"}, testSig}, true},
|
||||
} {
|
||||
rec := &fakeInbound{domains: map[string]bool{"in.example.com": true}}
|
||||
s := inboundSessionFor(t, rec, true)
|
||||
message(t, s, "sender@example.net", []string{"a@in.example.com"}, append(append([][2]string{}, verdict...), c.extra...))
|
||||
if len(rec.rows) != 1 {
|
||||
t.Fatalf("%s: rows = %+v", name, rec.rows)
|
||||
}
|
||||
got := rec.rows[0]
|
||||
if c.signed && (got.Engine != store.SpamEngineRspamd || got.Decision != store.SpamDecisionQuarantine) {
|
||||
t.Errorf("%s: a signed verdict was not read: %+v", name, got)
|
||||
}
|
||||
if !c.signed && (got.Engine != store.SpamEngineSelfPost || got.Decision != store.SpamDecisionAccept || got.Detail != "") {
|
||||
t.Errorf("%s: an unsigned verdict was recorded: %+v", name, got)
|
||||
}
|
||||
}
|
||||
|
||||
// A signature is of no use to the next message on the connection.
|
||||
rec := &fakeInbound{domains: map[string]bool{"in.example.com": true}}
|
||||
s := inboundSessionFor(t, rec, true)
|
||||
message(t, s, "one@example.net", []string{"a@in.example.com"}, append(append([][2]string{}, verdict...), testSig))
|
||||
message(t, s, "two@example.net", []string{"a@in.example.com"}, verdict)
|
||||
if len(rec.rows) != 2 || rec.rows[1].Engine != store.SpamEngineSelfPost {
|
||||
t.Errorf("rows = %+v", rec.rows)
|
||||
}
|
||||
|
||||
// With no queue id nothing can be checked, so nothing is trusted.
|
||||
rec = &fakeInbound{domains: map[string]bool{"in.example.com": true}}
|
||||
s = inboundSessionFor(t, rec, true)
|
||||
_, _ = s.MailFrom("sender@example.net", mods(nil))
|
||||
_, _ = s.RcptTo("a@in.example.com", mods(nil))
|
||||
_, _ = s.Header("X-Rspamd-Action", "reject", mods(nil))
|
||||
_, _ = s.Header(SignatureHeader, FilterSignature(testKey, ""), mods(nil))
|
||||
_, _ = s.Body(mods(nil))
|
||||
if len(rec.rows) != 1 || rec.rows[0].Engine != store.SpamEngineSelfPost {
|
||||
t.Errorf("no queue id: rows = %+v", rec.rows)
|
||||
}
|
||||
}
|
||||
|
||||
// What rspamd 3.11 wrote for this key and queue id in a throwaway container
|
||||
// (dev trial, 2026-10-12): the two ends must compute the same thing.
|
||||
func TestFilterSignatureIsWhatTheFilterWrites(t *testing.T) {
|
||||
if got := FilterSignature(testKey, "4A1B2C3D5E"); got != "2aa1ca6dd1545d3f16c8c0c29ba59bfc28c050224ac2c5c953bd2617f73b6de2" {
|
||||
t.Errorf("FilterSignature = %s", got)
|
||||
}
|
||||
}
|
||||
@@ -53,7 +53,11 @@ type session struct {
|
||||
// valid zero value (no in-flight accounting).
|
||||
flight *inflight
|
||||
|
||||
clientIP string // captured once per connection
|
||||
// clientIP is the client's address. Postfix gives it at Connect, once per
|
||||
// connection, and again with every MAIL FROM as the {client_addr} macro
|
||||
// (milter_mail_macros in build/postfix-config.sh). The second is the one
|
||||
// that can be relied on: see MailFrom.
|
||||
clientIP string
|
||||
|
||||
login string
|
||||
from string
|
||||
@@ -62,9 +66,8 @@ type session struct {
|
||||
reserved []*reservation // level-2 slots held by the current message
|
||||
}
|
||||
|
||||
// Connect captures the client IP, which comes from the addr parameter rather
|
||||
// than a macro (the {client_addr} macro was empty in testing). It is the
|
||||
// rate-limit key; here it is recorded for completeness.
|
||||
// Connect captures the client IP of a connection's first message. It is not
|
||||
// enough on its own - see MailFrom.
|
||||
func (s *session) Connect(host, family string, port uint16, addr net.IP, m *milter.Modifier) (milter.Response, error) {
|
||||
if addr != nil {
|
||||
s.clientIP = addr.String()
|
||||
@@ -79,8 +82,21 @@ func (s *session) Connect(host, family string, port uint16, addr net.IP, m *milt
|
||||
// limit is enforced here: over the limit, the message is refused with a 4xx
|
||||
// tempfail before recipients are even offered (guide § Rate limiting).
|
||||
// Enforcement is fail-open — see overLimit.
|
||||
//
|
||||
// The client address is read again here, from the {client_addr} macro. The
|
||||
// milter library replaces the session with a new one after every reply that is
|
||||
// not "continue" - a refusal, or the accept that used to end a message - and
|
||||
// Postfix does not send Connect again on the same connection. A session made
|
||||
// that way never saw the client, so every message of a connection after the
|
||||
// first, and every MAIL FROM after a refused one, used to be judged with no
|
||||
// client address: the level-2 limit and an application's client-address list
|
||||
// both let it through. (Found on 2026-10-12; test/e2e sends two MAIL FROM on
|
||||
// one connection.)
|
||||
func (s *session) MailFrom(from string, m *milter.Modifier) (milter.Response, error) {
|
||||
s.releaseReservations() // a previous transaction that ended without EOM/ABORT
|
||||
if ip := clientAddr(m); ip != "" {
|
||||
s.clientIP = ip
|
||||
}
|
||||
s.from = cleanAddress(from)
|
||||
s.login = macro(m, "auth_authen")
|
||||
s.rcpts = nil
|
||||
@@ -113,13 +129,15 @@ func (s *session) Header(name, value string, m *milter.Modifier) (milter.Respons
|
||||
|
||||
// Body fires at end-of-message, when the queue-id macro {i} is set and the
|
||||
// message is about to be committed to the queue. This is where the "queued"
|
||||
// rows are written. We accept (this milter is done) without ever rejecting.
|
||||
// rows are written. The reply is "continue", which at end-of-message lets the
|
||||
// message through just as "accept" does - and, unlike it, leaves this session
|
||||
// in place for the connection's next message (see MailFrom).
|
||||
func (s *session) Body(m *milter.Modifier) (milter.Response, error) {
|
||||
s.record(macro(m, "i"))
|
||||
// The rows are in the send log now, so the stored count sees this message
|
||||
// and its level-2 slots are no longer needed.
|
||||
s.releaseReservations()
|
||||
return milter.RespAccept, nil
|
||||
return milter.RespContinue, nil
|
||||
}
|
||||
|
||||
// Abort ends the current transaction without an end-of-message (client RSET, or
|
||||
@@ -143,6 +161,21 @@ func macro(m *milter.Modifier, name string) string {
|
||||
return m.Macros["{"+name+"}"]
|
||||
}
|
||||
|
||||
// clientAddr is the client's address from the {client_addr} macro, or "" when
|
||||
// Postfix did not send one that reads as an address. An IPv6 address may come
|
||||
// in Sendmail's form, with "IPv6:" in front.
|
||||
func clientAddr(m *milter.Modifier) string {
|
||||
raw := strings.TrimSpace(macro(m, "client_addr"))
|
||||
if len(raw) > 5 && strings.EqualFold(raw[:5], "IPv6:") {
|
||||
raw = raw[5:]
|
||||
}
|
||||
ip := net.ParseIP(raw)
|
||||
if ip == nil {
|
||||
return ""
|
||||
}
|
||||
return ip.String()
|
||||
}
|
||||
|
||||
// record writes one send-log row per recipient. Failures are logged, never
|
||||
// propagated: journalling must not affect mail acceptance (architecture.md §
|
||||
// Mail path).
|
||||
|
||||
@@ -166,8 +166,10 @@ func TestBodyAcceptsEvenWhenRecorderFails(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("Body returned error, must fail open: %v", err)
|
||||
}
|
||||
if resp != milter.RespAccept {
|
||||
t.Fatalf("Body response = %v, want Accept", resp)
|
||||
// "Continue" at end-of-message lets the message through, and keeps the
|
||||
// session for the connection's next message.
|
||||
if resp != milter.RespContinue {
|
||||
t.Fatalf("Body response = %v, want Continue", resp)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -389,21 +391,77 @@ func TestRateLimitFailsOpenOnCountError(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRateLimitNoIPKeyDoesNotApply(t *testing.T) {
|
||||
// A ceiling is on a domain or an application, not on an address: a message that
|
||||
// arrives with no client address is counted like any other. It used to be let
|
||||
// through, which is how the second MAIL FROM of a connection got past a limit.
|
||||
func TestRateLimitAppliesWithoutAClientIP(t *testing.T) {
|
||||
rec := &fakeRecorder{
|
||||
limits: map[string]store.RateLimit{
|
||||
store.RateLimitScopeDomain + "|example.com": domainLimit(),
|
||||
},
|
||||
counts: map[string]int64{store.RateLimitScopeDomain + "|example.com": 999},
|
||||
}
|
||||
// A session with no client IP (e.g. local submission) cannot be keyed.
|
||||
s := &session{rec: rec}
|
||||
resp, err := s.MailFrom("a@example.com", mods(map[string]string{"auth_authen": "app1"}))
|
||||
if err != nil {
|
||||
t.Fatalf("MailFrom: %v", err)
|
||||
}
|
||||
if resp != milter.RespContinue {
|
||||
t.Fatalf("no-IP session = %v, want Continue", resp)
|
||||
if resp != milter.RespTempFail {
|
||||
t.Fatalf("over the limit with no client IP = %v, want TempFail", resp)
|
||||
}
|
||||
}
|
||||
|
||||
// The milter library hands the milter a fresh session after every reply that
|
||||
// is not "continue", and Postfix sends Connect once per connection. Such a
|
||||
// session knows the client only from the {client_addr} macro of MAIL FROM, and
|
||||
// that has to be enough for an application's client-address list.
|
||||
func TestAuthIPRestrictHoldsForASessionThatNeverSawConnect(t *testing.T) {
|
||||
rec := &fakeRecorder{apps: map[string]store.Application{
|
||||
"app1": {Login: "app1", AuthIPRestrict: true, AuthAllowedIPs: []string{"198.51.100.1"}},
|
||||
}}
|
||||
for name, c := range map[string]struct {
|
||||
macros map[string]string
|
||||
want milter.Response
|
||||
}{
|
||||
"an unlisted address in the macro": {map[string]string{"auth_authen": "app1", "{client_addr}": limitIP}, milter.RespTempFail},
|
||||
"the listed address in the macro": {map[string]string{"auth_authen": "app1", "{client_addr}": "198.51.100.1"}, milter.RespContinue},
|
||||
// No address at all is not a pass: a list an unknown address satisfies is no list.
|
||||
"no client address": {map[string]string{"auth_authen": "app1"}, milter.RespTempFail},
|
||||
"not an address": {map[string]string{"auth_authen": "app1", "{client_addr}": "unknown"}, milter.RespTempFail},
|
||||
} {
|
||||
s := &session{rec: rec} // no Connect
|
||||
resp, err := s.MailFrom("a@example.com", mods(c.macros))
|
||||
if err != nil || resp != c.want {
|
||||
t.Errorf("%s: %v, %v; want %v", name, resp, err, c.want)
|
||||
}
|
||||
}
|
||||
// An application with no list is not held up by a missing address.
|
||||
open := &fakeRecorder{apps: map[string]store.Application{"app1": {Login: "app1"}}}
|
||||
s := &session{rec: open}
|
||||
if resp, _ := s.MailFrom("a@example.com", mods(map[string]string{"auth_authen": "app1"})); resp != milter.RespContinue {
|
||||
t.Errorf("no list, no address: %v, want Continue", resp)
|
||||
}
|
||||
}
|
||||
|
||||
// The macro wins over what Connect said, and reads both spellings of IPv6.
|
||||
func TestClientAddrFromTheMailMacro(t *testing.T) {
|
||||
for raw, want := range map[string]string{
|
||||
"203.0.113.7": "203.0.113.7", "2001:db8::1": "2001:db8::1", "IPv6:2001:DB8::1": "2001:db8::1",
|
||||
"ipv6:::1": "::1", "": "", "unknown": "", "IPv6:": "",
|
||||
} {
|
||||
if got := clientAddr(mods(map[string]string{"{client_addr}": raw})); got != want {
|
||||
t.Errorf("clientAddr(%q) = %q, want %q", raw, got, want)
|
||||
}
|
||||
}
|
||||
s := &session{rec: &fakeRecorder{}}
|
||||
_, _ = s.Connect("h", "tcp4", 0, net.ParseIP("192.0.2.1"), mods(nil))
|
||||
_, _ = s.MailFrom("a@example.com", mods(map[string]string{"client_addr": "192.0.2.2"}))
|
||||
if s.clientIP != "192.0.2.2" {
|
||||
t.Errorf("clientIP = %q, want the macro's", s.clientIP)
|
||||
}
|
||||
_, _ = s.MailFrom("a@example.com", mods(nil))
|
||||
if s.clientIP != "192.0.2.2" {
|
||||
t.Errorf("clientIP = %q after a MAIL FROM with no macro, want it kept", s.clientIP)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -26,10 +26,9 @@ import (
|
||||
// reaches the send log (or is abandoned) — see inflight for why the stored
|
||||
// count alone is not enough.
|
||||
func (s *session) overLimit() bool {
|
||||
if s.clientIP == "" {
|
||||
return false // no client IP to key on; level-2 does not apply
|
||||
}
|
||||
|
||||
// A ceiling is on a domain or an application, whatever address the mail
|
||||
// comes from, so the client address is not asked for here. It used to be,
|
||||
// and a message that arrived without one was let through uncounted.
|
||||
if s.login != "" {
|
||||
rl, ok, err := s.rec.RateLimit(store.RateLimitScopeApp, s.login)
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"mime"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
sendmailPath = "/usr/sbin/sendmail"
|
||||
sendmailTimeout = 20 * time.Second
|
||||
)
|
||||
|
||||
// Mailer hands notifications to Postfix by local submission, as
|
||||
// postmaster@<hostname>. They are the panel's own mail: they pass no SASL
|
||||
// login, are not rows of the send log and count against no rate limit.
|
||||
// Postfix signs them through OpenDKIM with the key of the server's own name
|
||||
// (non_smtpd_milters in build/postfix-config.sh, domain.Service.SetServerName).
|
||||
type Mailer struct {
|
||||
hostname string
|
||||
// run submits a finished message; a field so tests can read it instead.
|
||||
run func(from, to string, msg []byte) error
|
||||
}
|
||||
|
||||
// NewMailer returns a mailer for the server named hostname.
|
||||
func NewMailer(hostname string) *Mailer {
|
||||
return &Mailer{hostname: hostname, run: sendmail}
|
||||
}
|
||||
|
||||
// From is the address notifications are sent from.
|
||||
func From(hostname string) string { return "postmaster@" + hostname }
|
||||
|
||||
// Send submits one message to one recipient.
|
||||
func (m *Mailer) Send(to, subject, body string) error {
|
||||
// The address is a panel user's, validated when it was saved. It goes into
|
||||
// a header and onto a command line all the same, so what cannot be part of
|
||||
// an address is refused here as well (security.md).
|
||||
if to == "" || strings.ContainsAny(to, " \t\r\n<>,;\"") || strings.HasPrefix(to, "-") || !strings.Contains(to, "@") {
|
||||
return fmt.Errorf("not an address a notification can go to: %q", to)
|
||||
}
|
||||
from := From(m.hostname)
|
||||
id, err := messageID(m.hostname)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var b bytes.Buffer
|
||||
fmt.Fprintf(&b, "From: SelfPost <%s>\r\n", from)
|
||||
fmt.Fprintf(&b, "To: <%s>\r\n", to)
|
||||
fmt.Fprintf(&b, "Subject: %s\r\n", mime.QEncoding.Encode("utf-8", oneLine(subject)))
|
||||
fmt.Fprintf(&b, "Date: %s\r\n", time.Now().UTC().Format(time.RFC1123Z))
|
||||
fmt.Fprintf(&b, "Message-ID: %s\r\n", id)
|
||||
// Tells a vacation responder and a ticket system that nobody wrote this.
|
||||
b.WriteString("Auto-Submitted: auto-generated\r\n")
|
||||
b.WriteString("MIME-Version: 1.0\r\n")
|
||||
b.WriteString("Content-Type: text/plain; charset=utf-8\r\n")
|
||||
b.WriteString("Content-Transfer-Encoding: 8bit\r\n")
|
||||
b.WriteString("\r\n")
|
||||
b.WriteString(strings.ReplaceAll(strings.ReplaceAll(body, "\r\n", "\n"), "\n", "\r\n"))
|
||||
return m.run(from, to, b.Bytes())
|
||||
}
|
||||
|
||||
// oneLine keeps a subject on its line whatever went into it.
|
||||
func oneLine(s string) string {
|
||||
return strings.Join(strings.Fields(s), " ")
|
||||
}
|
||||
|
||||
func messageID(hostname string) (string, error) {
|
||||
var raw [12]byte
|
||||
if _, err := rand.Read(raw[:]); err != nil {
|
||||
return "", fmt.Errorf("message id: %w", err)
|
||||
}
|
||||
return fmt.Sprintf("<%s.%d@%s>", hex.EncodeToString(raw[:]), time.Now().Unix(), hostname), nil
|
||||
}
|
||||
|
||||
// sendmail submits msg through Postfix's sendmail command. The argument list is
|
||||
// fixed apart from the two addresses, which follow "--" so that neither can be
|
||||
// read as an option; nothing goes through a shell (security.md). -i keeps a
|
||||
// line holding a single dot from ending the message.
|
||||
func sendmail(from, to string, msg []byte) error {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), sendmailTimeout)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, sendmailPath, "-i", "-f", from, "--", to)
|
||||
cmd.Stdin = bytes.NewReader(msg)
|
||||
out, err := cmd.CombinedOutput()
|
||||
if ctx.Err() == context.DeadlineExceeded {
|
||||
return fmt.Errorf("sendmail timed out after %s", sendmailTimeout)
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("sendmail: %w: %s", err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,416 @@
|
||||
// Package notify writes to the panel's users when something needs them
|
||||
// (architecture.md § Notifications): a sending domain's DNS record stops
|
||||
// matching, a level-2 rate limit refuses mail, the TLS certificate is about to
|
||||
// expire, a message has waited in the queue for more than three days.
|
||||
//
|
||||
// Each subject has a row of state in the database, so a thing is said once and
|
||||
// a restart says nothing again. The rules of when a message goes out are the
|
||||
// owner's (docs/plans/panel-notifications.md § When a message is sent) and
|
||||
// live in this file; what the messages say is in text.go, how they leave in
|
||||
// mailer.go.
|
||||
package notify
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/mixeme/selfpost/internal/dnscheck"
|
||||
"github.com/mixeme/selfpost/internal/health"
|
||||
"github.com/mixeme/selfpost/internal/postfix"
|
||||
"github.com/mixeme/selfpost/internal/store"
|
||||
)
|
||||
|
||||
const (
|
||||
// CheckInterval is how often DNS, the certificate and the queue are looked
|
||||
// at. A var-free constant: the rules below count in checks.
|
||||
CheckInterval = time.Hour
|
||||
// LimitInterval is how often refused mail is looked for. A refusal is
|
||||
// reported within the minute rather than on the hour: it is mail not being
|
||||
// sent now. The look is one indexed query.
|
||||
LimitInterval = time.Minute
|
||||
|
||||
// dnsChecks is how many checks in a row must agree before a DNS mismatch is
|
||||
// reported, and before it is reported gone. One check is a resolver's bad
|
||||
// minute; two, an hour apart, is a record that is wrong.
|
||||
dnsChecks = 2
|
||||
|
||||
// QueueAge is how long a message waits before the queue is reported. Postfix
|
||||
// keeps a message for five days by default, so two are left to act in.
|
||||
QueueAge = 3 * 24 * time.Hour
|
||||
|
||||
// The certificate is reported this many days before it expires, and again at
|
||||
// the second number if it is still the same certificate.
|
||||
certFirstDays = 14
|
||||
certLastDays = 3
|
||||
|
||||
// limitRepeat is the shortest time between two messages about the same
|
||||
// level-2 limit.
|
||||
limitRepeat = 24 * time.Hour
|
||||
|
||||
phasePending = "pending" // seen wrong, not yet said
|
||||
phaseOpen = "open" // said
|
||||
)
|
||||
|
||||
// Store is what the notifier needs of the database. *store.Store satisfies it.
|
||||
type Store interface {
|
||||
NotificationsEnabled() (bool, error)
|
||||
NotifyRecipients(domainName string) ([]string, error)
|
||||
GetNotifyState(key string) (store.NotifyState, bool, error)
|
||||
SetNotifyState(st store.NotifyState) error
|
||||
DeleteNotifyState(key string) error
|
||||
ListNotifyStateKeys(prefix string) ([]string, error)
|
||||
LimitRefusalsSince(since time.Time) ([]store.LimitRefusal, error)
|
||||
ListDomains() ([]store.Domain, error)
|
||||
}
|
||||
|
||||
// Sender hands one message to the mail system. *Mailer does; tests collect.
|
||||
type Sender interface {
|
||||
Send(to, subject, body string) error
|
||||
}
|
||||
|
||||
// Checks are the readings the notifier acts on, as functions so that tests
|
||||
// need neither DNS nor a certificate nor a queue.
|
||||
type Checks struct {
|
||||
// Domain is a fresh check of a sending domain's published records.
|
||||
Domain func(d store.Domain) (dnscheck.Domain, error)
|
||||
// Certificate is the state of the TLS certificate Postfix serves.
|
||||
Certificate func() health.Certificate
|
||||
// Queue lists the messages Postfix holds; an error says nothing about any.
|
||||
Queue func() ([]postfix.QueuedMessage, error)
|
||||
}
|
||||
|
||||
// Notifier applies the rules. Create one with New and call Run.
|
||||
type Notifier struct {
|
||||
st Store
|
||||
checks Checks
|
||||
sender Sender
|
||||
hostname string
|
||||
now func() time.Time
|
||||
}
|
||||
|
||||
// New builds a notifier for the server named hostname.
|
||||
func New(st Store, checks Checks, sender Sender, hostname string) *Notifier {
|
||||
return &Notifier{st: st, checks: checks, sender: sender, hostname: hostname, now: func() time.Time { return time.Now().UTC() }}
|
||||
}
|
||||
|
||||
// Run checks on both cadences until ctx is cancelled. The first hourly check
|
||||
// runs a few minutes after the start, not at it: a container that has just
|
||||
// come up is still starting Postfix and may have no network yet.
|
||||
func (n *Notifier) Run(ctx context.Context) error {
|
||||
first := time.NewTimer(3 * time.Minute)
|
||||
defer first.Stop()
|
||||
hourly := time.NewTicker(CheckInterval)
|
||||
defer hourly.Stop()
|
||||
limits := time.NewTicker(LimitInterval)
|
||||
defer limits.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return nil
|
||||
case <-first.C:
|
||||
n.Check()
|
||||
case <-hourly.C:
|
||||
n.Check()
|
||||
case <-limits.C:
|
||||
n.CheckLimits()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// enabled reports whether anything is to be done at all. With the instance
|
||||
// switch off nothing is checked and no state moves, so that what is wrong
|
||||
// when it is turned on again is said then.
|
||||
func (n *Notifier) enabled() bool {
|
||||
on, err := n.st.NotificationsEnabled()
|
||||
if err != nil {
|
||||
log.Printf("notify: read the switch: %v", err)
|
||||
return false
|
||||
}
|
||||
return on
|
||||
}
|
||||
|
||||
// Check is the hourly pass: DNS of every sending domain, the certificate, the
|
||||
// queue.
|
||||
func (n *Notifier) Check() {
|
||||
if !n.enabled() {
|
||||
return
|
||||
}
|
||||
n.checkDNS()
|
||||
n.checkCertificate()
|
||||
n.checkQueue()
|
||||
}
|
||||
|
||||
// send writes one message to every recipient of a subject, each in a message
|
||||
// of their own - users are not shown each other's addresses. It reports
|
||||
// whether the subject counts as said: nobody to tell is said, a message that
|
||||
// could not be handed over to anyone is not, and is tried at the next check.
|
||||
func (n *Notifier) send(domainName string, m message) bool {
|
||||
to, err := n.st.NotifyRecipients(domainName)
|
||||
if err != nil {
|
||||
log.Printf("notify: recipients: %v", err)
|
||||
return false
|
||||
}
|
||||
if len(to) == 0 {
|
||||
return true
|
||||
}
|
||||
sent := 0
|
||||
for _, addr := range to {
|
||||
if err := n.sender.Send(addr, m.subject, m.body); err != nil {
|
||||
log.Printf("notify: send %q to %s: %v", m.subject, addr, err)
|
||||
continue
|
||||
}
|
||||
sent++
|
||||
}
|
||||
if sent > 0 {
|
||||
log.Printf("notify: %q sent to %d of %d", m.subject, sent, len(to))
|
||||
}
|
||||
return sent > 0
|
||||
}
|
||||
|
||||
func (n *Notifier) save(st store.NotifyState) {
|
||||
if err := n.st.SetNotifyState(st); err != nil {
|
||||
log.Printf("notify: save state %s: %v", st.Key, err)
|
||||
}
|
||||
}
|
||||
|
||||
func (n *Notifier) forget(key string) {
|
||||
if err := n.st.DeleteNotifyState(key); err != nil {
|
||||
log.Printf("notify: forget state %s: %v", key, err)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- DNS
|
||||
|
||||
// dnsFault is what is wrong with a domain's records: the checks that failed,
|
||||
// in the order the page shows them.
|
||||
type dnsFault struct {
|
||||
Record string // "DKIM", "SPF", "DMARC"
|
||||
Detail string
|
||||
}
|
||||
|
||||
// dnsReading sorts a check into wrong, right, or nothing learned. A record
|
||||
// that fails is wrong whatever the others say. A lookup that did not answer is
|
||||
// nothing learned - unless another record is plainly wrong. A warning (no DMARC
|
||||
// record, an SPF include this check does not follow) is advice the page gives
|
||||
// and not a thing to write about.
|
||||
func dnsReading(d dnscheck.Domain) (faults []dnsFault, known bool) {
|
||||
known = true
|
||||
for _, r := range []struct {
|
||||
name string
|
||||
res dnscheck.Result
|
||||
}{{"DKIM", d.DKIM}, {"SPF", d.SPF}, {"DMARC", d.DMARC}} {
|
||||
switch r.res.Status {
|
||||
case health.StatusError:
|
||||
faults = append(faults, dnsFault{Record: r.name, Detail: r.res.Detail})
|
||||
case health.StatusUnknown:
|
||||
known = false
|
||||
}
|
||||
}
|
||||
if len(faults) > 0 {
|
||||
known = true
|
||||
}
|
||||
return faults, known
|
||||
}
|
||||
|
||||
func (n *Notifier) checkDNS() {
|
||||
domains, err := n.st.ListDomains()
|
||||
if err != nil {
|
||||
log.Printf("notify: list domains: %v", err)
|
||||
return
|
||||
}
|
||||
names := make(map[string]bool, len(domains))
|
||||
for _, d := range domains {
|
||||
names[d.Name] = true
|
||||
res, err := n.checks.Domain(d)
|
||||
if err != nil {
|
||||
log.Printf("notify: check %s: %v", d.Name, err)
|
||||
continue
|
||||
}
|
||||
n.dnsStep(d.Name, res)
|
||||
}
|
||||
// A deleted domain has no records to be right or wrong.
|
||||
keys, err := n.st.ListNotifyStateKeys("dns:")
|
||||
if err != nil {
|
||||
log.Printf("notify: list dns state: %v", err)
|
||||
return
|
||||
}
|
||||
for _, key := range keys {
|
||||
if !names[strings.TrimPrefix(key, "dns:")] {
|
||||
n.forget(key)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// dnsStep moves one domain's state by one check.
|
||||
func (n *Notifier) dnsStep(name string, res dnscheck.Domain) {
|
||||
faults, known := dnsReading(res)
|
||||
if !known {
|
||||
return
|
||||
}
|
||||
bad := len(faults) > 0
|
||||
key := "dns:" + name
|
||||
st, have, err := n.st.GetNotifyState(key)
|
||||
if err != nil {
|
||||
log.Printf("notify: %v", err)
|
||||
return
|
||||
}
|
||||
now := n.now()
|
||||
|
||||
switch {
|
||||
case !have && !bad:
|
||||
// Nothing wrong and nothing said.
|
||||
case !have && bad:
|
||||
n.save(store.NotifyState{Key: key, Phase: phasePending, Count: 1, Since: now})
|
||||
case st.Phase == phasePending && !bad:
|
||||
n.forget(key) // it was the resolver's bad minute
|
||||
case st.Phase == phasePending && bad:
|
||||
st.Count++
|
||||
if st.Count >= dnsChecks {
|
||||
if !n.send(name, dnsProblem(n.hostname, name, faults, st.Since)) {
|
||||
n.save(st)
|
||||
return
|
||||
}
|
||||
st.Phase, st.Count, st.SentAt = phaseOpen, 0, now
|
||||
}
|
||||
n.save(st)
|
||||
case st.Phase == phaseOpen && bad:
|
||||
if st.Count != 0 {
|
||||
st.Count = 0 // it went right for a check and wrong again: still the same trouble
|
||||
n.save(st)
|
||||
}
|
||||
case st.Phase == phaseOpen && !bad:
|
||||
st.Count++
|
||||
if st.Count >= dnsChecks {
|
||||
if n.send(name, dnsCleared(n.hostname, name, st.Since)) {
|
||||
n.forget(key)
|
||||
return
|
||||
}
|
||||
}
|
||||
n.save(st)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- certificate
|
||||
|
||||
func (n *Notifier) checkCertificate() {
|
||||
c := n.checks.Certificate()
|
||||
if c.NotAfter.IsZero() {
|
||||
return // no certificate configured, or one that cannot be read: Health says so
|
||||
}
|
||||
const key = "cert"
|
||||
mark := c.NotAfter.UTC().Format(time.RFC3339)
|
||||
st, have, err := n.st.GetNotifyState(key)
|
||||
if err != nil {
|
||||
log.Printf("notify: %v", err)
|
||||
return
|
||||
}
|
||||
if have && st.Mark != mark {
|
||||
// A different certificate: what was said about the old one is over.
|
||||
n.forget(key)
|
||||
have, st = false, store.NotifyState{}
|
||||
}
|
||||
|
||||
left := c.NotAfter.Sub(n.now())
|
||||
level := ""
|
||||
switch {
|
||||
case left <= certLastDays*24*time.Hour:
|
||||
level = "last"
|
||||
case left <= certFirstDays*24*time.Hour:
|
||||
level = "first"
|
||||
}
|
||||
if level == "" || (have && (st.Phase == level || st.Phase == "last")) {
|
||||
return
|
||||
}
|
||||
if !n.send("", certExpiring(n.hostname, c.NotAfter, left)) {
|
||||
return
|
||||
}
|
||||
n.save(store.NotifyState{Key: key, Phase: level, Mark: mark, Since: n.now(), SentAt: n.now()})
|
||||
}
|
||||
|
||||
// ---- queue
|
||||
|
||||
func (n *Notifier) checkQueue() {
|
||||
msgs, err := n.checks.Queue()
|
||||
if err != nil {
|
||||
log.Printf("notify: read the queue: %v", err)
|
||||
return // no listing is no information
|
||||
}
|
||||
now := n.now()
|
||||
var old []postfix.QueuedMessage
|
||||
for _, m := range msgs {
|
||||
if now.Sub(m.Arrival) > QueueAge {
|
||||
old = append(old, m)
|
||||
}
|
||||
}
|
||||
sort.Slice(old, func(i, j int) bool { return old[i].Arrival.Before(old[j].Arrival) })
|
||||
|
||||
const key = "queue"
|
||||
st, have, err := n.st.GetNotifyState(key)
|
||||
if err != nil {
|
||||
log.Printf("notify: %v", err)
|
||||
return
|
||||
}
|
||||
switch {
|
||||
case len(old) > 0 && !have:
|
||||
if n.send("", queueStuck(n.hostname, old, now)) {
|
||||
n.save(store.NotifyState{Key: key, Phase: phaseOpen, Since: now, SentAt: now})
|
||||
}
|
||||
case len(old) == 0 && have:
|
||||
if n.send("", queueCleared(n.hostname, st.Since)) {
|
||||
n.forget(key)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---- level-2 limits
|
||||
|
||||
// CheckLimits looks for mail refused under a level-2 rate limit and reports
|
||||
// each limit that refused, at most once in limitRepeat.
|
||||
func (n *Notifier) CheckLimits() {
|
||||
if !n.enabled() {
|
||||
return
|
||||
}
|
||||
now := n.now()
|
||||
refusals, err := n.st.LimitRefusalsSince(now.Add(-limitRepeat))
|
||||
if err != nil {
|
||||
log.Printf("notify: %v", err)
|
||||
return
|
||||
}
|
||||
for _, r := range refusals {
|
||||
if r.Domain == "" {
|
||||
continue
|
||||
}
|
||||
key := fmt.Sprintf("limit:%s:%s", r.Domain, r.AppLogin)
|
||||
st, have, err := n.st.GetNotifyState(key)
|
||||
if err != nil {
|
||||
log.Printf("notify: %v", err)
|
||||
continue
|
||||
}
|
||||
// Told within the day: the refusals counted now are the ones it was
|
||||
// told about, or more of the same. Told longer ago: everything in the
|
||||
// window is new since then.
|
||||
if have && now.Sub(st.SentAt) < limitRepeat {
|
||||
continue
|
||||
}
|
||||
if n.send(r.Domain, limitRefused(n.hostname, r)) {
|
||||
n.save(store.NotifyState{Key: key, Phase: phaseOpen, Since: r.First, SentAt: now})
|
||||
}
|
||||
}
|
||||
|
||||
// A limit nobody has hit for two days is not a subject any more.
|
||||
keys, err := n.st.ListNotifyStateKeys("limit:")
|
||||
if err != nil {
|
||||
log.Printf("notify: list limit state: %v", err)
|
||||
return
|
||||
}
|
||||
for _, key := range keys {
|
||||
st, have, err := n.st.GetNotifyState(key)
|
||||
if err == nil && have && now.Sub(st.SentAt) > 2*limitRepeat {
|
||||
n.forget(key)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,380 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mixeme/selfpost/internal/dnscheck"
|
||||
"github.com/mixeme/selfpost/internal/health"
|
||||
"github.com/mixeme/selfpost/internal/postfix"
|
||||
"github.com/mixeme/selfpost/internal/store"
|
||||
)
|
||||
|
||||
// outbox collects what would have been handed to Postfix.
|
||||
type outbox struct {
|
||||
sent []string // "to|subject"
|
||||
body []string
|
||||
fail bool
|
||||
}
|
||||
|
||||
func (o *outbox) Send(to, subject, body string) error {
|
||||
if o.fail {
|
||||
return errors.New("sendmail: queue file write error")
|
||||
}
|
||||
o.sent = append(o.sent, to+"|"+subject)
|
||||
o.body = append(o.body, body)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (o *outbox) take() []string {
|
||||
s := o.sent
|
||||
o.sent, o.body = nil, nil
|
||||
return s
|
||||
}
|
||||
|
||||
// bench is a notifier over a real database with a clock and readings the test
|
||||
// sets: one global user, one user of example.com, one user of another domain.
|
||||
type bench struct {
|
||||
n *Notifier
|
||||
st *store.Store
|
||||
out *outbox
|
||||
now time.Time
|
||||
dns map[string]dnscheck.Domain
|
||||
cert health.Certificate
|
||||
q []postfix.QueuedMessage
|
||||
qErr error
|
||||
}
|
||||
|
||||
func newBench(t *testing.T) *bench {
|
||||
t.Helper()
|
||||
st, err := store.Open(filepath.Join(t.TempDir(), "test.db"))
|
||||
if err != nil {
|
||||
t.Fatalf("open store: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = st.Close() })
|
||||
|
||||
b := &bench{st: st, out: &outbox{}, now: time.Date(2026, 10, 12, 12, 0, 0, 0, time.UTC), dns: map[string]dnscheck.Domain{}}
|
||||
mine, err := st.AddDomain("example.com", "sp")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
other, err := st.AddDomain("other.example", "sp")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, u := range []struct {
|
||||
name string
|
||||
role store.Role
|
||||
reach store.Reach
|
||||
}{
|
||||
{"root", store.RoleGlobal, store.Reach{}},
|
||||
{"mine", store.RoleDomain, store.Reach{DomainIDs: []int64{mine.ID}}},
|
||||
{"theirs", store.RoleDomain, store.Reach{DomainIDs: []int64{other.ID}}},
|
||||
} {
|
||||
id, err := st.CreateUser(u.name, "hash", u.role, u.reach)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.UpdateUser(id, u.name, "hash", u.name+"@example.org"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
b.n = New(st, Checks{
|
||||
Domain: func(d store.Domain) (dnscheck.Domain, error) {
|
||||
res, ok := b.dns[d.Name]
|
||||
if !ok {
|
||||
return good(), nil
|
||||
}
|
||||
return res, nil
|
||||
},
|
||||
Certificate: func() health.Certificate { return b.cert },
|
||||
Queue: func() ([]postfix.QueuedMessage, error) { return b.q, b.qErr },
|
||||
}, b.out, "mail.example.org")
|
||||
b.n.now = func() time.Time { return b.now }
|
||||
return b
|
||||
}
|
||||
|
||||
// hour runs the hourly check and moves the clock on by an hour.
|
||||
func (b *bench) hour() []string {
|
||||
b.n.Check()
|
||||
b.now = b.now.Add(time.Hour)
|
||||
return b.out.take()
|
||||
}
|
||||
|
||||
func good() dnscheck.Domain {
|
||||
ok := dnscheck.Result{Status: health.StatusOK}
|
||||
return dnscheck.Domain{DKIM: ok, SPF: ok, DMARC: ok}
|
||||
}
|
||||
|
||||
func badDKIM() dnscheck.Domain {
|
||||
d := good()
|
||||
d.DKIM = dnscheck.Result{Status: health.StatusError, Detail: "A TXT record exists but its public key is not the one this server signs with."}
|
||||
return d
|
||||
}
|
||||
|
||||
func want(t *testing.T, what string, got []string, wanted ...string) {
|
||||
t.Helper()
|
||||
if strings.Join(got, "\n") != strings.Join(wanted, "\n") {
|
||||
t.Errorf("%s: sent\n %s\nwant\n %s", what, strings.Join(got, "\n "), strings.Join(wanted, "\n "))
|
||||
}
|
||||
}
|
||||
|
||||
// A DNS mismatch is said once, after two checks in a row found it, to the
|
||||
// global users and the domain's own; it is said to be over once, after two
|
||||
// checks in a row found the records right.
|
||||
func TestDNSMismatchIsSaidOnceAndClearedOnce(t *testing.T) {
|
||||
b := newBench(t)
|
||||
b.dns["example.com"] = badDKIM()
|
||||
|
||||
want(t, "first bad check", b.hour())
|
||||
problem := "[SelfPost] example.com: DKIM no longer matches"
|
||||
want(t, "second bad check", b.hour(), "mine@example.org|"+problem, "root@example.org|"+problem)
|
||||
want(t, "third bad check", b.hour())
|
||||
|
||||
b.dns["example.com"] = good()
|
||||
want(t, "first good check", b.hour())
|
||||
// One good check does not end it: wrong again is the same trouble, not a new one.
|
||||
b.dns["example.com"] = badDKIM()
|
||||
want(t, "wrong again", b.hour())
|
||||
b.dns["example.com"] = good()
|
||||
want(t, "good once more", b.hour())
|
||||
cleared := "[SelfPost] example.com: DNS records match again"
|
||||
want(t, "second good check in a row", b.hour(), "mine@example.org|"+cleared, "root@example.org|"+cleared)
|
||||
want(t, "after it cleared", b.hour())
|
||||
|
||||
if _, have, _ := b.st.GetNotifyState("dns:example.com"); have {
|
||||
t.Error("state left behind after the trouble was over")
|
||||
}
|
||||
}
|
||||
|
||||
// One bad check followed by a good one was the resolver, and nothing is said.
|
||||
// A lookup that did not answer says nothing either way; a warning is advice
|
||||
// the page gives, not a thing to write about.
|
||||
func TestDNSNoiseIsNotSaid(t *testing.T) {
|
||||
b := newBench(t)
|
||||
|
||||
b.dns["example.com"] = badDKIM()
|
||||
want(t, "one bad check", b.hour())
|
||||
b.dns["example.com"] = good()
|
||||
want(t, "then good", b.hour())
|
||||
b.dns["example.com"] = badDKIM()
|
||||
want(t, "bad again, counted from one", b.hour())
|
||||
|
||||
unknown := good()
|
||||
unknown.SPF = dnscheck.Result{Status: health.StatusUnknown, Detail: "Could not look up the SPF record."}
|
||||
b.dns["example.com"] = unknown
|
||||
want(t, "resolver did not answer", b.hour())
|
||||
if st, have, _ := b.st.GetNotifyState("dns:example.com"); !have || st.Count != 1 {
|
||||
t.Errorf("an unanswered lookup moved the state: %+v, %v", st, have)
|
||||
}
|
||||
|
||||
warn := good()
|
||||
warn.DMARC = dnscheck.Result{Status: health.StatusWarn, Detail: "No DMARC record."}
|
||||
b.dns["other.example"] = warn
|
||||
want(t, "a warning", b.hour())
|
||||
want(t, "a warning, twice", b.hour())
|
||||
}
|
||||
|
||||
// The instance switch stops everything, state included; a domain that is
|
||||
// deleted takes its state with it; a message that could not be handed over is
|
||||
// tried again at the next check.
|
||||
func TestDNSSwitchDeletionAndRetry(t *testing.T) {
|
||||
b := newBench(t)
|
||||
b.dns["example.com"] = badDKIM()
|
||||
|
||||
if err := b.st.SetNotificationsEnabled(false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want(t, "switched off", b.hour())
|
||||
want(t, "switched off", b.hour())
|
||||
if _, have, _ := b.st.GetNotifyState("dns:example.com"); have {
|
||||
t.Fatal("state moved while notifications were off")
|
||||
}
|
||||
if err := b.st.SetNotificationsEnabled(true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
want(t, "on again, first check", b.hour())
|
||||
b.out.fail = true
|
||||
want(t, "second check, mail system down", b.hour())
|
||||
b.out.fail = false
|
||||
if got := b.hour(); len(got) != 2 {
|
||||
t.Errorf("after the mail system came back: %v, want the two messages", got)
|
||||
}
|
||||
|
||||
domains, _ := b.st.ListDomains()
|
||||
for _, d := range domains {
|
||||
if d.Name == "example.com" {
|
||||
if err := b.st.DeleteDomain(d.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
want(t, "domain deleted", b.hour())
|
||||
if _, have, _ := b.st.GetNotifyState("dns:example.com"); have {
|
||||
t.Error("a deleted domain kept its state")
|
||||
}
|
||||
}
|
||||
|
||||
// The certificate is said at 14 days and again at 3 if it is still the same
|
||||
// one, to global users only; a new certificate starts over.
|
||||
func TestCertificateIsSaidTwice(t *testing.T) {
|
||||
b := newBench(t)
|
||||
expiry := b.now.Add(20 * 24 * time.Hour)
|
||||
b.cert = health.Certificate{NotAfter: expiry}
|
||||
want(t, "20 days left", b.hour())
|
||||
|
||||
b.now = expiry.Add(-14 * 24 * time.Hour)
|
||||
want(t, "14 days left", b.hour(), "root@example.org|[SelfPost] TLS certificate of mail.example.org expires in 14 days")
|
||||
want(t, "an hour later", b.hour())
|
||||
b.now = expiry.Add(-8 * 24 * time.Hour)
|
||||
want(t, "8 days left", b.hour())
|
||||
|
||||
b.now = expiry.Add(-3 * 24 * time.Hour)
|
||||
want(t, "3 days left", b.hour(), "root@example.org|[SelfPost] TLS certificate of mail.example.org expires in 3 days")
|
||||
b.now = expiry.Add(time.Hour)
|
||||
want(t, "expired, already said", b.hour())
|
||||
|
||||
// Renewed: nothing to say, and the next expiry is a new subject.
|
||||
renewed := b.now.Add(90 * 24 * time.Hour)
|
||||
b.cert = health.Certificate{NotAfter: renewed}
|
||||
want(t, "renewed", b.hour())
|
||||
b.now = renewed.Add(-2 * 24 * time.Hour)
|
||||
want(t, "the new one, first seen with 2 days left", b.hour(), "root@example.org|[SelfPost] TLS certificate of mail.example.org expires in 2 days")
|
||||
|
||||
// No certificate configured: Health says so, mail does not.
|
||||
b.cert = health.Certificate{Status: health.StatusUnknown}
|
||||
want(t, "no certificate", b.hour())
|
||||
}
|
||||
|
||||
// A message older than three days is said once, and its going once; a queue
|
||||
// that cannot be listed says nothing.
|
||||
func TestQueueIsSaidOnceAndClearedOnce(t *testing.T) {
|
||||
b := newBench(t)
|
||||
b.q = []postfix.QueuedMessage{
|
||||
{ID: "FRESH001", Queue: "deferred", Sender: "app@example.com", Arrival: b.now.Add(-2 * 24 * time.Hour)},
|
||||
}
|
||||
want(t, "two days old", b.hour())
|
||||
|
||||
b.q = append(b.q, postfix.QueuedMessage{ID: "OLD00001", Queue: "deferred", Sender: "app@example.com", Arrival: b.now.Add(-4 * 24 * time.Hour)})
|
||||
b.n.Check()
|
||||
if len(b.out.sent) != 1 || b.out.sent[0] != "root@example.org|[SelfPost] A message has been in the queue of mail.example.org for more than 3 days" {
|
||||
t.Fatalf("four days old: %v", b.out.sent)
|
||||
}
|
||||
if body := b.out.body[0]; !strings.Contains(body, "OLD00001") || strings.Contains(body, "FRESH001") {
|
||||
t.Errorf("the message names the wrong mail:\n%s", body)
|
||||
}
|
||||
b.out.take()
|
||||
want(t, "still there", b.hour())
|
||||
|
||||
b.qErr = errors.New("postqueue: Permission denied")
|
||||
b.q = nil
|
||||
want(t, "queue cannot be listed", b.hour())
|
||||
b.qErr = nil
|
||||
want(t, "gone", b.hour(), "root@example.org|[SelfPost] The queue of mail.example.org holds nothing older than 3 days")
|
||||
want(t, "after", b.hour())
|
||||
}
|
||||
|
||||
// A limit that refuses mail is said to the domain's users, at most once a day.
|
||||
func TestLimitRefusalIsSaidOnceADay(t *testing.T) {
|
||||
b := newBench(t)
|
||||
b.now = time.Now().UTC() // refusals are dated by the store's clock
|
||||
refuse := func() {
|
||||
t.Helper()
|
||||
if err := b.st.InsertRejected(store.SendLogEntry{Domain: "example.com", AppLogin: "app1", From: "a@example.com"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
b.n.CheckLimits()
|
||||
want(t, "nothing refused", b.out.take())
|
||||
|
||||
refuse()
|
||||
b.n.CheckLimits()
|
||||
subject := "[SelfPost] example.com: a rate limit is refusing mail"
|
||||
want(t, "first refusal", b.out.take(), "mine@example.org|"+subject, "root@example.org|"+subject)
|
||||
|
||||
refuse()
|
||||
b.now = b.now.Add(2 * time.Hour)
|
||||
b.n.CheckLimits()
|
||||
want(t, "more of it two hours later", b.out.take())
|
||||
|
||||
// A day after it was said, mail is still being refused: said again. The
|
||||
// refusals are dated by the store's clock, so it is the telling that is
|
||||
// moved back, not the clock forward.
|
||||
st, have, err := b.st.GetNotifyState("limit:example.com:app1")
|
||||
if err != nil || !have {
|
||||
t.Fatalf("no state for the limit: %v, %v", have, err)
|
||||
}
|
||||
b.now = time.Now().UTC()
|
||||
st.SentAt = b.now.Add(-25 * time.Hour)
|
||||
if err := b.st.SetNotifyState(st); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b.n.CheckLimits()
|
||||
want(t, "a day later, still refusing", b.out.take(), "mine@example.org|"+subject, "root@example.org|"+subject)
|
||||
if body := limitRefused("mail.example.org", store.LimitRefusal{Domain: "example.com", AppLogin: "app1", Count: 3, First: b.now}).body; !strings.Contains(body, "3 messages from the application app1 of example.com") {
|
||||
t.Errorf("the message does not say who was refused how much:\n%s", body)
|
||||
}
|
||||
|
||||
// Nobody has hit the limit for two days: it is not a subject any more.
|
||||
st.SentAt = b.now.Add(-49 * time.Hour)
|
||||
if err := b.st.SetNotifyState(st); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := b.st.LimitRefusalsSince(b.now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b.now = b.now.Add(25 * time.Hour) // past the window the refusals lie in
|
||||
b.n.CheckLimits()
|
||||
want(t, "long quiet", b.out.take())
|
||||
if _, have, _ := b.st.GetNotifyState("limit:example.com:app1"); have {
|
||||
t.Error("a limit nobody hits kept its state")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMailerComposesAndRefuses(t *testing.T) {
|
||||
var from, to string
|
||||
var msg []byte
|
||||
m := NewMailer("mail.example.org")
|
||||
m.run = func(f, t string, b []byte) error { from, to, msg = f, t, b; return nil }
|
||||
|
||||
if err := m.Send("root@example.org", "[SelfPost] example.com:\r\nBcc: x@evil.example", "line one\n.\nline three\n"); err != nil {
|
||||
t.Fatalf("Send: %v", err)
|
||||
}
|
||||
if from != "postmaster@mail.example.org" || to != "root@example.org" {
|
||||
t.Errorf("envelope %s -> %s", from, to)
|
||||
}
|
||||
text := string(msg)
|
||||
head, body, ok := strings.Cut(text, "\r\n\r\n")
|
||||
if !ok {
|
||||
t.Fatalf("no header/body break:\n%s", text)
|
||||
}
|
||||
for _, h := range []string{
|
||||
"From: SelfPost <postmaster@mail.example.org>",
|
||||
"To: <root@example.org>",
|
||||
"Subject: [SelfPost] example.com: Bcc: x@evil.example",
|
||||
"Auto-Submitted: auto-generated",
|
||||
"Content-Type: text/plain; charset=utf-8",
|
||||
} {
|
||||
if !strings.Contains(head, h+"\r\n") && !strings.HasSuffix(head, h) {
|
||||
t.Errorf("header %q missing from:\n%s", h, head)
|
||||
}
|
||||
}
|
||||
if strings.Count(head, "\r\n")+1 != 9 {
|
||||
t.Errorf("a subject broke out of its line; headers:\n%s", head)
|
||||
}
|
||||
if body != "line one\r\n.\r\nline three\r\n" {
|
||||
t.Errorf("body = %q", body)
|
||||
}
|
||||
|
||||
for _, bad := range []string{"", "no-at-sign", "-oQ/tmp@example.org", "a@example.org\nBcc: b@example.org", "a b@example.org", "<a@example.org>"} {
|
||||
if err := m.Send(bad, "s", "b"); err == nil {
|
||||
t.Errorf("address %q was accepted", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,149 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/mixeme/selfpost/internal/postfix"
|
||||
"github.com/mixeme/selfpost/internal/store"
|
||||
)
|
||||
|
||||
// message is one notification: a subject line and a plain-text body.
|
||||
//
|
||||
// The texts name the page of the panel to open rather than link to it: the
|
||||
// panel does not know its own address - it sits behind the operator's reverse
|
||||
// proxy - and a link built from a guess would be a wrong link in a message
|
||||
// people are asked to trust.
|
||||
type message struct {
|
||||
subject string
|
||||
body string
|
||||
}
|
||||
|
||||
// maxListed is how many messages of the queue a notification names. The rest
|
||||
// are counted: the page shows them all.
|
||||
const maxListed = 10
|
||||
|
||||
func compose(hostname, subject string, paragraphs ...string) message {
|
||||
paragraphs = append(paragraphs,
|
||||
fmt.Sprintf("Sent by the SelfPost panel on %s. You receive this because notifications are on in your Account there; turn them off on the same page.", hostname))
|
||||
return message{
|
||||
subject: "[SelfPost] " + subject,
|
||||
body: strings.Join(paragraphs, "\n\n") + "\n",
|
||||
}
|
||||
}
|
||||
|
||||
func stamp(t time.Time) string { return t.UTC().Format("2006-01-02 15:04 UTC") }
|
||||
|
||||
// plain makes text that came from outside fit for a message body: a queued
|
||||
// message's sender is whatever its sender wrote, and a DNS finding may quote a
|
||||
// name. What is not printable ASCII becomes "?", and it is cut to max bytes -
|
||||
// a notification is read by a person, and nothing in it is theirs to act on
|
||||
// but the panel's own words.
|
||||
func plain(s string, max int) string {
|
||||
if len(s) > max {
|
||||
s = s[:max]
|
||||
}
|
||||
b := []byte(s)
|
||||
for i, c := range b {
|
||||
if c < 0x20 || c > 0x7e {
|
||||
b[i] = '?'
|
||||
}
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// days words a duration in whole days, the unit these messages count in.
|
||||
func days(d time.Duration) string {
|
||||
n := int(d.Hours() / 24)
|
||||
if n == 1 {
|
||||
return "1 day"
|
||||
}
|
||||
return fmt.Sprintf("%d days", n)
|
||||
}
|
||||
|
||||
func dnsProblem(hostname, domain string, faults []dnsFault, since time.Time) message {
|
||||
names := make([]string, len(faults))
|
||||
lines := make([]string, len(faults))
|
||||
for i, f := range faults {
|
||||
names[i] = f.Record
|
||||
lines[i] = f.Record + ": " + plain(f.Detail, 600)
|
||||
}
|
||||
return compose(hostname,
|
||||
fmt.Sprintf("%s: %s no longer matches", domain, strings.Join(names, ", ")),
|
||||
fmt.Sprintf("The DNS records of %s do not match what this server sends with. Two checks an hour apart found the same, the first at %s.", domain, stamp(since)),
|
||||
strings.Join(lines, "\n\n"),
|
||||
fmt.Sprintf("Until the records are right, receivers may refuse mail from %s or file it as spam. The records to publish are in the panel under Outbound > Domains > %s.", domain, domain),
|
||||
"One more message follows when the records match again.",
|
||||
)
|
||||
}
|
||||
|
||||
func dnsCleared(hostname, domain string, since time.Time) message {
|
||||
return compose(hostname,
|
||||
fmt.Sprintf("%s: DNS records match again", domain),
|
||||
fmt.Sprintf("The DNS records of %s match what this server sends with again; two checks an hour apart agree. They had not since %s.", domain, stamp(since)),
|
||||
)
|
||||
}
|
||||
|
||||
func certExpiring(hostname string, notAfter time.Time, left time.Duration) message {
|
||||
subject := fmt.Sprintf("TLS certificate of %s expires in %s", hostname, days(left))
|
||||
first := fmt.Sprintf("The TLS certificate this server presents on its submission ports expires on %s, in %s.", stamp(notAfter), days(left))
|
||||
if left <= 0 {
|
||||
subject = fmt.Sprintf("TLS certificate of %s has expired", hostname)
|
||||
first = fmt.Sprintf("The TLS certificate this server presents on its submission ports expired on %s.", stamp(notAfter))
|
||||
}
|
||||
return compose(hostname, subject,
|
||||
first,
|
||||
"Once it has expired, applications that verify the certificate can no longer submit mail. Renewal normally happens on the host, at the reverse proxy the certificate comes from; check that it still runs. The panel shows the certificate under Server > Health.",
|
||||
)
|
||||
}
|
||||
|
||||
func queueStuck(hostname string, old []postfix.QueuedMessage, now time.Time) message {
|
||||
var b strings.Builder
|
||||
for i, m := range old {
|
||||
if i == maxListed {
|
||||
fmt.Fprintf(&b, "... and %d more\n", len(old)-maxListed)
|
||||
break
|
||||
}
|
||||
sender := m.Sender
|
||||
if sender == "" {
|
||||
sender = "(bounce)"
|
||||
}
|
||||
fmt.Fprintf(&b, "%s from %s waiting %s, since %s\n", plain(m.ID, 24), plain(sender, 120), days(now.Sub(m.Arrival)), stamp(m.Arrival))
|
||||
}
|
||||
what := "A message has"
|
||||
if len(old) > 1 {
|
||||
what = fmt.Sprintf("%d messages have", len(old))
|
||||
}
|
||||
return compose(hostname,
|
||||
fmt.Sprintf("%s been in the queue of %s for more than %s", what, hostname, days(QueueAge)),
|
||||
fmt.Sprintf("%s waited in the mail queue for more than %s. Postfix goes on retrying and returns a message to its sender when the queue lifetime runs out.", what, days(QueueAge)),
|
||||
strings.TrimRight(b.String(), "\n"),
|
||||
"The reason the receiving server gives is in the panel: Outbound > Queue lists what is held, and the message's page in Outbound > Log shows what Postfix logged about it.",
|
||||
"One more message follows when nothing that old is left in the queue.",
|
||||
)
|
||||
}
|
||||
|
||||
func queueCleared(hostname string, since time.Time) message {
|
||||
return compose(hostname,
|
||||
fmt.Sprintf("The queue of %s holds nothing older than %s", hostname, days(QueueAge)),
|
||||
fmt.Sprintf("No message in the mail queue is older than %s any more: what was waiting since the notification of %s was delivered, or returned to its sender.", days(QueueAge), stamp(since)),
|
||||
)
|
||||
}
|
||||
|
||||
func limitRefused(hostname string, r store.LimitRefusal) message {
|
||||
who := "the domain " + r.Domain
|
||||
if r.AppLogin != "" {
|
||||
who = fmt.Sprintf("the application %s of %s", r.AppLogin, r.Domain)
|
||||
}
|
||||
count := "1 message"
|
||||
if r.Count != 1 {
|
||||
count = fmt.Sprintf("%d messages", r.Count)
|
||||
}
|
||||
return compose(hostname,
|
||||
fmt.Sprintf("%s: a rate limit is refusing mail", r.Domain),
|
||||
fmt.Sprintf("In the last 24 hours this server refused %s from %s under a level-2 rate limit, the first at %s. Refused mail is not queued: the application has to send it again.", count, who, stamp(r.First)),
|
||||
fmt.Sprintf("If the volume is expected, raise the limit in the panel: Outbound > Domains > %s, on the domain or on the application. If it is not, the application is sending more than it should.", r.Domain),
|
||||
"This limit is reported at most once a day.",
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
package postfix
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// QueuedMessage is one message Postfix is holding, as `postqueue -j` reports
|
||||
// it: where it is, and since when.
|
||||
type QueuedMessage struct {
|
||||
ID string
|
||||
Queue string // "deferred", "active", "hold", "incoming", "maildrop"
|
||||
Sender string
|
||||
Arrival time.Time
|
||||
}
|
||||
|
||||
// readQueueJSON runs `postqueue -j`. It is a var so tests can answer without a
|
||||
// running Postfix, the way readRetryConf is.
|
||||
var readQueueJSON = postqueueJSON
|
||||
|
||||
func postqueueJSON() ([]byte, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), postqueueTimeout)
|
||||
defer cancel()
|
||||
// A fixed flag and no user input, never through a shell (security.md).
|
||||
cmd := exec.CommandContext(ctx, "postqueue", "-j")
|
||||
out := &limitWriter{remaining: maxQueueOutput}
|
||||
errOut := &limitWriter{remaining: 4096}
|
||||
cmd.Stdout = out
|
||||
cmd.Stderr = errOut
|
||||
err := cmd.Run()
|
||||
if ctx.Err() == context.DeadlineExceeded {
|
||||
return nil, fmt.Errorf("postqueue -j timed out after %s", postqueueTimeout)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("postqueue -j: %w: %s", err, strings.TrimSpace(errOut.buf.String()))
|
||||
}
|
||||
// A listing cut at the size limit is still a list of whole lines up to the
|
||||
// cut; the last, broken one is dropped by the parser.
|
||||
return out.buf.Bytes(), nil
|
||||
}
|
||||
|
||||
// QueueMessages returns the messages Postfix holds with the time each arrived,
|
||||
// for the notification about mail that has waited too long. An error means the
|
||||
// queue could not be listed and says nothing about any message.
|
||||
func QueueMessages() ([]QueuedMessage, error) {
|
||||
out, err := readQueueJSON()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return parseQueueJSON(out), nil
|
||||
}
|
||||
|
||||
// parseQueueJSON reads `postqueue -j` output: one JSON object per line. A line
|
||||
// that is not one - the tail of a listing cut at its size limit - is skipped.
|
||||
func parseQueueJSON(out []byte) []QueuedMessage {
|
||||
var msgs []QueuedMessage
|
||||
sc := bufio.NewScanner(strings.NewReader(string(out)))
|
||||
sc.Buffer(make([]byte, 0, 64*1024), 1<<20)
|
||||
for sc.Scan() {
|
||||
line := strings.TrimSpace(sc.Text())
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
var e struct {
|
||||
QueueName string `json:"queue_name"`
|
||||
QueueID string `json:"queue_id"`
|
||||
ArrivalTime int64 `json:"arrival_time"`
|
||||
Sender string `json:"sender"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(line), &e); err != nil || e.QueueID == "" || e.ArrivalTime <= 0 {
|
||||
continue
|
||||
}
|
||||
msgs = append(msgs, QueuedMessage{
|
||||
ID: e.QueueID,
|
||||
Queue: e.QueueName,
|
||||
Sender: e.Sender,
|
||||
Arrival: time.Unix(e.ArrivalTime, 0).UTC(),
|
||||
})
|
||||
}
|
||||
return msgs
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
package postfix
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestParseQueueJSON(t *testing.T) {
|
||||
out := []byte(`{"queue_name": "deferred", "queue_id": "41E862C00D9E", "arrival_time": 1791000000, "message_size": 446, "forced_expire": false, "sender": "app@example.com", "recipients": [{"address": "a@example.net", "delay_reason": "connect to mx.example.net[203.0.113.9]:25: Connection timed out"}]}
|
||||
{"queue_name": "active", "queue_id": "5F00AA12", "arrival_time": 1791250000, "message_size": 900, "sender": "", "recipients": [{"address": "b@example.net"}]}
|
||||
|
||||
not json at all
|
||||
{"queue_name": "deferred", "queue_id": "", "arrival_time": 1791250000}
|
||||
{"queue_name": "deferred", "queue_id": "6C00BB34", "arrival_time": 17912`)
|
||||
|
||||
got := parseQueueJSON(out)
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("parsed %d messages, want 2: %+v", len(got), got)
|
||||
}
|
||||
if got[0].ID != "41E862C00D9E" || got[0].Queue != "deferred" || got[0].Sender != "app@example.com" ||
|
||||
!got[0].Arrival.Equal(time.Unix(1791000000, 0)) {
|
||||
t.Errorf("first message = %+v", got[0])
|
||||
}
|
||||
if got[1].ID != "5F00AA12" || got[1].Queue != "active" {
|
||||
t.Errorf("second message = %+v", got[1])
|
||||
}
|
||||
if len(parseQueueJSON(nil)) != 0 {
|
||||
t.Error("an empty queue is not empty")
|
||||
}
|
||||
}
|
||||
|
||||
// A queue that cannot be listed is an error, not an empty queue: the caller
|
||||
// must not conclude that nothing is waiting.
|
||||
func TestQueueMessagesReportsAFailedListing(t *testing.T) {
|
||||
old := readQueueJSON
|
||||
t.Cleanup(func() { readQueueJSON = old })
|
||||
readQueueJSON = func() ([]byte, error) { return nil, errors.New("postqueue: Permission denied") }
|
||||
|
||||
if msgs, err := QueueMessages(); err == nil {
|
||||
t.Fatalf("no error for a failed listing (got %v)", msgs)
|
||||
}
|
||||
}
|
||||
@@ -129,6 +129,17 @@ func (p RetryPolicy) BackoffCap() string {
|
||||
return FormatDuration(p.MaximalBackoff)
|
||||
}
|
||||
|
||||
// LongestLifetime is how long Postfix may go on working on a message: the
|
||||
// larger of the queue lifetime for ordinary mail and for mail from the null
|
||||
// sender, which an application may submit too. It bounds which stored log
|
||||
// lines can be about a send-log row (store.SendLogLines).
|
||||
func (p RetryPolicy) LongestLifetime() time.Duration {
|
||||
if p.BounceQueueLifetime > p.MaximalQueueLifetime {
|
||||
return p.BounceQueueLifetime
|
||||
}
|
||||
return p.MaximalQueueLifetime
|
||||
}
|
||||
|
||||
// QueueLifetime is the human string for maximal_queue_lifetime.
|
||||
func (p RetryPolicy) QueueLifetime() string {
|
||||
return FormatDuration(p.MaximalQueueLifetime)
|
||||
|
||||
@@ -131,14 +131,18 @@ func (s *Store) InsertDMARCReport(rep DMARCReport) (int64, error) {
|
||||
return id, nil
|
||||
}
|
||||
|
||||
// PruneDMARCReports enforces count and age caps.
|
||||
func (s *Store) PruneDMARCReports() error {
|
||||
// PruneDMARCReports enforces the age cap for every domain and the count cap
|
||||
// for domain. The count is kept per domain on purpose: reports are accepted
|
||||
// from anyone who knows a domain's report address, and with one cap for the
|
||||
// whole instance a stranger filing reports under one domain would push every
|
||||
// other domain's reports out.
|
||||
func (s *Store) PruneDMARCReports(domain string) error {
|
||||
cutoff := time.Now().UTC().AddDate(0, 0, -DMARCReportsMaxAgeDays).Format(time.RFC3339)
|
||||
if _, err := s.db.Exec(`DELETE FROM dmarc_reports WHERE received_at < ?`, cutoff); err != nil {
|
||||
return fmt.Errorf("prune dmarc by age: %w", err)
|
||||
}
|
||||
var count int
|
||||
if err := s.db.QueryRow(`SELECT COUNT(*) FROM dmarc_reports`).Scan(&count); err != nil {
|
||||
if err := s.db.QueryRow(`SELECT COUNT(*) FROM dmarc_reports WHERE domain = ?`, domain).Scan(&count); err != nil {
|
||||
return fmt.Errorf("count dmarc reports: %w", err)
|
||||
}
|
||||
if count <= DMARCReportsMaxKeep {
|
||||
@@ -147,8 +151,8 @@ func (s *Store) PruneDMARCReports() error {
|
||||
excess := count - DMARCReportsMaxKeep
|
||||
_, err := s.db.Exec(`
|
||||
DELETE FROM dmarc_reports WHERE id IN (
|
||||
SELECT id FROM dmarc_reports ORDER BY received_at ASC LIMIT ?
|
||||
)`, excess)
|
||||
SELECT id FROM dmarc_reports WHERE domain = ? ORDER BY received_at ASC LIMIT ?
|
||||
)`, domain, excess)
|
||||
if err != nil {
|
||||
return fmt.Errorf("prune dmarc by count: %w", err)
|
||||
}
|
||||
|
||||
@@ -58,7 +58,7 @@ func TestPruneDMARCReports(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := st.PruneDMARCReports(); err != nil {
|
||||
if err := st.PruneDMARCReports("example.com"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
list, _ := st.ListDMARCReports(DMARCReportScope{AllDomains: true}, 100)
|
||||
|
||||
@@ -0,0 +1,379 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Decisions of the inbound antispam journal. Kept in sync with the CHECK
|
||||
// constraint of inbound_spam_log.
|
||||
const (
|
||||
SpamDecisionAccept = "accept"
|
||||
SpamDecisionReject = "reject"
|
||||
SpamDecisionTempfail = "tempfail"
|
||||
SpamDecisionQuarantine = "quarantine"
|
||||
)
|
||||
|
||||
// Who made a decision of the inbound antispam journal. Kept in sync with the
|
||||
// CHECK constraint of inbound_spam_log.
|
||||
const (
|
||||
SpamEngineRspamd = "rspamd" // the external filter
|
||||
SpamEnginePostfix = "postfix" // Postfix's own policy: an unknown recipient, a sender domain that does not exist
|
||||
SpamEngineSelfPost = "selfpost" // nobody: relayed with no filter attached
|
||||
)
|
||||
|
||||
// InboundSpamEntry is one row of the inbound antispam journal: what happened
|
||||
// to a message offered on port 25 for one recipient. Metadata only - the
|
||||
// message itself is relayed, not stored. Subject and Detail may be empty.
|
||||
type InboundSpamEntry struct {
|
||||
InboundDomain string
|
||||
ClientIP string
|
||||
From string
|
||||
To string
|
||||
Subject string
|
||||
Decision string
|
||||
Engine string
|
||||
Detail string
|
||||
// At is when it happened; zero means now. The log tailer dates a refusal
|
||||
// by its mail.log line, which it may read long after.
|
||||
At time.Time
|
||||
}
|
||||
|
||||
// InsertInboundSpam records one journal row. It is called from the inbound
|
||||
// journal milter's hot path and from the log tailer, so it returns any error
|
||||
// for the caller to log; neither of them may hold up mail over it.
|
||||
func (s *Store) InsertInboundSpam(e InboundSpamEntry) error {
|
||||
at := e.At
|
||||
if at.IsZero() {
|
||||
at = time.Now()
|
||||
}
|
||||
_, err := s.db.Exec(
|
||||
`INSERT INTO inbound_spam_log
|
||||
(inbound_domain, client_ip, from_addr, to_addr, subject, decision, engine, detail, created_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
e.InboundDomain, e.ClientIP, e.From, e.To, e.Subject, e.Decision, e.Engine, e.Detail,
|
||||
at.UTC().Format(time.RFC3339),
|
||||
)
|
||||
if err != nil {
|
||||
return fmt.Errorf("insert inbound_spam_log: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// InsertInboundSpamOnce records a journal row unless the same row is there
|
||||
// already - same moment, client, sender, recipient, decision, engine and
|
||||
// detail - and says whether it wrote. The log tailer uses it for the lines it
|
||||
// reads when it starts: after an unclean stop those are lines it may have read
|
||||
// before, and a refusal read twice is otherwise two rows. e.At must be set; a
|
||||
// row dated "now" is never the same as another.
|
||||
func (s *Store) InsertInboundSpamOnce(e InboundSpamEntry) (bool, error) {
|
||||
at := e.At.UTC().Format(time.RFC3339)
|
||||
res, err := s.db.Exec(
|
||||
`INSERT INTO inbound_spam_log
|
||||
(inbound_domain, client_ip, from_addr, to_addr, subject, decision, engine, detail, created_at)
|
||||
SELECT ?, ?, ?, ?, ?, ?, ?, ?, ?
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM inbound_spam_log
|
||||
WHERE created_at = ? AND client_ip = ? AND from_addr = ? AND to_addr = ?
|
||||
AND decision = ? AND engine = ? AND detail = ?)`,
|
||||
e.InboundDomain, e.ClientIP, e.From, e.To, e.Subject, e.Decision, e.Engine, e.Detail, at,
|
||||
at, e.ClientIP, e.From, e.To, e.Decision, e.Engine, e.Detail,
|
||||
)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("insert inbound_spam_log once: %w", err)
|
||||
}
|
||||
n, _ := res.RowsAffected()
|
||||
return n > 0, nil
|
||||
}
|
||||
|
||||
// PruneInboundSpam deletes journal rows older than cutoff and then, if more
|
||||
// than keep rows are left, the oldest of those. It returns how many went.
|
||||
func (s *Store) PruneInboundSpam(cutoff time.Time, keep int) (int64, error) {
|
||||
res, err := s.db.Exec("DELETE FROM inbound_spam_log WHERE created_at < ?", cutoff.UTC().Format(time.RFC3339))
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("prune inbound_spam_log by age: %w", err)
|
||||
}
|
||||
n, _ := res.RowsAffected()
|
||||
// Oldest by when it happened, not by id: the tailer dates a refusal by its
|
||||
// log line, so after the panel was down the newest ids are old rows.
|
||||
res, err = s.db.Exec(
|
||||
`DELETE FROM inbound_spam_log WHERE id IN
|
||||
(SELECT id FROM inbound_spam_log ORDER BY created_at DESC, id DESC LIMIT -1 OFFSET ?)`, keep)
|
||||
if err != nil {
|
||||
return n, fmt.Errorf("prune inbound_spam_log by count: %w", err)
|
||||
}
|
||||
m, _ := res.RowsAffected()
|
||||
return n + m, nil
|
||||
}
|
||||
|
||||
// InboundSpamRow is one row of the inbound antispam journal as it is read
|
||||
// back for the panel: the entry as it was written, with its id and when it
|
||||
// happened.
|
||||
type InboundSpamRow struct {
|
||||
ID int64
|
||||
InboundDomain string
|
||||
ClientIP string
|
||||
From string
|
||||
To string
|
||||
Subject string
|
||||
Decision string
|
||||
Engine string
|
||||
Detail string
|
||||
CreatedAt time.Time
|
||||
}
|
||||
|
||||
// InboundSpamFilter narrows QueryInboundSpam and CountInboundSpam. Like
|
||||
// SendLogFilter it carries two kinds of narrowing that behave in opposite ways
|
||||
// on purpose.
|
||||
//
|
||||
// Domain, Decision, Since and Until are the operator's own filters, chosen in
|
||||
// the UI: an empty field matches everything. Since and Until are days, both
|
||||
// inclusive and read as UTC days whatever the time of day they carry.
|
||||
//
|
||||
// Domains is the authorization scope, which no query parameter may widen: the
|
||||
// exhaustive set of inbound domain names the caller may read, applied as an IN
|
||||
// constraint. nil lifts it (the global role reads the whole journal, including
|
||||
// rows of a domain deleted since); an empty set is a caller who reaches
|
||||
// nothing and gets nothing. Unlike SendLogFilter, forgetting the scope is
|
||||
// therefore the permissive case - the caller states "all" by leaving it out -
|
||||
// so every caller that serves someone other than the global role must set it.
|
||||
type InboundSpamFilter struct {
|
||||
Domain string
|
||||
Decision string
|
||||
Since time.Time
|
||||
Until time.Time
|
||||
Domains []string
|
||||
}
|
||||
|
||||
// QueryInboundSpam returns journal rows matching filter, newest first, for
|
||||
// the Inbound log's server-side pagination. A refusal is dated by the mail.log
|
||||
// line the tailer read it from, which can be older than rows written since, so
|
||||
// the order is by time and not by id.
|
||||
func (s *Store) QueryInboundSpam(filter InboundSpamFilter, limit, offset int) ([]InboundSpamRow, error) {
|
||||
where, args := inboundSpamWhere(filter)
|
||||
args = append(args, limit, offset)
|
||||
rows, err := s.db.Query(
|
||||
`SELECT id, inbound_domain, client_ip, from_addr, to_addr, subject, decision, engine, detail, created_at
|
||||
FROM inbound_spam_log`+where+`
|
||||
ORDER BY created_at DESC, id DESC
|
||||
LIMIT ? OFFSET ?`,
|
||||
args...,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("query inbound_spam_log: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var out []InboundSpamRow
|
||||
for rows.Next() {
|
||||
var (
|
||||
row InboundSpamRow
|
||||
created string
|
||||
)
|
||||
if err := rows.Scan(&row.ID, &row.InboundDomain, &row.ClientIP, &row.From, &row.To,
|
||||
&row.Subject, &row.Decision, &row.Engine, &row.Detail, &created); err != nil {
|
||||
return nil, fmt.Errorf("scan inbound_spam_log row: %w", err)
|
||||
}
|
||||
row.CreatedAt, _ = time.Parse(time.RFC3339, created)
|
||||
out = append(out, row)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// CountInboundSpam returns how many journal rows match filter, for the page
|
||||
// numbers under the Inbound log.
|
||||
func (s *Store) CountInboundSpam(filter InboundSpamFilter) (int64, error) {
|
||||
where, args := inboundSpamWhere(filter)
|
||||
var n int64
|
||||
if err := s.db.QueryRow(`SELECT COUNT(*) FROM inbound_spam_log`+where, args...).Scan(&n); err != nil {
|
||||
return 0, fmt.Errorf("count inbound_spam_log: %w", err)
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
|
||||
func inboundSpamWhere(f InboundSpamFilter) (string, []any) {
|
||||
var clauses []string
|
||||
var args []any
|
||||
if f.Domains != nil {
|
||||
if len(f.Domains) == 0 {
|
||||
return " WHERE 1 = 0", nil
|
||||
}
|
||||
marks := make([]string, len(f.Domains))
|
||||
for i, name := range f.Domains {
|
||||
marks[i] = "?"
|
||||
args = append(args, name)
|
||||
}
|
||||
clauses = append(clauses, "inbound_domain IN ("+strings.Join(marks, ", ")+")")
|
||||
}
|
||||
if f.Domain != "" {
|
||||
clauses = append(clauses, "inbound_domain = ?")
|
||||
args = append(args, f.Domain)
|
||||
}
|
||||
if f.Decision != "" {
|
||||
clauses = append(clauses, "decision = ?")
|
||||
args = append(args, f.Decision)
|
||||
}
|
||||
// created_at is RFC3339 UTC, so comparing it as text against the same
|
||||
// format is comparing moments: from midnight of the first day up to, not
|
||||
// including, midnight after the last.
|
||||
if !f.Since.IsZero() {
|
||||
y, m, d := f.Since.UTC().Date()
|
||||
clauses = append(clauses, "created_at >= ?")
|
||||
args = append(args, time.Date(y, m, d, 0, 0, 0, 0, time.UTC).Format(time.RFC3339))
|
||||
}
|
||||
if !f.Until.IsZero() {
|
||||
y, m, d := f.Until.UTC().Date()
|
||||
clauses = append(clauses, "created_at < ?")
|
||||
args = append(args, time.Date(y, m, d+1, 0, 0, 0, 0, time.UTC).Format(time.RFC3339))
|
||||
}
|
||||
if len(clauses) == 0 {
|
||||
return "", nil
|
||||
}
|
||||
return " WHERE " + strings.Join(clauses, " AND "), args
|
||||
}
|
||||
|
||||
// IsInboundDomain reports whether name is a configured inbound domain. Port 25
|
||||
// also takes mail that is not the relay's - DMARC reports for the server's own
|
||||
// name - and the journal keeps rows for inbound domains only.
|
||||
func (s *Store) IsInboundDomain(name string) (bool, error) {
|
||||
var one int
|
||||
err := s.db.QueryRow("SELECT 1 FROM inbound_domains WHERE name = ?", strings.ToLower(name)).Scan(&one)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("look up inbound domain: %w", err)
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// Lists and entry kinds of the filter lists. Kept in sync with the CHECK
|
||||
// constraints of inbound_spam_list.
|
||||
const (
|
||||
SpamListAllow = "allow"
|
||||
SpamListDeny = "deny"
|
||||
|
||||
SpamEntryAddress = "address" // sender address
|
||||
SpamEntryDomain = "domain" // sender domain
|
||||
SpamEntryIP = "ip" // client address or network
|
||||
)
|
||||
|
||||
// ErrSpamListEntryExists is returned when the entry is already on that list.
|
||||
var ErrSpamListEntryExists = errors.New("filter list entry already exists")
|
||||
|
||||
// ErrSpamListEntryNotFound is returned when a filter list entry id does not exist.
|
||||
var ErrSpamListEntryNotFound = errors.New("filter list entry not found")
|
||||
|
||||
// SpamListEntry is one allow or deny entry for the external filter. Value is
|
||||
// stored normalised by the caller (internal/inbound), which is what makes the
|
||||
// uniqueness of an entry on its list mean anything.
|
||||
type SpamListEntry struct {
|
||||
ID int64
|
||||
List string
|
||||
Kind string
|
||||
Value string
|
||||
Note string
|
||||
CreatedAt time.Time
|
||||
}
|
||||
|
||||
// AddSpamListEntry inserts an entry. The caller must have validated it.
|
||||
func (s *Store) AddSpamListEntry(list, kind, value, note string) (SpamListEntry, error) {
|
||||
now := time.Now().UTC()
|
||||
res, err := s.db.Exec(
|
||||
"INSERT INTO inbound_spam_list (list_type, entry_type, value, note, created_at) VALUES (?, ?, ?, ?, ?)",
|
||||
list, kind, value, note, now.Format(time.RFC3339),
|
||||
)
|
||||
if err != nil {
|
||||
if isUniqueViolation(err) {
|
||||
return SpamListEntry{}, ErrSpamListEntryExists
|
||||
}
|
||||
return SpamListEntry{}, fmt.Errorf("insert inbound_spam_list: %w", err)
|
||||
}
|
||||
id, err := res.LastInsertId()
|
||||
if err != nil {
|
||||
return SpamListEntry{}, fmt.Errorf("filter list entry id: %w", err)
|
||||
}
|
||||
return SpamListEntry{ID: id, List: list, Kind: kind, Value: value, Note: note, CreatedAt: now.Truncate(time.Second)}, nil
|
||||
}
|
||||
|
||||
// ListSpamListEntries returns every entry of both lists: by list, then kind,
|
||||
// then value.
|
||||
func (s *Store) ListSpamListEntries() ([]SpamListEntry, error) {
|
||||
rows, err := s.db.Query(
|
||||
"SELECT id, list_type, entry_type, value, note, created_at FROM inbound_spam_list ORDER BY list_type, entry_type, value")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("list inbound_spam_list: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []SpamListEntry
|
||||
for rows.Next() {
|
||||
var e SpamListEntry
|
||||
var created string
|
||||
if err := rows.Scan(&e.ID, &e.List, &e.Kind, &e.Value, &e.Note, &created); err != nil {
|
||||
return nil, fmt.Errorf("scan inbound_spam_list: %w", err)
|
||||
}
|
||||
e.CreatedAt, _ = time.Parse(time.RFC3339, created)
|
||||
out = append(out, e)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// DeleteSpamListEntry removes an entry by id.
|
||||
func (s *Store) DeleteSpamListEntry(id int64) error {
|
||||
res, err := s.db.Exec("DELETE FROM inbound_spam_list WHERE id = ?", id)
|
||||
if err != nil {
|
||||
return fmt.Errorf("delete inbound_spam_list: %w", err)
|
||||
}
|
||||
if n, _ := res.RowsAffected(); n == 0 {
|
||||
return ErrSpamListEntryNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SpamListHit is what the journal holds about one message a filter list
|
||||
// decided: who it was from, which server offered it, when, and the detail that
|
||||
// says which list it was.
|
||||
type SpamListHit struct {
|
||||
ClientIP string
|
||||
From string
|
||||
Decision string
|
||||
Detail string
|
||||
CreatedAt time.Time
|
||||
}
|
||||
|
||||
// ListSpamListHits returns the journal rows of the filter in whose detail one
|
||||
// of the marks occurs - the marks the filter lists leave (internal/inbound).
|
||||
// Which entry a row belongs to is for the caller to work out; here the rows
|
||||
// are only narrowed, so that a page view does not read the whole journal.
|
||||
func (s *Store) ListSpamListHits(marks ...string) ([]SpamListHit, error) {
|
||||
if len(marks) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
likes := make([]string, len(marks))
|
||||
args := []any{SpamEngineRspamd}
|
||||
for i, m := range marks {
|
||||
likes[i] = "instr(detail, ?) > 0"
|
||||
args = append(args, m)
|
||||
}
|
||||
rows, err := s.db.Query(
|
||||
`SELECT client_ip, from_addr, decision, detail, created_at FROM inbound_spam_log
|
||||
WHERE engine = ? AND (`+strings.Join(likes, " OR ")+`)`, args...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("query filter list hits: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []SpamListHit
|
||||
for rows.Next() {
|
||||
var h SpamListHit
|
||||
var created string
|
||||
if err := rows.Scan(&h.ClientIP, &h.From, &h.Decision, &h.Detail, &created); err != nil {
|
||||
return nil, fmt.Errorf("scan filter list hit: %w", err)
|
||||
}
|
||||
h.CreatedAt, _ = time.Parse(time.RFC3339, created)
|
||||
out = append(out, h)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
@@ -0,0 +1,338 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestInsertInboundSpam(t *testing.T) {
|
||||
st := openTestStore(t)
|
||||
e := InboundSpamEntry{
|
||||
InboundDomain: "in.example.com", ClientIP: "203.0.113.7",
|
||||
From: "sender@example.net", To: "a@in.example.com", Subject: "Hello",
|
||||
Decision: SpamDecisionAccept, Engine: SpamEngineRspamd, Detail: "no action, score 0.10 / 15.00",
|
||||
}
|
||||
if err := st.InsertInboundSpam(e); err != nil {
|
||||
t.Fatalf("InsertInboundSpam: %v", err)
|
||||
}
|
||||
var got InboundSpamEntry
|
||||
var created string
|
||||
err := st.db.QueryRow(
|
||||
"SELECT inbound_domain, client_ip, from_addr, to_addr, subject, decision, engine, detail, created_at FROM inbound_spam_log",
|
||||
).Scan(&got.InboundDomain, &got.ClientIP, &got.From, &got.To, &got.Subject, &got.Decision, &got.Engine, &got.Detail, &created)
|
||||
if err != nil {
|
||||
t.Fatalf("read back: %v", err)
|
||||
}
|
||||
if got != e {
|
||||
t.Errorf("row = %+v\nwant %+v", got, e)
|
||||
}
|
||||
if at, err := time.Parse(time.RFC3339, created); err != nil || time.Since(at) > time.Minute {
|
||||
t.Errorf("created_at = %q (%v)", created, err)
|
||||
}
|
||||
|
||||
// Every decision and engine the code names is one the table takes.
|
||||
for _, d := range []string{SpamDecisionAccept, SpamDecisionReject, SpamDecisionTempfail, SpamDecisionQuarantine} {
|
||||
for _, eng := range []string{SpamEngineRspamd, SpamEnginePostfix, SpamEngineSelfPost} {
|
||||
e.Decision, e.Engine = d, eng
|
||||
if err := st.InsertInboundSpam(e); err != nil {
|
||||
t.Errorf("decision %s by %s: %v", d, eng, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
e.Decision = "bounce"
|
||||
if err := st.InsertInboundSpam(e); err == nil {
|
||||
t.Error("an unknown decision was stored")
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsInboundDomain(t *testing.T) {
|
||||
st := openTestStore(t)
|
||||
if _, err := st.AddInboundDomain("in.example.com"); err != nil {
|
||||
t.Fatalf("AddInboundDomain: %v", err)
|
||||
}
|
||||
for name, want := range map[string]bool{
|
||||
"in.example.com": true, "IN.Example.COM": true,
|
||||
"example.com": false, "": false, "other.example.com": false,
|
||||
} {
|
||||
if got, err := st.IsInboundDomain(name); err != nil || got != want {
|
||||
t.Errorf("IsInboundDomain(%q) = %v, %v; want %v", name, got, err, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestInsertInboundSpamKeepsItsOwnTime(t *testing.T) {
|
||||
st := openTestStore(t)
|
||||
at := time.Date(2026, 10, 1, 9, 30, 0, 0, time.FixedZone("msk", 3*3600))
|
||||
err := st.InsertInboundSpam(InboundSpamEntry{
|
||||
InboundDomain: "in.example.com", ClientIP: "203.0.113.7", To: "a@in.example.com",
|
||||
Decision: SpamDecisionReject, Engine: SpamEnginePostfix, At: at,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("InsertInboundSpam: %v", err)
|
||||
}
|
||||
var created string
|
||||
if err := st.db.QueryRow("SELECT created_at FROM inbound_spam_log").Scan(&created); err != nil || created != "2026-10-01T06:30:00Z" {
|
||||
t.Errorf("created_at = %q, %v", created, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPruneInboundSpam(t *testing.T) {
|
||||
st := openTestStore(t)
|
||||
now := time.Now().UTC()
|
||||
add := func(to string, age time.Duration) {
|
||||
t.Helper()
|
||||
err := st.InsertInboundSpam(InboundSpamEntry{
|
||||
InboundDomain: "in.example.com", ClientIP: "203.0.113.7", To: to,
|
||||
Decision: SpamDecisionReject, Engine: SpamEnginePostfix, At: now.Add(-age),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("InsertInboundSpam: %v", err)
|
||||
}
|
||||
}
|
||||
left := func() string {
|
||||
t.Helper()
|
||||
rows, err := st.db.Query("SELECT to_addr FROM inbound_spam_log ORDER BY id")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out string
|
||||
for rows.Next() {
|
||||
var to string
|
||||
if err := rows.Scan(&to); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
out += to + " "
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
add("old", 100*24*time.Hour)
|
||||
add("a", 3*time.Hour)
|
||||
add("b", 2*time.Hour)
|
||||
add("c", time.Hour)
|
||||
|
||||
// Age first: nothing else goes while the rest fits.
|
||||
if n, err := st.PruneInboundSpam(now.AddDate(0, 0, -90), 10); err != nil || n != 1 {
|
||||
t.Fatalf("prune by age = %d, %v", n, err)
|
||||
}
|
||||
if got := left(); got != "a b c " {
|
||||
t.Fatalf("left = %q", got)
|
||||
}
|
||||
// Then the cap, oldest first.
|
||||
if n, err := st.PruneInboundSpam(now.AddDate(0, 0, -90), 2); err != nil || n != 1 {
|
||||
t.Fatalf("prune by count = %d, %v", n, err)
|
||||
}
|
||||
if got := left(); got != "b c " {
|
||||
t.Fatalf("left = %q", got)
|
||||
}
|
||||
if n, err := st.PruneInboundSpam(now.AddDate(0, 0, -90), 2); err != nil || n != 0 {
|
||||
t.Errorf("a second prune removed %d, %v", n, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInboundSpamLogRetentionDays(t *testing.T) {
|
||||
st := openTestStore(t)
|
||||
if days, err := st.GetInboundSpamLogRetentionDays(30); err != nil || days != 30 {
|
||||
t.Errorf("unset, env 30: %d, %v", days, err)
|
||||
}
|
||||
if days, err := st.GetInboundSpamLogRetentionDays(3); err != nil || days != InboundSpamLogRetentionDaysDefault {
|
||||
t.Errorf("unset, env out of range: %d, %v", days, err)
|
||||
}
|
||||
if err := st.EnsureInboundSpamLogRetentionDays(30); err != nil {
|
||||
t.Fatalf("Ensure: %v", err)
|
||||
}
|
||||
if err := st.EnsureInboundSpamLogRetentionDays(60); err != nil {
|
||||
t.Fatalf("Ensure again: %v", err)
|
||||
}
|
||||
if days, _ := st.GetInboundSpamLogRetentionDays(90); days != 30 {
|
||||
t.Errorf("after Ensure(30), Ensure(60): %d, want the first seed kept", days)
|
||||
}
|
||||
if err := st.SetInboundSpamLogRetentionDays(6); err == nil {
|
||||
t.Error("6 days was accepted")
|
||||
}
|
||||
if err := st.SetInboundSpamLogRetentionDays(365); err != nil {
|
||||
t.Errorf("365 days: %v", err)
|
||||
}
|
||||
if days, _ := st.GetInboundSpamLogRetentionDays(90); days != 365 {
|
||||
t.Errorf("after Set(365): %d", days)
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueryInboundSpamFilters(t *testing.T) {
|
||||
st := openTestStore(t)
|
||||
day := func(d, h int) time.Time { return time.Date(2026, 9, d, h, 0, 0, 0, time.UTC) }
|
||||
add := func(domain, to, decision string, at time.Time) {
|
||||
t.Helper()
|
||||
err := st.InsertInboundSpam(InboundSpamEntry{
|
||||
InboundDomain: domain, ClientIP: "203.0.113.7", From: "s@example.net", To: to,
|
||||
Subject: "S " + to, Decision: decision, Engine: SpamEngineRspamd, Detail: "d", At: at,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("InsertInboundSpam: %v", err)
|
||||
}
|
||||
}
|
||||
add("a.example", "1@a.example", SpamDecisionAccept, day(20, 23))
|
||||
add("a.example", "2@a.example", SpamDecisionReject, day(21, 0))
|
||||
add("b.example", "3@b.example", SpamDecisionAccept, day(21, 12))
|
||||
add("b.example", "4@b.example", SpamDecisionTempfail, day(22, 0))
|
||||
// Written last but dated earliest: the tailer reads a refusal after the fact.
|
||||
add("a.example", "5@a.example", SpamDecisionReject, day(19, 8))
|
||||
|
||||
to := func(f InboundSpamFilter) string {
|
||||
t.Helper()
|
||||
rows, err := st.QueryInboundSpam(f, 50, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("QueryInboundSpam(%+v): %v", f, err)
|
||||
}
|
||||
n, err := st.CountInboundSpam(f)
|
||||
if err != nil || n != int64(len(rows)) {
|
||||
t.Fatalf("CountInboundSpam(%+v) = %d, %v; the query returned %d", f, n, err, len(rows))
|
||||
}
|
||||
var out string
|
||||
for _, r := range rows {
|
||||
out += r.To[:1]
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Newest first by time, not by the order of writing.
|
||||
if got := to(InboundSpamFilter{}); got != "43215" {
|
||||
t.Errorf("no filter = %q, want 43215", got)
|
||||
}
|
||||
if got := to(InboundSpamFilter{Domain: "a.example"}); got != "215" {
|
||||
t.Errorf("domain = %q, want 215", got)
|
||||
}
|
||||
if got := to(InboundSpamFilter{Decision: SpamDecisionReject}); got != "25" {
|
||||
t.Errorf("decision = %q, want 25", got)
|
||||
}
|
||||
// Days are whole UTC days and both ends are in: 21 st is 00:00 up to 23:59.
|
||||
if got := to(InboundSpamFilter{Since: day(21, 15), Until: day(21, 3)}); got != "32" {
|
||||
t.Errorf("one day = %q, want 32", got)
|
||||
}
|
||||
if got := to(InboundSpamFilter{Since: day(22, 0)}); got != "4" {
|
||||
t.Errorf("since = %q, want 4", got)
|
||||
}
|
||||
if got := to(InboundSpamFilter{Until: day(20, 0)}); got != "15" {
|
||||
t.Errorf("until = %q, want 15", got)
|
||||
}
|
||||
// A time in another zone is read as the UTC day it falls on.
|
||||
msk := time.FixedZone("msk", 3*3600)
|
||||
if got := to(InboundSpamFilter{Since: time.Date(2026, 9, 22, 2, 0, 0, 0, msk)}); got != "432" {
|
||||
t.Errorf("since in another zone = %q, want 432 (it is the 21st in UTC)", got)
|
||||
}
|
||||
if got := to(InboundSpamFilter{Domain: "b.example", Decision: SpamDecisionAccept, Since: day(21, 0), Until: day(21, 0)}); got != "3" {
|
||||
t.Errorf("all filters = %q, want 3", got)
|
||||
}
|
||||
|
||||
// The scope: nil is everything, a set is its domains, an empty set nothing,
|
||||
// and a chosen domain outside the set is nothing too.
|
||||
if got := to(InboundSpamFilter{Domains: []string{"b.example"}}); got != "43" {
|
||||
t.Errorf("scope = %q, want 43", got)
|
||||
}
|
||||
if got := to(InboundSpamFilter{Domains: []string{}}); got != "" {
|
||||
t.Errorf("empty scope = %q, want nothing", got)
|
||||
}
|
||||
if got := to(InboundSpamFilter{Domains: []string{"b.example"}, Domain: "a.example"}); got != "" {
|
||||
t.Errorf("a chosen domain outside the scope = %q, want nothing", got)
|
||||
}
|
||||
|
||||
// Paging.
|
||||
rows, err := st.QueryInboundSpam(InboundSpamFilter{}, 2, 2)
|
||||
if err != nil || len(rows) != 2 || rows[0].To != "2@a.example" || rows[1].To != "1@a.example" {
|
||||
t.Errorf("page 2 of 2 rows = %+v, %v", rows, err)
|
||||
}
|
||||
if rows[0].ID == 0 || !rows[0].CreatedAt.Equal(day(21, 0)) || rows[0].Subject != "S 2@a.example" ||
|
||||
rows[0].Engine != SpamEngineRspamd || rows[0].Detail != "d" || rows[0].ClientIP != "203.0.113.7" || rows[0].From != "s@example.net" {
|
||||
t.Errorf("a row read back = %+v", rows[0])
|
||||
}
|
||||
|
||||
// A value is data, not SQL.
|
||||
if got := to(InboundSpamFilter{Domain: "a.example' OR '1'='1"}); got != "" {
|
||||
t.Errorf("injected domain = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestListSpamListHits(t *testing.T) {
|
||||
st := openTestStore(t)
|
||||
add := func(engine, decision, detail string) {
|
||||
t.Helper()
|
||||
err := st.InsertInboundSpam(InboundSpamEntry{
|
||||
InboundDomain: "in.example.com", ClientIP: "203.0.113.7", From: "a@example.net", To: "a@in.example.com",
|
||||
Decision: decision, Engine: engine, Detail: detail, At: time.Date(2026, 10, 1, 9, 0, 0, 0, time.UTC),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
add(SpamEngineRspamd, SpamDecisionReject, "5.7.1 Refused by this server's deny list")
|
||||
add(SpamEngineRspamd, SpamDecisionAccept, "no action, score -99.00 / 15.00; SELFPOST_ALLOW_IP(-100.00)")
|
||||
add(SpamEngineRspamd, SpamDecisionReject, "5.7.1 Spam message rejected")
|
||||
add(SpamEnginePostfix, SpamDecisionReject, "550 5.1.1 this server's deny list")
|
||||
// a mark with the characters LIKE would treat as wildcards is text
|
||||
add(SpamEngineRspamd, SpamDecisionAccept, "no action; SELFPOSTXALLOWX(1.00)")
|
||||
|
||||
hits, err := st.ListSpamListHits("this server's deny list", "SELFPOST_ALLOW_")
|
||||
if err != nil || len(hits) != 2 {
|
||||
t.Fatalf("hits = %+v, %v; want the two rows a list decided", hits, err)
|
||||
}
|
||||
h := hits[0]
|
||||
if h.From != "a@example.net" || h.ClientIP != "203.0.113.7" || h.Decision != SpamDecisionReject || h.CreatedAt.Hour() != 9 {
|
||||
t.Errorf("hit = %+v", h)
|
||||
}
|
||||
if hits, err := st.ListSpamListHits(); err != nil || hits != nil {
|
||||
t.Errorf("no marks: %v, %v", hits, err)
|
||||
}
|
||||
}
|
||||
|
||||
// The cap removes the oldest by when it happened. After the panel was down the
|
||||
// tailer writes old refusals with the newest ids, and they are not what stays.
|
||||
func TestPruneInboundSpamCapIsByTime(t *testing.T) {
|
||||
st := openTestStore(t)
|
||||
now := time.Now().UTC()
|
||||
for _, r := range []struct {
|
||||
to string
|
||||
age time.Duration
|
||||
}{{"new-1", time.Minute}, {"new-2", 2 * time.Minute}, {"backdated-old", 5 * time.Hour}} {
|
||||
err := st.InsertInboundSpam(InboundSpamEntry{
|
||||
InboundDomain: "in.example.com", ClientIP: "203.0.113.7", To: r.to,
|
||||
Decision: SpamDecisionReject, Engine: SpamEnginePostfix, At: now.Add(-r.age),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if n, err := st.PruneInboundSpam(time.Time{}, 2); err != nil || n != 1 {
|
||||
t.Fatalf("prune = %d, %v", n, err)
|
||||
}
|
||||
rows, err := st.QueryInboundSpam(InboundSpamFilter{}, 10, 0)
|
||||
if err != nil || len(rows) != 2 || rows[0].To != "new-1" || rows[1].To != "new-2" {
|
||||
t.Errorf("left = %+v, %v; want the two newest by time", rows, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInsertInboundSpamOnce(t *testing.T) {
|
||||
st := openTestStore(t)
|
||||
e := InboundSpamEntry{
|
||||
InboundDomain: "in.example.com", ClientIP: "203.0.113.7", From: "a@example.net", To: "nobody@in.example.com",
|
||||
Decision: SpamDecisionReject, Engine: SpamEnginePostfix, Detail: "550 5.1.1 no",
|
||||
At: time.Date(2026, 10, 12, 9, 0, 5, 0, time.UTC),
|
||||
}
|
||||
for i, want := range []bool{true, false, false} {
|
||||
if wrote, err := st.InsertInboundSpamOnce(e); err != nil || wrote != want {
|
||||
t.Fatalf("attempt %d: wrote %v, %v; want %v", i+1, wrote, err, want)
|
||||
}
|
||||
}
|
||||
// Anything that tells two refusals apart makes a second row.
|
||||
later, other := e, e
|
||||
later.At = e.At.Add(time.Second)
|
||||
other.To = "somebody@in.example.com"
|
||||
for name, v := range map[string]InboundSpamEntry{"a second later": later, "another recipient": other} {
|
||||
if wrote, err := st.InsertInboundSpamOnce(v); err != nil || !wrote {
|
||||
t.Errorf("%s: wrote %v, %v", name, wrote, err)
|
||||
}
|
||||
}
|
||||
if n, err := st.CountInboundSpam(InboundSpamFilter{}); err != nil || n != 3 {
|
||||
t.Errorf("rows = %d, %v; want 3", n, err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
-- Inbound antispam journal and filter lists (inbound-antispam-panel, 2.1.0).
|
||||
-- The first migration after the 2.0 baseline: DDL only, so a 2.0.x database
|
||||
-- gains two empty tables and nothing in it changes.
|
||||
|
||||
-- What happened to a message offered on port 25, one row per recipient.
|
||||
-- Metadata only - SelfPost relays inbound mail and stores none of it. Accepted
|
||||
-- messages are written by the inbound journal milter, refused ones by the
|
||||
-- mail.log tailer, which is why a row can lack a subject (refused before DATA)
|
||||
-- and why engine says who decided: the external filter, Postfix's own policy,
|
||||
-- or nobody (selfpost: relayed with no filter attached).
|
||||
CREATE TABLE inbound_spam_log (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
inbound_domain TEXT NOT NULL,
|
||||
client_ip TEXT NOT NULL,
|
||||
from_addr TEXT NOT NULL,
|
||||
to_addr TEXT NOT NULL,
|
||||
subject TEXT NOT NULL,
|
||||
decision TEXT NOT NULL CHECK (decision IN ('accept', 'reject', 'tempfail', 'quarantine')),
|
||||
engine TEXT NOT NULL CHECK (engine IN ('rspamd', 'postfix', 'selfpost')),
|
||||
detail TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL
|
||||
);
|
||||
-- Retention prunes by age; the Log page lists newest first, for the whole
|
||||
-- server or for one domain.
|
||||
CREATE INDEX idx_inbound_spam_log_created_at ON inbound_spam_log (created_at);
|
||||
CREATE INDEX idx_inbound_spam_log_domain_created_at ON inbound_spam_log (inbound_domain, created_at);
|
||||
|
||||
-- Allow / deny entries for the external filter, instance-wide. The panel
|
||||
-- writes them out as map files under /data/antispam/; SelfPost itself does not
|
||||
-- consult them. value is stored normalised, so the same entry cannot be added
|
||||
-- twice to one list.
|
||||
CREATE TABLE inbound_spam_list (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
list_type TEXT NOT NULL CHECK (list_type IN ('allow', 'deny')),
|
||||
entry_type TEXT NOT NULL CHECK (entry_type IN ('address', 'domain', 'ip')),
|
||||
value TEXT NOT NULL,
|
||||
note TEXT NOT NULL DEFAULT '',
|
||||
created_at TEXT NOT NULL,
|
||||
UNIQUE (list_type, entry_type, value)
|
||||
);
|
||||
@@ -0,0 +1,22 @@
|
||||
-- What Postfix wrote about a message, kept with its send-log rows
|
||||
-- (delivery-log-storage, 2.2.0). DDL only: a 2.1.0 database gains one empty
|
||||
-- table and nothing in it changes.
|
||||
|
||||
-- One row per mail.log line headed by a queue id, written by the log tailer as
|
||||
-- it reads the log. mail.log is rotated daily and fourteen files are kept; a
|
||||
-- send-log row lives for the retention window, ninety days by default, and
|
||||
-- its page shows these lines for as long.
|
||||
--
|
||||
-- There is no reference to send_log: the first lines of a message are logged
|
||||
-- before the journal milter writes its row, and a message with several
|
||||
-- recipients is several rows under one queue id. A line belongs to a row by
|
||||
-- queue id and by time (Postfix reuses queue ids), and a line whose queue id
|
||||
-- has no send-log row is deleted by the tailer's prune.
|
||||
CREATE TABLE send_log_lines (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
queue_id TEXT NOT NULL,
|
||||
logged_at TEXT NOT NULL,
|
||||
line TEXT NOT NULL
|
||||
);
|
||||
-- A message's page reads the lines of one queue id within a span of time.
|
||||
CREATE INDEX idx_send_log_lines_queue_id ON send_log_lines (queue_id, logged_at);
|
||||
@@ -0,0 +1,28 @@
|
||||
-- Notifications: the panel writes to its users when something needs them
|
||||
-- (panel-notifications, 2.3.0). DDL only; nothing existing changes, and every
|
||||
-- user starts with notifications on.
|
||||
|
||||
-- A user's own switch, on Account. It decides nothing by itself: a user is
|
||||
-- written to when it is on, the profile has an e-mail and the instance switch
|
||||
-- (settings key "notifications") is on.
|
||||
ALTER TABLE users ADD COLUMN notify INTEGER NOT NULL DEFAULT 1 CHECK (notify IN (0, 1));
|
||||
|
||||
-- What the panel has already said, so that it says each thing once and a
|
||||
-- restart announces nothing again. One row per subject:
|
||||
-- dns:<domain> a sending domain's records
|
||||
-- cert the TLS certificate
|
||||
-- queue a message older than three days
|
||||
-- limit:<domain>:<login> a level-2 limit that refused mail
|
||||
-- phase is where the subject stands ("pending": seen wrong, not yet said;
|
||||
-- "open": said; for the certificate, the threshold last said). count is the
|
||||
-- checks in a row that agree with the phase's direction, mark what the phase
|
||||
-- refers to (the certificate's expiry), since when it began, sent_at when the
|
||||
-- last message about it went out.
|
||||
CREATE TABLE notify_state (
|
||||
key TEXT PRIMARY KEY,
|
||||
phase TEXT NOT NULL,
|
||||
count INTEGER NOT NULL DEFAULT 0,
|
||||
mark TEXT NOT NULL DEFAULT '',
|
||||
since TEXT NOT NULL,
|
||||
sent_at TEXT NOT NULL DEFAULT ''
|
||||
);
|
||||
@@ -0,0 +1,194 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// settingNotifications is the instance switch for notification e-mails. A
|
||||
// missing value means on: the switch exists to turn them off.
|
||||
const settingNotifications = "notifications"
|
||||
|
||||
// NotificationsEnabled reports whether the instance sends notification e-mails
|
||||
// at all (Server › Settings).
|
||||
func (s *Store) NotificationsEnabled() (bool, error) {
|
||||
v, err := s.GetSetting(settingNotifications)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
return v != "off", nil
|
||||
}
|
||||
|
||||
// SetNotificationsEnabled turns notification e-mails on or off for everyone.
|
||||
func (s *Store) SetNotificationsEnabled(on bool) error {
|
||||
v := "on"
|
||||
if !on {
|
||||
v = "off"
|
||||
}
|
||||
return s.SetSetting(settingNotifications, v)
|
||||
}
|
||||
|
||||
// SetUserNotify sets a user's own switch (Account).
|
||||
func (s *Store) SetUserNotify(id int64, on bool) error {
|
||||
res, err := s.db.Exec("UPDATE users SET notify = ? WHERE id = ?", on, id)
|
||||
if err != nil {
|
||||
return fmt.Errorf("set user notify: %w", err)
|
||||
}
|
||||
if n, _ := res.RowsAffected(); n == 0 {
|
||||
return ErrUserNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// NotifyRecipients returns the addresses a notification goes to: every global
|
||||
// user, and for a subject that is a sending domain's (domainName not empty)
|
||||
// also the domain users it is assigned to - by name on their list, or by
|
||||
// "all domains". Only users with an e-mail and their own switch on; an address
|
||||
// two users share is returned once.
|
||||
func (s *Store) NotifyRecipients(domainName string) ([]string, error) {
|
||||
query := `SELECT DISTINCT email FROM users
|
||||
WHERE notify = 1 AND email <> '' AND role = ?`
|
||||
args := []any{RoleGlobal}
|
||||
if domainName != "" {
|
||||
query = `SELECT DISTINCT email FROM users
|
||||
WHERE notify = 1 AND email <> '' AND (
|
||||
role = ?
|
||||
OR all_domains = 1
|
||||
OR EXISTS (SELECT 1 FROM user_domains ud JOIN domains d ON d.id = ud.domain_id
|
||||
WHERE ud.user_id = users.id AND d.name = ?))`
|
||||
args = append(args, domainName)
|
||||
}
|
||||
rows, err := s.db.Query(query+" ORDER BY email", args...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("notify recipients: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []string
|
||||
for rows.Next() {
|
||||
var email string
|
||||
if err := rows.Scan(&email); err != nil {
|
||||
return nil, fmt.Errorf("notify recipients scan: %w", err)
|
||||
}
|
||||
out = append(out, email)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// NotifyState is what the panel remembers about one subject of notification
|
||||
// (migration 0004 describes the keys and the fields).
|
||||
type NotifyState struct {
|
||||
Key string
|
||||
Phase string
|
||||
Count int
|
||||
Mark string
|
||||
Since time.Time
|
||||
SentAt time.Time // zero when nothing was sent yet
|
||||
}
|
||||
|
||||
// GetNotifyState returns the state kept under key; ok is false when there is
|
||||
// none, which is the state of a subject nothing is wrong with.
|
||||
func (s *Store) GetNotifyState(key string) (NotifyState, bool, error) {
|
||||
st := NotifyState{Key: key}
|
||||
var since, sentAt string
|
||||
err := s.db.QueryRow(
|
||||
"SELECT phase, count, mark, since, sent_at FROM notify_state WHERE key = ?", key,
|
||||
).Scan(&st.Phase, &st.Count, &st.Mark, &since, &sentAt)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return NotifyState{}, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return NotifyState{}, false, fmt.Errorf("get notify state %s: %w", key, err)
|
||||
}
|
||||
st.Since, _ = time.Parse(time.RFC3339, since)
|
||||
if sentAt != "" {
|
||||
st.SentAt, _ = time.Parse(time.RFC3339, sentAt)
|
||||
}
|
||||
return st, true, nil
|
||||
}
|
||||
|
||||
// SetNotifyState writes the state of one subject, replacing what was there.
|
||||
func (s *Store) SetNotifyState(st NotifyState) error {
|
||||
sentAt := ""
|
||||
if !st.SentAt.IsZero() {
|
||||
sentAt = st.SentAt.UTC().Format(time.RFC3339)
|
||||
}
|
||||
_, err := s.db.Exec(
|
||||
`INSERT INTO notify_state (key, phase, count, mark, since, sent_at) VALUES (?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(key) DO UPDATE SET phase = excluded.phase, count = excluded.count,
|
||||
mark = excluded.mark, since = excluded.since, sent_at = excluded.sent_at`,
|
||||
st.Key, st.Phase, st.Count, st.Mark, st.Since.UTC().Format(time.RFC3339), sentAt,
|
||||
)
|
||||
if err != nil {
|
||||
return fmt.Errorf("set notify state %s: %w", st.Key, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// DeleteNotifyState forgets a subject: nothing is wrong with it any more, or
|
||||
// it no longer exists.
|
||||
func (s *Store) DeleteNotifyState(key string) error {
|
||||
if _, err := s.db.Exec("DELETE FROM notify_state WHERE key = ?", key); err != nil {
|
||||
return fmt.Errorf("delete notify state %s: %w", key, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ListNotifyStateKeys returns the keys that begin with prefix, so that the
|
||||
// states of subjects that are gone - a deleted domain - can be found.
|
||||
func (s *Store) ListNotifyStateKeys(prefix string) ([]string, error) {
|
||||
rows, err := s.db.Query(
|
||||
"SELECT key FROM notify_state WHERE substr(key, 1, ?) = ? ORDER BY key", len(prefix), prefix)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("list notify state: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []string
|
||||
for rows.Next() {
|
||||
var key string
|
||||
if err := rows.Scan(&key); err != nil {
|
||||
return nil, fmt.Errorf("list notify state scan: %w", err)
|
||||
}
|
||||
out = append(out, key)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// LimitRefusal is the mail one application was refused under a level-2 rate
|
||||
// limit in a span of time: how many messages and when the first.
|
||||
type LimitRefusal struct {
|
||||
Domain string
|
||||
AppLogin string
|
||||
Count int
|
||||
First time.Time
|
||||
}
|
||||
|
||||
// LimitRefusalsSince returns, per domain and application, the messages the
|
||||
// journal milter refused under a level-2 limit at or after since (send-log
|
||||
// rows with status "rejected").
|
||||
func (s *Store) LimitRefusalsSince(since time.Time) ([]LimitRefusal, error) {
|
||||
rows, err := s.db.Query(
|
||||
`SELECT COALESCE(domain, ''), COALESCE(app_login, ''), COUNT(*), MIN(created_at) FROM send_log
|
||||
WHERE status = ? AND created_at >= ?
|
||||
GROUP BY domain, app_login ORDER BY domain, app_login`,
|
||||
StatusRejected, since.UTC().Format(time.RFC3339),
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("limit refusals: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []LimitRefusal
|
||||
for rows.Next() {
|
||||
var (
|
||||
r LimitRefusal
|
||||
first string
|
||||
)
|
||||
if err := rows.Scan(&r.Domain, &r.AppLogin, &r.Count, &first); err != nil {
|
||||
return nil, fmt.Errorf("limit refusals scan: %w", err)
|
||||
}
|
||||
r.First, _ = time.Parse(time.RFC3339, first)
|
||||
out = append(out, r)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
@@ -0,0 +1,166 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The instance switch is on until someone turns it off: a database that never
|
||||
// saw the setting sends notifications.
|
||||
func TestNotificationsEnabledByDefault(t *testing.T) {
|
||||
s := openTestStore(t)
|
||||
if on, err := s.NotificationsEnabled(); err != nil || !on {
|
||||
t.Fatalf("a fresh database: enabled = %v, %v; want on", on, err)
|
||||
}
|
||||
if err := s.SetNotificationsEnabled(false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if on, err := s.NotificationsEnabled(); err != nil || on {
|
||||
t.Errorf("after turning off: enabled = %v, %v", on, err)
|
||||
}
|
||||
if err := s.SetNotificationsEnabled(true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if on, err := s.NotificationsEnabled(); err != nil || !on {
|
||||
t.Errorf("after turning on again: enabled = %v, %v", on, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Who is written to: every global user about everything, a domain user about
|
||||
// the domains on their list or all of them - and nobody without an e-mail or
|
||||
// with their own switch off.
|
||||
func TestNotifyRecipients(t *testing.T) {
|
||||
s := openTestStore(t)
|
||||
mine, err := s.AddDomain("mine.example.com", "sp")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
other, err := s.AddDomain("other.example.com", "sp")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
user := func(name string, role Role, reach Reach, email string, notify bool) {
|
||||
t.Helper()
|
||||
id, err := s.CreateUser(name, "hash", role, reach)
|
||||
if err != nil {
|
||||
t.Fatalf("create %s: %v", name, err)
|
||||
}
|
||||
if err := s.UpdateUser(id, name, "hash", email); err != nil {
|
||||
t.Fatalf("update %s: %v", name, err)
|
||||
}
|
||||
if u, err := s.GetUser(id); err != nil || !u.Notify {
|
||||
t.Fatalf("%s: a new user's switch is %v, %v; want on", name, u.Notify, err)
|
||||
}
|
||||
if !notify {
|
||||
if err := s.SetUserNotify(id, false); err != nil {
|
||||
t.Fatalf("switch off %s: %v", name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
user("root", RoleGlobal, Reach{}, "root@example.org", true)
|
||||
user("root2", RoleGlobal, Reach{}, "root@example.org", true) // the same mailbox
|
||||
user("quiet", RoleGlobal, Reach{}, "quiet@example.org", false)
|
||||
user("nomail", RoleGlobal, Reach{}, "", true)
|
||||
user("mine", RoleDomain, Reach{DomainIDs: []int64{mine.ID}}, "mine@example.org", true)
|
||||
user("theirs", RoleDomain, Reach{DomainIDs: []int64{other.ID}}, "theirs@example.org", true)
|
||||
user("every", RoleDomain, Reach{AllDomains: true}, "every@example.org", true)
|
||||
user("mineoff", RoleDomain, Reach{DomainIDs: []int64{mine.ID}}, "mineoff@example.org", false)
|
||||
|
||||
for _, c := range []struct{ domain, want string }{
|
||||
{"", "root@example.org"},
|
||||
{"mine.example.com", "every@example.org mine@example.org root@example.org"},
|
||||
{"other.example.com", "every@example.org root@example.org theirs@example.org"},
|
||||
{"gone.example.com", "every@example.org root@example.org"},
|
||||
} {
|
||||
got, err := s.NotifyRecipients(c.domain)
|
||||
if err != nil {
|
||||
t.Fatalf("recipients for %q: %v", c.domain, err)
|
||||
}
|
||||
if strings.Join(got, " ") != c.want {
|
||||
t.Errorf("recipients for %q = %q, want %q", c.domain, strings.Join(got, " "), c.want)
|
||||
}
|
||||
}
|
||||
|
||||
if err := s.SetUserNotify(99999, true); err != ErrUserNotFound {
|
||||
t.Errorf("switch of a user that does not exist: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// What has been said is remembered by key, replaced whole, and forgotten.
|
||||
func TestNotifyStateRoundTrip(t *testing.T) {
|
||||
s := openTestStore(t)
|
||||
if _, ok, err := s.GetNotifyState("dns:example.com"); err != nil || ok {
|
||||
t.Fatalf("no state yet: ok = %v, %v", ok, err)
|
||||
}
|
||||
since := time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC)
|
||||
st := NotifyState{Key: "dns:example.com", Phase: "pending", Count: 1, Since: since}
|
||||
if err := s.SetNotifyState(st); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, ok, err := s.GetNotifyState("dns:example.com")
|
||||
if err != nil || !ok || got.Phase != "pending" || got.Count != 1 || !got.Since.Equal(since) || !got.SentAt.IsZero() {
|
||||
t.Fatalf("state = %+v, %v, %v", got, ok, err)
|
||||
}
|
||||
|
||||
st.Phase, st.Count, st.Mark, st.SentAt = "open", 0, "x", since.Add(time.Hour)
|
||||
if err := s.SetNotifyState(st); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, _, _ = s.GetNotifyState("dns:example.com")
|
||||
if got.Phase != "open" || got.Mark != "x" || !got.SentAt.Equal(since.Add(time.Hour)) {
|
||||
t.Errorf("replaced state = %+v", got)
|
||||
}
|
||||
|
||||
for _, key := range []string{"dns:other.example.com", "queue", "limit:example.com:app"} {
|
||||
if err := s.SetNotifyState(NotifyState{Key: key, Phase: "open", Since: since}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
keys, err := s.ListNotifyStateKeys("dns:")
|
||||
if err != nil || strings.Join(keys, " ") != "dns:example.com dns:other.example.com" {
|
||||
t.Errorf("dns keys = %v, %v", keys, err)
|
||||
}
|
||||
if err := s.DeleteNotifyState("dns:example.com"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, ok, _ := s.GetNotifyState("dns:example.com"); ok {
|
||||
t.Error("state still there after delete")
|
||||
}
|
||||
}
|
||||
|
||||
// Refused mail is counted per domain and application from a point in time;
|
||||
// delivered mail and older refusals are not.
|
||||
func TestLimitRefusalsSince(t *testing.T) {
|
||||
s := openTestStore(t)
|
||||
for _, e := range []SendLogEntry{
|
||||
{Domain: "example.com", AppLogin: "app1", From: "a@example.com"},
|
||||
{Domain: "example.com", AppLogin: "app1", From: "a@example.com"},
|
||||
{Domain: "example.com", AppLogin: "app2", From: "b@example.com"},
|
||||
} {
|
||||
if err := s.InsertRejected(e); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := s.InsertQueued(SendLogEntry{QueueID: "ABC123", Domain: "example.com", AppLogin: "app3", To: "x@example.net"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.InsertRejected(SendLogEntry{Domain: "old.example.com", AppLogin: "old"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.db.Exec(`UPDATE send_log SET created_at = '2026-01-01T00:00:00Z' WHERE app_login = 'old'`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
got, err := s.LimitRefusalsSince(time.Now().UTC().Add(-time.Hour))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) != 2 || got[0].AppLogin != "app1" || got[0].Count != 2 || got[1].AppLogin != "app2" || got[1].Count != 1 {
|
||||
t.Fatalf("refusals = %+v", got)
|
||||
}
|
||||
if got[0].Domain != "example.com" || got[0].First.IsZero() {
|
||||
t.Errorf("first refusal = %+v", got[0])
|
||||
}
|
||||
}
|
||||
@@ -40,7 +40,7 @@ func TestBaselineSchema(t *testing.T) {
|
||||
st := openTestStore(t)
|
||||
want := map[string]string{
|
||||
"settings": "key value",
|
||||
"users": "id username password_hash role email all_domains all_inbound_domains created_at",
|
||||
"users": "id username password_hash role email all_domains all_inbound_domains created_at notify",
|
||||
"sessions": "token_hash user_id created_at expires_at",
|
||||
"domains": "id name dkim_selector dmarc_rua created_at",
|
||||
"user_domains": "user_id domain_id",
|
||||
@@ -55,6 +55,10 @@ func TestBaselineSchema(t *testing.T) {
|
||||
"user_inbound_domains": "user_id inbound_domain_id",
|
||||
"dmarc_reports": "id domain reporter report_id period_begin period_end received_at contact_email policy_p policy_sp policy_pct policy_adkim policy_aspf pass_count fail_count recipient",
|
||||
"dmarc_report_records": "id report_row_id source_ip count disposition spf_result dkim_result header_from",
|
||||
"inbound_spam_log": "id inbound_domain client_ip from_addr to_addr subject decision engine detail created_at",
|
||||
"inbound_spam_list": "id list_type entry_type value note created_at",
|
||||
"send_log_lines": "id queue_id logged_at line",
|
||||
"notify_state": "key phase count mark since sent_at",
|
||||
}
|
||||
rows, err := st.db.Query("SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%' ORDER BY name")
|
||||
if err != nil {
|
||||
@@ -99,7 +103,8 @@ func TestBaselineSchema(t *testing.T) {
|
||||
rows.Close()
|
||||
wantIdx := []string{
|
||||
"idx_dmarc_report_records_report", "idx_dmarc_reports_domain", "idx_dmarc_reports_received",
|
||||
"idx_send_log_app_login_created_at", "idx_send_log_created_at", "idx_send_log_domain", "idx_send_log_queue_id",
|
||||
"idx_inbound_spam_log_created_at", "idx_inbound_spam_log_domain_created_at",
|
||||
"idx_send_log_app_login_created_at", "idx_send_log_created_at", "idx_send_log_domain", "idx_send_log_lines_queue_id", "idx_send_log_queue_id",
|
||||
"idx_sessions_expires_at",
|
||||
}
|
||||
sort.Strings(wantIdx)
|
||||
@@ -108,8 +113,8 @@ func TestBaselineSchema(t *testing.T) {
|
||||
}
|
||||
|
||||
var version, appID int64
|
||||
if err := st.db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 1 {
|
||||
t.Errorf("user_version = %d, %v; want 1", version, err)
|
||||
if err := st.db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 4 {
|
||||
t.Errorf("user_version = %d, %v; want 4", version, err)
|
||||
}
|
||||
if err := st.db.QueryRow("PRAGMA application_id").Scan(&appID); err != nil || appID != applicationID {
|
||||
t.Errorf("application_id = %#x, %v; want %#x", appID, err, applicationID)
|
||||
@@ -156,6 +161,96 @@ func TestReopenKeepsTheDatabase(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A 2.0.x data directory - the baseline and nothing after it - is brought to
|
||||
// the head by opening it, with what it holds left alone.
|
||||
func TestOpenMigratesA20Database(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "test.db")
|
||||
baseline, err := migrationsFS.ReadFile("migrations/0001_init.sql")
|
||||
if err != nil {
|
||||
t.Fatalf("read the baseline: %v", err)
|
||||
}
|
||||
db, err := sql.Open("sqlite", "file:"+path)
|
||||
if err != nil {
|
||||
t.Fatalf("create a 2.0 db: %v", err)
|
||||
}
|
||||
for _, stmt := range []string{
|
||||
string(baseline),
|
||||
"PRAGMA user_version = 1",
|
||||
"INSERT INTO users (username, password_hash, role, created_at) VALUES ('admin', 'hash', 'global', '2026-10-11T00:00:00Z')",
|
||||
} {
|
||||
if _, err := db.Exec(stmt); err != nil {
|
||||
t.Fatalf("prepare a 2.0 db: %v", err)
|
||||
}
|
||||
}
|
||||
db.Close()
|
||||
|
||||
st, err := Open(path)
|
||||
if err != nil {
|
||||
t.Fatalf("open a 2.0 database: %v", err)
|
||||
}
|
||||
defer st.Close()
|
||||
var version int
|
||||
if err := st.db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 4 {
|
||||
t.Errorf("user_version = %d, %v; want 4", version, err)
|
||||
}
|
||||
if ok, err := st.UserExists(); err != nil || !ok {
|
||||
t.Errorf("the user of the 2.0 database: exists = %v, %v", ok, err)
|
||||
}
|
||||
for _, table := range []string{"inbound_spam_log", "inbound_spam_list", "send_log_lines", "notify_state"} {
|
||||
if len(tableColumns(t, st, table)) == 0 {
|
||||
t.Errorf("table %s was not created", table)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// What the antispam tables refuse: a decision, engine or list the panel does
|
||||
// not know, and the same entry twice in one list.
|
||||
func TestInboundSpamConstraints(t *testing.T) {
|
||||
st := openTestStore(t)
|
||||
logRow := func(decision, engine string) error {
|
||||
_, err := st.db.Exec(
|
||||
"INSERT INTO inbound_spam_log (inbound_domain, client_ip, from_addr, to_addr, subject, decision, engine, detail, created_at)"+
|
||||
" VALUES ('in.example.com', '192.0.2.1', 'a@example.net', 'b@in.example.com', '', ?, ?, '', '2026-10-11T00:00:00Z')",
|
||||
decision, engine)
|
||||
return err
|
||||
}
|
||||
if err := logRow("reject", "postfix"); err != nil {
|
||||
t.Fatalf("a reject by postfix: %v", err)
|
||||
}
|
||||
if err := logRow("bounce", "postfix"); err == nil {
|
||||
t.Error("decision bounce was accepted")
|
||||
}
|
||||
if err := logRow("accept", "spamassassin"); err == nil {
|
||||
t.Error("engine spamassassin was accepted")
|
||||
}
|
||||
|
||||
entry := func(list, kind, value string) error {
|
||||
_, err := st.db.Exec(
|
||||
"INSERT INTO inbound_spam_list (list_type, entry_type, value, created_at) VALUES (?, ?, ?, '2026-10-11T00:00:00Z')",
|
||||
list, kind, value)
|
||||
return err
|
||||
}
|
||||
if err := entry("allow", "address", "billing@vendor.example"); err != nil {
|
||||
t.Fatalf("an allow entry: %v", err)
|
||||
}
|
||||
if err := entry("allow", "address", "billing@vendor.example"); err == nil {
|
||||
t.Error("the same entry was added to one list twice")
|
||||
}
|
||||
if err := entry("deny", "address", "billing@vendor.example"); err != nil {
|
||||
t.Errorf("the same value on the other list: %v", err)
|
||||
}
|
||||
if err := entry("block", "address", "x@example.net"); err == nil {
|
||||
t.Error("list type block was accepted")
|
||||
}
|
||||
if err := entry("deny", "subject", "pills"); err == nil {
|
||||
t.Error("entry type subject was accepted")
|
||||
}
|
||||
var note string
|
||||
if err := st.db.QueryRow("SELECT note FROM inbound_spam_list WHERE list_type = 'allow'").Scan(¬e); err != nil || note != "" {
|
||||
t.Errorf("note of an entry added without one = %q, %v", note, err)
|
||||
}
|
||||
}
|
||||
|
||||
// A 1.x data directory must be refused, loudly. Its user_version (10 at the
|
||||
// last 1.x release) would otherwise read as "the baseline is already applied",
|
||||
// and the panel would start on tables it does not know.
|
||||
|
||||
@@ -0,0 +1,152 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
// SendLogLine is one mail.log line Postfix wrote under a queue id, kept so a
|
||||
// message's page can show it after mail.log has rotated (architecture.md § Log
|
||||
// tailer). At is the line's own time, not when the tailer read it.
|
||||
type SendLogLine struct {
|
||||
QueueID string
|
||||
At time.Time
|
||||
Line string
|
||||
}
|
||||
|
||||
// SendLogLineMax is how many bytes of a line are kept. Postfix quotes the far
|
||||
// end's reply in a delivery line, so a line is as long as someone else cared
|
||||
// to make it; what a page needs of it is the beginning.
|
||||
const SendLogLineMax = 4096
|
||||
|
||||
// sendLogLineBefore is how long before a row was written a line may be dated
|
||||
// and still be the row's. The journal milter writes the row at end-of-message;
|
||||
// the smtpd line that opens the queue id is written when the first recipient
|
||||
// is accepted, and the message body can take its time after that.
|
||||
const sendLogLineBefore = time.Hour
|
||||
|
||||
// cutSendLogLine cuts a line to SendLogLineMax bytes, and not through the
|
||||
// middle of a character.
|
||||
func cutSendLogLine(line string) string {
|
||||
if len(line) <= SendLogLineMax {
|
||||
return line
|
||||
}
|
||||
end := SendLogLineMax
|
||||
for end > 0 && !utf8.RuneStart(line[end]) {
|
||||
end--
|
||||
}
|
||||
return line[:end]
|
||||
}
|
||||
|
||||
// InsertSendLogLines stores lines in one transaction and returns how many it
|
||||
// wrote. The tailer calls it with what it read in one pass over the log,
|
||||
// whether or not send-log rows exist for the queue ids yet: a commit per line
|
||||
// would be a disk sync per line, on the one connection the journal milter
|
||||
// writes through.
|
||||
//
|
||||
// With once set, a line is written only if the same line is not there already
|
||||
// under its queue id. That is for the lines the tailer reads again after an
|
||||
// unclean stop; a line carries its time and its process id, so two that are
|
||||
// equal are one line read twice.
|
||||
func (s *Store) InsertSendLogLines(lines []SendLogLine, once bool) (int, error) {
|
||||
if len(lines) == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
query := `INSERT INTO send_log_lines (queue_id, logged_at, line) VALUES (?1, ?2, ?3)`
|
||||
if once {
|
||||
query = `INSERT INTO send_log_lines (queue_id, logged_at, line)
|
||||
SELECT ?1, ?2, ?3
|
||||
WHERE NOT EXISTS (SELECT 1 FROM send_log_lines WHERE queue_id = ?1 AND line = ?3)`
|
||||
}
|
||||
tx, err := s.db.Begin()
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("insert send_log_lines: %w", err)
|
||||
}
|
||||
defer tx.Rollback()
|
||||
stmt, err := tx.Prepare(query)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("insert send_log_lines: %w", err)
|
||||
}
|
||||
defer stmt.Close()
|
||||
wrote := 0
|
||||
for _, l := range lines {
|
||||
res, err := stmt.Exec(l.QueueID, l.At.UTC().Format(time.RFC3339), cutSendLogLine(l.Line))
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("insert send_log_lines: %w", err)
|
||||
}
|
||||
n, _ := res.RowsAffected()
|
||||
wrote += int(n)
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
return 0, fmt.Errorf("insert send_log_lines: %w", err)
|
||||
}
|
||||
return wrote, nil
|
||||
}
|
||||
|
||||
// SendLogLines returns the stored lines of the message a send-log row is
|
||||
// about, oldest first and at most n of them - the latest n when there are
|
||||
// more, since what a message did last is what its page is opened for.
|
||||
//
|
||||
// Postfix reuses a queue id once the message has left the queue, and a row
|
||||
// lives far longer than that, so the id alone does not say whose a line is.
|
||||
// A line is the row's when it is dated from shortly before the row was written
|
||||
// to lifetime after it; lifetime is how long Postfix may keep working on a
|
||||
// message, with a margin, and the caller's to state (postfix.RetryPolicy).
|
||||
func (s *Store) SendLogLines(row SendLogRow, lifetime time.Duration, n int) ([]SendLogLine, error) {
|
||||
if row.QueueID == "" || n <= 0 {
|
||||
return nil, nil
|
||||
}
|
||||
rows, err := s.db.Query(
|
||||
`SELECT logged_at, line FROM (
|
||||
SELECT id, logged_at, line FROM send_log_lines
|
||||
WHERE queue_id = ? AND logged_at >= ? AND logged_at <= ?
|
||||
ORDER BY id DESC LIMIT ?)
|
||||
ORDER BY id`,
|
||||
row.QueueID,
|
||||
row.CreatedAt.Add(-sendLogLineBefore).UTC().Format(time.RFC3339),
|
||||
row.CreatedAt.Add(lifetime).UTC().Format(time.RFC3339),
|
||||
n,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("query send_log_lines: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var out []SendLogLine
|
||||
for rows.Next() {
|
||||
var at string
|
||||
l := SendLogLine{QueueID: row.QueueID}
|
||||
if err := rows.Scan(&at, &l.Line); err != nil {
|
||||
return nil, fmt.Errorf("scan send_log_lines row: %w", err)
|
||||
}
|
||||
l.At, _ = time.Parse(time.RFC3339, at)
|
||||
out = append(out, l)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// PruneSendLogLines deletes the lines dated before cutoff whose queue id has
|
||||
// no send-log row, and returns how many went. That one rule is the whole of
|
||||
// the table's retention: it removes the lines of rows the send-log sweep has
|
||||
// deleted, and the lines of messages that never had a row - the bounces
|
||||
// Postfix generates. cutoff leaves alone the lines of a message still being
|
||||
// received, whose row the journal milter has yet to write.
|
||||
//
|
||||
// The send log is asked once per queue id, not once per line: the ids are read
|
||||
// off the index, and a message is several lines.
|
||||
func (s *Store) PruneSendLogLines(cutoff time.Time) (int64, error) {
|
||||
res, err := s.db.Exec(
|
||||
`DELETE FROM send_log_lines
|
||||
WHERE logged_at < ?
|
||||
AND queue_id IN (
|
||||
SELECT q.queue_id FROM (SELECT DISTINCT queue_id FROM send_log_lines) AS q
|
||||
WHERE NOT EXISTS (SELECT 1 FROM send_log WHERE send_log.queue_id = q.queue_id))`,
|
||||
cutoff.UTC().Format(time.RFC3339),
|
||||
)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("prune send_log_lines: %w", err)
|
||||
}
|
||||
n, _ := res.RowsAffected()
|
||||
return n, nil
|
||||
}
|
||||
@@ -0,0 +1,203 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
func sendLogLineTexts(lines []SendLogLine) []string {
|
||||
out := make([]string, len(lines))
|
||||
for i, l := range lines {
|
||||
out[i] = l.Line
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func countSendLogLines(t *testing.T, s *Store) int {
|
||||
t.Helper()
|
||||
var n int
|
||||
if err := s.db.QueryRow(`SELECT COUNT(*) FROM send_log_lines`).Scan(&n); err != nil {
|
||||
t.Fatalf("count send_log_lines: %v", err)
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// A row's page gets the lines of its queue id, oldest first, and not the lines
|
||||
// of another message.
|
||||
func TestSendLogLinesOfOneMessage(t *testing.T) {
|
||||
s := openTestStore(t)
|
||||
at := time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC)
|
||||
if n, err := s.InsertSendLogLines([]SendLogLine{
|
||||
{QueueID: "41E862C00D9E", At: at, Line: "client=app"},
|
||||
{QueueID: "5F00AA", At: at, Line: "someone else's"},
|
||||
{QueueID: "41E862C00D9E", At: at.Add(time.Second), Line: "status=sent"},
|
||||
{QueueID: "41E862C00D9E", At: at.Add(2 * time.Second), Line: "removed"},
|
||||
}, false); err != nil || n != 4 {
|
||||
t.Fatalf("insert: wrote %d, %v", n, err)
|
||||
}
|
||||
row := SendLogRow{QueueID: "41E862C00D9E", CreatedAt: at}
|
||||
|
||||
lines, err := s.SendLogLines(row, 6*24*time.Hour, 200)
|
||||
if err != nil {
|
||||
t.Fatalf("SendLogLines: %v", err)
|
||||
}
|
||||
if got := strings.Join(sendLogLineTexts(lines), "|"); got != "client=app|status=sent|removed" {
|
||||
t.Errorf("lines = %q", got)
|
||||
}
|
||||
if len(lines) > 0 && !lines[0].At.Equal(at) {
|
||||
t.Errorf("first line dated %v, want %v", lines[0].At, at)
|
||||
}
|
||||
|
||||
// More than the page shows: the latest are kept, still oldest first.
|
||||
lines, err = s.SendLogLines(row, 6*24*time.Hour, 2)
|
||||
if err != nil {
|
||||
t.Fatalf("SendLogLines: %v", err)
|
||||
}
|
||||
if got := strings.Join(sendLogLineTexts(lines), "|"); got != "status=sent|removed" {
|
||||
t.Errorf("latest two = %q", got)
|
||||
}
|
||||
|
||||
// A row the milter refused has no queue id and so no lines - not every line.
|
||||
lines, err = s.SendLogLines(SendLogRow{CreatedAt: at}, 6*24*time.Hour, 200)
|
||||
if err != nil || len(lines) != 0 {
|
||||
t.Errorf("lines of a row without a queue id = %v, %v", lines, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Postfix reuses a queue id once the message has left the queue. The lines of
|
||||
// a later message under the same id are not the earlier row's, nor the other
|
||||
// way round.
|
||||
func TestSendLogLinesBoundedByTheRowsLifetime(t *testing.T) {
|
||||
s := openTestStore(t)
|
||||
first := time.Date(2026, 9, 1, 9, 0, 0, 0, time.UTC)
|
||||
second := first.AddDate(0, 0, 30)
|
||||
lifetime := 6 * 24 * time.Hour
|
||||
if _, err := s.InsertSendLogLines([]SendLogLine{
|
||||
{QueueID: "ABC123", At: first.Add(-30 * time.Minute), Line: "first: client"},
|
||||
{QueueID: "ABC123", At: first.Add(5 * 24 * time.Hour), Line: "first: expired"},
|
||||
{QueueID: "ABC123", At: second.Add(-2 * time.Hour), Line: "between the two"},
|
||||
{QueueID: "ABC123", At: second, Line: "second: sent"},
|
||||
}, false); err != nil {
|
||||
t.Fatalf("insert: %v", err)
|
||||
}
|
||||
|
||||
lines, err := s.SendLogLines(SendLogRow{QueueID: "ABC123", CreatedAt: first}, lifetime, 200)
|
||||
if err != nil {
|
||||
t.Fatalf("SendLogLines: %v", err)
|
||||
}
|
||||
if got := strings.Join(sendLogLineTexts(lines), "|"); got != "first: client|first: expired" {
|
||||
t.Errorf("the first message's lines = %q", got)
|
||||
}
|
||||
lines, err = s.SendLogLines(SendLogRow{QueueID: "ABC123", CreatedAt: second}, lifetime, 200)
|
||||
if err != nil {
|
||||
t.Fatalf("SendLogLines: %v", err)
|
||||
}
|
||||
if got := strings.Join(sendLogLineTexts(lines), "|"); got != "second: sent" {
|
||||
t.Errorf("the second message's lines = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// After an unclean stop the tailer reads some lines twice; the second reading
|
||||
// writes nothing, also when the line comes twice in one batch. The same text
|
||||
// under another queue id is another line. Without once, what is given is
|
||||
// written.
|
||||
func TestInsertSendLogLinesOnce(t *testing.T) {
|
||||
s := openTestStore(t)
|
||||
l := SendLogLine{QueueID: "ABC123", At: time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC), Line: "Oct 12 09:00:00 mail postfix/qmgr[7]: ABC123: removed"}
|
||||
other := l
|
||||
other.QueueID = "DEF456"
|
||||
|
||||
if n, err := s.InsertSendLogLines([]SendLogLine{l, l, other}, true); err != nil || n != 2 {
|
||||
t.Errorf("first reading wrote %d, %v; want 2", n, err)
|
||||
}
|
||||
if n, err := s.InsertSendLogLines([]SendLogLine{l, other}, true); err != nil || n != 0 {
|
||||
t.Errorf("second reading wrote %d, %v; want 0", n, err)
|
||||
}
|
||||
if n := countSendLogLines(t, s); n != 2 {
|
||||
t.Errorf("%d lines stored, want 2", n)
|
||||
}
|
||||
if n, err := s.InsertSendLogLines([]SendLogLine{l}, false); err != nil || n != 1 {
|
||||
t.Errorf("a line as it is written: wrote %d, %v; want 1", n, err)
|
||||
}
|
||||
if n, err := s.InsertSendLogLines(nil, true); err != nil || n != 0 {
|
||||
t.Errorf("nothing to write: %d, %v", n, err)
|
||||
}
|
||||
}
|
||||
|
||||
// The far end writes the reply a delivery line quotes; what is kept of a line
|
||||
// has a ceiling, not set through the middle of a character, and a line read
|
||||
// twice is still recognised past it.
|
||||
func TestSendLogLineIsCut(t *testing.T) {
|
||||
s := openTestStore(t)
|
||||
at := time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC)
|
||||
// Two bytes a character, starting at an odd offset: the ceiling falls inside one.
|
||||
l := SendLogLine{QueueID: "ABC123", At: at, Line: "x" + strings.Repeat("é", SendLogLineMax)}
|
||||
if _, err := s.InsertSendLogLines([]SendLogLine{l}, false); err != nil {
|
||||
t.Fatalf("insert: %v", err)
|
||||
}
|
||||
if n, err := s.InsertSendLogLines([]SendLogLine{l}, true); err != nil || n != 0 {
|
||||
t.Errorf("the same long line again: wrote %d, %v", n, err)
|
||||
}
|
||||
lines, err := s.SendLogLines(SendLogRow{QueueID: "ABC123", CreatedAt: at}, time.Hour, 10)
|
||||
if err != nil || len(lines) != 1 {
|
||||
t.Fatalf("lines = %d, %v", len(lines), err)
|
||||
}
|
||||
if got := lines[0].Line; len(got) != SendLogLineMax-1 || !utf8.ValidString(got) {
|
||||
t.Errorf("kept %d bytes (valid UTF-8: %v), want %d whole characters", len(got), utf8.ValidString(got), SendLogLineMax-1)
|
||||
}
|
||||
}
|
||||
|
||||
// The prune is the table's retention: lines go when no send-log row carries
|
||||
// their queue id - the row was pruned, or there never was one - and a message
|
||||
// still being received is left alone.
|
||||
func TestPruneSendLogLines(t *testing.T) {
|
||||
s := openTestStore(t)
|
||||
now := time.Date(2026, 10, 12, 12, 0, 0, 0, time.UTC)
|
||||
old := now.Add(-3 * time.Hour)
|
||||
|
||||
if err := s.InsertQueued(SendLogEntry{QueueID: "KEPT01", Domain: "example.com", To: "a@example.net"}); err != nil {
|
||||
t.Fatalf("insert row: %v", err)
|
||||
}
|
||||
// A row the milter refused has no queue id; it keeps no line alive.
|
||||
if err := s.InsertRejected(SendLogEntry{Domain: "example.com", From: "a@example.com"}); err != nil {
|
||||
t.Fatalf("insert rejected row: %v", err)
|
||||
}
|
||||
if _, err := s.InsertSendLogLines([]SendLogLine{
|
||||
{QueueID: "KEPT01", At: old, Line: "has a row"},
|
||||
{QueueID: "INBOUND1", At: old, Line: "never had a row"},
|
||||
{QueueID: "ARRIVING", At: now.Add(-time.Minute), Line: "its row is not written yet"},
|
||||
}, false); err != nil {
|
||||
t.Fatalf("insert: %v", err)
|
||||
}
|
||||
|
||||
n, err := s.PruneSendLogLines(now.Add(-time.Hour))
|
||||
if err != nil || n != 1 {
|
||||
t.Fatalf("prune removed %d, %v; want 1", n, err)
|
||||
}
|
||||
var left []string
|
||||
rows, err := s.db.Query(`SELECT queue_id FROM send_log_lines ORDER BY id`)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for rows.Next() {
|
||||
var id string
|
||||
if err := rows.Scan(&id); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
left = append(left, id)
|
||||
}
|
||||
rows.Close()
|
||||
if got := strings.Join(left, " "); got != "KEPT01 ARRIVING" {
|
||||
t.Errorf("left after the prune: %q", got)
|
||||
}
|
||||
|
||||
// The send-log sweep deletes the row; the next prune takes its lines.
|
||||
if _, err := s.DeleteSendLogBefore(time.Now().UTC().Add(time.Hour)); err != nil {
|
||||
t.Fatalf("delete rows: %v", err)
|
||||
}
|
||||
if n, err := s.PruneSendLogLines(now.Add(-time.Hour)); err != nil || n != 1 {
|
||||
t.Errorf("prune after the row went removed %d, %v; want 1", n, err)
|
||||
}
|
||||
}
|
||||
@@ -77,6 +77,72 @@ func (s *Store) EnsureSendLogRetentionDays(envDefault int) error {
|
||||
return s.SetSendLogRetentionDays(sendLogRetentionFallback(envDefault))
|
||||
}
|
||||
|
||||
const (
|
||||
// InboundSpamLogRetentionDaysKey is the settings table key for the
|
||||
// inbound antispam journal's retention. Same range as the send log's.
|
||||
InboundSpamLogRetentionDaysKey = "inbound_spam_log_retention_days"
|
||||
InboundSpamLogRetentionDaysMin = 7
|
||||
InboundSpamLogRetentionDaysMax = 365
|
||||
// InboundSpamLogRetentionDaysDefault matches
|
||||
// INBOUND_SPAM_LOG_RETENTION_DAYS when unset.
|
||||
InboundSpamLogRetentionDaysDefault = 90
|
||||
)
|
||||
|
||||
// ErrInboundSpamLogRetentionDaysOutOfRange is returned when the inbound
|
||||
// journal's retention is outside the allowed panel range.
|
||||
var ErrInboundSpamLogRetentionDaysOutOfRange = errors.New("inbound journal retention days out of range")
|
||||
|
||||
// ValidateInboundSpamLogRetentionDays checks the panel-allowed window.
|
||||
func ValidateInboundSpamLogRetentionDays(days int) error {
|
||||
if days < InboundSpamLogRetentionDaysMin || days > InboundSpamLogRetentionDaysMax {
|
||||
return fmt.Errorf("%w: must be between %d and %d days", ErrInboundSpamLogRetentionDaysOutOfRange, InboundSpamLogRetentionDaysMin, InboundSpamLogRetentionDaysMax)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func inboundSpamLogRetentionFallback(envDefault int) int {
|
||||
if ValidateInboundSpamLogRetentionDays(envDefault) == nil {
|
||||
return envDefault
|
||||
}
|
||||
return InboundSpamLogRetentionDaysDefault
|
||||
}
|
||||
|
||||
// GetInboundSpamLogRetentionDays returns the effective retention window of
|
||||
// the inbound journal. When the setting is missing or invalid, envDefault is
|
||||
// used (INBOUND_SPAM_LOG_RETENTION_DAYS).
|
||||
func (s *Store) GetInboundSpamLogRetentionDays(envDefault int) (int, error) {
|
||||
raw, err := s.GetSetting(InboundSpamLogRetentionDaysKey)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
days, err := strconv.Atoi(raw)
|
||||
if err != nil || ValidateInboundSpamLogRetentionDays(days) != nil {
|
||||
return inboundSpamLogRetentionFallback(envDefault), nil
|
||||
}
|
||||
return days, nil
|
||||
}
|
||||
|
||||
// SetInboundSpamLogRetentionDays persists the inbound journal's retention.
|
||||
func (s *Store) SetInboundSpamLogRetentionDays(days int) error {
|
||||
if err := ValidateInboundSpamLogRetentionDays(days); err != nil {
|
||||
return err
|
||||
}
|
||||
return s.SetSetting(InboundSpamLogRetentionDaysKey, strconv.Itoa(days))
|
||||
}
|
||||
|
||||
// EnsureInboundSpamLogRetentionDays seeds the setting from envDefault when it
|
||||
// has never been written.
|
||||
func (s *Store) EnsureInboundSpamLogRetentionDays(envDefault int) error {
|
||||
raw, err := s.GetSetting(InboundSpamLogRetentionDaysKey)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if raw != "" {
|
||||
return nil
|
||||
}
|
||||
return s.SetInboundSpamLogRetentionDays(inboundSpamLogRetentionFallback(envDefault))
|
||||
}
|
||||
|
||||
// GetSetting returns a settings value or empty string when missing.
|
||||
func (s *Store) GetSetting(key string) (string, error) {
|
||||
var value string
|
||||
|
||||
@@ -43,12 +43,15 @@ type User struct {
|
||||
// are then ignored.
|
||||
AllDomains bool
|
||||
AllInboundDomains bool
|
||||
CreatedAt time.Time
|
||||
DomainIDs []int64
|
||||
InboundDomainIDs []int64
|
||||
// Notify is the user's own switch for notification e-mails (Account). It
|
||||
// is on for a new user and does nothing without an e-mail.
|
||||
Notify bool
|
||||
CreatedAt time.Time
|
||||
DomainIDs []int64
|
||||
InboundDomainIDs []int64
|
||||
}
|
||||
|
||||
const userColumns = "id, username, password_hash, role, email, all_domains, all_inbound_domains, created_at"
|
||||
const userColumns = "id, username, password_hash, role, email, all_domains, all_inbound_domains, notify, created_at"
|
||||
|
||||
func scanUser(r scanRow) (User, error) {
|
||||
var (
|
||||
@@ -56,7 +59,7 @@ func scanUser(r scanRow) (User, error) {
|
||||
createdAt string
|
||||
)
|
||||
err := r.Scan(&u.ID, &u.Username, &u.PasswordHash, &u.Role, &u.Email,
|
||||
&u.AllDomains, &u.AllInboundDomains, &createdAt)
|
||||
&u.AllDomains, &u.AllInboundDomains, &u.Notify, &createdAt)
|
||||
if err != nil {
|
||||
return User{}, err
|
||||
}
|
||||
|
||||
@@ -56,6 +56,9 @@ var routes = struct{ always, inbound, dmarc []string }{
|
||||
"POST /inbound/domains/{id}/upstream",
|
||||
"POST /inbound/domains/{id}/recipients",
|
||||
"GET /inbound/domains/{id}/delete", "POST /inbound/domains/{id}/delete",
|
||||
"GET /inbound/log",
|
||||
"GET /inbound/filter-lists", "POST /inbound/filter-lists",
|
||||
"POST /inbound/filter-lists/{id}/delete",
|
||||
},
|
||||
dmarc: []string{
|
||||
"GET /outbound/dmarc", "GET /outbound/dmarc/domains/{id}", "GET /outbound/dmarc/reports/{id}",
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"net"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// fakeZone is a DNS server on the loopback that answers from a table, so that a
|
||||
// page can be rendered against records that are published, wrong or absent
|
||||
// without leaving the machine. Names are lower case without the trailing dot;
|
||||
// a name the zone does not hold is NXDOMAIN, which is how a record that is not
|
||||
// published looks to the checker.
|
||||
type fakeZone struct {
|
||||
mu sync.Mutex
|
||||
txt map[string][]string
|
||||
addr map[string]string // name -> IPv4 address
|
||||
}
|
||||
|
||||
// newFakeZone starts the server for the length of the test and returns the
|
||||
// resolver address to give dnscheck.New.
|
||||
func newFakeZone(t *testing.T) (*fakeZone, string) {
|
||||
t.Helper()
|
||||
conn, err := net.ListenPacket("udp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatalf("fake DNS: listen: %v", err)
|
||||
}
|
||||
z := &fakeZone{txt: map[string][]string{}, addr: map[string]string{}}
|
||||
t.Cleanup(func() { _ = conn.Close() })
|
||||
go func() {
|
||||
buf := make([]byte, 1500)
|
||||
for {
|
||||
n, from, err := conn.ReadFrom(buf)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if resp := z.answer(buf[:n]); resp != nil {
|
||||
_, _ = conn.WriteTo(resp, from)
|
||||
}
|
||||
}
|
||||
}()
|
||||
return z, conn.LocalAddr().String()
|
||||
}
|
||||
|
||||
// publishTXT sets the TXT record of a name; publishA its IPv4 address.
|
||||
func (z *fakeZone) publishTXT(name, value string) {
|
||||
z.mu.Lock()
|
||||
defer z.mu.Unlock()
|
||||
z.txt[strings.ToLower(name)] = append(z.txt[strings.ToLower(name)], value)
|
||||
}
|
||||
|
||||
func (z *fakeZone) publishA(name, ip string) {
|
||||
z.mu.Lock()
|
||||
defer z.mu.Unlock()
|
||||
z.addr[strings.ToLower(name)] = ip
|
||||
}
|
||||
|
||||
// answer builds the reply to one query: the question is copied back, then the
|
||||
// records of the asked type if the zone has any. A name with nothing of the
|
||||
// asked type but known under another type is an empty answer (NOERROR), an
|
||||
// unknown name is NXDOMAIN.
|
||||
func (z *fakeZone) answer(q []byte) []byte {
|
||||
if len(q) < 12 || binary.BigEndian.Uint16(q[4:6]) != 1 {
|
||||
return nil
|
||||
}
|
||||
i, labels := 12, []string{}
|
||||
for {
|
||||
if i >= len(q) {
|
||||
return nil
|
||||
}
|
||||
l := int(q[i])
|
||||
if l == 0 {
|
||||
i++
|
||||
break
|
||||
}
|
||||
if i+1+l > len(q) {
|
||||
return nil
|
||||
}
|
||||
labels = append(labels, string(q[i+1:i+1+l]))
|
||||
i += 1 + l
|
||||
}
|
||||
if i+4 > len(q) {
|
||||
return nil
|
||||
}
|
||||
name, qtype := strings.ToLower(strings.Join(labels, ".")), binary.BigEndian.Uint16(q[i:i+2])
|
||||
question := q[12 : i+4]
|
||||
|
||||
z.mu.Lock()
|
||||
txt, ip := z.txt[name], z.addr[name]
|
||||
z.mu.Unlock()
|
||||
|
||||
var answers [][]byte
|
||||
switch {
|
||||
case qtype == 16:
|
||||
for _, v := range txt {
|
||||
var rdata []byte
|
||||
for len(v) > 255 {
|
||||
rdata = append(rdata, 255)
|
||||
rdata = append(rdata, v[:255]...)
|
||||
v = v[255:]
|
||||
}
|
||||
rdata = append(rdata, byte(len(v)))
|
||||
rdata = append(rdata, v...)
|
||||
answers = append(answers, record(16, rdata))
|
||||
}
|
||||
case qtype == 1 && ip != "":
|
||||
answers = append(answers, record(1, net.ParseIP(ip).To4()))
|
||||
}
|
||||
rcode := uint16(0)
|
||||
if len(txt) == 0 && ip == "" {
|
||||
rcode = 3
|
||||
}
|
||||
resp := make([]byte, 12, 512)
|
||||
copy(resp, q[:2])
|
||||
binary.BigEndian.PutUint16(resp[2:], 0x8180|rcode)
|
||||
binary.BigEndian.PutUint16(resp[4:], 1)
|
||||
binary.BigEndian.PutUint16(resp[6:], uint16(len(answers)))
|
||||
resp = append(resp, question...)
|
||||
for _, a := range answers {
|
||||
resp = append(resp, a...)
|
||||
}
|
||||
return resp
|
||||
}
|
||||
|
||||
// record is one answer record named by the question (a pointer to offset 12),
|
||||
// class IN, with a TTL of a minute.
|
||||
func record(rtype uint16, rdata []byte) []byte {
|
||||
r := []byte{0xC0, 0x0C, 0, 0, 0, 1, 0, 0, 0, 60, 0, 0}
|
||||
binary.BigEndian.PutUint16(r[2:], rtype)
|
||||
binary.BigEndian.PutUint16(r[10:], uint16(len(rdata)))
|
||||
return append(r, rdata...)
|
||||
}
|
||||
@@ -50,6 +50,9 @@ type Config struct {
|
||||
// SendLogRetentionEnvDefault is SEND_LOG_RETENTION_DAYS at panel start; used
|
||||
// as bootstrap and fallback when the settings row is missing or invalid.
|
||||
SendLogRetentionEnvDefault int
|
||||
// InboundSpamLogRetentionEnvDefault is INBOUND_SPAM_LOG_RETENTION_DAYS at
|
||||
// panel start, the same bootstrap and fallback for the inbound journal.
|
||||
InboundSpamLogRetentionEnvDefault int
|
||||
}
|
||||
|
||||
// Handlers holds dependencies for authenticated panel routes.
|
||||
@@ -58,6 +61,7 @@ type Handlers struct {
|
||||
domains *domain.Service
|
||||
apps *app.Service
|
||||
inbound *inbound.Service
|
||||
lists *inbound.Lists
|
||||
dmarc *dmarc.Service
|
||||
cfg Config
|
||||
view *view.Engine
|
||||
@@ -72,6 +76,7 @@ func New(
|
||||
domains *domain.Service,
|
||||
apps *app.Service,
|
||||
inboundSvc *inbound.Service,
|
||||
lists *inbound.Lists,
|
||||
dmarcSvc *dmarc.Service,
|
||||
cfg Config,
|
||||
v *view.Engine,
|
||||
@@ -84,6 +89,7 @@ func New(
|
||||
domains: domains,
|
||||
apps: apps,
|
||||
inbound: inboundSvc,
|
||||
lists: lists,
|
||||
dmarc: dmarcSvc,
|
||||
cfg: cfg,
|
||||
view: v,
|
||||
|
||||
@@ -26,6 +26,7 @@ type accountForm struct {
|
||||
Err string
|
||||
Username string
|
||||
Email string
|
||||
Notify bool
|
||||
}
|
||||
|
||||
func (h *Handlers) accountUser(w http.ResponseWriter, r *http.Request) (store.User, bool) {
|
||||
@@ -44,7 +45,7 @@ func (h *Handlers) accountUser(w http.ResponseWriter, r *http.Request) (store.Us
|
||||
}
|
||||
|
||||
func formOf(u store.User) accountForm {
|
||||
return accountForm{Username: u.Username, Email: u.Email}
|
||||
return accountForm{Username: u.Username, Email: u.Email, Notify: u.Notify}
|
||||
}
|
||||
|
||||
// HandleAccount shows the Account page.
|
||||
@@ -58,7 +59,7 @@ func (h *Handlers) HandleAccount(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
func (h *Handlers) renderAccount(w http.ResponseWriter, r *http.Request, status int, u store.User, f accountForm) {
|
||||
page := view.NewAccount(h.shellMeta(r), string(u.Role))
|
||||
page.Username, page.Email = f.Username, f.Email
|
||||
page.Username, page.Email, page.Notify = f.Username, f.Email, f.Notify
|
||||
page.WithResult(accountFlash(r), f.Err)
|
||||
h.view.Render(w, status, "account", page)
|
||||
}
|
||||
@@ -90,6 +91,8 @@ func (h *Handlers) HandleAccountProfile(w http.ResponseWriter, r *http.Request)
|
||||
}
|
||||
f.Username = strings.TrimSpace(r.PostFormValue("username"))
|
||||
f.Email = strings.TrimSpace(r.PostFormValue("email"))
|
||||
// An unticked checkbox is not sent at all: absent is off.
|
||||
f.Notify = r.PostFormValue("notify") != ""
|
||||
if f.Username == "" {
|
||||
f.Username = u.Username
|
||||
}
|
||||
@@ -112,7 +115,14 @@ func (h *Handlers) HandleAccountProfile(w http.ResponseWriter, r *http.Request)
|
||||
fail(http.StatusInternalServerError, "Could not save the profile. Please check the logs and try again.")
|
||||
return
|
||||
}
|
||||
logf("panel: user %d profile updated (username: %t, e-mail: %t)", u.ID, f.Username != u.Username, f.Email != u.Email)
|
||||
if f.Notify != u.Notify {
|
||||
if err := h.store.SetUserNotify(u.ID, f.Notify); err != nil {
|
||||
logf("panel: account: set notifications of user %d: %v", u.ID, err)
|
||||
fail(http.StatusInternalServerError, "Could not save the profile. Please check the logs and try again.")
|
||||
return
|
||||
}
|
||||
}
|
||||
logf("panel: user %d profile updated (username: %t, e-mail: %t, notifications: %t)", u.ID, f.Username != u.Username, f.Email != u.Email, f.Notify)
|
||||
http.Redirect(w, r, "/account?done=profile", http.StatusSeeOther)
|
||||
}
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user