inbound-antispam-panel: Filter lists, kind column, matches, note on a sheet

Follows the design commit before it (owner's words quoted there).

The lists' tables are Value, Kind, Matched and Remove. The kind tag
opens the entry's sheet: note, day added, how often it decided a
message in the days the log keeps, and the last match.

Matches are read off the journal on each view, nothing is stored
(inbound.Lists.Usage, store.ListSpamListHits). A row counts for an
entry when the list decided the message - a rejected row whose reply is
the deny rule's message, an accepted row whose verdict carries the
allow rule's symbol for that kind - and the entry fits the row's sender
or client address the way rspamd matches it: an address whole, a domain
as the sender's domain, a network containing the client (an
IPv4-mapped address as IPv4). A failure to count shows the lists with
"never" and logs why.

For that the journal milter keeps the lists' own symbols ahead of its
six-symbol cap: rspamd writes symbols heaviest first, so an allow
entry's -100 came last and was cut behind any six others.

The plan gains "Filter list usage": what counting from the journal
implies, and - the owner asked for it in writing - the conditions under
which a counter kept with each entry becomes worth a migration (the
journal cut by its row cap and not by age; a short retention; counts
that must outlive the journal; a slow page). multimap.conf says which
of its texts SelfPost reads back.

Tests: Usage over every kind on both lists and the rows that are
nobody's; the milter's symbol order; the store query; the page and its
sheets; a handler test from journal rows to the numbers on the page.

Seen: the fixture page with a sheet open, headless Edge at 1360 px -
the same as the accepted drawing.

Checked on the test server, uncommitted tree: go vet ./... ok,
go test ./... ok (22 packages), make e2e ok.

Not seen in a real run: a count above zero. It needs an entry added in
the panel, which I cannot sign in to, and then mail that the list
decides; in particular the deny rule's message arriving in mail.log as
the reply of a milter-reject line is what the trial with rspamc showed,
not yet what Postfix logged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
mix
2026-10-12 01:08:26 +03:00
co-authored by Claude Opus 5.5
parent e0ccff257d
commit 7192c3861e
18 changed files with 501 additions and 45 deletions
+5 -1
View File
@@ -52,7 +52,11 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); version
*Deny sender* on accepted mail, *Allow sender* on mail the filter refused. *Deny sender* on accepted mail, *Allow sender* on mail the filter refused.
- **Inbound › Filter lists** (global role): add and remove allow and deny - **Inbound › Filter lists** (global role): add and remove allow and deny
entries; the page says where the lists go and whether the filter is on to entries; the page says where the lists go and whether the filter is on to
read them. read them. Each entry shows how often it decided a message - *Matched*, or
*never* - and its kind opens a sheet with the note, the day it was added and
the last match, so that entries nothing matches any more can be found and
removed. The count is read from the inbound log and reaches back as far as
the log does.
- **Server › Settings**: *Keep inbound log rows, days*, shown with the inbound - **Server › Settings**: *Keep inbound log rows, days*, shown with the inbound
relay on. relay on.
- Help: *Inbound log* and *Filter lists*. - Help: *Inbound log* and *Filter lists*.
+4
View File
@@ -11,6 +11,10 @@
# by the deny list still overrides. # by the deny list still overrides.
# #
# A domain entry is that domain, not its subdomains. # A domain entry is that domain, not its subdomains.
#
# SelfPost counts how often an entry decided a message by what these rules
# leave behind: the deny message below and the names of the allow symbols.
# Change either and Inbound > Filter lists shows "never" for every entry.
SELFPOST_DENY_IP { SELFPOST_DENY_IP {
type = "ip"; type = "ip";
+7
View File
@@ -114,6 +114,13 @@ smtp/inet` removes Debian's stock smtpd). Outbound delivery still uses the
prefilters rspamd lets the one that ran first decide, and deny then won or prefilters rspamd lets the one that ran first decide, and deny then won or
lost by rule order (seen on rspamd 3.11). A refusal by the deny list lost by rule order (seen on rspamd 3.11). A refusal by the deny list
overrides any score, so deny wins whichever entries match. overrides any score, so deny wins whichever entries match.
How often an entry decided a message is not stored: `Lists.Usage` reads it
off the journal on each view of Filter lists - the rows a list decided (the
deny rule's reply, the allow rule's symbol, which the journal milter keeps
ahead of its six-symbol cap) matched against the entries as rspamd matches
them. When that stops being good enough is written in
[plans/inbound-antispam-panel.md](plans/inbound-antispam-panel.md) § Filter
list usage.
**Inbound relay** (`INBOUND_RELAY_ENABLE=true`): accepts only **Inbound relay** (`INBOUND_RELAY_ENABLE=true`): accepts only
`relay_domains` + `relay_recipient_maps` (`reject_unauth_destination`, `relay_domains` + `relay_recipient_maps` (`reject_unauth_destination`,
`reject_unlisted_recipient`). Maps under `/data/postfix/` (`relay_domains`, `reject_unlisted_recipient`). Maps under `/data/postfix/` (`relay_domains`,
+7 -1
View File
@@ -876,7 +876,13 @@ anyone on the host; they hold addresses, not secrets.
In the panel the lists are **Inbound › Filter lists** (global role), and what In the panel the lists are **Inbound › Filter lists** (global role), and what
port 25 decided about each message is **Inbound › Log**: accepted mail with the port 25 decided about each message is **Inbound › Log**: accepted mail with the
filter's verdict, refused mail with the reply it was given. A row of the log filter's verdict, refused mail with the reply it was given. A row of the log
leads to the lists with its sender filled in. A user with the *domain* role leads to the lists with its sender filled in. Beside each entry of a list is
how often it decided a message, counted from the log - so over the days the
log keeps - and *never* marks an entry that may have outlived its use. The
count depends on two things in
[multimap.conf](../deploy/antispam/rspamd/multimap.conf): the message of the
deny rules and the names of the allow symbols; change them and every entry
reads *never*. A user with the *domain* role
sees the log of the inbound domains assigned to them, and not the lists. How sees the log of the inbound domains assigned to them, and not the lists. How
long the log keeps a row is set under **Server › Settings**. long the log keeps a row is set under **Server › Settings**.
+43 -2
View File
@@ -97,8 +97,9 @@ changes to it: the sender column is headed *From*; *Why* is not a column but
a sheet over the page, opened by the decision tag, holding a centred table of a sheet over the page, opened by the decision tag, holding a centred table of
who decided, the verdict and symbols or the reply, and the subject whole who decided, the verdict and symbols or the reply, and the subject whole
(kept on the sheet at his word: the table clips it); a message Postfix itself (kept on the sheet at his word: the table clips it); a message Postfix itself
refused offers no *Deny sender*. Inbound › Filter lists is built and not yet refused offers no *Deny sender*. Inbound › Filter lists was redrawn at his word
accepted. the same day - kind in a column of its own, the note on the entry's sheet,
matches counted (§ Filter list usage) - and awaits his look at the result.
### Retention ### Retention
@@ -126,6 +127,46 @@ Mirror send-log retention:
List change → atomic map rewrite (same pattern as Postfix maps). No reload: List change → atomic map rewrite (same pattern as Postfix maps). No reload:
rspamd rereads a changed map file by itself (about ten seconds on 3.11). rspamd rereads a changed map file by itself (about ten seconds on 3.11).
### Filter list usage (owner, 2026-10-12)
Each entry shows how often it decided a message (*Matched*, or *never*) and,
on its sheet, when it last did - so that entries nothing matches any more can
be removed. **The count is read off the journal on every view of the page;
nothing is stored for it.** A journal row counts for an entry when the list
decided the message - a rejected row whose reply is the deny rule's message,
an accepted row whose verdict carries the allow rule's symbol
(`SELFPOST_ALLOW_IP` / `_FROM` / `_DOMAIN`, which the journal milter always
keeps) - and the entry fits the row's sender or client address the way rspamd
matches it (`inbound.Lists.Usage`).
What follows from counting this way, and is accepted:
- the count reaches back as far as the journal does: its retention (90 days by
default), which is what the sheet says;
- a row counts for every fitting entry of the deciding list, since the journal
does not say which of two the filter met first;
- a message refused at end-of-message is one row whatever its recipient count,
so a count is of messages, roughly.
**When to move to a counter kept with each entry** (a migration adding hits
and last-hit to `inbound_spam_list`, bumped where the journal row is written;
the owner asked for the condition to be written down). Any one of:
1. **The journal is cut by its 10 000-row cap and not by age.** Then it holds
fewer days than its retention, the sheet's "in the last 90 days" is no
longer true, and *never* stops being evidence that an entry is stale. The
sign: the log tailer's prune line keeps reporting rows removed while the
oldest row of Inbound › Log is younger than the retention.
2. **The retention is set short** (towards its minimum of 7 days) for reasons
of its own, privacy or disk. A week without a match does not make an entry
stale.
3. **Counts are wanted that outlive the journal** - since the entry was added.
4. **The page is slow.** A view reads every journal row a list decided and
tries it against every entry. With the cap that is at most 10 000 rows
times the number of entries; past a few hundred entries, measure it.
Until then no migration is spent on it.
### Activation ### Activation
| Condition | Behaviour | | Condition | Behaviour |
+113
View File
@@ -21,6 +21,7 @@ type ListStore interface {
AddSpamListEntry(list, kind, value, note string) (store.SpamListEntry, error) AddSpamListEntry(list, kind, value, note string) (store.SpamListEntry, error)
ListSpamListEntries() ([]store.SpamListEntry, error) ListSpamListEntries() ([]store.SpamListEntry, error)
DeleteSpamListEntry(id int64) error DeleteSpamListEntry(id int64) error
ListSpamListHits(marks ...string) ([]store.SpamListHit, error)
} }
// Lists owns the allow and deny lists of the external spam filter: the rows in // Lists owns the allow and deny lists of the external spam filter: the rows in
@@ -236,3 +237,115 @@ func normalizeNote(raw string) (string, error) {
} }
return note, nil return note, nil
} }
// What a list leaves in the journal when it decides a message - the two ends
// of deploy/antispam/rspamd/multimap.conf this package depends on. A deny rule
// refuses with a message, which the log tailer files as the reply of a
// rejected row. An allow rule adds a symbol named after the kind of entry,
// which the journal milter keeps in the verdict of an accepted row.
const (
denyReplyMark = "this server's deny list"
allowMark = "SELFPOST_ALLOW_"
)
var allowSymbol = map[string]string{
store.SpamEntryIP: "SELFPOST_ALLOW_IP(",
store.SpamEntryAddress: "SELFPOST_ALLOW_FROM(",
store.SpamEntryDomain: "SELFPOST_ALLOW_DOMAIN(",
}
// Usage is how often an entry decided a message, as far back as the journal
// reaches, and when it last did. The zero value is an entry that never did.
type Usage struct {
Hits int
Last time.Time
}
// Usage counts, for every entry, the journal rows it decided. Nothing is
// stored for this: the journal already says which list decided a message and
// who the message was from, and an entry is matched against that the way the
// filter matches it - an address whole, a domain as the sender's domain, an
// address or network against the client's. So the count reaches back as far
// as the journal does (its retention, or less when it is full), and a row
// counts for every entry of the deciding list that fits it: the journal does
// not say which of two fitting entries the filter met first.
//
// A counter kept with each entry would not depend on the journal; the plan
// (docs/plans/inbound-antispam-panel.md, Filter list usage) says when that is
// worth a migration.
func (l *Lists) Usage() (map[int64]Usage, error) {
entries, err := l.store.ListSpamListEntries()
if err != nil {
return nil, err
}
out := make(map[int64]Usage, len(entries))
if len(entries) == 0 {
return out, nil
}
hits, err := l.store.ListSpamListHits(denyReplyMark, allowMark)
if err != nil {
return nil, err
}
type matcher struct {
id int64
list string
kind string
value string
prefix netip.Prefix
}
matchers := make([]matcher, 0, len(entries))
for _, e := range entries {
m := matcher{id: e.ID, list: e.List, kind: e.Kind, value: e.Value}
if e.Kind == store.SpamEntryIP {
if p, err := netip.ParsePrefix(e.Value); err == nil {
m.prefix = p
} else if a, err := netip.ParseAddr(e.Value); err == nil {
m.prefix = netip.PrefixFrom(a, a.BitLen())
} else {
continue // not an entry this package wrote
}
}
matchers = append(matchers, m)
}
for _, h := range hits {
from := strings.ToLower(h.From)
domain := ""
if i := strings.LastIndexByte(from, '@'); i >= 0 {
domain = from[i+1:]
}
client, clientErr := netip.ParseAddr(h.ClientIP)
client = client.Unmap()
denied := h.Decision != store.SpamDecisionAccept && strings.Contains(h.Detail, denyReplyMark)
for _, m := range matchers {
switch m.list {
case store.SpamListDeny:
if !denied {
continue
}
case store.SpamListAllow:
if h.Decision != store.SpamDecisionAccept || !strings.Contains(h.Detail, allowSymbol[m.kind]) {
continue
}
}
fits := false
switch m.kind {
case store.SpamEntryAddress:
fits = from == m.value
case store.SpamEntryDomain:
fits = domain == m.value
case store.SpamEntryIP:
fits = clientErr == nil && m.prefix.Contains(client)
}
if !fits {
continue
}
u := out[m.id]
u.Hits++
if h.CreatedAt.After(u.Last) {
u.Last = h.CreatedAt
}
out[m.id] = u
}
}
return out, nil
}
+92
View File
@@ -276,3 +276,95 @@ func TestListsLastWritten(t *testing.T) {
t.Errorf("LastWritten counts a stray file: %v", got) t.Errorf("LastWritten counts a stray file: %v", got)
} }
} }
// How often an entry decided a message is read off the journal: the list that
// decided, and whether the entry fits who the message was from.
func TestListsUsage(t *testing.T) {
l, st, _ := testLists(t)
add := func(list, kind, value string) int64 {
t.Helper()
e, err := l.Add(list, kind, value, "")
if err != nil {
t.Fatalf("Add(%s, %s, %s): %v", list, kind, value, err)
}
return e.ID
}
denyDomain := add(store.SpamListDeny, store.SpamEntryDomain, "evil.example")
denyAddr := add(store.SpamListDeny, store.SpamEntryAddress, "spammer@evil.example")
denyNet := add(store.SpamListDeny, store.SpamEntryIP, "198.51.100.0/24")
denyV6 := add(store.SpamListDeny, store.SpamEntryIP, "2001:db8::/32")
allowAddr := add(store.SpamListAllow, store.SpamEntryAddress, "billing@vendor.example")
allowDomain := add(store.SpamListAllow, store.SpamEntryDomain, "vendor.example")
allowIP := add(store.SpamListAllow, store.SpamEntryIP, "203.0.113.5")
unused := add(store.SpamListAllow, store.SpamEntryDomain, "quiet.example")
const denied = "5.7.1 Refused by this server's deny list"
at := time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC)
hour := 0
row := func(from, ip, decision, engine, detail string) {
t.Helper()
hour++
err := st.InsertInboundSpam(store.InboundSpamEntry{
InboundDomain: "in.example.com", To: "a@in.example.com", From: from, ClientIP: ip,
Decision: decision, Engine: engine, Detail: detail, At: at.Add(time.Duration(hour) * time.Hour),
})
if err != nil {
t.Fatal(err)
}
}
// hour 1: the address and its domain are both on the deny list, and both fit.
row("Spammer@Evil.Example", "192.0.2.9", store.SpamDecisionReject, store.SpamEngineRspamd, denied)
// hour 2: another address of the domain.
row("other@evil.example", "192.0.2.9", store.SpamDecisionReject, store.SpamEngineRspamd, denied)
// hour 3, 4: the networks, v4 (as Postfix may log it, mapped) and v6.
row("x@elsewhere.example", "::ffff:198.51.100.77", store.SpamDecisionReject, store.SpamEngineRspamd, denied)
row("x@elsewhere.example", "2001:db8::1", store.SpamDecisionReject, store.SpamEngineRspamd, denied)
// hour 5: a subdomain is not the domain.
row("x@sub.evil.example", "192.0.2.9", store.SpamDecisionReject, store.SpamEngineRspamd, denied)
// hour 6: allowed by its address and its domain - two symbols, two entries.
row("billing@vendor.example", "192.0.2.50", store.SpamDecisionAccept, store.SpamEngineRspamd,
"no action, score -199.00 / 15.00; SELFPOST_ALLOW_DOMAIN(-100.00) SELFPOST_ALLOW_FROM(-100.00)")
// hour 7: allowed by its domain alone.
row("accounts@vendor.example", "192.0.2.50", store.SpamDecisionAccept, store.SpamEngineRspamd,
"no action, score -99.00 / 15.00; SELFPOST_ALLOW_DOMAIN(-100.00)")
// hour 8: allowed by the client address; the sender's domain is not why.
row("x@quiet.example", "203.0.113.5", store.SpamDecisionAccept, store.SpamEngineRspamd,
"no action, score -99.00 / 15.00; SELFPOST_ALLOW_IP(-100.00)")
// Nobody's: the filter's own refusal, an accept with no list symbol, a
// refusal by Postfix, and a sender who spells the mark in his own address.
row("spammer@evil.example", "192.0.2.9", store.SpamDecisionReject, store.SpamEngineRspamd, "5.7.1 Spam message rejected")
row("billing@vendor.example", "192.0.2.50", store.SpamDecisionAccept, store.SpamEngineRspamd, "no action, score 0.10 / 15.00")
row("spammer@evil.example", "198.51.100.1", store.SpamDecisionReject, store.SpamEnginePostfix, "550 5.1.1 <nobody@in.example.com>: this server's deny list")
row("billing@vendor.example", "203.0.113.5", store.SpamDecisionAccept, store.SpamEngineSelfPost, "")
usage, err := l.Usage()
if err != nil {
t.Fatalf("Usage: %v", err)
}
for name, c := range map[string]struct {
id int64
hits int
last int // hour
}{
"deny domain": {denyDomain, 2, 2}, "deny address": {denyAddr, 1, 1},
"deny network": {denyNet, 1, 3}, "deny v6 network": {denyV6, 1, 4},
"allow address": {allowAddr, 1, 6}, "allow domain": {allowDomain, 2, 7}, "allow ip": {allowIP, 1, 8},
"unused": {unused, 0, 0},
} {
got := usage[c.id]
wantLast := time.Time{}
if c.hits > 0 {
wantLast = at.Add(time.Duration(c.last) * time.Hour)
}
if got.Hits != c.hits || !got.Last.Equal(wantLast) {
t.Errorf("%s: %d hits, last %v; want %d, %v", name, got.Hits, got.Last, c.hits, wantLast)
}
}
}
func TestListsUsageWithNoEntries(t *testing.T) {
l, _, _ := testLists(t)
if usage, err := l.Usage(); err != nil || len(usage) != 0 {
t.Errorf("Usage = %v, %v", usage, err)
}
}
+20 -3
View File
@@ -142,12 +142,16 @@ func (s *inboundSession) record() {
} }
} }
// What the journal keeps of a verdict: enough to see why, on one table row. // What the journal keeps of a verdict: enough to see why, on one line.
const ( const (
verdictMaxSymbols = 6 verdictMaxSymbols = 6
verdictMaxBytes = 255 verdictMaxBytes = 255
) )
// ListSymbolPrefix starts the names of the symbols the filter lists' rules add
// (deploy/antispam/rspamd/multimap.conf): SELFPOST_ALLOW_IP, _FROM, _DOMAIN.
const ListSymbolPrefix = "SELFPOST_"
var ( var (
// "default: False [7.20 / 15.00]" - the score and the reject threshold. // "default: False [7.20 / 15.00]" - the score and the reject threshold.
rspamdScore = regexp.MustCompile(`\[(-?[0-9]+(?:\.[0-9]+)?) / (-?[0-9]+(?:\.[0-9]+)?)\]`) rspamdScore = regexp.MustCompile(`\[(-?[0-9]+(?:\.[0-9]+)?) / (-?[0-9]+(?:\.[0-9]+)?)\]`)
@@ -182,15 +186,28 @@ func rspamdVerdict(actionHeader, resultHeader string) (action, detail string) {
} }
detail = strings.Join(parts, ", ") detail = strings.Join(parts, ", ")
var symbols []string // The filter lists' own symbols come first and are kept whatever else there
// is: Filter lists counts how often an entry decided by finding them here,
// and rspamd writes symbols heaviest first, which puts an allow entry's
// -100 last, behind any six others.
var symbols, rest []string
for _, m := range rspamdSymbol.FindAllStringSubmatch(resultHeader, -1) { for _, m := range rspamdSymbol.FindAllStringSubmatch(resultHeader, -1) {
if strings.Trim(m[2], "-0.") == "" { // weight zero: informational if strings.Trim(m[2], "-0.") == "" { // weight zero: informational
continue continue
} }
symbols = append(symbols, m[1]+"("+m[2]+")") if strings.HasPrefix(m[1], ListSymbolPrefix) {
if len(symbols) < verdictMaxSymbols {
symbols = append(symbols, m[1]+"("+m[2]+")")
}
continue
}
rest = append(rest, m[1]+"("+m[2]+")")
}
for _, sym := range rest {
if len(symbols) == verdictMaxSymbols { if len(symbols) == verdictMaxSymbols {
break break
} }
symbols = append(symbols, sym)
} }
if len(symbols) > 0 { if len(symbols) > 0 {
if detail != "" { if detail != "" {
+12
View File
@@ -222,3 +222,15 @@ func TestInboundAcceptsWhenTheStoreFails(t *testing.T) {
} }
} }
} }
// rspamd writes symbols heaviest first, so an allow entry's -100 comes last.
// Filter lists counts an entry's matches by finding that symbol in the journal:
// it is kept whatever else there is.
func TestRspamdVerdictKeepsTheListSymbols(t *testing.T) {
result := "default: False [-91.00 / 15.00]; SYM_A(3.00)[]; SYM_B(2.00)[]; SYM_C(1.50)[]; SYM_D(1.00)[]; SYM_E(0.80)[]; SYM_F(0.50)[]; SYM_G(0.20)[]; ARC_NA(0.00)[]; SELFPOST_ALLOW_FROM(-100.00)[billing@vendor.example]"
_, detail := rspamdVerdict("no action", result)
want := "no action, score -91.00 / 15.00; SELFPOST_ALLOW_FROM(-100.00) SYM_A(3.00) SYM_B(2.00) SYM_C(1.50) SYM_D(1.00) SYM_E(0.80)"
if detail != want {
t.Errorf("detail = %q\nwant %q", detail, want)
}
}
+45
View File
@@ -304,3 +304,48 @@ func (s *Store) DeleteSpamListEntry(id int64) error {
} }
return nil return nil
} }
// SpamListHit is what the journal holds about one message a filter list
// decided: who it was from, which server offered it, when, and the detail that
// says which list it was.
type SpamListHit struct {
ClientIP string
From string
Decision string
Detail string
CreatedAt time.Time
}
// ListSpamListHits returns the journal rows of the filter in whose detail one
// of the marks occurs - the marks the filter lists leave (internal/inbound).
// Which entry a row belongs to is for the caller to work out; here the rows
// are only narrowed, so that a page view does not read the whole journal.
func (s *Store) ListSpamListHits(marks ...string) ([]SpamListHit, error) {
if len(marks) == 0 {
return nil, nil
}
likes := make([]string, len(marks))
args := []any{SpamEngineRspamd}
for i, m := range marks {
likes[i] = "instr(detail, ?) > 0"
args = append(args, m)
}
rows, err := s.db.Query(
`SELECT client_ip, from_addr, decision, detail, created_at FROM inbound_spam_log
WHERE engine = ? AND (`+strings.Join(likes, " OR ")+`)`, args...)
if err != nil {
return nil, fmt.Errorf("query filter list hits: %w", err)
}
defer rows.Close()
var out []SpamListHit
for rows.Next() {
var h SpamListHit
var created string
if err := rows.Scan(&h.ClientIP, &h.From, &h.Decision, &h.Detail, &created); err != nil {
return nil, fmt.Errorf("scan filter list hit: %w", err)
}
h.CreatedAt, _ = time.Parse(time.RFC3339, created)
out = append(out, h)
}
return out, rows.Err()
}
+32
View File
@@ -252,3 +252,35 @@ func TestQueryInboundSpamFilters(t *testing.T) {
t.Errorf("injected domain = %q", got) t.Errorf("injected domain = %q", got)
} }
} }
func TestListSpamListHits(t *testing.T) {
st := openTestStore(t)
add := func(engine, decision, detail string) {
t.Helper()
err := st.InsertInboundSpam(InboundSpamEntry{
InboundDomain: "in.example.com", ClientIP: "203.0.113.7", From: "a@example.net", To: "a@in.example.com",
Decision: decision, Engine: engine, Detail: detail, At: time.Date(2026, 10, 1, 9, 0, 0, 0, time.UTC),
})
if err != nil {
t.Fatal(err)
}
}
add(SpamEngineRspamd, SpamDecisionReject, "5.7.1 Refused by this server's deny list")
add(SpamEngineRspamd, SpamDecisionAccept, "no action, score -99.00 / 15.00; SELFPOST_ALLOW_IP(-100.00)")
add(SpamEngineRspamd, SpamDecisionReject, "5.7.1 Spam message rejected")
add(SpamEnginePostfix, SpamDecisionReject, "550 5.1.1 this server's deny list")
// a mark with the characters LIKE would treat as wildcards is text
add(SpamEngineRspamd, SpamDecisionAccept, "no action; SELFPOSTXALLOWX(1.00)")
hits, err := st.ListSpamListHits("this server's deny list", "SELFPOST_ALLOW_")
if err != nil || len(hits) != 2 {
t.Fatalf("hits = %+v, %v; want the two rows a list decided", hits, err)
}
h := hits[0]
if h.From != "a@example.net" || h.ClientIP != "203.0.113.7" || h.Decision != SpamDecisionReject || h.CreatedAt.Hour() != 9 {
t.Errorf("hit = %+v", h)
}
if hits, err := st.ListSpamListHits(); err != nil || hits != nil {
t.Errorf("no marks: %v, %v", hits, err)
}
}
+11 -2
View File
@@ -167,12 +167,21 @@ func (h *Handlers) renderFilterLists(w http.ResponseWriter, r *http.Request, sta
http.Error(w, "internal error", http.StatusInternalServerError) http.Error(w, "internal error", http.StatusInternalServerError)
return return
} }
// How often each entry decided a message is counted from the journal. A
// failure there is not a reason to hide the lists: they are shown without
// the counts' worth - every entry "never" - and the log says why.
usage, err := h.lists.Usage()
if err != nil {
logf("panel: filter lists: usage: %v", err)
}
on, _, _ := h.inboundFilter() on, _, _ := h.inboundFilter()
in := make([]view.FilterEntryInput, len(entries)) in := make([]view.FilterEntryInput, len(entries))
for i, e := range entries { for i, e := range entries {
in[i] = view.FilterEntryInput{ID: e.ID, List: e.List, Kind: e.Kind, Value: e.Value, Note: e.Note, Added: e.CreatedAt} in[i] = view.FilterEntryInput{ID: e.ID, List: e.List, Kind: e.Kind, Value: e.Value, Note: e.Note, Added: e.CreatedAt,
Hits: usage[e.ID].Hits, LastHit: usage[e.ID].Last}
} }
page := view.NewFilterLists(h.shellMeta(r), view.FilterListsInput{FilterOn: on, LastWritten: h.lists.LastWritten()}). page := view.NewFilterLists(h.shellMeta(r), view.FilterListsInput{
FilterOn: on, LastWritten: h.lists.LastWritten(), RetentionDays: h.inboundSpamLogRetentionDays()}).
WithEntries(in). WithEntries(in).
WithResult(flash, formErr) WithResult(flash, formErr)
// A blank list or kind leaves the form on its first choice. // A blank list or kind leaves the form on its first choice.
@@ -364,9 +364,10 @@ func TestFilterListsAddAndRemove(t *testing.T) {
has(t, "lists", page, has(t, "lists", page,
`<div class="notification is-success is-light">`, `Entry added.`, `<div class="notification is-success is-light">`, `Entry added.`,
`<h2>Deny</h2><div class="sp-end">1 entry · wins over Allow</div>`, `<h2>Allow</h2><div class="sp-end">1 entry</div>`, `<h2>Deny</h2><div class="sp-end">1 entry · wins over Allow</div>`, `<h2>Allow</h2><div class="sp-end">1 entry</div>`,
`bulk.example <span class="tag is-light">domain</span></td><td class="sp-muted">newsletter</td>`, `<td class="sp-mono sp-small">bulk.example</td><td><button type="button" class="tag is-light" popovertarget="entry-`,
`198.51.100.0/24 <span class="tag is-light">ip</span></td><td class="sp-muted">—</td>`, `<tr><th>Note</th><td>newsletter</td></tr><tr><th>Added</th><td class="sp-mono sp-small">`+time.Now().UTC().Format("2006-01-02")+`</td></tr><tr><th>Matched</th><td>never in the last 90 days</td></tr>`,
`<td class="sp-muted sp-nowrap sp-small">`+time.Now().UTC().Format("2006-01-02")+`</td>`, `<td class="sp-mono sp-small">198.51.100.0/24</td>`, `<tr><th>Note</th><td>—</td></tr>`,
`<td class="sp-muted sp-nowrap">never</td>`,
) )
lacks(t, "lists", page, `No sender is turned away`) lacks(t, "lists", page, `No sender is turned away`)
@@ -635,3 +636,39 @@ func TestHelpHasTheInboundLogAndFilterListsTopics(t *testing.T) {
has(t, "log", s.get(h.HandleInboundLog, "/inbound/log", globalPrincipal).Body.String(), `href="/help#inbound-log"`) has(t, "log", s.get(h.HandleInboundLog, "/inbound/log", globalPrincipal).Body.String(), `href="/help#inbound-log"`)
has(t, "lists", s.get(h.HandleFilterLists, "/inbound/filter-lists", globalPrincipal).Body.String(), `href="/help#filter-lists"`) has(t, "lists", s.get(h.HandleFilterLists, "/inbound/filter-lists", globalPrincipal).Body.String(), `href="/help#filter-lists"`)
} }
// An entry's count comes from the journal: a row counts for the entry that
// fits it on the list that decided it.
func TestFilterListsCountMatchesFromTheJournal(t *testing.T) {
s := newInboundLogStand(t)
for _, e := range [][3]string{{"deny", "domain", "bulk.example"}, {"allow", "address", "billing@vendor.example"}, {"deny", "ip", "203.0.113.0/24"}} {
if _, err := s.h.lists.Add(e[0], e[1], e[2], ""); err != nil {
t.Fatalf("Add %v: %v", e, err)
}
}
at := time.Date(2026, 10, 1, 9, 30, 0, 0, time.UTC)
for i, row := range []store.InboundSpamEntry{
{From: "promo@bulk.example", ClientIP: "198.51.100.9", Decision: store.SpamDecisionReject, Engine: store.SpamEngineRspamd, Detail: "5.7.1 Refused by this server's deny list"},
{From: "other@BULK.example", ClientIP: "198.51.100.9", Decision: store.SpamDecisionReject, Engine: store.SpamEngineRspamd, Detail: "5.7.1 Refused by this server's deny list"},
{From: "billing@vendor.example", ClientIP: "192.0.2.5", Decision: store.SpamDecisionAccept, Engine: store.SpamEngineRspamd, Detail: "no action, score -99.00 / 15.00; SELFPOST_ALLOW_FROM(-100.00)"},
// refused by the filter for its own reasons: nobody's match
{From: "promo@bulk.example", ClientIP: "198.51.100.9", Decision: store.SpamDecisionReject, Engine: store.SpamEngineRspamd, Detail: "5.7.1 Spam message rejected"},
} {
row.InboundDomain, row.To, row.At = "a.example.com", "alice@a.example.com", at.Add(time.Duration(i)*time.Hour)
if err := s.st.InsertInboundSpam(row); err != nil {
t.Fatal(err)
}
}
page := s.get(s.h.HandleFilterLists, "/inbound/filter-lists", globalPrincipal).Body.String()
has(t, "lists", page,
`<tr><th>Matched</th><td>2 in the last 90 days</td></tr><tr><th>Last match</th><td class="sp-mono sp-small">2026-10-01 10:30</td></tr>`,
`<td class="sp-nowrap">2</td>`,
`<tr><th>Matched</th><td>1 in the last 90 days</td></tr><tr><th>Last match</th><td class="sp-mono sp-small">2026-10-01 11:30</td></tr>`,
`<td class="sp-nowrap">1</td>`,
// the network nobody came from
`<td class="sp-muted sp-nowrap">never</td>`,
)
if got := strings.Count(page, `<td class="sp-muted sp-nowrap">never</td>`); got != 1 {
t.Errorf("%d entries say never, want the one network", got)
}
}
+6 -5
View File
@@ -450,12 +450,13 @@ func inLogFixture() *InLog {
// allowed, and a filter that reads them. // allowed, and a filter that reads them.
func inFilterListsFixture() *FilterLists { func inFilterListsFixture() *FilterLists {
day := func(m time.Month, d int) time.Time { return time.Date(2026, m, d, 0, 0, 0, 0, time.UTC) } day := func(m time.Month, d int) time.Time { return time.Date(2026, m, d, 0, 0, 0, 0, time.UTC) }
return NewFilterLists(admin(), FilterListsInput{FilterOn: true, LastWritten: time.Date(2026, 9, 11, 8, 40, 0, 0, time.UTC)}). at := func(d, h, m int) time.Time { return time.Date(2026, 9, d, h, m, 0, 0, time.UTC) }
return NewFilterLists(admin(), FilterListsInput{FilterOn: true, LastWritten: time.Date(2026, 9, 11, 8, 40, 0, 0, time.UTC), RetentionDays: 90}).
WithEntries([]FilterEntryInput{ WithEntries([]FilterEntryInput{
{ID: 1, List: "deny", Kind: "domain", Value: "bulk.example", Note: "newsletter nobody subscribed to", Added: day(9, 2)}, {ID: 1, List: "deny", Kind: "domain", Value: "bulk.example", Note: "newsletter nobody subscribed to", Added: day(9, 2), Hits: 41, LastHit: at(21, 13, 57)},
{ID: 2, List: "deny", Kind: "ip", Value: "198.51.100.0/24", Note: "dictionary attack on lists.example.org", Added: day(8, 19)}, {ID: 2, List: "deny", Kind: "ip", Value: "198.51.100.0/24", Note: "dictionary attack on lists.example.org", Added: day(8, 19), Hits: 1203, LastHit: at(21, 9, 12)},
{ID: 3, List: "deny", Kind: "address", Value: "win@lottery.example", Added: day(7, 30)}, {ID: 3, List: "deny", Kind: "address", Value: "win@lottery.example", Added: day(7, 30)},
{ID: 4, List: "allow", Kind: "domain", Value: "partner.example", Note: "invoices were scored as spam", Added: day(9, 11)}, {ID: 4, List: "allow", Kind: "domain", Value: "partner.example", Note: "invoices were scored as spam", Added: day(9, 11), Hits: 7, LastHit: at(19, 16, 40)},
{ID: 5, List: "allow", Kind: "address", Value: "noreply@github.example", Note: "security advisories", Added: day(6, 4)}, {ID: 5, List: "allow", Kind: "address", Value: "noreply@github.example", Note: "security advisories", Added: day(6, 4), Hits: 18, LastHit: at(21, 13, 30)},
}) })
} }
+1 -1
View File
@@ -205,7 +205,7 @@ var inboundTopics = []HelpTopic{
Strong("Deny wins"), ".", Strong("Deny wins"), ".",
Br(), Br(), Br(), Br(),
"SelfPost does not apply the lists itself: it writes them as files under ", Code("/data/antispam/"), "SelfPost does not apply the lists itself: it writes them as files under ", Code("/data/antispam/"),
", and rspamd rereads a file within seconds of a change. With the spam filter off the lists are kept and can be edited, but nothing reads them. The sender matched is the envelope sender — the one in the log — not the From line a mail program shows."), ", and rspamd rereads a file within seconds of a change. With the spam filter off the lists are kept and can be edited, but nothing reads them. The sender matched is the envelope sender — the one in the log — not the From line a mail program shows. Matched counts the messages in the Inbound log that the entry decided, so it reaches back as far as the log does; never means not once in that time, and such an entry is one to consider removing."),
}}, }},
} }
+43 -17
View File
@@ -46,6 +46,8 @@ type FilterLists struct {
Where Box Where Box
WhereFacts []Fact WhereFacts []Fact
retentionDays int
} }
// FilterList is one of the two boxes of entries. // FilterList is one of the two boxes of entries.
@@ -55,24 +57,29 @@ type FilterList struct {
Empty EmptyState Empty EmptyState
} }
// FilterRow is one entry of a list: the value with the kind of entry as a tag, // FilterRow is one entry of a list: the value, the kind of entry as a tag that
// the operator's note, the day it was added and the way to remove it. // opens the entry's sheet (the note, the day it was added, how often it decided
// a message and when last), how often it decided, and the way to remove it.
// Matched is empty for an entry that never did; the table then says "never".
type FilterRow struct { type FilterRow struct {
Value string Value string
Kind Tag Entry Sheet
Note string Matched string
Added string Remove RowAction
Remove RowAction
} }
// FilterEntryInput is what the page knows of one entry of a list. // FilterEntryInput is what the page knows of one entry of a list. Hits and
// LastHit are counted from the Inbound log, so they reach back as far as it
// does.
type FilterEntryInput struct { type FilterEntryInput struct {
ID int64 ID int64
List string List string
Kind string Kind string
Value string Value string
Note string Note string
Added time.Time Added time.Time
Hits int
LastHit time.Time
} }
// FilterListsInput is the state of the filter the lists are written for: whether // FilterListsInput is the state of the filter the lists are written for: whether
@@ -80,6 +87,9 @@ type FilterEntryInput struct {
type FilterListsInput struct { type FilterListsInput struct {
FilterOn bool FilterOn bool
LastWritten time.Time LastWritten time.Time
// RetentionDays is how long the Inbound log keeps a row: how far back an
// entry's matches are counted.
RetentionDays int
} }
// NewFilterLists builds the page. The form starts on the Deny list for a sender // NewFilterLists builds the page. The form starts on the Deny list for a sender
@@ -113,6 +123,7 @@ func NewFilterLists(m Meta, in FilterListsInput) *FilterLists {
} }
p.Deny.Box.End = Plain(plural(0, "entry", "entries") + " · wins over Allow") p.Deny.Box.End = Plain(plural(0, "entry", "entries") + " · wins over Allow")
p.Allow.Box.End = Plain(plural(0, "entry", "entries")) p.Allow.Box.End = Plain(plural(0, "entry", "entries"))
p.retentionDays = in.RetentionDays
p.setWhere(in) p.setWhere(in)
return p return p
} }
@@ -144,10 +155,25 @@ func (p *FilterLists) setWhere(in FilterListsInput) {
func (p *FilterLists) WithEntries(entries []FilterEntryInput) *FilterLists { func (p *FilterLists) WithEntries(entries []FilterEntryInput) *FilterLists {
p.Deny.Rows, p.Allow.Rows = nil, nil p.Deny.Rows, p.Allow.Rows = nil, nil
for _, e := range entries { for _, e := range entries {
row := FilterRow{ id := strconv.FormatInt(e.ID, 10)
Value: e.Value, Kind: Tag{Label: e.Kind}, Note: e.Note, Added: e.Added.UTC().Format("2006-01-02"), window := "in the last " + plural(int64(p.retentionDays), "day", "days")
Remove: RowAction{Label: "Remove", Post: inboundFilterLists + "/" + strconv.FormatInt(e.ID, 10) + "/delete", Danger: true}, sheet := []SheetRow{
{Label: "Note", Lines: []string{orDash(e.Note)}},
{Label: "Added", Lines: []string{e.Added.UTC().Format("2006-01-02")}, Mono: true},
} }
row := FilterRow{
Value: e.Value,
Remove: RowAction{Label: "Remove", Post: inboundFilterLists + "/" + id + "/delete", Danger: true},
}
if e.Hits > 0 {
row.Matched = strconv.Itoa(e.Hits)
sheet = append(sheet,
SheetRow{Label: "Matched", Lines: []string{row.Matched + " " + window}},
SheetRow{Label: "Last match", Lines: []string{e.LastHit.UTC().Format("2006-01-02 15:04")}, Mono: true})
} else {
sheet = append(sheet, SheetRow{Label: "Matched", Lines: []string{"never " + window}})
}
row.Entry = Sheet{ID: "entry-" + id, Tag: Tag{Label: e.Kind}, Title: e.Value, Rows: sheet}
switch e.List { switch e.List {
case "deny": case "deny":
p.Deny.Rows = append(p.Deny.Rows, row) p.Deny.Rows = append(p.Deny.Rows, row)
+16 -7
View File
@@ -183,18 +183,27 @@ func TestFilterListsShowsTheFormTheListsAndWhereTheyGo(t *testing.T) {
`href="/help#filter-lists" title="How the lists are applied"`, `href="/help#filter-lists" title="How the lists are applied"`,
`<h2>Deny</h2><div class="sp-end">3 entries · wins over Allow</div>`, `<h2>Deny</h2><div class="sp-end">3 entries · wins over Allow</div>`,
`<h2>Allow</h2><div class="sp-end">2 entries</div>`, `<h2>Allow</h2><div class="sp-end">2 entries</div>`,
`<td class="sp-mono sp-small">bulk.example <span class="tag is-light">domain</span></td><td class="sp-muted">newsletter nobody subscribed to</td><td class="sp-muted sp-nowrap sp-small">2026-09-02</td>`, `<thead><tr><th>Value</th><th>Kind</th><th>Matched</th><th></th></tr></thead>`,
`198.51.100.0/24 <span class="tag is-light">ip</span>`, // the kind is a tag that opens the entry's sheet; the count is beside it
`win@lottery.example <span class="tag is-light">address</span></td><td class="sp-muted">—</td>`, `<td class="sp-mono sp-small">bulk.example</td><td><button type="button" class="tag is-light" popovertarget="entry-1">domain</button><div class="sp-sheet" id="entry-1" popover>`,
`<div class="sp-sheet-head"><span class="tag is-light">domain</span><h2>bulk.example</h2>`,
`<tr><th>Note</th><td>newsletter nobody subscribed to</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-09-02</td></tr><tr><th>Matched</th><td>41 in the last 90 days</td></tr><tr><th>Last match</th><td class="sp-mono sp-small">2026-09-21 13:57</td></tr></tbody>`,
`</div></td><td class="sp-nowrap">41</td><td class="sp-actions">`,
`<td class="sp-nowrap">1203</td>`,
// an entry that never decided anything says so, in the row and on its sheet
`<td class="sp-mono sp-small">win@lottery.example</td><td><button type="button" class="tag is-light" popovertarget="entry-3">address</button>`,
`<tr><th>Note</th><td>—</td></tr><tr><th>Added</th><td class="sp-mono sp-small">2026-07-30</td></tr><tr><th>Matched</th><td>never in the last 90 days</td></tr></tbody>`,
`</div></td><td class="sp-muted sp-nowrap">never</td><td class="sp-actions">`,
`<form method="post" action="/inbound/filter-lists/3/delete"><button type="submit" class="has-text-danger">Remove</button></form>`, `<form method="post" action="/inbound/filter-lists/3/delete"><button type="submit" class="has-text-danger">Remove</button></form>`,
`<h2>Where the lists go</h2><div class="sp-end"><span class="tag is-success is-light">applied</span></div>`, `<h2>Where the lists go</h2><div class="sp-end"><span class="tag is-success is-light">applied</span></div>`,
`rspamd, from map files it rereads within seconds of a change`, `rspamd, from map files it rereads within seconds of a change`,
`<span class="sp-mono">/data/antispam/</span> — six, one for each list and kind of entry`, `<span class="sp-mono">/data/antispam/</span> — six, one for each list and kind of entry`,
`2026-09-11 08:40 UTC`, `Deny wins, whichever entries they are.`, `2026-09-11 08:40 UTC`, `Deny wins, whichever entries they are.`,
) )
if got := strings.Count(out, `<table`); got != 2 { if got := strings.Count(out, `<table class="table is-fullwidth is-hoverable">`); got != 2 {
t.Errorf("%d tables, want the two lists", got) t.Errorf("%d list tables, want the two lists", got)
} }
pageLacks(t, "Filter lists", out, `<th>Note</th><th>`, `<th>Added</th><th>`)
} }
// With no filter attached the lists are still the operator's to edit, and the // With no filter attached the lists are still the operator's to edit, and the
@@ -213,8 +222,8 @@ func TestFilterListsOffAndEmpty(t *testing.T) {
`No sender is turned away by hand.`, `No sender is turned away by hand.`,
) )
pageLacks(t, "Filter lists", out, `rspamd, from map files`, `is-success is-light">applied`, `No sender is let through by hand.`) pageLacks(t, "Filter lists", out, `rspamd, from map files`, `is-success is-light">applied`, `No sender is let through by hand.`)
if got := strings.Count(out, `<table`); got != 1 { if got := strings.Count(out, `<table class="table is-fullwidth is-hoverable">`); got != 1 {
t.Errorf("%d tables, want the one list that has entries", got) t.Errorf("%d list tables, want the one list that has entries", got)
} }
} }
@@ -1,7 +1,8 @@
{{/* Inbound > Filter lists (mockup in-filter-lists.html): the allow and deny {{/* Inbound > Filter lists (mockup in-filter-lists.html): the allow and deny
lists that correct the spam filter by hand, for the whole instance, and lists that correct the spam filter by hand, for the whole instance, and
where they go. Global role only. The form adds an entry; each row of a list where they go. Global role only. The form adds an entry; each row of a list
removes its own. */}} removes its own, says how often the entry decided a message, and opens a
sheet with the entry's note from the tag of its kind. */}}
{{define "content"}} {{define "content"}}
{{template "page_head" .Head}} {{template "page_head" .Head}}
{{if .Flash}}{{template "flash" .Flash}}{{end}} {{if .Flash}}{{template "flash" .Flash}}{{end}}
@@ -54,9 +55,9 @@
table, or what is absent when the list is empty. */}} table, or what is absent when the list is empty. */}}
{{define "filter_list"}}{{template "box_open" .Box}} {{define "filter_list"}}{{template "box_open" .Box}}
{{if .Rows}}<div class="table-container"><table class="table is-fullwidth is-hoverable"> {{if .Rows}}<div class="table-container"><table class="table is-fullwidth is-hoverable">
<thead><tr><th>Value</th><th>Note</th><th>Added</th><th></th></tr></thead> <thead><tr><th>Value</th><th>Kind</th><th>Matched</th><th></th></tr></thead>
<tbody>{{range .Rows}} <tbody>{{range .Rows}}
<tr><td class="sp-mono sp-small">{{.Value}} {{template "tag" .Kind}}</td>{{if .Note}}<td class="sp-muted">{{.Note}}</td>{{else}}<td class="sp-muted">—</td>{{end}}<td class="sp-muted sp-nowrap sp-small">{{.Added}}</td><td class="sp-actions">{{template "row_action" .Remove}}</td></tr>{{end}} <tr><td class="sp-mono sp-small">{{.Value}}</td><td>{{template "sheet_tag" .Entry}}</td>{{if .Matched}}<td class="sp-nowrap">{{.Matched}}</td>{{else}}<td class="sp-muted sp-nowrap">never</td>{{end}}<td class="sp-actions">{{template "row_action" .Remove}}</td></tr>{{end}}
</tbody> </tbody>
</table></div> </table></div>
{{else}}{{template "empty_state" .Empty}} {{else}}{{template "empty_state" .Empty}}