FreeBSD, Server and OS, Spam Fighting

Spamilter in a Jail

If you’ve been following my two sub-threads on this blog regarding FreeBSD jails and fighting spam, you know the status: I have a FreeBSD server called joker that has multiple jails on it.  One jail is riddler, and it handles all outbound SMTP connections.  But not inbound.  Further, you know that I have a reasonably complex anti-spam system set up on joker using a combination of SpamAssassin along with Neal Horman’s spamilter.  I won’t summarize them any further; if you need to catch up, read the previous entries.

I have all incoming SMTP connections going to joker, because of spamilter and its excellent companion: ipfwmtad.  As previously explained, ipfwmtad takes input from spamilter to block incoming connections from known bad MTAs via FreeBSD’s ipfw or Linux’s iptables.  FreeBSD jails can’t have ipfw running (without jumping through a lot of hoops).  They must rely on their host for it.  So, it means that spamilter and ipfwmtad have to run on joker, not riddler.

Separate Spamilter and Ipfwmtad

My goal with jails on FreeBSD was to spin up a jail per externally facing service such as SMTP, DNS, web servers, etc.  However, with the aforementioned restriction, the host joker was still taking inbound SMTP.  I figured the key to moving incoming into a jail was to separate the spamilter daemon from the ipfwmtad daemon.  In other words: have spamilter running on one server (where sendmail is) and have ipfwmtad running on another.ipfw2

More to the point: riddler would be running sendmail to allow incoming SMTP, and it would have spamilter running to provide anti-spam coverage.  But joker would have to be running ipfwmtad because it’s the host, and the host has ipfw running.  I’d also have to get SpamAssassin running on riddler along with procmail.  That way all of the message filtering could be done within riddler, but only after the port 25 connection made it through joker‘s ipfw.

Fortunately, Neal’s code made it look like this could be an reasonably easy change.  In the hndlrs.c file, spamilter makes a network socket connection to localhost, port 4739 to talk to ipfwmtad:

int sd = NetSockOpenTcpPeer(INADDR_LOOPBACK,4739);

It seemed to me that perhaps he could change that a bit to allow the target address to be configured by the end user, but default to  And if so, I could have spamilter on riddler talk to an ipfwmtad process running on joker.  I sent a note off to Neal asking him to consider it, and within a short period of time…

New Spamilter Code and New Config Options

After Neal’s code change, the ipfwmtad process could be started and told to listen on a specific IP address, using the -I <address> option.  Further, there’s a new config file: /usr/local/etc/spamilter/ipfwmtad.acl which defines which IP addresses (or networks) are allowed to connect to ipfwmtad.  Finally, the /usr/local/etc/spamilter/spamilter.conf file got a few more entries in it to help it talk to a remote ipfwmtad.

Spamilter on Riddler

When I created the riddler jail, I had the forethought to do so with the FreeBSD source code included.  Before snagging a copy of Neal’s latest code for riddler, I had to install the git package.  From within the jail:

pkg install git
cd /usr/src
git clone

The rest of the install and configuration was identical to the previous one on joker, as outlined in the first spam fighting blog entry.  Only this time, I had to make some slight changes to the /etc/rc.conf:

# Spamilter

I specifically didn’t want or need ipfwmtad starting in the riddler jail, so I set its variable to “NO”. Then in /usr/local/etc/spamilter/spamilter.conf:

IpfwHost = joker's_IP
IpfwPort = 4739
IpfwUser = ipfw_user
IpfwPass = ipfw_pass

The key points here are that I’m using joker’s IP address, not And in order to do that, I have to include a user and password in the config file. That user/pass combination should correspond to an existing entry in the target server’s /etc/passwd file. But that user doesn’t need to have any rights or access to anything. It’s purely for authentication. With that, I created a user on joker and set its password accordingly, but its home directory was set to /nonexistent, and its login shell to /usr/sbin/nologin.

At this point, having followed all of the installation and configuration steps outline in the first blog entry, I had sendmail talking to spamilter on riddler.  I was almost ready to cut the MX record in my DNS entry over to riddler, but I had a few more things to do.

Ipfwmtad on Joker

The /etc/rc.conf file on joker had to change a little bit.  Namely, I didn’t need spamilter, or even sendmail or SpamAssassin starting any longer.  However, I did still need ipfwmtad to start:

# Sendmail
# Spam Assassin and Milter
# Spamilter needed just for ipfwmtad
ipfwmtad_flags="-I joker's_IP"

Neal’s code distribution didn’t include an rc file just for ipfwmtad.  The spamilter rc file wouldn’t launch anything if it saw the “NO” spamilter_enable variable.  So I quickly hacked together my own /usr/local/etc/rc.d/ipfwmtad:


# PROVIDE: ipfwmtad
# REQUIRE: psql
# KEYWORD: nojail shutdown

. /etc/rc.subr




	if [ "${ipfwmtad_enable}" = "YES" ]; then
		pid_ipfwmtad="`check_process ${proc_ipfwmtad}`"
		if [ -n "$pid_ipfwmtad" ]; then
			echo "ipfwmtad is running as pid ${pid_ipfwmtad}"
			echo "ipfwmtad is not running."

	return $rc


	if [ "${ipfwmtad_enable}" = "YES" ]; then
		if [ -z "`check_process ${proc_ipfwmtad}`" ]; then
			${proc_ipfwmtad} ${ipfwmtad_flags}

	return $rc

	local pids

	if [ "${ipfwmtad_enable}" = "YES" ]; then
		pid_ipfwmtad="`check_process ${proc_ipfwmtad}`"
		if [ ! -z "${pid_ipfwmtad}" ]; then
			kill $sig_stop ${pid_ipfwmtad}
			pids="${pids} ${pid_ipfwmtad}"

	if [ -n "${pids}" ]; then
		wait_for_pids ${pids}
		return 0

	return 1

load_rc_config $name
run_rc_command "$1"

With that, ipfwmtad would now start on joker, latching on to joker‘s public IP.  One last config file on joker needed to be edited so that riddler could connect to joker’s ipfwmtad.  In the file /usr/local/etc/spamilter/ipfwmtad.acl:

	allow		|	riddler_IPv4/32	|
	allow		|	riddler_IPv6/128	|

SpamAssassin in Jail

The previous steps allowed me to split the spamilter and ipfwmtad up onto different machines, one of which being a jail.  However, I still needed to get SpamAssassin running on riddler, which meant procmail as well.  Further, I needed to make sure that riddler was writing the mail to a spool file that joker could read from because I intended to leave my Dovecot IMAP daemon on the latter.

Fortunately this was fairly easy.  SpamAssassin initially complained about not being able to latch on to the loopback interface of the jail, but that was easily solved by telling it to use riddler’s public IP instead.  In riddler’s /etc/rc.conf:

# Spam Assassin
spamd_flags="-i -A riddler_IPv4 -A [riddler_IPv6]"

The last bit on riddler was to add my user’s home directory.  Previously, I didn’t need it to have a home directory within the jail; the only reason I had an entry in riddler‘s /etc/passwd was for saslauthd to authenticate outbound SMTP.  However, procmail needed to be able to read my ~/.procmailrc file, which meant I needed a home directory.  I created one, copied the .procmailrc file over from my home directory on joker, and then started SpamAssassin.

One More Nullfs on Joker

Recall from one of the previous entries that I had a series of nullfs mounts on joker for the jails to access an NFS volume.  My NFS server bane has, among its exported filesystem, joker‘s /var/mail spool directory:

joker# ls -la /var/mail
lrwxr-xr-x 1 root wheel 19 Nov 13 16:59 /var/mail@ -> /opt/var/spool/mail
joker# df /opt/var/spool/mail
Filesystem                                 Size    Used   Avail Capacity  Mounted on     11T    629G     10T     6%    /opt

I wanted the sendmail process on riddler to write incoming mail to the same spool directory. I needed another nullfs mount. From joker:

joker# mount -t nullfs /opt/var/spool/mail /local/jails/riddler/var/mail

And from within riddler:

riddler# df /var/mail
Filesystem            1K-blocks      Used       Avail Capacity  Mounted on
/opt/var/spool/mail 11757442224 659306944 11098135280     6%    [restricted]

Move MX Record to Riddler

After completing all of these steps, I was ready to move the MX record for my domains over to riddler.  I made the appropriate changes on my authoritative DNS server, and within a short amount of time, riddler was taking all inbound.  Spamilter and SpamAssassin were working fine together from within the jail, and the former was communicating with ipfwmtad on joker to block incoming SMTP connections from bad MTAs.

Of course, this gave me another idea: if I could move the ipfwmtad process to the jail’s host, what’s stopping me from moving it to a different machine altogether?  Like, perhaps, a router running FreeBSD? …

Stay tuned.

Leave a Reply