FreeBSD Jails: Putting Them to Work

In the last entry, I explained how I decided to remove the Linux hypervisor from my network and replace it with a beefy FreeBSD server named joker.  Instead of all of the VMs running on the former KVM server, I instead created a collection of FreeBSD jails.  This post will serve to document what I did to get them performing actual work.

More experienced FreeBSD jail admins might read this and other posts of mine and think, “Why the hell did he do it that way?!”  The answer: I’m still learning how this works.  And the way I figure things out might not necessarily be the way you do.

Secure the Network From the Jails

Remember in my last post that I have joker dual-homed on the public side of my network as well as the private.  It’s a big risk, but one I’m willing to live with.  Each of the jails were given their own public IP, but were not given a private one.  However, that didn’t stop them from being able to contact the private network through joker‘s second interface.  The source of the connection would be the jail’s public IP, but it would egress joker via its private interface and hit the private network.

No good.

To block that outbound, I changed some of the pf rules on joker to specifically block all of the jails’ public IPs from reaching outbound to the private network.  Remember that because it comes up later:

block out quick from <riddler> to <local_lan>
block out quick from <clayface> to <local_lan>
block out quick from <penguin> to <local_lan>
block out quick from <scarecrow> to <local_lan>
block out quick from <catwoman> to <local_lan>

Riddler: Outbound SMTP

Some time ago, the “fine” folks at Google decided that my joker VM would no longer be able to send mail to GMail users.  Their anti-spam tool somehow tagged joker as an originator of “bulk email”, and from that point forward, I was no longer able to send email to anyone at GMail.  I contacted an old AOL co-worker of mine who’s been at Google for a bunch of years and asked him to look into it for me.  Unfortunately, nothing happened.  To solve this problem, I spun up a new VM called riddler, gave it its own public IP address (I have a block), and configured it to send outbound mail.  Voila: I could email GMail again.

The new riddler jail was the first to get configured.  I used pkg to install the saslauthd and the sendmail that uses it:

pkg install cyrus-sasl-saslauthd sendmail+tls+sasl2

The /etc/mail/mailer.conf file had to be adjusted so that the proper sendmail would get called:

# $FreeBSD: releng/10.2/etc/mail/mailer.conf 93858 2002-04-05 04:25:14Z gshapiro $
#
# Execute the "real" sendmail program, named /usr/libexec/sendmail/sendmail
#
sendmail	/usr/local/sbin/sendmail
send-mail	/usr/local/sbin/sendmail
mailq		/usr/local/sbin/sendmail
newaliases	/usr/local/sbin/sendmail
hoststat	/usr/local/sbin/sendmail
purgestat	/usr/local/sbin/sendmail

A couple of quick edits to /etc/mail/riddler.lateapex.net.mc:

dnl set SASL options
TRUST_AUTH_MECH(`GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN')dnl
define(`confAUTH_MECHANISMS', `GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN')dnl

And then run this in /etc/mail to make a new sendmail.cf:

make install

Add these lines to /etc/rc.conf:

# Sendmail
sendmail_enable="YES"

# SASL Auth for sendmail
saslauthd_enable="YES"

And start them both up:

service sendmail start
service saslauthd start

But I wasn’t done. I also had to add my user to riddler’s local passwd file so that my IMAP client could authenticate when I sent email. I performed the usual steps of running vipw, adding the new user, and setting a password. But in this case, the user didn’t need a real shell, nor a home directory. My user would never be logging into the riddler jail, namely because neither it nor any of the other jails would be running sshd.

After all of that, I had a useable outbound SMTP server, all within a jail. One of the great things about jails is that they only have the processes running that they actually need. For instance:

riddler# ps auxww
USER     PID %CPU %MEM    VSZ   RSS TT  STAT STARTED    TIME COMMAND
root    3858  0.0  0.0  14512  2156  -  SsJ  24Nov15 0:07.52 /usr/sbin/syslogd -s
root    3978  0.0  0.0  47404  5288  -  IsJ  24Nov15 0:00.26 /usr/local/sbin/saslauthd -a pam
root    4045  0.0  0.0  47404  5224  -  IJ   24Nov15 0:00.26 /usr/local/sbin/saslauthd -a pam
root    4049  0.0  0.0  47404  5224  -  IJ   24Nov15 0:00.26 /usr/local/sbin/saslauthd -a pam
root    4050  0.0  0.0  47404  5224  -  IJ   24Nov15 0:00.26 /usr/local/sbin/saslauthd -a pam
root    4052  0.0  0.0  47404  5224  -  IJ   24Nov15 0:00.26 /usr/local/sbin/saslauthd -a pam
root    4425  0.0  0.0  51344  6120  -  SsJ  24Nov15 0:11.93 sendmail: accepting connections (sendmail)
smmsp   4428  0.0  0.0  26312  5284  -  IsJ  24Nov15 0:00.28 sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (sendmail)
root    4432  0.0  0.0  16612  2284  -  IsJ  24Nov15 0:02.69 /usr/sbin/cron -s
root   96271  0.0  0.0  23592  3456  0  SJ    7:18PM 0:00.02 /bin/tcsh
root   96366  0.0  0.0  18760  2180  0  R+J   7:30PM 0:00.00 ps auxww
riddler#

Jails for Apache: Clayface, Catwoman, and Penguin

While joker was a FreeBSD (and formerly a Linux) VM under KVM, I had the apache web server running on it, serving up 4 different web sites.  One is a site I use internally, and with the new joker physical build, it’s still there.  The other 3 are sites that I run either for buddies, or for myself.  For instance: this blog.  I decided that since I’m paying for a block of public IP addresses, I should go ahead and use them.  So with that, I decided to put each of the 3 public web servers in their own jail.

I still had the web page data (eg: PHPBB3, blog software, etc) in the same directories as before: NFS-mounted from bane.  The joker VM didn’t have any of those web pages on a local filesystem.  They were (and still are) all stored on bane.  But how to get that data mounted within the jail?  Doing mounts from within a jail isn’t allowed (usually).  And of course I blocked traffic from the jails to anything on the private network (such as the NAS).

Fortunately, FreeBSD has a nullfs mount that allowed me to work around this.  Since joker is the jail host, it has direct access to the jails’ filesystems.  So using a null mount, I could attach a directory from the NAS mount (remember: /opt) to each of the jails’ filesystems.

For instance, this blog is on jail catwoman.  On joker, I made a directory:

mkdir -p /local/jails/catwoman/usr/local/www/apache24/jasonvanpatten.com

And then using the mount command:

mount -t nullfs -o rw /opt/www/joker.apache24/jasonvanpatten.com /local/jails/catwoman/usr/local/www/apache24/jasonvanpatten.com

From within the catwoman jail:

joker# jexec catwoman /bin/tcsh
# source .cshrc
catwoman# cd /usr/local/www/apache24/jasonvanpatten.com/
catwoman# ls -la index.php
-rw-r--r--  1 www  www  418 Sep 24  2013 index.php
catwoman# df .
Filesystem                                   1K-blocks      Used       Avail Capacity  Mounted on
/opt/www/joker.apache24/jasonvanpatten.com 11757450956 660062840 11097388116     6%    [restricted]
catwoman#

Similar mounts were done for penguin and clayface.  I also installed apache24, various php56 packages, and whatever else I’d need on each jail to run their local web server.  However, each of them needed access to the MySQL database running on bane.  Hm.  Well that was easy enough to fix in joker‘s pf file: just punch mysql holes in from each jail to bane.  Done.

Before too long, I had 3 more jails running with apache on each of them.

Null Filesystems Causing Boot Problems

I wanted these filesystems mounted at boot time, so I added them all to joker‘s /etc/fstab:

/opt/www/joker.apache24/jvpvideoproductions.com         /local/jails/clayface/usr/local/www/apache24/jvpvideoproductions.com    nullfs  rw       0       0
/opt/www/joker.apache24/jasonvanpatten.com              /local/jails/catwoman/usr/local/www/apache24/jasonvanpatten.com nullfs  rw      0       0
/opt/www/joker.apache24/trackdogs       /local/jails/penguin/usr/local/www/apache24/trackdogs   nullfs  rw       0       0

At some point shortly thereafter, I rebooted joker and it didn’t come up properly. Its console was stuck in single user mode because it couldn’t mount the filesystems in /etc/fstab. Huh?

The problem was two fold.  First: The nullfs entries in /etc/fstab were being executed before the /local ZFS volume was mounted.  Secondly, they were being executed before /opt was NFS mounted.  Hm.  I had to wait until the ZFS and NFS mounts were executed before mounting the new nullfs entries.

I solved this in my usual sledge hammer approach.  The first step was to add the keyword noauto to each of the nullfs entries’ options.  Like so:

/opt/www/joker.apache24/jvpvideoproductions.com         /local/jails/clayface/usr/local/www/apache24/jvpvideoproductions.com    nullfs  rw,noauto       0       0
/opt/www/joker.apache24/jasonvanpatten.com              /local/jails/catwoman/usr/local/www/apache24/jasonvanpatten.com nullfs  rw,noauto      0       0
/opt/www/joker.apache24/trackdogs       /local/jails/penguin/usr/local/www/apache24/trackdogs   nullfs  rw,noauto       0       0

Obviously, this meant that the filesystems would not get mounted at boot time. So to address that, I wrote /usr/local/etc/rc.d/jailfs:

#!/bin/sh

# PROVIDE: jailfs
# REQUIRE: mountcritlocal mountcritremote
# BEFORE: jail

. /etc/rc.subr

name="jailfs"
rcvar="jailfs_enable"

start_cmd="jailfs_start"
stop_cmd="jailfs_stop"

jailfs_start()
{
	echo "Mounting jail filesystems:"
	for FS in `grep nullfs /etc/fstab | grep noauto | awk '{print $2}'`
	do
		echo "		${FS}"
		mount ${FS}
	done
}

jailfs_stop()
{
	echo "Unmounting jail filesystems:"
	for FS in `grep nullfs /etc/fstab | grep noauto | awk '{print $2}'`
	do
		echo "		${FS}"
		umount ${FS}
	done
}

load_rc_config $name
run_rc_command "$1"

Of course, with that, I had to add another entry in joker‘s /etc/rc.conf:

# Mount filesystems needed for jails
jailfs_enable="YES"

That seemed to do the trick. The next reboot saw the /local filesystem get mounted first, then /opt, and then all of the nullfs entries in /etc/fstab.

Scarecrow: Authoritative DNS Jail

The last jail on the list was scarecrow.  I created him because I wanted to convert the named process on joker to be internal only.  Within the scarecrow jail, I installed the bind910 pkg, and then from joker, just copied all of the /usr/local/etc/namedb files to scarecrow‘s /usr/local/etc.  I made the appropriate edits to the config files, spun the bind process up and then changed all of my domain SOA records to point to scarecrow instead of joker.

joker# jexec scarecrow /bin/tcsh
# source .cshrc
scarecrow# ps auxww
USER   PID %CPU %MEM    VSZ   RSS TT  STAT STARTED    TIME COMMAND
root  3844  0.0  0.0  14512  2084  -  SsJ  24Nov15 0:01.70 /usr/sbin/syslogd -s
root  4258  0.0  0.0  16612  2200  -  IsJ  24Nov15 0:02.43 /usr/sbin/cron -s
bind 81587  0.0  0.0 201712 51736  -  IsJ  Sun09AM 0:15.87 /usr/local/sbin/named -u bind -c /usr/local/etc/namedb/named.conf
root 96865  0.0  0.0  23592  3344  0  SJ    8:26PM 0:00.01 /bin/tcsh
root 96867  0.0  0.0  18760  2120  0  R+J   8:26PM 0:00.00 ps auxww
scarecrow#

PF Holes for the Jails

Joker’s PF rules were originally written to handle incoming connections for him.  Once the jails were launched and running, I had to punch a series of holes in the filter for each of them.  Incoming port 80 for the apache servers.  Incoming port 587 for riddler.  Port 53 for scarecrow.

# Services riddler will listen on
riddler_tcp_ports = "{ smtp, 587 }"

# Services penguin will listen on
penguin_tcp_ports = "{ smtp, www }"

# Services clayface will listen on
clayface_tcp_ports = "{ www }"

# Services catwoman will listen on
catwoman_tcp_ports = "{ www }"

# Service scarecrow will listen on
scarecrow_tcp_ports = "{ domain }"

# Riddler (SMTP forwarder)
pass in quick proto tcp from any to <riddler> port $riddler_tcp_ports

# penguin
pass in quick proto tcp from any to <penguin> port $penguin_tcp_ports

# clayface
pass in quick proto tcp from any to <clayface> port $clayface_tcp_ports

# catwoman
pass in quick proto tcp from any to <catwoman> port $catwoman_tcp_ports

# scarecrow
pass in quick proto tcp from any to <scarecrow> port $scarecrow_tcp_ports

More to Come

This work provided me the separation of most of my internal and external services.  At this point, the main host joker was running services that only I use.  Each of the jails were running services that others use to communicate with me.  Except incoming mail; that was still running on joker.  There’s a reason for that and it has to do with the spamilter I’m running.  But I have a lot more to tell about that spamilter before I explain how I ultimately stuffed it and all incoming email into a jail.  Stay tuned.