FreeBSD, Network Architecture, Server and OS

FRR Patched And Working

This entry will be another fairly quick one.  In the previous entry regarding the server, routing, etc, I described how I had to work around the fact that FRR didn’t honor FreeBSD’s assigning and deletion of IP addresses from interfaces.  Specifically: I had the line

redistribute connected

in the server’s BGP configuration.  That means any of the server’s interfaces with IP addresses on them would have those IP addresses added to BGP.  And, it should mean that if an IP address is removed from one of those interfaces, the corresponding entry would be withdrawn from BGP.  But as I found: that wasn’t happening.  I’ll try to explain with a small amount of detail.

When a FreeBSD jail is started, you can configure it to add an IP address to the server’s physical interface.  And when that jail is stopped, the IP address is removed from the interface.  FRR was seeing the new IP address, but it was NOT seeing the IP address being deleted.  So when the jail was stopped, FRR thought its corresponding IP address was still on the interface.

Here’s an example, using an RFC1918 IP address on my public interface.

arkham# ifconfig lagg0.50 alias 10.200.0.1/32 
arkham# ifconfig lagg0.50 
lagg0.50: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 
options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6> 
ether a0:36:9f:0d:ea:c0 
inet 108.28.193.217 netmask 0xffffffff broadcast 108.28.193.217 
inet 108.28.193.219 netmask 0xffffffff broadcast 108.28.193.219 
inet 108.28.193.212 netmask 0xffffffff broadcast 108.28.193.212 
inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255 
inet 108.28.193.210 netmask 0xffffffff broadcast 108.28.193.210 
inet 108.28.193.213 netmask 0xffffffff broadcast 108.28.193.213 
inet 10.200.0.1 netmask 0xffffffff broadcast 10.200.0.1 
groups: vlan 
vlan: 50 vlanpcp: 0 parent interface: lagg0 
fib: 1 
media: Ethernet autoselect 
status: active 
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

The key point here is the line in the interface’s output showing 10.200.0.1 is configured as an IP alias.  If I check FRR’s vtysh shell:

arkham.private.lateapex.net# show int lagg0.50 
Interface lagg0.50 is up, line protocol is up 
Link ups: 1 last: 2020/09/15 09:24:23.39 
Link downs: 0 last: (never) 
vrf: default 
index 7 metric 1 mtu 1500 speed 10000 
flags: <UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> 
Type: Unknown 
HWaddr: a0:36:9f:0d:ea:c0 
inet 10.0.0.2/24 
inet 10.200.0.1/32 unnumbered 
inet 108.28.193.210/32 unnumbered 
inet 108.28.193.212/32 unnumbered 
inet 108.28.193.213/32 unnumbered 
inet 108.28.193.217/32 unnumbered 
inet 108.28.193.219/32 unnumbered 
Interface Type Other 
input packets 8192879, bytes 32838236838, dropped 0, multicast packets 99463 
input errors 0 
output packets 7065411, bytes 1194089187, multicast packets 139 
output errors 0 
collisions 0

The line that reads:

inet 10.200.0.1/32 unnumbered

means that FRR is seeing the new /32. Cool.  If I had redistribute connected in BGP, my server would start announcing 10.200.0.1/32 to it.  Now let me remove it from the OS:

arkham# ifconfig lagg0.50 inet 10.200.0.1/32 delete 
arkham# ifconfig lagg0.50 | grep 10.200.0.1 
arkham#

Gone.  It doesn’t show up in the ifconfig output.  But how about in vtysh?

arkham.private.lateapex.net# show int lagg0.50
Interface lagg0.50 is up, line protocol is up
Link ups: 1 last: 2020/09/15 09:24:23.39
Link downs: 0 last: (never)
vrf: default
index 7 metric 1 mtu 1500 speed 10000
flags: <UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>
Type: Unknown
HWaddr: a0:36:9f:0d:ea:c0
inet 10.0.0.2/24
inet 10.200.0.1/32 unnumbered
inet 108.28.193.210/32 unnumbered
inet 108.28.193.212/32 unnumbered
inet 108.28.193.213/32 unnumbered
inet 108.28.193.217/32 unnumbered
inet 108.28.193.219/32 unnumbered
Interface Type Other
input packets 8193318, bytes 32838657020, dropped 0, multicast packets 99463
input errors 0
output packets 7065693, bytes 1194127091, multicast packets 139
output errors 0
collisions 0

It’s still there.  Which means it’s still being announced to my router even though the IP is no longer on the interface.

I joined the FRR slack channel this weekend and started chatting with the developers there.  Fortunately I know a few of them because they’re Cumulus guys.  Donald Sharp volunteered to run with the fix, and he figured it out pretty quickly.  He had a patch submitted to their git repository, awaiting approval.  It turns out it was a few lines in one of the C files that needed changing, so I hand-jammed them in myself.  Once I had FRR rebuilt on my server, it was doing exactly what I needed.

Now, with redistribute connected in the server’s BGP config, I can start and stop jails and have their IPs appear and disappear from the router.  For instance, my jail madhatter has IP 108.28.193.213, and is currently seen by the upstream router:

lateapex-gw.lateapex.net# show ip route 108.28.193.213/32
Routing entry for 108.28.193.213/32
Known via "bgp", distance 20, metric 1, best
Last update 02:36:53 ago
* 10.0.0.2, via ix1, weight 1

If I tell arkham to stop that jail:

arkham# service jail stop madhatter
Stopping jails: madhatter.
arkham#

I can check the router again to see if the /32 is still there:

lateapex-gw.lateapex.net# show ip route 108.28.193.213/32
% Network not in table

Gone.  Fire the jail back up and check the router:

arkham# service jail start madhatter
Starting jails: madhatter.
lateapex-gw.lateapex.net# show ip route 108.28.193.213/32
Routing entry for 108.28.193.213/32
Known via "bgp", distance 20, metric 1, best
Last update 00:00:16 ago
* 10.0.0.2, via ix1, weight 1

It’s back.

Cool.  Yes kids, routing on your servers is fun and useful.  Do it!

Leave a Reply