FreeBSD, Network Architecture, Server and OS

Running FRR On Home Router


I’ve been building up my Cumulus Linux and FreeBSD virtual network over the course of the last several months.  As you’ll recall from an earlier post, I decided to put a single aggregation router at the top of the network as an ingress and egress point.  This allowed me to put a single static route for the entire virtual LAN and set it to the interface on the router.  And it also allowed me access to the network even if I had to reboot one of the spines.  However, with my recent work adding Free Range Routing (FRR) to my FreeBSD VMs, I thought of a different way to handle ingress and egress and remove a router hop.

The Virtual Network: Before

The following image has been used a bunch of times in my blog posts, but here’s what the “before” looked like:

The AGG router at the top of the network served as the ingress and egress for that virtual setup.  I had a set of static routes on my host server as well as my home router, and even on my laptop all pointing towards the interface on the AGG.  The AGG interface served as a single point for those static routes to be set to, and I knew I wouldn’t be doing much with the AGG router at all once it was up and running.  All of my work with EVPN, VXLAN, etc would happen at the layer below the AGG.  The advantage there is that the static routes wouldn’t need to change if I, say, had to reboot SPINE01.

FRR On The Router

After finishing up all of the work on my previous posts which included installing Free Range Routing on FreeBSD, I decided to try it on my router as well.  I rebuilt my router as a FreeBSD device a few years ago, as outlined in this post.  Since I knew how easy it was to get FRR running on FreeBSD, I installed it on the router.  Once FRR was installed, I had to figure out how I wanted to peer it with the virtual network, how I wanted to handle the routing, etc.

The Physical Layout

The physical layout looks roughly like this:

The router is connected to the same L2 switch as my FreeBSD hypervisor is.  You can see that I have EBGP arrows drawn between the router and the two spine VMs.  I’ll explain that in a bit.  But ultimately, the internal interfaces on both the server and router are on the same VLAN, of course.  And because of that, the router has L2 connectivity into the virtual network due to the bridging on the physical server.  Since L2 exists, it’ll be simple to get the routers talking BGP with one another.

Router Configuration


The router already has an IP on my internal 192.168 VLAN.  Each of the VMs on my server also have their Management Ethernet interface on that same VLAN, so that I can perform maintenance.  I decided to add an alias to the router’s interface on a completely different L3 broadcast domain.  Since I planned to have three interfaces on that broadcast domain, I needed to use a /29.  In the /etc/rc.conf:

ifconfig_em0_alias0=" netmask"

And then from the CLI so that I didn’t have to restart anything:

# ifconfig em0 inet alias


The intent was to peer two different spine routers with this device, which would make unnumbered not possible.  Therefore, the /usr/local/etc/frr/bgpd.conf:

router bgp 65000
 bgp router-id
 bgp bestpath as-path multipath-relax
 neighbor remote-as 65200
 neighbor remote-as 65200
 address-family ipv4 unicast
  neighbor default-originate
  neighbor soft-reconfiguration inbound
  neighbor default-originate
  neighbor soft-reconfiguration inbound

Pretty simple: EBGP peer with two down-stream IP addresses and set EBGP multi-path.  That last bit didn’t work, and I’ll explain why in a moment.

The Virtual Network: After

As you can see from the above diagram, it looks pretty similar to the “Before”, except the AGG layer is gone.  Instead, the swp1 interfaces on the two SPINE routers that were connected to the AGG router are now bridged into the host’s Ethernet interface.

VM Config Change

The VM configuration change to make this happen was very easy.  In the config files for both of the spine VMs, I merely changed which switch their “network1” interfaces were connected to.  Previously, they were connected to switches that were shared with the AGG router.  But after the change:


That “private” switch is the one that’s shared with the hypervisor’s physical Ethernet interface.  In other words: the VMs would be directly connected to my internal private VLAN.

Spine Config Changes

The config changes for the spines were handled by Ansible, of course.

Host File

The lines for the two spine routers were changed to reflect the IP addresses that would live on the swp1 interfaces:

spine01 lo0= hostname=spine01 vl100= swp1=
spine02 lo0= hostname=spine02 vl100= swp1=

Spine Interface File

I had to change the subnet on the swp1 interface from a /31 to a /29:

      - add interface swp1 ip address {{ swp1 }}/29

Spine BGP File

Since both spines will be peering with the same IP address on the router, their respective BGP configurations would be identical:

      - add bgp neighbor remote-as 65000
      - add bgp neighbor capability extended-nexthop
      - add bgp neighbor bfd

Cheating: Reseting Spine Configs Manually

In order for me to fire off the above Ansible changes and have them work properly, I would also need to write a series of commands to reset the previous configurations.  In other words, remove the previous swp1 IP address and remove the previous BGP peer with the AGG router.  Instead of spending the time writing those Ansible “reset” lines, I just went to each spine (there are, after all, only two of them) and manually reset those configurations.  I changed /etc/network/interfaces and /etc/frr/frr.conf respectively, and then:

# ifreload -a
# service frr restart

on each.  Done.  Once that was done, I fired off the Ansible playbook command to get the interfaces and BGP configurations added.

Did It Work?

It did.  Sort of.  From the respect of the spine routers it’s perfect.  They’ve peered right up with my home router and are exchanging routes without an issue:

root@spine01:mgmt-vrf:/home/jvp# net show bgp sum
show bgp ipv4 unicast summary
BGP router identifier, local AS number 65200 vrf-id 0
BGP table version 27
RIB entries 35, using 5320 bytes of memory
Peers 5, using 96 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor                             V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd
leaf01(swp3)                         4      65201   22607   22608        0    0    0 18:49:17            5
leaf02(swp4)                         4      65201   22605   22606        0    0    0 18:49:15            5
leaf03(swp5)                         4      65201   22609   22610        0    0    0 18:49:24            5
leaf04(swp6)                         4      65201   22608   22608        0    0    0 18:49:18            5
lateapex-gw(              4      65000   22286   22609        0    0    0 18:43:06            3

Total number of neighbors 5


root@spine01:mgmt-vrf:/home/jvp# net show route
RIB entry for
Routing entry for
  Known via "bgp", distance 20, metric 0, best
  Last update 18:43:45 ago
  *, via swp1

FIB entry for
default via dev swp1  proto bgp  metric 20

However, remember I mentioned the EBGP multi-path not really working properly on my router?  Well, it’s due to the fact that the FreeBSD kernel, by default, doesn’t support ECMP.  A compile option needs to be added at build time in order to support it, and I’m not concerned enough to try and do that.  My router has a very slow, 2-core CPU in it, so building a new kernel on it isn’t really high on my priority list.  Since the kernel doesn’t support ECMP by default, the pre-built FRR packages for FreeBSD don’t, either.  So while there are no errors thrown in BGP when I attempted to enable ECMP, only one route was installed:

lateapex-gw# show ip bgp sum

IPv4 Unicast Summary:
BGP router identifier, local AS number 65000 vrf-id 0
BGP table version 18
RIB entries 35, using 5600 bytes of memory
Peers 2, using 27 KiB of memory

Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd      4      65200   22765   22439        0    0    0 18:57:50           16      4      65200   22765   22439        0    0    0 18:57:50           17

Total number of neighbors 2

lateapex-gw# show ip bgp
BGP routing table entry for
Paths: (2 available, best #2, table Default-IP-Routing-Table)
  Advertised to non peer-group peers:
  65200 65201 65300 (metric 1) from (
      Origin IGP, valid, external
      AddPath ID: RX 0, TX 34
      Last update: Fri May 31 15:54:37 2019

  65200 65201 65300 (metric 1) from (
      Origin IGP, valid, external, best
      AddPath ID: RX 0, TX 17
      Last update: Fri May 31 15:54:37 2019

lateapex-gw# show ip route
Routing entry for
  Known via "bgp", distance 20, metric 0, best
  Last update 18:58:18 ago
  *, via em0

As you can see from the output above, I checked the forwarding table for the Anycast route I’ve been working with over the past few posts:  The router sees two BGP announcements for it, and they’re actually identical from an AS path perspective.  But because I don’t have ECMP enabled in FRR (or the router’s kernel), BGP’s best path selection went all the way to the lowest router ID of

Ultimately, that won’t make or break my lab network.  If I have to reboot spine01 for some reason, or re-do its BGP, or whatever, then the peer with spine02 will take over and I’ll still be able to get to the virtual network.  So BGP is still the right solution here, it’s just not quite as elegant as it would be if ECMP were supported.


Leave a Reply