FreeBSD, Network Architecture, Server and OS

Cumulus VX on FreeBSD’s BHyve


Over the course of the last few weeks or so, I’ve been working with Cumulus Linux’s VX image on VirtualBox, on my Macbook Pro.  As my virtual lab grew from a few VX images to the count of ten or so, my laptop began to complain.  My laptop has a 4-core 2.8Ghz Core i7 with 16GB of RAM.  For most things, it’s perfectly OK.  But running ten 1GB VMs began to take its toll.  I really wanted a way to offload this lab to one of the servers in my basement, and I figured out a way.

FreeBSD Server

About three years ago, I built a new server here at home, and I went through the process in this post.  You’ll note in that entry that I decided to shun virtualizing all of my servers in favor of FreeBSD’s jail system.  At the time, the bhyve hypervisor wasn’t really where I wanted it to be as far as usability and ease of configuration.  And to be fair, I really didn’t need a hypervisor and a bunch of VMs when jails were (and still are) the better solution.

In fact, this blog is running in one of the jails.

Since then, FreeBSD has been updated quite a bit, and so has bhyve.  I’m not going to go into the specifics of upgrading my server, but suffice it to say: the first step was upgrading to FreeBSD 11.2 (from the 10.3 it was running… big jump).  That went without too many hitches, to be fair.

joker$ freebsd-version 

VirtualBox or Bhyve?

Due to my success with VBox on my Mac, I figured I would try to get everything moved over to FreeBSD and its VBox distribution.  There’s already a package set up to make installing VBox easy:

joker# pkg search virtualbox
phpvirtualbox-5.2.0_1          AJAX Web Interface for VirtualBox
virtualbox-ose-5.2.18_1        General-purpose full virtualizer for x86 hardware
virtualbox-ose-additions-5.2.18_1 VirtualBox additions for FreeBSD guests
virtualbox-ose-additions-nox11-5.2.18_1 VirtualBox additions for FreeBSD guests
virtualbox-ose-kmod-5.2.18_1   VirtualBox kernel module for FreeBSD
virtualbox-ose-nox11-5.2.18_1  General-purpose full virtualizer for x86 hardware

After installing the VBox package, I created a new VM using the same VBox image from Cumulus that I’d worked with on my Mac.  That installation went fine.  But when I tried to launch the VM, it failed outright.  VBox refuses to load a VM if any of VBox’s shared libraries are available over soft (logical) links.  Those libraries are all stored in /usr/local/lib, and on my system, /usr/local is a link to another filesystem: /local.

joker# cd /usr
joker# ls -lat local
lrwxr-xr-x  1 root  wheel  6 May  6  2016 local@ -> /local

And there’s nothing I can do about that.  It’s a moronic limitation, done with the concerns of “security” in mind.  Which is asinine.  This is UNIX, no?  Soft links are used everywhere.

The problem is that VBox on FreeBSD isn’t an officially supported product from Oracle.  And I didn’t want to spend time trying to hunt down the maintainers of the package, to see if they could fix this little annoyance.  So with that, I abandoned the plan to use VBox on FreeBSD.


Admittedly: my last exposure to bhyve was three years ago when I rebuilt the joker server in FreeBSD.  At the time, it wasn’t really ready for prime time, in my opinion.  So I was a bit hesitant to try it again.  Fortunately, this is for a lab, meaning it wouldn’t need to be rock solid.  And also: bhyve has been updated quite a bit in the past three years.  So perhaps it’s time to take another look.

Installing Bhyve

I followed Dan’s Blog entry to get the appropriate packages installed and the OS configured for use.  I won’t repeat the steps here; he did a great job with outlining what needs to be done.  What I didn’t bother with were all of the ZFS steps he followed.  I am using ZFS quite heavily on my server; in fact each jail running on it has its own ZFS volume.  But the intent was to just run the VMs out of raw disk image files, so having the ZFS volumes built wouldn’t be necessary.

Instead, I created a directory in /local called bhyve, where I intended to put all of the VMs.  And in the /etc/rc.conf file:

#BHyve stuff

Then from the CLI:

vm init

and I was ready to go.

The Cumulus Disk Image

Bhyve won’t work natively with disk images from other hypervisors.  Fortunately, it’s really easy to convert the images between the various hypervisor formats.  I downloaded the Cumulus 3.7 KVM image from their website; that comes in QCOW2 format.  It needs to be converted to a raw disk image with:

qemu-img convert -f qcow2 -O raw cumulus-image.qcow2 cumulus-image.raw

This took a bit of time; the raw file ends up being a little over 6GB in size.  But fortunately, it only needs to be done once.  From there, I had a base raw image that I could make lots of copies of; one for each VM.  But it also means that each VM would take up a little over 6GB of disk space; something to keep in mind.  My server has plenty of storage space available, so it’s not an issue for me.  But, again: something to keep in mind.

Now that I had the disk image ready, I went into the /local/bhyve directory and made a series of sub-directories.  When done:

joker# cd /local/bhyve/
joker# ls -l
total 56
drwxr-xr-x  2 root  wheel   5 Oct 27 13:33 .config/
drwxr-xr-x  2 root  wheel   2 Oct 26 11:19 .iso/
drwxr-xr-x  2 root  wheel  18 Oct 26 11:19 .templates/
drwxr-xr-x  2 root  wheel   8 Oct 27 12:37 agg/
drwxr-xr-x  2 root  wheel   8 Oct 27 13:17 leaf01/
drwxr-xr-x  2 root  wheel   8 Oct 27 13:17 leaf02/
drwxr-xr-x  2 root  wheel   8 Oct 27 13:17 leaf03/
drwxr-xr-x  2 root  wheel   8 Oct 27 13:36 leaf04/
drwxr-xr-x  2 root  wheel   8 Oct 27 13:05 server01/
drwxr-xr-x  2 root  wheel   9 Oct 27 13:05 server02/
drwxr-xr-x  2 root  wheel   8 Oct 27 13:05 server03/
drwxr-xr-x  2 root  wheel   8 Oct 27 12:18 spine01/
drwxr-xr-x  2 root  wheel   8 Oct 27 12:27 spine02/

The first three are created by the vm command.  I created the rest of them; each directory would represent a running Cumulus image when all was said and done.

In each sub-directory, I placed a copy of the new raw disk image and named it accordingly: sub-directory/sub-directory.img.  So, for instance, in the agg sub-directory, I placed a file called agg.raw.

VM Networking

Before I performed any VM networking, I needed to take care of the server’s real interface on my private network.  I wanted to bridge the VMs’ eth0 mgmt interface with my server’s physical interface, so that the VMs would have access to my internal network (and DHCP server, and web server, and …).  My server’s interface is a lagg of two Intel NICs: em0 and em1.  It’s suggested that if you’re going to do that, you move the physical interface’s IP address to the bridge interface.  So with that:

ifconfig bridge0
ifconfig bridge0 inet addm lagg1
ifconfig lagg1 delete

The last line does NOT delete the lagg interface.  It removes the IP configuration from it.  The second line puts the IP on the bridge0 interface and adds lagg1 to it.  Once these commands were done, bridge0 existed with a single lagg interface (which is actually 2 physicals) in it.

VM “Switches”

The vm-bhyve package that I installed earlier in this adventure provides a simple way to create virtual “switches” for bhyve to use.  These aren’t actual vswitches in the typical sense of a hypervisor; instead they’re just bridge interfaces.  Each VM that’s attached to a given switch does so through a FreeBSD tap interface.  The tap interfaces sort of correspond to the interfaces on each VM.  So if a VM has five different connected interfaces, it’ll have five different FreeBSD tap interfaces associated with it.

The first switch to create was the management switch.  I unimaginatively called it “private” (instead of “management”… duh).  Since I was going to do this manually and use the existing bridge0 interface that I already created, the command line looked like:

vm switch create -t manual -b bridge0 private

After that, I could attach VMs to the switch named “private” and they’d all have access to my internal LAN here at home.

I wasn’t done.  There are a lot of point-to-point connections in my little lab.  Each one would need a switch created so that the VMs could communicate with one another as if they were directly connected.  For example:

vm switch create spn01-lf01

created the bridge that VMs spine01 and leaf01 would use to connect to each other.  All said and done:

agg-spine01  standard  vm-agg-spine01  -        no       -    -     -
agg-spine02  standard  vm-agg-spine02  -        no       -    -     -
spn01-spn02  standard  vm-spn01-spn02  -        no       -    -     -
spn01-lf01   standard  vm-spn01-lf01   -        no       -    -     -
spn01-lf02   standard  vm-spn01-lf02   -        no       -    -     -
spn01-lf03   standard  vm-spn01-lf03   -        no       -    -     -
spn01-lf04   standard  vm-spn01-lf04   -        no       -    -     -
spn02-lf01   standard  vm-spn02-lf01   -        no       -    -     -
spn02-lf02   standard  vm-spn02-lf02   -        no       -    -     -
spn02-lf03   standard  vm-spn02-lf03   -        no       -    -     -
spn02-lf04   standard  vm-spn02-lf04   -        no       -    -     -
lf01-srv01   standard  vm-lf01-srv01   -        no       -    -     -
lf02-srv02   standard  vm-lf02-srv02   -        no       -    -     -
lf03-srv03   standard  vm-lf03-srv03   -        no       -    -     -
private      manual    bridge0         n/a      no       n/a  n/a   n/a
lf04-srv04   standard  vm-lf04-srv04   -        no       -    -     -

fifteen bridges created, ignoring the “private” one.  You can see the third column in the table is called IFACE.  That’s what references the actual bridge interface created in the host OS.

joker# ifconfig vm-spn01-lf01
vm-spn01-lf01: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 4a:78:50:ca:00:e2
	nd6 options=1<PERFORMNUD>
	groups: bridge vm-switch viid-1c2d6@ 
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	        ifmaxaddr 0 port 46 priority 128 path cost 2000000
	        ifmaxaddr 0 port 31 priority 128 path cost 2000000

And here, you can see that the bridge has two tap interfaces as part of it: tap22 and tap7.  I’ll get into those in a bit.

Why Use Separate Switches?

Good question.  Why did I create one switch per interconnect?  The answer is that I want to continue using things like BGP unnumbered in the Cumulus config as well as a few other “direct connect” technologies.  If I put all of the interfaces in the same bridge, which I could have done, then those technologies would get confused and not work properly.

VM Config File

I’ll admit: this took me the longest time to get working properly.  Mostly due to my own ignorance and not anything specifically wrong with bhyve or the Cumulus KVM image.  Here are the important bits from the /local/bhyve/agg/agg.conf file:


Remember from my former setup that my agg router has a management interface (eth0) and a direct uplink to the private network.  I did this so that I could route into and out of the entire lab network via the agg router’s swp1 interface, not its management eth0 interface.  That’s why the network0_switch and network1_switch are set to the same “private” switch; the one I created using the OS’ bridge0 interface.  The rest of the VMs will only have one connection that switch except the “servers”.  They won’t have a connection to it at all.

The next two interfaces, which are represented by network3 and network4 are attached to the appropriate point-to-point switches that I created previously.  Hopefully that’s pretty simple to understand.

The disk0 lines tell bhyve what kind of HDD device to use, and what image file.  As you can see: it’s agg.img; the one I created earlier.  Each VM’s config file will reference its own respective image file, of course.

The guest type is Linux, since Cumulus is Debian Linux.  And the loader… this one caused me a lot of unnecessary trouble and took the longest to figure out.  Again, that’s due to my own ignorance; ultimately “grub” is the right answer.  Key to making this work, though, is telling bhyve which partition the /boot directory is on.  Otherwise, grub won’t load properly and the OS can’t boot.  To determine this, I cheated and booted up one of my idle VBox images on my Mac.  It turns out the device is /dev/sda4, or the 4th partition.  Done.

Finally, you’ll see that I defined the MAC address for the first NIC.  I didn’t do this at first, and instead chose to edit my static DHCP reservations.  Bhyve will create new MAC addresses the first time the VM is booted.  But why keep editing the DHCP reservations with each first boot when I could just re-use the MAC addresses from my VBox VMs?  Duh.


The command:

vm start agg

fired the agg VM up after a lot of trial and error.  And once I had the config file nailed, the OS did finally boot up.  It’s a somewhat slow process the first time, but it did run.  As root from the host OS, I ran this to get to the console of the VM.  I tried logging in as “jvp” to see if the ZTP process was followed; unfortunately it wasn’t.  So I logged in as user “cumulus”:

joker# vm console agg

Debian GNU/Linux 8 agg ttyS0

agg login: cumulus
Last login: Sat Oct 27 14:58:56 UTC 2018 on ttyS0
Linux agg 4.1.0-cl-7-amd64 #1 SMP Debian 4.1.33-1+cl3u16 (2018-09-15) x86_64

Welcome to Cumulus VX (TM)

Cumulus VX (TM) is a community supported virtual appliance designed for
experiencing, testing and prototyping Cumulus Networks' latest technology.
For any questions or technical support, visit our community site at:

The registered trademark Linux (R) is used pursuant to a sublicense from LMI,
the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide

Did it get a DHCP address?

cumulus@agg:~$ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 58:9c:fc:02:84:7f  
          inet addr:  Bcast:  Mask:
          inet6 addr: fe80::5a9c:fcff:fe02:847f/64 Scope:Link
          RX packets:259290 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3629 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:72668538 (69.3 MiB)  TX bytes:733370 (716.1 KiB)

It did.  And interestingly, unlike the VX images running on my Mac’s VBox, this image properly interpreted the “host-name” option line in my DHCP file.  It set the hostname to agg.  With that I ran the commands:

ztp -R

And a minute or two later, I had a ZTP’d VX image running on bhyve.  Remember that as part of my ZTP script, I add my username and the SSH key I use on my laptop.  Well, can I ssh into the “new” agg router?

deadshot$ ssh agg

Welcome to Cumulus VX (TM)

Cumulus VX (TM) is a community supported virtual appliance designed for
experiencing, testing and prototyping Cumulus Networks' latest technology.
For any questions or technical support, visit our community site at:

The registered trademark Linux (R) is used pursuant to a sublicense from LMI,
the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide
Last login: Sat Oct 27 16:36:49 2018 from


Remember I mentioned the tap interfaces previously?  Well now that there’s a running VM, let’s ask the vm command about it:

joker# vm info agg | egrep network-interface -A 6
    number: 0
    emulation: e1000
    virtual-switch: private
    fixed-mac-address: 58:9c:fc:02:84:7f
    fixed-device: -
    active-device: tap0
    number: 1
    emulation: e1000
    virtual-switch: private
    fixed-mac-address: 58:9c:fc:0a:34:ae
    fixed-device: -
    active-device: tap1
    number: 2
    emulation: e1000
    virtual-switch: agg-spine01
    fixed-mac-address: 58:9c:fc:01:49:58
    fixed-device: -
    active-device: tap2
    number: 3
    emulation: e1000
    virtual-switch: agg-spine02
    fixed-mac-address: 58:9c:fc:0f:3b:ba
    fixed-device: -
    active-device: tap3

You can see from the output that each interface on the agg router has an associated tap interface assigned to it.  And you can look at them from FreeBSD if you want to:

joker# ifconfig tap0
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: vmnet-agg-0-private
	ether 00:bd:39:d5:9b:00
	hwaddr 00:bd:39:d5:9b:00
	inet6 fe80::2bd:39ff:fed5:9b00%tap0 prefixlen 64 tentative scopeid 0xa 
	media: Ethernet autoselect
	status: active
	groups: tap vm-port 
	Opened by PID 88452

Agg Router: Static Config

Remember that originally my intent with the agg router is to mimic a router at the “top” of the data center; one that may or may not (likely not) be running Cumulus.  So it has IP addresses on its downlinks to the spines, and a static pointed at my internal network.  I also did not write any ansible playbooks for it; instead figuring it would only take me a few minutes to configure it up again if I ever had to.

That was the next step: configuring the agg router with IP addresses, BGP, etc.  All pretty easy, and all outlined in my previous blog entries.

The Servers: Static Config

My next step was to set up something that would indicate total success in this endeavor: the servers.  Like my previous work, the servers (which are Cumulus with a single eth0 interface) would be statically configured.  There’d be one hanging off leaf nodes 1, 2, and 3.  Their address range would be, and VXLAN would provide their L2 broadcast domain across the leaf nodes.

Since they wouldn’t be DHCP’ing out to get an IP, I wasn’t worried about what MAC addresses bhyve would assign to them.  I whipped up their respective config files, using the agg.conf as a guide.  But the main difference, of course: network0_switch would be the VM’s specific bridge to its uplinked switch.  So, for instance:


for server01’s network.

These booted right up, and required some manual config over the console.  But that was fine.  To each, I added:

  • IP address
  • Default route
  • Hostname
  • My “jvp” user and SSH key

And with that, I left the VMs running but ignored them.

The Spines: Ansible

I spent the entirety of the last entry discussing learning ansible and configuring the leaf nodes with it.  I didn’t do anything with the spine nodes.  I knew going into it that some stuff would need to be added to my inventory file, and I might need to write a couple of different “roles” for the interface and BGP configurations respectively.  They’d be a bit different than the leaf nodes.

In the cumulus directory of my laptop:

mkdir -p roles/spine-bgp/tasks
mkdir -p roles/spine-interfaces/tasks

I changed the hosts inventory file to look like this; note the new variable at the end of each spine line:

spine01 lo0=	hostname=spine01	vl100=	swp1=
spine02 lo0=	hostname=spine02	vl100=	swp1=

leaf01 lo0=	hostname=leaf01		vl100=
leaf02 lo0=	hostname=leaf02		vl100=
leaf03 lo0=	hostname=leaf03		vl100=
leaf04 lo0=	hostname=leaf04		vl100=

The reason: remember that the uplink router for the spines isn’t acting like a cumulus device, and therefore I can’t guarantee the whole BGP unnumbered bit is even present.  In other words: interface swp1 on both spines would need a static IP.

The setup.yaml file was edited for the spines, with a whole new -hosts: section:

- hosts: leaf
  become: true
  - loopback
  - interfaces
  - routing
  - bgp
  - mgmt

- hosts: spine
  become: true
  - loopback
  - spine-interfaces
  - routing
  - spine-bgp
  - mgmt

As you can see, where it differs from the leaf entry is the call to spine-interfaces and spine-bgp.  Let’s look at those files.

The spine-interfaces File

  - name: Configure swp interfaces
      - add interface swp1-6
      - add interface swp1 ip address {{ swp1 }}/31
      - add vxlan vni10100 vxlan id 10100
      - add bridge bridge ports vni10100
      - add bridge bridge vids 100
      - add bridge bridge vlan-aware
      - add interface swp2-6 mtu 9216
      - add vlan 100 ip address {{ vl100  }}/24
      - add vlan 100 vlan-id 100
      - add vlan 100 vlan-raw-device bridge
      - add vxlan vni10100 mtu 9000
      - add vxlan vni10100 vxlan local-tunnelip {{ lo0 }}
      - add vxlan vni10100 bridge access 100
      - add vxlan vni10100 bridge arp-nd-suppress on
      - add vxlan vni10100 bridge learning off 
      atomic: true

Fairly similar to the leaf nodes’ interface file, except it’s setting configs for interfaces swp1-6.  The first interface gets a static IP noted by the {{ swp1 }} variable.  The MTUs on interfaces swp2-6 are set to 9216 and I’m not adding an access port like I am on the leaf nodes.  Finally, the VLAN100 interface doesn’t need an Anycast IP for the servers to use as a default route; they’ll be using thelr locally-attached leaf switch for that instead.  I just have an IP on the VLAN100 interface so that the spines can perform VXLAN routing to get packets to the servers more efficiently.

The spine-bgp File

  - name: BGP with EVPN
      - add bgp autonomous-system 65200
      - add bgp router-id {{ lo0 }}
      - add bgp bestpath as-path multipath-relax
      - add bgp neighbor leaf peer-group
      - add bgp neighbor leaf remote-as external
      - add bgp neighbor leaf bfd
      - add bgp neighbor swp1 interface
      - add bgp neighbor swp1 remote-as external
      - add bgp neighbor swp1 capability extended-nexthop
      - add bgp neighbor swp1 bfd
      - add bgp neighbor swp2 interface
      - add bgp neighbor swp2 remote-as internal
      - add bgp neighbor swp2 next-hop-self
      - add bgp neighbor swp2 bfd
      - add bgp neighbor swp3 interface peer-group leaf
      - add bgp neighbor swp4 interface peer-group leaf
      - add bgp neighbor swp5 interface peer-group leaf
      - add bgp neighbor swp6 interface peer-group leaf
      - add bgp ipv4 unicast redistribute connected
      - add bgp l2vpn evpn  neighbor leaf activate
      - add bgp l2vpn evpn neighbor swp2 activate
      - add bgp l2vpn evpn  advertise-all-vni
      atomic: true

  - name: reload FRR
        name: frr
        state: restarted

Again, a bit different from the leaf nodes.  In this case I needed to create the leaf peer-group, set up the uplink peer and the cross-connect peer, then use the peer-group with interfaces swp3-6.  What I didn’t need to do is add the leaf node’s “allow ASN in” line for EVPN since the spines wouldn’t be announcing locally-known MAC addresses in EVPN.

Executing Ansible on Spines

From my laptop, the command:

ansible-playbook setup.yaml -i hosts -l spine -K

prompted me for my sudo password, and then quickly configured the two spines.  I wish I’d saved the output from the command; unfortunately I didn’t think to.  But suffice to say: it worked.  I had two running spines in almost no time flat.  No manual configuration required at all.

The Leaf Nodes: Ansible

There’s no need to go through the details of the ansible configuration for the leaf nodes.  From my previous blog entry, we know it already works.  With the lead nodes booted and already ZTP’d, this command:

ansible-playbook setup.yaml -i hosts -l leaf -K

did the deed.

Checking The Work

I was almost tempted to just blindly try and ssh into one of the servers to see if it worked, but I wanted to make sure the leaf nodes and spines were talking to each other.  This was pretty easy to verify:

root@spine01:mgmt-vrf:/home/jvp# net show bgp sum

show bgp ipv4 unicast summary
BGP router identifier, local AS number 65200 vrf-id 0
BGP table version 48
RIB entries 21, using 3192 bytes of memory
Peers 6, using 116 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd
leaf01(swp3)    4      65201   30997   31033        0    0    0 1d01h47m            2
leaf02(swp4)    4      65201   31003   31033        0    0    0 1d01h47m            2
leaf03(swp5)    4      65201   31003   31033        0    0    0 1d01h47m            2
leaf04(swp6)    4      65201   29814   29826        0    0    0 23:46:40            2
agg(swp1)       4      65210   32223   32211        0    0    0 00:11:20            5
spine02(swp2)   4      65200   31373   31420        0    0    0 1d02h07m           10

Total number of neighbors 6

show bgp ipv6 unicast summary
% No BGP neighbors found

show bgp l2vpn evpn summary
BGP router identifier, local AS number 65200 vrf-id 0
BGP table version 0
RIB entries 11, using 1672 bytes of memory
Peers 6, using 116 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd
leaf01(swp3)    4      65201   30997   31034        0    0    0 1d01h47m            4
leaf02(swp4)    4      65201   31003   31033        0    0    0 1d01h47m            4
leaf03(swp5)    4      65201   31003   31033        0    0    0 1d01h47m            4
leaf04(swp6)    4      65201   29814   29826        0    0    0 23:46:40            1
spine02(swp2)   4      65200   31373   31420        0    0    0 1d02h07m           14

Total number of neighbors 5

Looks like it.  Well, let’s SSH into one of the servers from my laptop and see if it actually worked:

deadshot$ ssh

Welcome to Cumulus VX (TM)

Cumulus VX (TM) is a community supported virtual appliance designed for
experiencing, testing and prototyping Cumulus Networks' latest technology.
For any questions or technical support, visit our community site at:

The registered trademark Linux (R) is used pursuant to a sublicense from LMI,
the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide
Last login: Sun Oct 28 15:37:50 2018 from

That’s certainly promising.  I already know the answer, but let’s make sure we have L2 between the servers:

jvp@server02:~$ ping
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=7.04 ms
64 bytes from icmp_seq=2 ttl=64 time=3.40 ms
64 bytes from icmp_seq=3 ttl=64 time=2.65 ms
--- ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 2.656/4.369/7.047/1.918 ms
jvp@server02:~$ arp -an
? ( at 58:9c:fc:00:5e:43 [ether] on eth0

Boom.  Done.

A Note About LLDP

LLDP won’t work properly between the VMs with this setup.  The FreeBSD bridge interfaces that the VMs are using as “virtual cables” will just eat the LLDP packets that each switch sends out.  The pair switch won’t ever see them, so no LLDP:

root@spine01:mgmt-vrf:/home/jvp# net show int
State  Name      Spd  MTU    Mode           LLDP  Summary
-----  --------  ---  -----  -------------  ----  ---------------------------
UP     lo        N/A  65536  Loopback             IP:
       lo                                         IP:
       lo                                         IP: ::1/128
UP     eth0      1G   1500   Mgmt                 Master: mgmt(UP)
       eth0                                       IP:
UP     swp1      1G   1500   Interface/L3         IP:
UP     swp2      1G   9216   NotConfigured
UP     swp3      1G   9216   NotConfigured
UP     swp4      1G   9216   NotConfigured
UP     swp5      1G   9216   NotConfigured
UP     swp6      1G   9216   NotConfigured
UP     bridge    N/A  9000   Bridge/L2
UP     mgmt      N/A  65536  Interface/L3         IP:
UP     vlan100   N/A  9000   Interface/L3         IP:
UP     vni10100  N/A  9000   Access/L2            Master: bridge(UP)

You’ll see the “LLDP” column is completely blank.  At this time, and with FreeBSD as the host OS, I don’t know any way around this.  There may not be, unfortunately.  But, for my lab use, this isn’t a deal breaker.  Slightly annoying, yes.  But not bhyve’s fault; more a fault with how FreeBSD handles bridge interfaces.


My goal with this exercise was to move my 10-node “lab” from my Macbook Pro laptop over to my FreeBSD server.  The server has significantly more resources available to it and doesn’t overheat or page-fault itself into oblivion when the VMs are running, unlike my Mac.  This all required me to learn about the bhyve hypervisor and deploying Linux images to it.  Fortunately, it’s not difficult at all once you realize how to edit the config files.  And since I did all that work with ansible in my previous blog entry, re-deploying these virtual devices was a matter of a reboot and running a command.

Easy peazy.

agg       default    grub    1    1024M   -    Yes [1]    Running (88452)
leaf01    default    grub    1    1024M   -    No         Running (93524)
leaf02    default    grub    1    1024M   -    No         Running (93604)
leaf03    default    grub    1    1024M   -    No         Running (93648)
leaf04    default    grub    1    1024M   -    No         Running (96064)
server01  default    grub    1    1024M   -    No         Running (20715)
server02  default    grub    1    1024M   -    No         Running (12516)
server03  default    grub    1    1024M   -    No         Running (21741)
spine01   default    grub    1    1024M   -    No         Running (85241)
spine02   default    grub    1    1024M   -    No         Running (86359)

Leave a Reply