Introduction

In a previous entry, I chronicled my self-education regarding Cumulus’ OS and how to do EVPN VXLAN with it.  Throughout this entry, I’ll be referring back to that document, including some of the diagrams, configurations, etc.  This entry will explain another use for VXLAN, and that’s running it instead of MPLS to handle L2VPNs across a network.  That network can be intra-data center, inter-data center, or even via a backbone.  VXLAN is vastly more simple to configure than MPLS is, and VXLAN is being included in more and more network devices and operating systems.

What this document won’t touch upon is the Traffic Engineering aspect of MPLS; VXLAN can’t help with that at all.  So if you’re using MPLS for its TE capabilities, this won’t apply.  My personal opinion, though, is that if you need TE on your backbone: you’re doing it wrong.

Also of note: I’m not the first guy to think of this.  I’m just going to provide my experimentation and thought processes behind it.

Using EVPN and VXLAN

Initially, I’m going to stick with the network architecture from the previous blog entry.  It looks like this, but I’ll be making some changes to it:

Agg Router Becomes Customer Router

The first thing I did was break the IP connectivity between the Agg router and Router 1.  That also broke the EBGP peer between those two routers.  I left the connectivity up between the Agg and Router 2 because I wanted a way to be SSH into the system, still.  But ultimately, I need that link between the Agg and Router 1 to act like a link between your backbone and a customer’s router.

The change to /etc/network/interfaces was focused only on interface swp5:

auto swp5
iface swp5
    address 172.16.0.100/24

You’ll note from the previous entry that 172.16.0.0/24 is part of VLAN 100 for my “servers”.  I’ll be making use of that.

The change to /etc/frr/frr.conf was also simple: I removed the swp5 interface peer from the “spine” group, and added it to a newly-created “over-vxlan” group:

	neighbor over-vxlan peer-group
	neighbor over-vxlan remote-as external
	neighbor over-vxlan default-originate
!
	neighbor swp5 interface peer-group over-vxlan

SV1 Becomes Customer Router

Once done, I needed to promote the server SV1 to router status.  Since it’s just running Cumulus’ OS, that was pretty easy to do.  Since it’ll be running BGP, it needs a loopback for its router ID.  In /etc/network/interfaces:

auto lo
iface lo inet loopback
	address 10.100.0.20/32

Like the previous Cumulus devices that are routing, I had to change the /etc/frr/daemons file to include BGP.  I’m not going to show the config snippet here, though; it’s pretty self-explanatory.  But the /etc/frr/frr.conf:

router bgp 65230
	bgp router-id 10.100.0.20
	redistribute connected

	neighbor over-vxlan peer-group
	neighbor over-vxlan remote-as external
!
	neighbor swp4 interface peer-group over-vxlan

Router 1 Changes

Because Router 1 is part of an EVPN VXLAN network, it’s basically ready to go.  But a simple /etc/network/interfaces change had to be made to put the interface facing the Agg router into VLAN 100:

auto swp7
iface swp7
	bridge-access 100

auto bridge
iface bridge
    bridge-ports vni10100 swp7
    bridge-vids 100
    bridge-vlan-aware yes

Agg and SV1 “Directly” Connected

With this done, interface swp7 is now an Access port on VLAN 100.  And within short order, the Agg router and SV1 could see each other directly and form an EBGP peer:

root@agg:/etc/network# ping 172.16.0.10
PING 172.16.0.10 (172.16.0.10) 56(84) bytes of data.
64 bytes from 172.16.0.10: icmp_seq=1 ttl=64 time=1.03 ms
64 bytes from 172.16.0.10: icmp_seq=2 ttl=64 time=1.00 ms
64 bytes from 172.16.0.10: icmp_seq=3 ttl=64 time=1.01 ms

--- 172.16.0.10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.000/1.018/1.035/0.014 ms

root@agg:/etc/network# arp -an 172.16.0.10
? (172.16.0.10) at 08:00:27:36:62:99 [ether] on swp5

root@agg:/etc/network# net show bgp sum

show bgp ipv4 unicast summary
=============================
BGP router identifier 10.100.0.10, local AS number 65210 vrf-id 0
BGP table version 8
RIB entries 15, using 2280 bytes of memory
Peers 2, using 39 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd
sv1(swp5)       4      65230     867     870        0    0    0 00:42:27            2
router2(swp6)   4      65200     866     867        0    0    0 00:42:50            5

Total number of neighbors 2


show bgp ipv6 unicast summary
=============================


show bgp l2vpn evpn summary
===========================

root@agg:/etc/network# net show route 10.100.0.20
RIB entry for 10.100.0.20
=========================
Routing entry for 10.100.0.20/32
  Known via "bgp", distance 20, metric 0, best
  Last update 00:43:00 ago
  * fe80::a00:27ff:fe36:6299, via swp5


FIB entry for 10.100.0.20
=========================
10.100.0.20 via 169.254.0.1 dev swp5  proto bgp  metric 20 onlink

Success.  The Agg and SV1 routers have L2 knowledge of each other, as shown by the arp command.  And they have an EBGP peer with one another as well.  I can show that even further with a traceroute from the host Mac, to the loopback on SV1:

$ traceroute -n 10.100.0.20
traceroute to 10.100.0.20 (10.100.0.20), 64 hops max, 52 byte packets
 1  192.168.10.80  0.340 ms  0.111 ms  0.116 ms
 2  10.100.0.20  0.878 ms  0.571 ms  0.610 ms

Two hops: Agg and then SV1.  The inter-connecting EVPN network isn’t seen at all.

VXLAN Without EVPN

The previous example was easy for me to whip up because I already had the EVPN lab configured the way I wanted it.  VXLAN was working perfectly, and I just needed to change a couple of configuration bits on a few of the virtual devices.  What if your backbone can’t support an EVPN control plane across it?  There are numerous reasons why that might be, but it’s not actually a big deal.  This is where unicast VXLAN configurations can come into to play.  And, quite frankly, it’s far more likely to work on a backbone than an EVPN control plane would.

I’m not going to show configuration snippets, but it should be pretty simple to imagine at this point.  Imagine a backbone with two customers, each having two different buildings.  Each customer needs L2 connectivity between their respective buildings.  Well, that’s pretty easy assuming the device on your backbone that’s connected to the customers supports unicast VXLAN:

Each device labeled “Router” in that diagram is something that can support unicast VXLAN, and the dotted arrows represent VXLAN tunnels.  As long as it has connectivity with the respective router at the other site, the VXLAN tunnel will stay up.  You can set the egress port on your router to be in any VLAN, as long as it’s the same VLAN on the other side.  It might also make sense from a management perspective to keep each customer in their own VLAN.  For example: Customer 1 is in VLAN 100, 2 is in VLAN 200, etc.  This will become more important and, actually, required, when you start using a router to talk to more than one customer.

The benefit of this configuration is that no other device on your entire network has to know anything about VXLAN.  They just need to forward packets accordingly.  The only caveat to this is that you’ll need to remember to configure all possible links with jumbo frames because of the overlay overhead.  But you’re doing that across your backbone already, anyway.  Right?