Table Of Content
In a previous entry, I chronicled my self-education regarding Cumulus’ OS and how to do EVPN VXLAN with it. Throughout this entry, I’ll be referring back to that document, including some of the diagrams, configurations, etc. This entry will explain another use for VXLAN, and that’s running it instead of MPLS to handle L2VPNs across a network. That network can be intra-data center, inter-data center, or even via a backbone. VXLAN is vastly more simple to configure than MPLS is, and VXLAN is being included in more and more network devices and operating systems.
What this document won’t touch upon is the Traffic Engineering aspect of MPLS; VXLAN can’t help with that at all. So if you’re using MPLS for its TE capabilities, this won’t apply. My personal opinion, though, is that if you need TE on your backbone: you’re doing it wrong.
Also of note: I’m not the first guy to think of this. I’m just going to provide my experimentation and thought processes behind it.
Using EVPN and VXLAN
Initially, I’m going to stick with the network architecture from the previous blog entry. It looks like this, but I’ll be making some changes to it:
Agg Router Becomes Customer Router
The first thing I did was break the IP connectivity between the Agg router and Router 1. That also broke the EBGP peer between those two routers. I left the connectivity up between the Agg and Router 2 because I wanted a way to be SSH into the system, still. But ultimately, I need that link between the Agg and Router 1 to act like a link between your backbone and a customer’s router.
The change to /etc/network/interfaces was focused only on interface swp5:
auto swp5 iface swp5 address 172.16.0.100/24
You’ll note from the previous entry that 172.16.0.0/24 is part of VLAN 100 for my “servers”. I’ll be making use of that.
The change to /etc/frr/frr.conf was also simple: I removed the swp5 interface peer from the “spine” group, and added it to a newly-created “over-vxlan” group:
neighbor over-vxlan peer-group neighbor over-vxlan remote-as external neighbor over-vxlan default-originate ! neighbor swp5 interface peer-group over-vxlan
SV1 Becomes Customer Router
Once done, I needed to promote the server SV1 to router status. Since it’s just running Cumulus’ OS, that was pretty easy to do. Since it’ll be running BGP, it needs a loopback for its router ID. In /etc/network/interfaces:
auto lo iface lo inet loopback address 10.100.0.20/32
Like the previous Cumulus devices that are routing, I had to change the /etc/frr/daemons file to include BGP. I’m not going to show the config snippet here, though; it’s pretty self-explanatory. But the /etc/frr/frr.conf:
router bgp 65230 bgp router-id 10.100.0.20 redistribute connected neighbor over-vxlan peer-group neighbor over-vxlan remote-as external ! neighbor swp4 interface peer-group over-vxlan
Router 1 Changes
Because Router 1 is part of an EVPN VXLAN network, it’s basically ready to go. But a simple /etc/network/interfaces change had to be made to put the interface facing the Agg router into VLAN 100:
auto swp7 iface swp7 bridge-access 100 auto bridge iface bridge bridge-ports vni10100 swp7 bridge-vids 100 bridge-vlan-aware yes
Agg and SV1 “Directly” Connected
With this done, interface swp7 is now an Access port on VLAN 100. And within short order, the Agg router and SV1 could see each other directly and form an EBGP peer:
root@agg:/etc/network# ping 172.16.0.10 PING 172.16.0.10 (172.16.0.10) 56(84) bytes of data. 64 bytes from 172.16.0.10: icmp_seq=1 ttl=64 time=1.03 ms 64 bytes from 172.16.0.10: icmp_seq=2 ttl=64 time=1.00 ms 64 bytes from 172.16.0.10: icmp_seq=3 ttl=64 time=1.01 ms --- 172.16.0.10 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.000/1.018/1.035/0.014 ms root@agg:/etc/network# arp -an 172.16.0.10 ? (172.16.0.10) at 08:00:27:36:62:99 [ether] on swp5
root@agg:/etc/network# net show bgp sum show bgp ipv4 unicast summary ============================= BGP router identifier 10.100.0.10, local AS number 65210 vrf-id 0 BGP table version 8 RIB entries 15, using 2280 bytes of memory Peers 2, using 39 KiB of memory Peer groups 2, using 128 bytes of memory Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd sv1(swp5) 4 65230 867 870 0 0 0 00:42:27 2 router2(swp6) 4 65200 866 867 0 0 0 00:42:50 5 Total number of neighbors 2 show bgp ipv6 unicast summary ============================= show bgp l2vpn evpn summary ===========================
root@agg:/etc/network# net show route 10.100.0.20 RIB entry for 10.100.0.20 ========================= Routing entry for 10.100.0.20/32 Known via "bgp", distance 20, metric 0, best Last update 00:43:00 ago * fe80::a00:27ff:fe36:6299, via swp5 FIB entry for 10.100.0.20 ========================= 10.100.0.20 via 169.254.0.1 dev swp5 proto bgp metric 20 onlink
Success. The Agg and SV1 routers have L2 knowledge of each other, as shown by the arp command. And they have an EBGP peer with one another as well. I can show that even further with a traceroute from the host Mac, to the loopback on SV1:
$ traceroute -n 10.100.0.20 traceroute to 10.100.0.20 (10.100.0.20), 64 hops max, 52 byte packets 1 192.168.10.80 0.340 ms 0.111 ms 0.116 ms 2 10.100.0.20 0.878 ms 0.571 ms 0.610 ms
Two hops: Agg and then SV1. The inter-connecting EVPN network isn’t seen at all.
VXLAN Without EVPN
The previous example was easy for me to whip up because I already had the EVPN lab configured the way I wanted it. VXLAN was working perfectly, and I just needed to change a couple of configuration bits on a few of the virtual devices. What if your backbone can’t support an EVPN control plane across it? There are numerous reasons why that might be, but it’s not actually a big deal. This is where unicast VXLAN configurations can come into to play. And, quite frankly, it’s far more likely to work on a backbone than an EVPN control plane would.
I’m not going to show configuration snippets, but it should be pretty simple to imagine at this point. Imagine a backbone with two customers, each having two different buildings. Each customer needs L2 connectivity between their respective buildings. Well, that’s pretty easy assuming the device on your backbone that’s connected to the customers supports unicast VXLAN:
Each device labeled “Router” in that diagram is something that can support unicast VXLAN, and the dotted arrows represent VXLAN tunnels. As long as it has connectivity with the respective router at the other site, the VXLAN tunnel will stay up. You can set the egress port on your router to be in any VLAN, as long as it’s the same VLAN on the other side. It might also make sense from a management perspective to keep each customer in their own VLAN. For example: Customer 1 is in VLAN 100, 2 is in VLAN 200, etc. This will become more important and, actually, required, when you start using a router to talk to more than one customer.
The benefit of this configuration is that no other device on your entire network has to know anything about VXLAN. They just need to forward packets accordingly. The only caveat to this is that you’ll need to remember to configure all possible links with jumbo frames because of the overlay overhead. But you’re doing that across your backbone already, anyway. Right?