Mvendor-VXLAN-lab
Introduction
These labs were realized end of 2020. So different behaviors could be observed today with last releases, in a good or a bad way.
Archives posted are not production-ready and can carry unused/incomplete config parts
Analyzes and conclusions are not an absolute truth. So if you have remarks/advices, please share :)
Lab Target
The target of the lab is to build a multivendor lab, with an architecture reflecting those deployed in customers' environment.
This lab will be constructed in several steps, starting in a virtual version, followed by a physical one.
For the moment, we are working on typical "modern" datacenters implementations :
- EVPN/VXLAN layer 2 only
- EVPN/VXLAN layer 2 + layer 3 per rack (ou layer 3 anycast gw)
- EVPN within MPLS en L2/L3
- "standard" IP Fabric
Host specifications
Server : - Intel Xeon E5-1660v3 - 64GB DDR4 ECC 2133MHz - 2x HDD SATA 4TB Datacenter Class Soft RA System : VMware ESXi 6.7 U3
Software versions :
All network devices instances run within EVE-NG software , here EVE-NG 2.0.3-110 The software has been installed with an OVF available on the website https://www.eve-ng.net/
Regarding the vendors images, they have been downloaded on the vendors portals (or provided by them directly).
- Arista : veos-4.24.1F
- Juniper MX : 20.1R1.11-limited-VCP
- Juniper QFX : 18.4R1.8
- Juniper SRX : next-gen 20.1R1.11
- Nokia : 7750 VSR-I timos-20.7R1
Password used :
- Nokia : admin:nokia2020
- Arista : admin: <-- no password
- Juniper : root:Juniper
- Cumulus : cumulus:CumulusLinux!
GOAL
The target of this lab is to put in place a setup providing a "cross-dc" switch, multi-vendor, based on EVPN/VXLAN technology.
In this lab, we have :
- Arista devices as Spine nodes
- Spine nodes carry directly DCI connections
- Local Spine share the same autonomous system
- Leaf devices are from different vendors : Juniper, Arista, Nokia and Cumulus
- attached host are single-homed and are simulated by Arista instances. Thus allowing to test N vlans at the same time
- EVPN uses VXLAN encapsulation, the control plane is BGP base. All nodes have dedicated private ASN (except for local SPINE)
- Underlay is managed by eBGP sessions, where only Loopback IPs are distributed. Those IPs are used as VTEP IPs by LEAF nodes
- Overlay is managed by eBGP multihop sessions between loopback IPs. Well think to only activate EVPN family for those sessions + define the upsate-source IP (loopback IP)
Configurations + diagrams
[Attach:Mvendor-lab-4_vendors.zip]
[Attach:Mvendor-lab_5-vendors.zip]
Observations
Juniper vQFX supports only vlan-aware-bundle service in VXLAN (one EVPN instance for N VLAN)
On Arista you can choose either use VLAN-BASED or VLAN-AWARE-BUNDLE
vlan 100
rd 10.255.255.5:1
route-target both 65000:100
redistribute learned
or
vlan-aware-bundle LAB
rd 10.255.255.5:1
route-target both 65000:100
redistribute learned
vlan 100
On Nokia I've only found the way to do vlan-based.
It can be seen in type-routes (ether tag id part in yellow bellow) :
Arista vlan-based :
{master:0}
root@QFX_DC1> show route | match 2:10.255.255.5
2:10.255.255.5:1::0::50:01:00:3e:d9:dc/304 MAC/IP
2:10.255.255.5:1::0::50:01:00:3e:d9:dc/304 MAC/IP
Arista vlan-bundle (the ::0:: is replaced by ::100::):
{master:0}
root@QFX_DC1> show route | match 2:10.255.255.8
2:10.255.255.8:1::100::50:01:00:cd:ff:68/304 MAC/IP
2:10.255.255.8:1::100::50:01:00:cd:ff:68/304 MAC/IP
Juniper vlan-bundle :
{master:0}
root@QFX_DC1> show route | match 2:10.255.255.10
2:10.255.255.10:1::100::50:01:00:36:5f:a1/304 MAC/IP
2:10.255.255.10:1::100::50:01:00:36:5f:a1/304 MAC/IP
Nokia vlan based :
{master:0}
root@QFX_DC1> show route | match 2:10.255.255.9
2:10.255.255.9:100::0::50:01:00:06:09:e7/304 MAC/IP<div></div>
So, the border effect is that on Arista devices, fdb install routes basing on ether tag id. Juniper doesn't do that (as Nokia)
Juniper DC1 output
{master:0}
root@QFX_DC1> show ethernet-switching table
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static<div></div>
SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
Ethernet switching table : 6 entries, 6 learned
Routing instance : default-switch<div></div>
Vlan MAC MAC Logical Active
name address flags interface source
VL_100 50:01:00:06:09:e7 D vtep.32770 10.255.255.9 NOKIA-DC2
VL_100 50:01:00:36:5f:a1 D vtep.32773 10.255.255.10 JUNIPER-DC2
VL_100 50:01:00:3e:d9:dc D vtep.32769 10.255.255.5 ARISTA-DC1
VL_100 50:01:00:65:ae:e8 D vtep.32772 10.255.255.6 NOKIA-DC1
VL_100 50:01:00:96:6b:29 D xe-0/0/3.0
VL_100 50:01:00:cd:ff:68 D vtep.32771 10.255.255.8 ARISTA-DC2
Nokia DC1 output
<syntaxhighlight lang="bash">
Forwarding Database, Service 100 ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
----100 50:01:00:06:09:e7 vxlan-1: Evpn 10/15/20 08:50:03 NOKIA-DC2<div></div>
10.255.255.9:100
100 50:01:00:36:5f:a1 vxlan-1: Evpn 10/15/20 08:50:03 JUNIPER-DC2<div></div>
10.255.255.10:100
100 50:01:00:3e:d9:dc vxlan-1: Evpn 10/15/20 09:16:27 ARISTA-DC1<div></div>
10.255.255.5:100
100 50:01:00:65:ae:e8 sap:1/1/c2/1:100 L/150 10/13/20 16:51:07
100 50:01:00:96:6b:29 vxlan-1: Evpn 10/15/20 08:50:03 JUNIPER-DC1<div></div>
10.255.255.7:100
100 50:01:00:cd:ff:68 vxlan-1: Evpn 10/15/20 08:51:55 ARISTA-DC2<div></div>
10.255.255.8:100
Arista DC2 output
vlan-aware bundle :
LEAF-ARS-DC2#show mac address-table
Mac Address Table
<div></div>
----Vlan Mac Address Type Ports Moves Last Move<div></div>
--------------- ---- ----- ----- ---------<div></div>
100 5001.0036.5fa1 DYNAMIC Vx1 1 0:37:45 ago JUNIPER-DC2
100 5001.003e.d9dc DYNAMIC Vx1 1 0:13:14 ago ARISTA-DC1
100 5001.0096.6b29 DYNAMIC Vx1 1 0:37:45 ago JUNIPER-DC1
100 5001.00cd.ff68 DYNAMIC Et2 1 1 day, 19:22:18 ago
Only mac @ for Arista / Juniper are installed even the Nokia routes are well received
LEAF-ARS-DC2# show bgp evpn rd 10.255.255.6:100
BGP routing table information for VRF default
Router identifier 10.255.255.8, local AS number 65021
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP<div></div>
S - Stale, c - Contributing to ECMP, b - backup
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop<div></div>
Network Next Hop Metric AIGP LocPref Weight Path
* >Ec RD: 10.255.255.6:100 mac-ip 5001.0065.aee8
10.255.255.6 - 100 0 65002 65001 65012 i
* ec RD: 10.255.255.6:100 mac-ip 5001.0065.aee8
10.255.255.6 - 100 0 65002 65001 65012 i
* >Ec RD: 10.255.255.6:100 imet 10.255.255.6
10.255.255.6 - 100 0 65002 65001 65012 i
* ec RD: 10.255.255.6:100 imet 10.255.255.6
10.255.255.6 - 100 0 65002 65001 65012 i
compared to Juniper routes for example :
LEAF-ARS-DC2# show bgp evpn rd 10.255.255.7:1
BGP routing table information for VRF default
Router identifier 10.255.255.8, local AS number 65021
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP<div></div>
S - Stale, c - Contributing to ECMP, b - backup
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop<div></div>
Network Next Hop Metric AIGP LocPref Weight Path
* >Ec RD: 10.255.255.7:1 mac-ip 100 5001.0096.6b29
10.255.255.7 - 100 0 65002 65001 65013 i
[...]
Ether tag id makes the difference for me.
So, since no type 2 routes is received, the packets from Arista to Nokia are processed as BUM (so sent to all VTEP from the VXLAN instance).
Some tests wee done with L3 routing, details coming
