NesevoWiki - User contributions [en]

VyosAristaCVX-VXLAN-lab

2022-04-05T12:42:31Z

PierreLancastre:

=== Introduction ===
More and more, EVPN is making its place in Datacenter field. Mainstream vendors integrate its support, with VXLAN or MPLS for data plane + BGP for control plane. In previous labs, we were able to observe that interoperability was quite good between vendors. Nevertheless, even we've already tested CumulusVX, there are also other open sources NOS which can support EVPN.

Today we'll test VyOS in it's latest rolling version, which handles the support of EVPN.

The targets of this lab are to :

* create a VXLAN Fabric with VyOS nodes
* provide a L2VPN and a L3VPN to client nodes within the fabric
* integrate some other vendors node into the fabric, here Arista and Cumulus VX
* test that all works as expected

=== Lab setup ===
To reach those goals, we'll use the EVE-NG tool in community edition (https://www.eve-ng.net/index.php/download/#DL-COMM) and use the following NOS version :

* Vyatta VyOS 1.4 rolling <-- free to download https://vyos.net/get/nightly-builds/
* Arista vEOS 4.26.0F lab edition <-- need a support account
* Nvidia Cumulus VX 4.2.1 lab edition <-- free to download https://www.nvidia.com/en-us/networking/ethernet-switching/cumulus-vx/download/

We'll build a 3-tiers fabric, with leaf/spine/superspine devices, VXLAN/EVPN running on all nodes. It gives the following setup:

[[File:nesevo-Lab-Vyatta-Arista-CVX.png]]

On EVE-NG, we add IP information, ASN, loopback IPs. All other information will be past after when dealing with technical details.

So, it gives:

[[File:2022-02_eveng-lab-vyatta-cum-arista.png]]

==== <span class="mw-headline">VXLAN/BGP infos</span> ====
Note: in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>
{| class="wikitable"
|+
!service
!vlan id
!route-target
!VNI
!Commentaires
|-
|L2 cross-DC
|100
|target:65000:100
|100
|the 6 clients are in the same broadcast domain / subnet
|-
|L3VPN cros-DC
|200
|target:65000:200
|200
|the 6 clients have their own subnets. Here we use only one vni because no ESI will be implemented. One leaf one subnet.
|}
Client devices are represented by computers on EVE-NG. IPs in green are using the L2VPN service (extended layer 2 between pods) and IPs in orange the L3VPN service (routing service).

Below some diagrams showing the logic topologies :

[[File:nesevo-Lab-Vyatta-Arista-CVX-l2vpn.png]]

[[File:nesevo-Lab-Vyatta-Arista-CVX-l3vpn.png]]

Below the link to the configurations to allow you to quickly reproduce the tab.

To be noted that the configurations are not "ready-to-prod " and need to be optimized (adding bfd, bgp optimization, etc.). Some typos can subsist. Whatever, the lab works and gives the expected results :)

Link to devices configurations : [https://wiki.nesevo.com/images/3/32/02.lab-vyos-eos-cvx_v2.zip lab-vyos-eos-cvx_v2.zip]

=== Lab details ===
Considering that the lab is up and running, we'll now come back to the architecture.

We have 3 kinds of network nodes :

* leaf switches, on edge, which provide L2 and L3 services.
* couple of spine switches which allow leaf from a same pod to exchange traffic. They are also connected to superspine to reach other pods
* superspine switches which interconnext the different spine switches from the different pods, thus allowing interpods traffic

All those network devices control plane can exchange routes' information within 2 layers of bgp : underlay and overlay.

The role of the "underlay" is to dynamically route the traffic between loopback interfaces from the different nodes. Loopback interfaces are used to mount BGP sessions for the underlay and are used as VTEP interfaces on leaf nodes. Generally, it's better to have 2 different Loopback for this usage (in case of anycast vtep setup for example), but some vendors don't allow to 2 loopback interfaces in the same routing context. Whatever, in our setup, having only one loopback is fine.

Here, the underlay is done in eBGP, <u>ipv4 unicast family only + route-map to only redistribute loopback interfaces IP</u>.

* Each node mount a session to the directly connected peers.
* Each leaf node has a dedicated ASN.
* Each couple of spine/superspine has a ASN for two, to follow the best practices to optimize the routing table (ECMP).

The role of the "overlay" is to exchange EVPN routes type 2,3,5 here (type 1/4 routes seem to not be managed by VyOS at the moment, TBC).

* eBGP multihop sessions are mounted to directly neighbor loopback IPs.
* policies/options are set to not change the original nexthop when propagating routes
* for L2 services we only allow the redistribution of type 2 (macip) and type 3 (inclusive mcast for BUM management) routes
* for L3 services we only allow the redistribution of type 5 (prefix) routes
* regarding the setup, to enhance the scalability, it could be nice to use dedicated nodes (VM or phys) to take the role of route-servers. Here, we could connect them to superspine. Thus, spine/superspine would only do underlay and could not be a bottleneck for the control plane scalability

Moreover, regarding the lab setup, no bgp pic / bfd / VMTO / other option are implemented to optimize the convergence / traffic delivery. The goal was to verify that VyOS is compatible in a basic but functionnal VXLAN setup.

Now, we'll take a look on the different output from the 2 vendors leaf / spine / superspine to verify that all works as expected.

We verify that underlay works well. All is fine if:

* we have an eBGP session per directly connected peer
* we receive the same number of prefixes per peer type:
** a leef has to receive the same number of prefixes from its spine neighbors
** spine neigbors have to receive the same amount of prefixes from superspine, and reciprocally
** all VTEP IPs have to be known into the GRT (global routing table / default routing instance)

====== On VyOS ======
Below we can see that we receive 17 prefixes from both neighbors and that we know all loopback IPs from the setup (13 remote, 1 local). We have a bit too much routes, because on Cumulus nodes the filters that I have configured seem misconstructed.

To be noted that spine will announce all the Loopback IPs except the one from it's twin, since they are not directly connected and superspine respect as loop protection constraint (don't reflect routes originated by an ASN "X" to a peer from the same ASN "X").<syntaxhighlight lang="bash">
vyos@leaf1-pod1:~$ show ip bgp summary established

IPv4 Unicast Summary (VRF default):
BGP router identifier 10.255.255.5, local AS number 65011 vrf-id 0
BGP table version 20
RIB entries 37, using 6808 bytes of memory
Peers 2, using 1446 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
10.254.254.9 4 65001 29993 29988 0 0 0 02w6d19h 17 19 N/A
10.254.254.11 4 65001 17218 17220 0 0 0 01w4d22h 17 19 N/A

Displayed neighbors 2
Total number of neighbors 2

vyos@leaf1-pod1:~$ show ip route | match /32
B>* 10.255.255.1/32 [20/0] via 10.254.254.9, eth0, weight 1, 02w6d19h
B>* 10.255.255.2/32 [20/0] via 10.254.254.11, eth1, weight 1, 01w4d22h
B>* 10.255.255.3/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.4/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
C>* 10.255.255.5/32 is directly connected, dum0, 02w6d19h
B>* 10.255.255.6/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d22h
B>* 10.255.255.7/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.8/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.9/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.10/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.11/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w2d01h
B>* 10.255.255.12/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w2d00h
B>* 10.255.255.13/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w2d01h
B>* 10.255.255.14/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w2d00h

</syntaxhighlight>

====== On Arista ======
We verify that we receive prefix + same leaf / superspine / remote pods spine IPs (we note there misses some filters on spine since we receive /31 interco routes): <syntaxhighlight lang="bash">
leaf1-pod3#show ip bgp neighbors 10.254.254.43 received-routes
BGP routing table information for VRF default
Router identifier 10.255.255.11, local AS number 65015
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup, L - labeled-unicast
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI Origin Validation codes: V - valid, I - invalid, U - unknown
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric AIGP LocPref Weight Path
10.254.254.36/31 10.254.254.43 0 - - - 65004 ?
10.254.254.38/31 10.254.254.43 0 - - - 65004 ?
10.254.254.42/31 10.254.254.43 0 - - - 65004 ?
10.254.254.44/31 10.254.254.43 - - - - 65004 65016 ?
10.254.254.46/31 10.254.254.43 0 - - - 65004 ?
* 10.255.255.1/32 10.254.254.43 - - - - 65004 65003 65001 ?
* 10.255.255.2/32 10.254.254.43 - - - - 65004 65003 65001 ?
* 10.255.255.3/32 10.254.254.43 - - - - 65004 65003 65002 ?
* 10.255.255.4/32 10.254.254.43 - - - - 65004 65003 65002 ?
* 10.255.255.5/32 10.254.254.43 - - - - 65004 65003 65001 65011 ?
* 10.255.255.6/32 10.254.254.43 - - - - 65004 65003 65001 65012 ?
* 10.255.255.7/32 10.254.254.43 - - - - 65004 65003 65002 65013 ?
* 10.255.255.8/32 10.254.254.43 - - - - 65004 65003 65002 65014 ?
* 10.255.255.9/32 10.254.254.43 - - - - 65004 65003 ?
* 10.255.255.10/32 10.254.254.43 - - - - 65004 65003 ?
* 10.255.255.12/32 10.254.254.43 - - - - 65004 65016 ?
* > 10.255.255.14/32 10.254.254.43 0 - - - 65004 ?
leaf1-pod3#show ip bgp neighbors 10.254.254.41 received-routes
BGP routing table information for VRF default
Router identifier 10.255.255.11, local AS number 65015
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup, L - labeled-unicast
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI Origin Validation codes: V - valid, I - invalid, U - unknown
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric AIGP LocPref Weight Path
* > 10.255.255.1/32 10.254.254.41 - - - - 65004 65003 65001 ?
* > 10.255.255.2/32 10.254.254.41 - - - - 65004 65003 65001 ?
* > 10.255.255.3/32 10.254.254.41 - - - - 65004 65003 65002 ?
* > 10.255.255.4/32 10.254.254.41 - - - - 65004 65003 65002 ?
* > 10.255.255.5/32 10.254.254.41 - - - - 65004 65003 65001 65011 ?
* > 10.255.255.6/32 10.254.254.41 - - - - 65004 65003 65001 65012 ?
* > 10.255.255.7/32 10.254.254.41 - - - - 65004 65003 65002 65013 ?
* > 10.255.255.8/32 10.254.254.41 - - - - 65004 65003 65002 65014 ?
* > 10.255.255.9/32 10.254.254.41 - - - - 65004 65003 ?
* > 10.255.255.10/32 10.254.254.41 - - - - 65004 65003 ?
* > 10.255.255.12/32 10.254.254.41 - - - - 65004 65016 ?
* > 10.255.255.13/32 10.254.254.41 - - - - 65004 i

</syntaxhighlight>

====== On Cumulus ======
<syntaxhighlight lang="bash">
cumulus@cumulus:mgmt:~$ net show bgp summary
show bgp ipv4 unicast summary
=============================
BGP router identifier 10.255.255.12, local AS number 65016 vrf-id 0
BGP table version 103
RIB entries 37, using 7104 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.254.254.45 4 65004 306764 261331 0 0 0 01:39:02 12
spine1-pod3(10.254.254.47) 4 65004 261602 261664 0 0 0 01w2d02h 16

Total number of neighbors 2

[...]

cumulus@cumulus:mgmt:~$ net show route | grep /32
B>* 10.255.255.1/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.2/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.3/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.4/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.5/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.6/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.7/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.8/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.9/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.10/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.11/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
C>* 10.255.255.12/32 is directly connected, lo, 01w2d02h
B>* 10.255.255.13/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.14/32 [20/0] via 10.254.254.47, swp2, weight 1, 01w2d02h

</syntaxhighlight>We can also verify that routing is ok by pinging remote loopback IPs from local loopback + check eBGP EVPN session status: <syntaxhighlight lang="bash">
# ping example from VyOS leaf
vyos@leaf1-pod1:~$ ping 10.255.255.12 source-address 10.255.255.5
PING 10.255.255.12 (10.255.255.12) from 10.255.255.5 : 56(84) bytes of data.
64 bytes from 10.255.255.12: icmp_seq=1 ttl=61 time=2.60 ms
64 bytes from 10.255.255.12: icmp_seq=2 ttl=61 time=2.54 ms
64 bytes from 10.255.255.12: icmp_seq=3 ttl=61 time=2.44 ms

# check of EVPN sessions on the different vendors

# vyos

vyos@leaf1-pod1:~$ show bgp l2vpn evpn summary
BGP router identifier 10.255.255.5, local AS number 65011 vrf-id 0
BGP table version 0
RIB entries 31, using 5704 bytes of memory
Peers 2, using 1446 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
10.255.255.1 4 65001 30205 30208 0 0 0 02w6d21h 13 13 N/A
10.255.255.2 4 65001 17392 17429 0 0 0 01w5d00h 11 13 N/A

# Arista

leaf1-pod3#show bgp evpn summary
BGP summary information for VRF default
Router identifier 10.255.255.11, local AS number 65015
Neighbor Status Codes: m - Under maintenance
Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
OVL_spine1-pod3 10.255.255.13 4 65004 15608 15503 0 0 00:03:47 Estab 17 17
OVL_spine2-pod3 10.255.255.14 4 65004 262780 308558 0 0 9d02h Estab 13 13

# Cumulus

cumulus@cumulus:mgmt:~$ net show bgp l2vpn evpn summary
BGP router identifier 10.255.255.12, local AS number 65016 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.255.255.13 4 65004 307243 261693 0 0 0 00:04:25 14
spine1-pod3(10.255.255.14) 4 65004 262001 262025 0 0 0 01w2d02h 10

</syntaxhighlight>

On EVPN, we should receive quite the same number of prefixes from both spine, but looks like there differences of best path election + redistribution between vendors. Perhaps something to dig. The most important point is to verify that the FIB for the services are consistent.

==== L2 service implementation ====
On that topic, we'll firstly focus on the Layer 2 service. To be sure that all works as expected we have to check that:

* mac-address-table are consistent on the whole fabric. In traditionnal L2 setup, switch FIBs depend of the traffic passing through them and a mac @ can be see "everywhere" if it does a BUM on the network. In VXLAN, in a mac-vrf topology without filer, even a host doesn't send BUM, the mac address is propagated on the whole fabric (within BGP update). But, as it is on traditional L2 topology, you have to be sure that mac @ table aging time are consistent on all devices, with an aging time superior to ARP timer on hosts.
* routes type 2 are visible on each leaf nodes for a same vni, routes type 3 to reach remote VTEP have to be learned
* a common route-target for the VRF routes need to be used/set within policies. In effect, there are differences between vendors on route "tagging" (features to auto generate route-target, "hard coded" alogos, etc.). Here, on VyOS, we got to add adminstrative extended communities within route-map for the L2 VNI. We'll show it below
* all fabric connections (interconnections between network nodes) have the sufficient "IP MTU". Please keep in mind that VXLAN overhead has a size of 50 bytes. So, to be sure that Jumbo traffic for host is ok, configure it at least at 9050 Bytes. I would advise to push it at 9100 Bytes in case of VXLAN usage on client HOSTS for Virtualized setup on servers.

Check FDB on leaf switch + debug type 2 routes' propagation for leaf1 pod1<syntaxhighlight lang="bash">
##############
# VyOS
##############

vyos@leaf1-pod1:~$ show bridge br100 fdb
50:01:00:01:00:02 dev eth2 vlan 1 master br100 permanent
50:01:00:01:00:02 dev eth2 master br100 permanent
33:33:00:00:00:01 dev eth2 self permanent
33:33:00:00:00:02 dev eth2 self permanent
01:00:5e:00:00:01 dev eth2 self permanent
01:80:c2:00:00:0e dev eth2 self permanent
01:80:c2:00:00:03 dev eth2 self permanent
01:80:c2:00:00:00 dev eth2 self permanent
33:33:00:00:00:01 dev br100 self permanent
33:33:00:00:00:02 dev br100 self permanent
33:33:ff:01:00:02 dev br100 self permanent
01:00:5e:00:00:6a dev br100 self permanent
33:33:00:00:00:6a dev br100 self permanent
01:00:5e:00:00:01 dev br100 self permanent
33:33:ff:00:00:00 dev br100 self permanent
00:50:79:66:68:15 dev vxlan100 vlan 1 extern_learn master br100
00:50:79:66:68:15 dev vxlan100 extern_learn master br100
8a:d0:00:16:1c:17 dev vxlan100 vlan 1 master br100 permanent
8a:d0:00:16:1c:17 dev vxlan100 master br100 permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.6 self permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.11 self permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.12 self permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.7 self permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.8 self permanent
00:50:79:66:68:15 dev vxlan100 dst 10.255.255.12 self extern_learn

# and to focus on remote nodes

vyos@leaf1-pod1:~$ show bridge br100 fdb | grep "self extern_learn"
00:50:79:66:68:15 dev vxlan100 dst 10.255.255.12 self extern_learn
00:50:79:66:68:13 dev vxlan100 dst 10.255.255.11 self extern_learn
00:50:79:66:68:19 dev vxlan100 dst 10.255.255.8 self extern_learn
00:50:79:66:68:0c dev vxlan100 dst 10.255.255.6 self extern_learn
00:50:79:66:68:17 dev vxlan100 dst 10.255.255.7 self extern_learn

# or on local clients

vyos@leaf1-pod1:~$ show bridge br100 fdb | grep eth | grep -v perm
00:50:79:66:68:0b dev eth2 master br100

# Now if we want to focus on route type 3 learned

Displayed 7 prefixes (12 paths) (of requested type)
vyos@leaf1-pod1:~$ show bgp l2vpn evpn route type 3 | grep Dis
Route Distinguisher: 10.255.255.5:3
Route Distinguisher: 10.255.255.6:3
Route Distinguisher: 10.255.255.7:2
Route Distinguisher: 10.255.255.8:2
Route Distinguisher: 10.255.255.11:100
Route Distinguisher: 10.255.255.12:100

# we notice that RDs for VyOS nodes don't respect the logic of <Loopback:vni>. It s because for the moment we can't set it by vni (or I didn't find how to do that)

##############
# Arista
##############

# we check the fdb. Remote host are behind the port Vx1

leaf1-pod3#show mac address-table vlan 100
Mac Address Table
------------------------------------------------------------------

Vlan Mac Address Type Ports Moves Last Move
---- ----------- ---- ----- ----- ---------
100 0050.7966.680c DYNAMIC Vx1 1 0:00:38 ago
100 0050.7966.6813 DYNAMIC Et3 1 0:14:08 ago
100 0050.7966.6815 DYNAMIC Vx1 1 9 days, 2:36:39 ago
100 0050.7966.6817 DYNAMIC Vx1 1 0:00:36 ago
100 0050.7966.6819 DYNAMIC Vx1 1 0:00:35 ago
Total Mac Addresses for this criterion: 5

# we check that all VTEP are ready for BUM

leaf1-pod3#show bgp evpn route-type imet | i >
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
* >Ec RD: 10.255.255.5:3 imet 10.255.255.5
* >Ec RD: 10.255.255.6:3 imet 10.255.255.6
* >Ec RD: 10.255.255.7:2 imet 10.255.255.7
* >Ec RD: 10.255.255.8:2 imet 10.255.255.8
* > RD: 10.255.255.11:100 imet 10.255.255.11
* >Ec RD: 10.255.255.12:100 imet 10.255.255.12
[...]

leaf1-pod3#show vxlan vtep detail
Remote VTEPS for Vxlan1:

VTEP Learned Via MAC Address Learning Tunnel Type(s)
------------------ ------------------ -------------------------- --------------
10.255.255.5 control plane control plane flood, unicast
10.255.255.6 control plane control plane flood, unicast
10.255.255.7 control plane control plane flood, unicast
10.255.255.8 control plane control plane flood, unicast
10.255.255.12 control plane control plane flood, unicast

##############
# Cumulus
##############

# we check the fdb. Remote host are behind the port Vx1

cumulus@cumulus:mgmt:~$ net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ ----------------
100 bridge bridge 50:01:00:11:00:03 permanent 9 days, 02:47:42
100 bridge swp3 00:50:79:66:68:15 00:00:06
100 bridge vni100 00:50:79:66:68:0c extern_learn 00:00:40
100 bridge vni100 00:50:79:66:68:13 extern_learn 00:21:01
100 bridge vni100 00:50:79:66:68:17 extern_learn 00:00:38
100 bridge vni100 00:50:79:66:68:19 extern_learn 00:00:36

# --> here we see that it misses the @mac from the host on leaf1 pod1.
# we check the evpn routes on the Cumulus node and we see that from leaf1 pod1 vni 100 we have only those routes

Route Distinguisher: 10.255.255.5:3
* [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
* [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
*> [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8

# --> something is wrong on the remote leaf.
# on the leaf we check the policy for the EVPN routes export

route-map RM-OUT-VNI {
rule 1 {
action permit
match {
evpn {
route-type multicast
vni 100
}
}
set {
extcommunity {
rt 65000:100
}
}
}
rule 2 {
action permit
match {
evpn {
route-type prefix
vni 100
}
}
set {
extcommunity {
rt 65000:100
}
}
}
rule 3 {
action permit
match {
evpn {
vni 200
}
}
set {
extcommunity {
rt 65000:200
}

# we notice that a typo in rule numbering removed the export of macip routes for vni 100.
# we correct that :

route-map RM-OUT-VNI {
rule 1 {
action permit
match {
evpn {
route-type multicast
vni 100
}
}
set {
extcommunity {
rt 65000:100
}
}
}
rule 2 {
action permit
match {
evpn {
route-type macip
vni 100
}
}
set {
extcommunity {
rt 65000:100
}
}
}
rule 3 {
action permit
match {
evpn {
route-type prefix
vni 200
}
}
set {
extcommunity {
rt 65000:200
}

# once corrected we check the fdb on the cumulus node + evpn routes

cumulus@cumulus:mgmt:~$ net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ ----------------
100 bridge bridge 50:01:00:11:00:03 permanent 9 days, 02:51:48
100 bridge swp3 00:50:79:66:68:15 00:02:01
100 bridge vni100 00:50:79:66:68:0b extern_learn 00:00:26
100 bridge vni100 00:50:79:66:68:0c extern_learn 00:04:46
100 bridge vni100 00:50:79:66:68:13 extern_learn 00:25:08
100 bridge vni100 00:50:79:66:68:17 extern_learn 00:04:45
100 bridge vni100 00:50:79:66:68:19 extern_learn 00:04:43

Route Distinguisher: 10.255.255.5:3
* [2]:[0]:[48]:[00:50:79:66:68:0b]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
* [2]:[0]:[48]:[00:50:79:66:68:0b]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
*> [2]:[0]:[48]:[00:50:79:66:68:0b]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
* [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
* [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
*> [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8

# we have now Type 2 routes, all is ok now and we learn the client mac 00:50:79:66:68:0b
# Nevertheless, the trafic was working between the client hosts, but trafic from Cumulus leaf client to VyOS leaf client was broadcasted to all leaf since the @mac was not advertised on the fabric

</syntaxhighlight>It could be interesting to check how each node is configured to deliver the service, but the article would be too long. So, to sum the steps to provide a L2 service:

# create a VLAN (on Arista) / bridge (VyOS and Cumulus) + associate it to the VNI/VXLAN instances/interface
# create the mac vrf + define the route-target into bgp (Arista + Cumulus) / update the route-maps + community list to allow the redistribution of the VNI routes (type 2 and type 3)

==== L3 service implementation ====
Now, we'll now look at the L3 service. On that kind of setup we need to implement / check:
* leaf IP subnets are well redistributed betweeb all leaf (here we are in a vrf "any to any" topology)
* we only redistribute routes for IP prefix. Routes type 2 (mac @) don't need to be redistributed between leaf
* that all clients can ping each other through the L3VPN

To achieve that, we need to implement analog configuration than for the L2 service, but with the following differences:

* create a VLAN (on Arista) / bridge (VyOS and Cumulus) + associate it to the VNI/VXLAN instances|interface
* create the L3 VRF + define the route-target into bgp (Arista + Cumulus + VyOS) + allow connected routes distribution (Arista / Cumulus) / update the route-maps + community list to allow the redistribution of the VNI routes (type 5) for VyOS / restrict the routes redistribution to type 5 routes into the VNI (Cumulus)

Now we check the routing table with common cli commands + into the EVPN RIB :

<syntaxhighlight lang="bash">
# check routes into the vrf

vyos@leaf1-pod1:~$ show ip route vrf VRF-L3-1
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF VRF-L3-1:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 02w6d22h
C>* 192.168.10.0/24 is directly connected, br200, 02w6d22h
B>* 192.168.20.0/24 [20/0] via 10.255.255.6, br200 onlink, weight 1, 05:11:08
B>* 192.168.30.0/24 [20/0] via 10.255.255.7, br200 onlink, weight 1, 05:04:15
B>* 192.168.40.0/24 [20/0] via 10.255.255.8, br200 onlink, weight 1, 04:59:41
B>* 192.168.50.0/24 [20/0] via 10.255.255.11, br200 onlink, weight 1, 05:11:08
B>* 192.168.60.0/24 [20/0] via 10.255.255.12, br200 onlink, weight 1, 05:11:08

leaf1-pod3#show ip route vrf VRF-L3-1

VRF: VRF-L3-1
Codes: C - connected, S - static, K - kernel,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,
R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
NG - Nexthop Group Static Route, V - VXLAN Control Service,
DH - DHCP client installed default route, M - Martian,
DP - Dynamic Policy Route, L - VRF Leaked,
G - gRIBI, RC - Route Cache Route

Gateway of last resort is not set

B E 192.168.10.0/24 [200/0] via VTEP 10.255.255.5 VNI 200 router-mac 12:9b:9b:d8:f6:86 local-interface Vxlan1
B E 192.168.20.0/24 [200/0] via VTEP 10.255.255.6 VNI 200 router-mac 50:01:00:02:00:03 local-interface Vxlan1
B E 192.168.30.0/24 [200/0] via VTEP 10.255.255.7 VNI 200 router-mac 50:01:00:09:00:03 local-interface Vxlan1
B E 192.168.40.0/24 [200/0] via VTEP 10.255.255.8 VNI 200 router-mac 50:01:00:0a:00:03 local-interface Vxlan1
C 192.168.50.0/24 is directly connected, Vlan200
B E 192.168.60.0/24 [200/0] via VTEP 10.255.255.12 VNI 200 router-mac 50:01:00:11:00:03 local-interface Vxlan1

cumulus@cumulus:mgmt:~$ net show route vrf VRF-L3-1
show ip route vrf VRF-L3-1
===========================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued route, r - rejected route

VRF VRF-L3-1:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 01w2d03h
B>* 192.168.10.0/24 [20/0] via 10.255.255.5, vlan200 onlink, weight 1, 03:18:37
B>* 192.168.20.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 03:18:37
B>* 192.168.30.0/24 [20/0] via 10.255.255.7, vlan200 onlink, weight 1, 03:18:37
B>* 192.168.40.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:18:37
B>* 192.168.50.0/24 [20/0] via 10.255.255.11, vlan200 onlink, weight 1, 03:18:37
C>* 192.168.60.0/24 is directly connected, vlan200, 01w2d03h

# we can notice that the output are quite similar, except on Arista which indicates the VNI for remote targets, instead of referencing the local bridge / vlan200

# Now we check the routes for the VNI 200

# VyOS

vyos@leaf1-pod1:~$ show bgp l2vpn evpn route type prefix
BGP table version is 1, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:200
*> [5]:[0]:[24]:[192.168.10.0]
10.255.255.5 0 32768 ?
ET:8 RT:65000:200 Rmac:12:9b:9b:d8:f6:86
Route Distinguisher: 10.255.255.6:200
*> [5]:[0]:[24]:[192.168.20.0]
10.255.255.6 0 65001 65012 ?
RT:65000:200 ET:8 Rmac:50:01:00:02:00:03
Route Distinguisher: 10.255.255.7:200
* [5]:[0]:[24]:[192.168.30.0]
10.255.255.7 0 65001 65003 65002 65013 ?
RT:65000:200 RT:65013:200 ET:8 Rmac:50:01:00:09:00:03
*> [5]:[0]:[24]:[192.168.30.0]
10.255.255.7 0 65001 65003 65002 65013 ?
RT:65000:200 RT:65013:200 ET:8 Rmac:50:01:00:09:00:03
Route Distinguisher: 10.255.255.8:200
*> [5]:[0]:[24]:[192.168.40.0]
10.255.255.8 0 65001 65003 65002 65014 ?
RT:65000:200 RT:65014:200 ET:8 Rmac:50:01:00:0a:00:03
* [5]:[0]:[24]:[192.168.40.0]
10.255.255.8 0 65001 65003 65002 65014 ?
RT:65000:200 RT:65014:200 ET:8 Rmac:50:01:00:0a:00:03
Route Distinguisher: 10.255.255.11:200
* [5]:[0]:[24]:[192.168.50.0]
10.255.255.11 0 65001 65003 65004 65015 i
RT:65000:200 ET:8 Rmac:50:01:00:cd:ff:68
*> [5]:[0]:[24]:[192.168.50.0]
10.255.255.11 0 65001 65003 65004 65015 i
RT:65000:200 ET:8 Rmac:50:01:00:cd:ff:68
Route Distinguisher: 10.255.255.12:10200
* [5]:[0]:[24]:[192.168.60.0]
10.255.255.12 0 65001 65003 65004 65016 i
RT:65000:200 ET:8 Rmac:50:01:00:11:00:03
*> [5]:[0]:[24]:[192.168.60.0]
10.255.255.12 0 65001 65003 65004 65016 i
RT:65000:200 ET:8 Rmac:50:01:00:11:00:03

# Arista

leaf1-pod3#show bgp evpn route ip-prefix ipv4
BGP routing table information for VRF default
Router identifier 10.255.255.11, local AS number 65015
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric LocPref Weight Path
* > RD: 10.255.255.5:200 ip-prefix 192.168.10.0/24
10.255.255.5 - 100 0 65004 65003 65001 65011 ?
* RD: 10.255.255.5:200 ip-prefix 192.168.10.0/24
10.255.255.5 - 100 0 65004 65003 65001 65011 ?
* > RD: 10.255.255.6:200 ip-prefix 192.168.20.0/24
10.255.255.6 - 100 0 65004 65003 65001 65012 ?
* RD: 10.255.255.6:200 ip-prefix 192.168.20.0/24
10.255.255.6 - 100 0 65004 65003 65001 65012 ?
* > RD: 10.255.255.7:200 ip-prefix 192.168.30.0/24
10.255.255.7 - 100 0 65004 65003 65002 65013 ?
* RD: 10.255.255.7:200 ip-prefix 192.168.30.0/24
10.255.255.7 - 100 0 65004 65003 65002 65013 ?
* > RD: 10.255.255.8:200 ip-prefix 192.168.40.0/24
10.255.255.8 - 100 0 65004 65003 65002 65014 ?
* RD: 10.255.255.8:200 ip-prefix 192.168.40.0/24
10.255.255.8 - 100 0 65004 65003 65002 65014 ?
* > RD: 10.255.255.11:200 ip-prefix 192.168.50.0/24
- - - 0 i
* > RD: 10.255.255.12:10200 ip-prefix 192.168.60.0/24
10.255.255.12 - 100 0 65004 65016 i
* RD: 10.255.255.12:10200 ip-prefix 192.168.60.0/24
10.255.255.12 - 100 0 65004 65016 i

# CVX

cumulus@cumulus:mgmt:~$ net show bgp l2vpn evpn route type prefix
BGP table version is 7, local router ID is 10.255.255.12
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:200
* [5]:[0]:[24]:[192.168.10.0]
10.255.255.5 0 65004 65003 65001 65011 ?
RT:65000:200 ET:8 Rmac:12:9b:9b:d8:f6:86
*> [5]:[0]:[24]:[192.168.10.0]
10.255.255.5 0 65004 65003 65001 65011 ?
RT:65000:200 ET:8 Rmac:12:9b:9b:d8:f6:86
Route Distinguisher: 10.255.255.6:200
* [5]:[0]:[24]:[192.168.20.0]
10.255.255.6 0 65004 65003 65001 65012 ?
RT:65000:200 ET:8 Rmac:50:01:00:02:00:03
*> [5]:[0]:[24]:[192.168.20.0]
10.255.255.6 0 65004 65003 65001 65012 ?
RT:65000:200 ET:8 Rmac:50:01:00:02:00:03
Route Distinguisher: 10.255.255.7:200
* [5]:[0]:[24]:[192.168.30.0]
10.255.255.7 0 65004 65003 65002 65013 ?
RT:65000:200 RT:65013:200 ET:8 Rmac:50:01:00:09:00:03
*> [5]:[0]:[24]:[192.168.30.0]
10.255.255.7 0 65004 65003 65002 65013 ?
RT:65000:200 RT:65013:200 ET:8 Rmac:50:01:00:09:00:03
Route Distinguisher: 10.255.255.8:200
* [5]:[0]:[24]:[192.168.40.0]
10.255.255.8 0 65004 65003 65002 65014 ?
RT:65000:200 RT:65014:200 ET:8 Rmac:50:01:00:0a:00:03
*> [5]:[0]:[24]:[192.168.40.0]
10.255.255.8 0 65004 65003 65002 65014 ?
RT:65000:200 RT:65014:200 ET:8 Rmac:50:01:00:0a:00:03
Route Distinguisher: 10.255.255.11:200
* [5]:[0]:[24]:[192.168.50.0]
10.255.255.11 0 65004 65015 i
RT:65000:200 ET:8 Rmac:50:01:00:cd:ff:68
*> [5]:[0]:[24]:[192.168.50.0]
10.255.255.11 0 65004 65015 i
RT:65000:200 ET:8 Rmac:50:01:00:cd:ff:68
Route Distinguisher: 10.255.255.12:10200
*> [5]:[0]:[24]:[192.168.60.0]
10.255.255.12 0 32768 i
ET:8 RT:65000:200 Rmac:50:01:00:11:00:03

# Now we check with traceroute on the 3 clients connected on focused leaf switches

# client leaf VyOS

VPCS> ping 192.168.50.2

84 bytes from 192.168.50.2 icmp_seq=1 ttl=62 time=16.456 ms
84 bytes from 192.168.50.2 icmp_seq=2 ttl=62 time=11.852 ms
^C
VPCS> ping 192.168.60.2

84 bytes from 192.168.60.2 icmp_seq=1 ttl=62 time=5.554 ms
84 bytes from 192.168.60.2 icmp_seq=2 ttl=62 time=4.432 ms
^C
VPCS> trace 192.168.50.2
trace to 192.168.50.2, 8 hops max, press Ctrl+C to stop
1 192.168.10.1 0.499 ms 3.143 ms 1.432 ms
2 192.168.50.1 10.465 ms 12.512 ms 8.992 ms
3 *192.168.50.2 11.056 ms (ICMP type:3, code:3, Destination port unreachable)

VPCS> trace 192.168.60.2
trace to 192.168.60.2, 8 hops max, press Ctrl+C to stop
1 192.168.10.1 0.525 ms 0.312 ms 0.278 ms
2 192.168.60.1 4.949 ms 4.232 ms 4.022 ms
3 *192.168.60.2 4.167 ms (ICMP type:3, code:3, Destination port unreachable)

# client leaf Arista

VPCS> ping 192.168.10.2

84 bytes from 192.168.10.2 icmp_seq=1 ttl=62 time=10.914 ms
^C
VPCS> ping 192.168.60.2

84 bytes from 192.168.60.2 icmp_seq=1 ttl=62 time=11.755 ms
^C
VPCS> trace 192.168.10.2
trace to 192.168.10.2, 8 hops max, press Ctrl+C to stop
1 192.168.50.1 4.742 ms 4.123 ms 4.293 ms
2 192.168.10.1 12.550 ms 15.039 ms 9.411 ms
3 *192.168.10.2 8.939 ms (ICMP type:3, code:3, Destination port unreachable)

VPCS>
VPCS> trace 192.168.60.2
trace to 192.168.60.2, 8 hops max, press Ctrl+C to stop
1 192.168.50.1 3.073 ms 2.901 ms 2.585 ms
2 192.168.60.1 10.815 ms 11.493 ms 12.144 ms
3 *192.168.60.2 10.429 ms (ICMP type:3, code:3, Destination port unreachable)

# client leaf Cumulus

VPCS> ping 192.168.10.2

84 bytes from 192.168.10.2 icmp_seq=1 ttl=62 time=4.731 ms
^C
VPCS> ping 192.168.50.2

84 bytes from 192.168.50.2 icmp_seq=1 ttl=62 time=11.722 ms
^C
VPCS> trace 192.168.50.2
trace to 192.168.50.2, 8 hops max, press Ctrl+C to stop
1 192.168.60.1 0.487 ms 0.266 ms 0.248 ms
2 192.168.50.1 6.691 ms 6.281 ms 6.613 ms
3 *192.168.50.2 9.355 ms (ICMP type:3, code:3, Destination port unreachable)

VPCS>
VPCS> trace 192.168.10.2
trace to 192.168.10.2, 8 hops max, press Ctrl+C to stop
1 192.168.60.1 0.436 ms 0.315 ms 0.380 ms
2 192.168.10.1 4.535 ms 4.661 ms 3.994 ms
3 *192.168.10.2 4.292 ms (ICMP type:3, code:3, Destination port unreachable)

</syntaxhighlight>

=== Conclusion of the lab ===
These first results are satisfying, it comforts the idea that VXLAN is a strong option to deliver L2/L3 services in datacenters, but not only. This kind of architecture is more and more chosen in campus designs, PRA, NFV solutions.

The support of this technology on more and more open source NOS are encouraging regarding the innovation. If we look at Vyatta project for example, whom OS can be compiled "brick per brick", we could imagine to create NFV CPEs providing VXLAN fabric extensions of some client VRFs. CPE could be connected to the central Fabric within encrypted tunnels (wireguard,ipsec) or through direct link over MacSec (this feature in on the Vyatta Roadmap).

Please contact us if you want to feed the topic / discuss the lab.

Cheers

Pierre L.

VyosAristaCVX-VXLAN-lab

2022-04-05T12:40:19Z

PierreLancastre:

=== Introduction ===
More and more, EVPN is making its place in Datacenter field. Mainstream vendors integrate its support, with VXLAN or MPLS for data plane + BGP for control plane. In previous labs, we were able to observe that interoperability was quite good between vendors. Nevertheless, even we've already tested CumulusVX, there are also other open sources NOS which can support EVPN.

Today we'll test VyOS in it's latest rolling version, which handles the support of EVPN.

The targets of this lab are to :

* create a VXLAN Fabric with VyOS nodes
* provide a L2VPN and a L3VPN to client nodes within the fabric
* integrate some other vendors node into the fabric, here Arista and Cumulus VX
* test that all works as expected

=== Lab setup ===
To reach those goals, we'll use the EVE-NG tool in community edition (https://www.eve-ng.net/index.php/download/#DL-COMM) and use the following NOS version :

* Vyatta VyOS 1.4 rolling <-- free to download https://vyos.net/get/nightly-builds/
* Arista vEOS 4.26.0F lab edition <-- need a support account
* Nvidia Cumulus VX 4.2.1 lab edition <-- free to download https://www.nvidia.com/en-us/networking/ethernet-switching/cumulus-vx/download/

We'll build a 3-tiers fabric, with leaf/spine/superspine devices, VXLAN/EVPN running on all nodes. It gives the following setup:

[[File:nesevo-Lab-Vyatta-Arista-CVX.png]]

On EVE-NG, we add IP information, ASN, loopback IPs. All other information will be past after when dealing with technical details.

So, it gives:

[[File:2022-02_eveng-lab-vyatta-cum-arista.png]]

==== <span class="mw-headline">VXLAN/BGP infos</span> ====
Note: in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>
{| class="wikitable"
|+
!service
!vlan id
!route-target
!VNI
!Commentaires
|-
|L2 cross-DC
|100
|target:65000:100
|100
|the 6 clients are in the same broadcast domain / subnet
|-
|L3VPN cros-DC
|200
|target:65000:200
|200
|the 6 clients have their own subnets. Here we use only one vni because no ESI will be implemented. One leaf one subnet.
|}
Client devices are represented by computers on EVE-NG. IPs in green are using the L2VPN service (extended layer 2 between pods) and IPs in orange the L3VPN service (routing service).

Below some diagrams showing the logic topologies :

[[File:Nesevo-Lab-Vyatta-Arista-CVX-l2vpn.png]]

[[File:Nesevo-Lab-Vyatta-Arista-CVX-l3vpn.png]]

Below the link to the configurations to allow you to quickly reproduce the tab.

To be noted that the configurations are not "ready-to-prod " and need to be optimized (adding bfd, bgp optimization, etc.). Some typos can subsist. Whatever, the lab works and gives the expected results :)

Link to devices configurations : [[File:02.lab-vyos-eos-cvx_v2.zip]]

=== Lab details ===
Considering that the lab is up and running, we'll now come back to the architecture.

We have 3 kinds of network nodes :

* leaf switches, on edge, which provide L2 and L3 services.
* couple of spine switches which allow leaf from a same pod to exchange traffic. They are also connected to superspine to reach other pods
* superspine switches which interconnext the different spine switches from the different pods, thus allowing interpods traffic

All those network devices control plane can exchange routes' information within 2 layers of bgp : underlay and overlay.

The role of the "underlay" is to dynamically route the traffic between loopback interfaces from the different nodes. Loopback interfaces are used to mount BGP sessions for the underlay and are used as VTEP interfaces on leaf nodes. Generally, it's better to have 2 different Loopback for this usage (in case of anycast vtep setup for example), but some vendors don't allow to 2 loopback interfaces in the same routing context. Whatever, in our setup, having only one loopback is fine.

Here, the underlay is done in eBGP, <u>ipv4 unicast family only + route-map to only redistribute loopback interfaces IP</u>.

* Each node mount a session to the directly connected peers.
* Each leaf node has a dedicated ASN.
* Each couple of spine/superspine has a ASN for two, to follow the best practices to optimize the routing table (ECMP).

The role of the "overlay" is to exchange EVPN routes type 2,3,5 here (type 1/4 routes seem to not be managed by VyOS at the moment, TBC).

* eBGP multihop sessions are mounted to directly neighbor loopback IPs.
* policies/options are set to not change the original nexthop when propagating routes
* for L2 services we only allow the redistribution of type 2 (macip) and type 3 (inclusive mcast for BUM management) routes
* for L3 services we only allow the redistribution of type 5 (prefix) routes
* regarding the setup, to enhance the scalability, it could be nice to use dedicated nodes (VM or phys) to take the role of route-servers. Here, we could connect them to superspine. Thus, spine/superspine would only do underlay and could not be a bottleneck for the control plane scalability

Moreover, regarding the lab setup, no bgp pic / bfd / VMTO / other option are implemented to optimize the convergence / traffic delivery. The goal was to verify that VyOS is compatible in a basic but functionnal VXLAN setup.

Now, we'll take a look on the different output from the 2 vendors leaf / spine / superspine to verify that all works as expected.

We verify that underlay works well. All is fine if:

* we have an eBGP session per directly connected peer
* we receive the same number of prefixes per peer type:
** a leef has to receive the same number of prefixes from its spine neighbors
** spine neigbors have to receive the same amount of prefixes from superspine, and reciprocally
** all VTEP IPs have to be known into the GRT (global routing table / default routing instance)

====== On VyOS ======
Below we can see that we receive 17 prefixes from both neighbors and that we know all loopback IPs from the setup (13 remote, 1 local). We have a bit too much routes, because on Cumulus nodes the filters that I have configured seem misconstructed.

To be noted that spine will announce all the Loopback IPs except the one from it's twin, since they are not directly connected and superspine respect as loop protection constraint (don't reflect routes originated by an ASN "X" to a peer from the same ASN "X").<syntaxhighlight lang="bash">
vyos@leaf1-pod1:~$ show ip bgp summary established

IPv4 Unicast Summary (VRF default):
BGP router identifier 10.255.255.5, local AS number 65011 vrf-id 0
BGP table version 20
RIB entries 37, using 6808 bytes of memory
Peers 2, using 1446 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
10.254.254.9 4 65001 29993 29988 0 0 0 02w6d19h 17 19 N/A
10.254.254.11 4 65001 17218 17220 0 0 0 01w4d22h 17 19 N/A

Displayed neighbors 2
Total number of neighbors 2

vyos@leaf1-pod1:~$ show ip route | match /32
B>* 10.255.255.1/32 [20/0] via 10.254.254.9, eth0, weight 1, 02w6d19h
B>* 10.255.255.2/32 [20/0] via 10.254.254.11, eth1, weight 1, 01w4d22h
B>* 10.255.255.3/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.4/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
C>* 10.255.255.5/32 is directly connected, dum0, 02w6d19h
B>* 10.255.255.6/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d22h
B>* 10.255.255.7/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.8/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.9/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.10/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.11/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w2d01h
B>* 10.255.255.12/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w2d00h
B>* 10.255.255.13/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w2d01h
B>* 10.255.255.14/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w2d00h

</syntaxhighlight>

====== On Arista ======
We verify that we receive prefix + same leaf / superspine / remote pods spine IPs (we note there misses some filters on spine since we receive /31 interco routes): <syntaxhighlight lang="bash">
leaf1-pod3#show ip bgp neighbors 10.254.254.43 received-routes
BGP routing table information for VRF default
Router identifier 10.255.255.11, local AS number 65015
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup, L - labeled-unicast
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI Origin Validation codes: V - valid, I - invalid, U - unknown
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric AIGP LocPref Weight Path
10.254.254.36/31 10.254.254.43 0 - - - 65004 ?
10.254.254.38/31 10.254.254.43 0 - - - 65004 ?
10.254.254.42/31 10.254.254.43 0 - - - 65004 ?
10.254.254.44/31 10.254.254.43 - - - - 65004 65016 ?
10.254.254.46/31 10.254.254.43 0 - - - 65004 ?
* 10.255.255.1/32 10.254.254.43 - - - - 65004 65003 65001 ?
* 10.255.255.2/32 10.254.254.43 - - - - 65004 65003 65001 ?
* 10.255.255.3/32 10.254.254.43 - - - - 65004 65003 65002 ?
* 10.255.255.4/32 10.254.254.43 - - - - 65004 65003 65002 ?
* 10.255.255.5/32 10.254.254.43 - - - - 65004 65003 65001 65011 ?
* 10.255.255.6/32 10.254.254.43 - - - - 65004 65003 65001 65012 ?
* 10.255.255.7/32 10.254.254.43 - - - - 65004 65003 65002 65013 ?
* 10.255.255.8/32 10.254.254.43 - - - - 65004 65003 65002 65014 ?
* 10.255.255.9/32 10.254.254.43 - - - - 65004 65003 ?
* 10.255.255.10/32 10.254.254.43 - - - - 65004 65003 ?
* 10.255.255.12/32 10.254.254.43 - - - - 65004 65016 ?
* > 10.255.255.14/32 10.254.254.43 0 - - - 65004 ?
leaf1-pod3#show ip bgp neighbors 10.254.254.41 received-routes
BGP routing table information for VRF default
Router identifier 10.255.255.11, local AS number 65015
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup, L - labeled-unicast
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI Origin Validation codes: V - valid, I - invalid, U - unknown
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric AIGP LocPref Weight Path
* > 10.255.255.1/32 10.254.254.41 - - - - 65004 65003 65001 ?
* > 10.255.255.2/32 10.254.254.41 - - - - 65004 65003 65001 ?
* > 10.255.255.3/32 10.254.254.41 - - - - 65004 65003 65002 ?
* > 10.255.255.4/32 10.254.254.41 - - - - 65004 65003 65002 ?
* > 10.255.255.5/32 10.254.254.41 - - - - 65004 65003 65001 65011 ?
* > 10.255.255.6/32 10.254.254.41 - - - - 65004 65003 65001 65012 ?
* > 10.255.255.7/32 10.254.254.41 - - - - 65004 65003 65002 65013 ?
* > 10.255.255.8/32 10.254.254.41 - - - - 65004 65003 65002 65014 ?
* > 10.255.255.9/32 10.254.254.41 - - - - 65004 65003 ?
* > 10.255.255.10/32 10.254.254.41 - - - - 65004 65003 ?
* > 10.255.255.12/32 10.254.254.41 - - - - 65004 65016 ?
* > 10.255.255.13/32 10.254.254.41 - - - - 65004 i

</syntaxhighlight>

====== On Cumulus ======
<syntaxhighlight lang="bash">
cumulus@cumulus:mgmt:~$ net show bgp summary
show bgp ipv4 unicast summary
=============================
BGP router identifier 10.255.255.12, local AS number 65016 vrf-id 0
BGP table version 103
RIB entries 37, using 7104 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.254.254.45 4 65004 306764 261331 0 0 0 01:39:02 12
spine1-pod3(10.254.254.47) 4 65004 261602 261664 0 0 0 01w2d02h 16

Total number of neighbors 2

[...]

cumulus@cumulus:mgmt:~$ net show route | grep /32
B>* 10.255.255.1/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.2/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.3/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.4/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.5/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.6/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.7/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.8/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.9/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.10/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.11/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
C>* 10.255.255.12/32 is directly connected, lo, 01w2d02h
B>* 10.255.255.13/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.14/32 [20/0] via 10.254.254.47, swp2, weight 1, 01w2d02h

</syntaxhighlight>We can also verify that routing is ok by pinging remote loopback IPs from local loopback + check eBGP EVPN session status: <syntaxhighlight lang="bash">
# ping example from VyOS leaf
vyos@leaf1-pod1:~$ ping 10.255.255.12 source-address 10.255.255.5
PING 10.255.255.12 (10.255.255.12) from 10.255.255.5 : 56(84) bytes of data.
64 bytes from 10.255.255.12: icmp_seq=1 ttl=61 time=2.60 ms
64 bytes from 10.255.255.12: icmp_seq=2 ttl=61 time=2.54 ms
64 bytes from 10.255.255.12: icmp_seq=3 ttl=61 time=2.44 ms

# check of EVPN sessions on the different vendors

# vyos

vyos@leaf1-pod1:~$ show bgp l2vpn evpn summary
BGP router identifier 10.255.255.5, local AS number 65011 vrf-id 0
BGP table version 0
RIB entries 31, using 5704 bytes of memory
Peers 2, using 1446 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
10.255.255.1 4 65001 30205 30208 0 0 0 02w6d21h 13 13 N/A
10.255.255.2 4 65001 17392 17429 0 0 0 01w5d00h 11 13 N/A

# Arista

leaf1-pod3#show bgp evpn summary
BGP summary information for VRF default
Router identifier 10.255.255.11, local AS number 65015
Neighbor Status Codes: m - Under maintenance
Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
OVL_spine1-pod3 10.255.255.13 4 65004 15608 15503 0 0 00:03:47 Estab 17 17
OVL_spine2-pod3 10.255.255.14 4 65004 262780 308558 0 0 9d02h Estab 13 13

# Cumulus

cumulus@cumulus:mgmt:~$ net show bgp l2vpn evpn summary
BGP router identifier 10.255.255.12, local AS number 65016 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.255.255.13 4 65004 307243 261693 0 0 0 00:04:25 14
spine1-pod3(10.255.255.14) 4 65004 262001 262025 0 0 0 01w2d02h 10

</syntaxhighlight>

On EVPN, we should receive quite the same number of prefixes from both spine, but looks like there differences of best path election + redistribution between vendors. Perhaps something to dig. The most important point is to verify that the FIB for the services are consistent.

==== L2 service implementation ====
On that topic, we'll firstly focus on the Layer 2 service. To be sure that all works as expected we have to check that:

* mac-address-table are consistent on the whole fabric. In traditionnal L2 setup, switch FIBs depend of the traffic passing through them and a mac @ can be see "everywhere" if it does a BUM on the network. In VXLAN, in a mac-vrf topology without filer, even a host doesn't send BUM, the mac address is propagated on the whole fabric (within BGP update). But, as it is on traditional L2 topology, you have to be sure that mac @ table aging time are consistent on all devices, with an aging time superior to ARP timer on hosts.
* routes type 2 are visible on each leaf nodes for a same vni, routes type 3 to reach remote VTEP have to be learned
* a common route-target for the VRF routes need to be used/set within policies. In effect, there are differences between vendors on route "tagging" (features to auto generate route-target, "hard coded" alogos, etc.). Here, on VyOS, we got to add adminstrative extended communities within route-map for the L2 VNI. We'll show it below
* all fabric connections (interconnections between network nodes) have the sufficient "IP MTU". Please keep in mind that VXLAN overhead has a size of 50 bytes. So, to be sure that Jumbo traffic for host is ok, configure it at least at 9050 Bytes. I would advise to push it at 9100 Bytes in case of VXLAN usage on client HOSTS for Virtualized setup on servers.

Check FDB on leaf switch + debug type 2 routes' propagation for leaf1 pod1<syntaxhighlight lang="bash">
##############
# VyOS
##############

vyos@leaf1-pod1:~$ show bridge br100 fdb
50:01:00:01:00:02 dev eth2 vlan 1 master br100 permanent
50:01:00:01:00:02 dev eth2 master br100 permanent
33:33:00:00:00:01 dev eth2 self permanent
33:33:00:00:00:02 dev eth2 self permanent
01:00:5e:00:00:01 dev eth2 self permanent
01:80:c2:00:00:0e dev eth2 self permanent
01:80:c2:00:00:03 dev eth2 self permanent
01:80:c2:00:00:00 dev eth2 self permanent
33:33:00:00:00:01 dev br100 self permanent
33:33:00:00:00:02 dev br100 self permanent
33:33:ff:01:00:02 dev br100 self permanent
01:00:5e:00:00:6a dev br100 self permanent
33:33:00:00:00:6a dev br100 self permanent
01:00:5e:00:00:01 dev br100 self permanent
33:33:ff:00:00:00 dev br100 self permanent
00:50:79:66:68:15 dev vxlan100 vlan 1 extern_learn master br100
00:50:79:66:68:15 dev vxlan100 extern_learn master br100
8a:d0:00:16:1c:17 dev vxlan100 vlan 1 master br100 permanent
8a:d0:00:16:1c:17 dev vxlan100 master br100 permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.6 self permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.11 self permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.12 self permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.7 self permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.8 self permanent
00:50:79:66:68:15 dev vxlan100 dst 10.255.255.12 self extern_learn

# and to focus on remote nodes

vyos@leaf1-pod1:~$ show bridge br100 fdb | grep "self extern_learn"
00:50:79:66:68:15 dev vxlan100 dst 10.255.255.12 self extern_learn
00:50:79:66:68:13 dev vxlan100 dst 10.255.255.11 self extern_learn
00:50:79:66:68:19 dev vxlan100 dst 10.255.255.8 self extern_learn
00:50:79:66:68:0c dev vxlan100 dst 10.255.255.6 self extern_learn
00:50:79:66:68:17 dev vxlan100 dst 10.255.255.7 self extern_learn

# or on local clients

vyos@leaf1-pod1:~$ show bridge br100 fdb | grep eth | grep -v perm
00:50:79:66:68:0b dev eth2 master br100

# Now if we want to focus on route type 3 learned

Displayed 7 prefixes (12 paths) (of requested type)
vyos@leaf1-pod1:~$ show bgp l2vpn evpn route type 3 | grep Dis
Route Distinguisher: 10.255.255.5:3
Route Distinguisher: 10.255.255.6:3
Route Distinguisher: 10.255.255.7:2
Route Distinguisher: 10.255.255.8:2
Route Distinguisher: 10.255.255.11:100
Route Distinguisher: 10.255.255.12:100

# we notice that RDs for VyOS nodes don't respect the logic of <Loopback:vni>. It s because for the moment we can't set it by vni (or I didn't find how to do that)

##############
# Arista
##############

# we check the fdb. Remote host are behind the port Vx1

leaf1-pod3#show mac address-table vlan 100
Mac Address Table
------------------------------------------------------------------

Vlan Mac Address Type Ports Moves Last Move
---- ----------- ---- ----- ----- ---------
100 0050.7966.680c DYNAMIC Vx1 1 0:00:38 ago
100 0050.7966.6813 DYNAMIC Et3 1 0:14:08 ago
100 0050.7966.6815 DYNAMIC Vx1 1 9 days, 2:36:39 ago
100 0050.7966.6817 DYNAMIC Vx1 1 0:00:36 ago
100 0050.7966.6819 DYNAMIC Vx1 1 0:00:35 ago
Total Mac Addresses for this criterion: 5

# we check that all VTEP are ready for BUM

leaf1-pod3#show bgp evpn route-type imet | i >
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
* >Ec RD: 10.255.255.5:3 imet 10.255.255.5
* >Ec RD: 10.255.255.6:3 imet 10.255.255.6
* >Ec RD: 10.255.255.7:2 imet 10.255.255.7
* >Ec RD: 10.255.255.8:2 imet 10.255.255.8
* > RD: 10.255.255.11:100 imet 10.255.255.11
* >Ec RD: 10.255.255.12:100 imet 10.255.255.12
[...]

leaf1-pod3#show vxlan vtep detail
Remote VTEPS for Vxlan1:

VTEP Learned Via MAC Address Learning Tunnel Type(s)
------------------ ------------------ -------------------------- --------------
10.255.255.5 control plane control plane flood, unicast
10.255.255.6 control plane control plane flood, unicast
10.255.255.7 control plane control plane flood, unicast
10.255.255.8 control plane control plane flood, unicast
10.255.255.12 control plane control plane flood, unicast

##############
# Cumulus
##############

# we check the fdb. Remote host are behind the port Vx1

cumulus@cumulus:mgmt:~$ net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ ----------------
100 bridge bridge 50:01:00:11:00:03 permanent 9 days, 02:47:42
100 bridge swp3 00:50:79:66:68:15 00:00:06
100 bridge vni100 00:50:79:66:68:0c extern_learn 00:00:40
100 bridge vni100 00:50:79:66:68:13 extern_learn 00:21:01
100 bridge vni100 00:50:79:66:68:17 extern_learn 00:00:38
100 bridge vni100 00:50:79:66:68:19 extern_learn 00:00:36

# --> here we see that it misses the @mac from the host on leaf1 pod1.
# we check the evpn routes on the Cumulus node and we see that from leaf1 pod1 vni 100 we have only those routes

Route Distinguisher: 10.255.255.5:3
* [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
* [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
*> [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8

# --> something is wrong on the remote leaf.
# on the leaf we check the policy for the EVPN routes export

route-map RM-OUT-VNI {
rule 1 {
action permit
match {
evpn {
route-type multicast
vni 100
}
}
set {
extcommunity {
rt 65000:100
}
}
}
rule 2 {
action permit
match {
evpn {
route-type prefix
vni 100
}
}
set {
extcommunity {
rt 65000:100
}
}
}
rule 3 {
action permit
match {
evpn {
vni 200
}
}
set {
extcommunity {
rt 65000:200
}

# we notice that a typo in rule numbering removed the export of macip routes for vni 100.
# we correct that :

route-map RM-OUT-VNI {
rule 1 {
action permit
match {
evpn {
route-type multicast
vni 100
}
}
set {
extcommunity {
rt 65000:100
}
}
}
rule 2 {
action permit
match {
evpn {
route-type macip
vni 100
}
}
set {
extcommunity {
rt 65000:100
}
}
}
rule 3 {
action permit
match {
evpn {
route-type prefix
vni 200
}
}
set {
extcommunity {
rt 65000:200
}

# once corrected we check the fdb on the cumulus node + evpn routes

cumulus@cumulus:mgmt:~$ net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ ----------------
100 bridge bridge 50:01:00:11:00:03 permanent 9 days, 02:51:48
100 bridge swp3 00:50:79:66:68:15 00:02:01
100 bridge vni100 00:50:79:66:68:0b extern_learn 00:00:26
100 bridge vni100 00:50:79:66:68:0c extern_learn 00:04:46
100 bridge vni100 00:50:79:66:68:13 extern_learn 00:25:08
100 bridge vni100 00:50:79:66:68:17 extern_learn 00:04:45
100 bridge vni100 00:50:79:66:68:19 extern_learn 00:04:43

Route Distinguisher: 10.255.255.5:3
* [2]:[0]:[48]:[00:50:79:66:68:0b]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
* [2]:[0]:[48]:[00:50:79:66:68:0b]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
*> [2]:[0]:[48]:[00:50:79:66:68:0b]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
* [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
* [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
*> [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8

# we have now Type 2 routes, all is ok now and we learn the client mac 00:50:79:66:68:0b
# Nevertheless, the trafic was working between the client hosts, but trafic from Cumulus leaf client to VyOS leaf client was broadcasted to all leaf since the @mac was not advertised on the fabric

</syntaxhighlight>It could be interesting to check how each node is configured to deliver the service, but the article would be too long. So, to sum the steps to provide a L2 service:

# create a VLAN (on Arista) / bridge (VyOS and Cumulus) + associate it to the VNI/VXLAN instances/interface
# create the mac vrf + define the route-target into bgp (Arista + Cumulus) / update the route-maps + community list to allow the redistribution of the VNI routes (type 2 and type 3)

==== L3 service implementation ====
Now, we'll now look at the L3 service. On that kind of setup we need to implement / check:
* leaf IP subnets are well redistributed betweeb all leaf (here we are in a vrf "any to any" topology)
* we only redistribute routes for IP prefix. Routes type 2 (mac @) don't need to be redistributed between leaf
* that all clients can ping each other through the L3VPN

To achieve that, we need to implement analog configuration than for the L2 service, but with the following differences:

* create a VLAN (on Arista) / bridge (VyOS and Cumulus) + associate it to the VNI/VXLAN instances|interface
* create the L3 VRF + define the route-target into bgp (Arista + Cumulus + VyOS) + allow connected routes distribution (Arista / Cumulus) / update the route-maps + community list to allow the redistribution of the VNI routes (type 5) for VyOS / restrict the routes redistribution to type 5 routes into the VNI (Cumulus)

Now we check the routing table with common cli commands + into the EVPN RIB :

<syntaxhighlight lang="bash">
# check routes into the vrf

vyos@leaf1-pod1:~$ show ip route vrf VRF-L3-1
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF VRF-L3-1:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 02w6d22h
C>* 192.168.10.0/24 is directly connected, br200, 02w6d22h
B>* 192.168.20.0/24 [20/0] via 10.255.255.6, br200 onlink, weight 1, 05:11:08
B>* 192.168.30.0/24 [20/0] via 10.255.255.7, br200 onlink, weight 1, 05:04:15
B>* 192.168.40.0/24 [20/0] via 10.255.255.8, br200 onlink, weight 1, 04:59:41
B>* 192.168.50.0/24 [20/0] via 10.255.255.11, br200 onlink, weight 1, 05:11:08
B>* 192.168.60.0/24 [20/0] via 10.255.255.12, br200 onlink, weight 1, 05:11:08

leaf1-pod3#show ip route vrf VRF-L3-1

VRF: VRF-L3-1
Codes: C - connected, S - static, K - kernel,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,
R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
NG - Nexthop Group Static Route, V - VXLAN Control Service,
DH - DHCP client installed default route, M - Martian,
DP - Dynamic Policy Route, L - VRF Leaked,
G - gRIBI, RC - Route Cache Route

Gateway of last resort is not set

B E 192.168.10.0/24 [200/0] via VTEP 10.255.255.5 VNI 200 router-mac 12:9b:9b:d8:f6:86 local-interface Vxlan1
B E 192.168.20.0/24 [200/0] via VTEP 10.255.255.6 VNI 200 router-mac 50:01:00:02:00:03 local-interface Vxlan1
B E 192.168.30.0/24 [200/0] via VTEP 10.255.255.7 VNI 200 router-mac 50:01:00:09:00:03 local-interface Vxlan1
B E 192.168.40.0/24 [200/0] via VTEP 10.255.255.8 VNI 200 router-mac 50:01:00:0a:00:03 local-interface Vxlan1
C 192.168.50.0/24 is directly connected, Vlan200
B E 192.168.60.0/24 [200/0] via VTEP 10.255.255.12 VNI 200 router-mac 50:01:00:11:00:03 local-interface Vxlan1

cumulus@cumulus:mgmt:~$ net show route vrf VRF-L3-1
show ip route vrf VRF-L3-1
===========================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued route, r - rejected route

VRF VRF-L3-1:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 01w2d03h
B>* 192.168.10.0/24 [20/0] via 10.255.255.5, vlan200 onlink, weight 1, 03:18:37
B>* 192.168.20.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 03:18:37
B>* 192.168.30.0/24 [20/0] via 10.255.255.7, vlan200 onlink, weight 1, 03:18:37
B>* 192.168.40.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:18:37
B>* 192.168.50.0/24 [20/0] via 10.255.255.11, vlan200 onlink, weight 1, 03:18:37
C>* 192.168.60.0/24 is directly connected, vlan200, 01w2d03h

# we can notice that the output are quite similar, except on Arista which indicates the VNI for remote targets, instead of referencing the local bridge / vlan200

# Now we check the routes for the VNI 200

# VyOS

vyos@leaf1-pod1:~$ show bgp l2vpn evpn route type prefix
BGP table version is 1, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:200
*> [5]:[0]:[24]:[192.168.10.0]
10.255.255.5 0 32768 ?
ET:8 RT:65000:200 Rmac:12:9b:9b:d8:f6:86
Route Distinguisher: 10.255.255.6:200
*> [5]:[0]:[24]:[192.168.20.0]
10.255.255.6 0 65001 65012 ?
RT:65000:200 ET:8 Rmac:50:01:00:02:00:03
Route Distinguisher: 10.255.255.7:200
* [5]:[0]:[24]:[192.168.30.0]
10.255.255.7 0 65001 65003 65002 65013 ?
RT:65000:200 RT:65013:200 ET:8 Rmac:50:01:00:09:00:03
*> [5]:[0]:[24]:[192.168.30.0]
10.255.255.7 0 65001 65003 65002 65013 ?
RT:65000:200 RT:65013:200 ET:8 Rmac:50:01:00:09:00:03
Route Distinguisher: 10.255.255.8:200
*> [5]:[0]:[24]:[192.168.40.0]
10.255.255.8 0 65001 65003 65002 65014 ?
RT:65000:200 RT:65014:200 ET:8 Rmac:50:01:00:0a:00:03
* [5]:[0]:[24]:[192.168.40.0]
10.255.255.8 0 65001 65003 65002 65014 ?
RT:65000:200 RT:65014:200 ET:8 Rmac:50:01:00:0a:00:03
Route Distinguisher: 10.255.255.11:200
* [5]:[0]:[24]:[192.168.50.0]
10.255.255.11 0 65001 65003 65004 65015 i
RT:65000:200 ET:8 Rmac:50:01:00:cd:ff:68
*> [5]:[0]:[24]:[192.168.50.0]
10.255.255.11 0 65001 65003 65004 65015 i
RT:65000:200 ET:8 Rmac:50:01:00:cd:ff:68
Route Distinguisher: 10.255.255.12:10200
* [5]:[0]:[24]:[192.168.60.0]
10.255.255.12 0 65001 65003 65004 65016 i
RT:65000:200 ET:8 Rmac:50:01:00:11:00:03
*> [5]:[0]:[24]:[192.168.60.0]
10.255.255.12 0 65001 65003 65004 65016 i
RT:65000:200 ET:8 Rmac:50:01:00:11:00:03

# Arista

leaf1-pod3#show bgp evpn route ip-prefix ipv4
BGP routing table information for VRF default
Router identifier 10.255.255.11, local AS number 65015
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric LocPref Weight Path
* > RD: 10.255.255.5:200 ip-prefix 192.168.10.0/24
10.255.255.5 - 100 0 65004 65003 65001 65011 ?
* RD: 10.255.255.5:200 ip-prefix 192.168.10.0/24
10.255.255.5 - 100 0 65004 65003 65001 65011 ?
* > RD: 10.255.255.6:200 ip-prefix 192.168.20.0/24
10.255.255.6 - 100 0 65004 65003 65001 65012 ?
* RD: 10.255.255.6:200 ip-prefix 192.168.20.0/24
10.255.255.6 - 100 0 65004 65003 65001 65012 ?
* > RD: 10.255.255.7:200 ip-prefix 192.168.30.0/24
10.255.255.7 - 100 0 65004 65003 65002 65013 ?
* RD: 10.255.255.7:200 ip-prefix 192.168.30.0/24
10.255.255.7 - 100 0 65004 65003 65002 65013 ?
* > RD: 10.255.255.8:200 ip-prefix 192.168.40.0/24
10.255.255.8 - 100 0 65004 65003 65002 65014 ?
* RD: 10.255.255.8:200 ip-prefix 192.168.40.0/24
10.255.255.8 - 100 0 65004 65003 65002 65014 ?
* > RD: 10.255.255.11:200 ip-prefix 192.168.50.0/24
- - - 0 i
* > RD: 10.255.255.12:10200 ip-prefix 192.168.60.0/24
10.255.255.12 - 100 0 65004 65016 i
* RD: 10.255.255.12:10200 ip-prefix 192.168.60.0/24
10.255.255.12 - 100 0 65004 65016 i

# CVX

cumulus@cumulus:mgmt:~$ net show bgp l2vpn evpn route type prefix
BGP table version is 7, local router ID is 10.255.255.12
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:200
* [5]:[0]:[24]:[192.168.10.0]
10.255.255.5 0 65004 65003 65001 65011 ?
RT:65000:200 ET:8 Rmac:12:9b:9b:d8:f6:86
*> [5]:[0]:[24]:[192.168.10.0]
10.255.255.5 0 65004 65003 65001 65011 ?
RT:65000:200 ET:8 Rmac:12:9b:9b:d8:f6:86
Route Distinguisher: 10.255.255.6:200
* [5]:[0]:[24]:[192.168.20.0]
10.255.255.6 0 65004 65003 65001 65012 ?
RT:65000:200 ET:8 Rmac:50:01:00:02:00:03
*> [5]:[0]:[24]:[192.168.20.0]
10.255.255.6 0 65004 65003 65001 65012 ?
RT:65000:200 ET:8 Rmac:50:01:00:02:00:03
Route Distinguisher: 10.255.255.7:200
* [5]:[0]:[24]:[192.168.30.0]
10.255.255.7 0 65004 65003 65002 65013 ?
RT:65000:200 RT:65013:200 ET:8 Rmac:50:01:00:09:00:03
*> [5]:[0]:[24]:[192.168.30.0]
10.255.255.7 0 65004 65003 65002 65013 ?
RT:65000:200 RT:65013:200 ET:8 Rmac:50:01:00:09:00:03
Route Distinguisher: 10.255.255.8:200
* [5]:[0]:[24]:[192.168.40.0]
10.255.255.8 0 65004 65003 65002 65014 ?
RT:65000:200 RT:65014:200 ET:8 Rmac:50:01:00:0a:00:03
*> [5]:[0]:[24]:[192.168.40.0]
10.255.255.8 0 65004 65003 65002 65014 ?
RT:65000:200 RT:65014:200 ET:8 Rmac:50:01:00:0a:00:03
Route Distinguisher: 10.255.255.11:200
* [5]:[0]:[24]:[192.168.50.0]
10.255.255.11 0 65004 65015 i
RT:65000:200 ET:8 Rmac:50:01:00:cd:ff:68
*> [5]:[0]:[24]:[192.168.50.0]
10.255.255.11 0 65004 65015 i
RT:65000:200 ET:8 Rmac:50:01:00:cd:ff:68
Route Distinguisher: 10.255.255.12:10200
*> [5]:[0]:[24]:[192.168.60.0]
10.255.255.12 0 32768 i
ET:8 RT:65000:200 Rmac:50:01:00:11:00:03

# Now we check with traceroute on the 3 clients connected on focused leaf switches

# client leaf VyOS

VPCS> ping 192.168.50.2

84 bytes from 192.168.50.2 icmp_seq=1 ttl=62 time=16.456 ms
84 bytes from 192.168.50.2 icmp_seq=2 ttl=62 time=11.852 ms
^C
VPCS> ping 192.168.60.2

84 bytes from 192.168.60.2 icmp_seq=1 ttl=62 time=5.554 ms
84 bytes from 192.168.60.2 icmp_seq=2 ttl=62 time=4.432 ms
^C
VPCS> trace 192.168.50.2
trace to 192.168.50.2, 8 hops max, press Ctrl+C to stop
1 192.168.10.1 0.499 ms 3.143 ms 1.432 ms
2 192.168.50.1 10.465 ms 12.512 ms 8.992 ms
3 *192.168.50.2 11.056 ms (ICMP type:3, code:3, Destination port unreachable)

VPCS> trace 192.168.60.2
trace to 192.168.60.2, 8 hops max, press Ctrl+C to stop
1 192.168.10.1 0.525 ms 0.312 ms 0.278 ms
2 192.168.60.1 4.949 ms 4.232 ms 4.022 ms
3 *192.168.60.2 4.167 ms (ICMP type:3, code:3, Destination port unreachable)

# client leaf Arista

VPCS> ping 192.168.10.2

84 bytes from 192.168.10.2 icmp_seq=1 ttl=62 time=10.914 ms
^C
VPCS> ping 192.168.60.2

84 bytes from 192.168.60.2 icmp_seq=1 ttl=62 time=11.755 ms
^C
VPCS> trace 192.168.10.2
trace to 192.168.10.2, 8 hops max, press Ctrl+C to stop
1 192.168.50.1 4.742 ms 4.123 ms 4.293 ms
2 192.168.10.1 12.550 ms 15.039 ms 9.411 ms
3 *192.168.10.2 8.939 ms (ICMP type:3, code:3, Destination port unreachable)

VPCS>
VPCS> trace 192.168.60.2
trace to 192.168.60.2, 8 hops max, press Ctrl+C to stop
1 192.168.50.1 3.073 ms 2.901 ms 2.585 ms
2 192.168.60.1 10.815 ms 11.493 ms 12.144 ms
3 *192.168.60.2 10.429 ms (ICMP type:3, code:3, Destination port unreachable)

# client leaf Cumulus

VPCS> ping 192.168.10.2

84 bytes from 192.168.10.2 icmp_seq=1 ttl=62 time=4.731 ms
^C
VPCS> ping 192.168.50.2

84 bytes from 192.168.50.2 icmp_seq=1 ttl=62 time=11.722 ms
^C
VPCS> trace 192.168.50.2
trace to 192.168.50.2, 8 hops max, press Ctrl+C to stop
1 192.168.60.1 0.487 ms 0.266 ms 0.248 ms
2 192.168.50.1 6.691 ms 6.281 ms 6.613 ms
3 *192.168.50.2 9.355 ms (ICMP type:3, code:3, Destination port unreachable)

VPCS>
VPCS> trace 192.168.10.2
trace to 192.168.10.2, 8 hops max, press Ctrl+C to stop
1 192.168.60.1 0.436 ms 0.315 ms 0.380 ms
2 192.168.10.1 4.535 ms 4.661 ms 3.994 ms
3 *192.168.10.2 4.292 ms (ICMP type:3, code:3, Destination port unreachable)

</syntaxhighlight>

=== Conclusion of the lab ===
These first results are satisfying, it comforts the idea that VXLAN is a strong option to deliver L2/L3 services in datacenters, but not only. This kind of architecture is more and more chosen in campus designs, PRA, NFV solutions.

The support of this technology on more and more open source NOS are encouraging regarding the innovation. If we look at Vyatta project for example, whom OS can be compiled "brick per brick", we could imagine to create NFV CPEs providing VXLAN fabric extensions of some client VRFs. CPE could be connected to the central Fabric within encrypted tunnels (wireguard,ipsec) or through direct link over MacSec (this feature in on the Vyatta Roadmap).

Please contact us if you want to feed the topic / discuss the lab.

Cheers

Pierre L.

VyosAristaCVX-VXLAN-lab

2022-04-05T12:37:45Z

PierreLancastre:

=== Introduction ===
More and more, EVPN is making its place in Datacenter field. Mainstream vendors integrate its support, with VXLAN or MPLS for data plane + BGP for control plane. In previous labs, we were able to observe that interoperability was quite good between vendors. Nevertheless, even we've already tested CumulusVX, there are also other open sources NOS which can support EVPN.

Today we'll test VyOS in it's latest rolling version, which handles the support of EVPN.

The targets of this lab are to :

* create a VXLAN Fabric with VyOS nodes
* provide a L2VPN and a L3VPN to client nodes within the fabric
* integrate some other vendors node into the fabric, here Arista and Cumulus VX
* test that all works as expected

=== Lab setup ===
To reach those goals, we'll use the EVE-NG tool in community edition (https://www.eve-ng.net/index.php/download/#DL-COMM) and use the following NOS version :

* Vyatta VyOS 1.4 rolling <-- free to download https://vyos.net/get/nightly-builds/
* Arista vEOS 4.26.0F lab edition <-- need a support account
* Nvidia Cumulus VX 4.2.1 lab edition <-- free to download https://www.nvidia.com/en-us/networking/ethernet-switching/cumulus-vx/download/

We'll build a 3-tiers fabric, with leaf/spine/superspine devices, VXLAN/EVPN running on all nodes. It gives the following setup:

[[Datei:nesevo-Lab-Vyatta-Arista-CVX.png|rahmenlos|600x600px]]

On EVE-NG, we add IP information, ASN, loopback IPs. All other information will be past after when dealing with technical details.

So, it gives:

[[Datei:2022-02_eveng-lab-vyatta-cum-arista.png|rahmenlos|600x600px]]

==== <span class="mw-headline">VXLAN/BGP infos</span> ====
Note: in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>
{| class="wikitable"
|+
!service
!vlan id
!route-target
!VNI
!Commentaires
|-
|L2 cross-DC
|100
|target:65000:100
|100
|the 6 clients are in the same broadcast domain / subnet
|-
|L3VPN cros-DC
|200
|target:65000:200
|200
|the 6 clients have their own subnets. Here we use only one vni because no ESI will be implemented. One leaf one subnet.
|}
Client devices are represented by computers on EVE-NG. IPs in green are using the L2VPN service (extended layer 2 between pods) and IPs in orange the L3VPN service (routing service).

Below some diagrams showing the logic topologies :

[[Datei:Nesevo-Lab-Vyatta-Arista-CVX-l2vpn.png|rahmenlos|400x400px]]

[[https://wiki.nesevo.com/images/0/04/Nesevo-Lab-Vyatta-Arista-CVX-l3vpn.png|rahmenlos|400x400px]]

Below the link to the configurations to allow you to quickly reproduce the tab.

To be noted that the configurations are not "ready-to-prod " and need to be optimized (adding bfd, bgp optimization, etc.). Some typos can subsist. Whatever, the lab works and gives the expected results :)

Link to devices configurations : http://wiki.intra.nesevo.com/images/3/32/02.lab-vyos-eos-cvx_v2.zip

=== Lab details ===
Considering that the lab is up and running, we'll now come back to the architecture.

We have 3 kinds of network nodes :

* leaf switches, on edge, which provide L2 and L3 services.
* couple of spine switches which allow leaf from a same pod to exchange traffic. They are also connected to superspine to reach other pods
* superspine switches which interconnext the different spine switches from the different pods, thus allowing interpods traffic

All those network devices control plane can exchange routes' information within 2 layers of bgp : underlay and overlay.

The role of the "underlay" is to dynamically route the traffic between loopback interfaces from the different nodes. Loopback interfaces are used to mount BGP sessions for the underlay and are used as VTEP interfaces on leaf nodes. Generally, it's better to have 2 different Loopback for this usage (in case of anycast vtep setup for example), but some vendors don't allow to 2 loopback interfaces in the same routing context. Whatever, in our setup, having only one loopback is fine.

Here, the underlay is done in eBGP, <u>ipv4 unicast family only + route-map to only redistribute loopback interfaces IP</u>.

* Each node mount a session to the directly connected peers.
* Each leaf node has a dedicated ASN.
* Each couple of spine/superspine has a ASN for two, to follow the best practices to optimize the routing table (ECMP).

The role of the "overlay" is to exchange EVPN routes type 2,3,5 here (type 1/4 routes seem to not be managed by VyOS at the moment, TBC).

* eBGP multihop sessions are mounted to directly neighbor loopback IPs.
* policies/options are set to not change the original nexthop when propagating routes
* for L2 services we only allow the redistribution of type 2 (macip) and type 3 (inclusive mcast for BUM management) routes
* for L3 services we only allow the redistribution of type 5 (prefix) routes
* regarding the setup, to enhance the scalability, it could be nice to use dedicated nodes (VM or phys) to take the role of route-servers. Here, we could connect them to superspine. Thus, spine/superspine would only do underlay and could not be a bottleneck for the control plane scalability

Moreover, regarding the lab setup, no bgp pic / bfd / VMTO / other option are implemented to optimize the convergence / traffic delivery. The goal was to verify that VyOS is compatible in a basic but functionnal VXLAN setup.

Now, we'll take a look on the different output from the 2 vendors leaf / spine / superspine to verify that all works as expected.

We verify that underlay works well. All is fine if:

* we have an eBGP session per directly connected peer
* we receive the same number of prefixes per peer type:
** a leef has to receive the same number of prefixes from its spine neighbors
** spine neigbors have to receive the same amount of prefixes from superspine, and reciprocally
** all VTEP IPs have to be known into the GRT (global routing table / default routing instance)

====== On VyOS ======
Below we can see that we receive 17 prefixes from both neighbors and that we know all loopback IPs from the setup (13 remote, 1 local). We have a bit too much routes, because on Cumulus nodes the filters that I have configured seem misconstructed.

To be noted that spine will announce all the Loopback IPs except the one from it's twin, since they are not directly connected and superspine respect as loop protection constraint (don't reflect routes originated by an ASN "X" to a peer from the same ASN "X").<syntaxhighlight lang="bash">
vyos@leaf1-pod1:~$ show ip bgp summary established

IPv4 Unicast Summary (VRF default):
BGP router identifier 10.255.255.5, local AS number 65011 vrf-id 0
BGP table version 20
RIB entries 37, using 6808 bytes of memory
Peers 2, using 1446 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
10.254.254.9 4 65001 29993 29988 0 0 0 02w6d19h 17 19 N/A
10.254.254.11 4 65001 17218 17220 0 0 0 01w4d22h 17 19 N/A

Displayed neighbors 2
Total number of neighbors 2

vyos@leaf1-pod1:~$ show ip route | match /32
B>* 10.255.255.1/32 [20/0] via 10.254.254.9, eth0, weight 1, 02w6d19h
B>* 10.255.255.2/32 [20/0] via 10.254.254.11, eth1, weight 1, 01w4d22h
B>* 10.255.255.3/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.4/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
C>* 10.255.255.5/32 is directly connected, dum0, 02w6d19h
B>* 10.255.255.6/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d22h
B>* 10.255.255.7/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.8/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.9/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.10/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.11/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w2d01h
B>* 10.255.255.12/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w2d00h
B>* 10.255.255.13/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w2d01h
B>* 10.255.255.14/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w2d00h

</syntaxhighlight>

====== On Arista ======
We verify that we receive prefix + same leaf / superspine / remote pods spine IPs (we note there misses some filters on spine since we receive /31 interco routes): <syntaxhighlight lang="bash">
leaf1-pod3#show ip bgp neighbors 10.254.254.43 received-routes
BGP routing table information for VRF default
Router identifier 10.255.255.11, local AS number 65015
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup, L - labeled-unicast
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI Origin Validation codes: V - valid, I - invalid, U - unknown
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric AIGP LocPref Weight Path
10.254.254.36/31 10.254.254.43 0 - - - 65004 ?
10.254.254.38/31 10.254.254.43 0 - - - 65004 ?
10.254.254.42/31 10.254.254.43 0 - - - 65004 ?
10.254.254.44/31 10.254.254.43 - - - - 65004 65016 ?
10.254.254.46/31 10.254.254.43 0 - - - 65004 ?
* 10.255.255.1/32 10.254.254.43 - - - - 65004 65003 65001 ?
* 10.255.255.2/32 10.254.254.43 - - - - 65004 65003 65001 ?
* 10.255.255.3/32 10.254.254.43 - - - - 65004 65003 65002 ?
* 10.255.255.4/32 10.254.254.43 - - - - 65004 65003 65002 ?
* 10.255.255.5/32 10.254.254.43 - - - - 65004 65003 65001 65011 ?
* 10.255.255.6/32 10.254.254.43 - - - - 65004 65003 65001 65012 ?
* 10.255.255.7/32 10.254.254.43 - - - - 65004 65003 65002 65013 ?
* 10.255.255.8/32 10.254.254.43 - - - - 65004 65003 65002 65014 ?
* 10.255.255.9/32 10.254.254.43 - - - - 65004 65003 ?
* 10.255.255.10/32 10.254.254.43 - - - - 65004 65003 ?
* 10.255.255.12/32 10.254.254.43 - - - - 65004 65016 ?
* > 10.255.255.14/32 10.254.254.43 0 - - - 65004 ?
leaf1-pod3#show ip bgp neighbors 10.254.254.41 received-routes
BGP routing table information for VRF default
Router identifier 10.255.255.11, local AS number 65015
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup, L - labeled-unicast
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI Origin Validation codes: V - valid, I - invalid, U - unknown
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric AIGP LocPref Weight Path
* > 10.255.255.1/32 10.254.254.41 - - - - 65004 65003 65001 ?
* > 10.255.255.2/32 10.254.254.41 - - - - 65004 65003 65001 ?
* > 10.255.255.3/32 10.254.254.41 - - - - 65004 65003 65002 ?
* > 10.255.255.4/32 10.254.254.41 - - - - 65004 65003 65002 ?
* > 10.255.255.5/32 10.254.254.41 - - - - 65004 65003 65001 65011 ?
* > 10.255.255.6/32 10.254.254.41 - - - - 65004 65003 65001 65012 ?
* > 10.255.255.7/32 10.254.254.41 - - - - 65004 65003 65002 65013 ?
* > 10.255.255.8/32 10.254.254.41 - - - - 65004 65003 65002 65014 ?
* > 10.255.255.9/32 10.254.254.41 - - - - 65004 65003 ?
* > 10.255.255.10/32 10.254.254.41 - - - - 65004 65003 ?
* > 10.255.255.12/32 10.254.254.41 - - - - 65004 65016 ?
* > 10.255.255.13/32 10.254.254.41 - - - - 65004 i

</syntaxhighlight>

====== On Cumulus ======
<syntaxhighlight lang="bash">
cumulus@cumulus:mgmt:~$ net show bgp summary
show bgp ipv4 unicast summary
=============================
BGP router identifier 10.255.255.12, local AS number 65016 vrf-id 0
BGP table version 103
RIB entries 37, using 7104 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.254.254.45 4 65004 306764 261331 0 0 0 01:39:02 12
spine1-pod3(10.254.254.47) 4 65004 261602 261664 0 0 0 01w2d02h 16

Total number of neighbors 2

[...]

cumulus@cumulus:mgmt:~$ net show route | grep /32
B>* 10.255.255.1/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.2/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.3/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.4/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.5/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.6/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.7/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.8/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.9/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.10/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.11/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
C>* 10.255.255.12/32 is directly connected, lo, 01w2d02h
B>* 10.255.255.13/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.14/32 [20/0] via 10.254.254.47, swp2, weight 1, 01w2d02h

</syntaxhighlight>We can also verify that routing is ok by pinging remote loopback IPs from local loopback + check eBGP EVPN session status: <syntaxhighlight lang="bash">
# ping example from VyOS leaf
vyos@leaf1-pod1:~$ ping 10.255.255.12 source-address 10.255.255.5
PING 10.255.255.12 (10.255.255.12) from 10.255.255.5 : 56(84) bytes of data.
64 bytes from 10.255.255.12: icmp_seq=1 ttl=61 time=2.60 ms
64 bytes from 10.255.255.12: icmp_seq=2 ttl=61 time=2.54 ms
64 bytes from 10.255.255.12: icmp_seq=3 ttl=61 time=2.44 ms

# check of EVPN sessions on the different vendors

# vyos

vyos@leaf1-pod1:~$ show bgp l2vpn evpn summary
BGP router identifier 10.255.255.5, local AS number 65011 vrf-id 0
BGP table version 0
RIB entries 31, using 5704 bytes of memory
Peers 2, using 1446 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
10.255.255.1 4 65001 30205 30208 0 0 0 02w6d21h 13 13 N/A
10.255.255.2 4 65001 17392 17429 0 0 0 01w5d00h 11 13 N/A

# Arista

leaf1-pod3#show bgp evpn summary
BGP summary information for VRF default
Router identifier 10.255.255.11, local AS number 65015
Neighbor Status Codes: m - Under maintenance
Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
OVL_spine1-pod3 10.255.255.13 4 65004 15608 15503 0 0 00:03:47 Estab 17 17
OVL_spine2-pod3 10.255.255.14 4 65004 262780 308558 0 0 9d02h Estab 13 13

# Cumulus

cumulus@cumulus:mgmt:~$ net show bgp l2vpn evpn summary
BGP router identifier 10.255.255.12, local AS number 65016 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.255.255.13 4 65004 307243 261693 0 0 0 00:04:25 14
spine1-pod3(10.255.255.14) 4 65004 262001 262025 0 0 0 01w2d02h 10

</syntaxhighlight>

On EVPN, we should receive quite the same number of prefixes from both spine, but looks like there differences of best path election + redistribution between vendors. Perhaps something to dig. The most important point is to verify that the FIB for the services are consistent.

==== L2 service implementation ====
On that topic, we'll firstly focus on the Layer 2 service. To be sure that all works as expected we have to check that:

* mac-address-table are consistent on the whole fabric. In traditionnal L2 setup, switch FIBs depend of the traffic passing through them and a mac @ can be see "everywhere" if it does a BUM on the network. In VXLAN, in a mac-vrf topology without filer, even a host doesn't send BUM, the mac address is propagated on the whole fabric (within BGP update). But, as it is on traditional L2 topology, you have to be sure that mac @ table aging time are consistent on all devices, with an aging time superior to ARP timer on hosts.
* routes type 2 are visible on each leaf nodes for a same vni, routes type 3 to reach remote VTEP have to be learned
* a common route-target for the VRF routes need to be used/set within policies. In effect, there are differences between vendors on route "tagging" (features to auto generate route-target, "hard coded" alogos, etc.). Here, on VyOS, we got to add adminstrative extended communities within route-map for the L2 VNI. We'll show it below
* all fabric connections (interconnections between network nodes) have the sufficient "IP MTU". Please keep in mind that VXLAN overhead has a size of 50 bytes. So, to be sure that Jumbo traffic for host is ok, configure it at least at 9050 Bytes. I would advise to push it at 9100 Bytes in case of VXLAN usage on client HOSTS for Virtualized setup on servers.

Check FDB on leaf switch + debug type 2 routes' propagation for leaf1 pod1<syntaxhighlight lang="bash">
##############
# VyOS
##############

vyos@leaf1-pod1:~$ show bridge br100 fdb
50:01:00:01:00:02 dev eth2 vlan 1 master br100 permanent
50:01:00:01:00:02 dev eth2 master br100 permanent
33:33:00:00:00:01 dev eth2 self permanent
33:33:00:00:00:02 dev eth2 self permanent
01:00:5e:00:00:01 dev eth2 self permanent
01:80:c2:00:00:0e dev eth2 self permanent
01:80:c2:00:00:03 dev eth2 self permanent
01:80:c2:00:00:00 dev eth2 self permanent
33:33:00:00:00:01 dev br100 self permanent
33:33:00:00:00:02 dev br100 self permanent
33:33:ff:01:00:02 dev br100 self permanent
01:00:5e:00:00:6a dev br100 self permanent
33:33:00:00:00:6a dev br100 self permanent
01:00:5e:00:00:01 dev br100 self permanent
33:33:ff:00:00:00 dev br100 self permanent
00:50:79:66:68:15 dev vxlan100 vlan 1 extern_learn master br100
00:50:79:66:68:15 dev vxlan100 extern_learn master br100
8a:d0:00:16:1c:17 dev vxlan100 vlan 1 master br100 permanent
8a:d0:00:16:1c:17 dev vxlan100 master br100 permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.6 self permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.11 self permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.12 self permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.7 self permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.8 self permanent
00:50:79:66:68:15 dev vxlan100 dst 10.255.255.12 self extern_learn

# and to focus on remote nodes

vyos@leaf1-pod1:~$ show bridge br100 fdb | grep "self extern_learn"
00:50:79:66:68:15 dev vxlan100 dst 10.255.255.12 self extern_learn
00:50:79:66:68:13 dev vxlan100 dst 10.255.255.11 self extern_learn
00:50:79:66:68:19 dev vxlan100 dst 10.255.255.8 self extern_learn
00:50:79:66:68:0c dev vxlan100 dst 10.255.255.6 self extern_learn
00:50:79:66:68:17 dev vxlan100 dst 10.255.255.7 self extern_learn

# or on local clients

vyos@leaf1-pod1:~$ show bridge br100 fdb | grep eth | grep -v perm
00:50:79:66:68:0b dev eth2 master br100

# Now if we want to focus on route type 3 learned

Displayed 7 prefixes (12 paths) (of requested type)
vyos@leaf1-pod1:~$ show bgp l2vpn evpn route type 3 | grep Dis
Route Distinguisher: 10.255.255.5:3
Route Distinguisher: 10.255.255.6:3
Route Distinguisher: 10.255.255.7:2
Route Distinguisher: 10.255.255.8:2
Route Distinguisher: 10.255.255.11:100
Route Distinguisher: 10.255.255.12:100

# we notice that RDs for VyOS nodes don't respect the logic of <Loopback:vni>. It s because for the moment we can't set it by vni (or I didn't find how to do that)

##############
# Arista
##############

# we check the fdb. Remote host are behind the port Vx1

leaf1-pod3#show mac address-table vlan 100
Mac Address Table
------------------------------------------------------------------

Vlan Mac Address Type Ports Moves Last Move
---- ----------- ---- ----- ----- ---------
100 0050.7966.680c DYNAMIC Vx1 1 0:00:38 ago
100 0050.7966.6813 DYNAMIC Et3 1 0:14:08 ago
100 0050.7966.6815 DYNAMIC Vx1 1 9 days, 2:36:39 ago
100 0050.7966.6817 DYNAMIC Vx1 1 0:00:36 ago
100 0050.7966.6819 DYNAMIC Vx1 1 0:00:35 ago
Total Mac Addresses for this criterion: 5

# we check that all VTEP are ready for BUM

leaf1-pod3#show bgp evpn route-type imet | i >
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
* >Ec RD: 10.255.255.5:3 imet 10.255.255.5
* >Ec RD: 10.255.255.6:3 imet 10.255.255.6
* >Ec RD: 10.255.255.7:2 imet 10.255.255.7
* >Ec RD: 10.255.255.8:2 imet 10.255.255.8
* > RD: 10.255.255.11:100 imet 10.255.255.11
* >Ec RD: 10.255.255.12:100 imet 10.255.255.12
[...]

leaf1-pod3#show vxlan vtep detail
Remote VTEPS for Vxlan1:

VTEP Learned Via MAC Address Learning Tunnel Type(s)
------------------ ------------------ -------------------------- --------------
10.255.255.5 control plane control plane flood, unicast
10.255.255.6 control plane control plane flood, unicast
10.255.255.7 control plane control plane flood, unicast
10.255.255.8 control plane control plane flood, unicast
10.255.255.12 control plane control plane flood, unicast

##############
# Cumulus
##############

# we check the fdb. Remote host are behind the port Vx1

cumulus@cumulus:mgmt:~$ net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ ----------------
100 bridge bridge 50:01:00:11:00:03 permanent 9 days, 02:47:42
100 bridge swp3 00:50:79:66:68:15 00:00:06
100 bridge vni100 00:50:79:66:68:0c extern_learn 00:00:40
100 bridge vni100 00:50:79:66:68:13 extern_learn 00:21:01
100 bridge vni100 00:50:79:66:68:17 extern_learn 00:00:38
100 bridge vni100 00:50:79:66:68:19 extern_learn 00:00:36

# --> here we see that it misses the @mac from the host on leaf1 pod1.
# we check the evpn routes on the Cumulus node and we see that from leaf1 pod1 vni 100 we have only those routes

Route Distinguisher: 10.255.255.5:3
* [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
* [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
*> [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8

# --> something is wrong on the remote leaf.
# on the leaf we check the policy for the EVPN routes export

route-map RM-OUT-VNI {
rule 1 {
action permit
match {
evpn {
route-type multicast
vni 100
}
}
set {
extcommunity {
rt 65000:100
}
}
}
rule 2 {
action permit
match {
evpn {
route-type prefix
vni 100
}
}
set {
extcommunity {
rt 65000:100
}
}
}
rule 3 {
action permit
match {
evpn {
vni 200
}
}
set {
extcommunity {
rt 65000:200
}

# we notice that a typo in rule numbering removed the export of macip routes for vni 100.
# we correct that :

route-map RM-OUT-VNI {
rule 1 {
action permit
match {
evpn {
route-type multicast
vni 100
}
}
set {
extcommunity {
rt 65000:100
}
}
}
rule 2 {
action permit
match {
evpn {
route-type macip
vni 100
}
}
set {
extcommunity {
rt 65000:100
}
}
}
rule 3 {
action permit
match {
evpn {
route-type prefix
vni 200
}
}
set {
extcommunity {
rt 65000:200
}

# once corrected we check the fdb on the cumulus node + evpn routes

cumulus@cumulus:mgmt:~$ net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ ----------------
100 bridge bridge 50:01:00:11:00:03 permanent 9 days, 02:51:48
100 bridge swp3 00:50:79:66:68:15 00:02:01
100 bridge vni100 00:50:79:66:68:0b extern_learn 00:00:26
100 bridge vni100 00:50:79:66:68:0c extern_learn 00:04:46
100 bridge vni100 00:50:79:66:68:13 extern_learn 00:25:08
100 bridge vni100 00:50:79:66:68:17 extern_learn 00:04:45
100 bridge vni100 00:50:79:66:68:19 extern_learn 00:04:43

Route Distinguisher: 10.255.255.5:3
* [2]:[0]:[48]:[00:50:79:66:68:0b]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
* [2]:[0]:[48]:[00:50:79:66:68:0b]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
*> [2]:[0]:[48]:[00:50:79:66:68:0b]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
* [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
* [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
*> [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8

# we have now Type 2 routes, all is ok now and we learn the client mac 00:50:79:66:68:0b
# Nevertheless, the trafic was working between the client hosts, but trafic from Cumulus leaf client to VyOS leaf client was broadcasted to all leaf since the @mac was not advertised on the fabric

</syntaxhighlight>It could be interesting to check how each node is configured to deliver the service, but the article would be too long. So, to sum the steps to provide a L2 service:

# create a VLAN (on Arista) / bridge (VyOS and Cumulus) + associate it to the VNI/VXLAN instances/interface
# create the mac vrf + define the route-target into bgp (Arista + Cumulus) / update the route-maps + community list to allow the redistribution of the VNI routes (type 2 and type 3)

==== L3 service implementation ====
Now, we'll now look at the L3 service. On that kind of setup we need to implement / check:
* leaf IP subnets are well redistributed betweeb all leaf (here we are in a vrf "any to any" topology)
* we only redistribute routes for IP prefix. Routes type 2 (mac @) don't need to be redistributed between leaf
* that all clients can ping each other through the L3VPN

To achieve that, we need to implement analog configuration than for the L2 service, but with the following differences:

* create a VLAN (on Arista) / bridge (VyOS and Cumulus) + associate it to the VNI/VXLAN instances|interface
* create the L3 VRF + define the route-target into bgp (Arista + Cumulus + VyOS) + allow connected routes distribution (Arista / Cumulus) / update the route-maps + community list to allow the redistribution of the VNI routes (type 5) for VyOS / restrict the routes redistribution to type 5 routes into the VNI (Cumulus)

Now we check the routing table with common cli commands + into the EVPN RIB :

<syntaxhighlight lang="bash">
# check routes into the vrf

vyos@leaf1-pod1:~$ show ip route vrf VRF-L3-1
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF VRF-L3-1:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 02w6d22h
C>* 192.168.10.0/24 is directly connected, br200, 02w6d22h
B>* 192.168.20.0/24 [20/0] via 10.255.255.6, br200 onlink, weight 1, 05:11:08
B>* 192.168.30.0/24 [20/0] via 10.255.255.7, br200 onlink, weight 1, 05:04:15
B>* 192.168.40.0/24 [20/0] via 10.255.255.8, br200 onlink, weight 1, 04:59:41
B>* 192.168.50.0/24 [20/0] via 10.255.255.11, br200 onlink, weight 1, 05:11:08
B>* 192.168.60.0/24 [20/0] via 10.255.255.12, br200 onlink, weight 1, 05:11:08

leaf1-pod3#show ip route vrf VRF-L3-1

VRF: VRF-L3-1
Codes: C - connected, S - static, K - kernel,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,
R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
NG - Nexthop Group Static Route, V - VXLAN Control Service,
DH - DHCP client installed default route, M - Martian,
DP - Dynamic Policy Route, L - VRF Leaked,
G - gRIBI, RC - Route Cache Route

Gateway of last resort is not set

B E 192.168.10.0/24 [200/0] via VTEP 10.255.255.5 VNI 200 router-mac 12:9b:9b:d8:f6:86 local-interface Vxlan1
B E 192.168.20.0/24 [200/0] via VTEP 10.255.255.6 VNI 200 router-mac 50:01:00:02:00:03 local-interface Vxlan1
B E 192.168.30.0/24 [200/0] via VTEP 10.255.255.7 VNI 200 router-mac 50:01:00:09:00:03 local-interface Vxlan1
B E 192.168.40.0/24 [200/0] via VTEP 10.255.255.8 VNI 200 router-mac 50:01:00:0a:00:03 local-interface Vxlan1
C 192.168.50.0/24 is directly connected, Vlan200
B E 192.168.60.0/24 [200/0] via VTEP 10.255.255.12 VNI 200 router-mac 50:01:00:11:00:03 local-interface Vxlan1

cumulus@cumulus:mgmt:~$ net show route vrf VRF-L3-1
show ip route vrf VRF-L3-1
===========================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued route, r - rejected route

VRF VRF-L3-1:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 01w2d03h
B>* 192.168.10.0/24 [20/0] via 10.255.255.5, vlan200 onlink, weight 1, 03:18:37
B>* 192.168.20.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 03:18:37
B>* 192.168.30.0/24 [20/0] via 10.255.255.7, vlan200 onlink, weight 1, 03:18:37
B>* 192.168.40.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:18:37
B>* 192.168.50.0/24 [20/0] via 10.255.255.11, vlan200 onlink, weight 1, 03:18:37
C>* 192.168.60.0/24 is directly connected, vlan200, 01w2d03h

# we can notice that the output are quite similar, except on Arista which indicates the VNI for remote targets, instead of referencing the local bridge / vlan200

# Now we check the routes for the VNI 200

# VyOS

vyos@leaf1-pod1:~$ show bgp l2vpn evpn route type prefix
BGP table version is 1, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:200
*> [5]:[0]:[24]:[192.168.10.0]
10.255.255.5 0 32768 ?
ET:8 RT:65000:200 Rmac:12:9b:9b:d8:f6:86
Route Distinguisher: 10.255.255.6:200
*> [5]:[0]:[24]:[192.168.20.0]
10.255.255.6 0 65001 65012 ?
RT:65000:200 ET:8 Rmac:50:01:00:02:00:03
Route Distinguisher: 10.255.255.7:200
* [5]:[0]:[24]:[192.168.30.0]
10.255.255.7 0 65001 65003 65002 65013 ?
RT:65000:200 RT:65013:200 ET:8 Rmac:50:01:00:09:00:03
*> [5]:[0]:[24]:[192.168.30.0]
10.255.255.7 0 65001 65003 65002 65013 ?
RT:65000:200 RT:65013:200 ET:8 Rmac:50:01:00:09:00:03
Route Distinguisher: 10.255.255.8:200
*> [5]:[0]:[24]:[192.168.40.0]
10.255.255.8 0 65001 65003 65002 65014 ?
RT:65000:200 RT:65014:200 ET:8 Rmac:50:01:00:0a:00:03
* [5]:[0]:[24]:[192.168.40.0]
10.255.255.8 0 65001 65003 65002 65014 ?
RT:65000:200 RT:65014:200 ET:8 Rmac:50:01:00:0a:00:03
Route Distinguisher: 10.255.255.11:200
* [5]:[0]:[24]:[192.168.50.0]
10.255.255.11 0 65001 65003 65004 65015 i
RT:65000:200 ET:8 Rmac:50:01:00:cd:ff:68
*> [5]:[0]:[24]:[192.168.50.0]
10.255.255.11 0 65001 65003 65004 65015 i
RT:65000:200 ET:8 Rmac:50:01:00:cd:ff:68
Route Distinguisher: 10.255.255.12:10200
* [5]:[0]:[24]:[192.168.60.0]
10.255.255.12 0 65001 65003 65004 65016 i
RT:65000:200 ET:8 Rmac:50:01:00:11:00:03
*> [5]:[0]:[24]:[192.168.60.0]
10.255.255.12 0 65001 65003 65004 65016 i
RT:65000:200 ET:8 Rmac:50:01:00:11:00:03

# Arista

leaf1-pod3#show bgp evpn route ip-prefix ipv4
BGP routing table information for VRF default
Router identifier 10.255.255.11, local AS number 65015
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric LocPref Weight Path
* > RD: 10.255.255.5:200 ip-prefix 192.168.10.0/24
10.255.255.5 - 100 0 65004 65003 65001 65011 ?
* RD: 10.255.255.5:200 ip-prefix 192.168.10.0/24
10.255.255.5 - 100 0 65004 65003 65001 65011 ?
* > RD: 10.255.255.6:200 ip-prefix 192.168.20.0/24
10.255.255.6 - 100 0 65004 65003 65001 65012 ?
* RD: 10.255.255.6:200 ip-prefix 192.168.20.0/24
10.255.255.6 - 100 0 65004 65003 65001 65012 ?
* > RD: 10.255.255.7:200 ip-prefix 192.168.30.0/24
10.255.255.7 - 100 0 65004 65003 65002 65013 ?
* RD: 10.255.255.7:200 ip-prefix 192.168.30.0/24
10.255.255.7 - 100 0 65004 65003 65002 65013 ?
* > RD: 10.255.255.8:200 ip-prefix 192.168.40.0/24
10.255.255.8 - 100 0 65004 65003 65002 65014 ?
* RD: 10.255.255.8:200 ip-prefix 192.168.40.0/24
10.255.255.8 - 100 0 65004 65003 65002 65014 ?
* > RD: 10.255.255.11:200 ip-prefix 192.168.50.0/24
- - - 0 i
* > RD: 10.255.255.12:10200 ip-prefix 192.168.60.0/24
10.255.255.12 - 100 0 65004 65016 i
* RD: 10.255.255.12:10200 ip-prefix 192.168.60.0/24
10.255.255.12 - 100 0 65004 65016 i

# CVX

cumulus@cumulus:mgmt:~$ net show bgp l2vpn evpn route type prefix
BGP table version is 7, local router ID is 10.255.255.12
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:200
* [5]:[0]:[24]:[192.168.10.0]
10.255.255.5 0 65004 65003 65001 65011 ?
RT:65000:200 ET:8 Rmac:12:9b:9b:d8:f6:86
*> [5]:[0]:[24]:[192.168.10.0]
10.255.255.5 0 65004 65003 65001 65011 ?
RT:65000:200 ET:8 Rmac:12:9b:9b:d8:f6:86
Route Distinguisher: 10.255.255.6:200
* [5]:[0]:[24]:[192.168.20.0]
10.255.255.6 0 65004 65003 65001 65012 ?
RT:65000:200 ET:8 Rmac:50:01:00:02:00:03
*> [5]:[0]:[24]:[192.168.20.0]
10.255.255.6 0 65004 65003 65001 65012 ?
RT:65000:200 ET:8 Rmac:50:01:00:02:00:03
Route Distinguisher: 10.255.255.7:200
* [5]:[0]:[24]:[192.168.30.0]
10.255.255.7 0 65004 65003 65002 65013 ?
RT:65000:200 RT:65013:200 ET:8 Rmac:50:01:00:09:00:03
*> [5]:[0]:[24]:[192.168.30.0]
10.255.255.7 0 65004 65003 65002 65013 ?
RT:65000:200 RT:65013:200 ET:8 Rmac:50:01:00:09:00:03
Route Distinguisher: 10.255.255.8:200
* [5]:[0]:[24]:[192.168.40.0]
10.255.255.8 0 65004 65003 65002 65014 ?
RT:65000:200 RT:65014:200 ET:8 Rmac:50:01:00:0a:00:03
*> [5]:[0]:[24]:[192.168.40.0]
10.255.255.8 0 65004 65003 65002 65014 ?
RT:65000:200 RT:65014:200 ET:8 Rmac:50:01:00:0a:00:03
Route Distinguisher: 10.255.255.11:200
* [5]:[0]:[24]:[192.168.50.0]
10.255.255.11 0 65004 65015 i
RT:65000:200 ET:8 Rmac:50:01:00:cd:ff:68
*> [5]:[0]:[24]:[192.168.50.0]
10.255.255.11 0 65004 65015 i
RT:65000:200 ET:8 Rmac:50:01:00:cd:ff:68
Route Distinguisher: 10.255.255.12:10200
*> [5]:[0]:[24]:[192.168.60.0]
10.255.255.12 0 32768 i
ET:8 RT:65000:200 Rmac:50:01:00:11:00:03

# Now we check with traceroute on the 3 clients connected on focused leaf switches

# client leaf VyOS

VPCS> ping 192.168.50.2

84 bytes from 192.168.50.2 icmp_seq=1 ttl=62 time=16.456 ms
84 bytes from 192.168.50.2 icmp_seq=2 ttl=62 time=11.852 ms
^C
VPCS> ping 192.168.60.2

84 bytes from 192.168.60.2 icmp_seq=1 ttl=62 time=5.554 ms
84 bytes from 192.168.60.2 icmp_seq=2 ttl=62 time=4.432 ms
^C
VPCS> trace 192.168.50.2
trace to 192.168.50.2, 8 hops max, press Ctrl+C to stop
1 192.168.10.1 0.499 ms 3.143 ms 1.432 ms
2 192.168.50.1 10.465 ms 12.512 ms 8.992 ms
3 *192.168.50.2 11.056 ms (ICMP type:3, code:3, Destination port unreachable)

VPCS> trace 192.168.60.2
trace to 192.168.60.2, 8 hops max, press Ctrl+C to stop
1 192.168.10.1 0.525 ms 0.312 ms 0.278 ms
2 192.168.60.1 4.949 ms 4.232 ms 4.022 ms
3 *192.168.60.2 4.167 ms (ICMP type:3, code:3, Destination port unreachable)

# client leaf Arista

VPCS> ping 192.168.10.2

84 bytes from 192.168.10.2 icmp_seq=1 ttl=62 time=10.914 ms
^C
VPCS> ping 192.168.60.2

84 bytes from 192.168.60.2 icmp_seq=1 ttl=62 time=11.755 ms
^C
VPCS> trace 192.168.10.2
trace to 192.168.10.2, 8 hops max, press Ctrl+C to stop
1 192.168.50.1 4.742 ms 4.123 ms 4.293 ms
2 192.168.10.1 12.550 ms 15.039 ms 9.411 ms
3 *192.168.10.2 8.939 ms (ICMP type:3, code:3, Destination port unreachable)

VPCS>
VPCS> trace 192.168.60.2
trace to 192.168.60.2, 8 hops max, press Ctrl+C to stop
1 192.168.50.1 3.073 ms 2.901 ms 2.585 ms
2 192.168.60.1 10.815 ms 11.493 ms 12.144 ms
3 *192.168.60.2 10.429 ms (ICMP type:3, code:3, Destination port unreachable)

# client leaf Cumulus

VPCS> ping 192.168.10.2

84 bytes from 192.168.10.2 icmp_seq=1 ttl=62 time=4.731 ms
^C
VPCS> ping 192.168.50.2

84 bytes from 192.168.50.2 icmp_seq=1 ttl=62 time=11.722 ms
^C
VPCS> trace 192.168.50.2
trace to 192.168.50.2, 8 hops max, press Ctrl+C to stop
1 192.168.60.1 0.487 ms 0.266 ms 0.248 ms
2 192.168.50.1 6.691 ms 6.281 ms 6.613 ms
3 *192.168.50.2 9.355 ms (ICMP type:3, code:3, Destination port unreachable)

VPCS>
VPCS> trace 192.168.10.2
trace to 192.168.10.2, 8 hops max, press Ctrl+C to stop
1 192.168.60.1 0.436 ms 0.315 ms 0.380 ms
2 192.168.10.1 4.535 ms 4.661 ms 3.994 ms
3 *192.168.10.2 4.292 ms (ICMP type:3, code:3, Destination port unreachable)

</syntaxhighlight>

=== Conclusion of the lab ===
These first results are satisfying, it comforts the idea that VXLAN is a strong option to deliver L2/L3 services in datacenters, but not only. This kind of architecture is more and more chosen in campus designs, PRA, NFV solutions.

The support of this technology on more and more open source NOS are encouraging regarding the innovation. If we look at Vyatta project for example, whom OS can be compiled "brick per brick", we could imagine to create NFV CPEs providing VXLAN fabric extensions of some client VRFs. CPE could be connected to the central Fabric within encrypted tunnels (wireguard,ipsec) or through direct link over MacSec (this feature in on the Vyatta Roadmap).

Please contact us if you want to feed the topic / discuss the lab.

Cheers

Pierre L.

File:02.lab-vyos-eos-cvx v2.zip

2022-04-05T12:34:47Z

PierreLancastre:

File:Nesevo-Lab-Vyatta-Arista-CVX-l3vpn.png

2022-04-05T12:33:31Z

PierreLancastre:

File:Nesevo-Lab-Vyatta-Arista-CVX-l2vpn.png

2022-04-05T12:33:14Z

PierreLancastre:

File:2022-02 eveng-lab-vyatta-cum-arista.png

2022-04-05T12:32:03Z

PierreLancastre:

File:Nesevo-Lab-Vyatta-Arista-CVX.png

2022-04-05T12:31:36Z

PierreLancastre:

VyosAristaCVX-VXLAN-lab

2022-04-05T12:28:14Z

PierreLancastre:

=== Introduction ===
More and more, EVPN is making its place in Datacenter field. Mainstream vendors integrate its support, with VXLAN or MPLS for data plane + BGP for control plane. In previous labs, we were able to observe that interoperability was quite good between vendors. Nevertheless, even we've already tested CumulusVX, there are also other open sources NOS which can support EVPN.

Today we'll test VyOS in it's latest rolling version, which handles the support of EVPN.

The targets of this lab are to :

* create a VXLAN Fabric with VyOS nodes
* provide a L2VPN and a L3VPN to client nodes within the fabric
* integrate some other vendors node into the fabric, here Arista and Cumulus VX
* test that all works as expected

=== Lab setup ===
To reach those goals, we'll use the EVE-NG tool in community edition (https://www.eve-ng.net/index.php/download/#DL-COMM) and use the following NOS version :

* Vyatta VyOS 1.4 rolling <-- free to download https://vyos.net/get/nightly-builds/
* Arista vEOS 4.26.0F lab edition <-- need a support account
* Nvidia Cumulus VX 4.2.1 lab edition <-- free to download https://www.nvidia.com/en-us/networking/ethernet-switching/cumulus-vx/download/

We'll build a 3-tiers fabric, with leaf/spine/superspine devices, VXLAN/EVPN running on all nodes. It gives the following setup:

[[Datei:Nesevo-Lab-Vyatta-Arista-CVX.png|rahmenlos|600x600px]]

On EVE-NG, we add IP information, ASN, loopback IPs. All other information will be past after when dealing with technical details.

So, it gives:

[[Datei:2022-02_eveng-lab-vyatta-cum-arista.png|rahmenlos|600x600px]]

==== <span class="mw-headline">VXLAN/BGP infos</span> ====
Note: in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>
{| class="wikitable"
|+
!service
!vlan id
!route-target
!VNI
!Commentaires
|-
|L2 cross-DC
|100
|target:65000:100
|100
|the 6 clients are in the same broadcast domain / subnet
|-
|L3VPN cros-DC
|200
|target:65000:200
|200
|the 6 clients have their own subnets. Here we use only one vni because no ESI will be implemented. One leaf one subnet.
|}
Client devices are represented by computers on EVE-NG. IPs in green are using the L2VPN service (extended layer 2 between pods) and IPs in orange the L3VPN service (routing service).

Below some diagrams showing the logic topologies :

[[Datei:Nesevo-Lab-Vyatta-Arista-CVX-l2vpn.png|rahmenlos|400x400px]]

[[Datei:Nesevo-Lab-Vyatta-Arista-CVX-l3vpn.png|rahmenlos|400x400px]]

Below the link to the configurations to allow you to quickly reproduce the tab.

To be noted that the configurations are not "ready-to-prod " and need to be optimized (adding bfd, bgp optimization, etc.). Some typos can subsist. Whatever, the lab works and gives the expected results :)

Link to devices configurations : http://wiki.intra.nesevo.com/images/3/32/02.lab-vyos-eos-cvx_v2.zip

=== Lab details ===
Considering that the lab is up and running, we'll now come back to the architecture.

We have 3 kinds of network nodes :

* leaf switches, on edge, which provide L2 and L3 services.
* couple of spine switches which allow leaf from a same pod to exchange traffic. They are also connected to superspine to reach other pods
* superspine switches which interconnext the different spine switches from the different pods, thus allowing interpods traffic

All those network devices control plane can exchange routes' information within 2 layers of bgp : underlay and overlay.

The role of the "underlay" is to dynamically route the traffic between loopback interfaces from the different nodes. Loopback interfaces are used to mount BGP sessions for the underlay and are used as VTEP interfaces on leaf nodes. Generally, it's better to have 2 different Loopback for this usage (in case of anycast vtep setup for example), but some vendors don't allow to 2 loopback interfaces in the same routing context. Whatever, in our setup, having only one loopback is fine.

Here, the underlay is done in eBGP, <u>ipv4 unicast family only + route-map to only redistribute loopback interfaces IP</u>.

* Each node mount a session to the directly connected peers.
* Each leaf node has a dedicated ASN.
* Each couple of spine/superspine has a ASN for two, to follow the best practices to optimize the routing table (ECMP).

The role of the "overlay" is to exchange EVPN routes type 2,3,5 here (type 1/4 routes seem to not be managed by VyOS at the moment, TBC).

* eBGP multihop sessions are mounted to directly neighbor loopback IPs.
* policies/options are set to not change the original nexthop when propagating routes
* for L2 services we only allow the redistribution of type 2 (macip) and type 3 (inclusive mcast for BUM management) routes
* for L3 services we only allow the redistribution of type 5 (prefix) routes
* regarding the setup, to enhance the scalability, it could be nice to use dedicated nodes (VM or phys) to take the role of route-servers. Here, we could connect them to superspine. Thus, spine/superspine would only do underlay and could not be a bottleneck for the control plane scalability

Moreover, regarding the lab setup, no bgp pic / bfd / VMTO / other option are implemented to optimize the convergence / traffic delivery. The goal was to verify that VyOS is compatible in a basic but functionnal VXLAN setup.

Now, we'll take a look on the different output from the 2 vendors leaf / spine / superspine to verify that all works as expected.

We verify that underlay works well. All is fine if:

* we have an eBGP session per directly connected peer
* we receive the same number of prefixes per peer type:
** a leef has to receive the same number of prefixes from its spine neighbors
** spine neigbors have to receive the same amount of prefixes from superspine, and reciprocally
** all VTEP IPs have to be known into the GRT (global routing table / default routing instance)

====== On VyOS ======
Below we can see that we receive 17 prefixes from both neighbors and that we know all loopback IPs from the setup (13 remote, 1 local). We have a bit too much routes, because on Cumulus nodes the filters that I have configured seem misconstructed.

To be noted that spine will announce all the Loopback IPs except the one from it's twin, since they are not directly connected and superspine respect as loop protection constraint (don't reflect routes originated by an ASN "X" to a peer from the same ASN "X").<syntaxhighlight lang="bash">
vyos@leaf1-pod1:~$ show ip bgp summary established

IPv4 Unicast Summary (VRF default):
BGP router identifier 10.255.255.5, local AS number 65011 vrf-id 0
BGP table version 20
RIB entries 37, using 6808 bytes of memory
Peers 2, using 1446 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
10.254.254.9 4 65001 29993 29988 0 0 0 02w6d19h 17 19 N/A
10.254.254.11 4 65001 17218 17220 0 0 0 01w4d22h 17 19 N/A

Displayed neighbors 2
Total number of neighbors 2

vyos@leaf1-pod1:~$ show ip route | match /32
B>* 10.255.255.1/32 [20/0] via 10.254.254.9, eth0, weight 1, 02w6d19h
B>* 10.255.255.2/32 [20/0] via 10.254.254.11, eth1, weight 1, 01w4d22h
B>* 10.255.255.3/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.4/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
C>* 10.255.255.5/32 is directly connected, dum0, 02w6d19h
B>* 10.255.255.6/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d22h
B>* 10.255.255.7/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.8/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.9/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.10/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.11/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w2d01h
B>* 10.255.255.12/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w2d00h
B>* 10.255.255.13/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w2d01h
B>* 10.255.255.14/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w2d00h

</syntaxhighlight>

====== On Arista ======
We verify that we receive prefix + same leaf / superspine / remote pods spine IPs (we note there misses some filters on spine since we receive /31 interco routes): <syntaxhighlight lang="bash">
leaf1-pod3#show ip bgp neighbors 10.254.254.43 received-routes
BGP routing table information for VRF default
Router identifier 10.255.255.11, local AS number 65015
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup, L - labeled-unicast
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI Origin Validation codes: V - valid, I - invalid, U - unknown
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric AIGP LocPref Weight Path
10.254.254.36/31 10.254.254.43 0 - - - 65004 ?
10.254.254.38/31 10.254.254.43 0 - - - 65004 ?
10.254.254.42/31 10.254.254.43 0 - - - 65004 ?
10.254.254.44/31 10.254.254.43 - - - - 65004 65016 ?
10.254.254.46/31 10.254.254.43 0 - - - 65004 ?
* 10.255.255.1/32 10.254.254.43 - - - - 65004 65003 65001 ?
* 10.255.255.2/32 10.254.254.43 - - - - 65004 65003 65001 ?
* 10.255.255.3/32 10.254.254.43 - - - - 65004 65003 65002 ?
* 10.255.255.4/32 10.254.254.43 - - - - 65004 65003 65002 ?
* 10.255.255.5/32 10.254.254.43 - - - - 65004 65003 65001 65011 ?
* 10.255.255.6/32 10.254.254.43 - - - - 65004 65003 65001 65012 ?
* 10.255.255.7/32 10.254.254.43 - - - - 65004 65003 65002 65013 ?
* 10.255.255.8/32 10.254.254.43 - - - - 65004 65003 65002 65014 ?
* 10.255.255.9/32 10.254.254.43 - - - - 65004 65003 ?
* 10.255.255.10/32 10.254.254.43 - - - - 65004 65003 ?
* 10.255.255.12/32 10.254.254.43 - - - - 65004 65016 ?
* > 10.255.255.14/32 10.254.254.43 0 - - - 65004 ?
leaf1-pod3#show ip bgp neighbors 10.254.254.41 received-routes
BGP routing table information for VRF default
Router identifier 10.255.255.11, local AS number 65015
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup, L - labeled-unicast
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI Origin Validation codes: V - valid, I - invalid, U - unknown
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric AIGP LocPref Weight Path
* > 10.255.255.1/32 10.254.254.41 - - - - 65004 65003 65001 ?
* > 10.255.255.2/32 10.254.254.41 - - - - 65004 65003 65001 ?
* > 10.255.255.3/32 10.254.254.41 - - - - 65004 65003 65002 ?
* > 10.255.255.4/32 10.254.254.41 - - - - 65004 65003 65002 ?
* > 10.255.255.5/32 10.254.254.41 - - - - 65004 65003 65001 65011 ?
* > 10.255.255.6/32 10.254.254.41 - - - - 65004 65003 65001 65012 ?
* > 10.255.255.7/32 10.254.254.41 - - - - 65004 65003 65002 65013 ?
* > 10.255.255.8/32 10.254.254.41 - - - - 65004 65003 65002 65014 ?
* > 10.255.255.9/32 10.254.254.41 - - - - 65004 65003 ?
* > 10.255.255.10/32 10.254.254.41 - - - - 65004 65003 ?
* > 10.255.255.12/32 10.254.254.41 - - - - 65004 65016 ?
* > 10.255.255.13/32 10.254.254.41 - - - - 65004 i

</syntaxhighlight>

====== On Cumulus ======
<syntaxhighlight lang="bash">
cumulus@cumulus:mgmt:~$ net show bgp summary
show bgp ipv4 unicast summary
=============================
BGP router identifier 10.255.255.12, local AS number 65016 vrf-id 0
BGP table version 103
RIB entries 37, using 7104 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.254.254.45 4 65004 306764 261331 0 0 0 01:39:02 12
spine1-pod3(10.254.254.47) 4 65004 261602 261664 0 0 0 01w2d02h 16

Total number of neighbors 2

[...]

cumulus@cumulus:mgmt:~$ net show route | grep /32
B>* 10.255.255.1/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.2/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.3/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.4/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.5/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.6/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.7/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.8/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.9/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.10/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.11/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
C>* 10.255.255.12/32 is directly connected, lo, 01w2d02h
B>* 10.255.255.13/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.14/32 [20/0] via 10.254.254.47, swp2, weight 1, 01w2d02h

</syntaxhighlight>We can also verify that routing is ok by pinging remote loopback IPs from local loopback + check eBGP EVPN session status: <syntaxhighlight lang="bash">
# ping example from VyOS leaf
vyos@leaf1-pod1:~$ ping 10.255.255.12 source-address 10.255.255.5
PING 10.255.255.12 (10.255.255.12) from 10.255.255.5 : 56(84) bytes of data.
64 bytes from 10.255.255.12: icmp_seq=1 ttl=61 time=2.60 ms
64 bytes from 10.255.255.12: icmp_seq=2 ttl=61 time=2.54 ms
64 bytes from 10.255.255.12: icmp_seq=3 ttl=61 time=2.44 ms

# check of EVPN sessions on the different vendors

# vyos

vyos@leaf1-pod1:~$ show bgp l2vpn evpn summary
BGP router identifier 10.255.255.5, local AS number 65011 vrf-id 0
BGP table version 0
RIB entries 31, using 5704 bytes of memory
Peers 2, using 1446 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
10.255.255.1 4 65001 30205 30208 0 0 0 02w6d21h 13 13 N/A
10.255.255.2 4 65001 17392 17429 0 0 0 01w5d00h 11 13 N/A

# Arista

leaf1-pod3#show bgp evpn summary
BGP summary information for VRF default
Router identifier 10.255.255.11, local AS number 65015
Neighbor Status Codes: m - Under maintenance
Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
OVL_spine1-pod3 10.255.255.13 4 65004 15608 15503 0 0 00:03:47 Estab 17 17
OVL_spine2-pod3 10.255.255.14 4 65004 262780 308558 0 0 9d02h Estab 13 13

# Cumulus

cumulus@cumulus:mgmt:~$ net show bgp l2vpn evpn summary
BGP router identifier 10.255.255.12, local AS number 65016 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.255.255.13 4 65004 307243 261693 0 0 0 00:04:25 14
spine1-pod3(10.255.255.14) 4 65004 262001 262025 0 0 0 01w2d02h 10

</syntaxhighlight>

On EVPN, we should receive quite the same number of prefixes from both spine, but looks like there differences of best path election + redistribution between vendors. Perhaps something to dig. The most important point is to verify that the FIB for the services are consistent.

==== L2 service implementation ====
On that topic, we'll firstly focus on the Layer 2 service. To be sure that all works as expected we have to check that:

* mac-address-table are consistent on the whole fabric. In traditionnal L2 setup, switch FIBs depend of the traffic passing through them and a mac @ can be see "everywhere" if it does a BUM on the network. In VXLAN, in a mac-vrf topology without filer, even a host doesn't send BUM, the mac address is propagated on the whole fabric (within BGP update). But, as it is on traditional L2 topology, you have to be sure that mac @ table aging time are consistent on all devices, with an aging time superior to ARP timer on hosts.
* routes type 2 are visible on each leaf nodes for a same vni, routes type 3 to reach remote VTEP have to be learned
* a common route-target for the VRF routes need to be used/set within policies. In effect, there are differences between vendors on route "tagging" (features to auto generate route-target, "hard coded" alogos, etc.). Here, on VyOS, we got to add adminstrative extended communities within route-map for the L2 VNI. We'll show it below
* all fabric connections (interconnections between network nodes) have the sufficient "IP MTU". Please keep in mind that VXLAN overhead has a size of 50 bytes. So, to be sure that Jumbo traffic for host is ok, configure it at least at 9050 Bytes. I would advise to push it at 9100 Bytes in case of VXLAN usage on client HOSTS for Virtualized setup on servers.

Check FDB on leaf switch + debug type 2 routes' propagation for leaf1 pod1<syntaxhighlight lang="bash">
##############
# VyOS
##############

vyos@leaf1-pod1:~$ show bridge br100 fdb
50:01:00:01:00:02 dev eth2 vlan 1 master br100 permanent
50:01:00:01:00:02 dev eth2 master br100 permanent
33:33:00:00:00:01 dev eth2 self permanent
33:33:00:00:00:02 dev eth2 self permanent
01:00:5e:00:00:01 dev eth2 self permanent
01:80:c2:00:00:0e dev eth2 self permanent
01:80:c2:00:00:03 dev eth2 self permanent
01:80:c2:00:00:00 dev eth2 self permanent
33:33:00:00:00:01 dev br100 self permanent
33:33:00:00:00:02 dev br100 self permanent
33:33:ff:01:00:02 dev br100 self permanent
01:00:5e:00:00:6a dev br100 self permanent
33:33:00:00:00:6a dev br100 self permanent
01:00:5e:00:00:01 dev br100 self permanent
33:33:ff:00:00:00 dev br100 self permanent
00:50:79:66:68:15 dev vxlan100 vlan 1 extern_learn master br100
00:50:79:66:68:15 dev vxlan100 extern_learn master br100
8a:d0:00:16:1c:17 dev vxlan100 vlan 1 master br100 permanent
8a:d0:00:16:1c:17 dev vxlan100 master br100 permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.6 self permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.11 self permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.12 self permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.7 self permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.8 self permanent
00:50:79:66:68:15 dev vxlan100 dst 10.255.255.12 self extern_learn

# and to focus on remote nodes

vyos@leaf1-pod1:~$ show bridge br100 fdb | grep "self extern_learn"
00:50:79:66:68:15 dev vxlan100 dst 10.255.255.12 self extern_learn
00:50:79:66:68:13 dev vxlan100 dst 10.255.255.11 self extern_learn
00:50:79:66:68:19 dev vxlan100 dst 10.255.255.8 self extern_learn
00:50:79:66:68:0c dev vxlan100 dst 10.255.255.6 self extern_learn
00:50:79:66:68:17 dev vxlan100 dst 10.255.255.7 self extern_learn

# or on local clients

vyos@leaf1-pod1:~$ show bridge br100 fdb | grep eth | grep -v perm
00:50:79:66:68:0b dev eth2 master br100

# Now if we want to focus on route type 3 learned

Displayed 7 prefixes (12 paths) (of requested type)
vyos@leaf1-pod1:~$ show bgp l2vpn evpn route type 3 | grep Dis
Route Distinguisher: 10.255.255.5:3
Route Distinguisher: 10.255.255.6:3
Route Distinguisher: 10.255.255.7:2
Route Distinguisher: 10.255.255.8:2
Route Distinguisher: 10.255.255.11:100
Route Distinguisher: 10.255.255.12:100

# we notice that RDs for VyOS nodes don't respect the logic of <Loopback:vni>. It s because for the moment we can't set it by vni (or I didn't find how to do that)

##############
# Arista
##############

# we check the fdb. Remote host are behind the port Vx1

leaf1-pod3#show mac address-table vlan 100
Mac Address Table
------------------------------------------------------------------

Vlan Mac Address Type Ports Moves Last Move
---- ----------- ---- ----- ----- ---------
100 0050.7966.680c DYNAMIC Vx1 1 0:00:38 ago
100 0050.7966.6813 DYNAMIC Et3 1 0:14:08 ago
100 0050.7966.6815 DYNAMIC Vx1 1 9 days, 2:36:39 ago
100 0050.7966.6817 DYNAMIC Vx1 1 0:00:36 ago
100 0050.7966.6819 DYNAMIC Vx1 1 0:00:35 ago
Total Mac Addresses for this criterion: 5

# we check that all VTEP are ready for BUM

leaf1-pod3#show bgp evpn route-type imet | i >
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
* >Ec RD: 10.255.255.5:3 imet 10.255.255.5
* >Ec RD: 10.255.255.6:3 imet 10.255.255.6
* >Ec RD: 10.255.255.7:2 imet 10.255.255.7
* >Ec RD: 10.255.255.8:2 imet 10.255.255.8
* > RD: 10.255.255.11:100 imet 10.255.255.11
* >Ec RD: 10.255.255.12:100 imet 10.255.255.12
[...]

leaf1-pod3#show vxlan vtep detail
Remote VTEPS for Vxlan1:

VTEP Learned Via MAC Address Learning Tunnel Type(s)
------------------ ------------------ -------------------------- --------------
10.255.255.5 control plane control plane flood, unicast
10.255.255.6 control plane control plane flood, unicast
10.255.255.7 control plane control plane flood, unicast
10.255.255.8 control plane control plane flood, unicast
10.255.255.12 control plane control plane flood, unicast

##############
# Cumulus
##############

# we check the fdb. Remote host are behind the port Vx1

cumulus@cumulus:mgmt:~$ net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ ----------------
100 bridge bridge 50:01:00:11:00:03 permanent 9 days, 02:47:42
100 bridge swp3 00:50:79:66:68:15 00:00:06
100 bridge vni100 00:50:79:66:68:0c extern_learn 00:00:40
100 bridge vni100 00:50:79:66:68:13 extern_learn 00:21:01
100 bridge vni100 00:50:79:66:68:17 extern_learn 00:00:38
100 bridge vni100 00:50:79:66:68:19 extern_learn 00:00:36

# --> here we see that it misses the @mac from the host on leaf1 pod1.
# we check the evpn routes on the Cumulus node and we see that from leaf1 pod1 vni 100 we have only those routes

Route Distinguisher: 10.255.255.5:3
* [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
* [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
*> [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8

# --> something is wrong on the remote leaf.
# on the leaf we check the policy for the EVPN routes export

route-map RM-OUT-VNI {
rule 1 {
action permit
match {
evpn {
route-type multicast
vni 100
}
}
set {
extcommunity {
rt 65000:100
}
}
}
rule 2 {
action permit
match {
evpn {
route-type prefix
vni 100
}
}
set {
extcommunity {
rt 65000:100
}
}
}
rule 3 {
action permit
match {
evpn {
vni 200
}
}
set {
extcommunity {
rt 65000:200
}

# we notice that a typo in rule numbering removed the export of macip routes for vni 100.
# we correct that :

route-map RM-OUT-VNI {
rule 1 {
action permit
match {
evpn {
route-type multicast
vni 100
}
}
set {
extcommunity {
rt 65000:100
}
}
}
rule 2 {
action permit
match {
evpn {
route-type macip
vni 100
}
}
set {
extcommunity {
rt 65000:100
}
}
}
rule 3 {
action permit
match {
evpn {
route-type prefix
vni 200
}
}
set {
extcommunity {
rt 65000:200
}

# once corrected we check the fdb on the cumulus node + evpn routes

cumulus@cumulus:mgmt:~$ net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ ----------------
100 bridge bridge 50:01:00:11:00:03 permanent 9 days, 02:51:48
100 bridge swp3 00:50:79:66:68:15 00:02:01
100 bridge vni100 00:50:79:66:68:0b extern_learn 00:00:26
100 bridge vni100 00:50:79:66:68:0c extern_learn 00:04:46
100 bridge vni100 00:50:79:66:68:13 extern_learn 00:25:08
100 bridge vni100 00:50:79:66:68:17 extern_learn 00:04:45
100 bridge vni100 00:50:79:66:68:19 extern_learn 00:04:43

Route Distinguisher: 10.255.255.5:3
* [2]:[0]:[48]:[00:50:79:66:68:0b]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
* [2]:[0]:[48]:[00:50:79:66:68:0b]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
*> [2]:[0]:[48]:[00:50:79:66:68:0b]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
* [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
* [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
*> [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8

# we have now Type 2 routes, all is ok now and we learn the client mac 00:50:79:66:68:0b
# Nevertheless, the trafic was working between the client hosts, but trafic from Cumulus leaf client to VyOS leaf client was broadcasted to all leaf since the @mac was not advertised on the fabric

</syntaxhighlight>It could be interesting to check how each node is configured to deliver the service, but the article would be too long. So, to sum the steps to provide a L2 service:

# create a VLAN (on Arista) / bridge (VyOS and Cumulus) + associate it to the VNI/VXLAN instances/interface
# create the mac vrf + define the route-target into bgp (Arista + Cumulus) / update the route-maps + community list to allow the redistribution of the VNI routes (type 2 and type 3)

==== L3 service implementation ====
Now, we'll now look at the L3 service. On that kind of setup we need to implement / check:
* leaf IP subnets are well redistributed betweeb all leaf (here we are in a vrf "any to any" topology)
* we only redistribute routes for IP prefix. Routes type 2 (mac @) don't need to be redistributed between leaf
* that all clients can ping each other through the L3VPN

To achieve that, we need to implement analog configuration than for the L2 service, but with the following differences:

* create a VLAN (on Arista) / bridge (VyOS and Cumulus) + associate it to the VNI/VXLAN instances|interface
* create the L3 VRF + define the route-target into bgp (Arista + Cumulus + VyOS) + allow connected routes distribution (Arista / Cumulus) / update the route-maps + community list to allow the redistribution of the VNI routes (type 5) for VyOS / restrict the routes redistribution to type 5 routes into the VNI (Cumulus)

Now we check the routing table with common cli commands + into the EVPN RIB :

<syntaxhighlight lang="bash">
# check routes into the vrf

vyos@leaf1-pod1:~$ show ip route vrf VRF-L3-1
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF VRF-L3-1:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 02w6d22h
C>* 192.168.10.0/24 is directly connected, br200, 02w6d22h
B>* 192.168.20.0/24 [20/0] via 10.255.255.6, br200 onlink, weight 1, 05:11:08
B>* 192.168.30.0/24 [20/0] via 10.255.255.7, br200 onlink, weight 1, 05:04:15
B>* 192.168.40.0/24 [20/0] via 10.255.255.8, br200 onlink, weight 1, 04:59:41
B>* 192.168.50.0/24 [20/0] via 10.255.255.11, br200 onlink, weight 1, 05:11:08
B>* 192.168.60.0/24 [20/0] via 10.255.255.12, br200 onlink, weight 1, 05:11:08

leaf1-pod3#show ip route vrf VRF-L3-1

VRF: VRF-L3-1
Codes: C - connected, S - static, K - kernel,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,
R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
NG - Nexthop Group Static Route, V - VXLAN Control Service,
DH - DHCP client installed default route, M - Martian,
DP - Dynamic Policy Route, L - VRF Leaked,
G - gRIBI, RC - Route Cache Route

Gateway of last resort is not set

B E 192.168.10.0/24 [200/0] via VTEP 10.255.255.5 VNI 200 router-mac 12:9b:9b:d8:f6:86 local-interface Vxlan1
B E 192.168.20.0/24 [200/0] via VTEP 10.255.255.6 VNI 200 router-mac 50:01:00:02:00:03 local-interface Vxlan1
B E 192.168.30.0/24 [200/0] via VTEP 10.255.255.7 VNI 200 router-mac 50:01:00:09:00:03 local-interface Vxlan1
B E 192.168.40.0/24 [200/0] via VTEP 10.255.255.8 VNI 200 router-mac 50:01:00:0a:00:03 local-interface Vxlan1
C 192.168.50.0/24 is directly connected, Vlan200
B E 192.168.60.0/24 [200/0] via VTEP 10.255.255.12 VNI 200 router-mac 50:01:00:11:00:03 local-interface Vxlan1

cumulus@cumulus:mgmt:~$ net show route vrf VRF-L3-1
show ip route vrf VRF-L3-1
===========================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued route, r - rejected route

VRF VRF-L3-1:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 01w2d03h
B>* 192.168.10.0/24 [20/0] via 10.255.255.5, vlan200 onlink, weight 1, 03:18:37
B>* 192.168.20.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 03:18:37
B>* 192.168.30.0/24 [20/0] via 10.255.255.7, vlan200 onlink, weight 1, 03:18:37
B>* 192.168.40.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:18:37
B>* 192.168.50.0/24 [20/0] via 10.255.255.11, vlan200 onlink, weight 1, 03:18:37
C>* 192.168.60.0/24 is directly connected, vlan200, 01w2d03h

# we can notice that the output are quite similar, except on Arista which indicates the VNI for remote targets, instead of referencing the local bridge / vlan200

# Now we check the routes for the VNI 200

# VyOS

vyos@leaf1-pod1:~$ show bgp l2vpn evpn route type prefix
BGP table version is 1, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:200
*> [5]:[0]:[24]:[192.168.10.0]
10.255.255.5 0 32768 ?
ET:8 RT:65000:200 Rmac:12:9b:9b:d8:f6:86
Route Distinguisher: 10.255.255.6:200
*> [5]:[0]:[24]:[192.168.20.0]
10.255.255.6 0 65001 65012 ?
RT:65000:200 ET:8 Rmac:50:01:00:02:00:03
Route Distinguisher: 10.255.255.7:200
* [5]:[0]:[24]:[192.168.30.0]
10.255.255.7 0 65001 65003 65002 65013 ?
RT:65000:200 RT:65013:200 ET:8 Rmac:50:01:00:09:00:03
*> [5]:[0]:[24]:[192.168.30.0]
10.255.255.7 0 65001 65003 65002 65013 ?
RT:65000:200 RT:65013:200 ET:8 Rmac:50:01:00:09:00:03
Route Distinguisher: 10.255.255.8:200
*> [5]:[0]:[24]:[192.168.40.0]
10.255.255.8 0 65001 65003 65002 65014 ?
RT:65000:200 RT:65014:200 ET:8 Rmac:50:01:00:0a:00:03
* [5]:[0]:[24]:[192.168.40.0]
10.255.255.8 0 65001 65003 65002 65014 ?
RT:65000:200 RT:65014:200 ET:8 Rmac:50:01:00:0a:00:03
Route Distinguisher: 10.255.255.11:200
* [5]:[0]:[24]:[192.168.50.0]
10.255.255.11 0 65001 65003 65004 65015 i
RT:65000:200 ET:8 Rmac:50:01:00:cd:ff:68
*> [5]:[0]:[24]:[192.168.50.0]
10.255.255.11 0 65001 65003 65004 65015 i
RT:65000:200 ET:8 Rmac:50:01:00:cd:ff:68
Route Distinguisher: 10.255.255.12:10200
* [5]:[0]:[24]:[192.168.60.0]
10.255.255.12 0 65001 65003 65004 65016 i
RT:65000:200 ET:8 Rmac:50:01:00:11:00:03
*> [5]:[0]:[24]:[192.168.60.0]
10.255.255.12 0 65001 65003 65004 65016 i
RT:65000:200 ET:8 Rmac:50:01:00:11:00:03

# Arista

leaf1-pod3#show bgp evpn route ip-prefix ipv4
BGP routing table information for VRF default
Router identifier 10.255.255.11, local AS number 65015
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric LocPref Weight Path
* > RD: 10.255.255.5:200 ip-prefix 192.168.10.0/24
10.255.255.5 - 100 0 65004 65003 65001 65011 ?
* RD: 10.255.255.5:200 ip-prefix 192.168.10.0/24
10.255.255.5 - 100 0 65004 65003 65001 65011 ?
* > RD: 10.255.255.6:200 ip-prefix 192.168.20.0/24
10.255.255.6 - 100 0 65004 65003 65001 65012 ?
* RD: 10.255.255.6:200 ip-prefix 192.168.20.0/24
10.255.255.6 - 100 0 65004 65003 65001 65012 ?
* > RD: 10.255.255.7:200 ip-prefix 192.168.30.0/24
10.255.255.7 - 100 0 65004 65003 65002 65013 ?
* RD: 10.255.255.7:200 ip-prefix 192.168.30.0/24
10.255.255.7 - 100 0 65004 65003 65002 65013 ?
* > RD: 10.255.255.8:200 ip-prefix 192.168.40.0/24
10.255.255.8 - 100 0 65004 65003 65002 65014 ?
* RD: 10.255.255.8:200 ip-prefix 192.168.40.0/24
10.255.255.8 - 100 0 65004 65003 65002 65014 ?
* > RD: 10.255.255.11:200 ip-prefix 192.168.50.0/24
- - - 0 i
* > RD: 10.255.255.12:10200 ip-prefix 192.168.60.0/24
10.255.255.12 - 100 0 65004 65016 i
* RD: 10.255.255.12:10200 ip-prefix 192.168.60.0/24
10.255.255.12 - 100 0 65004 65016 i

# CVX

cumulus@cumulus:mgmt:~$ net show bgp l2vpn evpn route type prefix
BGP table version is 7, local router ID is 10.255.255.12
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:200
* [5]:[0]:[24]:[192.168.10.0]
10.255.255.5 0 65004 65003 65001 65011 ?
RT:65000:200 ET:8 Rmac:12:9b:9b:d8:f6:86
*> [5]:[0]:[24]:[192.168.10.0]
10.255.255.5 0 65004 65003 65001 65011 ?
RT:65000:200 ET:8 Rmac:12:9b:9b:d8:f6:86
Route Distinguisher: 10.255.255.6:200
* [5]:[0]:[24]:[192.168.20.0]
10.255.255.6 0 65004 65003 65001 65012 ?
RT:65000:200 ET:8 Rmac:50:01:00:02:00:03
*> [5]:[0]:[24]:[192.168.20.0]
10.255.255.6 0 65004 65003 65001 65012 ?
RT:65000:200 ET:8 Rmac:50:01:00:02:00:03
Route Distinguisher: 10.255.255.7:200
* [5]:[0]:[24]:[192.168.30.0]
10.255.255.7 0 65004 65003 65002 65013 ?
RT:65000:200 RT:65013:200 ET:8 Rmac:50:01:00:09:00:03
*> [5]:[0]:[24]:[192.168.30.0]
10.255.255.7 0 65004 65003 65002 65013 ?
RT:65000:200 RT:65013:200 ET:8 Rmac:50:01:00:09:00:03
Route Distinguisher: 10.255.255.8:200
* [5]:[0]:[24]:[192.168.40.0]
10.255.255.8 0 65004 65003 65002 65014 ?
RT:65000:200 RT:65014:200 ET:8 Rmac:50:01:00:0a:00:03
*> [5]:[0]:[24]:[192.168.40.0]
10.255.255.8 0 65004 65003 65002 65014 ?
RT:65000:200 RT:65014:200 ET:8 Rmac:50:01:00:0a:00:03
Route Distinguisher: 10.255.255.11:200
* [5]:[0]:[24]:[192.168.50.0]
10.255.255.11 0 65004 65015 i
RT:65000:200 ET:8 Rmac:50:01:00:cd:ff:68
*> [5]:[0]:[24]:[192.168.50.0]
10.255.255.11 0 65004 65015 i
RT:65000:200 ET:8 Rmac:50:01:00:cd:ff:68
Route Distinguisher: 10.255.255.12:10200
*> [5]:[0]:[24]:[192.168.60.0]
10.255.255.12 0 32768 i
ET:8 RT:65000:200 Rmac:50:01:00:11:00:03

# Now we check with traceroute on the 3 clients connected on focused leaf switches

# client leaf VyOS

VPCS> ping 192.168.50.2

84 bytes from 192.168.50.2 icmp_seq=1 ttl=62 time=16.456 ms
84 bytes from 192.168.50.2 icmp_seq=2 ttl=62 time=11.852 ms
^C
VPCS> ping 192.168.60.2

84 bytes from 192.168.60.2 icmp_seq=1 ttl=62 time=5.554 ms
84 bytes from 192.168.60.2 icmp_seq=2 ttl=62 time=4.432 ms
^C
VPCS> trace 192.168.50.2
trace to 192.168.50.2, 8 hops max, press Ctrl+C to stop
1 192.168.10.1 0.499 ms 3.143 ms 1.432 ms
2 192.168.50.1 10.465 ms 12.512 ms 8.992 ms
3 *192.168.50.2 11.056 ms (ICMP type:3, code:3, Destination port unreachable)

VPCS> trace 192.168.60.2
trace to 192.168.60.2, 8 hops max, press Ctrl+C to stop
1 192.168.10.1 0.525 ms 0.312 ms 0.278 ms
2 192.168.60.1 4.949 ms 4.232 ms 4.022 ms
3 *192.168.60.2 4.167 ms (ICMP type:3, code:3, Destination port unreachable)

# client leaf Arista

VPCS> ping 192.168.10.2

84 bytes from 192.168.10.2 icmp_seq=1 ttl=62 time=10.914 ms
^C
VPCS> ping 192.168.60.2

84 bytes from 192.168.60.2 icmp_seq=1 ttl=62 time=11.755 ms
^C
VPCS> trace 192.168.10.2
trace to 192.168.10.2, 8 hops max, press Ctrl+C to stop
1 192.168.50.1 4.742 ms 4.123 ms 4.293 ms
2 192.168.10.1 12.550 ms 15.039 ms 9.411 ms
3 *192.168.10.2 8.939 ms (ICMP type:3, code:3, Destination port unreachable)

VPCS>
VPCS> trace 192.168.60.2
trace to 192.168.60.2, 8 hops max, press Ctrl+C to stop
1 192.168.50.1 3.073 ms 2.901 ms 2.585 ms
2 192.168.60.1 10.815 ms 11.493 ms 12.144 ms
3 *192.168.60.2 10.429 ms (ICMP type:3, code:3, Destination port unreachable)

# client leaf Cumulus

VPCS> ping 192.168.10.2

84 bytes from 192.168.10.2 icmp_seq=1 ttl=62 time=4.731 ms
^C
VPCS> ping 192.168.50.2

84 bytes from 192.168.50.2 icmp_seq=1 ttl=62 time=11.722 ms
^C
VPCS> trace 192.168.50.2
trace to 192.168.50.2, 8 hops max, press Ctrl+C to stop
1 192.168.60.1 0.487 ms 0.266 ms 0.248 ms
2 192.168.50.1 6.691 ms 6.281 ms 6.613 ms
3 *192.168.50.2 9.355 ms (ICMP type:3, code:3, Destination port unreachable)

VPCS>
VPCS> trace 192.168.10.2
trace to 192.168.10.2, 8 hops max, press Ctrl+C to stop
1 192.168.60.1 0.436 ms 0.315 ms 0.380 ms
2 192.168.10.1 4.535 ms 4.661 ms 3.994 ms
3 *192.168.10.2 4.292 ms (ICMP type:3, code:3, Destination port unreachable)

</syntaxhighlight>

=== Conclusion of the lab ===
These first results are satisfying, it comforts the idea that VXLAN is a strong option to deliver L2/L3 services in datacenters, but not only. This kind of architecture is more and more chosen in campus designs, PRA, NFV solutions.

The support of this technology on more and more open source NOS are encouraging regarding the innovation. If we look at Vyatta project for example, whom OS can be compiled "brick per brick", we could imagine to create NFV CPEs providing VXLAN fabric extensions of some client VRFs. CPE could be connected to the central Fabric within encrypted tunnels (wireguard,ipsec) or through direct link over MacSec (this feature in on the Vyatta Roadmap).

Please contact us if you want to feed the topic / discuss the lab.

Cheers

Pierre L.

VyosAristaCVX-VXLAN-lab

2022-04-05T12:26:02Z

PierreLancastre: Created page with "Introduction More and more, EVPN is making its place in Datacenter field. Mainstream vendors integrate its support, with VXLAN or MPLS for data plane + BGP for control plane...."

Introduction

More and more, EVPN is making its place in Datacenter field. Mainstream vendors integrate its support, with VXLAN or MPLS for data plane + BGP for control plane. In previous labs, we were able to observe that interoperability was quite good between vendors. Nevertheless, even we've already tested CumulusVX, there are also other open sources NOS which can support EVPN.

Today we'll test VyOS in it's latest rolling version, which handles the support of EVPN.

The targets of this lab are to :

create a VXLAN Fabric with VyOS nodes
provide a L2VPN and a L3VPN to client nodes within the fabric
integrate some other vendors node into the fabric, here Arista and Cumulus VX
test that all works as expected
Lab setup

To reach those goals, we'll use the EVE-NG tool in community edition (https://www.eve-ng.net/index.php/download/#DL-COMM) and use the following NOS version :

Vyatta VyOS 1.4 rolling <-- free to download https://vyos.net/get/nightly-builds/
Arista vEOS 4.26.0F lab edition <-- need a support account
Nvidia Cumulus VX 4.2.1 lab edition <-- free to download https://www.nvidia.com/en-us/networking/ethernet-switching/cumulus-vx/download/

We'll build a 3-tiers fabric, with leaf/spine/superspine devices, VXLAN/EVPN running on all nodes. It gives the following setup:

On EVE-NG, we add IP information, ASN, loopback IPs. All other information will be past after when dealing with technical details.

So, it gives:

VXLAN/BGP infos

Note: in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>

service vlan id route-target VNI Commentaires
L2 cross-DC 100 target:65000:100 100 the 6 clients are in the same broadcast domain / subnet
L3VPN cros-DC 200 target:65000:200 200 the 6 clients have their own subnets. Here we use only one vni because no ESI will be implemented. One leaf one subnet.

Client devices are represented by computers on EVE-NG. IPs in green are using the L2VPN service (extended layer 2 between pods) and IPs in orange the L3VPN service (routing service).

Below some diagrams showing the logic topologies :

Below the link to the configurations to allow you to quickly reproduce the tab.

To be noted that the configurations are not "ready-to-prod " and need to be optimized (adding bfd, bgp optimization, etc.). Some typos can subsist. Whatever, the lab works and gives the expected results :)

Link to devices configurations : http://wiki.intra.nesevo.com/images/3/32/02.lab-vyos-eos-cvx_v2.zip

Lab details

Considering that the lab is up and running, we'll now come back to the architecture.

We have 3 kinds of network nodes :

leaf switches, on edge, which provide L2 and L3 services.
couple of spine switches which allow leaf from a same pod to exchange traffic. They are also connected to superspine to reach other pods
superspine switches which interconnext the different spine switches from the different pods, thus allowing interpods traffic

All those network devices control plane can exchange routes' information within 2 layers of bgp : underlay and overlay.

The role of the "underlay" is to dynamically route the traffic between loopback interfaces from the different nodes. Loopback interfaces are used to mount BGP sessions for the underlay and are used as VTEP interfaces on leaf nodes. Generally, it's better to have 2 different Loopback for this usage (in case of anycast vtep setup for example), but some vendors don't allow to 2 loopback interfaces in the same routing context. Whatever, in our setup, having only one loopback is fine.

Here, the underlay is done in eBGP, ipv4 unicast family only + route-map to only redistribute loopback interfaces IP.

Each node mount a session to the directly connected peers.
Each leaf node has a dedicated ASN.
Each couple of spine/superspine has a ASN for two, to follow the best practices to optimize the routing table (ECMP).

The role of the "overlay" is to exchange EVPN routes type 2,3,5 here (type 1/4 routes seem to not be managed by VyOS at the moment, TBC).

eBGP multihop sessions are mounted to directly neighbor loopback IPs.
policies/options are set to not change the original nexthop when propagating routes
for L2 services we only allow the redistribution of type 2 (macip) and type 3 (inclusive mcast for BUM management) routes
for L3 services we only allow the redistribution of type 5 (prefix) routes
regarding the setup, to enhance the scalability, it could be nice to use dedicated nodes (VM or phys) to take the role of route-servers. Here, we could connect them to superspine. Thus, spine/superspine would only do underlay and could not be a bottleneck for the control plane scalability

Moreover, regarding the lab setup, no bgp pic / bfd / VMTO / other option are implemented to optimize the convergence / traffic delivery. The goal was to verify that VyOS is compatible in a basic but functionnal VXLAN setup.

Now, we'll take a look on the different output from the 2 vendors leaf / spine / superspine to verify that all works as expected.

We verify that underlay works well. All is fine if:

we have an eBGP session per directly connected peer
we receive the same number of prefixes per peer type:
a leef has to receive the same number of prefixes from its spine neighbors
spine neigbors have to receive the same amount of prefixes from superspine, and reciprocally
all VTEP IPs have to be known into the GRT (global routing table / default routing instance)
On VyOS

Below we can see that we receive 17 prefixes from both neighbors and that we know all loopback IPs from the setup (13 remote, 1 local). We have a bit too much routes, because on Cumulus nodes the filters that I have configured seem misconstructed.

To be noted that spine will announce all the Loopback IPs except the one from it's twin, since they are not directly connected and superspine respect as loop protection constraint (don't reflect routes originated by an ASN "X" to a peer from the same ASN "X").

vyos@leaf1-pod1:~$ show ip bgp summary established

IPv4 Unicast Summary (VRF default):
BGP router identifier 10.255.255.5, local AS number 65011 vrf-id 0
BGP table version 20
RIB entries 37, using 6808 bytes of memory
Peers 2, using 1446 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
10.254.254.9 4 65001 29993 29988 0 0 0 02w6d19h 17 19 N/A
10.254.254.11 4 65001 17218 17220 0 0 0 01w4d22h 17 19 N/A

Displayed neighbors 2
Total number of neighbors 2

vyos@leaf1-pod1:~$ show ip route | match /32
B>* 10.255.255.1/32 [20/0] via 10.254.254.9, eth0, weight 1, 02w6d19h
B>* 10.255.255.2/32 [20/0] via 10.254.254.11, eth1, weight 1, 01w4d22h
B>* 10.255.255.3/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.4/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
C>* 10.255.255.5/32 is directly connected, dum0, 02w6d19h
B>* 10.255.255.6/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d22h
B>* 10.255.255.7/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.8/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.9/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.10/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w4d18h
B>* 10.255.255.11/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w2d01h
B>* 10.255.255.12/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w2d00h
B>* 10.255.255.13/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w2d01h
B>* 10.255.255.14/32 [20/0] via 10.254.254.9, eth0, weight 1, 01w2d00h

On Arista

We verify that we receive prefix + same leaf / superspine / remote pods spine IPs (we note there misses some filters on spine since we receive /31 interco routes):

leaf1-pod3#show ip bgp neighbors 10.254.254.43 received-routes
BGP routing table information for VRF default
Router identifier 10.255.255.11, local AS number 65015
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup, L - labeled-unicast
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI Origin Validation codes: V - valid, I - invalid, U - unknown
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric AIGP LocPref Weight Path
10.254.254.36/31 10.254.254.43 0 - - - 65004 ?
10.254.254.38/31 10.254.254.43 0 - - - 65004 ?
10.254.254.42/31 10.254.254.43 0 - - - 65004 ?
10.254.254.44/31 10.254.254.43 - - - - 65004 65016 ?
10.254.254.46/31 10.254.254.43 0 - - - 65004 ?
* 10.255.255.1/32 10.254.254.43 - - - - 65004 65003 65001 ?
* 10.255.255.2/32 10.254.254.43 - - - - 65004 65003 65001 ?
* 10.255.255.3/32 10.254.254.43 - - - - 65004 65003 65002 ?
* 10.255.255.4/32 10.254.254.43 - - - - 65004 65003 65002 ?
* 10.255.255.5/32 10.254.254.43 - - - - 65004 65003 65001 65011 ?
* 10.255.255.6/32 10.254.254.43 - - - - 65004 65003 65001 65012 ?
* 10.255.255.7/32 10.254.254.43 - - - - 65004 65003 65002 65013 ?
* 10.255.255.8/32 10.254.254.43 - - - - 65004 65003 65002 65014 ?
* 10.255.255.9/32 10.254.254.43 - - - - 65004 65003 ?
* 10.255.255.10/32 10.254.254.43 - - - - 65004 65003 ?
* 10.255.255.12/32 10.254.254.43 - - - - 65004 65016 ?
* > 10.255.255.14/32 10.254.254.43 0 - - - 65004 ?
leaf1-pod3#show ip bgp neighbors 10.254.254.41 received-routes
BGP routing table information for VRF default
Router identifier 10.255.255.11, local AS number 65015
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup, L - labeled-unicast
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI Origin Validation codes: V - valid, I - invalid, U - unknown
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric AIGP LocPref Weight Path
* > 10.255.255.1/32 10.254.254.41 - - - - 65004 65003 65001 ?
* > 10.255.255.2/32 10.254.254.41 - - - - 65004 65003 65001 ?
* > 10.255.255.3/32 10.254.254.41 - - - - 65004 65003 65002 ?
* > 10.255.255.4/32 10.254.254.41 - - - - 65004 65003 65002 ?
* > 10.255.255.5/32 10.254.254.41 - - - - 65004 65003 65001 65011 ?
* > 10.255.255.6/32 10.254.254.41 - - - - 65004 65003 65001 65012 ?
* > 10.255.255.7/32 10.254.254.41 - - - - 65004 65003 65002 65013 ?
* > 10.255.255.8/32 10.254.254.41 - - - - 65004 65003 65002 65014 ?
* > 10.255.255.9/32 10.254.254.41 - - - - 65004 65003 ?
* > 10.255.255.10/32 10.254.254.41 - - - - 65004 65003 ?
* > 10.255.255.12/32 10.254.254.41 - - - - 65004 65016 ?
* > 10.255.255.13/32 10.254.254.41 - - - - 65004 i

On Cumulus
cumulus@cumulus:mgmt:~$ net show bgp summary
show bgp ipv4 unicast summary
=============================
BGP router identifier 10.255.255.12, local AS number 65016 vrf-id 0
BGP table version 103
RIB entries 37, using 7104 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.254.254.45 4 65004 306764 261331 0 0 0 01:39:02 12
spine1-pod3(10.254.254.47) 4 65004 261602 261664 0 0 0 01w2d02h 16

Total number of neighbors 2

[...]

cumulus@cumulus:mgmt:~$ net show route | grep /32
B>* 10.255.255.1/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.2/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.3/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.4/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.5/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.6/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.7/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.8/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.9/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.10/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.11/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
C>* 10.255.255.12/32 is directly connected, lo, 01w2d02h
B>* 10.255.255.13/32 [20/0] via 10.254.254.45, swp1, weight 1, 01:39:03
B>* 10.255.255.14/32 [20/0] via 10.254.254.47, swp2, weight 1, 01w2d02h

We can also verify that routing is ok by pinging remote loopback IPs from local loopback + check eBGP EVPN session status:

# ping example from VyOS leaf
vyos@leaf1-pod1:~$ ping 10.255.255.12 source-address 10.255.255.5
PING 10.255.255.12 (10.255.255.12) from 10.255.255.5 : 56(84) bytes of data.
64 bytes from 10.255.255.12: icmp_seq=1 ttl=61 time=2.60 ms
64 bytes from 10.255.255.12: icmp_seq=2 ttl=61 time=2.54 ms
64 bytes from 10.255.255.12: icmp_seq=3 ttl=61 time=2.44 ms

# check of EVPN sessions on the different vendors

# vyos

vyos@leaf1-pod1:~$ show bgp l2vpn evpn summary
BGP router identifier 10.255.255.5, local AS number 65011 vrf-id 0
BGP table version 0
RIB entries 31, using 5704 bytes of memory
Peers 2, using 1446 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
10.255.255.1 4 65001 30205 30208 0 0 0 02w6d21h 13 13 N/A
10.255.255.2 4 65001 17392 17429 0 0 0 01w5d00h 11 13 N/A

# Arista

leaf1-pod3#show bgp evpn summary
BGP summary information for VRF default
Router identifier 10.255.255.11, local AS number 65015
Neighbor Status Codes: m - Under maintenance
Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
OVL_spine1-pod3 10.255.255.13 4 65004 15608 15503 0 0 00:03:47 Estab 17 17
OVL_spine2-pod3 10.255.255.14 4 65004 262780 308558 0 0 9d02h Estab 13 13

# Cumulus

cumulus@cumulus:mgmt:~$ net show bgp l2vpn evpn summary
BGP router identifier 10.255.255.12, local AS number 65016 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.255.255.13 4 65004 307243 261693 0 0 0 00:04:25 14
spine1-pod3(10.255.255.14) 4 65004 262001 262025 0 0 0 01w2d02h 10

On EVPN, we should receive quite the same number of prefixes from both spine, but looks like there differences of best path election + redistribution between vendors. Perhaps something to dig. The most important point is to verify that the FIB for the services are consistent.

L2 service implementation

On that topic, we'll firstly focus on the Layer 2 service. To be sure that all works as expected we have to check that:

mac-address-table are consistent on the whole fabric. In traditionnal L2 setup, switch FIBs depend of the traffic passing through them and a mac @ can be see "everywhere" if it does a BUM on the network. In VXLAN, in a mac-vrf topology without filer, even a host doesn't send BUM, the mac address is propagated on the whole fabric (within BGP update). But, as it is on traditional L2 topology, you have to be sure that mac @ table aging time are consistent on all devices, with an aging time superior to ARP timer on hosts.
routes type 2 are visible on each leaf nodes for a same vni, routes type 3 to reach remote VTEP have to be learned
a common route-target for the VRF routes need to be used/set within policies. In effect, there are differences between vendors on route "tagging" (features to auto generate route-target, "hard coded" alogos, etc.). Here, on VyOS, we got to add adminstrative extended communities within route-map for the L2 VNI. We'll show it below
all fabric connections (interconnections between network nodes) have the sufficient "IP MTU". Please keep in mind that VXLAN overhead has a size of 50 bytes. So, to be sure that Jumbo traffic for host is ok, configure it at least at 9050 Bytes. I would advise to push it at 9100 Bytes in case of VXLAN usage on client HOSTS for Virtualized setup on servers.

Check FDB on leaf switch + debug type 2 routes' propagation for leaf1 pod1

##############
# VyOS
##############

vyos@leaf1-pod1:~$ show bridge br100 fdb
50:01:00:01:00:02 dev eth2 vlan 1 master br100 permanent
50:01:00:01:00:02 dev eth2 master br100 permanent
33:33:00:00:00:01 dev eth2 self permanent
33:33:00:00:00:02 dev eth2 self permanent
01:00:5e:00:00:01 dev eth2 self permanent
01:80:c2:00:00:0e dev eth2 self permanent
01:80:c2:00:00:03 dev eth2 self permanent
01:80:c2:00:00:00 dev eth2 self permanent
33:33:00:00:00:01 dev br100 self permanent
33:33:00:00:00:02 dev br100 self permanent
33:33:ff:01:00:02 dev br100 self permanent
01:00:5e:00:00:6a dev br100 self permanent
33:33:00:00:00:6a dev br100 self permanent
01:00:5e:00:00:01 dev br100 self permanent
33:33:ff:00:00:00 dev br100 self permanent
00:50:79:66:68:15 dev vxlan100 vlan 1 extern_learn master br100
00:50:79:66:68:15 dev vxlan100 extern_learn master br100
8a:d0:00:16:1c:17 dev vxlan100 vlan 1 master br100 permanent
8a:d0:00:16:1c:17 dev vxlan100 master br100 permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.6 self permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.11 self permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.12 self permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.7 self permanent
00:00:00:00:00:00 dev vxlan100 dst 10.255.255.8 self permanent
00:50:79:66:68:15 dev vxlan100 dst 10.255.255.12 self extern_learn

# and to focus on remote nodes

vyos@leaf1-pod1:~$ show bridge br100 fdb | grep "self extern_learn"
00:50:79:66:68:15 dev vxlan100 dst 10.255.255.12 self extern_learn
00:50:79:66:68:13 dev vxlan100 dst 10.255.255.11 self extern_learn
00:50:79:66:68:19 dev vxlan100 dst 10.255.255.8 self extern_learn
00:50:79:66:68:0c dev vxlan100 dst 10.255.255.6 self extern_learn
00:50:79:66:68:17 dev vxlan100 dst 10.255.255.7 self extern_learn

# or on local clients

vyos@leaf1-pod1:~$ show bridge br100 fdb | grep eth | grep -v perm
00:50:79:66:68:0b dev eth2 master br100

# Now if we want to focus on route type 3 learned

Displayed 7 prefixes (12 paths) (of requested type)
vyos@leaf1-pod1:~$ show bgp l2vpn evpn route type 3 | grep Dis
Route Distinguisher: 10.255.255.5:3
Route Distinguisher: 10.255.255.6:3
Route Distinguisher: 10.255.255.7:2
Route Distinguisher: 10.255.255.8:2
Route Distinguisher: 10.255.255.11:100
Route Distinguisher: 10.255.255.12:100

# we notice that RDs for VyOS nodes don't respect the logic of <Loopback:vni>. It s because for the moment we can't set it by vni (or I didn't find how to do that)

##############
# Arista
##############

# we check the fdb. Remote host are behind the port Vx1

leaf1-pod3#show mac address-table vlan 100
Mac Address Table
------------------------------------------------------------------

Vlan Mac Address Type Ports Moves Last Move
---- ----------- ---- ----- ----- ---------
100 0050.7966.680c DYNAMIC Vx1 1 0:00:38 ago
100 0050.7966.6813 DYNAMIC Et3 1 0:14:08 ago
100 0050.7966.6815 DYNAMIC Vx1 1 9 days, 2:36:39 ago
100 0050.7966.6817 DYNAMIC Vx1 1 0:00:36 ago
100 0050.7966.6819 DYNAMIC Vx1 1 0:00:35 ago
Total Mac Addresses for this criterion: 5

# we check that all VTEP are ready for BUM

leaf1-pod3#show bgp evpn route-type imet | i >
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
* >Ec RD: 10.255.255.5:3 imet 10.255.255.5
* >Ec RD: 10.255.255.6:3 imet 10.255.255.6
* >Ec RD: 10.255.255.7:2 imet 10.255.255.7
* >Ec RD: 10.255.255.8:2 imet 10.255.255.8
* > RD: 10.255.255.11:100 imet 10.255.255.11
* >Ec RD: 10.255.255.12:100 imet 10.255.255.12
[...]

leaf1-pod3#show vxlan vtep detail
Remote VTEPS for Vxlan1:

VTEP Learned Via MAC Address Learning Tunnel Type(s)
------------------ ------------------ -------------------------- --------------
10.255.255.5 control plane control plane flood, unicast
10.255.255.6 control plane control plane flood, unicast
10.255.255.7 control plane control plane flood, unicast
10.255.255.8 control plane control plane flood, unicast
10.255.255.12 control plane control plane flood, unicast

##############
# Cumulus
##############

# we check the fdb. Remote host are behind the port Vx1

cumulus@cumulus:mgmt:~$ net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ ----------------
100 bridge bridge 50:01:00:11:00:03 permanent 9 days, 02:47:42
100 bridge swp3 00:50:79:66:68:15 00:00:06
100 bridge vni100 00:50:79:66:68:0c extern_learn 00:00:40
100 bridge vni100 00:50:79:66:68:13 extern_learn 00:21:01
100 bridge vni100 00:50:79:66:68:17 extern_learn 00:00:38
100 bridge vni100 00:50:79:66:68:19 extern_learn 00:00:36

# --> here we see that it misses the @mac from the host on leaf1 pod1.
# we check the evpn routes on the Cumulus node and we see that from leaf1 pod1 vni 100 we have only those routes

Route Distinguisher: 10.255.255.5:3
* [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
* [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
*> [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8

# --> something is wrong on the remote leaf.
# on the leaf we check the policy for the EVPN routes export

route-map RM-OUT-VNI {
rule 1 {
action permit
match {
evpn {
route-type multicast
vni 100
}
}
set {
extcommunity {
rt 65000:100
}
}
}
rule 2 {
action permit
match {
evpn {
route-type prefix
vni 100
}
}
set {
extcommunity {
rt 65000:100
}
}
}
rule 3 {
action permit
match {
evpn {
vni 200
}
}
set {
extcommunity {
rt 65000:200
}

# we notice that a typo in rule numbering removed the export of macip routes for vni 100.
# we correct that :

route-map RM-OUT-VNI {
rule 1 {
action permit
match {
evpn {
route-type multicast
vni 100
}
}
set {
extcommunity {
rt 65000:100
}
}
}
rule 2 {
action permit
match {
evpn {
route-type macip
vni 100
}
}
set {
extcommunity {
rt 65000:100
}
}
}
rule 3 {
action permit
match {
evpn {
route-type prefix
vni 200
}
}
set {
extcommunity {
rt 65000:200
}

# once corrected we check the fdb on the cumulus node + evpn routes

cumulus@cumulus:mgmt:~$ net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ ----------------
100 bridge bridge 50:01:00:11:00:03 permanent 9 days, 02:51:48
100 bridge swp3 00:50:79:66:68:15 00:02:01
100 bridge vni100 00:50:79:66:68:0b extern_learn 00:00:26
100 bridge vni100 00:50:79:66:68:0c extern_learn 00:04:46
100 bridge vni100 00:50:79:66:68:13 extern_learn 00:25:08
100 bridge vni100 00:50:79:66:68:17 extern_learn 00:04:45
100 bridge vni100 00:50:79:66:68:19 extern_learn 00:04:43

Route Distinguisher: 10.255.255.5:3
* [2]:[0]:[48]:[00:50:79:66:68:0b]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
* [2]:[0]:[48]:[00:50:79:66:68:0b]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
*> [2]:[0]:[48]:[00:50:79:66:68:0b]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
* [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
* [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8
*> [3]:[0]:[32]:[10.255.255.5]
10.255.255.5 0 65004 65003 65001 65011 i
RT:65000:100 RT:65011:100 ET:8

# we have now Type 2 routes, all is ok now and we learn the client mac 00:50:79:66:68:0b
# Nevertheless, the trafic was working between the client hosts, but trafic from Cumulus leaf client to VyOS leaf client was broadcasted to all leaf since the @mac was not advertised on the fabric

It could be interesting to check how each node is configured to deliver the service, but the article would be too long. So, to sum the steps to provide a L2 service:

create a VLAN (on Arista) / bridge (VyOS and Cumulus) + associate it to the VNI/VXLAN instances/interface
create the mac vrf + define the route-target into bgp (Arista + Cumulus) / update the route-maps + community list to allow the redistribution of the VNI routes (type 2 and type 3)
L3 service implementation

Now, we'll now look at the L3 service. On that kind of setup we need to implement / check:

leaf IP subnets are well redistributed betweeb all leaf (here we are in a vrf "any to any" topology)
we only redistribute routes for IP prefix. Routes type 2 (mac @) don't need to be redistributed between leaf
that all clients can ping each other through the L3VPN

To achieve that, we need to implement analog configuration than for the L2 service, but with the following differences:

create a VLAN (on Arista) / bridge (VyOS and Cumulus) + associate it to the VNI/VXLAN instances|interface
create the L3 VRF + define the route-target into bgp (Arista + Cumulus + VyOS) + allow connected routes distribution (Arista / Cumulus) / update the route-maps + community list to allow the redistribution of the VNI routes (type 5) for VyOS / restrict the routes redistribution to type 5 routes into the VNI (Cumulus)

Now we check the routing table with common cli commands + into the EVPN RIB :

# check routes into the vrf

vyos@leaf1-pod1:~$ show ip route vrf VRF-L3-1
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF VRF-L3-1:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 02w6d22h
C>* 192.168.10.0/24 is directly connected, br200, 02w6d22h
B>* 192.168.20.0/24 [20/0] via 10.255.255.6, br200 onlink, weight 1, 05:11:08
B>* 192.168.30.0/24 [20/0] via 10.255.255.7, br200 onlink, weight 1, 05:04:15
B>* 192.168.40.0/24 [20/0] via 10.255.255.8, br200 onlink, weight 1, 04:59:41
B>* 192.168.50.0/24 [20/0] via 10.255.255.11, br200 onlink, weight 1, 05:11:08
B>* 192.168.60.0/24 [20/0] via 10.255.255.12, br200 onlink, weight 1, 05:11:08

leaf1-pod3#show ip route vrf VRF-L3-1

VRF: VRF-L3-1
Codes: C - connected, S - static, K - kernel,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type2, B - BGP, B I - iBGP, B E - eBGP,
R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
NG - Nexthop Group Static Route, V - VXLAN Control Service,
DH - DHCP client installed default route, M - Martian,
DP - Dynamic Policy Route, L - VRF Leaked,
G - gRIBI, RC - Route Cache Route

Gateway of last resort is not set

B E 192.168.10.0/24 [200/0] via VTEP 10.255.255.5 VNI 200 router-mac 12:9b:9b:d8:f6:86 local-interface Vxlan1
B E 192.168.20.0/24 [200/0] via VTEP 10.255.255.6 VNI 200 router-mac 50:01:00:02:00:03 local-interface Vxlan1
B E 192.168.30.0/24 [200/0] via VTEP 10.255.255.7 VNI 200 router-mac 50:01:00:09:00:03 local-interface Vxlan1
B E 192.168.40.0/24 [200/0] via VTEP 10.255.255.8 VNI 200 router-mac 50:01:00:0a:00:03 local-interface Vxlan1
C 192.168.50.0/24 is directly connected, Vlan200
B E 192.168.60.0/24 [200/0] via VTEP 10.255.255.12 VNI 200 router-mac 50:01:00:11:00:03 local-interface Vxlan1

cumulus@cumulus:mgmt:~$ net show route vrf VRF-L3-1
show ip route vrf VRF-L3-1
===========================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued route, r - rejected route

VRF VRF-L3-1:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 01w2d03h
B>* 192.168.10.0/24 [20/0] via 10.255.255.5, vlan200 onlink, weight 1, 03:18:37
B>* 192.168.20.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 03:18:37
B>* 192.168.30.0/24 [20/0] via 10.255.255.7, vlan200 onlink, weight 1, 03:18:37
B>* 192.168.40.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:18:37
B>* 192.168.50.0/24 [20/0] via 10.255.255.11, vlan200 onlink, weight 1, 03:18:37
C>* 192.168.60.0/24 is directly connected, vlan200, 01w2d03h

# we can notice that the output are quite similar, except on Arista which indicates the VNI for remote targets, instead of referencing the local bridge / vlan200

# Now we check the routes for the VNI 200

# VyOS

vyos@leaf1-pod1:~$ show bgp l2vpn evpn route type prefix
BGP table version is 1, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:200
*> [5]:[0]:[24]:[192.168.10.0]
10.255.255.5 0 32768 ?
ET:8 RT:65000:200 Rmac:12:9b:9b:d8:f6:86
Route Distinguisher: 10.255.255.6:200
*> [5]:[0]:[24]:[192.168.20.0]
10.255.255.6 0 65001 65012 ?
RT:65000:200 ET:8 Rmac:50:01:00:02:00:03
Route Distinguisher: 10.255.255.7:200
* [5]:[0]:[24]:[192.168.30.0]
10.255.255.7 0 65001 65003 65002 65013 ?
RT:65000:200 RT:65013:200 ET:8 Rmac:50:01:00:09:00:03
*> [5]:[0]:[24]:[192.168.30.0]
10.255.255.7 0 65001 65003 65002 65013 ?
RT:65000:200 RT:65013:200 ET:8 Rmac:50:01:00:09:00:03
Route Distinguisher: 10.255.255.8:200
*> [5]:[0]:[24]:[192.168.40.0]
10.255.255.8 0 65001 65003 65002 65014 ?
RT:65000:200 RT:65014:200 ET:8 Rmac:50:01:00:0a:00:03
* [5]:[0]:[24]:[192.168.40.0]
10.255.255.8 0 65001 65003 65002 65014 ?
RT:65000:200 RT:65014:200 ET:8 Rmac:50:01:00:0a:00:03
Route Distinguisher: 10.255.255.11:200
* [5]:[0]:[24]:[192.168.50.0]
10.255.255.11 0 65001 65003 65004 65015 i
RT:65000:200 ET:8 Rmac:50:01:00:cd:ff:68
*> [5]:[0]:[24]:[192.168.50.0]
10.255.255.11 0 65001 65003 65004 65015 i
RT:65000:200 ET:8 Rmac:50:01:00:cd:ff:68
Route Distinguisher: 10.255.255.12:10200
* [5]:[0]:[24]:[192.168.60.0]
10.255.255.12 0 65001 65003 65004 65016 i
RT:65000:200 ET:8 Rmac:50:01:00:11:00:03
*> [5]:[0]:[24]:[192.168.60.0]
10.255.255.12 0 65001 65003 65004 65016 i
RT:65000:200 ET:8 Rmac:50:01:00:11:00:03

# Arista

leaf1-pod3#show bgp evpn route ip-prefix ipv4
BGP routing table information for VRF default
Router identifier 10.255.255.11, local AS number 65015
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop

Network Next Hop Metric LocPref Weight Path
* > RD: 10.255.255.5:200 ip-prefix 192.168.10.0/24
10.255.255.5 - 100 0 65004 65003 65001 65011 ?
* RD: 10.255.255.5:200 ip-prefix 192.168.10.0/24
10.255.255.5 - 100 0 65004 65003 65001 65011 ?
* > RD: 10.255.255.6:200 ip-prefix 192.168.20.0/24
10.255.255.6 - 100 0 65004 65003 65001 65012 ?
* RD: 10.255.255.6:200 ip-prefix 192.168.20.0/24
10.255.255.6 - 100 0 65004 65003 65001 65012 ?
* > RD: 10.255.255.7:200 ip-prefix 192.168.30.0/24
10.255.255.7 - 100 0 65004 65003 65002 65013 ?
* RD: 10.255.255.7:200 ip-prefix 192.168.30.0/24
10.255.255.7 - 100 0 65004 65003 65002 65013 ?
* > RD: 10.255.255.8:200 ip-prefix 192.168.40.0/24
10.255.255.8 - 100 0 65004 65003 65002 65014 ?
* RD: 10.255.255.8:200 ip-prefix 192.168.40.0/24
10.255.255.8 - 100 0 65004 65003 65002 65014 ?
* > RD: 10.255.255.11:200 ip-prefix 192.168.50.0/24
- - - 0 i
* > RD: 10.255.255.12:10200 ip-prefix 192.168.60.0/24
10.255.255.12 - 100 0 65004 65016 i
* RD: 10.255.255.12:10200 ip-prefix 192.168.60.0/24
10.255.255.12 - 100 0 65004 65016 i

# CVX

cumulus@cumulus:mgmt:~$ net show bgp l2vpn evpn route type prefix
BGP table version is 7, local router ID is 10.255.255.12
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:200
* [5]:[0]:[24]:[192.168.10.0]
10.255.255.5 0 65004 65003 65001 65011 ?
RT:65000:200 ET:8 Rmac:12:9b:9b:d8:f6:86
*> [5]:[0]:[24]:[192.168.10.0]
10.255.255.5 0 65004 65003 65001 65011 ?
RT:65000:200 ET:8 Rmac:12:9b:9b:d8:f6:86
Route Distinguisher: 10.255.255.6:200
* [5]:[0]:[24]:[192.168.20.0]
10.255.255.6 0 65004 65003 65001 65012 ?
RT:65000:200 ET:8 Rmac:50:01:00:02:00:03
*> [5]:[0]:[24]:[192.168.20.0]
10.255.255.6 0 65004 65003 65001 65012 ?
RT:65000:200 ET:8 Rmac:50:01:00:02:00:03
Route Distinguisher: 10.255.255.7:200
* [5]:[0]:[24]:[192.168.30.0]
10.255.255.7 0 65004 65003 65002 65013 ?
RT:65000:200 RT:65013:200 ET:8 Rmac:50:01:00:09:00:03
*> [5]:[0]:[24]:[192.168.30.0]
10.255.255.7 0 65004 65003 65002 65013 ?
RT:65000:200 RT:65013:200 ET:8 Rmac:50:01:00:09:00:03
Route Distinguisher: 10.255.255.8:200
* [5]:[0]:[24]:[192.168.40.0]
10.255.255.8 0 65004 65003 65002 65014 ?
RT:65000:200 RT:65014:200 ET:8 Rmac:50:01:00:0a:00:03
*> [5]:[0]:[24]:[192.168.40.0]
10.255.255.8 0 65004 65003 65002 65014 ?
RT:65000:200 RT:65014:200 ET:8 Rmac:50:01:00:0a:00:03
Route Distinguisher: 10.255.255.11:200
* [5]:[0]:[24]:[192.168.50.0]
10.255.255.11 0 65004 65015 i
RT:65000:200 ET:8 Rmac:50:01:00:cd:ff:68
*> [5]:[0]:[24]:[192.168.50.0]
10.255.255.11 0 65004 65015 i
RT:65000:200 ET:8 Rmac:50:01:00:cd:ff:68
Route Distinguisher: 10.255.255.12:10200
*> [5]:[0]:[24]:[192.168.60.0]
10.255.255.12 0 32768 i
ET:8 RT:65000:200 Rmac:50:01:00:11:00:03

# Now we check with traceroute on the 3 clients connected on focused leaf switches

# client leaf VyOS

VPCS> ping 192.168.50.2

84 bytes from 192.168.50.2 icmp_seq=1 ttl=62 time=16.456 ms
84 bytes from 192.168.50.2 icmp_seq=2 ttl=62 time=11.852 ms
^C
VPCS> ping 192.168.60.2

84 bytes from 192.168.60.2 icmp_seq=1 ttl=62 time=5.554 ms
84 bytes from 192.168.60.2 icmp_seq=2 ttl=62 time=4.432 ms
^C
VPCS> trace 192.168.50.2
trace to 192.168.50.2, 8 hops max, press Ctrl+C to stop
1 192.168.10.1 0.499 ms 3.143 ms 1.432 ms
2 192.168.50.1 10.465 ms 12.512 ms 8.992 ms
3 *192.168.50.2 11.056 ms (ICMP type:3, code:3, Destination port unreachable)

VPCS> trace 192.168.60.2
trace to 192.168.60.2, 8 hops max, press Ctrl+C to stop
1 192.168.10.1 0.525 ms 0.312 ms 0.278 ms
2 192.168.60.1 4.949 ms 4.232 ms 4.022 ms
3 *192.168.60.2 4.167 ms (ICMP type:3, code:3, Destination port unreachable)

# client leaf Arista

VPCS> ping 192.168.10.2

84 bytes from 192.168.10.2 icmp_seq=1 ttl=62 time=10.914 ms
^C
VPCS> ping 192.168.60.2

84 bytes from 192.168.60.2 icmp_seq=1 ttl=62 time=11.755 ms
^C
VPCS> trace 192.168.10.2
trace to 192.168.10.2, 8 hops max, press Ctrl+C to stop
1 192.168.50.1 4.742 ms 4.123 ms 4.293 ms
2 192.168.10.1 12.550 ms 15.039 ms 9.411 ms
3 *192.168.10.2 8.939 ms (ICMP type:3, code:3, Destination port unreachable)

VPCS>
VPCS> trace 192.168.60.2
trace to 192.168.60.2, 8 hops max, press Ctrl+C to stop
1 192.168.50.1 3.073 ms 2.901 ms 2.585 ms
2 192.168.60.1 10.815 ms 11.493 ms 12.144 ms
3 *192.168.60.2 10.429 ms (ICMP type:3, code:3, Destination port unreachable)

# client leaf Cumulus

VPCS> ping 192.168.10.2

84 bytes from 192.168.10.2 icmp_seq=1 ttl=62 time=4.731 ms
^C
VPCS> ping 192.168.50.2

84 bytes from 192.168.50.2 icmp_seq=1 ttl=62 time=11.722 ms
^C
VPCS> trace 192.168.50.2
trace to 192.168.50.2, 8 hops max, press Ctrl+C to stop
1 192.168.60.1 0.487 ms 0.266 ms 0.248 ms
2 192.168.50.1 6.691 ms 6.281 ms 6.613 ms
3 *192.168.50.2 9.355 ms (ICMP type:3, code:3, Destination port unreachable)

VPCS>
VPCS> trace 192.168.10.2
trace to 192.168.10.2, 8 hops max, press Ctrl+C to stop
1 192.168.60.1 0.436 ms 0.315 ms 0.380 ms
2 192.168.10.1 4.535 ms 4.661 ms 3.994 ms
3 *192.168.10.2 4.292 ms (ICMP type:3, code:3, Destination port unreachable)

Conclusion of the lab

These first results are satisfying, it comforts the idea that VXLAN is a strong option to deliver L2/L3 services in datacenters, but not only. This kind of architecture is more and more chosen in campus designs, PRA, NFV solutions.

The support of this technology on more and more open source NOS are encouraging regarding the innovation. If we look at Vyatta project for example, whom OS can be compiled "brick per brick", we could imagine to create NFV CPEs providing VXLAN fabric extensions of some client VRFs. CPE could be connected to the central Fabric within encrypted tunnels (wireguard,ipsec) or through direct link over MacSec (this feature in on the Vyatta Roadmap).

Please contact us if you want to feed the topic / discuss the lab.

Cheers

Pierre L.

Nesevo Virtual Labs

2022-04-05T12:24:15Z

PierreLancastre:

[[File:nesevo-labs-v2.png]]

Welcome to our Labs Wiki page.

Here you'll find our shares about what we've experimented in lab. Results / observations could change today with latest versions and our knowledge improvement.

Don't hesitate to share your feedbacks, thanks.

= VyOS/Arista/CVX Lab with EVE-NG =

[[VyosAristaCVX-VXLAN-lab]]

= Nokia/Cumulus ContainerLab =

[[Cumulus-Nokia-Lab]]

= SONiC with Docker =

[[Sonic-lab]]

= M-Vendor VXLAN Lab with EVE-NG =

[[Mvendor-VXLAN-lab]]

Cumulus-Nokia-Lab

2021-12-03T08:46:34Z

PierreLancastre:

The targets of this lab are to :

* mix Nokia SR-Linux and Cumulus VXLAN nodes, in spine or leaf roles
* make this environment work for different kinds of setup : cross-DC L2, differentiated L3 gateways per leaf per VRF, anycast gateway cross or intra DC, etc.
* spot compatibility issues if existing

To reach those goals, we'll use the Nokia lab tool "containerlab" https://containerlab.srlinux.dev/, labs from https://clabs.netdevops.me/ helped a lot. Many thanks to the contributors.

We'll need a Linux host with Docker installed on it + kvm virtualization enhancements activated (important if your host is a VM). If virtualization optimization is not ok, you won't be able to launch the most of devices.

Installation details are referenced there : https://containerlab.srlinux.dev/install/

Once installed, you'll need to register the different NOS images you'll use. Cumulus image is present on a public registrery, but Nokia/Arista/Juniper are not, you'll need a vendor support account to retreive them. Once downloaded (.tgz file typically) you can "docker load" the image :

For private images :<syntaxhighlight lang="bash">
sudo docker load -i 21.6.1-235.tar.xz /* (Nokia) */
sudo docker load -i cEOS64-lab-4.26.1F.tar /* (Arista) */
</syntaxhighlight>For public images :<syntaxhighlight lang="bash">
sudo docker pull networkop/cx:4.3.0 /* (Cumulus) */
</syntaxhighlight>

To verify your installed images, you can run "sudo docker images"<syntaxhighlight lang="bash">
plancastre@containerlab:~/00.LAB/senss$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
networkop/cx 4.3.0 a40541ed15ba 2 weeks ago 733MB
srlinux 21.6.1-235 e29aed30e83f 3 weeks ago 1.14GB
networkop/ignite dev dcc5eff29355 4 weeks ago 37.9MB
networkop/kernel 4.19 e794d8875f83 3 months ago 88MB
frrouting/frr v7.5.1 c3e13a4c5918 3 months ago 123MB

</syntaxhighlight>Once you've got all your images, to run the lab, you'll need a .yml file defining the topology to run.

In our lab, we've started with the 5-clos lab file from Nokia site, which we've forked.

We can see there is a branch "kinds" where we can define templates of nodes, in order to not rewrite whole parameters when creating nodes. For the images, it's important to take the repository name + tag.

'''Note:''' If the tag is "latest" it is not needed.

All supported "kinds" are referenced there : https://containerlab.srlinux.dev/manual/kinds/kinds/

'''Note:''' yml is indentation sensitive, so please respect the "double space" used to define hierarchy. Below the lab diagram + the yml file associated

== Infos on nodes ==

=== L3 infos ===
{| class="wikitable"
|+
!Device
!OS
!ASN
!Loopback/VTEP
!swp1 or eth1/1
!swp2 or eth1/2
!swp3 or eth1/3
!SVI
|-
|client1
|Linux
|<none>
|<none>
|vlan100: 192.168.1.1/24
vlan200: 192.168.2.1/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client2
|Linux
|<none>
|<none>
|vlan100: 192.168.1.2/24
vlan200: 192.168.2.2/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client3
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.1/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|client4
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.2/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|leaf1
|CVX
|65011
|10.255.255.5/32
|10.254.254.8/31
|10.254.254.10/31
|trunk vlan 100,200
|vlan200 : 192.168.2.254/24
|-
|leaf2
|SRLinux
|65012
|10.255.255.6/32
|10.254.254.12/31
|10.254.254.14/31
|trunk vlan 100,200
|irb0.200 : 192.168.2.254/24
|-
|leaf3
|CVX
|65013
|10.255.255.7/32
|10.254.254.22/31
|10.254.254.24/31
|trunk vlan 100,200
|vlan200 : 192.168.200.254/24
|-
|leaf4
|SRLinux
|65014
|10.255.255.8/32
|10.254.254.26/31
|10.254.254.28/31
|trunk vlan 100,200
|irb0.200 : 192.168.200.254/24
|-
|spine1
|CVX
|65001
|10.255.255.1/32
|10.254.254.9/31
|10.254.254.13/31
|10.254.254.0/31
|<none>
|-
|spine2
|CVX
|65001
|10.255.255.2/32
|10.254.254.11/31
|10.254.254.15/31
|10.254.254.16/31
|<none>
|-
|spine3
|SRLinux
|65002
|10.255.255.3/32
|10.254.254.23/31
|10.254.254.27/31
|10.254.254.18/31
|<none>
|-
|spine4
|SRLinux
|65002
|10.255.255.4/32
|10.254.254.25/31
|10.254.254.29/31
|10.254.254.20/31
|<none>
|-
|superspine1
|CVX
|65003
|10.255.255.9/32
|10.254.254.1/31
|10.254.254.19/31
|<none>
|<none>
|-
|superspine2
|SRLinux
|65003
|10.255.255.10/32
|10.254.254.17/31
|10.254.254.21/31
|<none>
|<none>
|}

=== VXLAN/BGP infos ===
Note : in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>
{| class="wikitable"
|+
!service
!vlan id
!route-target
!VNI
!Commentaires
|-
|L2 cross-DC
|100
|target:65000:100
|100
|the 4 clients are in the same broadcast domain / subnet
|-
|L3VPN cros-DC
|200
|target:65000:200
target:65000:10200
|200 (for L2)
10200 (for routing)
|the 4 clients are separated in 2 different subnets. Traffic is routed within dedicated VNI
To be noted that in the current setup VMTO is not implemented, so cross DC traffic is not optimized.
|}

== Lab diagram ==
[[File:Lab-srl-diagram.png|ohne|mini|600x600px]]

== yml lab file ==
<syntaxhighlight lang="yaml">
# topology documentation: http://containerlab.srlinux.dev/lab-examples/min-5clos/
name: lab-srl-cum

topology:
kinds:
srl:
image: srlinux:21.6.1-235
linux:
image: ghcr.io/hellt/network-multitool
cvx:
image: networkop/cx:4.3.0
runtime: docker
nodes:
leaf1:
kind: cvx
leaf2:
kind: srl
type: ixrd2
leaf3:
kind: cvx
leaf4:
kind: srl
type: ixrd2

spine1:
kind: cvx
spine2:
kind: cvx
spine3:
kind: srl
type: ixr6
spine4:
kind: srl
type: ixr6

superspine1:
kind: cvx
superspine2:
kind: srl
type: ixr6

client1:
kind: linux
client2:
kind: linux
client3:
kind: linux
client4:
kind: linux

links:
# leaf to spine links POD1
- endpoints: ["leaf1:swp1", "spine1:swp1"]
- endpoints: ["leaf1:swp2", "spine2:swp1"]
- endpoints: ["leaf2:e1-1", "spine1:swp2"]
- endpoints: ["leaf2:e1-2", "spine2:swp2"]

# spine to superspine links POD1
- endpoints: ["spine1:swp3", "superspine1:swp1"]
- endpoints: ["spine2:swp3", "superspine2:e1-1"]

# leaf to spine links POD2
- endpoints: ["leaf3:swp1", "spine3:e1-1"]
- endpoints: ["leaf3:swp2", "spine4:e1-1"]
- endpoints: ["leaf4:e1-1", "spine3:e1-2"]
- endpoints: ["leaf4:e1-2", "spine4:e1-2"]

# spine to superspine links POD2
- endpoints: ["spine3:e1-3", "superspine1:swp2"]
- endpoints: ["spine4:e1-3", "superspine2:e1-2"]

# client connection links
- endpoints: ["client1:eth1", "leaf1:swp3"]
- endpoints: ["client2:eth1", "leaf2:e1-3"]
- endpoints: ["client3:eth1", "leaf3:swp3"]
- endpoints: ["client4:eth1", "leaf4:e1-3"]

</syntaxhighlight>

Once the nodes are defined, we have to define the needed connections between the nodes.

Each endpoint is defined as '<nowiki/>'''<nowiki/>'<node_name>:<interface_name><nowiki>''</nowiki>'''

When all is ok, you can launch in suduoer the topology.

For example :<syntaxhighlight lang="bash">
sudo containerlab deploy --topo topo.clab.yml --reconfigure

Note: The option "reconfigure" is used there in order to not have to destroy instances when updating topology/relaunching.
</syntaxhighlight>Another info regarding the yml file, it's possible to quickly generate files with the '''"containerlab gen -n 3tier --nodes 4,2,1"''' command for example. https://containerlab.srlinux.dev/cmd/generate/

To be noted that the "tier-x" notion in containerlab is inversed regarding the rfc 7938 (tier1 is superspine level into the rfc, tier1 is leaf level in nokia tool)

At the end of the topology launch, you can retreive the details :<syntaxhighlight lang="yaml">
INFO[0011] Writing /etc/hosts file
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| # | Name | Container ID | Image | Kind | Group | State | IPv4 Address | IPv6 Address |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| 1 | clab-lab-srl-cum-client1 | 25e2bbf6b13f | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
| 2 | clab-lab-srl-cum-client2 | 17d82d5a6e21 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.4/24 | 2001:172:20:20::4/64 |
| 3 | clab-lab-srl-cum-client3 | f8d78527c1b1 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.6/24 | 2001:172:20:20::6/64 |
| 4 | clab-lab-srl-cum-client4 | 913a0172d0c5 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 5 | clab-lab-srl-cum-leaf1 | 9ffc334af2b1 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.9/24 | 2001:172:20:20::9/64 |
| 6 | clab-lab-srl-cum-leaf2 | 7d3eb855d749 | srlinux:21.6.1-235 | srl | | running | 172.20.20.14/24 | 2001:172:20:20::e/64 |
| 7 | clab-lab-srl-cum-leaf3 | e0d36ffc606a | networkop/cx:4.3.0 | cvx | | running | 172.20.20.7/24 | 2001:172:20:20::7/64 |
| 8 | clab-lab-srl-cum-leaf4 | e6753b1d099e | srlinux:21.6.1-235 | srl | | running | 172.20.20.12/24 | 2001:172:20:20::c/64 |
| 9 | clab-lab-srl-cum-spine1 | a7e63993d052 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.5/24 | 2001:172:20:20::5/64 |
| 10 | clab-lab-srl-cum-spine2 | 3124250d5b75 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
| 11 | clab-lab-srl-cum-spine3 | 5e300ca3f66f | srlinux:21.6.1-235 | srl | | running | 172.20.20.11/24 | 2001:172:20:20::b/64 |
| 12 | clab-lab-srl-cum-spine4 | 0a59e60a596b | srlinux:21.6.1-235 | srl | | running | 172.20.20.13/24 | 2001:172:20:20::d/64 |
| 13 | clab-lab-srl-cum-superspine1 | 3d9935413fc2 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.8/24 | 2001:172:20:20::8/64 |
| 14 | clab-lab-srl-cum-superspine2 | 5bc58d423e2c | srlinux:21.6.1-235 | srl | | running | 172.20.20.15/24 | 2001:172:20:20::f/64 |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
</syntaxhighlight>The containerlab program creates the updates into the host file + generates an ansible inventory file.

There are different ways to connect onto the docker instances, within docker exec commands or SSH directly the IP address related above.

For example :<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-spine1 bash
</syntaxhighlight>-->>> allows to launch a bash on the cumulus device (then you run the different net commands)<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-leaf2 sr_cli
</syntaxhighlight>-->>> allows to launch the cli on a SRLinux device

To end this setup preparation, the containerlab tool integrates an option to launch a webserver showing the lab topology in graph + node details

By default, it's listening on port 50080 of the host

Then, you can access to the following web page showing the topology + lab infos

[[File:Lab graph tool.jpg|ohne|mini|900x900px]]

Finally, if you want to stop a lab, please run '''<nowiki>''</nowiki>containerlab destroy --topo <topology>.yml<nowiki>''</nowiki>'''

At this step, nodes are running, connections are done, but configurations are empty.

'''So, it's time to configure them :)'''

Configuration files are attached to this blog. Once validated, they are exportable and can be called by the tool when launching the lab. Below how to do it :

Firstable, once you ve finished your "hand made config", you can run the following command to save it :<syntaxhighlight lang="bash">
containerlab save -t quickstart.clab.yml
</syntaxhighlight>'''To be noted it's only working for Nokia nodes in this lab'''. CVX configurations will have to be called by another method (or pushed manually)

It will create one folder per node into the lab folder which gives the following tree :
[[File:Topology-srl.jpg|ohne|mini]]

'''Note''': For Cumulus, you can apparently retrieve configurations files within adding the following parameters into lab config file. '''To be tested.''' some lab samples here : https://github.com/hellt/clabs/blob/main/labs/cvx04/symm-mh.clab.yml

Now we have the lab running, below the different commands to check that all is working as expected : <syntaxhighlight lang="yaml">
-------------------------------------------
Check L2 tables ///
-------------------------------------------

Nokia :

A:leaf2# show network-instance mac-vrf-100 bridge-table mac-table all
----------------------------------------------------------------------------------------------------------------------------------------------------------
Mac-table of network instance mac-vrf-100
----------------------------------------------------------------------------------------------------------------------------------------------------------
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
| Address | Destination | Dest Index | Type | Active | Aging | Last Update |
+====================+=========================================+============+=============+=========+========+=========================================+
| AA:C1:AB:0C:60:D6 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T13:06:17.000Z |
| | vtep:10.255.255.8 vni:100 | 67 | | | | |
| AA:C1:AB:23:31:04 | ethernet-1/3.100 | 2 | learnt | true | 300 | 2021-10-08T13:06:10.000Z |
| AA:C1:AB:4B:FF:71 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:55:28.000Z |
| | vtep:10.255.255.5 vni:100 | 60 | | | | |
| AA:C1:AB:B5:79:50 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:43:37.000Z |
| | vtep:10.255.255.7 vni:100 | 64 | | | | |
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 0 Total 0 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 1 Total 1 Active
Total Evpn Macs : 3 Total 3 Active
Total Evpn static Macs : 0 Total 0 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 4 Total 4 Active
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : type learnt matches the local learnt mac, evpn are learnt within VXLAN

Cumulus :

root@leaf1-cum-dc1:/# net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
100 bridge swp3 aa:c1:ab:4b:ff:71 00:00:02
100 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:17:21
100 bridge vni100 22:2d:67:f1:bb:26 permanent 02:17:21
100 bridge vni100 aa:c1:ab:0c:60:d6 extern_learn <1 sec
100 bridge vni100 aa:c1:ab:23:31:04 extern_learn 00:00:07
100 bridge vni100 aa:c1:ab:b5:79:50 extern_learn 00:13:28

root@leaf1-cum-dc1:/# net show bridge macs vlan 200

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
200 bridge swp3 aa:c1:ab:4b:ff:71 00:08:46
200 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:16:36
200 bridge vni200 00:00:5e:00:01:01 static sticky 00:14:26
200 bridge vni200 1e:0d:aa:46:53:b1 permanent 02:16:36
200 bridge vni200 02:77:6b:ff:00:41 static sticky 01:00:52
200 bridge vni200 02:f8:b8:ff:00:41 static sticky 00:14:26
200 bridge vni200 aa:c1:ab:b5:79:50 extern_learn 00:11:51
200 bridge vni10200 02:77:6b:ff:00:00 extern_learn 00:53:02
200 bridge vni10200 02:f8:b8:ff:00:00 extern_learn 00:14:26
200 bridge vni10200 86:5e:aa:ca:50:be permanent 02:16:36

Note : the macs with permanent flag are local interfaces, physical ou logic (vtep)
Note2 : extern_learn mac are the one learned via the VXLAN infra (and mapped to the vnixxx interface)

-------------------------------------------
Check arp tables ///
-------------------------------------------

Nokia :

:leaf2# show arpnd arp-entries
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| ethernet-1/1 | 0 | 10.254.254.13 | dynamic | AA:C1:AB:59:65:EA | 3 hours from now |
| ethernet-1/2 | 0 | 10.254.254.15 | dynamic | AA:C1:AB:CB:F8:8A | 3 hours from now |
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
| mgmt0 | 0 | 172.20.20.1 | dynamic | 02:42:63:9E:74:03 | 2 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 5 (0 static, 5 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : on Nokia you have to sort the entries basing on subinterface and the retrieve the L3VPN attached

A:leaf2# show arpnd arp-entries subinterface 200
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 2 (0 static, 2 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Cumulus :

Here we retrieve all arp infos linked to evpn

root@leaf1-cum-dc1:/# net show evpn arp-cache vni all
VNI 100 #ARP (IPv4 and IPv6, local and remote) 2

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.1.3 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0
192.168.1.1 local active aa:c1:ab:4b:ff:71 0/0

VNI 200 #ARP (IPv4 and IPv6, local and remote) 4

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.2.1 local active aa:c1:ab:4b:ff:71 0/0
192.168.200.254 remote active 00:00:5e:00:01:01 10.255.255.8 0/0
192.168.2.254 remote active 00:00:5e:00:01:01 10.255.255.6 0/0
192.168.200.1 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0

VNI 400 #ARP (IPv4 and IPv6, local and remote) 0

-------------------------------------------
Check bgp sessions' status - underlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep ipv4
| default | 10.254.254.19 | udl-superspine | S | 65003 | established | 0d:0h:22m:44s | ipv4-unicast | [13/4/4] |
| default | 10.254.254.22 | udl-leaf | S | 65013 | established | 0d:0h:22m:52s | ipv4-unicast | [13/1/4] |

Cumulus :

root@leaf3:/# net show bgp ipv4 unicast summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 13
RIB entries 25, using 4800 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.254.254.23 4 65002 466 403 0 0 0 00:19:34 4 13
10.254.254.25 4 65002 434 355 0 0 0 00:17:10 8 13

-------------------------------------------
Check bgp sessions' status - overlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep evpn
| default | 10.255.255.7 | ovl-leaf | S | 65013 | established | 0d:0h:24m:14s| evpn | [15/0/11] |
| default | 10.255.255.9 | ovl-superspine | S | 65003 | established | 0d:0h:24m:7s | evpn | [11/0/4] |

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.255.255.3 4 65002 5618 4742 0 0 0 03:53:35 19 27
10.255.255.4 4 65002 5565 4694 0 0 0 03:51:11 19 27

-------------------------------------------
Check EVPN L2 tables///
-------------------------------------------

Nokia :

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn route type 2
BGP table version is 17, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:100
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
Route Distinguisher: 10.255.255.5:200
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
Route Distinguisher: 10.255.255.6:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
Route Distinguisher: 10.255.255.7:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100
Route Distinguisher: 10.255.255.7:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.200.1]
10.255.255.7 32768 i
ET:8 RT:65000:200
Route Distinguisher: 10.255.255.8:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC

and if we focus on our vni 100

root@leaf3-cum-dc1:/# net show bgp evpn route vni 100 type 2
BGP table version is 48, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100

The additional route type "MAC+IP" is optional and is useful/mandatory when you do anycast irb.

-------------------------------------------
Check tables EVPN L3 ///
-------------------------------------------

Nokia :

--{ + running }--[ ]--
A:leaf2# show network-instance ip-vrf-200 route-table all
-----------------------------------------------------------------------------------------------------------------------------------------------------------
IPv4 Unicast route table of network instance ip-vrf-200
-----------------------------------------------------------------------------------------------------------------------------------------------------------
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
| Prefix | ID | Route Type | Route Owner | Best/Fib- | Metric | Pref | Next-hop | Next-hop |
| | | | | status(slot) | | | (Type) | Interface |
+=======================+=======+============+======================+======================+==========+=========+==============+==============+
| 192.168.2.0/24 | 9 | local | net_inst_mgr | True/success | 0 | 0 | 192.168.2.25 | irb0.200 |
| | | | | | | | 4 (direct) | |
| 192.168.2.254/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (extract) | |
| 192.168.2.255/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (broadcast) | |
| 192.168.200.0/24 | 0 | bgp-evpn | bgp_evpn_mgr | True/success | 0 | 170 | 10.255.255.8 | None |
| | | | | | | | (indirect/vx | |
| | | | | | | | lan) | |
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
-----------------------------------------------------------------------------------------------------------------------------------------------------------
4 IPv4 routes total
4 IPv4 prefixes with active routes
0 IPv4 prefixes with active ECMP routes
-----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--
A:leaf2#

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf1-cum-dc1:/# net show route vrf ip-vrf-200
show ip route vrf ip-vrf-200
=============================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF ip-vrf-200:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 04:48:42
B 192.168.2.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 04:27:26
C>* 192.168.2.0/24 is directly connected, vlan200, 04:48:42
B>* 192.168.200.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:48:49

root@leaf1-cum-dc1:/# net show bgp evpn route type 5
BGP table version is 3, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.6:10200
* [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
*> [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
Route Distinguisher: 10.255.255.8:10200
*> [5]:[0]:[24]:[192.168.200.0]
10.255.255.8 0 65001 65003 65002 65014 i
RT:65000:10200 ET:8 Rmac:02:f8:b8:ff:00:00

-------------------------------------------
Check arp hosts ///
-------------------------------------------

From client #1 /

bash-5.0# arp
Address HWtype HWaddress Flags Mask Iface
192.168.1.3 ether aa:c1:ab:b5:79:50 C eth1.100
192.168.1.4 ether aa:c1:ab:0c:60:d6 C eth1.100
192.168.2.2 ether aa:c1:ab:23:31:04 C eth1.200
192.168.2.254 ether aa:c1:ab:4f:f2:e7 C eth1.200
192.168.1.2 ether aa:c1:ab:23:31:04 C eth1.100
containerlab.local ether 02:42:63:9e:74:03 C eth0

-------------------------------------------
Ping ///
-------------------------------------------

From client #1 /

L2 service ping (vlan 100) ----------------------

bash-5.0# ping -c 1 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.812 ms

--- 192.168.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.812/0.812/0.812/0.000 MsgRcvd
bash-5.0# ping -c 1 192.168.1.3
PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_seq=1 ttl=64 time=1.40 ms

--- 192.168.1.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.403/1.403/1.403/0.000 ms
bash-5.0# ping -c 1 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=1.62 ms

--- 192.168.1.4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.619/1.619/1.619/0.000 ms

L3 service ping (vlan 200)

bash-5.0# ping -c 1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.053 ms

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms
bash-5.0# ^C
bash-5.0# ping -c 1 192.168.2.2
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.611 ms

--- 192.168.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.611/0.611/0.611/0.000 ms

-------------------------------------------
Traceroute ///
-------------------------------------------

L3 service traceroute (vlan 200)

bash-5.0# traceroute 192.168.200.1
traceroute to 192.168.200.1 (192.168.200.1), 30 hops max, 46 byte packets
1 192.168.2.254 (192.168.2.254) 0.012 ms 0.010 ms 0.007 ms
2 192.168.200.254 (192.168.200.254) 2.751 ms * 2.549 ms
3 192.168.200.1 (192.168.200.1) 2.084 ms 1.805 ms 1.761 ms

</syntaxhighlight>To come back on the setup, we wanted to implement solutions inspired by the <nowiki>RFC 7938</nowiki> rfc7938 (see section 5.2.1)

* eBGP will be used for Underlay and Overlay
* eBGP single-hop sessions are established over direct point-to-point links interconnecting the network nodes, no multi-hop or loopback sessions are used, even in the case of multiple links between the same pair of nodes.
* Private Use ASNs from the range 64512-65534 are used to avoid ASN conflicts.
* A single ASN is allocated to all of the Clos topology's Tier 1 devices.
* A unique ASN is allocated to each set of Tier 2 devices in the same cluster.
* A unique ASN is allocated to every Tier 3 device (e.g., ToR) in this topology.

In add, depending of network nodes, it will be interesting to :

* Add fast-failover option for underlay + add link-debounce on links to limit possible impacts in case of link flaps. BFD could be considered in some cases
* Have the best ECMP hashing algorythm on leaf nodes. On tier-1/tier-2 levels hashing will be based on underlay part (trafic inter-vtep), so L3-L4 decision, but some models can retreive the vni-id if I well remember.
* Implement the "multipath relax" since we could have multiple AS paths for a same destination.
* Implement gshut community on underlay (on tier-1/tier-2) or at least policies to prepend underlay routes to avoid some nodes in case of maintenance.

== Thoughts/Points to deepend ==
On leaf nodes, LACP over multiple nodes is feasible within ESI, but impact on incidents/maintenance can vary depending on the vendors.

'''Tests need to be done to check that.'''

Some sysadmin would like to KISS and have client servers connected in active/passive bonding, with eventually single-leaf ports aggregation.

'''Note :''' NH ESI in FIB management on ingress nodes can have some limitations (numbers of installable next-hop, load-balancing, etc. oftenly linked to the node chipset)

Another important engineering point to define at the beginning is : '''Do we authorize multisite L2 propagation or not ?'''

From my point-of-view, the smaller are the L2 segments the better it is. I would suggest Local L2 segments wit anycast gateways provided by Leaf switches, with L3 routing between DCs within EVPN type5 routes sharing only. Nevertheless, some business requirements for endcustomers can keep extended L2 alive.

We have to be careful when implementing Consistent hashing. ex: <nowiki>https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-44/Layer-3/Routing/Equal-Cost-Multipath-Load-Sharing-Hardware-ECMP/</nowiki>

If possible activate it per VRF, not globally, and restrict on dedicated subnets << depends of the vendor, it can be dangerous regarding memory management

Think at configuring the bgp option to not advertise routes to peers before they are installed in FIB.

We can implement leaf roles in some cases, to be able to do "light traffic engineering " on DCI links in some setup.

These roles will be managed within bgp communities added to VTEP /32. Look at the diagram below. Depending of the colour, the eBGP sessions between SupS will prepend or not the VTEP IPs announced to the remote DC

[[File:Fabric-roles.jpg|ohne|mini|1200x1200px]]

<blockquote>'''To be continued'''</blockquote>

== Lab archives ==
<blockquote>Reminder : following config are for lab, so please don't pay attention on typos/lack of security params/unused config. Thanks \o/</blockquote>

=== Credentials ===
<syntaxhighlight lang="bash">
SRLinux : admin/<no_password>

CVX : cumulus/cumulus
</syntaxhighlight>

=== Lab archives ===
layer 2 (vlan 100) + layer 3 (vlan 200) lab, 10/12/2021
[[https://wiki.nesevo.com/images/6/66/20211008_lab-nokia-cvx.zip 20211008_lab-nokia-cvx.zip]]
<u>If you want to exchange about that lab, those topics, please don't hesitate to contact us</u>

Nesevo Virtual Labs

2021-10-29T13:44:35Z

PierreLancastre:

[[File:nesevo-labs-v2.png]]

Welcome to our Labs Wiki page.

Here you'll find our shares about what we've experimented in lab. Results / observations could change today with latest versions and our knowledge improvement.

Don't hesitate to share your feedbacks, thanks.

= Nokia/Cumulus ContainerLab =

[[Cumulus-Nokia-Lab]]

= SONiC with Docker =

[[Sonic-lab]]

= M-Vendor VXLAN Lab with EVE-NG =

[[Mvendor-VXLAN-lab]]

Mvendor-VXLAN-lab

2021-10-29T13:38:22Z

PierreLancastre:

= Introduction =

These labs were realized end of 2020. So different behaviors could be observed today with last releases, in a good or a bad way.

Archives posted are not production-ready and can carry unused/incomplete config parts

Analyzes and conclusions are not an absolute truth. So if you have remarks/advices, please share :)

== <span>Lab Target</span> ==

The target of the lab is to build a multivendor lab, with an architecture reflecting those deployed in customers' environment.

This lab will be constructed in several steps, starting in a virtual version, followed by a physical one.

For the moment, we are working on typical "modern" datacenters implementations :

* EVPN/VXLAN layer 2 only
* EVPN/VXLAN layer 2 + layer 3 per rack (ou layer 3 anycast gw)
* EVPN within MPLS en L2/L3
* "standard" IP Fabric

== <span>Host specifications</span> ==

Server : - Intel Xeon E5-1660v3 - 64GB DDR4 ECC 2133MHz - 2x HDD SATA 4TB Datacenter Class Soft RA
System : VMware ESXi 6.7 U3

Software versions :

All network devices instances run within EVE-NG software , here EVE-NG 2.0.3-110 The software has been installed with an OVF available on the website https://www.eve-ng.net/

Regarding the vendors images, they have been downloaded on the vendors portals (or provided by them directly).

* Arista : veos-4.24.1F
* Juniper MX : 20.1R1.11-limited-VCP
* Juniper QFX : 18.4R1.8
* Juniper SRX : next-gen 20.1R1.11
* Nokia : 7750 VSR-I timos-20.7R1

<span>Password used :</span><div></div>

* Nokia : admin:nokia2020
* Arista : admin: <-- no password
* Juniper : root:Juniper
* Cumulus : cumulus:CumulusLinux!

= GOAL =
The target of this lab is to put in place a setup providing a "cross-dc" switch, multi-vendor, based on EVPN/VXLAN technology.

In this lab, we have :

* Arista devices as Spine nodes
* Spine nodes carry directly DCI connections
* Local Spine share the same autonomous system
* Leaf devices are from different vendors : Juniper, Arista, Nokia and Cumulus
* attached host are single-homed and are simulated by Arista instances. Thus allowing to test N vlans at the same time
* EVPN uses VXLAN encapsulation, the control plane is BGP base. All nodes have dedicated private ASN (except for local SPINE)
* Underlay is managed by eBGP sessions, where only Loopback IPs are distributed. Those IPs are used as VTEP IPs by LEAF nodes
* Overlay is managed by eBGP multihop sessions between loopback IPs. Well think to only activate EVPN family for those sessions + define the upsate-source IP (loopback IP)

== Configurations + diagrams ==

[[File:Lab-mvendor-vxlan-l2-nok-ars-jun-cum.jpeg]]
[[https://wiki.nesevo.com/images/9/96/Mvendor-lab-4_vendors.zip Attach:Mvendor-lab-4_vendors.zip]]

[[File:L2-L3-mvendor-vxlan-direct-dci.png]]
[[https://wiki.nesevo.com/images/3/38/Mvendor-lab_5-vendors.zip Attach:Mvendor-lab_5-vendors.zip]]

== Observations ==
Juniper vQFX supports only vlan-aware-bundle service in VXLAN (one EVPN instance for N VLAN)

On Arista you can choose either use VLAN-BASED or VLAN-AWARE-BUNDLE

<syntaxhighlight lang="bash">
vlan 100
rd 10.255.255.5:1
route-target both 65000:100
redistribute learned
</syntaxhighlight>
or
<syntaxhighlight lang="bash">
vlan-aware-bundle LAB
rd 10.255.255.5:1
route-target both 65000:100
redistribute learned
vlan 100
</syntaxhighlight>

On Nokia I've only found the way to do vlan-based.

It can be seen in type-routes (ether tag id part in yellow bellow) :

Arista vlan-based :

<syntaxhighlight lang="bash">
{master:0}
root@QFX_DC1> show route | match 2:10.255.255.5
2:10.255.255.5:1::0::50:01:00:3e:d9:dc/304 MAC/IP
2:10.255.255.5:1::0::50:01:00:3e:d9:dc/304 MAC/IP
</syntaxhighlight>

Arista vlan-bundle (the ::0:: is replaced by ::100::):
<syntaxhighlight lang="bash">
{master:0}
root@QFX_DC1> show route | match 2:10.255.255.8
2:10.255.255.8:1::100::50:01:00:cd:ff:68/304 MAC/IP
2:10.255.255.8:1::100::50:01:00:cd:ff:68/304 MAC/IP
</syntaxhighlight>

Juniper vlan-bundle :

<syntaxhighlight lang="bash">
{master:0}
root@QFX_DC1> show route | match 2:10.255.255.10
2:10.255.255.10:1::100::50:01:00:36:5f:a1/304 MAC/IP
2:10.255.255.10:1::100::50:01:00:36:5f:a1/304 MAC/IP
</syntaxhighlight>

Nokia vlan based :

<syntaxhighlight lang="bash">
{master:0}
root@QFX_DC1> show route | match 2:10.255.255.9

2:10.255.255.9:100::0::50:01:00:06:09:e7/304 MAC/IP<div></div>
</syntaxhighlight>

So, the border effect is that on Arista devices, fdb install routes basing on ether tag id. Juniper doesn't do that (as Nokia)

Juniper DC1 output

<syntaxhighlight lang="bash">

{master:0}
root@QFX_DC1> show ethernet-switching table
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static<div></div>
SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
Ethernet switching table : 6 entries, 6 learned
Routing instance : default-switch<div></div>
Vlan MAC MAC Logical Active
name address flags interface source
VL_100 50:01:00:06:09:e7 D vtep.32770 10.255.255.9 NOKIA-DC2
VL_100 50:01:00:36:5f:a1 D vtep.32773 10.255.255.10 JUNIPER-DC2
VL_100 50:01:00:3e:d9:dc D vtep.32769 10.255.255.5 ARISTA-DC1
VL_100 50:01:00:65:ae:e8 D vtep.32772 10.255.255.6 NOKIA-DC1
VL_100 50:01:00:96:6b:29 D xe-0/0/3.0
VL_100 50:01:00:cd:ff:68 D vtep.32771 10.255.255.8 ARISTA-DC2

Nokia DC1 output

<syntaxhighlight lang="bash">
Forwarding Database, Service 100 ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age

----100 50:01:00:06:09:e7 vxlan-1: Evpn 10/15/20 08:50:03 NOKIA-DC2<div></div>
10.255.255.9:100
100 50:01:00:36:5f:a1 vxlan-1: Evpn 10/15/20 08:50:03 JUNIPER-DC2<div></div>
10.255.255.10:100
100 50:01:00:3e:d9:dc vxlan-1: Evpn 10/15/20 09:16:27 ARISTA-DC1<div></div>
10.255.255.5:100
100 50:01:00:65:ae:e8 sap:1/1/c2/1:100 L/150 10/13/20 16:51:07

100 50:01:00:96:6b:29 vxlan-1: Evpn 10/15/20 08:50:03 JUNIPER-DC1<div></div>
10.255.255.7:100
100 50:01:00:cd:ff:68 vxlan-1: Evpn 10/15/20 08:51:55 ARISTA-DC2<div></div>
10.255.255.8:100

</syntaxhighlight>

Arista DC2 output

vlan-aware bundle :

<syntaxhighlight lang="bash">
LEAF-ARS-DC2#show mac address-table
Mac Address Table
<div></div>
----Vlan Mac Address Type Ports Moves Last Move<div></div>
--------------- ---- ----- ----- ---------<div></div>
100 5001.0036.5fa1 DYNAMIC Vx1 1 0:37:45 ago JUNIPER-DC2

100 5001.003e.d9dc DYNAMIC Vx1 1 0:13:14 ago ARISTA-DC1

100 5001.0096.6b29 DYNAMIC Vx1 1 0:37:45 ago JUNIPER-DC1

100 5001.00cd.ff68 DYNAMIC Et2 1 1 day, 19:22:18 ago
</syntaxhighlight>

Only mac @ for Arista / Juniper are installed even the Nokia routes are well received

<syntaxhighlight lang="bash">
LEAF-ARS-DC2# show bgp evpn rd 10.255.255.6:100

BGP routing table information for VRF default

Router identifier 10.255.255.8, local AS number 65021

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP<div></div>
S - Stale, c - Contributing to ECMP, b - backup

% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop<div></div>
Network Next Hop Metric AIGP LocPref Weight Path

* >Ec RD: 10.255.255.6:100 mac-ip 5001.0065.aee8

10.255.255.6 - 100 0 65002 65001 65012 i

* ec RD: 10.255.255.6:100 mac-ip 5001.0065.aee8

10.255.255.6 - 100 0 65002 65001 65012 i

* >Ec RD: 10.255.255.6:100 imet 10.255.255.6

10.255.255.6 - 100 0 65002 65001 65012 i

* ec RD: 10.255.255.6:100 imet 10.255.255.6

10.255.255.6 - 100 0 65002 65001 65012 i
</syntaxhighlight>

compared to Juniper routes for example :

<syntaxhighlight lang="bash">
LEAF-ARS-DC2# show bgp evpn rd 10.255.255.7:1

BGP routing table information for VRF default

Router identifier 10.255.255.8, local AS number 65021

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP<div></div>
S - Stale, c - Contributing to ECMP, b - backup

% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop<div></div>
Network Next Hop Metric AIGP LocPref Weight Path

* >Ec RD: 10.255.255.7:1 mac-ip 100 5001.0096.6b29

10.255.255.7 - 100 0 65002 65001 65013 i
[...]
</syntaxhighlight>

Ether tag id makes the difference for me.</div>

So, since no type 2 routes is received, the packets from Arista to Nokia are processed as BUM (so sent to all VTEP from the VXLAN instance).

Some tests wee done with L3 routing, details coming

Mvendor-VXLAN-lab

2021-10-29T13:38:05Z

PierreLancastre:

Mvendor-VXLAN-lab

2021-10-29T13:35:59Z

PierreLancastre:

= Introduction =

These labs were realized end of 2020. So different behaviors could be observed today with last releases, in a good or a bad way.

Archives posted are not production-ready and can carry unused/incomplete config parts

Analyzes and conclusions are not an absolute truth. So if you have remarks/advices, please share :)

== <span>Lab Target</span> ==

The target of the lab is to build a multivendor lab, with an architecture reflecting those deployed in customers' environment.

This lab will be constructed in several steps, starting in a virtual version, followed by a physical one.

For the moment, we are working on typical "modern" datacenters implementations :

* EVPN/VXLAN layer 2 only
* EVPN/VXLAN layer 2 + layer 3 per rack (ou layer 3 anycast gw)
* EVPN within MPLS en L2/L3
* "standard" IP Fabric

== <span>Host specifications</span> ==

Server : - Intel Xeon E5-1660v3 - 64GB DDR4 ECC 2133MHz - 2x HDD SATA 4TB Datacenter Class Soft RA
System : VMware ESXi 6.7 U3

Software versions :

All network devices instances run within EVE-NG software , here EVE-NG 2.0.3-110 The software has been installed with an OVF available on the website https://www.eve-ng.net/

Regarding the vendors images, they have been downloaded on the vendors portals (or provided by them directly).

* Arista : veos-4.24.1F
* Juniper MX : 20.1R1.11-limited-VCP
* Juniper QFX : 18.4R1.8
* Juniper SRX : next-gen 20.1R1.11
* Nokia : 7750 VSR-I timos-20.7R1

<span>Password used :</span><div></div>

* Nokia : admin:nokia2020
* Arista : admin: <-- no password
* Juniper : root:Juniper
* Cumulus : cumulus:CumulusLinux!

= GOAL =
The target of this lab is to put in place a setup providing a "cross-dc" switch, multi-vendor, based on EVPN/VXLAN technology.

In this lab, we have :

* Arista devices as Spine nodes
* Spine nodes carry directly DCI connections
* Local Spine share the same autonomous system
* Leaf devices are from different vendors : Juniper, Arista, Nokia and Cumulus
* attached host are single-homed and are simulated by Arista instances. Thus allowing to test N vlans at the same time
* EVPN uses VXLAN encapsulation, the control plane is BGP base. All nodes have dedicated private ASN (except for local SPINE)
* Underlay is managed by eBGP sessions, where only Loopback IPs are distributed. Those IPs are used as VTEP IPs by LEAF nodes
* Overlay is managed by eBGP multihop sessions between loopback IPs. Well think to only activate EVPN family for those sessions + define the upsate-source IP (loopback IP)

= Configurations + diagrams =

[[File:Lab-mvendor-vxlan-l2-nok-ars-jun-cum.jpeg]]
[[https://wiki.nesevo.com/images/9/96/Mvendor-lab-4_vendors.zip Attach:Mvendor-lab-4_vendors.zip]]

[[File:L2-L3-mvendor-vxlan-direct-dci.png]]
[[https://wiki.nesevo.com/images/3/38/Mvendor-lab_5-vendors.zip Attach:Mvendor-lab_5-vendors.zip]]

== Note ==
Juniper vQFX supports only vlan-aware-bundle service in VXLAN (one EVPN instance for N VLAN)

On Arista you can choose either use VLAN-BASED or VLAN-AWARE-BUNDLE

<syntaxhighlight lang="bash">
vlan 100
rd 10.255.255.5:1
route-target both 65000:100
redistribute learned
</syntaxhighlight>
or
<syntaxhighlight lang="bash">
vlan-aware-bundle LAB
rd 10.255.255.5:1
route-target both 65000:100
redistribute learned
vlan 100
</syntaxhighlight>

On Nokia I've only found the way to do vlan-based.

It can be seen in type-routes (ether tag id part in yellow bellow) :

Arista vlan-based :

<syntaxhighlight lang="bash">
{master:0}
root@QFX_DC1> show route | match 2:10.255.255.5
2:10.255.255.5:1::0::50:01:00:3e:d9:dc/304 MAC/IP
2:10.255.255.5:1::0::50:01:00:3e:d9:dc/304 MAC/IP
</syntaxhighlight>

Arista vlan-bundle (the ::0:: is replaced by ::100::):
<syntaxhighlight lang="bash">
{master:0}
root@QFX_DC1> show route | match 2:10.255.255.8
2:10.255.255.8:1::100::50:01:00:cd:ff:68/304 MAC/IP
2:10.255.255.8:1::100::50:01:00:cd:ff:68/304 MAC/IP
</syntaxhighlight>

Juniper vlan-bundle :

<syntaxhighlight lang="bash">
{master:0}
root@QFX_DC1> show route | match 2:10.255.255.10
2:10.255.255.10:1::100::50:01:00:36:5f:a1/304 MAC/IP
2:10.255.255.10:1::100::50:01:00:36:5f:a1/304 MAC/IP
</syntaxhighlight>

Nokia vlan based :

<syntaxhighlight lang="bash">
{master:0}
root@QFX_DC1> show route | match 2:10.255.255.9

2:10.255.255.9:100::0::50:01:00:06:09:e7/304 MAC/IP<div></div>
</syntaxhighlight>

So, the border effect is that on Arista devices, fdb install routes basing on ether tag id. Juniper doesn't do that (as Nokia)

Juniper DC1 output

<syntaxhighlight lang="bash">

{master:0}
root@QFX_DC1> show ethernet-switching table
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static<div></div>
SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
Ethernet switching table : 6 entries, 6 learned
Routing instance : default-switch<div></div>
Vlan MAC MAC Logical Active
name address flags interface source
VL_100 50:01:00:06:09:e7 D vtep.32770 10.255.255.9 NOKIA-DC2
VL_100 50:01:00:36:5f:a1 D vtep.32773 10.255.255.10 JUNIPER-DC2
VL_100 50:01:00:3e:d9:dc D vtep.32769 10.255.255.5 ARISTA-DC1
VL_100 50:01:00:65:ae:e8 D vtep.32772 10.255.255.6 NOKIA-DC1
VL_100 50:01:00:96:6b:29 D xe-0/0/3.0
VL_100 50:01:00:cd:ff:68 D vtep.32771 10.255.255.8 ARISTA-DC2

Nokia DC1 output

<syntaxhighlight lang="bash">
Forwarding Database, Service 100 ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age

----100 50:01:00:06:09:e7 vxlan-1: Evpn 10/15/20 08:50:03 NOKIA-DC2<div></div>
10.255.255.9:100
100 50:01:00:36:5f:a1 vxlan-1: Evpn 10/15/20 08:50:03 JUNIPER-DC2<div></div>
10.255.255.10:100
100 50:01:00:3e:d9:dc vxlan-1: Evpn 10/15/20 09:16:27 ARISTA-DC1<div></div>
10.255.255.5:100
100 50:01:00:65:ae:e8 sap:1/1/c2/1:100 L/150 10/13/20 16:51:07

100 50:01:00:96:6b:29 vxlan-1: Evpn 10/15/20 08:50:03 JUNIPER-DC1<div></div>
10.255.255.7:100
100 50:01:00:cd:ff:68 vxlan-1: Evpn 10/15/20 08:51:55 ARISTA-DC2<div></div>
10.255.255.8:100

</syntaxhighlight>

Arista DC2 output

vlan-aware bundle :

<syntaxhighlight lang="bash">
LEAF-ARS-DC2#show mac address-table
Mac Address Table
<div></div>
----Vlan Mac Address Type Ports Moves Last Move<div></div>
--------------- ---- ----- ----- ---------<div></div>
100 5001.0036.5fa1 DYNAMIC Vx1 1 0:37:45 ago JUNIPER-DC2

100 5001.003e.d9dc DYNAMIC Vx1 1 0:13:14 ago ARISTA-DC1

100 5001.0096.6b29 DYNAMIC Vx1 1 0:37:45 ago JUNIPER-DC1

100 5001.00cd.ff68 DYNAMIC Et2 1 1 day, 19:22:18 ago
</syntaxhighlight>

Only mac @ for Arista / Juniper are installed even the Nokia routes are well received

<syntaxhighlight lang="bash">
LEAF-ARS-DC2# show bgp evpn rd 10.255.255.6:100

BGP routing table information for VRF default

Router identifier 10.255.255.8, local AS number 65021

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP<div></div>
S - Stale, c - Contributing to ECMP, b - backup

% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop<div></div>
Network Next Hop Metric AIGP LocPref Weight Path

* >Ec RD: 10.255.255.6:100 mac-ip 5001.0065.aee8

10.255.255.6 - 100 0 65002 65001 65012 i

* ec RD: 10.255.255.6:100 mac-ip 5001.0065.aee8

10.255.255.6 - 100 0 65002 65001 65012 i

* >Ec RD: 10.255.255.6:100 imet 10.255.255.6

10.255.255.6 - 100 0 65002 65001 65012 i

* ec RD: 10.255.255.6:100 imet 10.255.255.6

10.255.255.6 - 100 0 65002 65001 65012 i
</syntaxhighlight>

compared to Juniper routes for example :

<syntaxhighlight lang="bash">
LEAF-ARS-DC2# show bgp evpn rd 10.255.255.7:1

BGP routing table information for VRF default

Router identifier 10.255.255.8, local AS number 65021

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP<div></div>
S - Stale, c - Contributing to ECMP, b - backup

% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop<div></div>
Network Next Hop Metric AIGP LocPref Weight Path

* >Ec RD: 10.255.255.7:1 mac-ip 100 5001.0096.6b29

10.255.255.7 - 100 0 65002 65001 65013 i
[...]
</syntaxhighlight>

Ether tag id makes the difference for me.</div>

So, since no type 2 routes is received, the packets from Arista to Nokia are processed as BUM (so sent to all VTEP from the VXLAN instance).

Below the sources of different versions of the lab :

File:L2-L3-mvendor-vxlan-direct-dci.png

2021-10-29T13:29:33Z

PierreLancastre:

File:Lab-mvendor-vxlan-l2-nok-ars-jun-cum.jpeg

2021-10-29T13:28:54Z

PierreLancastre:

File:Mvendor-lab 5-vendors.zip

2021-10-29T13:28:31Z

PierreLancastre:

File:Mvendor-lab-4 vendors.zip

2021-10-29T13:27:59Z

PierreLancastre:

Mvendor-VXLAN-lab

2021-10-29T13:26:26Z

PierreLancastre:

= Introduction =

These labs were realized end of 2020. So different behaviors could be observed today with last releases, in a good or a bad way.

== <span>Lab Target</span> ==

The target of the lab is to build a multivendor lab, with an architecture reflecting those deployed in customers' environment.

This lab will be constructed in several steps, starting in a virtual version, followed by a physical one.

For the moment, we are working on typical "modern" datacenters implementations :

* EVPN/VXLAN layer 2 only
* EVPN/VXLAN layer 2 + layer 3 per rack (ou layer 3 anycast gw)
* EVPN within MPLS en L2/L3
* "standard" IP Fabric

== <span>Host specifications</span> ==

Server : - Intel Xeon E5-1660v3 - 64GB DDR4 ECC 2133MHz - 2x HDD SATA 4TB Datacenter Class Soft RA
System : VMware ESXi 6.7 U3

Software versions :

All network devices instances run within EVE-NG software , here EVE-NG 2.0.3-110 The software has been installed with an OVF available on the website https://www.eve-ng.net/

Regarding the vendors images, they have been downloaded on the vendors portals (or provided by them directly).

* Arista : veos-4.24.1F
* Juniper MX : 20.1R1.11-limited-VCP
* Juniper QFX : 18.4R1.8
* Juniper SRX : next-gen 20.1R1.11
* Nokia : 7750 VSR-I timos-20.7R1

<span>Password used :</span><div></div>

* Nokia : admin:nokia2020
* Arista : admin: <-- no password
* Juniper : root:Juniper
* Cumulus : cumulus:CumulusLinux!

= GOAL =
The target of this lab is to put in place a setup providing a "cross-dc" switch, multi-vendor, based on EVPN/VXLAN technology.

In this lab, we have :

* Arista devices as Spine nodes
* Spine nodes carry directly DCI connections
* Local Spine share the same autonomous system
* Leaf devices are from different vendors : Juniper, Arista, Nokia and Cumulus
* attached host are single-homed and are simulated by Arista instances. Thus allowing to test N vlans at the same time
* EVPN uses VXLAN encapsulation, the control plane is BGP base. All nodes have dedicated private ASN (except for local SPINE)
* Underlay is managed by eBGP sessions, where only Loopback IPs are distributed. Those IPs are used as VTEP IPs by LEAF nodes
* Overlay is managed by eBGP multihop sessions between loopback IPs. Well think to only activate EVPN family for those sessions + define the upsate-source IP (loopback IP)

= Configurations + diagrams =

== Note ==
Juniper vQFX supports only vlan-aware-bundle service in VXLAN (one EVPN instance for N VLAN)

On Arista you can choose either use VLAN-BASED or VLAN-AWARE-BUNDLE

<syntaxhighlight lang="bash">
vlan 100
rd 10.255.255.5:1
route-target both 65000:100
redistribute learned
</syntaxhighlight>
or
<syntaxhighlight lang="bash">
vlan-aware-bundle LAB
rd 10.255.255.5:1
route-target both 65000:100
redistribute learned
vlan 100
</syntaxhighlight>

On Nokia I've only found the way to do vlan-based.

It can be seen in type-routes (ether tag id part in yellow bellow) :

Arista vlan-based :

<syntaxhighlight lang="bash">
{master:0}
root@QFX_DC1> show route | match 2:10.255.255.5
2:10.255.255.5:1::0::50:01:00:3e:d9:dc/304 MAC/IP
2:10.255.255.5:1::0::50:01:00:3e:d9:dc/304 MAC/IP
</syntaxhighlight>

Arista vlan-bundle (the ::0:: is replaced by ::100::):
<syntaxhighlight lang="bash">
{master:0}
root@QFX_DC1> show route | match 2:10.255.255.8
2:10.255.255.8:1::100::50:01:00:cd:ff:68/304 MAC/IP
2:10.255.255.8:1::100::50:01:00:cd:ff:68/304 MAC/IP
</syntaxhighlight>

Juniper vlan-bundle :

<syntaxhighlight lang="bash">
{master:0}
root@QFX_DC1> show route | match 2:10.255.255.10
2:10.255.255.10:1::100::50:01:00:36:5f:a1/304 MAC/IP
2:10.255.255.10:1::100::50:01:00:36:5f:a1/304 MAC/IP
</syntaxhighlight>

Nokia vlan based :

<syntaxhighlight lang="bash">
{master:0}
root@QFX_DC1> show route | match 2:10.255.255.9

2:10.255.255.9:100::0::50:01:00:06:09:e7/304 MAC/IP<div></div>
</syntaxhighlight>

So, the border effect is that on Arista devices, fdb install routes basing on ether tag id. Juniper doesn't do that (as Nokia)

Juniper DC1 output

<syntaxhighlight lang="bash">

{master:0}
root@QFX_DC1> show ethernet-switching table
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static<div></div>
SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
Ethernet switching table : 6 entries, 6 learned
Routing instance : default-switch<div></div>
Vlan MAC MAC Logical Active
name address flags interface source
VL_100 50:01:00:06:09:e7 D vtep.32770 10.255.255.9 NOKIA-DC2
VL_100 50:01:00:36:5f:a1 D vtep.32773 10.255.255.10 JUNIPER-DC2
VL_100 50:01:00:3e:d9:dc D vtep.32769 10.255.255.5 ARISTA-DC1
VL_100 50:01:00:65:ae:e8 D vtep.32772 10.255.255.6 NOKIA-DC1
VL_100 50:01:00:96:6b:29 D xe-0/0/3.0
VL_100 50:01:00:cd:ff:68 D vtep.32771 10.255.255.8 ARISTA-DC2

Nokia DC1 output

<syntaxhighlight lang="bash">
Forwarding Database, Service 100 ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age

----100 50:01:00:06:09:e7 vxlan-1: Evpn 10/15/20 08:50:03 NOKIA-DC2<div></div>
10.255.255.9:100
100 50:01:00:36:5f:a1 vxlan-1: Evpn 10/15/20 08:50:03 JUNIPER-DC2<div></div>
10.255.255.10:100
100 50:01:00:3e:d9:dc vxlan-1: Evpn 10/15/20 09:16:27 ARISTA-DC1<div></div>
10.255.255.5:100
100 50:01:00:65:ae:e8 sap:1/1/c2/1:100 L/150 10/13/20 16:51:07

100 50:01:00:96:6b:29 vxlan-1: Evpn 10/15/20 08:50:03 JUNIPER-DC1<div></div>
10.255.255.7:100
100 50:01:00:cd:ff:68 vxlan-1: Evpn 10/15/20 08:51:55 ARISTA-DC2<div></div>
10.255.255.8:100

</syntaxhighlight>

Arista DC2 output

vlan-aware bundle :

<syntaxhighlight lang="bash">
LEAF-ARS-DC2#show mac address-table
Mac Address Table
<div></div>
----Vlan Mac Address Type Ports Moves Last Move<div></div>
--------------- ---- ----- ----- ---------<div></div>
100 5001.0036.5fa1 DYNAMIC Vx1 1 0:37:45 ago JUNIPER-DC2

100 5001.003e.d9dc DYNAMIC Vx1 1 0:13:14 ago ARISTA-DC1

100 5001.0096.6b29 DYNAMIC Vx1 1 0:37:45 ago JUNIPER-DC1

100 5001.00cd.ff68 DYNAMIC Et2 1 1 day, 19:22:18 ago
</syntaxhighlight>

Only mac @ for Arista / Juniper are installed even the Nokia routes are well received

<syntaxhighlight lang="bash">
LEAF-ARS-DC2# show bgp evpn rd 10.255.255.6:100

BGP routing table information for VRF default

Router identifier 10.255.255.8, local AS number 65021

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP<div></div>
S - Stale, c - Contributing to ECMP, b - backup

% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop<div></div>
Network Next Hop Metric AIGP LocPref Weight Path

* >Ec RD: 10.255.255.6:100 mac-ip 5001.0065.aee8

10.255.255.6 - 100 0 65002 65001 65012 i

* ec RD: 10.255.255.6:100 mac-ip 5001.0065.aee8

10.255.255.6 - 100 0 65002 65001 65012 i

* >Ec RD: 10.255.255.6:100 imet 10.255.255.6

10.255.255.6 - 100 0 65002 65001 65012 i

* ec RD: 10.255.255.6:100 imet 10.255.255.6

10.255.255.6 - 100 0 65002 65001 65012 i
</syntaxhighlight>

compared to Juniper routes for example :

<syntaxhighlight lang="bash">
LEAF-ARS-DC2# show bgp evpn rd 10.255.255.7:1

BGP routing table information for VRF default

Router identifier 10.255.255.8, local AS number 65021

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP<div></div>
S - Stale, c - Contributing to ECMP, b - backup

% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop<div></div>
Network Next Hop Metric AIGP LocPref Weight Path

* >Ec RD: 10.255.255.7:1 mac-ip 100 5001.0096.6b29

10.255.255.7 - 100 0 65002 65001 65013 i
[...]
</syntaxhighlight>

Ether tag id makes the difference for me.</div>

So, since no type 2 routes is received, the packets from Arista to Nokia are processed as BUM (so sent to all VTEP from the VXLAN instance).

Below the sources of different versions of the lab :

Mvendor-VXLAN-lab

2021-10-29T13:21:49Z

PierreLancastre:

= Introduction =

This lab was realized early 2021. So different behaviors could be observed today with last releases, in a good or a bad way.

== <span>Lab Target</span> ==

The target of the lab is to build a multivendor lab, with an architecture reflecting those deployed in customers' environment.

This lab will be constructed in several steps, starting in a virtual version, followed by a physical one.

For the moment, we are working on typical "modern" datacenters implementations :

* EVPN/VXLAN layer 2 only
* EVPN/VXLAN layer 2 + layer 3 per rack (ou layer 3 anycast gw)
* EVPN within MPLS en L2/L3
* "standard" IP Fabric

== <span>Host specifications</span> ==

Server : - Intel Xeon E5-1660v3 - 64GB DDR4 ECC 2133MHz - 2x HDD SATA 4TB Datacenter Class Soft RA
System : VMware ESXi 6.7 U3

Software versions :<

All network devices instances run within EVE-NG software , here EVE-NG 2.0.3-110 The software has been installed with an OVF available on the website https://www.eve-ng.net/

Regarding the vendors images, they have been downloaded on the vendors portals (or provided by them directly).

* Arista : veos-4.24.1F
* Juniper MX : 20.1R1.11-limited-VCP
* Juniper QFX : 18.4R1.8
* Juniper SRX : next-gen 20.1R1.11
* Nokia : 7750 VSR-I timos-20.7R1

<span>Password used :</span><div></div>

* Nokia : admin:nokia2020
* Arista : admin: <-- no password
* Juniper : root:Juniper
* Cumulus : cumulus:CumulusLinux!

= GOAL =
The target of this lab is to put in place a setup providing a "cross-dc" switch, multi-vendor, based on EVPN/VXLAN technology.

In this lab, we have :

* Arista devices as Spine nodes
* Spine nodes carry directly DCI connections
* Local Spine share the same autonomous system
* Leaf devices are from different vendors : Juniper, Arista, Nokia and Cumulus
* attached host are single-homed and are simulated by Arista instances. Thus allowing to test N vlans at the same time
* EVPN uses VXLAN encapsulation, the control plane is BGP base. All nodes have dedicated private ASN (except for local SPINE)
* Underlay is managed by eBGP sessions, where only Loopback IPs are distributed. Those IPs are used as VTEP IPs by LEAF nodes
* Overlay is managed by eBGP multihop sessions between loopback IPs. Well think to only activate EVPN family for those sessions + define the upsate-source IP (loopback IP)

= LAB DIAGRAM =
[[Datei:Lab-mvendor-vxlan-l2-nok-ars-jun-cum.jpeg|klasse=ve-pasteProtect]]

= Configurations =
[[:Datei:Config-lab-mvendor-vxlan-l2-nok-ars-jun-cum.zip|Datei:config-lab-mvendor-vxlan-l2-nok-ars-jun-cum.zip]]

== Note ==
Juniper vQFX supports only vlan-aware-bundle service in VXLAN (one EVPN instance for N VLAN)

On Arista you can choose either use VLAN-BASED or VLAN-AWARE-BUNDLE

<syntaxhighlight lang="bash">
vlan 100
rd 10.255.255.5:1
route-target both 65000:100
redistribute learned
</syntaxhighlight>
or
<syntaxhighlight lang="bash">
vlan-aware-bundle LAB
rd 10.255.255.5:1
route-target both 65000:100
redistribute learned
vlan 100
</syntaxhighlight>

On Nokia I've only found the way to do vlan-based.

It can be seen in type-routes (ether tag id part in yellow bellow) :

Arista vlan-based :

<syntaxhighlight lang="bash">
{master:0}
root@QFX_DC1> show route | match 2:10.255.255.5
2:10.255.255.5:1::0::50:01:00:3e:d9:dc/304 MAC/IP
2:10.255.255.5:1::0::50:01:00:3e:d9:dc/304 MAC/IP
</syntaxhighlight>

Arista vlan-bundle (the ::0:: is replaced by ::100::):
<syntaxhighlight lang="bash">
{master:0}
root@QFX_DC1> show route | match 2:10.255.255.8
2:10.255.255.8:1::100::50:01:00:cd:ff:68/304 MAC/IP
2:10.255.255.8:1::100::50:01:00:cd:ff:68/304 MAC/IP
</syntaxhighlight>

Juniper vlan-bundle :

<syntaxhighlight lang="bash">
{master:0}
root@QFX_DC1> show route | match 2:10.255.255.10
2:10.255.255.10:1::100::50:01:00:36:5f:a1/304 MAC/IP
2:10.255.255.10:1::100::50:01:00:36:5f:a1/304 MAC/IP
</syntaxhighlight>

Nokia vlan based :

<syntaxhighlight lang="bash">
{master:0}
root@QFX_DC1> show route | match 2:10.255.255.9

2:10.255.255.9:100::0::50:01:00:06:09:e7/304 MAC/IP<div></div>
</syntaxhighlight>

So, the border effect is that on Arista devices, fdb install routes basing on ether tag id. Juniper doesn't do that (as Nokia)

Juniper DC1 output

<syntaxhighlight lang="bash">

{master:0}
root@QFX_DC1> show ethernet-switching table
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static<div></div>
SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
Ethernet switching table : 6 entries, 6 learned
Routing instance : default-switch<div></div>
Vlan MAC MAC Logical Active
name address flags interface source
VL_100 50:01:00:06:09:e7 D vtep.32770 10.255.255.9 NOKIA-DC2
VL_100 50:01:00:36:5f:a1 D vtep.32773 10.255.255.10 JUNIPER-DC2
VL_100 50:01:00:3e:d9:dc D vtep.32769 10.255.255.5 ARISTA-DC1
VL_100 50:01:00:65:ae:e8 D vtep.32772 10.255.255.6 NOKIA-DC1
VL_100 50:01:00:96:6b:29 D xe-0/0/3.0
VL_100 50:01:00:cd:ff:68 D vtep.32771 10.255.255.8 ARISTA-DC2

Nokia DC1 output

<syntaxhighlight lang="bash">
Forwarding Database, Service 100 ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age

----100 50:01:00:06:09:e7 vxlan-1: Evpn 10/15/20 08:50:03 NOKIA-DC2<div></div>
10.255.255.9:100
100 50:01:00:36:5f:a1 vxlan-1: Evpn 10/15/20 08:50:03 JUNIPER-DC2<div></div>
10.255.255.10:100
100 50:01:00:3e:d9:dc vxlan-1: Evpn 10/15/20 09:16:27 ARISTA-DC1<div></div>
10.255.255.5:100
100 50:01:00:65:ae:e8 sap:1/1/c2/1:100 L/150 10/13/20 16:51:07

100 50:01:00:96:6b:29 vxlan-1: Evpn 10/15/20 08:50:03 JUNIPER-DC1<div></div>
10.255.255.7:100
100 50:01:00:cd:ff:68 vxlan-1: Evpn 10/15/20 08:51:55 ARISTA-DC2<div></div>
10.255.255.8:100

</syntaxhighlight>

Arista DC2 output

vlan-aware bundle :

<syntaxhighlight lang="bash">
LEAF-ARS-DC2#show mac address-table
Mac Address Table
<div></div>
----Vlan Mac Address Type Ports Moves Last Move<div></div>
--------------- ---- ----- ----- ---------<div></div>
100 5001.0036.5fa1 DYNAMIC Vx1 1 0:37:45 ago JUNIPER-DC2

100 5001.003e.d9dc DYNAMIC Vx1 1 0:13:14 ago ARISTA-DC1

100 5001.0096.6b29 DYNAMIC Vx1 1 0:37:45 ago JUNIPER-DC1

100 5001.00cd.ff68 DYNAMIC Et2 1 1 day, 19:22:18 ago
</syntaxhighlight>

Only mac @ for Arista / Juniper are installed even the Nokia routes are well received

<syntaxhighlight lang="bash">
LEAF-ARS-DC2# show bgp evpn rd 10.255.255.6:100

BGP routing table information for VRF default

Router identifier 10.255.255.8, local AS number 65021

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP<div></div>
S - Stale, c - Contributing to ECMP, b - backup

% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop<div></div>
Network Next Hop Metric AIGP LocPref Weight Path

* >Ec RD: 10.255.255.6:100 mac-ip 5001.0065.aee8

10.255.255.6 - 100 0 65002 65001 65012 i

* ec RD: 10.255.255.6:100 mac-ip 5001.0065.aee8

10.255.255.6 - 100 0 65002 65001 65012 i

* >Ec RD: 10.255.255.6:100 imet 10.255.255.6

10.255.255.6 - 100 0 65002 65001 65012 i

* ec RD: 10.255.255.6:100 imet 10.255.255.6

10.255.255.6 - 100 0 65002 65001 65012 i
</syntaxhighlight>

compared to Juniper routes for example :

<syntaxhighlight lang="bash">
LEAF-ARS-DC2# show bgp evpn rd 10.255.255.7:1

BGP routing table information for VRF default

Router identifier 10.255.255.8, local AS number 65021

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP<div></div>
S - Stale, c - Contributing to ECMP, b - backup

% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop<div></div>
Network Next Hop Metric AIGP LocPref Weight Path

* >Ec RD: 10.255.255.7:1 mac-ip 100 5001.0096.6b29

10.255.255.7 - 100 0 65002 65001 65013 i
[...]
</syntaxhighlight>

Ether tag id makes the difference for me.</div>

So, since no type 2 routes is received, the packets from Arista to Nokia are processed as BUM (so sent to all VTEP from the VXLAN instance).

Below the sources of different versions of the lab :

Mvendor-VXLAN-lab

2021-10-29T13:07:44Z

PierreLancastre: Created page with "<div> = Useful Infos = == <span>Lab Target</span> == The target of the lab is to build a multivendor lab, with an architecture reflecting those deployed in customers' environ..."

<div>
= Useful Infos =

== <span>Lab Target</span> ==
The target of the lab is to build a multivendor lab, with an architecture reflecting those deployed in customers' environment?.

This lab will be constructed in several steps, starting in a virtual version, followed by a physical one.

For the moment, we are working on typical "modern" datacenters implementations :<div></div>

* EVPN/VXLAN layer 2 only
* EVPN/VXLAN layer 2 + layer 3 per rack (ou layer 3 anycast gw)
* EVPN en MPLS en L2/L3
* "standard" IP Fabric

but there are also labs for SP customers running LDP/OSPF or SR/ISIS for EVPN/L3VPN services.

One goal is to spot/identify the (in)compatibilities between the different vendors, and build the different configurations to be able to provide "ready-to-deploy" setup. In add, working permits to securize the project of vendors migration / NOS migration (ex : EOS to Sonic for Arista).

Once the different setup consolidated, we can begin an automation work, within tools like ansible/saltstack/other, in parallel with the implementation of propietary vendor tools (like Arista Cloudvision).

The idea behing of all of that is to gain a good knwoledge of the products, wihtou necessarily being expert, but ideally with front-desk-setup, reflecting "the real life". In add, for me a work has to be led to implement a ZTP/ material validation setup, in order to improve/securize at most the deployment/assets management.

Now, the more vendors we have the best it is. We have to sollicitate the different vendors and communicate about the different setup in a "high level manner", without taking publicly position. The idea is to stay neutral whatever happens.

== <span>Host specifications</span> ==
For the moment, I'm working on an OVH dedicated server.

<span>Commercial reference :</span> RISE-LE-1 - Intel Xeon E5-1660v3 - 64GB DDR4 ECC 2133MHz - 2x HDD SATA 4TB Datacenter Class Soft RA

Système (OS) VMware ESXi 6.7 U3

<span>Software versions :</span>

All network devices instances run within EVE-NG software , here EVE-NG 2.0.3-110 The software has been installed with an OVF available on the website https://www.eve-ng.net/

Regarding the vendors images, they have been downloaded on the vendors portals (or provided by them directly). For Juniper you need a "special" account to be allowed to download the images, on Arista it's free even you are a simple user. For Nokia I don't know, but I suppose it's just in deal contexts since the licences are mandatory and need to be generated by Nokia guys.<div></div>

* Arista : veos-4.24.1F
* Juniper MX : 20.1R1.11-limited-VCP
* Juniper QFX : 18.4R1.8
* Juniper SRX : next-gen 20.1R1.11
* Nokia : 7750 VSR-I timos-20.7R1

<span>Password used :</span><div></div>

* Nokia : admin:nokia2020
* Arista : admin: <-- no password
* Juniper : root:Juniper
* Cumulus : cumulus:CumulusLinux!
</div>

Nesevo Virtual Labs

2021-10-29T13:07:25Z

PierreLancastre:

[[File:nesevo-labs-v2.png]]

Welcome to our Labs Wiki page.

Here you'll find our shares about what we've experimented in lab.

Don't hesitate to share your feedbacks, thanks.

= Nokia/Cumulus ContainerLab =

[[Cumulus-Nokia-Lab]]

= SONiC with Docker =

[[Sonic-lab]]

= M-Vendor VXLAN Lab with EVE-NG =

[[Mvendor-VXLAN-lab]]

Sonic-lab

2021-10-29T08:47:37Z

PierreLancastre:

<span><to be noted that those informations are included into the readme.md file into the archive></span>

== Lab diagram ==
[[File:SONIC_LAB.jpg]]

Lab archive [https://wiki.nesevo.com/images/0/0a/Sonic-lab-20201030.zip Sonic-lab-20201030.zip]

This lab is a forked lab from many found on @, starting from Microsoft one : https://github.com/Azure/SONiC/wiki/SONiC-P4-Software-Switch

The target of this lab is to simulate a simple IP/Fabric in SPINE/LEAF topology. Each Leaf provide local L2 connectivity + L3 gateways to hosts. Flows between hosts from different leaf are L3 routed, in a "node path" LEAF-SPINE-LEAF.

Nothing complicated, but it allows to put a first foot into SONiC world :)

For the moment, as far as we know, MC-LAG + EVPN/VXLAN features are only available on entreprise SONiC versions. Tests to come once physic switches in enterprise version will be available in our lab.

The diagram of the architecture is into the zip file (SONIC_LAB.jpg)

To run the lab you need :

* to have installed docker by using ./install_docker_ovs.sh
* to run the ./load_image.sh in order to get the Sonic image

Note : don't change file / folders names of network devices (leafX, spineX)

Once you're ready, please run the lab by launching "./start.sh"

The script :
* launches the containers without Docket network features
* create network namespaces
* build linux bridges to interconnect the containers
* configure host IPs
* launch startup sonic scripts into sonic devices
* once devices started, checks + ping tests are run. If all BGP sessions are ok on spine + all ping are fine, the setup works well.

The "start" script has several comments, perhaps without the right vocabulary, so please be tolerant :)

If you want to stop the lab, please launch "./stop.sh"

Below some useful commands to run for sonic containers :

=== <span>run ping</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> ping <ip>
</syntaxhighlight>
This command triggers ping into the targetted container :
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it leaf1 ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.052 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.034 ms
^C--- 192.168.1.1 ping statistics ---
</syntaxhighlight>

=== <span>check bgp sessions</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip bgp sum"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print the bgp status.
Below, we run it on one spine, you should have 4 sessiosn UP (for the 4 leaves)
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip bgp sum"
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 6409 6411 0 0 0 01:46:46 3
10.1.0.2 4 65102 6406 6410 0 0 0 01:46:44 3
10.1.0.4 4 65103 6407 6410 0 0 0 01:46:44 4
10.1.0.6 4 65104 6406 6412 0 0 0 01:46:46 1
Total number of neighbors 4
</syntaxhighlight>

=== <span>check ip route</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip route"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print its routing table?

<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,
> - selected route, * - FIB route
C>* 10.1.0.0/31 is directly connected, Vlan11
C>* 10.1.0.2/31 is directly connected, Vlan21
C>* 10.1.0.4/31 is directly connected, Vlan31
C>* 10.1.0.6/31 is directly connected, Vlan41
C>* 127.0.0.0/8 is directly connected, lo
B>* 192.168.1.0/24 [20/0] via 10.1.0.0, Vlan11, 01:47:23
B>* 192.168.2.0/24 [20/0] via 10.1.0.2, Vlan21, 01:47:21
B>* 192.168.3.0/24 [20/0] via 10.1.0.4, Vlan31, 01:47:21
B>* 192.168.4.0/24 [20/0] via 10.1.0.6, Vlan41, 01:47:23
</syntaxhighlight>
For more commands check SONiC docs : https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md

To finish, below the output of the start.sh when all goes well<div></div>
plancastre@dockerhost:~/00.Projects/p4-version-pla$ ./start.sh
---------------------------------------------------
Creating the containers: starting
---------------------------------------------------
0b4bbb3c6cfc9895466ea13a9e940d1a176d66f45deee9752e9b4732d735bb03
5ca9486baab3e8a5259062167132a944eee717875d3cc1ef29fe490cba3184c1
46c9c09dba27a32715b9fbe0bc434a9284b9d59317d278df693f425cb669f037
898e6afbcb2977115acbba2a69d069796cfd70fb90228b6c18e09ba4584365f7
eb9b3ef6d14784c9ffe4d9d5df3e921a24559889cb8770d226263aad6761182a
ae5f3b2c628da687eaabb331dc2cd989e71bcb386854e7a0be6ec587df99d49a
122197bffd7566a01ce3b6565fdfb6eb893aab7d5236f14e5a9bbbe95037ced2
a604520a080e2a2184598ad1d915c0c182d16e5185033373e95a4772d8978a67
0c49e18d78b2aa7c8ef8e937b60285aa56ec0bf4a8530226285270a085e11d5c
eb66336b8c6bd8333f1f4f685369e6cac973b817c35bb7126ac4d7ba72ef71bf
---------------------------------------------------
Retreiving PID from docker containers
---------------------------------------------------
---------------------------------------------------
Creating network namespaces for containers
---------------------------------------------------
List of created namespaces
ns_host4
ns_host3
ns_host2
ns_host1
ns_spine2
ns_spine1
ns_leaf4
ns_leaf3
ns_leaf2
ns_leaf1
---------------------------------------------------
Creating network connectivity between containers
---------------------------------------------------
---------------------------------------------------
Now assigning IPs to hosts (host1 to host4)
---------------------------------------------------
---------------------------------------------------
Configuration of containers is now finished
---------------------------------------------------
---------------------------------------------------
Now booting switches, please wait 2 minutes for switches to load
---------------------------------------------------
---------------------------------------------------
2 minutes period is finished
---------------------------------------------------
---------------------------------------------------
Now fixing the iptables
---------------------------------------------------
---------------------------------------------------
Backend actions are now finished
---------------------------------------------------
---------------------------------------------------
Now we'll check in 15 sec that BGP peers are all Up
---------------------------------------------------
---------------------------------------------------
SPINE1 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 94 97 0 0 0 00:01:28 4
10.1.0.2 4 65102 95 97 0 0 0 00:01:28 5
10.1.0.4 4 65103 95 97 0 0 0 00:01:28 5
10.1.0.6 4 65104 95 97 0 0 0 00:01:28 5
Total number of neighbors 4
---------------------------------------------------
SPINE2 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.2.0.1, local AS number 65002
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.2.0.0 4 65101 93 96 0 0 0 00:01:29 2
10.2.0.2 4 65102 94 100 0 0 0 00:01:32 1
10.2.0.4 4 65103 94 100 0 0 0 00:01:32 1
10.2.0.6 4 65104 94 100 0 0 0 00:01:32 1
Total number of neighbors 4
---------------------------------------------------
Now if all is ok, we'll do some ping tests
---------------------------------------------------
---------------------------------------------------
host1 ping its gateway
---------------------------------------------------
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=17.6 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=9.14 ms
--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 9.145/13.390/17.636/4.247 ms
---------------------------------------------------
host2 ping its gateway
---------------------------------------------------
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=23.6 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=6.21 ms
--- 192.168.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 6.218/14.950/23.683/8.733 ms
---------------------------------------------------
host3 ping its gateway
---------------------------------------------------
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=18.8 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=12.7 ms
--- 192.168.3.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.716/15.764/18.812/3.048 ms
---------------------------------------------------
host4 ping its gateway
---------------------------------------------------
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.
64 bytes from 192.168.4.1: icmp_seq=1 ttl=64 time=17.7 ms
64 bytes from 192.168.4.1: icmp_seq=2 ttl=64 time=4.46 ms
--- 192.168.4.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.464/11.107/17.750/6.643 ms
---------------------------------------------------
host1 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=16.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.049/17.253/19.532/1.615 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=15.5 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=14.3 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.312/15.592/16.939/1.083 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=20.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=16.8 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.048/17.871/20.739/2.058 ms
---------------------------------------------------
host2 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=16.6 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.222/16.583/16.924/0.286 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.6 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=18.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=26.8 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.620/20.549/26.887/4.524 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=33.8 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=27.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=20.2 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 20.216/27.259/33.818/5.566 ms
---------------------------------------------------
host3 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=14.5 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=15.5 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.535/16.549/19.541/2.162 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.4 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=14.6 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=18.2 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.643/16.458/18.278/1.491 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=15.3 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=14.1 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=13.1 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 13.171/14.234/15.334/0.883 ms
---------------------------------------------------
host4 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=21.9 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=21.3 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=14.6 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 14.619/19.284/21.932/3.310 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=14.8 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=18.4 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=14.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.148/15.786/18.409/1.879 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=15.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=18.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 15.117/16.788/18.962/1.612 ms
---------------------------------------------------
If bot spine have all BGP sessions and all ping succeeded, all works well
Congrats ! please check readme file to find some details
---------------------------------------------------

Nesevo Virtual Labs

2021-10-29T08:39:47Z

PierreLancastre: /* SONiC Lab Docker */

Nesevo Virtual Labs

2021-10-29T08:38:35Z

PierreLancastre:

Nesevo Virtual Labs

2021-10-29T08:36:21Z

PierreLancastre: /* SONiC Lab Docker */

[[File:nesevo-labs-v2.png]]

= Nokia/Cumulus ContainerLab =

[[Cumulus-Nokia-Lab]]

= SONiC Lab Docker =

[[Sonic-lab]]

Nesevo Virtual Labs

2021-10-29T08:35:59Z

PierreLancastre: /* SONiC Lab */

[[File:nesevo-labs-v2.png]]

= Nokia/Cumulus ContainerLab =

[[Cumulus-Nokia-Lab]]

= SONiC Lab Docker =

[[sonic-lab]]

Nesevo Virtual Labs

2021-10-29T08:35:39Z

PierreLancastre: /* Nokia/Cumulus Multi-Vendor Lab */

[[File:nesevo-labs-v2.png]]

= Nokia/Cumulus ContainerLab =

[[Cumulus-Nokia-Lab]]

= SONiC Lab =

[[sonic-lab]]

Nesevo Virtual Labs

2021-10-29T08:35:14Z

PierreLancastre: /* SONiC Lab */

[[File:nesevo-labs-v2.png]]

= Nokia/Cumulus Multi-Vendor Lab =

[[Cumulus-Nokia-Lab]]

= SONiC Lab =

[[sonic-lab]]

Sonic-lab

2021-10-29T08:34:39Z

PierreLancastre: Created page with " <span><to be noted that those informations are included into the readme.md file into the archive></span> == Lab diagram == File:SONIC_LAB.jpg Lab archive [https://wiki...."

<span><to be noted that those informations are included into the readme.md file into the archive></span>

== Lab diagram ==
[[File:SONIC_LAB.jpg]]

Lab archive [https://wiki.nesevo.com/images/0/0a/Sonic-lab-20201030.zip Sonic-lab-20201030.zip]

This lab is a forked lab from many found on @, starting from Microsoft one : https://github.com/Azure/SONiC/wiki/SONiC-P4-Software-Switch

The target of this lab is to simulate a simple IP/Fabric in SPINE/LEAF topology. Each Leaf provide local L2 connectivity + L3 gateways to hosts. Flows between hosts from different leaf are L3 routed, in a "node path" LEAF-SPINE-LEAF.

Nothing complicated, but it allows to put a first foot into SONiC world :)

For the moment, as far as we know, MC-LAG + EVPN/VXLAN features are only available on entreprise SONiC versions. Tests to come once physic switches in enterprise version will be available in our lab.

The diagram of the architecture is into the zip file (SONIC_LAB.jpg)

To run the lab you need :

* to have installed docker by using ./install_docker_ovs.sh
* to run the ./load_image.sh in order to get the Sonic image

Note : don't change file / folders names of network devices (leafX, spineX)

Once you're ready, please run the lab by launching "./start.sh"

The script :<div></div>
* launches the containers without Docket network features
* create network namespaces
* build linux bridges to interconnect the containers
* configure host IPs
* launch startup sonic scripts into sonic devices
* once devices started, checks + ping tests are run. If all BGP sessions are ok on spine + all ping are fine, the setup works well.
The "start" script has several comments, perhaps without the right vocabulary, so please be tolerant :)

If you want to stop the lab, please launch "./stop.sh"

Below some useful commands to run for sonic containers :

=== <span>run ping</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> ping <ip>
</syntaxhighlight>
This command triggers ping into the targetted container :
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it leaf1 ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.052 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.034 ms
^C--- 192.168.1.1 ping statistics ---
</syntaxhighlight>

=== <span>check bgp sessions</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip bgp sum"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print the bgp status.
Below, we run it on one spine, you should have 4 sessiosn UP (for the 4 leaves)
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip bgp sum"
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 6409 6411 0 0 0 01:46:46 3
10.1.0.2 4 65102 6406 6410 0 0 0 01:46:44 3
10.1.0.4 4 65103 6407 6410 0 0 0 01:46:44 4
10.1.0.6 4 65104 6406 6412 0 0 0 01:46:46 1
Total number of neighbors 4
</syntaxhighlight>

=== <span>check ip route</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip route"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print its routing table?

<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,
> - selected route, * - FIB route
C>* 10.1.0.0/31 is directly connected, Vlan11
C>* 10.1.0.2/31 is directly connected, Vlan21
C>* 10.1.0.4/31 is directly connected, Vlan31
C>* 10.1.0.6/31 is directly connected, Vlan41
C>* 127.0.0.0/8 is directly connected, lo
B>* 192.168.1.0/24 [20/0] via 10.1.0.0, Vlan11, 01:47:23
B>* 192.168.2.0/24 [20/0] via 10.1.0.2, Vlan21, 01:47:21
B>* 192.168.3.0/24 [20/0] via 10.1.0.4, Vlan31, 01:47:21
B>* 192.168.4.0/24 [20/0] via 10.1.0.6, Vlan41, 01:47:23
</syntaxhighlight>
For more commands check SONiC docs : https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md

To finish, below the output of the start.sh when all goes well<div></div>
plancastre@dockerhost:~/00.Projects/p4-version-pla$ ./start.sh
---------------------------------------------------
Creating the containers: starting
---------------------------------------------------
0b4bbb3c6cfc9895466ea13a9e940d1a176d66f45deee9752e9b4732d735bb03
5ca9486baab3e8a5259062167132a944eee717875d3cc1ef29fe490cba3184c1
46c9c09dba27a32715b9fbe0bc434a9284b9d59317d278df693f425cb669f037
898e6afbcb2977115acbba2a69d069796cfd70fb90228b6c18e09ba4584365f7
eb9b3ef6d14784c9ffe4d9d5df3e921a24559889cb8770d226263aad6761182a
ae5f3b2c628da687eaabb331dc2cd989e71bcb386854e7a0be6ec587df99d49a
122197bffd7566a01ce3b6565fdfb6eb893aab7d5236f14e5a9bbbe95037ced2
a604520a080e2a2184598ad1d915c0c182d16e5185033373e95a4772d8978a67
0c49e18d78b2aa7c8ef8e937b60285aa56ec0bf4a8530226285270a085e11d5c
eb66336b8c6bd8333f1f4f685369e6cac973b817c35bb7126ac4d7ba72ef71bf
---------------------------------------------------
Retreiving PID from docker containers
---------------------------------------------------
---------------------------------------------------
Creating network namespaces for containers
---------------------------------------------------
List of created namespaces
ns_host4
ns_host3
ns_host2
ns_host1
ns_spine2
ns_spine1
ns_leaf4
ns_leaf3
ns_leaf2
ns_leaf1
---------------------------------------------------
Creating network connectivity between containers
---------------------------------------------------
---------------------------------------------------
Now assigning IPs to hosts (host1 to host4)
---------------------------------------------------
---------------------------------------------------
Configuration of containers is now finished
---------------------------------------------------
---------------------------------------------------
Now booting switches, please wait 2 minutes for switches to load
---------------------------------------------------
---------------------------------------------------
2 minutes period is finished
---------------------------------------------------
---------------------------------------------------
Now fixing the iptables
---------------------------------------------------
---------------------------------------------------
Backend actions are now finished
---------------------------------------------------
---------------------------------------------------
Now we'll check in 15 sec that BGP peers are all Up
---------------------------------------------------
---------------------------------------------------
SPINE1 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 94 97 0 0 0 00:01:28 4
10.1.0.2 4 65102 95 97 0 0 0 00:01:28 5
10.1.0.4 4 65103 95 97 0 0 0 00:01:28 5
10.1.0.6 4 65104 95 97 0 0 0 00:01:28 5
Total number of neighbors 4
---------------------------------------------------
SPINE2 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.2.0.1, local AS number 65002
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.2.0.0 4 65101 93 96 0 0 0 00:01:29 2
10.2.0.2 4 65102 94 100 0 0 0 00:01:32 1
10.2.0.4 4 65103 94 100 0 0 0 00:01:32 1
10.2.0.6 4 65104 94 100 0 0 0 00:01:32 1
Total number of neighbors 4
---------------------------------------------------
Now if all is ok, we'll do some ping tests
---------------------------------------------------
---------------------------------------------------
host1 ping its gateway
---------------------------------------------------
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=17.6 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=9.14 ms
--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 9.145/13.390/17.636/4.247 ms
---------------------------------------------------
host2 ping its gateway
---------------------------------------------------
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=23.6 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=6.21 ms
--- 192.168.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 6.218/14.950/23.683/8.733 ms
---------------------------------------------------
host3 ping its gateway
---------------------------------------------------
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=18.8 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=12.7 ms
--- 192.168.3.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.716/15.764/18.812/3.048 ms
---------------------------------------------------
host4 ping its gateway
---------------------------------------------------
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.
64 bytes from 192.168.4.1: icmp_seq=1 ttl=64 time=17.7 ms
64 bytes from 192.168.4.1: icmp_seq=2 ttl=64 time=4.46 ms
--- 192.168.4.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.464/11.107/17.750/6.643 ms
---------------------------------------------------
host1 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=16.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.049/17.253/19.532/1.615 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=15.5 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=14.3 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.312/15.592/16.939/1.083 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=20.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=16.8 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.048/17.871/20.739/2.058 ms
---------------------------------------------------
host2 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=16.6 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.222/16.583/16.924/0.286 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.6 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=18.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=26.8 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.620/20.549/26.887/4.524 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=33.8 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=27.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=20.2 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 20.216/27.259/33.818/5.566 ms
---------------------------------------------------
host3 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=14.5 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=15.5 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.535/16.549/19.541/2.162 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.4 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=14.6 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=18.2 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.643/16.458/18.278/1.491 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=15.3 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=14.1 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=13.1 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 13.171/14.234/15.334/0.883 ms
---------------------------------------------------
host4 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=21.9 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=21.3 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=14.6 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 14.619/19.284/21.932/3.310 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=14.8 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=18.4 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=14.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.148/15.786/18.409/1.879 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=15.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=18.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 15.117/16.788/18.962/1.612 ms
---------------------------------------------------
If bot spine have all BGP sessions and all ping succeeded, all works well
Congrats ! please check readme file to find some details
---------------------------------------------------

Nesevo Virtual Labs

2021-10-29T08:34:01Z

PierreLancastre: /* Nokia/Cumulus Multi-Vendor Lab */

[[File:nesevo-labs-v2.png]]

= Nokia/Cumulus Multi-Vendor Lab =

[[Cumulus-Nokia-Lab]]

= SONiC Lab =

<span><to be noted that those informations are included into the readme.md file into the archive></span>

== Lab diagram ==
[[File:SONIC_LAB.jpg]]

Lab archive [https://wiki.nesevo.com/images/0/0a/Sonic-lab-20201030.zip Sonic-lab-20201030.zip]

This lab is a forked lab from many found on @, starting from Microsoft one : https://github.com/Azure/SONiC/wiki/SONiC-P4-Software-Switch

The target of this lab is to simulate a simple IP/Fabric in SPINE/LEAF topology. Each Leaf provide local L2 connectivity + L3 gateways to hosts. Flows between hosts from different leaf are L3 routed, in a "node path" LEAF-SPINE-LEAF.

Nothing complicated, but it allows to put a first foot into SONiC world :)

For the moment, as far as we know, MC-LAG + EVPN/VXLAN features are only available on entreprise SONiC versions. Tests to come once physic switches in enterprise version will be available in our lab.

The diagram of the architecture is into the zip file (SONIC_LAB.jpg)

To run the lab you need :

* to have installed docker by using ./install_docker_ovs.sh
* to run the ./load_image.sh in order to get the Sonic image

Note : don't change file / folders names of network devices (leafX, spineX)

Once you're ready, please run the lab by launching "./start.sh"

The script :<div></div>
* launches the containers without Docket network features
* create network namespaces
* build linux bridges to interconnect the containers
* configure host IPs
* launch startup sonic scripts into sonic devices
* once devices started, checks + ping tests are run. If all BGP sessions are ok on spine + all ping are fine, the setup works well.
The "start" script has several comments, perhaps without the right vocabulary, so please be tolerant :)

If you want to stop the lab, please launch "./stop.sh"

Below some useful commands to run for sonic containers :

=== <span>run ping</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> ping <ip>
</syntaxhighlight>
This command triggers ping into the targetted container :
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it leaf1 ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.052 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.034 ms
^C--- 192.168.1.1 ping statistics ---
</syntaxhighlight>

=== <span>check bgp sessions</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip bgp sum"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print the bgp status.
Below, we run it on one spine, you should have 4 sessiosn UP (for the 4 leaves)
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip bgp sum"
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 6409 6411 0 0 0 01:46:46 3
10.1.0.2 4 65102 6406 6410 0 0 0 01:46:44 3
10.1.0.4 4 65103 6407 6410 0 0 0 01:46:44 4
10.1.0.6 4 65104 6406 6412 0 0 0 01:46:46 1
Total number of neighbors 4
</syntaxhighlight>

=== <span>check ip route</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip route"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print its routing table?

<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,
> - selected route, * - FIB route
C>* 10.1.0.0/31 is directly connected, Vlan11
C>* 10.1.0.2/31 is directly connected, Vlan21
C>* 10.1.0.4/31 is directly connected, Vlan31
C>* 10.1.0.6/31 is directly connected, Vlan41
C>* 127.0.0.0/8 is directly connected, lo
B>* 192.168.1.0/24 [20/0] via 10.1.0.0, Vlan11, 01:47:23
B>* 192.168.2.0/24 [20/0] via 10.1.0.2, Vlan21, 01:47:21
B>* 192.168.3.0/24 [20/0] via 10.1.0.4, Vlan31, 01:47:21
B>* 192.168.4.0/24 [20/0] via 10.1.0.6, Vlan41, 01:47:23
</syntaxhighlight>
For more commands check SONiC docs : https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md

To finish, below the output of the start.sh when all goes well<div></div>
plancastre@dockerhost:~/00.Projects/p4-version-pla$ ./start.sh
---------------------------------------------------
Creating the containers: starting
---------------------------------------------------
0b4bbb3c6cfc9895466ea13a9e940d1a176d66f45deee9752e9b4732d735bb03
5ca9486baab3e8a5259062167132a944eee717875d3cc1ef29fe490cba3184c1
46c9c09dba27a32715b9fbe0bc434a9284b9d59317d278df693f425cb669f037
898e6afbcb2977115acbba2a69d069796cfd70fb90228b6c18e09ba4584365f7
eb9b3ef6d14784c9ffe4d9d5df3e921a24559889cb8770d226263aad6761182a
ae5f3b2c628da687eaabb331dc2cd989e71bcb386854e7a0be6ec587df99d49a
122197bffd7566a01ce3b6565fdfb6eb893aab7d5236f14e5a9bbbe95037ced2
a604520a080e2a2184598ad1d915c0c182d16e5185033373e95a4772d8978a67
0c49e18d78b2aa7c8ef8e937b60285aa56ec0bf4a8530226285270a085e11d5c
eb66336b8c6bd8333f1f4f685369e6cac973b817c35bb7126ac4d7ba72ef71bf
---------------------------------------------------
Retreiving PID from docker containers
---------------------------------------------------
---------------------------------------------------
Creating network namespaces for containers
---------------------------------------------------
List of created namespaces
ns_host4
ns_host3
ns_host2
ns_host1
ns_spine2
ns_spine1
ns_leaf4
ns_leaf3
ns_leaf2
ns_leaf1
---------------------------------------------------
Creating network connectivity between containers
---------------------------------------------------
---------------------------------------------------
Now assigning IPs to hosts (host1 to host4)
---------------------------------------------------
---------------------------------------------------
Configuration of containers is now finished
---------------------------------------------------
---------------------------------------------------
Now booting switches, please wait 2 minutes for switches to load
---------------------------------------------------
---------------------------------------------------
2 minutes period is finished
---------------------------------------------------
---------------------------------------------------
Now fixing the iptables
---------------------------------------------------
---------------------------------------------------
Backend actions are now finished
---------------------------------------------------
---------------------------------------------------
Now we'll check in 15 sec that BGP peers are all Up
---------------------------------------------------
---------------------------------------------------
SPINE1 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 94 97 0 0 0 00:01:28 4
10.1.0.2 4 65102 95 97 0 0 0 00:01:28 5
10.1.0.4 4 65103 95 97 0 0 0 00:01:28 5
10.1.0.6 4 65104 95 97 0 0 0 00:01:28 5
Total number of neighbors 4
---------------------------------------------------
SPINE2 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.2.0.1, local AS number 65002
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.2.0.0 4 65101 93 96 0 0 0 00:01:29 2
10.2.0.2 4 65102 94 100 0 0 0 00:01:32 1
10.2.0.4 4 65103 94 100 0 0 0 00:01:32 1
10.2.0.6 4 65104 94 100 0 0 0 00:01:32 1
Total number of neighbors 4
---------------------------------------------------
Now if all is ok, we'll do some ping tests
---------------------------------------------------
---------------------------------------------------
host1 ping its gateway
---------------------------------------------------
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=17.6 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=9.14 ms
--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 9.145/13.390/17.636/4.247 ms
---------------------------------------------------
host2 ping its gateway
---------------------------------------------------
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=23.6 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=6.21 ms
--- 192.168.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 6.218/14.950/23.683/8.733 ms
---------------------------------------------------
host3 ping its gateway
---------------------------------------------------
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=18.8 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=12.7 ms
--- 192.168.3.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.716/15.764/18.812/3.048 ms
---------------------------------------------------
host4 ping its gateway
---------------------------------------------------
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.
64 bytes from 192.168.4.1: icmp_seq=1 ttl=64 time=17.7 ms
64 bytes from 192.168.4.1: icmp_seq=2 ttl=64 time=4.46 ms
--- 192.168.4.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.464/11.107/17.750/6.643 ms
---------------------------------------------------
host1 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=16.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.049/17.253/19.532/1.615 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=15.5 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=14.3 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.312/15.592/16.939/1.083 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=20.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=16.8 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.048/17.871/20.739/2.058 ms
---------------------------------------------------
host2 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=16.6 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.222/16.583/16.924/0.286 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.6 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=18.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=26.8 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.620/20.549/26.887/4.524 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=33.8 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=27.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=20.2 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 20.216/27.259/33.818/5.566 ms
---------------------------------------------------
host3 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=14.5 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=15.5 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.535/16.549/19.541/2.162 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.4 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=14.6 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=18.2 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.643/16.458/18.278/1.491 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=15.3 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=14.1 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=13.1 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 13.171/14.234/15.334/0.883 ms
---------------------------------------------------
host4 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=21.9 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=21.3 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=14.6 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 14.619/19.284/21.932/3.310 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=14.8 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=18.4 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=14.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.148/15.786/18.409/1.879 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=15.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=18.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 15.117/16.788/18.962/1.612 ms
---------------------------------------------------
If bot spine have all BGP sessions and all ping succeeded, all works well
Congrats ! please check readme file to find some details
---------------------------------------------------

Cumulus-Nokia-Lab

2021-10-29T08:33:39Z

PierreLancastre: Created page with "The targets of this lab are to : * mix Nokia SR-Linux and Cumulus VXLAN nodes, in spine or leaf roles * make this environment work for different kinds of setup : cross-DC L2,..."

The targets of this lab are to :

* mix Nokia SR-Linux and Cumulus VXLAN nodes, in spine or leaf roles
* make this environment work for different kinds of setup : cross-DC L2, differentiated L3 gateways per leaf per VRF, anycast gateway cross or intra DC, etc.
* spot compatibility issues if existing

To reach those goals, we'll use the Nokia lab tool "containerlab" https://containerlab.srlinux.dev/, labs from https://clabs.netdevops.me/ helped a lot. Many thanks to the contributors.

We'll need a Linux host with Docker installed on it + kvm virtualization enhancements activated (important if your host is a VM). If virtualization optimization is not ok, you won't be able to launch the most of devices.

Installation details are referenced there : https://containerlab.srlinux.dev/install/

Once installed, you'll need to register the different NOS images you'll use. Cumulus image is present on a public registrery, but Nokia/Arista/Juniper are not, you'll need a vendor support account to retreive them. Once downloaded (.tgz file typically) you can "docker load" the image :

For private images :<syntaxhighlight lang="bash">
sudo docker load -i 21.6.1-235.tar.xz /* (Nokia) */
sudo docker load -i cEOS64-lab-4.26.1F.tar /* (Arista) */
</syntaxhighlight>For public images :<syntaxhighlight lang="bash">
sudo docker pull networkop/cx:4.3.0 /* (Cumulus) */
</syntaxhighlight>

To verify your installed images, you can run "sudo docker images"<syntaxhighlight lang="bash">
plancastre@containerlab:~/00.LAB/senss$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
networkop/cx 4.3.0 a40541ed15ba 2 weeks ago 733MB
srlinux 21.6.1-235 e29aed30e83f 3 weeks ago 1.14GB
networkop/ignite dev dcc5eff29355 4 weeks ago 37.9MB
networkop/kernel 4.19 e794d8875f83 3 months ago 88MB
frrouting/frr v7.5.1 c3e13a4c5918 3 months ago 123MB

</syntaxhighlight>Once you've got all your images, to run the lab, you'll need a .yml file defining the topology to run.

In our lab, we've started with the 5-clos lab file from Nokia site, which we've forked.

We can see there is a branch "kinds" where we can define templates of nodes, in order to not rewrite whole parameters when creating nodes. For the images, it's important to take the repository name + tag.

'''Note:''' If the tag is "latest" it is not needed.

All supported "kinds" are referenced there : https://containerlab.srlinux.dev/manual/kinds/kinds/

'''Note:''' yml is indentation sensitive, so please respect the "double space" used to define hierarchy. Below the lab diagram + the yml file associated

== Infos on nodes ==

=== L3 infos ===
{| class="wikitable"
|+
!Device
!OS
!ASN
!VTEP IP
!swp1 or eth1/1
!swp2 or eth1/2
!swp3 or eth1/3
!SVI
|-
|client1
|Linux
|<none>
|<none>
|vlan100: 192.168.1.1/24
vlan200: 192.168.2.1/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client2
|Linux
|<none>
|<none>
|vlan100: 192.168.1.2/24
vlan200: 192.168.2.2/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client3
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.1/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|client4
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.2/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|leaf1
|CVX
|65011
|10.255.255.5/32
|10.254.254.8/31
|10.254.254.10/31
|trunk vlan 100,200
|vlan200 : 192.168.2.254/24
|-
|leaf2
|SRLinux
|65012
|10.255.255.6/32
|10.254.254.12/31
|10.254.254.14/31
|trunk vlan 100,200
|irb0.200 : 192.168.2.254/24
|-
|leaf3
|CVX
|65013
|10.255.255.7/32
|10.254.254.22/31
|10.254.254.24/31
|trunk vlan 100,200
|vlan200 : 192.168.200.254/24
|-
|leaf4
|SRLinux
|65014
|10.255.255.8/32
|10.254.254.26/31
|10.254.254.28/31
|trunk vlan 100,200
|irb0.200 : 192.168.200.254/24
|-
|spine1
|CVX
|65001
|10.255.255.1/32
|10.254.254.9/31
|10.254.254.13/31
|10.254.254.0/31
|<none>
|-
|spine2
|CVX
|65001
|10.255.255.2/32
|10.254.254.11/31
|10.254.254.15/31
|10.254.254.16/31
|<none>
|-
|spine3
|SRLinux
|65002
|10.255.255.3/32
|10.254.254.23/31
|10.254.254.27/31
|10.254.254.18/31
|<none>
|-
|spine4
|SRLinux
|65002
|10.255.255.4/32
|10.254.254.25/31
|10.254.254.29/31
|10.254.254.20/31
|<none>
|-
|superspine1
|CVX
|65003
|10.255.255.9/32
|10.254.254.1/31
|10.254.254.19/31
|<none>
|<none>
|-
|superspine2
|SRLinux
|65003
|10.255.255.10/32
|10.254.254.17/31
|10.254.254.21/31
|<none>
|<none>
|}

=== VXLAN/BGP infos ===
Note : in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>
{| class="wikitable"
|+
!service
!vlan id
!route-target
!VNI
!Commentaires
|-
|L2 cross-DC
|100
|target:65000:100
|100
|the 4 clients are in the same broadcast domain / subnet
|-
|L3VPN cros-DC
|200
|target:65000:200
target:65000:10200
|200 (for L2)
10200 (for routing)
|the 4 clients are separated in 2 different subnets. Traffic is routed within dedicated VNI
To be noted that in the current setup VMTO is not implemented, so cross DC traffic is not optimized.
|}

== Lab diagram ==
[[File:Lab-srl-diagram.png|ohne|mini|600x600px]]

== yml lab file ==
<syntaxhighlight lang="yaml">
# topology documentation: http://containerlab.srlinux.dev/lab-examples/min-5clos/
name: lab-srl-cum

topology:
kinds:
srl:
image: srlinux:21.6.1-235
linux:
image: ghcr.io/hellt/network-multitool
cvx:
image: networkop/cx:4.3.0
runtime: docker
nodes:
leaf1:
kind: cvx
leaf2:
kind: srl
type: ixrd2
leaf3:
kind: cvx
leaf4:
kind: srl
type: ixrd2

spine1:
kind: cvx
spine2:
kind: cvx
spine3:
kind: srl
type: ixr6
spine4:
kind: srl
type: ixr6

superspine1:
kind: cvx
superspine2:
kind: srl
type: ixr6

client1:
kind: linux
client2:
kind: linux
client3:
kind: linux
client4:
kind: linux

links:
# leaf to spine links POD1
- endpoints: ["leaf1:swp1", "spine1:swp1"]
- endpoints: ["leaf1:swp2", "spine2:swp1"]
- endpoints: ["leaf2:e1-1", "spine1:swp2"]
- endpoints: ["leaf2:e1-2", "spine2:swp2"]

# spine to superspine links POD1
- endpoints: ["spine1:swp3", "superspine1:swp1"]
- endpoints: ["spine2:swp3", "superspine2:e1-1"]

# leaf to spine links POD2
- endpoints: ["leaf3:swp1", "spine3:e1-1"]
- endpoints: ["leaf3:swp2", "spine4:e1-1"]
- endpoints: ["leaf4:e1-1", "spine3:e1-2"]
- endpoints: ["leaf4:e1-2", "spine4:e1-2"]

# spine to superspine links POD2
- endpoints: ["spine3:e1-3", "superspine1:swp2"]
- endpoints: ["spine4:e1-3", "superspine2:e1-2"]

# client connection links
- endpoints: ["client1:eth1", "leaf1:swp3"]
- endpoints: ["client2:eth1", "leaf2:e1-3"]
- endpoints: ["client3:eth1", "leaf3:swp3"]
- endpoints: ["client4:eth1", "leaf4:e1-3"]

</syntaxhighlight>

Once the nodes are defined, we have to define the needed connections between the nodes.

Each endpoint is defined as '<nowiki/>'''<nowiki/>'<node_name>:<interface_name><nowiki>''</nowiki>'''

When all is ok, you can launch in suduoer the topology.

For example :<syntaxhighlight lang="bash">
sudo containerlab deploy --topo topo.clab.yml --reconfigure

Note: The option "reconfigure" is used there in order to not have to destroy instances when updating topology/relaunching.
</syntaxhighlight>Another info regarding the yml file, it's possible to quickly generate files with the '''"containerlab gen -n 3tier --nodes 4,2,1"''' command for example. https://containerlab.srlinux.dev/cmd/generate/

To be noted that the "tier-x" notion in containerlab is inversed regarding the rfc 7938 (tier1 is superspine level into the rfc, tier1 is leaf level in nokia tool)

At the end of the topology launch, you can retreive the details :<syntaxhighlight lang="yaml">
INFO[0011] Writing /etc/hosts file
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| # | Name | Container ID | Image | Kind | Group | State | IPv4 Address | IPv6 Address |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| 1 | clab-lab-srl-cum-client1 | 25e2bbf6b13f | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
| 2 | clab-lab-srl-cum-client2 | 17d82d5a6e21 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.4/24 | 2001:172:20:20::4/64 |
| 3 | clab-lab-srl-cum-client3 | f8d78527c1b1 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.6/24 | 2001:172:20:20::6/64 |
| 4 | clab-lab-srl-cum-client4 | 913a0172d0c5 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 5 | clab-lab-srl-cum-leaf1 | 9ffc334af2b1 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.9/24 | 2001:172:20:20::9/64 |
| 6 | clab-lab-srl-cum-leaf2 | 7d3eb855d749 | srlinux:21.6.1-235 | srl | | running | 172.20.20.14/24 | 2001:172:20:20::e/64 |
| 7 | clab-lab-srl-cum-leaf3 | e0d36ffc606a | networkop/cx:4.3.0 | cvx | | running | 172.20.20.7/24 | 2001:172:20:20::7/64 |
| 8 | clab-lab-srl-cum-leaf4 | e6753b1d099e | srlinux:21.6.1-235 | srl | | running | 172.20.20.12/24 | 2001:172:20:20::c/64 |
| 9 | clab-lab-srl-cum-spine1 | a7e63993d052 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.5/24 | 2001:172:20:20::5/64 |
| 10 | clab-lab-srl-cum-spine2 | 3124250d5b75 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
| 11 | clab-lab-srl-cum-spine3 | 5e300ca3f66f | srlinux:21.6.1-235 | srl | | running | 172.20.20.11/24 | 2001:172:20:20::b/64 |
| 12 | clab-lab-srl-cum-spine4 | 0a59e60a596b | srlinux:21.6.1-235 | srl | | running | 172.20.20.13/24 | 2001:172:20:20::d/64 |
| 13 | clab-lab-srl-cum-superspine1 | 3d9935413fc2 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.8/24 | 2001:172:20:20::8/64 |
| 14 | clab-lab-srl-cum-superspine2 | 5bc58d423e2c | srlinux:21.6.1-235 | srl | | running | 172.20.20.15/24 | 2001:172:20:20::f/64 |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
</syntaxhighlight>The containerlab program creates the updates into the host file + generates an ansible inventory file.

There are different ways to connect onto the docker instances, within docker exec commands or SSH directly the IP address related above.

For example :<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-spine1 bash
</syntaxhighlight>-->>> allows to launch a bash on the cumulus device (then you run the different net commands)<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-leaf2 sr_cli
</syntaxhighlight>-->>> allows to launch the cli on a SRLinux device

To end this setup preparation, the containerlab tool integrates an option to launch a webserver showing the lab topology in graph + node details

By default, it's listening on port 50080 of the host

Then, you can access to the following web page showing the topology + lab infos

[[File:Lab graph tool.jpg|ohne|mini|900x900px]]

Finally, if you want to stop a lab, please run '''<nowiki>''</nowiki>containerlab destroy --topo <topology>.yml<nowiki>''</nowiki>'''

At this step, nodes are running, connections are done, but configurations are empty.

'''So, it's time to configure them :)'''

Configuration files are attached to this blog. Once validated, they are exportable and can be called by the tool when launching the lab. Below how to do it :

Firstable, once you ve finished your "hand made config", you can run the following command to save it :<syntaxhighlight lang="bash">
containerlab save -t quickstart.clab.yml
</syntaxhighlight>'''To be noted it's only working for Nokia nodes in this lab'''. CVX configurations will have to be called by another method (or pushed manually)

It will create one folder per node into the lab folder which gives the following tree :
[[File:Topology-srl.jpg|ohne|mini]]

'''Note''': For Cumulus, you can apparently retrieve configurations files within adding the following parameters into lab config file. '''To be tested.''' some lab samples here : https://github.com/hellt/clabs/blob/main/labs/cvx04/symm-mh.clab.yml

Now we have the lab running, below the different commands to check that all is working as expected : <syntaxhighlight lang="yaml">
-------------------------------------------
Check L2 tables ///
-------------------------------------------

Nokia :

A:leaf2# show network-instance mac-vrf-100 bridge-table mac-table all
----------------------------------------------------------------------------------------------------------------------------------------------------------
Mac-table of network instance mac-vrf-100
----------------------------------------------------------------------------------------------------------------------------------------------------------
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
| Address | Destination | Dest Index | Type | Active | Aging | Last Update |
+====================+=========================================+============+=============+=========+========+=========================================+
| AA:C1:AB:0C:60:D6 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T13:06:17.000Z |
| | vtep:10.255.255.8 vni:100 | 67 | | | | |
| AA:C1:AB:23:31:04 | ethernet-1/3.100 | 2 | learnt | true | 300 | 2021-10-08T13:06:10.000Z |
| AA:C1:AB:4B:FF:71 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:55:28.000Z |
| | vtep:10.255.255.5 vni:100 | 60 | | | | |
| AA:C1:AB:B5:79:50 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:43:37.000Z |
| | vtep:10.255.255.7 vni:100 | 64 | | | | |
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 0 Total 0 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 1 Total 1 Active
Total Evpn Macs : 3 Total 3 Active
Total Evpn static Macs : 0 Total 0 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 4 Total 4 Active
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : type learnt matches the local learnt mac, evpn are learnt within VXLAN

Cumulus :

root@leaf1-cum-dc1:/# net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
100 bridge swp3 aa:c1:ab:4b:ff:71 00:00:02
100 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:17:21
100 bridge vni100 22:2d:67:f1:bb:26 permanent 02:17:21
100 bridge vni100 aa:c1:ab:0c:60:d6 extern_learn <1 sec
100 bridge vni100 aa:c1:ab:23:31:04 extern_learn 00:00:07
100 bridge vni100 aa:c1:ab:b5:79:50 extern_learn 00:13:28

root@leaf1-cum-dc1:/# net show bridge macs vlan 200

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
200 bridge swp3 aa:c1:ab:4b:ff:71 00:08:46
200 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:16:36
200 bridge vni200 00:00:5e:00:01:01 static sticky 00:14:26
200 bridge vni200 1e:0d:aa:46:53:b1 permanent 02:16:36
200 bridge vni200 02:77:6b:ff:00:41 static sticky 01:00:52
200 bridge vni200 02:f8:b8:ff:00:41 static sticky 00:14:26
200 bridge vni200 aa:c1:ab:b5:79:50 extern_learn 00:11:51
200 bridge vni10200 02:77:6b:ff:00:00 extern_learn 00:53:02
200 bridge vni10200 02:f8:b8:ff:00:00 extern_learn 00:14:26
200 bridge vni10200 86:5e:aa:ca:50:be permanent 02:16:36

Note : the macs with permanent flag are local interfaces, physical ou logic (vtep)
Note2 : extern_learn mac are the one learned via the VXLAN infra (and mapped to the vnixxx interface)

-------------------------------------------
Check arp tables ///
-------------------------------------------

Nokia :

:leaf2# show arpnd arp-entries
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| ethernet-1/1 | 0 | 10.254.254.13 | dynamic | AA:C1:AB:59:65:EA | 3 hours from now |
| ethernet-1/2 | 0 | 10.254.254.15 | dynamic | AA:C1:AB:CB:F8:8A | 3 hours from now |
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
| mgmt0 | 0 | 172.20.20.1 | dynamic | 02:42:63:9E:74:03 | 2 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 5 (0 static, 5 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : on Nokia you have to sort the entries basing on subinterface and the retrieve the L3VPN attached

A:leaf2# show arpnd arp-entries subinterface 200
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 2 (0 static, 2 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Cumulus :

Here we retrieve all arp infos linked to evpn

root@leaf1-cum-dc1:/# net show evpn arp-cache vni all
VNI 100 #ARP (IPv4 and IPv6, local and remote) 2

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.1.3 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0
192.168.1.1 local active aa:c1:ab:4b:ff:71 0/0

VNI 200 #ARP (IPv4 and IPv6, local and remote) 4

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.2.1 local active aa:c1:ab:4b:ff:71 0/0
192.168.200.254 remote active 00:00:5e:00:01:01 10.255.255.8 0/0
192.168.2.254 remote active 00:00:5e:00:01:01 10.255.255.6 0/0
192.168.200.1 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0

VNI 400 #ARP (IPv4 and IPv6, local and remote) 0

-------------------------------------------
Check bgp sessions' status - underlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep ipv4
| default | 10.254.254.19 | udl-superspine | S | 65003 | established | 0d:0h:22m:44s | ipv4-unicast | [13/4/4] |
| default | 10.254.254.22 | udl-leaf | S | 65013 | established | 0d:0h:22m:52s | ipv4-unicast | [13/1/4] |

Cumulus :

root@leaf3:/# net show bgp ipv4 unicast summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 13
RIB entries 25, using 4800 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.254.254.23 4 65002 466 403 0 0 0 00:19:34 4 13
10.254.254.25 4 65002 434 355 0 0 0 00:17:10 8 13

-------------------------------------------
Check bgp sessions' status - overlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep evpn
| default | 10.255.255.7 | ovl-leaf | S | 65013 | established | 0d:0h:24m:14s| evpn | [15/0/11] |
| default | 10.255.255.9 | ovl-superspine | S | 65003 | established | 0d:0h:24m:7s | evpn | [11/0/4] |

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.255.255.3 4 65002 5618 4742 0 0 0 03:53:35 19 27
10.255.255.4 4 65002 5565 4694 0 0 0 03:51:11 19 27

-------------------------------------------
Check EVPN L2 tables///
-------------------------------------------

Nokia :

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn route type 2
BGP table version is 17, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:100
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
Route Distinguisher: 10.255.255.5:200
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
Route Distinguisher: 10.255.255.6:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
Route Distinguisher: 10.255.255.7:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100
Route Distinguisher: 10.255.255.7:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.200.1]
10.255.255.7 32768 i
ET:8 RT:65000:200
Route Distinguisher: 10.255.255.8:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC

and if we focus on our vni 100

root@leaf3-cum-dc1:/# net show bgp evpn route vni 100 type 2
BGP table version is 48, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100

The additional route type "MAC+IP" is optional and is useful/mandatory when you do anycast irb.

-------------------------------------------
Check tables EVPN L3 ///
-------------------------------------------

Nokia :

--{ + running }--[ ]--
A:leaf2# show network-instance ip-vrf-200 route-table all
-----------------------------------------------------------------------------------------------------------------------------------------------------------
IPv4 Unicast route table of network instance ip-vrf-200
-----------------------------------------------------------------------------------------------------------------------------------------------------------
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
| Prefix | ID | Route Type | Route Owner | Best/Fib- | Metric | Pref | Next-hop | Next-hop |
| | | | | status(slot) | | | (Type) | Interface |
+=======================+=======+============+======================+======================+==========+=========+==============+==============+
| 192.168.2.0/24 | 9 | local | net_inst_mgr | True/success | 0 | 0 | 192.168.2.25 | irb0.200 |
| | | | | | | | 4 (direct) | |
| 192.168.2.254/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (extract) | |
| 192.168.2.255/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (broadcast) | |
| 192.168.200.0/24 | 0 | bgp-evpn | bgp_evpn_mgr | True/success | 0 | 170 | 10.255.255.8 | None |
| | | | | | | | (indirect/vx | |
| | | | | | | | lan) | |
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
-----------------------------------------------------------------------------------------------------------------------------------------------------------
4 IPv4 routes total
4 IPv4 prefixes with active routes
0 IPv4 prefixes with active ECMP routes
-----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--
A:leaf2#

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf1-cum-dc1:/# net show route vrf ip-vrf-200
show ip route vrf ip-vrf-200
=============================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF ip-vrf-200:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 04:48:42
B 192.168.2.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 04:27:26
C>* 192.168.2.0/24 is directly connected, vlan200, 04:48:42
B>* 192.168.200.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:48:49

root@leaf1-cum-dc1:/# net show bgp evpn route type 5
BGP table version is 3, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.6:10200
* [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
*> [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
Route Distinguisher: 10.255.255.8:10200
*> [5]:[0]:[24]:[192.168.200.0]
10.255.255.8 0 65001 65003 65002 65014 i
RT:65000:10200 ET:8 Rmac:02:f8:b8:ff:00:00

-------------------------------------------
Check arp hosts ///
-------------------------------------------

From client #1 /

bash-5.0# arp
Address HWtype HWaddress Flags Mask Iface
192.168.1.3 ether aa:c1:ab:b5:79:50 C eth1.100
192.168.1.4 ether aa:c1:ab:0c:60:d6 C eth1.100
192.168.2.2 ether aa:c1:ab:23:31:04 C eth1.200
192.168.2.254 ether aa:c1:ab:4f:f2:e7 C eth1.200
192.168.1.2 ether aa:c1:ab:23:31:04 C eth1.100
containerlab.local ether 02:42:63:9e:74:03 C eth0

-------------------------------------------
Ping ///
-------------------------------------------

From client #1 /

L2 service ping (vlan 100) ----------------------

bash-5.0# ping -c 1 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.812 ms

--- 192.168.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.812/0.812/0.812/0.000 MsgRcvd
bash-5.0# ping -c 1 192.168.1.3
PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_seq=1 ttl=64 time=1.40 ms

--- 192.168.1.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.403/1.403/1.403/0.000 ms
bash-5.0# ping -c 1 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=1.62 ms

--- 192.168.1.4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.619/1.619/1.619/0.000 ms

L3 service ping (vlan 200)

bash-5.0# ping -c 1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.053 ms

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms
bash-5.0# ^C
bash-5.0# ping -c 1 192.168.2.2
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.611 ms

--- 192.168.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.611/0.611/0.611/0.000 ms

-------------------------------------------
Traceroute ///
-------------------------------------------

L3 service traceroute (vlan 200)

bash-5.0# traceroute 192.168.200.1
traceroute to 192.168.200.1 (192.168.200.1), 30 hops max, 46 byte packets
1 192.168.2.254 (192.168.2.254) 0.012 ms 0.010 ms 0.007 ms
2 192.168.200.254 (192.168.200.254) 2.751 ms * 2.549 ms
3 192.168.200.1 (192.168.200.1) 2.084 ms 1.805 ms 1.761 ms

</syntaxhighlight>To come back on the setup, we wanted to implement solutions inspired by the <nowiki>RFC 7938</nowiki> rfc7938 (see section 5.2.1)

* eBGP will be used for Underlay and Overlay
* eBGP single-hop sessions are established over direct point-to-point links interconnecting the network nodes, no multi-hop or loopback sessions are used, even in the case of multiple links between the same pair of nodes.
* Private Use ASNs from the range 64512-65534 are used to avoid ASN conflicts.
* A single ASN is allocated to all of the Clos topology's Tier 1 devices.
* A unique ASN is allocated to each set of Tier 2 devices in the same cluster.
* A unique ASN is allocated to every Tier 3 device (e.g., ToR) in this topology.

In add, depending of network nodes, it will be interesting to :

* Add fast-failover option for underlay + add link-debounce on links to limit possible impacts in case of link flaps. BFD could be considered in some cases
* Have the best ECMP hashing algorythm on leaf nodes. On tier-1/tier-2 levels hashing will be based on underlay part (trafic inter-vtep), so L3-L4 decision, but some models can retreive the vni-id if I well remember.
* Implement the "multipath relax" since we could have multiple AS paths for a same destination.
* Implement gshut community on underlay (on tier-1/tier-2) or at least policies to prepend underlay routes to avoid some nodes in case of maintenance.

== Thoughts/Points to deepend ==
On leaf nodes, LACP over multiple nodes is feasible within ESI, but impact on incidents/maintenance can vary depending on the vendors.

'''Tests need to be done to check that.'''

Some sysadmin would like to KISS and have client servers connected in active/passive bonding, with eventually single-leaf ports aggregation.

'''Note :''' NH ESI in FIB management on ingress nodes can have some limitations (numbers of installable next-hop, load-balancing, etc. oftenly linked to the node chipset)

Another important engineering point to define at the beginning is : '''Do we authorize multisite L2 propagation or not ?'''

From my point-of-view, the smaller are the L2 segments the better it is. I would suggest Local L2 segments wit anycast gateways provided by Leaf switches, with L3 routing between DCs within EVPN type5 routes sharing only. Nevertheless, some business requirements for endcustomers can keep extended L2 alive.

We have to be careful when implementing Consistent hashing. ex: <nowiki>https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-44/Layer-3/Routing/Equal-Cost-Multipath-Load-Sharing-Hardware-ECMP/</nowiki>

If possible activate it per VRF, not globally, and restrict on dedicated subnets << depends of the vendor, it can be dangerous regarding memory management

Think at configuring the bgp option to not advertise routes to peers before they are installed in FIB.

We can implement leaf roles in some cases, to be able to do "light traffic engineering " on DCI links in some setup.

These roles will be managed within bgp communities added to VTEP /32. Look at the diagram below. Depending of the colour, the eBGP sessions between SupS will prepend or not the VTEP IPs announced to the remote DC

[[File:Fabric-roles.jpg|ohne|mini|1200x1200px]]

<blockquote>'''To be continued'''</blockquote>

== Lab archives ==
<blockquote>Reminder : following config are for lab, so please don't pay attention on typos/lack of security params/unused config. Thanks \o/</blockquote>

=== Credentials ===
<syntaxhighlight lang="bash">
SRLinux : admin/<no_password>

CVX : cumulus/cumulus
</syntaxhighlight>

=== Lab archives ===
layer 2 (vlan 100) + layer 3 (vlan 200) lab, 10/12/2021
[[https://wiki.nesevo.com/images/6/66/20211008_lab-nokia-cvx.zip 20211008_lab-nokia-cvx.zip]]
<u>If you want to exchange about that lab, those topics, please don't hesitate to contact us</u>

Nesevo Virtual Labs

2021-10-29T08:32:58Z

PierreLancastre: /* Nokia/Cumulus Multi-Vendor Lab */

[[File:nesevo-labs-v2.png]]

= Nokia/Cumulus Multi-Vendor Lab =

[[Cumulus-Nokia-Lab]]

The targets of this lab are to :

* mix Nokia SR-Linux and Cumulus VXLAN nodes, in spine or leaf roles
* make this environment work for different kinds of setup : cross-DC L2, differentiated L3 gateways per leaf per VRF, anycast gateway cross or intra DC, etc.
* spot compatibility issues if existing

To reach those goals, we'll use the Nokia lab tool "containerlab" https://containerlab.srlinux.dev/, labs from https://clabs.netdevops.me/ helped a lot. Many thanks to the contributors.

We'll need a Linux host with Docker installed on it + kvm virtualization enhancements activated (important if your host is a VM). If virtualization optimization is not ok, you won't be able to launch the most of devices.

Installation details are referenced there : https://containerlab.srlinux.dev/install/

Once installed, you'll need to register the different NOS images you'll use. Cumulus image is present on a public registrery, but Nokia/Arista/Juniper are not, you'll need a vendor support account to retreive them. Once downloaded (.tgz file typically) you can "docker load" the image :

For private images :<syntaxhighlight lang="bash">
sudo docker load -i 21.6.1-235.tar.xz /* (Nokia) */
sudo docker load -i cEOS64-lab-4.26.1F.tar /* (Arista) */
</syntaxhighlight>For public images :<syntaxhighlight lang="bash">
sudo docker pull networkop/cx:4.3.0 /* (Cumulus) */
</syntaxhighlight>

To verify your installed images, you can run "sudo docker images"<syntaxhighlight lang="bash">
plancastre@containerlab:~/00.LAB/senss$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
networkop/cx 4.3.0 a40541ed15ba 2 weeks ago 733MB
srlinux 21.6.1-235 e29aed30e83f 3 weeks ago 1.14GB
networkop/ignite dev dcc5eff29355 4 weeks ago 37.9MB
networkop/kernel 4.19 e794d8875f83 3 months ago 88MB
frrouting/frr v7.5.1 c3e13a4c5918 3 months ago 123MB

</syntaxhighlight>Once you've got all your images, to run the lab, you'll need a .yml file defining the topology to run.

In our lab, we've started with the 5-clos lab file from Nokia site, which we've forked.

We can see there is a branch "kinds" where we can define templates of nodes, in order to not rewrite whole parameters when creating nodes. For the images, it's important to take the repository name + tag.

'''Note:''' If the tag is "latest" it is not needed.

All supported "kinds" are referenced there : https://containerlab.srlinux.dev/manual/kinds/kinds/

'''Note:''' yml is indentation sensitive, so please respect the "double space" used to define hierarchy. Below the lab diagram + the yml file associated

== Infos on nodes ==

=== L3 infos ===
{| class="wikitable"
|+
!Device
!OS
!ASN
!VTEP IP
!swp1 or eth1/1
!swp2 or eth1/2
!swp3 or eth1/3
!SVI
|-
|client1
|Linux
|<none>
|<none>
|vlan100: 192.168.1.1/24
vlan200: 192.168.2.1/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client2
|Linux
|<none>
|<none>
|vlan100: 192.168.1.2/24
vlan200: 192.168.2.2/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client3
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.1/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|client4
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.2/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|leaf1
|CVX
|65011
|10.255.255.5/32
|10.254.254.8/31
|10.254.254.10/31
|trunk vlan 100,200
|vlan200 : 192.168.2.254/24
|-
|leaf2
|SRLinux
|65012
|10.255.255.6/32
|10.254.254.12/31
|10.254.254.14/31
|trunk vlan 100,200
|irb0.200 : 192.168.2.254/24
|-
|leaf3
|CVX
|65013
|10.255.255.7/32
|10.254.254.22/31
|10.254.254.24/31
|trunk vlan 100,200
|vlan200 : 192.168.200.254/24
|-
|leaf4
|SRLinux
|65014
|10.255.255.8/32
|10.254.254.26/31
|10.254.254.28/31
|trunk vlan 100,200
|irb0.200 : 192.168.200.254/24
|-
|spine1
|CVX
|65001
|10.255.255.1/32
|10.254.254.9/31
|10.254.254.13/31
|10.254.254.0/31
|<none>
|-
|spine2
|CVX
|65001
|10.255.255.2/32
|10.254.254.11/31
|10.254.254.15/31
|10.254.254.16/31
|<none>
|-
|spine3
|SRLinux
|65002
|10.255.255.3/32
|10.254.254.23/31
|10.254.254.27/31
|10.254.254.18/31
|<none>
|-
|spine4
|SRLinux
|65002
|10.255.255.4/32
|10.254.254.25/31
|10.254.254.29/31
|10.254.254.20/31
|<none>
|-
|superspine1
|CVX
|65003
|10.255.255.9/32
|10.254.254.1/31
|10.254.254.19/31
|<none>
|<none>
|-
|superspine2
|SRLinux
|65003
|10.255.255.10/32
|10.254.254.17/31
|10.254.254.21/31
|<none>
|<none>
|}

=== VXLAN/BGP infos ===
Note : in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>
{| class="wikitable"
|+
!service
!vlan id
!route-target
!VNI
!Commentaires
|-
|L2 cross-DC
|100
|target:65000:100
|100
|the 4 clients are in the same broadcast domain / subnet
|-
|L3VPN cros-DC
|200
|target:65000:200
target:65000:10200
|200 (for L2)
10200 (for routing)
|the 4 clients are separated in 2 different subnets. Traffic is routed within dedicated VNI
To be noted that in the current setup VMTO is not implemented, so cross DC traffic is not optimized.
|}

== Lab diagram ==
[[File:Lab-srl-diagram.png|ohne|mini|600x600px]]

== yml lab file ==
<syntaxhighlight lang="yaml">
# topology documentation: http://containerlab.srlinux.dev/lab-examples/min-5clos/
name: lab-srl-cum

topology:
kinds:
srl:
image: srlinux:21.6.1-235
linux:
image: ghcr.io/hellt/network-multitool
cvx:
image: networkop/cx:4.3.0
runtime: docker
nodes:
leaf1:
kind: cvx
leaf2:
kind: srl
type: ixrd2
leaf3:
kind: cvx
leaf4:
kind: srl
type: ixrd2

spine1:
kind: cvx
spine2:
kind: cvx
spine3:
kind: srl
type: ixr6
spine4:
kind: srl
type: ixr6

superspine1:
kind: cvx
superspine2:
kind: srl
type: ixr6

client1:
kind: linux
client2:
kind: linux
client3:
kind: linux
client4:
kind: linux

links:
# leaf to spine links POD1
- endpoints: ["leaf1:swp1", "spine1:swp1"]
- endpoints: ["leaf1:swp2", "spine2:swp1"]
- endpoints: ["leaf2:e1-1", "spine1:swp2"]
- endpoints: ["leaf2:e1-2", "spine2:swp2"]

# spine to superspine links POD1
- endpoints: ["spine1:swp3", "superspine1:swp1"]
- endpoints: ["spine2:swp3", "superspine2:e1-1"]

# leaf to spine links POD2
- endpoints: ["leaf3:swp1", "spine3:e1-1"]
- endpoints: ["leaf3:swp2", "spine4:e1-1"]
- endpoints: ["leaf4:e1-1", "spine3:e1-2"]
- endpoints: ["leaf4:e1-2", "spine4:e1-2"]

# spine to superspine links POD2
- endpoints: ["spine3:e1-3", "superspine1:swp2"]
- endpoints: ["spine4:e1-3", "superspine2:e1-2"]

# client connection links
- endpoints: ["client1:eth1", "leaf1:swp3"]
- endpoints: ["client2:eth1", "leaf2:e1-3"]
- endpoints: ["client3:eth1", "leaf3:swp3"]
- endpoints: ["client4:eth1", "leaf4:e1-3"]

</syntaxhighlight>

Once the nodes are defined, we have to define the needed connections between the nodes.

Each endpoint is defined as '<nowiki/>'''<nowiki/>'<node_name>:<interface_name><nowiki>''</nowiki>'''

When all is ok, you can launch in suduoer the topology.

For example :<syntaxhighlight lang="bash">
sudo containerlab deploy --topo topo.clab.yml --reconfigure

Note: The option "reconfigure" is used there in order to not have to destroy instances when updating topology/relaunching.
</syntaxhighlight>Another info regarding the yml file, it's possible to quickly generate files with the '''"containerlab gen -n 3tier --nodes 4,2,1"''' command for example. https://containerlab.srlinux.dev/cmd/generate/

To be noted that the "tier-x" notion in containerlab is inversed regarding the rfc 7938 (tier1 is superspine level into the rfc, tier1 is leaf level in nokia tool)

At the end of the topology launch, you can retreive the details :<syntaxhighlight lang="yaml">
INFO[0011] Writing /etc/hosts file
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| # | Name | Container ID | Image | Kind | Group | State | IPv4 Address | IPv6 Address |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| 1 | clab-lab-srl-cum-client1 | 25e2bbf6b13f | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
| 2 | clab-lab-srl-cum-client2 | 17d82d5a6e21 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.4/24 | 2001:172:20:20::4/64 |
| 3 | clab-lab-srl-cum-client3 | f8d78527c1b1 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.6/24 | 2001:172:20:20::6/64 |
| 4 | clab-lab-srl-cum-client4 | 913a0172d0c5 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 5 | clab-lab-srl-cum-leaf1 | 9ffc334af2b1 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.9/24 | 2001:172:20:20::9/64 |
| 6 | clab-lab-srl-cum-leaf2 | 7d3eb855d749 | srlinux:21.6.1-235 | srl | | running | 172.20.20.14/24 | 2001:172:20:20::e/64 |
| 7 | clab-lab-srl-cum-leaf3 | e0d36ffc606a | networkop/cx:4.3.0 | cvx | | running | 172.20.20.7/24 | 2001:172:20:20::7/64 |
| 8 | clab-lab-srl-cum-leaf4 | e6753b1d099e | srlinux:21.6.1-235 | srl | | running | 172.20.20.12/24 | 2001:172:20:20::c/64 |
| 9 | clab-lab-srl-cum-spine1 | a7e63993d052 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.5/24 | 2001:172:20:20::5/64 |
| 10 | clab-lab-srl-cum-spine2 | 3124250d5b75 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
| 11 | clab-lab-srl-cum-spine3 | 5e300ca3f66f | srlinux:21.6.1-235 | srl | | running | 172.20.20.11/24 | 2001:172:20:20::b/64 |
| 12 | clab-lab-srl-cum-spine4 | 0a59e60a596b | srlinux:21.6.1-235 | srl | | running | 172.20.20.13/24 | 2001:172:20:20::d/64 |
| 13 | clab-lab-srl-cum-superspine1 | 3d9935413fc2 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.8/24 | 2001:172:20:20::8/64 |
| 14 | clab-lab-srl-cum-superspine2 | 5bc58d423e2c | srlinux:21.6.1-235 | srl | | running | 172.20.20.15/24 | 2001:172:20:20::f/64 |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
</syntaxhighlight>The containerlab program creates the updates into the host file + generates an ansible inventory file.

There are different ways to connect onto the docker instances, within docker exec commands or SSH directly the IP address related above.

For example :<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-spine1 bash
</syntaxhighlight>-->>> allows to launch a bash on the cumulus device (then you run the different net commands)<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-leaf2 sr_cli
</syntaxhighlight>-->>> allows to launch the cli on a SRLinux device

To end this setup preparation, the containerlab tool integrates an option to launch a webserver showing the lab topology in graph + node details

By default, it's listening on port 50080 of the host

Then, you can access to the following web page showing the topology + lab infos

[[File:Lab graph tool.jpg|ohne|mini|900x900px]]

Finally, if you want to stop a lab, please run '''<nowiki>''</nowiki>containerlab destroy --topo <topology>.yml<nowiki>''</nowiki>'''

At this step, nodes are running, connections are done, but configurations are empty.

'''So, it's time to configure them :)'''

Configuration files are attached to this blog. Once validated, they are exportable and can be called by the tool when launching the lab. Below how to do it :

Firstable, once you ve finished your "hand made config", you can run the following command to save it :<syntaxhighlight lang="bash">
containerlab save -t quickstart.clab.yml
</syntaxhighlight>'''To be noted it's only working for Nokia nodes in this lab'''. CVX configurations will have to be called by another method (or pushed manually)

It will create one folder per node into the lab folder which gives the following tree :
[[File:Topology-srl.jpg|ohne|mini]]

'''Note''': For Cumulus, you can apparently retrieve configurations files within adding the following parameters into lab config file. '''To be tested.''' some lab samples here : https://github.com/hellt/clabs/blob/main/labs/cvx04/symm-mh.clab.yml

Now we have the lab running, below the different commands to check that all is working as expected : <syntaxhighlight lang="yaml">
-------------------------------------------
Check L2 tables ///
-------------------------------------------

Nokia :

A:leaf2# show network-instance mac-vrf-100 bridge-table mac-table all
----------------------------------------------------------------------------------------------------------------------------------------------------------
Mac-table of network instance mac-vrf-100
----------------------------------------------------------------------------------------------------------------------------------------------------------
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
| Address | Destination | Dest Index | Type | Active | Aging | Last Update |
+====================+=========================================+============+=============+=========+========+=========================================+
| AA:C1:AB:0C:60:D6 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T13:06:17.000Z |
| | vtep:10.255.255.8 vni:100 | 67 | | | | |
| AA:C1:AB:23:31:04 | ethernet-1/3.100 | 2 | learnt | true | 300 | 2021-10-08T13:06:10.000Z |
| AA:C1:AB:4B:FF:71 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:55:28.000Z |
| | vtep:10.255.255.5 vni:100 | 60 | | | | |
| AA:C1:AB:B5:79:50 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:43:37.000Z |
| | vtep:10.255.255.7 vni:100 | 64 | | | | |
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 0 Total 0 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 1 Total 1 Active
Total Evpn Macs : 3 Total 3 Active
Total Evpn static Macs : 0 Total 0 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 4 Total 4 Active
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : type learnt matches the local learnt mac, evpn are learnt within VXLAN

Cumulus :

root@leaf1-cum-dc1:/# net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
100 bridge swp3 aa:c1:ab:4b:ff:71 00:00:02
100 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:17:21
100 bridge vni100 22:2d:67:f1:bb:26 permanent 02:17:21
100 bridge vni100 aa:c1:ab:0c:60:d6 extern_learn <1 sec
100 bridge vni100 aa:c1:ab:23:31:04 extern_learn 00:00:07
100 bridge vni100 aa:c1:ab:b5:79:50 extern_learn 00:13:28

root@leaf1-cum-dc1:/# net show bridge macs vlan 200

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
200 bridge swp3 aa:c1:ab:4b:ff:71 00:08:46
200 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:16:36
200 bridge vni200 00:00:5e:00:01:01 static sticky 00:14:26
200 bridge vni200 1e:0d:aa:46:53:b1 permanent 02:16:36
200 bridge vni200 02:77:6b:ff:00:41 static sticky 01:00:52
200 bridge vni200 02:f8:b8:ff:00:41 static sticky 00:14:26
200 bridge vni200 aa:c1:ab:b5:79:50 extern_learn 00:11:51
200 bridge vni10200 02:77:6b:ff:00:00 extern_learn 00:53:02
200 bridge vni10200 02:f8:b8:ff:00:00 extern_learn 00:14:26
200 bridge vni10200 86:5e:aa:ca:50:be permanent 02:16:36

Note : the macs with permanent flag are local interfaces, physical ou logic (vtep)
Note2 : extern_learn mac are the one learned via the VXLAN infra (and mapped to the vnixxx interface)

-------------------------------------------
Check arp tables ///
-------------------------------------------

Nokia :

:leaf2# show arpnd arp-entries
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| ethernet-1/1 | 0 | 10.254.254.13 | dynamic | AA:C1:AB:59:65:EA | 3 hours from now |
| ethernet-1/2 | 0 | 10.254.254.15 | dynamic | AA:C1:AB:CB:F8:8A | 3 hours from now |
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
| mgmt0 | 0 | 172.20.20.1 | dynamic | 02:42:63:9E:74:03 | 2 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 5 (0 static, 5 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : on Nokia you have to sort the entries basing on subinterface and the retrieve the L3VPN attached

A:leaf2# show arpnd arp-entries subinterface 200
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 2 (0 static, 2 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Cumulus :

Here we retrieve all arp infos linked to evpn

root@leaf1-cum-dc1:/# net show evpn arp-cache vni all
VNI 100 #ARP (IPv4 and IPv6, local and remote) 2

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.1.3 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0
192.168.1.1 local active aa:c1:ab:4b:ff:71 0/0

VNI 200 #ARP (IPv4 and IPv6, local and remote) 4

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.2.1 local active aa:c1:ab:4b:ff:71 0/0
192.168.200.254 remote active 00:00:5e:00:01:01 10.255.255.8 0/0
192.168.2.254 remote active 00:00:5e:00:01:01 10.255.255.6 0/0
192.168.200.1 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0

VNI 400 #ARP (IPv4 and IPv6, local and remote) 0

-------------------------------------------
Check bgp sessions' status - underlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep ipv4
| default | 10.254.254.19 | udl-superspine | S | 65003 | established | 0d:0h:22m:44s | ipv4-unicast | [13/4/4] |
| default | 10.254.254.22 | udl-leaf | S | 65013 | established | 0d:0h:22m:52s | ipv4-unicast | [13/1/4] |

Cumulus :

root@leaf3:/# net show bgp ipv4 unicast summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 13
RIB entries 25, using 4800 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.254.254.23 4 65002 466 403 0 0 0 00:19:34 4 13
10.254.254.25 4 65002 434 355 0 0 0 00:17:10 8 13

-------------------------------------------
Check bgp sessions' status - overlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep evpn
| default | 10.255.255.7 | ovl-leaf | S | 65013 | established | 0d:0h:24m:14s| evpn | [15/0/11] |
| default | 10.255.255.9 | ovl-superspine | S | 65003 | established | 0d:0h:24m:7s | evpn | [11/0/4] |

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.255.255.3 4 65002 5618 4742 0 0 0 03:53:35 19 27
10.255.255.4 4 65002 5565 4694 0 0 0 03:51:11 19 27

-------------------------------------------
Check EVPN L2 tables///
-------------------------------------------

Nokia :

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn route type 2
BGP table version is 17, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:100
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
Route Distinguisher: 10.255.255.5:200
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
Route Distinguisher: 10.255.255.6:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
Route Distinguisher: 10.255.255.7:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100
Route Distinguisher: 10.255.255.7:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.200.1]
10.255.255.7 32768 i
ET:8 RT:65000:200
Route Distinguisher: 10.255.255.8:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC

and if we focus on our vni 100

root@leaf3-cum-dc1:/# net show bgp evpn route vni 100 type 2
BGP table version is 48, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100

The additional route type "MAC+IP" is optional and is useful/mandatory when you do anycast irb.

-------------------------------------------
Check tables EVPN L3 ///
-------------------------------------------

Nokia :

--{ + running }--[ ]--
A:leaf2# show network-instance ip-vrf-200 route-table all
-----------------------------------------------------------------------------------------------------------------------------------------------------------
IPv4 Unicast route table of network instance ip-vrf-200
-----------------------------------------------------------------------------------------------------------------------------------------------------------
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
| Prefix | ID | Route Type | Route Owner | Best/Fib- | Metric | Pref | Next-hop | Next-hop |
| | | | | status(slot) | | | (Type) | Interface |
+=======================+=======+============+======================+======================+==========+=========+==============+==============+
| 192.168.2.0/24 | 9 | local | net_inst_mgr | True/success | 0 | 0 | 192.168.2.25 | irb0.200 |
| | | | | | | | 4 (direct) | |
| 192.168.2.254/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (extract) | |
| 192.168.2.255/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (broadcast) | |
| 192.168.200.0/24 | 0 | bgp-evpn | bgp_evpn_mgr | True/success | 0 | 170 | 10.255.255.8 | None |
| | | | | | | | (indirect/vx | |
| | | | | | | | lan) | |
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
-----------------------------------------------------------------------------------------------------------------------------------------------------------
4 IPv4 routes total
4 IPv4 prefixes with active routes
0 IPv4 prefixes with active ECMP routes
-----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--
A:leaf2#

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf1-cum-dc1:/# net show route vrf ip-vrf-200
show ip route vrf ip-vrf-200
=============================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF ip-vrf-200:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 04:48:42
B 192.168.2.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 04:27:26
C>* 192.168.2.0/24 is directly connected, vlan200, 04:48:42
B>* 192.168.200.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:48:49

root@leaf1-cum-dc1:/# net show bgp evpn route type 5
BGP table version is 3, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.6:10200
* [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
*> [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
Route Distinguisher: 10.255.255.8:10200
*> [5]:[0]:[24]:[192.168.200.0]
10.255.255.8 0 65001 65003 65002 65014 i
RT:65000:10200 ET:8 Rmac:02:f8:b8:ff:00:00

-------------------------------------------
Check arp hosts ///
-------------------------------------------

From client #1 /

bash-5.0# arp
Address HWtype HWaddress Flags Mask Iface
192.168.1.3 ether aa:c1:ab:b5:79:50 C eth1.100
192.168.1.4 ether aa:c1:ab:0c:60:d6 C eth1.100
192.168.2.2 ether aa:c1:ab:23:31:04 C eth1.200
192.168.2.254 ether aa:c1:ab:4f:f2:e7 C eth1.200
192.168.1.2 ether aa:c1:ab:23:31:04 C eth1.100
containerlab.local ether 02:42:63:9e:74:03 C eth0

-------------------------------------------
Ping ///
-------------------------------------------

From client #1 /

L2 service ping (vlan 100) ----------------------

bash-5.0# ping -c 1 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.812 ms

--- 192.168.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.812/0.812/0.812/0.000 MsgRcvd
bash-5.0# ping -c 1 192.168.1.3
PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_seq=1 ttl=64 time=1.40 ms

--- 192.168.1.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.403/1.403/1.403/0.000 ms
bash-5.0# ping -c 1 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=1.62 ms

--- 192.168.1.4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.619/1.619/1.619/0.000 ms

L3 service ping (vlan 200)

bash-5.0# ping -c 1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.053 ms

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms
bash-5.0# ^C
bash-5.0# ping -c 1 192.168.2.2
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.611 ms

--- 192.168.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.611/0.611/0.611/0.000 ms

-------------------------------------------
Traceroute ///
-------------------------------------------

L3 service traceroute (vlan 200)

bash-5.0# traceroute 192.168.200.1
traceroute to 192.168.200.1 (192.168.200.1), 30 hops max, 46 byte packets
1 192.168.2.254 (192.168.2.254) 0.012 ms 0.010 ms 0.007 ms
2 192.168.200.254 (192.168.200.254) 2.751 ms * 2.549 ms
3 192.168.200.1 (192.168.200.1) 2.084 ms 1.805 ms 1.761 ms

</syntaxhighlight>To come back on the setup, we wanted to implement solutions inspired by the <nowiki>RFC 7938</nowiki> rfc7938 (see section 5.2.1)

* eBGP will be used for Underlay and Overlay
* eBGP single-hop sessions are established over direct point-to-point links interconnecting the network nodes, no multi-hop or loopback sessions are used, even in the case of multiple links between the same pair of nodes.
* Private Use ASNs from the range 64512-65534 are used to avoid ASN conflicts.
* A single ASN is allocated to all of the Clos topology's Tier 1 devices.
* A unique ASN is allocated to each set of Tier 2 devices in the same cluster.
* A unique ASN is allocated to every Tier 3 device (e.g., ToR) in this topology.

In add, depending of network nodes, it will be interesting to :

* Add fast-failover option for underlay + add link-debounce on links to limit possible impacts in case of link flaps. BFD could be considered in some cases
* Have the best ECMP hashing algorythm on leaf nodes. On tier-1/tier-2 levels hashing will be based on underlay part (trafic inter-vtep), so L3-L4 decision, but some models can retreive the vni-id if I well remember.
* Implement the "multipath relax" since we could have multiple AS paths for a same destination.
* Implement gshut community on underlay (on tier-1/tier-2) or at least policies to prepend underlay routes to avoid some nodes in case of maintenance.

== Thoughts/Points to deepend ==
On leaf nodes, LACP over multiple nodes is feasible within ESI, but impact on incidents/maintenance can vary depending on the vendors.

'''Tests need to be done to check that.'''

Some sysadmin would like to KISS and have client servers connected in active/passive bonding, with eventually single-leaf ports aggregation.

'''Note :''' NH ESI in FIB management on ingress nodes can have some limitations (numbers of installable next-hop, load-balancing, etc. oftenly linked to the node chipset)

Another important engineering point to define at the beginning is : '''Do we authorize multisite L2 propagation or not ?'''

From my point-of-view, the smaller are the L2 segments the better it is. I would suggest Local L2 segments wit anycast gateways provided by Leaf switches, with L3 routing between DCs within EVPN type5 routes sharing only. Nevertheless, some business requirements for endcustomers can keep extended L2 alive.

We have to be careful when implementing Consistent hashing. ex: <nowiki>https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-44/Layer-3/Routing/Equal-Cost-Multipath-Load-Sharing-Hardware-ECMP/</nowiki>

If possible activate it per VRF, not globally, and restrict on dedicated subnets << depends of the vendor, it can be dangerous regarding memory management

Think at configuring the bgp option to not advertise routes to peers before they are installed in FIB.

We can implement leaf roles in some cases, to be able to do "light traffic engineering " on DCI links in some setup.

These roles will be managed within bgp communities added to VTEP /32. Look at the diagram below. Depending of the colour, the eBGP sessions between SupS will prepend or not the VTEP IPs announced to the remote DC

[[File:Fabric-roles.jpg|ohne|mini|1200x1200px]]

<blockquote>'''To be continued'''</blockquote>

== Lab archives ==
<blockquote>Reminder : following config are for lab, so please don't pay attention on typos/lack of security params/unused config. Thanks \o/</blockquote>

=== Credentials ===
<syntaxhighlight lang="bash">
SRLinux : admin/<no_password>

CVX : cumulus/cumulus
</syntaxhighlight>

=== Lab archives ===
layer 2 (vlan 100) + layer 3 (vlan 200) lab, 10/12/2021
[[https://wiki.nesevo.com/images/6/66/20211008_lab-nokia-cvx.zip 20211008_lab-nokia-cvx.zip]]
<u>If you want to exchange about that lab, those topics, please don't hesitate to contact us</u>

= SONiC Lab =

<span><to be noted that those informations are included into the readme.md file into the archive></span>

== Lab diagram ==
[[File:SONIC_LAB.jpg]]

Lab archive [https://wiki.nesevo.com/images/0/0a/Sonic-lab-20201030.zip Sonic-lab-20201030.zip]

This lab is a forked lab from many found on @, starting from Microsoft one : https://github.com/Azure/SONiC/wiki/SONiC-P4-Software-Switch

The target of this lab is to simulate a simple IP/Fabric in SPINE/LEAF topology. Each Leaf provide local L2 connectivity + L3 gateways to hosts. Flows between hosts from different leaf are L3 routed, in a "node path" LEAF-SPINE-LEAF.

Nothing complicated, but it allows to put a first foot into SONiC world :)

For the moment, as far as we know, MC-LAG + EVPN/VXLAN features are only available on entreprise SONiC versions. Tests to come once physic switches in enterprise version will be available in our lab.

The diagram of the architecture is into the zip file (SONIC_LAB.jpg)

To run the lab you need :

* to have installed docker by using ./install_docker_ovs.sh
* to run the ./load_image.sh in order to get the Sonic image

Note : don't change file / folders names of network devices (leafX, spineX)

Once you're ready, please run the lab by launching "./start.sh"

The script :<div></div>
* launches the containers without Docket network features
* create network namespaces
* build linux bridges to interconnect the containers
* configure host IPs
* launch startup sonic scripts into sonic devices
* once devices started, checks + ping tests are run. If all BGP sessions are ok on spine + all ping are fine, the setup works well.
The "start" script has several comments, perhaps without the right vocabulary, so please be tolerant :)

If you want to stop the lab, please launch "./stop.sh"

Below some useful commands to run for sonic containers :

=== <span>run ping</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> ping <ip>
</syntaxhighlight>
This command triggers ping into the targetted container :
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it leaf1 ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.052 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.034 ms
^C--- 192.168.1.1 ping statistics ---
</syntaxhighlight>

=== <span>check bgp sessions</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip bgp sum"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print the bgp status.
Below, we run it on one spine, you should have 4 sessiosn UP (for the 4 leaves)
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip bgp sum"
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 6409 6411 0 0 0 01:46:46 3
10.1.0.2 4 65102 6406 6410 0 0 0 01:46:44 3
10.1.0.4 4 65103 6407 6410 0 0 0 01:46:44 4
10.1.0.6 4 65104 6406 6412 0 0 0 01:46:46 1
Total number of neighbors 4
</syntaxhighlight>

=== <span>check ip route</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip route"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print its routing table?

<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,
> - selected route, * - FIB route
C>* 10.1.0.0/31 is directly connected, Vlan11
C>* 10.1.0.2/31 is directly connected, Vlan21
C>* 10.1.0.4/31 is directly connected, Vlan31
C>* 10.1.0.6/31 is directly connected, Vlan41
C>* 127.0.0.0/8 is directly connected, lo
B>* 192.168.1.0/24 [20/0] via 10.1.0.0, Vlan11, 01:47:23
B>* 192.168.2.0/24 [20/0] via 10.1.0.2, Vlan21, 01:47:21
B>* 192.168.3.0/24 [20/0] via 10.1.0.4, Vlan31, 01:47:21
B>* 192.168.4.0/24 [20/0] via 10.1.0.6, Vlan41, 01:47:23
</syntaxhighlight>
For more commands check SONiC docs : https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md

To finish, below the output of the start.sh when all goes well<div></div>
plancastre@dockerhost:~/00.Projects/p4-version-pla$ ./start.sh
---------------------------------------------------
Creating the containers: starting
---------------------------------------------------
0b4bbb3c6cfc9895466ea13a9e940d1a176d66f45deee9752e9b4732d735bb03
5ca9486baab3e8a5259062167132a944eee717875d3cc1ef29fe490cba3184c1
46c9c09dba27a32715b9fbe0bc434a9284b9d59317d278df693f425cb669f037
898e6afbcb2977115acbba2a69d069796cfd70fb90228b6c18e09ba4584365f7
eb9b3ef6d14784c9ffe4d9d5df3e921a24559889cb8770d226263aad6761182a
ae5f3b2c628da687eaabb331dc2cd989e71bcb386854e7a0be6ec587df99d49a
122197bffd7566a01ce3b6565fdfb6eb893aab7d5236f14e5a9bbbe95037ced2
a604520a080e2a2184598ad1d915c0c182d16e5185033373e95a4772d8978a67
0c49e18d78b2aa7c8ef8e937b60285aa56ec0bf4a8530226285270a085e11d5c
eb66336b8c6bd8333f1f4f685369e6cac973b817c35bb7126ac4d7ba72ef71bf
---------------------------------------------------
Retreiving PID from docker containers
---------------------------------------------------
---------------------------------------------------
Creating network namespaces for containers
---------------------------------------------------
List of created namespaces
ns_host4
ns_host3
ns_host2
ns_host1
ns_spine2
ns_spine1
ns_leaf4
ns_leaf3
ns_leaf2
ns_leaf1
---------------------------------------------------
Creating network connectivity between containers
---------------------------------------------------
---------------------------------------------------
Now assigning IPs to hosts (host1 to host4)
---------------------------------------------------
---------------------------------------------------
Configuration of containers is now finished
---------------------------------------------------
---------------------------------------------------
Now booting switches, please wait 2 minutes for switches to load
---------------------------------------------------
---------------------------------------------------
2 minutes period is finished
---------------------------------------------------
---------------------------------------------------
Now fixing the iptables
---------------------------------------------------
---------------------------------------------------
Backend actions are now finished
---------------------------------------------------
---------------------------------------------------
Now we'll check in 15 sec that BGP peers are all Up
---------------------------------------------------
---------------------------------------------------
SPINE1 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 94 97 0 0 0 00:01:28 4
10.1.0.2 4 65102 95 97 0 0 0 00:01:28 5
10.1.0.4 4 65103 95 97 0 0 0 00:01:28 5
10.1.0.6 4 65104 95 97 0 0 0 00:01:28 5
Total number of neighbors 4
---------------------------------------------------
SPINE2 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.2.0.1, local AS number 65002
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.2.0.0 4 65101 93 96 0 0 0 00:01:29 2
10.2.0.2 4 65102 94 100 0 0 0 00:01:32 1
10.2.0.4 4 65103 94 100 0 0 0 00:01:32 1
10.2.0.6 4 65104 94 100 0 0 0 00:01:32 1
Total number of neighbors 4
---------------------------------------------------
Now if all is ok, we'll do some ping tests
---------------------------------------------------
---------------------------------------------------
host1 ping its gateway
---------------------------------------------------
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=17.6 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=9.14 ms
--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 9.145/13.390/17.636/4.247 ms
---------------------------------------------------
host2 ping its gateway
---------------------------------------------------
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=23.6 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=6.21 ms
--- 192.168.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 6.218/14.950/23.683/8.733 ms
---------------------------------------------------
host3 ping its gateway
---------------------------------------------------
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=18.8 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=12.7 ms
--- 192.168.3.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.716/15.764/18.812/3.048 ms
---------------------------------------------------
host4 ping its gateway
---------------------------------------------------
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.
64 bytes from 192.168.4.1: icmp_seq=1 ttl=64 time=17.7 ms
64 bytes from 192.168.4.1: icmp_seq=2 ttl=64 time=4.46 ms
--- 192.168.4.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.464/11.107/17.750/6.643 ms
---------------------------------------------------
host1 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=16.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.049/17.253/19.532/1.615 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=15.5 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=14.3 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.312/15.592/16.939/1.083 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=20.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=16.8 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.048/17.871/20.739/2.058 ms
---------------------------------------------------
host2 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=16.6 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.222/16.583/16.924/0.286 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.6 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=18.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=26.8 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.620/20.549/26.887/4.524 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=33.8 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=27.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=20.2 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 20.216/27.259/33.818/5.566 ms
---------------------------------------------------
host3 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=14.5 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=15.5 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.535/16.549/19.541/2.162 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.4 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=14.6 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=18.2 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.643/16.458/18.278/1.491 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=15.3 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=14.1 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=13.1 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 13.171/14.234/15.334/0.883 ms
---------------------------------------------------
host4 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=21.9 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=21.3 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=14.6 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 14.619/19.284/21.932/3.310 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=14.8 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=18.4 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=14.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.148/15.786/18.409/1.879 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=15.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=18.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 15.117/16.788/18.962/1.612 ms
---------------------------------------------------
If bot spine have all BGP sessions and all ping succeeded, all works well
Congrats ! please check readme file to find some details
---------------------------------------------------

Nesevo Virtual Labs

2021-10-29T08:20:35Z

PierreLancastre: /* Nokia/Cumulus Multi-Vendor Lab */

[[File:nesevo-labs-v2.png]]

= Nokia/Cumulus Multi-Vendor Lab =

The targets of this lab are to :

* mix Nokia SR-Linux and Cumulus VXLAN nodes, in spine or leaf roles
* make this environment work for different kinds of setup : cross-DC L2, differentiated L3 gateways per leaf per VRF, anycast gateway cross or intra DC, etc.
* spot compatibility issues if existing

To reach those goals, we'll use the Nokia lab tool "containerlab" https://containerlab.srlinux.dev/, labs from https://clabs.netdevops.me/ helped a lot. Many thanks to the contributors.

We'll need a Linux host with Docker installed on it + kvm virtualization enhancements activated (important if your host is a VM). If virtualization optimization is not ok, you won't be able to launch the most of devices.

Installation details are referenced there : https://containerlab.srlinux.dev/install/

Once installed, you'll need to register the different NOS images you'll use. Cumulus image is present on a public registrery, but Nokia/Arista/Juniper are not, you'll need a vendor support account to retreive them. Once downloaded (.tgz file typically) you can "docker load" the image :

For private images :<syntaxhighlight lang="bash">
sudo docker load -i 21.6.1-235.tar.xz /* (Nokia) */
sudo docker load -i cEOS64-lab-4.26.1F.tar /* (Arista) */
</syntaxhighlight>For public images :<syntaxhighlight lang="bash">
sudo docker pull networkop/cx:4.3.0 /* (Cumulus) */
</syntaxhighlight>

To verify your installed images, you can run "sudo docker images"<syntaxhighlight lang="bash">
plancastre@containerlab:~/00.LAB/senss$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
networkop/cx 4.3.0 a40541ed15ba 2 weeks ago 733MB
srlinux 21.6.1-235 e29aed30e83f 3 weeks ago 1.14GB
networkop/ignite dev dcc5eff29355 4 weeks ago 37.9MB
networkop/kernel 4.19 e794d8875f83 3 months ago 88MB
frrouting/frr v7.5.1 c3e13a4c5918 3 months ago 123MB

</syntaxhighlight>Once you've got all your images, to run the lab, you'll need a .yml file defining the topology to run.

In our lab, we've started with the 5-clos lab file from Nokia site, which we've forked.

We can see there is a branch "kinds" where we can define templates of nodes, in order to not rewrite whole parameters when creating nodes. For the images, it's important to take the repository name + tag.

'''Note:''' If the tag is "latest" it is not needed.

All supported "kinds" are referenced there : https://containerlab.srlinux.dev/manual/kinds/kinds/

'''Note:''' yml is indentation sensitive, so please respect the "double space" used to define hierarchy. Below the lab diagram + the yml file associated

== Infos on nodes ==

=== L3 infos ===
{| class="wikitable"
|+
!Device
!OS
!ASN
!VTEP IP
!swp1 or eth1/1
!swp2 or eth1/2
!swp3 or eth1/3
!SVI
|-
|client1
|Linux
|<none>
|<none>
|vlan100: 192.168.1.1/24
vlan200: 192.168.2.1/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client2
|Linux
|<none>
|<none>
|vlan100: 192.168.1.2/24
vlan200: 192.168.2.2/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client3
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.1/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|client4
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.2/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|leaf1
|CVX
|65011
|10.255.255.5/32
|10.254.254.8/31
|10.254.254.10/31
|trunk vlan 100,200
|vlan200 : 192.168.2.254/24
|-
|leaf2
|SRLinux
|65012
|10.255.255.6/32
|10.254.254.12/31
|10.254.254.14/31
|trunk vlan 100,200
|irb0.200 : 192.168.2.254/24
|-
|leaf3
|CVX
|65013
|10.255.255.7/32
|10.254.254.22/31
|10.254.254.24/31
|trunk vlan 100,200
|vlan200 : 192.168.200.254/24
|-
|leaf4
|SRLinux
|65014
|10.255.255.8/32
|10.254.254.26/31
|10.254.254.28/31
|trunk vlan 100,200
|irb0.200 : 192.168.200.254/24
|-
|spine1
|CVX
|65001
|10.255.255.1/32
|10.254.254.9/31
|10.254.254.13/31
|10.254.254.0/31
|<none>
|-
|spine2
|CVX
|65001
|10.255.255.2/32
|10.254.254.11/31
|10.254.254.15/31
|10.254.254.16/31
|<none>
|-
|spine3
|SRLinux
|65002
|10.255.255.3/32
|10.254.254.23/31
|10.254.254.27/31
|10.254.254.18/31
|<none>
|-
|spine4
|SRLinux
|65002
|10.255.255.4/32
|10.254.254.25/31
|10.254.254.29/31
|10.254.254.20/31
|<none>
|-
|superspine1
|CVX
|65003
|10.255.255.9/32
|10.254.254.1/31
|10.254.254.19/31
|<none>
|<none>
|-
|superspine2
|SRLinux
|65003
|10.255.255.10/32
|10.254.254.17/31
|10.254.254.21/31
|<none>
|<none>
|}

=== VXLAN/BGP infos ===
Note : in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>
{| class="wikitable"
|+
!service
!vlan id
!route-target
!VNI
!Commentaires
|-
|L2 cross-DC
|100
|target:65000:100
|100
|the 4 clients are in the same broadcast domain / subnet
|-
|L3VPN cros-DC
|200
|target:65000:200
target:65000:10200
|200 (for L2)
10200 (for routing)
|the 4 clients are separated in 2 different subnets. Traffic is routed within dedicated VNI
To be noted that in the current setup VMTO is not implemented, so cross DC traffic is not optimized.
|}

== Lab diagram ==
[[File:Lab-srl-diagram.png|ohne|mini|600x600px]]

== yml lab file ==
<syntaxhighlight lang="yaml">
# topology documentation: http://containerlab.srlinux.dev/lab-examples/min-5clos/
name: lab-srl-cum

topology:
kinds:
srl:
image: srlinux:21.6.1-235
linux:
image: ghcr.io/hellt/network-multitool
cvx:
image: networkop/cx:4.3.0
runtime: docker
nodes:
leaf1:
kind: cvx
leaf2:
kind: srl
type: ixrd2
leaf3:
kind: cvx
leaf4:
kind: srl
type: ixrd2

spine1:
kind: cvx
spine2:
kind: cvx
spine3:
kind: srl
type: ixr6
spine4:
kind: srl
type: ixr6

superspine1:
kind: cvx
superspine2:
kind: srl
type: ixr6

client1:
kind: linux
client2:
kind: linux
client3:
kind: linux
client4:
kind: linux

links:
# leaf to spine links POD1
- endpoints: ["leaf1:swp1", "spine1:swp1"]
- endpoints: ["leaf1:swp2", "spine2:swp1"]
- endpoints: ["leaf2:e1-1", "spine1:swp2"]
- endpoints: ["leaf2:e1-2", "spine2:swp2"]

# spine to superspine links POD1
- endpoints: ["spine1:swp3", "superspine1:swp1"]
- endpoints: ["spine2:swp3", "superspine2:e1-1"]

# leaf to spine links POD2
- endpoints: ["leaf3:swp1", "spine3:e1-1"]
- endpoints: ["leaf3:swp2", "spine4:e1-1"]
- endpoints: ["leaf4:e1-1", "spine3:e1-2"]
- endpoints: ["leaf4:e1-2", "spine4:e1-2"]

# spine to superspine links POD2
- endpoints: ["spine3:e1-3", "superspine1:swp2"]
- endpoints: ["spine4:e1-3", "superspine2:e1-2"]

# client connection links
- endpoints: ["client1:eth1", "leaf1:swp3"]
- endpoints: ["client2:eth1", "leaf2:e1-3"]
- endpoints: ["client3:eth1", "leaf3:swp3"]
- endpoints: ["client4:eth1", "leaf4:e1-3"]

</syntaxhighlight>

Once the nodes are defined, we have to define the needed connections between the nodes.

Each endpoint is defined as '<nowiki/>'''<nowiki/>'<node_name>:<interface_name><nowiki>''</nowiki>'''

When all is ok, you can launch in suduoer the topology.

For example :<syntaxhighlight lang="bash">
sudo containerlab deploy --topo topo.clab.yml --reconfigure

Note: The option "reconfigure" is used there in order to not have to destroy instances when updating topology/relaunching.
</syntaxhighlight>Another info regarding the yml file, it's possible to quickly generate files with the '''"containerlab gen -n 3tier --nodes 4,2,1"''' command for example. https://containerlab.srlinux.dev/cmd/generate/

To be noted that the "tier-x" notion in containerlab is inversed regarding the rfc 7938 (tier1 is superspine level into the rfc, tier1 is leaf level in nokia tool)

At the end of the topology launch, you can retreive the details :<syntaxhighlight lang="yaml">
INFO[0011] Writing /etc/hosts file
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| # | Name | Container ID | Image | Kind | Group | State | IPv4 Address | IPv6 Address |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| 1 | clab-lab-srl-cum-client1 | 25e2bbf6b13f | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
| 2 | clab-lab-srl-cum-client2 | 17d82d5a6e21 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.4/24 | 2001:172:20:20::4/64 |
| 3 | clab-lab-srl-cum-client3 | f8d78527c1b1 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.6/24 | 2001:172:20:20::6/64 |
| 4 | clab-lab-srl-cum-client4 | 913a0172d0c5 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 5 | clab-lab-srl-cum-leaf1 | 9ffc334af2b1 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.9/24 | 2001:172:20:20::9/64 |
| 6 | clab-lab-srl-cum-leaf2 | 7d3eb855d749 | srlinux:21.6.1-235 | srl | | running | 172.20.20.14/24 | 2001:172:20:20::e/64 |
| 7 | clab-lab-srl-cum-leaf3 | e0d36ffc606a | networkop/cx:4.3.0 | cvx | | running | 172.20.20.7/24 | 2001:172:20:20::7/64 |
| 8 | clab-lab-srl-cum-leaf4 | e6753b1d099e | srlinux:21.6.1-235 | srl | | running | 172.20.20.12/24 | 2001:172:20:20::c/64 |
| 9 | clab-lab-srl-cum-spine1 | a7e63993d052 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.5/24 | 2001:172:20:20::5/64 |
| 10 | clab-lab-srl-cum-spine2 | 3124250d5b75 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
| 11 | clab-lab-srl-cum-spine3 | 5e300ca3f66f | srlinux:21.6.1-235 | srl | | running | 172.20.20.11/24 | 2001:172:20:20::b/64 |
| 12 | clab-lab-srl-cum-spine4 | 0a59e60a596b | srlinux:21.6.1-235 | srl | | running | 172.20.20.13/24 | 2001:172:20:20::d/64 |
| 13 | clab-lab-srl-cum-superspine1 | 3d9935413fc2 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.8/24 | 2001:172:20:20::8/64 |
| 14 | clab-lab-srl-cum-superspine2 | 5bc58d423e2c | srlinux:21.6.1-235 | srl | | running | 172.20.20.15/24 | 2001:172:20:20::f/64 |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
</syntaxhighlight>The containerlab program creates the updates into the host file + generates an ansible inventory file.

There are different ways to connect onto the docker instances, within docker exec commands or SSH directly the IP address related above.

For example :<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-spine1 bash
</syntaxhighlight>-->>> allows to launch a bash on the cumulus device (then you run the different net commands)<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-leaf2 sr_cli
</syntaxhighlight>-->>> allows to launch the cli on a SRLinux device

To end this setup preparation, the containerlab tool integrates an option to launch a webserver showing the lab topology in graph + node details

By default, it's listening on port 50080 of the host

Then, you can access to the following web page showing the topology + lab infos

[[File:Lab graph tool.jpg|ohne|mini|900x900px]]

Finally, if you want to stop a lab, please run '''<nowiki>''</nowiki>containerlab destroy --topo <topology>.yml<nowiki>''</nowiki>'''

At this step, nodes are running, connections are done, but configurations are empty.

'''So, it's time to configure them :)'''

Configuration files are attached to this blog. Once validated, they are exportable and can be called by the tool when launching the lab. Below how to do it :

Firstable, once you ve finished your "hand made config", you can run the following command to save it :<syntaxhighlight lang="bash">
containerlab save -t quickstart.clab.yml
</syntaxhighlight>'''To be noted it's only working for Nokia nodes in this lab'''. CVX configurations will have to be called by another method (or pushed manually)

It will create one folder per node into the lab folder which gives the following tree :
[[File:Topology-srl.jpg|ohne|mini]]

'''Note''': For Cumulus, you can apparently retrieve configurations files within adding the following parameters into lab config file. '''To be tested.''' some lab samples here : https://github.com/hellt/clabs/blob/main/labs/cvx04/symm-mh.clab.yml

Now we have the lab running, below the different commands to check that all is working as expected : <syntaxhighlight lang="yaml">
-------------------------------------------
Check L2 tables ///
-------------------------------------------

Nokia :

A:leaf2# show network-instance mac-vrf-100 bridge-table mac-table all
----------------------------------------------------------------------------------------------------------------------------------------------------------
Mac-table of network instance mac-vrf-100
----------------------------------------------------------------------------------------------------------------------------------------------------------
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
| Address | Destination | Dest Index | Type | Active | Aging | Last Update |
+====================+=========================================+============+=============+=========+========+=========================================+
| AA:C1:AB:0C:60:D6 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T13:06:17.000Z |
| | vtep:10.255.255.8 vni:100 | 67 | | | | |
| AA:C1:AB:23:31:04 | ethernet-1/3.100 | 2 | learnt | true | 300 | 2021-10-08T13:06:10.000Z |
| AA:C1:AB:4B:FF:71 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:55:28.000Z |
| | vtep:10.255.255.5 vni:100 | 60 | | | | |
| AA:C1:AB:B5:79:50 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:43:37.000Z |
| | vtep:10.255.255.7 vni:100 | 64 | | | | |
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 0 Total 0 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 1 Total 1 Active
Total Evpn Macs : 3 Total 3 Active
Total Evpn static Macs : 0 Total 0 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 4 Total 4 Active
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : type learnt matches the local learnt mac, evpn are learnt within VXLAN

Cumulus :

root@leaf1-cum-dc1:/# net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
100 bridge swp3 aa:c1:ab:4b:ff:71 00:00:02
100 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:17:21
100 bridge vni100 22:2d:67:f1:bb:26 permanent 02:17:21
100 bridge vni100 aa:c1:ab:0c:60:d6 extern_learn <1 sec
100 bridge vni100 aa:c1:ab:23:31:04 extern_learn 00:00:07
100 bridge vni100 aa:c1:ab:b5:79:50 extern_learn 00:13:28

root@leaf1-cum-dc1:/# net show bridge macs vlan 200

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
200 bridge swp3 aa:c1:ab:4b:ff:71 00:08:46
200 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:16:36
200 bridge vni200 00:00:5e:00:01:01 static sticky 00:14:26
200 bridge vni200 1e:0d:aa:46:53:b1 permanent 02:16:36
200 bridge vni200 02:77:6b:ff:00:41 static sticky 01:00:52
200 bridge vni200 02:f8:b8:ff:00:41 static sticky 00:14:26
200 bridge vni200 aa:c1:ab:b5:79:50 extern_learn 00:11:51
200 bridge vni10200 02:77:6b:ff:00:00 extern_learn 00:53:02
200 bridge vni10200 02:f8:b8:ff:00:00 extern_learn 00:14:26
200 bridge vni10200 86:5e:aa:ca:50:be permanent 02:16:36

Note : the macs with permanent flag are local interfaces, physical ou logic (vtep)
Note2 : extern_learn mac are the one learned via the VXLAN infra (and mapped to the vnixxx interface)

-------------------------------------------
Check arp tables ///
-------------------------------------------

Nokia :

:leaf2# show arpnd arp-entries
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| ethernet-1/1 | 0 | 10.254.254.13 | dynamic | AA:C1:AB:59:65:EA | 3 hours from now |
| ethernet-1/2 | 0 | 10.254.254.15 | dynamic | AA:C1:AB:CB:F8:8A | 3 hours from now |
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
| mgmt0 | 0 | 172.20.20.1 | dynamic | 02:42:63:9E:74:03 | 2 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 5 (0 static, 5 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : on Nokia you have to sort the entries basing on subinterface and the retrieve the L3VPN attached

A:leaf2# show arpnd arp-entries subinterface 200
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 2 (0 static, 2 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Cumulus :

Here we retrieve all arp infos linked to evpn

root@leaf1-cum-dc1:/# net show evpn arp-cache vni all
VNI 100 #ARP (IPv4 and IPv6, local and remote) 2

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.1.3 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0
192.168.1.1 local active aa:c1:ab:4b:ff:71 0/0

VNI 200 #ARP (IPv4 and IPv6, local and remote) 4

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.2.1 local active aa:c1:ab:4b:ff:71 0/0
192.168.200.254 remote active 00:00:5e:00:01:01 10.255.255.8 0/0
192.168.2.254 remote active 00:00:5e:00:01:01 10.255.255.6 0/0
192.168.200.1 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0

VNI 400 #ARP (IPv4 and IPv6, local and remote) 0

-------------------------------------------
Check bgp sessions' status - underlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep ipv4
| default | 10.254.254.19 | udl-superspine | S | 65003 | established | 0d:0h:22m:44s | ipv4-unicast | [13/4/4] |
| default | 10.254.254.22 | udl-leaf | S | 65013 | established | 0d:0h:22m:52s | ipv4-unicast | [13/1/4] |

Cumulus :

root@leaf3:/# net show bgp ipv4 unicast summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 13
RIB entries 25, using 4800 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.254.254.23 4 65002 466 403 0 0 0 00:19:34 4 13
10.254.254.25 4 65002 434 355 0 0 0 00:17:10 8 13

-------------------------------------------
Check bgp sessions' status - overlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep evpn
| default | 10.255.255.7 | ovl-leaf | S | 65013 | established | 0d:0h:24m:14s| evpn | [15/0/11] |
| default | 10.255.255.9 | ovl-superspine | S | 65003 | established | 0d:0h:24m:7s | evpn | [11/0/4] |

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.255.255.3 4 65002 5618 4742 0 0 0 03:53:35 19 27
10.255.255.4 4 65002 5565 4694 0 0 0 03:51:11 19 27

-------------------------------------------
Check EVPN L2 tables///
-------------------------------------------

Nokia :

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn route type 2
BGP table version is 17, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:100
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
Route Distinguisher: 10.255.255.5:200
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
Route Distinguisher: 10.255.255.6:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
Route Distinguisher: 10.255.255.7:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100
Route Distinguisher: 10.255.255.7:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.200.1]
10.255.255.7 32768 i
ET:8 RT:65000:200
Route Distinguisher: 10.255.255.8:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC

and if we focus on our vni 100

root@leaf3-cum-dc1:/# net show bgp evpn route vni 100 type 2
BGP table version is 48, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100

The additional route type "MAC+IP" is optional and is useful/mandatory when you do anycast irb.

-------------------------------------------
Check tables EVPN L3 ///
-------------------------------------------

Nokia :

--{ + running }--[ ]--
A:leaf2# show network-instance ip-vrf-200 route-table all
-----------------------------------------------------------------------------------------------------------------------------------------------------------
IPv4 Unicast route table of network instance ip-vrf-200
-----------------------------------------------------------------------------------------------------------------------------------------------------------
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
| Prefix | ID | Route Type | Route Owner | Best/Fib- | Metric | Pref | Next-hop | Next-hop |
| | | | | status(slot) | | | (Type) | Interface |
+=======================+=======+============+======================+======================+==========+=========+==============+==============+
| 192.168.2.0/24 | 9 | local | net_inst_mgr | True/success | 0 | 0 | 192.168.2.25 | irb0.200 |
| | | | | | | | 4 (direct) | |
| 192.168.2.254/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (extract) | |
| 192.168.2.255/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (broadcast) | |
| 192.168.200.0/24 | 0 | bgp-evpn | bgp_evpn_mgr | True/success | 0 | 170 | 10.255.255.8 | None |
| | | | | | | | (indirect/vx | |
| | | | | | | | lan) | |
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
-----------------------------------------------------------------------------------------------------------------------------------------------------------
4 IPv4 routes total
4 IPv4 prefixes with active routes
0 IPv4 prefixes with active ECMP routes
-----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--
A:leaf2#

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf1-cum-dc1:/# net show route vrf ip-vrf-200
show ip route vrf ip-vrf-200
=============================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF ip-vrf-200:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 04:48:42
B 192.168.2.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 04:27:26
C>* 192.168.2.0/24 is directly connected, vlan200, 04:48:42
B>* 192.168.200.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:48:49

root@leaf1-cum-dc1:/# net show bgp evpn route type 5
BGP table version is 3, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.6:10200
* [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
*> [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
Route Distinguisher: 10.255.255.8:10200
*> [5]:[0]:[24]:[192.168.200.0]
10.255.255.8 0 65001 65003 65002 65014 i
RT:65000:10200 ET:8 Rmac:02:f8:b8:ff:00:00

-------------------------------------------
Check arp hosts ///
-------------------------------------------

From client #1 /

bash-5.0# arp
Address HWtype HWaddress Flags Mask Iface
192.168.1.3 ether aa:c1:ab:b5:79:50 C eth1.100
192.168.1.4 ether aa:c1:ab:0c:60:d6 C eth1.100
192.168.2.2 ether aa:c1:ab:23:31:04 C eth1.200
192.168.2.254 ether aa:c1:ab:4f:f2:e7 C eth1.200
192.168.1.2 ether aa:c1:ab:23:31:04 C eth1.100
containerlab.local ether 02:42:63:9e:74:03 C eth0

-------------------------------------------
Ping ///
-------------------------------------------

From client #1 /

L2 service ping (vlan 100) ----------------------

bash-5.0# ping -c 1 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.812 ms

--- 192.168.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.812/0.812/0.812/0.000 MsgRcvd
bash-5.0# ping -c 1 192.168.1.3
PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_seq=1 ttl=64 time=1.40 ms

--- 192.168.1.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.403/1.403/1.403/0.000 ms
bash-5.0# ping -c 1 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=1.62 ms

--- 192.168.1.4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.619/1.619/1.619/0.000 ms

L3 service ping (vlan 200)

bash-5.0# ping -c 1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.053 ms

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms
bash-5.0# ^C
bash-5.0# ping -c 1 192.168.2.2
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.611 ms

--- 192.168.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.611/0.611/0.611/0.000 ms

-------------------------------------------
Traceroute ///
-------------------------------------------

L3 service traceroute (vlan 200)

bash-5.0# traceroute 192.168.200.1
traceroute to 192.168.200.1 (192.168.200.1), 30 hops max, 46 byte packets
1 192.168.2.254 (192.168.2.254) 0.012 ms 0.010 ms 0.007 ms
2 192.168.200.254 (192.168.200.254) 2.751 ms * 2.549 ms
3 192.168.200.1 (192.168.200.1) 2.084 ms 1.805 ms 1.761 ms

</syntaxhighlight>To come back on the setup, we wanted to implement solutions inspired by the <nowiki>RFC 7938</nowiki> rfc7938 (see section 5.2.1)

* eBGP will be used for Underlay and Overlay
* eBGP single-hop sessions are established over direct point-to-point links interconnecting the network nodes, no multi-hop or loopback sessions are used, even in the case of multiple links between the same pair of nodes.
* Private Use ASNs from the range 64512-65534 are used to avoid ASN conflicts.
* A single ASN is allocated to all of the Clos topology's Tier 1 devices.
* A unique ASN is allocated to each set of Tier 2 devices in the same cluster.
* A unique ASN is allocated to every Tier 3 device (e.g., ToR) in this topology.

In add, depending of network nodes, it will be interesting to :

* Add fast-failover option for underlay + add link-debounce on links to limit possible impacts in case of link flaps. BFD could be considered in some cases
* Have the best ECMP hashing algorythm on leaf nodes. On tier-1/tier-2 levels hashing will be based on underlay part (trafic inter-vtep), so L3-L4 decision, but some models can retreive the vni-id if I well remember.
* Implement the "multipath relax" since we could have multiple AS paths for a same destination.
* Implement gshut community on underlay (on tier-1/tier-2) or at least policies to prepend underlay routes to avoid some nodes in case of maintenance.

== Thoughts/Points to deepend ==
On leaf nodes, LACP over multiple nodes is feasible within ESI, but impact on incidents/maintenance can vary depending on the vendors.

'''Tests need to be done to check that.'''

Some sysadmin would like to KISS and have client servers connected in active/passive bonding, with eventually single-leaf ports aggregation.

'''Note :''' NH ESI in FIB management on ingress nodes can have some limitations (numbers of installable next-hop, load-balancing, etc. oftenly linked to the node chipset)

Another important engineering point to define at the beginning is : '''Do we authorize multisite L2 propagation or not ?'''

From my point-of-view, the smaller are the L2 segments the better it is. I would suggest Local L2 segments wit anycast gateways provided by Leaf switches, with L3 routing between DCs within EVPN type5 routes sharing only. Nevertheless, some business requirements for endcustomers can keep extended L2 alive.

We have to be careful when implementing Consistent hashing. ex: <nowiki>https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-44/Layer-3/Routing/Equal-Cost-Multipath-Load-Sharing-Hardware-ECMP/</nowiki>

If possible activate it per VRF, not globally, and restrict on dedicated subnets << depends of the vendor, it can be dangerous regarding memory management

Think at configuring the bgp option to not advertise routes to peers before they are installed in FIB.

We can implement leaf roles in some cases, to be able to do "light traffic engineering " on DCI links in some setup.

These roles will be managed within bgp communities added to VTEP /32. Look at the diagram below. Depending of the colour, the eBGP sessions between SupS will prepend or not the VTEP IPs announced to the remote DC

[[File:Fabric-roles.jpg|ohne|mini|1200x1200px]]

<blockquote>'''To be continued'''</blockquote>

== Lab archives ==
<blockquote>Reminder : following config are for lab, so please don't pay attention on typos/lack of security params/unused config. Thanks \o/</blockquote>

=== Credentials ===
<syntaxhighlight lang="bash">
SRLinux : admin/<no_password>

CVX : cumulus/cumulus
</syntaxhighlight>

=== Lab archives ===
layer 2 (vlan 100) + layer 3 (vlan 200) lab, 10/12/2021
[[https://wiki.nesevo.com/images/6/66/20211008_lab-nokia-cvx.zip 20211008_lab-nokia-cvx.zip]]
<u>If you want to exchange about that lab, those topics, please don't hesitate to contact us</u>

= SONiC Lab =

<span><to be noted that those informations are included into the readme.md file into the archive></span>

== Lab diagram ==
[[File:SONIC_LAB.jpg]]

Lab archive [https://wiki.nesevo.com/images/0/0a/Sonic-lab-20201030.zip Sonic-lab-20201030.zip]

This lab is a forked lab from many found on @, starting from Microsoft one : https://github.com/Azure/SONiC/wiki/SONiC-P4-Software-Switch

The target of this lab is to simulate a simple IP/Fabric in SPINE/LEAF topology. Each Leaf provide local L2 connectivity + L3 gateways to hosts. Flows between hosts from different leaf are L3 routed, in a "node path" LEAF-SPINE-LEAF.

Nothing complicated, but it allows to put a first foot into SONiC world :)

For the moment, as far as we know, MC-LAG + EVPN/VXLAN features are only available on entreprise SONiC versions. Tests to come once physic switches in enterprise version will be available in our lab.

The diagram of the architecture is into the zip file (SONIC_LAB.jpg)

To run the lab you need :

* to have installed docker by using ./install_docker_ovs.sh
* to run the ./load_image.sh in order to get the Sonic image

Note : don't change file / folders names of network devices (leafX, spineX)

Once you're ready, please run the lab by launching "./start.sh"

The script :<div></div>
* launches the containers without Docket network features
* create network namespaces
* build linux bridges to interconnect the containers
* configure host IPs
* launch startup sonic scripts into sonic devices
* once devices started, checks + ping tests are run. If all BGP sessions are ok on spine + all ping are fine, the setup works well.
The "start" script has several comments, perhaps without the right vocabulary, so please be tolerant :)

If you want to stop the lab, please launch "./stop.sh"

Below some useful commands to run for sonic containers :

=== <span>run ping</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> ping <ip>
</syntaxhighlight>
This command triggers ping into the targetted container :
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it leaf1 ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.052 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.034 ms
^C--- 192.168.1.1 ping statistics ---
</syntaxhighlight>

=== <span>check bgp sessions</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip bgp sum"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print the bgp status.
Below, we run it on one spine, you should have 4 sessiosn UP (for the 4 leaves)
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip bgp sum"
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 6409 6411 0 0 0 01:46:46 3
10.1.0.2 4 65102 6406 6410 0 0 0 01:46:44 3
10.1.0.4 4 65103 6407 6410 0 0 0 01:46:44 4
10.1.0.6 4 65104 6406 6412 0 0 0 01:46:46 1
Total number of neighbors 4
</syntaxhighlight>

=== <span>check ip route</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip route"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print its routing table?

<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,
> - selected route, * - FIB route
C>* 10.1.0.0/31 is directly connected, Vlan11
C>* 10.1.0.2/31 is directly connected, Vlan21
C>* 10.1.0.4/31 is directly connected, Vlan31
C>* 10.1.0.6/31 is directly connected, Vlan41
C>* 127.0.0.0/8 is directly connected, lo
B>* 192.168.1.0/24 [20/0] via 10.1.0.0, Vlan11, 01:47:23
B>* 192.168.2.0/24 [20/0] via 10.1.0.2, Vlan21, 01:47:21
B>* 192.168.3.0/24 [20/0] via 10.1.0.4, Vlan31, 01:47:21
B>* 192.168.4.0/24 [20/0] via 10.1.0.6, Vlan41, 01:47:23
</syntaxhighlight>
For more commands check SONiC docs : https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md

To finish, below the output of the start.sh when all goes well<div></div>
plancastre@dockerhost:~/00.Projects/p4-version-pla$ ./start.sh
---------------------------------------------------
Creating the containers: starting
---------------------------------------------------
0b4bbb3c6cfc9895466ea13a9e940d1a176d66f45deee9752e9b4732d735bb03
5ca9486baab3e8a5259062167132a944eee717875d3cc1ef29fe490cba3184c1
46c9c09dba27a32715b9fbe0bc434a9284b9d59317d278df693f425cb669f037
898e6afbcb2977115acbba2a69d069796cfd70fb90228b6c18e09ba4584365f7
eb9b3ef6d14784c9ffe4d9d5df3e921a24559889cb8770d226263aad6761182a
ae5f3b2c628da687eaabb331dc2cd989e71bcb386854e7a0be6ec587df99d49a
122197bffd7566a01ce3b6565fdfb6eb893aab7d5236f14e5a9bbbe95037ced2
a604520a080e2a2184598ad1d915c0c182d16e5185033373e95a4772d8978a67
0c49e18d78b2aa7c8ef8e937b60285aa56ec0bf4a8530226285270a085e11d5c
eb66336b8c6bd8333f1f4f685369e6cac973b817c35bb7126ac4d7ba72ef71bf
---------------------------------------------------
Retreiving PID from docker containers
---------------------------------------------------
---------------------------------------------------
Creating network namespaces for containers
---------------------------------------------------
List of created namespaces
ns_host4
ns_host3
ns_host2
ns_host1
ns_spine2
ns_spine1
ns_leaf4
ns_leaf3
ns_leaf2
ns_leaf1
---------------------------------------------------
Creating network connectivity between containers
---------------------------------------------------
---------------------------------------------------
Now assigning IPs to hosts (host1 to host4)
---------------------------------------------------
---------------------------------------------------
Configuration of containers is now finished
---------------------------------------------------
---------------------------------------------------
Now booting switches, please wait 2 minutes for switches to load
---------------------------------------------------
---------------------------------------------------
2 minutes period is finished
---------------------------------------------------
---------------------------------------------------
Now fixing the iptables
---------------------------------------------------
---------------------------------------------------
Backend actions are now finished
---------------------------------------------------
---------------------------------------------------
Now we'll check in 15 sec that BGP peers are all Up
---------------------------------------------------
---------------------------------------------------
SPINE1 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 94 97 0 0 0 00:01:28 4
10.1.0.2 4 65102 95 97 0 0 0 00:01:28 5
10.1.0.4 4 65103 95 97 0 0 0 00:01:28 5
10.1.0.6 4 65104 95 97 0 0 0 00:01:28 5
Total number of neighbors 4
---------------------------------------------------
SPINE2 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.2.0.1, local AS number 65002
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.2.0.0 4 65101 93 96 0 0 0 00:01:29 2
10.2.0.2 4 65102 94 100 0 0 0 00:01:32 1
10.2.0.4 4 65103 94 100 0 0 0 00:01:32 1
10.2.0.6 4 65104 94 100 0 0 0 00:01:32 1
Total number of neighbors 4
---------------------------------------------------
Now if all is ok, we'll do some ping tests
---------------------------------------------------
---------------------------------------------------
host1 ping its gateway
---------------------------------------------------
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=17.6 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=9.14 ms
--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 9.145/13.390/17.636/4.247 ms
---------------------------------------------------
host2 ping its gateway
---------------------------------------------------
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=23.6 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=6.21 ms
--- 192.168.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 6.218/14.950/23.683/8.733 ms
---------------------------------------------------
host3 ping its gateway
---------------------------------------------------
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=18.8 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=12.7 ms
--- 192.168.3.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.716/15.764/18.812/3.048 ms
---------------------------------------------------
host4 ping its gateway
---------------------------------------------------
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.
64 bytes from 192.168.4.1: icmp_seq=1 ttl=64 time=17.7 ms
64 bytes from 192.168.4.1: icmp_seq=2 ttl=64 time=4.46 ms
--- 192.168.4.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.464/11.107/17.750/6.643 ms
---------------------------------------------------
host1 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=16.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.049/17.253/19.532/1.615 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=15.5 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=14.3 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.312/15.592/16.939/1.083 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=20.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=16.8 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.048/17.871/20.739/2.058 ms
---------------------------------------------------
host2 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=16.6 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.222/16.583/16.924/0.286 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.6 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=18.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=26.8 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.620/20.549/26.887/4.524 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=33.8 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=27.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=20.2 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 20.216/27.259/33.818/5.566 ms
---------------------------------------------------
host3 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=14.5 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=15.5 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.535/16.549/19.541/2.162 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.4 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=14.6 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=18.2 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.643/16.458/18.278/1.491 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=15.3 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=14.1 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=13.1 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 13.171/14.234/15.334/0.883 ms
---------------------------------------------------
host4 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=21.9 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=21.3 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=14.6 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 14.619/19.284/21.932/3.310 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=14.8 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=18.4 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=14.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.148/15.786/18.409/1.879 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=15.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=18.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 15.117/16.788/18.962/1.612 ms
---------------------------------------------------
If bot spine have all BGP sessions and all ping succeeded, all works well
Congrats ! please check readme file to find some details
---------------------------------------------------

File:20211008 lab-nokia-cvx.zip

2021-10-29T08:19:06Z

PierreLancastre:

Nesevo Virtual Labs

2021-10-29T08:17:11Z

PierreLancastre: /* Lab diagram */

[[File:nesevo-labs-v2.png]]

= Nokia/Cumulus Multi-Vendor Lab =

The targets of this lab are to :

* mix Nokia SR-Linux and Cumulus VXLAN nodes, in spine or leaf roles
* make this environment work for different kinds of setup : cross-DC L2, differentiated L3 gateways per leaf per VRF, anycast gateway cross or intra DC, etc.
* spot compatibility issues if existing

To reach those goals, we'll use the Nokia lab tool "containerlab" https://containerlab.srlinux.dev/, labs from https://clabs.netdevops.me/ helped a lot. Many thanks to the contributors.

We'll need a Linux host with Docker installed on it + kvm virtualization enhancements activated (important if your host is a VM). If virtualization optimization is not ok, you won't be able to launch the most of devices.

Installation details are referenced there : https://containerlab.srlinux.dev/install/

Once installed, you'll need to register the different NOS images you'll use. Cumulus image is present on a public registrery, but Nokia/Arista/Juniper are not, you'll need a vendor support account to retreive them. Once downloaded (.tgz file typically) you can "docker load" the image :

For private images :<syntaxhighlight lang="bash">
sudo docker load -i 21.6.1-235.tar.xz /* (Nokia) */
sudo docker load -i cEOS64-lab-4.26.1F.tar /* (Arista) */
</syntaxhighlight>For public images :<syntaxhighlight lang="bash">
sudo docker pull networkop/cx:4.3.0 /* (Cumulus) */
</syntaxhighlight>

To verify your installed images, you can run "sudo docker images"<syntaxhighlight lang="bash">
plancastre@containerlab:~/00.LAB/senss$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
networkop/cx 4.3.0 a40541ed15ba 2 weeks ago 733MB
srlinux 21.6.1-235 e29aed30e83f 3 weeks ago 1.14GB
networkop/ignite dev dcc5eff29355 4 weeks ago 37.9MB
networkop/kernel 4.19 e794d8875f83 3 months ago 88MB
frrouting/frr v7.5.1 c3e13a4c5918 3 months ago 123MB

</syntaxhighlight>Once you've got all your images, to run the lab, you'll need a .yml file defining the topology to run.

In our lab, we've started with the 5-clos lab file from Nokia site, which we've forked.

We can see there is a branch "kinds" where we can define templates of nodes, in order to not rewrite whole parameters when creating nodes. For the images, it's important to take the repository name + tag.

'''Note:''' If the tag is "latest" it is not needed.

All supported "kinds" are referenced there : https://containerlab.srlinux.dev/manual/kinds/kinds/

'''Note:''' yml is indentation sensitive, so please respect the "double space" used to define hierarchy. Below the lab diagram + the yml file associated

== Infos on nodes ==

=== L3 infos ===
{| class="wikitable"
|+
!Device
!OS
!ASN
!VTEP IP
!swp1 or eth1/1
!swp2 or eth1/2
!swp3 or eth1/3
!SVI
|-
|client1
|Linux
|<none>
|<none>
|vlan100: 192.168.1.1/24
vlan200: 192.168.2.1/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client2
|Linux
|<none>
|<none>
|vlan100: 192.168.1.2/24
vlan200: 192.168.2.2/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client3
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.1/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|client4
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.2/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|leaf1
|CVX
|65011
|10.255.255.5/32
|10.254.254.8/31
|10.254.254.10/31
|trunk vlan 100,200
|vlan200 : 192.168.2.254/24
|-
|leaf2
|SRLinux
|65012
|10.255.255.6/32
|10.254.254.12/31
|10.254.254.14/31
|trunk vlan 100,200
|irb0.200 : 192.168.2.254/24
|-
|leaf3
|CVX
|65013
|10.255.255.7/32
|10.254.254.22/31
|10.254.254.24/31
|trunk vlan 100,200
|vlan200 : 192.168.200.254/24
|-
|leaf4
|SRLinux
|65014
|10.255.255.8/32
|10.254.254.26/31
|10.254.254.28/31
|trunk vlan 100,200
|irb0.200 : 192.168.200.254/24
|-
|spine1
|CVX
|65001
|10.255.255.1/32
|10.254.254.9/31
|10.254.254.13/31
|10.254.254.0/31
|<none>
|-
|spine2
|CVX
|65001
|10.255.255.2/32
|10.254.254.11/31
|10.254.254.15/31
|10.254.254.16/31
|<none>
|-
|spine3
|SRLinux
|65002
|10.255.255.3/32
|10.254.254.23/31
|10.254.254.27/31
|10.254.254.18/31
|<none>
|-
|spine4
|SRLinux
|65002
|10.255.255.4/32
|10.254.254.25/31
|10.254.254.29/31
|10.254.254.20/31
|<none>
|-
|superspine1
|CVX
|65003
|10.255.255.9/32
|10.254.254.1/31
|10.254.254.19/31
|<none>
|<none>
|-
|superspine2
|SRLinux
|65003
|10.255.255.10/32
|10.254.254.17/31
|10.254.254.21/31
|<none>
|<none>
|}

=== VXLAN/BGP infos ===
Note : in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>
{| class="wikitable"
|+
!service
!vlan id
!route-target
!VNI
!Commentaires
|-
|L2 cross-DC
|100
|target:65000:100
|100
|the 4 clients are in the same broadcast domain / subnet
|-
|L3VPN cros-DC
|200
|target:65000:200
target:65000:10200
|200 (for L2)
10200 (for routing)
|the 4 clients are separated in 2 different subnets. Traffic is routed within dedicated VNI
To be noted that in the current setup VMTO is not implemented, so cross DC traffic is not optimized.
|}

== Lab diagram ==
[[File:Lab-srl-diagram.png|ohne|mini|600x600px]]

== yml lab file ==
<syntaxhighlight lang="yaml">
# topology documentation: http://containerlab.srlinux.dev/lab-examples/min-5clos/
name: lab-srl-cum

topology:
kinds:
srl:
image: srlinux:21.6.1-235
linux:
image: ghcr.io/hellt/network-multitool
cvx:
image: networkop/cx:4.3.0
runtime: docker
nodes:
leaf1:
kind: cvx
leaf2:
kind: srl
type: ixrd2
leaf3:
kind: cvx
leaf4:
kind: srl
type: ixrd2

spine1:
kind: cvx
spine2:
kind: cvx
spine3:
kind: srl
type: ixr6
spine4:
kind: srl
type: ixr6

superspine1:
kind: cvx
superspine2:
kind: srl
type: ixr6

client1:
kind: linux
client2:
kind: linux
client3:
kind: linux
client4:
kind: linux

links:
# leaf to spine links POD1
- endpoints: ["leaf1:swp1", "spine1:swp1"]
- endpoints: ["leaf1:swp2", "spine2:swp1"]
- endpoints: ["leaf2:e1-1", "spine1:swp2"]
- endpoints: ["leaf2:e1-2", "spine2:swp2"]

# spine to superspine links POD1
- endpoints: ["spine1:swp3", "superspine1:swp1"]
- endpoints: ["spine2:swp3", "superspine2:e1-1"]

# leaf to spine links POD2
- endpoints: ["leaf3:swp1", "spine3:e1-1"]
- endpoints: ["leaf3:swp2", "spine4:e1-1"]
- endpoints: ["leaf4:e1-1", "spine3:e1-2"]
- endpoints: ["leaf4:e1-2", "spine4:e1-2"]

# spine to superspine links POD2
- endpoints: ["spine3:e1-3", "superspine1:swp2"]
- endpoints: ["spine4:e1-3", "superspine2:e1-2"]

# client connection links
- endpoints: ["client1:eth1", "leaf1:swp3"]
- endpoints: ["client2:eth1", "leaf2:e1-3"]
- endpoints: ["client3:eth1", "leaf3:swp3"]
- endpoints: ["client4:eth1", "leaf4:e1-3"]

</syntaxhighlight>

Once the nodes are defined, we have to define the needed connections between the nodes.

Each endpoint is defined as '<nowiki/>'''<nowiki/>'<node_name>:<interface_name><nowiki>''</nowiki>'''

When all is ok, you can launch in suduoer the topology.

For example :<syntaxhighlight lang="bash">
sudo containerlab deploy --topo topo.clab.yml --reconfigure

Note: The option "reconfigure" is used there in order to not have to destroy instances when updating topology/relaunching.
</syntaxhighlight>Another info regarding the yml file, it's possible to quickly generate files with the '''"containerlab gen -n 3tier --nodes 4,2,1"''' command for example. https://containerlab.srlinux.dev/cmd/generate/

To be noted that the "tier-x" notion in containerlab is inversed regarding the rfc 7938 (tier1 is superspine level into the rfc, tier1 is leaf level in nokia tool)

At the end of the topology launch, you can retreive the details :<syntaxhighlight lang="yaml">
INFO[0011] Writing /etc/hosts file
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| # | Name | Container ID | Image | Kind | Group | State | IPv4 Address | IPv6 Address |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| 1 | clab-lab-srl-cum-client1 | 25e2bbf6b13f | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
| 2 | clab-lab-srl-cum-client2 | 17d82d5a6e21 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.4/24 | 2001:172:20:20::4/64 |
| 3 | clab-lab-srl-cum-client3 | f8d78527c1b1 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.6/24 | 2001:172:20:20::6/64 |
| 4 | clab-lab-srl-cum-client4 | 913a0172d0c5 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 5 | clab-lab-srl-cum-leaf1 | 9ffc334af2b1 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.9/24 | 2001:172:20:20::9/64 |
| 6 | clab-lab-srl-cum-leaf2 | 7d3eb855d749 | srlinux:21.6.1-235 | srl | | running | 172.20.20.14/24 | 2001:172:20:20::e/64 |
| 7 | clab-lab-srl-cum-leaf3 | e0d36ffc606a | networkop/cx:4.3.0 | cvx | | running | 172.20.20.7/24 | 2001:172:20:20::7/64 |
| 8 | clab-lab-srl-cum-leaf4 | e6753b1d099e | srlinux:21.6.1-235 | srl | | running | 172.20.20.12/24 | 2001:172:20:20::c/64 |
| 9 | clab-lab-srl-cum-spine1 | a7e63993d052 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.5/24 | 2001:172:20:20::5/64 |
| 10 | clab-lab-srl-cum-spine2 | 3124250d5b75 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
| 11 | clab-lab-srl-cum-spine3 | 5e300ca3f66f | srlinux:21.6.1-235 | srl | | running | 172.20.20.11/24 | 2001:172:20:20::b/64 |
| 12 | clab-lab-srl-cum-spine4 | 0a59e60a596b | srlinux:21.6.1-235 | srl | | running | 172.20.20.13/24 | 2001:172:20:20::d/64 |
| 13 | clab-lab-srl-cum-superspine1 | 3d9935413fc2 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.8/24 | 2001:172:20:20::8/64 |
| 14 | clab-lab-srl-cum-superspine2 | 5bc58d423e2c | srlinux:21.6.1-235 | srl | | running | 172.20.20.15/24 | 2001:172:20:20::f/64 |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
</syntaxhighlight>The containerlab program creates the updates into the host file + generates an ansible inventory file.

There are different ways to connect onto the docker instances, within docker exec commands or SSH directly the IP address related above.

For example :<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-spine1 bash
</syntaxhighlight>-->>> allows to launch a bash on the cumulus device (then you run the different net commands)<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-leaf2 sr_cli
</syntaxhighlight>-->>> allows to launch the cli on a SRLinux device

To end this setup preparation, the containerlab tool integrates an option to launch a webserver showing the lab topology in graph + node details

By default, it's listening on port 50080 of the host

Then, you can access to the following web page showing the topology + lab infos

[[File:Lab graph tool.jpg|ohne|mini|900x900px]]

Finally, if you want to stop a lab, please run '''<nowiki>''</nowiki>containerlab destroy --topo <topology>.yml<nowiki>''</nowiki>'''

At this step, nodes are running, connections are done, but configurations are empty.

'''So, it's time to configure them :)'''

Configuration files are attached to this blog. Once validated, they are exportable and can be called by the tool when launching the lab. Below how to do it :

Firstable, once you ve finished your "hand made config", you can run the following command to save it :<syntaxhighlight lang="bash">
containerlab save -t quickstart.clab.yml
</syntaxhighlight>'''To be noted it's only working for Nokia nodes in this lab'''. CVX configurations will have to be called by another method (or pushed manually)

It will create one folder per node into the lab folder which gives the following tree :
[[File:Topology-srl.jpg|ohne|mini]]

'''Note''': For Cumulus, you can apparently retrieve configurations files within adding the following parameters into lab config file. '''To be tested.''' some lab samples here : https://github.com/hellt/clabs/blob/main/labs/cvx04/symm-mh.clab.yml

Now we have the lab running, below the different commands to check that all is working as expected : <syntaxhighlight lang="yaml">
-------------------------------------------
Check L2 tables ///
-------------------------------------------

Nokia :

A:leaf2# show network-instance mac-vrf-100 bridge-table mac-table all
----------------------------------------------------------------------------------------------------------------------------------------------------------
Mac-table of network instance mac-vrf-100
----------------------------------------------------------------------------------------------------------------------------------------------------------
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
| Address | Destination | Dest Index | Type | Active | Aging | Last Update |
+====================+=========================================+============+=============+=========+========+=========================================+
| AA:C1:AB:0C:60:D6 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T13:06:17.000Z |
| | vtep:10.255.255.8 vni:100 | 67 | | | | |
| AA:C1:AB:23:31:04 | ethernet-1/3.100 | 2 | learnt | true | 300 | 2021-10-08T13:06:10.000Z |
| AA:C1:AB:4B:FF:71 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:55:28.000Z |
| | vtep:10.255.255.5 vni:100 | 60 | | | | |
| AA:C1:AB:B5:79:50 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:43:37.000Z |
| | vtep:10.255.255.7 vni:100 | 64 | | | | |
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 0 Total 0 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 1 Total 1 Active
Total Evpn Macs : 3 Total 3 Active
Total Evpn static Macs : 0 Total 0 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 4 Total 4 Active
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : type learnt matches the local learnt mac, evpn are learnt within VXLAN

Cumulus :

root@leaf1-cum-dc1:/# net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
100 bridge swp3 aa:c1:ab:4b:ff:71 00:00:02
100 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:17:21
100 bridge vni100 22:2d:67:f1:bb:26 permanent 02:17:21
100 bridge vni100 aa:c1:ab:0c:60:d6 extern_learn <1 sec
100 bridge vni100 aa:c1:ab:23:31:04 extern_learn 00:00:07
100 bridge vni100 aa:c1:ab:b5:79:50 extern_learn 00:13:28

root@leaf1-cum-dc1:/# net show bridge macs vlan 200

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
200 bridge swp3 aa:c1:ab:4b:ff:71 00:08:46
200 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:16:36
200 bridge vni200 00:00:5e:00:01:01 static sticky 00:14:26
200 bridge vni200 1e:0d:aa:46:53:b1 permanent 02:16:36
200 bridge vni200 02:77:6b:ff:00:41 static sticky 01:00:52
200 bridge vni200 02:f8:b8:ff:00:41 static sticky 00:14:26
200 bridge vni200 aa:c1:ab:b5:79:50 extern_learn 00:11:51
200 bridge vni10200 02:77:6b:ff:00:00 extern_learn 00:53:02
200 bridge vni10200 02:f8:b8:ff:00:00 extern_learn 00:14:26
200 bridge vni10200 86:5e:aa:ca:50:be permanent 02:16:36

Note : the macs with permanent flag are local interfaces, physical ou logic (vtep)
Note2 : extern_learn mac are the one learned via the VXLAN infra (and mapped to the vnixxx interface)

-------------------------------------------
Check arp tables ///
-------------------------------------------

Nokia :

:leaf2# show arpnd arp-entries
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| ethernet-1/1 | 0 | 10.254.254.13 | dynamic | AA:C1:AB:59:65:EA | 3 hours from now |
| ethernet-1/2 | 0 | 10.254.254.15 | dynamic | AA:C1:AB:CB:F8:8A | 3 hours from now |
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
| mgmt0 | 0 | 172.20.20.1 | dynamic | 02:42:63:9E:74:03 | 2 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 5 (0 static, 5 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : on Nokia you have to sort the entries basing on subinterface and the retrieve the L3VPN attached

A:leaf2# show arpnd arp-entries subinterface 200
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 2 (0 static, 2 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Cumulus :

Here we retrieve all arp infos linked to evpn

root@leaf1-cum-dc1:/# net show evpn arp-cache vni all
VNI 100 #ARP (IPv4 and IPv6, local and remote) 2

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.1.3 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0
192.168.1.1 local active aa:c1:ab:4b:ff:71 0/0

VNI 200 #ARP (IPv4 and IPv6, local and remote) 4

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.2.1 local active aa:c1:ab:4b:ff:71 0/0
192.168.200.254 remote active 00:00:5e:00:01:01 10.255.255.8 0/0
192.168.2.254 remote active 00:00:5e:00:01:01 10.255.255.6 0/0
192.168.200.1 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0

VNI 400 #ARP (IPv4 and IPv6, local and remote) 0

-------------------------------------------
Check bgp sessions' status - underlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep ipv4
| default | 10.254.254.19 | udl-superspine | S | 65003 | established | 0d:0h:22m:44s | ipv4-unicast | [13/4/4] |
| default | 10.254.254.22 | udl-leaf | S | 65013 | established | 0d:0h:22m:52s | ipv4-unicast | [13/1/4] |

Cumulus :

root@leaf3:/# net show bgp ipv4 unicast summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 13
RIB entries 25, using 4800 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.254.254.23 4 65002 466 403 0 0 0 00:19:34 4 13
10.254.254.25 4 65002 434 355 0 0 0 00:17:10 8 13

-------------------------------------------
Check bgp sessions' status - overlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep evpn
| default | 10.255.255.7 | ovl-leaf | S | 65013 | established | 0d:0h:24m:14s| evpn | [15/0/11] |
| default | 10.255.255.9 | ovl-superspine | S | 65003 | established | 0d:0h:24m:7s | evpn | [11/0/4] |

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.255.255.3 4 65002 5618 4742 0 0 0 03:53:35 19 27
10.255.255.4 4 65002 5565 4694 0 0 0 03:51:11 19 27

-------------------------------------------
Check EVPN L2 tables///
-------------------------------------------

Nokia :

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn route type 2
BGP table version is 17, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:100
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
Route Distinguisher: 10.255.255.5:200
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
Route Distinguisher: 10.255.255.6:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
Route Distinguisher: 10.255.255.7:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100
Route Distinguisher: 10.255.255.7:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.200.1]
10.255.255.7 32768 i
ET:8 RT:65000:200
Route Distinguisher: 10.255.255.8:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC

and if we focus on our vni 100

root@leaf3-cum-dc1:/# net show bgp evpn route vni 100 type 2
BGP table version is 48, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100

The additional route type "MAC+IP" is optional and is useful/mandatory when you do anycast irb.

-------------------------------------------
Check tables EVPN L3 ///
-------------------------------------------

Nokia :

--{ + running }--[ ]--
A:leaf2# show network-instance ip-vrf-200 route-table all
-----------------------------------------------------------------------------------------------------------------------------------------------------------
IPv4 Unicast route table of network instance ip-vrf-200
-----------------------------------------------------------------------------------------------------------------------------------------------------------
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
| Prefix | ID | Route Type | Route Owner | Best/Fib- | Metric | Pref | Next-hop | Next-hop |
| | | | | status(slot) | | | (Type) | Interface |
+=======================+=======+============+======================+======================+==========+=========+==============+==============+
| 192.168.2.0/24 | 9 | local | net_inst_mgr | True/success | 0 | 0 | 192.168.2.25 | irb0.200 |
| | | | | | | | 4 (direct) | |
| 192.168.2.254/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (extract) | |
| 192.168.2.255/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (broadcast) | |
| 192.168.200.0/24 | 0 | bgp-evpn | bgp_evpn_mgr | True/success | 0 | 170 | 10.255.255.8 | None |
| | | | | | | | (indirect/vx | |
| | | | | | | | lan) | |
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
-----------------------------------------------------------------------------------------------------------------------------------------------------------
4 IPv4 routes total
4 IPv4 prefixes with active routes
0 IPv4 prefixes with active ECMP routes
-----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--
A:leaf2#

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf1-cum-dc1:/# net show route vrf ip-vrf-200
show ip route vrf ip-vrf-200
=============================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF ip-vrf-200:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 04:48:42
B 192.168.2.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 04:27:26
C>* 192.168.2.0/24 is directly connected, vlan200, 04:48:42
B>* 192.168.200.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:48:49

root@leaf1-cum-dc1:/# net show bgp evpn route type 5
BGP table version is 3, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.6:10200
* [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
*> [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
Route Distinguisher: 10.255.255.8:10200
*> [5]:[0]:[24]:[192.168.200.0]
10.255.255.8 0 65001 65003 65002 65014 i
RT:65000:10200 ET:8 Rmac:02:f8:b8:ff:00:00

-------------------------------------------
Check arp hosts ///
-------------------------------------------

From client #1 /

bash-5.0# arp
Address HWtype HWaddress Flags Mask Iface
192.168.1.3 ether aa:c1:ab:b5:79:50 C eth1.100
192.168.1.4 ether aa:c1:ab:0c:60:d6 C eth1.100
192.168.2.2 ether aa:c1:ab:23:31:04 C eth1.200
192.168.2.254 ether aa:c1:ab:4f:f2:e7 C eth1.200
192.168.1.2 ether aa:c1:ab:23:31:04 C eth1.100
containerlab.local ether 02:42:63:9e:74:03 C eth0

-------------------------------------------
Ping ///
-------------------------------------------

From client #1 /

L2 service ping (vlan 100) ----------------------

bash-5.0# ping -c 1 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.812 ms

--- 192.168.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.812/0.812/0.812/0.000 MsgRcvd
bash-5.0# ping -c 1 192.168.1.3
PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_seq=1 ttl=64 time=1.40 ms

--- 192.168.1.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.403/1.403/1.403/0.000 ms
bash-5.0# ping -c 1 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=1.62 ms

--- 192.168.1.4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.619/1.619/1.619/0.000 ms

L3 service ping (vlan 200)

bash-5.0# ping -c 1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.053 ms

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms
bash-5.0# ^C
bash-5.0# ping -c 1 192.168.2.2
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.611 ms

--- 192.168.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.611/0.611/0.611/0.000 ms

-------------------------------------------
Traceroute ///
-------------------------------------------

L3 service traceroute (vlan 200)

bash-5.0# traceroute 192.168.200.1
traceroute to 192.168.200.1 (192.168.200.1), 30 hops max, 46 byte packets
1 192.168.2.254 (192.168.2.254) 0.012 ms 0.010 ms 0.007 ms
2 192.168.200.254 (192.168.200.254) 2.751 ms * 2.549 ms
3 192.168.200.1 (192.168.200.1) 2.084 ms 1.805 ms 1.761 ms

</syntaxhighlight>To come back on the setup, we wanted to implement solutions inspired by the <nowiki>RFC 7938</nowiki> rfc7938 (see section 5.2.1)

* eBGP will be used for Underlay and Overlay
* eBGP single-hop sessions are established over direct point-to-point links interconnecting the network nodes, no multi-hop or loopback sessions are used, even in the case of multiple links between the same pair of nodes.
* Private Use ASNs from the range 64512-65534 are used to avoid ASN conflicts.
* A single ASN is allocated to all of the Clos topology's Tier 1 devices.
* A unique ASN is allocated to each set of Tier 2 devices in the same cluster.
* A unique ASN is allocated to every Tier 3 device (e.g., ToR) in this topology.

In add, depending of network nodes, it will be interesting to :

* Add fast-failover option for underlay + add link-debounce on links to limit possible impacts in case of link flaps. BFD could be considered in some cases
* Have the best ECMP hashing algorythm on leaf nodes. On tier-1/tier-2 levels hashing will be based on underlay part (trafic inter-vtep), so L3-L4 decision, but some models can retreive the vni-id if I well remember.
* Implement the "multipath relax" since we could have multiple AS paths for a same destination.
* Implement gshut community on underlay (on tier-1/tier-2) or at least policies to prepend underlay routes to avoid some nodes in case of maintenance.

== Thoughts/Points to deepend ==
On leaf nodes, LACP over multiple nodes is feasible within ESI, but impact on incidents/maintenance can vary depending on the vendors.

'''Tests need to be done to check that.'''

Some sysadmin would like to KISS and have client servers connected in active/passive bonding, with eventually single-leaf ports aggregation.

'''Note :''' NH ESI in FIB management on ingress nodes can have some limitations (numbers of installable next-hop, load-balancing, etc. oftenly linked to the node chipset)

Another important engineering point to define at the beginning is : '''Do we authorize multisite L2 propagation or not ?'''

From my point-of-view, the smaller are the L2 segments the better it is. I would suggest Local L2 segments wit anycast gateways provided by Leaf switches, with L3 routing between DCs within EVPN type5 routes sharing only. Nevertheless, some business requirements for endcustomers can keep extended L2 alive.

We have to be careful when implementing Consistent hashing. ex: <nowiki>https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-44/Layer-3/Routing/Equal-Cost-Multipath-Load-Sharing-Hardware-ECMP/</nowiki>

If possible activate it per VRF, not globally, and restrict on dedicated subnets << depends of the vendor, it can be dangerous regarding memory management

Think at configuring the bgp option to not advertise routes to peers before they are installed in FIB.

We can implement leaf roles in some cases, to be able to do "light traffic engineering " on DCI links in some setup.

These roles will be managed within bgp communities added to VTEP /32. Look at the diagram below. Depending of the colour, the eBGP sessions between SupS will prepend or not the VTEP IPs announced to the remote DC

[[File:Fabric-roles.jpg|ohne|mini|1200x1200px]]

<blockquote>'''To be continued'''</blockquote>

== Lab archives ==
<blockquote>Reminder : following config are for lab, so please don't pay attention on typos/lack of security params/unused config. Thanks \o/</blockquote>

=== Credentials ===
<syntaxhighlight lang="bash">
SRLinux : admin/<no_password>

CVX : cumulus/cumulus
</syntaxhighlight>

=== Lab archives ===
layer 2 (vlan 100) + layer 3 (vlan 200) lab, 10/12/2021
[[File:20211008 lab-nokia-cvx-containerlab.zip|ohne|mini]]
<u>If you want to exchange about that lab, those topics, please don't hesitate to contact us</u>

= SONiC Lab =

<span><to be noted that those informations are included into the readme.md file into the archive></span>

== Lab diagram ==
[[File:SONIC_LAB.jpg]]

Lab archive [https://wiki.nesevo.com/images/0/0a/Sonic-lab-20201030.zip Sonic-lab-20201030.zip]

This lab is a forked lab from many found on @, starting from Microsoft one : https://github.com/Azure/SONiC/wiki/SONiC-P4-Software-Switch

The target of this lab is to simulate a simple IP/Fabric in SPINE/LEAF topology. Each Leaf provide local L2 connectivity + L3 gateways to hosts. Flows between hosts from different leaf are L3 routed, in a "node path" LEAF-SPINE-LEAF.

Nothing complicated, but it allows to put a first foot into SONiC world :)

For the moment, as far as we know, MC-LAG + EVPN/VXLAN features are only available on entreprise SONiC versions. Tests to come once physic switches in enterprise version will be available in our lab.

The diagram of the architecture is into the zip file (SONIC_LAB.jpg)

To run the lab you need :

* to have installed docker by using ./install_docker_ovs.sh
* to run the ./load_image.sh in order to get the Sonic image

Note : don't change file / folders names of network devices (leafX, spineX)

Once you're ready, please run the lab by launching "./start.sh"

The script :<div></div>
* launches the containers without Docket network features
* create network namespaces
* build linux bridges to interconnect the containers
* configure host IPs
* launch startup sonic scripts into sonic devices
* once devices started, checks + ping tests are run. If all BGP sessions are ok on spine + all ping are fine, the setup works well.
The "start" script has several comments, perhaps without the right vocabulary, so please be tolerant :)

If you want to stop the lab, please launch "./stop.sh"

Below some useful commands to run for sonic containers :

=== <span>run ping</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> ping <ip>
</syntaxhighlight>
This command triggers ping into the targetted container :
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it leaf1 ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.052 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.034 ms
^C--- 192.168.1.1 ping statistics ---
</syntaxhighlight>

=== <span>check bgp sessions</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip bgp sum"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print the bgp status.
Below, we run it on one spine, you should have 4 sessiosn UP (for the 4 leaves)
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip bgp sum"
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 6409 6411 0 0 0 01:46:46 3
10.1.0.2 4 65102 6406 6410 0 0 0 01:46:44 3
10.1.0.4 4 65103 6407 6410 0 0 0 01:46:44 4
10.1.0.6 4 65104 6406 6412 0 0 0 01:46:46 1
Total number of neighbors 4
</syntaxhighlight>

=== <span>check ip route</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip route"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print its routing table?

<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,
> - selected route, * - FIB route
C>* 10.1.0.0/31 is directly connected, Vlan11
C>* 10.1.0.2/31 is directly connected, Vlan21
C>* 10.1.0.4/31 is directly connected, Vlan31
C>* 10.1.0.6/31 is directly connected, Vlan41
C>* 127.0.0.0/8 is directly connected, lo
B>* 192.168.1.0/24 [20/0] via 10.1.0.0, Vlan11, 01:47:23
B>* 192.168.2.0/24 [20/0] via 10.1.0.2, Vlan21, 01:47:21
B>* 192.168.3.0/24 [20/0] via 10.1.0.4, Vlan31, 01:47:21
B>* 192.168.4.0/24 [20/0] via 10.1.0.6, Vlan41, 01:47:23
</syntaxhighlight>
For more commands check SONiC docs : https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md

To finish, below the output of the start.sh when all goes well<div></div>
plancastre@dockerhost:~/00.Projects/p4-version-pla$ ./start.sh
---------------------------------------------------
Creating the containers: starting
---------------------------------------------------
0b4bbb3c6cfc9895466ea13a9e940d1a176d66f45deee9752e9b4732d735bb03
5ca9486baab3e8a5259062167132a944eee717875d3cc1ef29fe490cba3184c1
46c9c09dba27a32715b9fbe0bc434a9284b9d59317d278df693f425cb669f037
898e6afbcb2977115acbba2a69d069796cfd70fb90228b6c18e09ba4584365f7
eb9b3ef6d14784c9ffe4d9d5df3e921a24559889cb8770d226263aad6761182a
ae5f3b2c628da687eaabb331dc2cd989e71bcb386854e7a0be6ec587df99d49a
122197bffd7566a01ce3b6565fdfb6eb893aab7d5236f14e5a9bbbe95037ced2
a604520a080e2a2184598ad1d915c0c182d16e5185033373e95a4772d8978a67
0c49e18d78b2aa7c8ef8e937b60285aa56ec0bf4a8530226285270a085e11d5c
eb66336b8c6bd8333f1f4f685369e6cac973b817c35bb7126ac4d7ba72ef71bf
---------------------------------------------------
Retreiving PID from docker containers
---------------------------------------------------
---------------------------------------------------
Creating network namespaces for containers
---------------------------------------------------
List of created namespaces
ns_host4
ns_host3
ns_host2
ns_host1
ns_spine2
ns_spine1
ns_leaf4
ns_leaf3
ns_leaf2
ns_leaf1
---------------------------------------------------
Creating network connectivity between containers
---------------------------------------------------
---------------------------------------------------
Now assigning IPs to hosts (host1 to host4)
---------------------------------------------------
---------------------------------------------------
Configuration of containers is now finished
---------------------------------------------------
---------------------------------------------------
Now booting switches, please wait 2 minutes for switches to load
---------------------------------------------------
---------------------------------------------------
2 minutes period is finished
---------------------------------------------------
---------------------------------------------------
Now fixing the iptables
---------------------------------------------------
---------------------------------------------------
Backend actions are now finished
---------------------------------------------------
---------------------------------------------------
Now we'll check in 15 sec that BGP peers are all Up
---------------------------------------------------
---------------------------------------------------
SPINE1 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 94 97 0 0 0 00:01:28 4
10.1.0.2 4 65102 95 97 0 0 0 00:01:28 5
10.1.0.4 4 65103 95 97 0 0 0 00:01:28 5
10.1.0.6 4 65104 95 97 0 0 0 00:01:28 5
Total number of neighbors 4
---------------------------------------------------
SPINE2 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.2.0.1, local AS number 65002
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.2.0.0 4 65101 93 96 0 0 0 00:01:29 2
10.2.0.2 4 65102 94 100 0 0 0 00:01:32 1
10.2.0.4 4 65103 94 100 0 0 0 00:01:32 1
10.2.0.6 4 65104 94 100 0 0 0 00:01:32 1
Total number of neighbors 4
---------------------------------------------------
Now if all is ok, we'll do some ping tests
---------------------------------------------------
---------------------------------------------------
host1 ping its gateway
---------------------------------------------------
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=17.6 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=9.14 ms
--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 9.145/13.390/17.636/4.247 ms
---------------------------------------------------
host2 ping its gateway
---------------------------------------------------
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=23.6 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=6.21 ms
--- 192.168.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 6.218/14.950/23.683/8.733 ms
---------------------------------------------------
host3 ping its gateway
---------------------------------------------------
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=18.8 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=12.7 ms
--- 192.168.3.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.716/15.764/18.812/3.048 ms
---------------------------------------------------
host4 ping its gateway
---------------------------------------------------
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.
64 bytes from 192.168.4.1: icmp_seq=1 ttl=64 time=17.7 ms
64 bytes from 192.168.4.1: icmp_seq=2 ttl=64 time=4.46 ms
--- 192.168.4.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.464/11.107/17.750/6.643 ms
---------------------------------------------------
host1 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=16.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.049/17.253/19.532/1.615 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=15.5 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=14.3 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.312/15.592/16.939/1.083 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=20.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=16.8 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.048/17.871/20.739/2.058 ms
---------------------------------------------------
host2 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=16.6 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.222/16.583/16.924/0.286 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.6 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=18.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=26.8 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.620/20.549/26.887/4.524 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=33.8 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=27.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=20.2 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 20.216/27.259/33.818/5.566 ms
---------------------------------------------------
host3 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=14.5 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=15.5 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.535/16.549/19.541/2.162 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.4 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=14.6 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=18.2 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.643/16.458/18.278/1.491 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=15.3 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=14.1 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=13.1 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 13.171/14.234/15.334/0.883 ms
---------------------------------------------------
host4 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=21.9 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=21.3 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=14.6 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 14.619/19.284/21.932/3.310 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=14.8 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=18.4 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=14.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.148/15.786/18.409/1.879 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=15.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=18.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 15.117/16.788/18.962/1.612 ms
---------------------------------------------------
If bot spine have all BGP sessions and all ping succeeded, all works well
Congrats ! please check readme file to find some details
---------------------------------------------------

Nesevo Virtual Labs

2021-10-29T08:15:59Z

PierreLancastre: /* run ping */

[[File:nesevo-labs-v2.png]]

= Nokia/Cumulus Multi-Vendor Lab =

The targets of this lab are to :

* mix Nokia SR-Linux and Cumulus VXLAN nodes, in spine or leaf roles
* make this environment work for different kinds of setup : cross-DC L2, differentiated L3 gateways per leaf per VRF, anycast gateway cross or intra DC, etc.
* spot compatibility issues if existing

To reach those goals, we'll use the Nokia lab tool "containerlab" https://containerlab.srlinux.dev/, labs from https://clabs.netdevops.me/ helped a lot. Many thanks to the contributors.

We'll need a Linux host with Docker installed on it + kvm virtualization enhancements activated (important if your host is a VM). If virtualization optimization is not ok, you won't be able to launch the most of devices.

Installation details are referenced there : https://containerlab.srlinux.dev/install/

Once installed, you'll need to register the different NOS images you'll use. Cumulus image is present on a public registrery, but Nokia/Arista/Juniper are not, you'll need a vendor support account to retreive them. Once downloaded (.tgz file typically) you can "docker load" the image :

For private images :<syntaxhighlight lang="bash">
sudo docker load -i 21.6.1-235.tar.xz /* (Nokia) */
sudo docker load -i cEOS64-lab-4.26.1F.tar /* (Arista) */
</syntaxhighlight>For public images :<syntaxhighlight lang="bash">
sudo docker pull networkop/cx:4.3.0 /* (Cumulus) */
</syntaxhighlight>

To verify your installed images, you can run "sudo docker images"<syntaxhighlight lang="bash">
plancastre@containerlab:~/00.LAB/senss$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
networkop/cx 4.3.0 a40541ed15ba 2 weeks ago 733MB
srlinux 21.6.1-235 e29aed30e83f 3 weeks ago 1.14GB
networkop/ignite dev dcc5eff29355 4 weeks ago 37.9MB
networkop/kernel 4.19 e794d8875f83 3 months ago 88MB
frrouting/frr v7.5.1 c3e13a4c5918 3 months ago 123MB

</syntaxhighlight>Once you've got all your images, to run the lab, you'll need a .yml file defining the topology to run.

In our lab, we've started with the 5-clos lab file from Nokia site, which we've forked.

We can see there is a branch "kinds" where we can define templates of nodes, in order to not rewrite whole parameters when creating nodes. For the images, it's important to take the repository name + tag.

'''Note:''' If the tag is "latest" it is not needed.

All supported "kinds" are referenced there : https://containerlab.srlinux.dev/manual/kinds/kinds/

'''Note:''' yml is indentation sensitive, so please respect the "double space" used to define hierarchy. Below the lab diagram + the yml file associated

== Infos on nodes ==

=== L3 infos ===
{| class="wikitable"
|+
!Device
!OS
!ASN
!VTEP IP
!swp1 or eth1/1
!swp2 or eth1/2
!swp3 or eth1/3
!SVI
|-
|client1
|Linux
|<none>
|<none>
|vlan100: 192.168.1.1/24
vlan200: 192.168.2.1/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client2
|Linux
|<none>
|<none>
|vlan100: 192.168.1.2/24
vlan200: 192.168.2.2/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client3
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.1/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|client4
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.2/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|leaf1
|CVX
|65011
|10.255.255.5/32
|10.254.254.8/31
|10.254.254.10/31
|trunk vlan 100,200
|vlan200 : 192.168.2.254/24
|-
|leaf2
|SRLinux
|65012
|10.255.255.6/32
|10.254.254.12/31
|10.254.254.14/31
|trunk vlan 100,200
|irb0.200 : 192.168.2.254/24
|-
|leaf3
|CVX
|65013
|10.255.255.7/32
|10.254.254.22/31
|10.254.254.24/31
|trunk vlan 100,200
|vlan200 : 192.168.200.254/24
|-
|leaf4
|SRLinux
|65014
|10.255.255.8/32
|10.254.254.26/31
|10.254.254.28/31
|trunk vlan 100,200
|irb0.200 : 192.168.200.254/24
|-
|spine1
|CVX
|65001
|10.255.255.1/32
|10.254.254.9/31
|10.254.254.13/31
|10.254.254.0/31
|<none>
|-
|spine2
|CVX
|65001
|10.255.255.2/32
|10.254.254.11/31
|10.254.254.15/31
|10.254.254.16/31
|<none>
|-
|spine3
|SRLinux
|65002
|10.255.255.3/32
|10.254.254.23/31
|10.254.254.27/31
|10.254.254.18/31
|<none>
|-
|spine4
|SRLinux
|65002
|10.255.255.4/32
|10.254.254.25/31
|10.254.254.29/31
|10.254.254.20/31
|<none>
|-
|superspine1
|CVX
|65003
|10.255.255.9/32
|10.254.254.1/31
|10.254.254.19/31
|<none>
|<none>
|-
|superspine2
|SRLinux
|65003
|10.255.255.10/32
|10.254.254.17/31
|10.254.254.21/31
|<none>
|<none>
|}

=== VXLAN/BGP infos ===
Note : in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>
{| class="wikitable"
|+
!service
!vlan id
!route-target
!VNI
!Commentaires
|-
|L2 cross-DC
|100
|target:65000:100
|100
|the 4 clients are in the same broadcast domain / subnet
|-
|L3VPN cros-DC
|200
|target:65000:200
target:65000:10200
|200 (for L2)
10200 (for routing)
|the 4 clients are separated in 2 different subnets. Traffic is routed within dedicated VNI
To be noted that in the current setup VMTO is not implemented, so cross DC traffic is not optimized.
|}

== Lab diagram ==
[[File:Lab-srl-diagram.png|ohne|mini|600x600px]]

== yml lab file ==
<syntaxhighlight lang="yaml">
# topology documentation: http://containerlab.srlinux.dev/lab-examples/min-5clos/
name: lab-srl-cum

topology:
kinds:
srl:
image: srlinux:21.6.1-235
linux:
image: ghcr.io/hellt/network-multitool
cvx:
image: networkop/cx:4.3.0
runtime: docker
nodes:
leaf1:
kind: cvx
leaf2:
kind: srl
type: ixrd2
leaf3:
kind: cvx
leaf4:
kind: srl
type: ixrd2

spine1:
kind: cvx
spine2:
kind: cvx
spine3:
kind: srl
type: ixr6
spine4:
kind: srl
type: ixr6

superspine1:
kind: cvx
superspine2:
kind: srl
type: ixr6

client1:
kind: linux
client2:
kind: linux
client3:
kind: linux
client4:
kind: linux

links:
# leaf to spine links POD1
- endpoints: ["leaf1:swp1", "spine1:swp1"]
- endpoints: ["leaf1:swp2", "spine2:swp1"]
- endpoints: ["leaf2:e1-1", "spine1:swp2"]
- endpoints: ["leaf2:e1-2", "spine2:swp2"]

# spine to superspine links POD1
- endpoints: ["spine1:swp3", "superspine1:swp1"]
- endpoints: ["spine2:swp3", "superspine2:e1-1"]

# leaf to spine links POD2
- endpoints: ["leaf3:swp1", "spine3:e1-1"]
- endpoints: ["leaf3:swp2", "spine4:e1-1"]
- endpoints: ["leaf4:e1-1", "spine3:e1-2"]
- endpoints: ["leaf4:e1-2", "spine4:e1-2"]

# spine to superspine links POD2
- endpoints: ["spine3:e1-3", "superspine1:swp2"]
- endpoints: ["spine4:e1-3", "superspine2:e1-2"]

# client connection links
- endpoints: ["client1:eth1", "leaf1:swp3"]
- endpoints: ["client2:eth1", "leaf2:e1-3"]
- endpoints: ["client3:eth1", "leaf3:swp3"]
- endpoints: ["client4:eth1", "leaf4:e1-3"]

</syntaxhighlight>

Once the nodes are defined, we have to define the needed connections between the nodes.

Each endpoint is defined as '<nowiki/>'''<nowiki/>'<node_name>:<interface_name><nowiki>''</nowiki>'''

When all is ok, you can launch in suduoer the topology.

For example :<syntaxhighlight lang="bash">
sudo containerlab deploy --topo topo.clab.yml --reconfigure

Note: The option "reconfigure" is used there in order to not have to destroy instances when updating topology/relaunching.
</syntaxhighlight>Another info regarding the yml file, it's possible to quickly generate files with the '''"containerlab gen -n 3tier --nodes 4,2,1"''' command for example. https://containerlab.srlinux.dev/cmd/generate/

To be noted that the "tier-x" notion in containerlab is inversed regarding the rfc 7938 (tier1 is superspine level into the rfc, tier1 is leaf level in nokia tool)

At the end of the topology launch, you can retreive the details :<syntaxhighlight lang="yaml">
INFO[0011] Writing /etc/hosts file
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| # | Name | Container ID | Image | Kind | Group | State | IPv4 Address | IPv6 Address |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| 1 | clab-lab-srl-cum-client1 | 25e2bbf6b13f | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
| 2 | clab-lab-srl-cum-client2 | 17d82d5a6e21 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.4/24 | 2001:172:20:20::4/64 |
| 3 | clab-lab-srl-cum-client3 | f8d78527c1b1 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.6/24 | 2001:172:20:20::6/64 |
| 4 | clab-lab-srl-cum-client4 | 913a0172d0c5 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 5 | clab-lab-srl-cum-leaf1 | 9ffc334af2b1 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.9/24 | 2001:172:20:20::9/64 |
| 6 | clab-lab-srl-cum-leaf2 | 7d3eb855d749 | srlinux:21.6.1-235 | srl | | running | 172.20.20.14/24 | 2001:172:20:20::e/64 |
| 7 | clab-lab-srl-cum-leaf3 | e0d36ffc606a | networkop/cx:4.3.0 | cvx | | running | 172.20.20.7/24 | 2001:172:20:20::7/64 |
| 8 | clab-lab-srl-cum-leaf4 | e6753b1d099e | srlinux:21.6.1-235 | srl | | running | 172.20.20.12/24 | 2001:172:20:20::c/64 |
| 9 | clab-lab-srl-cum-spine1 | a7e63993d052 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.5/24 | 2001:172:20:20::5/64 |
| 10 | clab-lab-srl-cum-spine2 | 3124250d5b75 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
| 11 | clab-lab-srl-cum-spine3 | 5e300ca3f66f | srlinux:21.6.1-235 | srl | | running | 172.20.20.11/24 | 2001:172:20:20::b/64 |
| 12 | clab-lab-srl-cum-spine4 | 0a59e60a596b | srlinux:21.6.1-235 | srl | | running | 172.20.20.13/24 | 2001:172:20:20::d/64 |
| 13 | clab-lab-srl-cum-superspine1 | 3d9935413fc2 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.8/24 | 2001:172:20:20::8/64 |
| 14 | clab-lab-srl-cum-superspine2 | 5bc58d423e2c | srlinux:21.6.1-235 | srl | | running | 172.20.20.15/24 | 2001:172:20:20::f/64 |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
</syntaxhighlight>The containerlab program creates the updates into the host file + generates an ansible inventory file.

There are different ways to connect onto the docker instances, within docker exec commands or SSH directly the IP address related above.

For example :<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-spine1 bash
</syntaxhighlight>-->>> allows to launch a bash on the cumulus device (then you run the different net commands)<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-leaf2 sr_cli
</syntaxhighlight>-->>> allows to launch the cli on a SRLinux device

To end this setup preparation, the containerlab tool integrates an option to launch a webserver showing the lab topology in graph + node details

By default, it's listening on port 50080 of the host

Then, you can access to the following web page showing the topology + lab infos

[[File:Lab graph tool.jpg|ohne|mini|900x900px]]

Finally, if you want to stop a lab, please run '''<nowiki>''</nowiki>containerlab destroy --topo <topology>.yml<nowiki>''</nowiki>'''

At this step, nodes are running, connections are done, but configurations are empty.

'''So, it's time to configure them :)'''

Configuration files are attached to this blog. Once validated, they are exportable and can be called by the tool when launching the lab. Below how to do it :

Firstable, once you ve finished your "hand made config", you can run the following command to save it :<syntaxhighlight lang="bash">
containerlab save -t quickstart.clab.yml
</syntaxhighlight>'''To be noted it's only working for Nokia nodes in this lab'''. CVX configurations will have to be called by another method (or pushed manually)

It will create one folder per node into the lab folder which gives the following tree :
[[File:Topology-srl.jpg|ohne|mini]]

'''Note''': For Cumulus, you can apparently retrieve configurations files within adding the following parameters into lab config file. '''To be tested.''' some lab samples here : https://github.com/hellt/clabs/blob/main/labs/cvx04/symm-mh.clab.yml

Now we have the lab running, below the different commands to check that all is working as expected : <syntaxhighlight lang="yaml">
-------------------------------------------
Check L2 tables ///
-------------------------------------------

Nokia :

A:leaf2# show network-instance mac-vrf-100 bridge-table mac-table all
----------------------------------------------------------------------------------------------------------------------------------------------------------
Mac-table of network instance mac-vrf-100
----------------------------------------------------------------------------------------------------------------------------------------------------------
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
| Address | Destination | Dest Index | Type | Active | Aging | Last Update |
+====================+=========================================+============+=============+=========+========+=========================================+
| AA:C1:AB:0C:60:D6 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T13:06:17.000Z |
| | vtep:10.255.255.8 vni:100 | 67 | | | | |
| AA:C1:AB:23:31:04 | ethernet-1/3.100 | 2 | learnt | true | 300 | 2021-10-08T13:06:10.000Z |
| AA:C1:AB:4B:FF:71 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:55:28.000Z |
| | vtep:10.255.255.5 vni:100 | 60 | | | | |
| AA:C1:AB:B5:79:50 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:43:37.000Z |
| | vtep:10.255.255.7 vni:100 | 64 | | | | |
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 0 Total 0 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 1 Total 1 Active
Total Evpn Macs : 3 Total 3 Active
Total Evpn static Macs : 0 Total 0 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 4 Total 4 Active
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : type learnt matches the local learnt mac, evpn are learnt within VXLAN

Cumulus :

root@leaf1-cum-dc1:/# net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
100 bridge swp3 aa:c1:ab:4b:ff:71 00:00:02
100 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:17:21
100 bridge vni100 22:2d:67:f1:bb:26 permanent 02:17:21
100 bridge vni100 aa:c1:ab:0c:60:d6 extern_learn <1 sec
100 bridge vni100 aa:c1:ab:23:31:04 extern_learn 00:00:07
100 bridge vni100 aa:c1:ab:b5:79:50 extern_learn 00:13:28

root@leaf1-cum-dc1:/# net show bridge macs vlan 200

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
200 bridge swp3 aa:c1:ab:4b:ff:71 00:08:46
200 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:16:36
200 bridge vni200 00:00:5e:00:01:01 static sticky 00:14:26
200 bridge vni200 1e:0d:aa:46:53:b1 permanent 02:16:36
200 bridge vni200 02:77:6b:ff:00:41 static sticky 01:00:52
200 bridge vni200 02:f8:b8:ff:00:41 static sticky 00:14:26
200 bridge vni200 aa:c1:ab:b5:79:50 extern_learn 00:11:51
200 bridge vni10200 02:77:6b:ff:00:00 extern_learn 00:53:02
200 bridge vni10200 02:f8:b8:ff:00:00 extern_learn 00:14:26
200 bridge vni10200 86:5e:aa:ca:50:be permanent 02:16:36

Note : the macs with permanent flag are local interfaces, physical ou logic (vtep)
Note2 : extern_learn mac are the one learned via the VXLAN infra (and mapped to the vnixxx interface)

-------------------------------------------
Check arp tables ///
-------------------------------------------

Nokia :

:leaf2# show arpnd arp-entries
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| ethernet-1/1 | 0 | 10.254.254.13 | dynamic | AA:C1:AB:59:65:EA | 3 hours from now |
| ethernet-1/2 | 0 | 10.254.254.15 | dynamic | AA:C1:AB:CB:F8:8A | 3 hours from now |
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
| mgmt0 | 0 | 172.20.20.1 | dynamic | 02:42:63:9E:74:03 | 2 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 5 (0 static, 5 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : on Nokia you have to sort the entries basing on subinterface and the retrieve the L3VPN attached

A:leaf2# show arpnd arp-entries subinterface 200
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 2 (0 static, 2 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Cumulus :

Here we retrieve all arp infos linked to evpn

root@leaf1-cum-dc1:/# net show evpn arp-cache vni all
VNI 100 #ARP (IPv4 and IPv6, local and remote) 2

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.1.3 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0
192.168.1.1 local active aa:c1:ab:4b:ff:71 0/0

VNI 200 #ARP (IPv4 and IPv6, local and remote) 4

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.2.1 local active aa:c1:ab:4b:ff:71 0/0
192.168.200.254 remote active 00:00:5e:00:01:01 10.255.255.8 0/0
192.168.2.254 remote active 00:00:5e:00:01:01 10.255.255.6 0/0
192.168.200.1 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0

VNI 400 #ARP (IPv4 and IPv6, local and remote) 0

-------------------------------------------
Check bgp sessions' status - underlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep ipv4
| default | 10.254.254.19 | udl-superspine | S | 65003 | established | 0d:0h:22m:44s | ipv4-unicast | [13/4/4] |
| default | 10.254.254.22 | udl-leaf | S | 65013 | established | 0d:0h:22m:52s | ipv4-unicast | [13/1/4] |

Cumulus :

root@leaf3:/# net show bgp ipv4 unicast summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 13
RIB entries 25, using 4800 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.254.254.23 4 65002 466 403 0 0 0 00:19:34 4 13
10.254.254.25 4 65002 434 355 0 0 0 00:17:10 8 13

-------------------------------------------
Check bgp sessions' status - overlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep evpn
| default | 10.255.255.7 | ovl-leaf | S | 65013 | established | 0d:0h:24m:14s| evpn | [15/0/11] |
| default | 10.255.255.9 | ovl-superspine | S | 65003 | established | 0d:0h:24m:7s | evpn | [11/0/4] |

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.255.255.3 4 65002 5618 4742 0 0 0 03:53:35 19 27
10.255.255.4 4 65002 5565 4694 0 0 0 03:51:11 19 27

-------------------------------------------
Check EVPN L2 tables///
-------------------------------------------

Nokia :

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn route type 2
BGP table version is 17, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:100
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
Route Distinguisher: 10.255.255.5:200
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
Route Distinguisher: 10.255.255.6:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
Route Distinguisher: 10.255.255.7:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100
Route Distinguisher: 10.255.255.7:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.200.1]
10.255.255.7 32768 i
ET:8 RT:65000:200
Route Distinguisher: 10.255.255.8:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC

and if we focus on our vni 100

root@leaf3-cum-dc1:/# net show bgp evpn route vni 100 type 2
BGP table version is 48, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100

The additional route type "MAC+IP" is optional and is useful/mandatory when you do anycast irb.

-------------------------------------------
Check tables EVPN L3 ///
-------------------------------------------

Nokia :

--{ + running }--[ ]--
A:leaf2# show network-instance ip-vrf-200 route-table all
-----------------------------------------------------------------------------------------------------------------------------------------------------------
IPv4 Unicast route table of network instance ip-vrf-200
-----------------------------------------------------------------------------------------------------------------------------------------------------------
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
| Prefix | ID | Route Type | Route Owner | Best/Fib- | Metric | Pref | Next-hop | Next-hop |
| | | | | status(slot) | | | (Type) | Interface |
+=======================+=======+============+======================+======================+==========+=========+==============+==============+
| 192.168.2.0/24 | 9 | local | net_inst_mgr | True/success | 0 | 0 | 192.168.2.25 | irb0.200 |
| | | | | | | | 4 (direct) | |
| 192.168.2.254/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (extract) | |
| 192.168.2.255/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (broadcast) | |
| 192.168.200.0/24 | 0 | bgp-evpn | bgp_evpn_mgr | True/success | 0 | 170 | 10.255.255.8 | None |
| | | | | | | | (indirect/vx | |
| | | | | | | | lan) | |
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
-----------------------------------------------------------------------------------------------------------------------------------------------------------
4 IPv4 routes total
4 IPv4 prefixes with active routes
0 IPv4 prefixes with active ECMP routes
-----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--
A:leaf2#

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf1-cum-dc1:/# net show route vrf ip-vrf-200
show ip route vrf ip-vrf-200
=============================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF ip-vrf-200:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 04:48:42
B 192.168.2.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 04:27:26
C>* 192.168.2.0/24 is directly connected, vlan200, 04:48:42
B>* 192.168.200.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:48:49

root@leaf1-cum-dc1:/# net show bgp evpn route type 5
BGP table version is 3, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.6:10200
* [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
*> [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
Route Distinguisher: 10.255.255.8:10200
*> [5]:[0]:[24]:[192.168.200.0]
10.255.255.8 0 65001 65003 65002 65014 i
RT:65000:10200 ET:8 Rmac:02:f8:b8:ff:00:00

-------------------------------------------
Check arp hosts ///
-------------------------------------------

From client #1 /

bash-5.0# arp
Address HWtype HWaddress Flags Mask Iface
192.168.1.3 ether aa:c1:ab:b5:79:50 C eth1.100
192.168.1.4 ether aa:c1:ab:0c:60:d6 C eth1.100
192.168.2.2 ether aa:c1:ab:23:31:04 C eth1.200
192.168.2.254 ether aa:c1:ab:4f:f2:e7 C eth1.200
192.168.1.2 ether aa:c1:ab:23:31:04 C eth1.100
containerlab.local ether 02:42:63:9e:74:03 C eth0

-------------------------------------------
Ping ///
-------------------------------------------

From client #1 /

L2 service ping (vlan 100) ----------------------

bash-5.0# ping -c 1 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.812 ms

--- 192.168.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.812/0.812/0.812/0.000 MsgRcvd
bash-5.0# ping -c 1 192.168.1.3
PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_seq=1 ttl=64 time=1.40 ms

--- 192.168.1.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.403/1.403/1.403/0.000 ms
bash-5.0# ping -c 1 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=1.62 ms

--- 192.168.1.4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.619/1.619/1.619/0.000 ms

L3 service ping (vlan 200)

bash-5.0# ping -c 1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.053 ms

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms
bash-5.0# ^C
bash-5.0# ping -c 1 192.168.2.2
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.611 ms

--- 192.168.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.611/0.611/0.611/0.000 ms

-------------------------------------------
Traceroute ///
-------------------------------------------

L3 service traceroute (vlan 200)

bash-5.0# traceroute 192.168.200.1
traceroute to 192.168.200.1 (192.168.200.1), 30 hops max, 46 byte packets
1 192.168.2.254 (192.168.2.254) 0.012 ms 0.010 ms 0.007 ms
2 192.168.200.254 (192.168.200.254) 2.751 ms * 2.549 ms
3 192.168.200.1 (192.168.200.1) 2.084 ms 1.805 ms 1.761 ms

</syntaxhighlight>To come back on the setup, we wanted to implement solutions inspired by the <nowiki>RFC 7938</nowiki> rfc7938 (see section 5.2.1)

* eBGP will be used for Underlay and Overlay
* eBGP single-hop sessions are established over direct point-to-point links interconnecting the network nodes, no multi-hop or loopback sessions are used, even in the case of multiple links between the same pair of nodes.
* Private Use ASNs from the range 64512-65534 are used to avoid ASN conflicts.
* A single ASN is allocated to all of the Clos topology's Tier 1 devices.
* A unique ASN is allocated to each set of Tier 2 devices in the same cluster.
* A unique ASN is allocated to every Tier 3 device (e.g., ToR) in this topology.

In add, depending of network nodes, it will be interesting to :

* Add fast-failover option for underlay + add link-debounce on links to limit possible impacts in case of link flaps. BFD could be considered in some cases
* Have the best ECMP hashing algorythm on leaf nodes. On tier-1/tier-2 levels hashing will be based on underlay part (trafic inter-vtep), so L3-L4 decision, but some models can retreive the vni-id if I well remember.
* Implement the "multipath relax" since we could have multiple AS paths for a same destination.
* Implement gshut community on underlay (on tier-1/tier-2) or at least policies to prepend underlay routes to avoid some nodes in case of maintenance.

== Thoughts/Points to deepend ==
On leaf nodes, LACP over multiple nodes is feasible within ESI, but impact on incidents/maintenance can vary depending on the vendors.

'''Tests need to be done to check that.'''

Some sysadmin would like to KISS and have client servers connected in active/passive bonding, with eventually single-leaf ports aggregation.

'''Note :''' NH ESI in FIB management on ingress nodes can have some limitations (numbers of installable next-hop, load-balancing, etc. oftenly linked to the node chipset)

Another important engineering point to define at the beginning is : '''Do we authorize multisite L2 propagation or not ?'''

From my point-of-view, the smaller are the L2 segments the better it is. I would suggest Local L2 segments wit anycast gateways provided by Leaf switches, with L3 routing between DCs within EVPN type5 routes sharing only. Nevertheless, some business requirements for endcustomers can keep extended L2 alive.

We have to be careful when implementing Consistent hashing. ex: <nowiki>https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-44/Layer-3/Routing/Equal-Cost-Multipath-Load-Sharing-Hardware-ECMP/</nowiki>

If possible activate it per VRF, not globally, and restrict on dedicated subnets << depends of the vendor, it can be dangerous regarding memory management

Think at configuring the bgp option to not advertise routes to peers before they are installed in FIB.

We can implement leaf roles in some cases, to be able to do "light traffic engineering " on DCI links in some setup.

These roles will be managed within bgp communities added to VTEP /32. Look at the diagram below. Depending of the colour, the eBGP sessions between SupS will prepend or not the VTEP IPs announced to the remote DC

[[File:Fabric-roles.jpg|ohne|mini|1200x1200px]]

<blockquote>'''To be continued'''</blockquote>

== Lab archives ==
<blockquote>Reminder : following config are for lab, so please don't pay attention on typos/lack of security params/unused config. Thanks \o/</blockquote>

=== Credentials ===
<syntaxhighlight lang="bash">
SRLinux : admin/<no_password>

CVX : cumulus/cumulus
</syntaxhighlight>

=== Lab archives ===
layer 2 (vlan 100) + layer 3 (vlan 200) lab, 10/12/2021
[[File:20211008 lab-nokia-cvx-containerlab.zip|ohne|mini]]
<u>If you want to exchange about that lab, those topics, please don't hesitate to contact us</u>

= SONiC Lab =

<span><to be noted that those informations are included into the readme.md file into the archive></span>

== Lab diagram ==
[[File:SONIC_LAB.jpg]]

Lab archive [https://wiki.nesevo.com/images/0/0a/Sonic-lab-20201030.zip Sonic-lab-20201030.zip]

This lab is a forked lab from many found on @, starting from Microsoft one : https://github.com/Azure/SONiC/wiki/SONiC-P4-Software-Switch

The target of this lab is to simulate a simple IP/Fabric in SPINE/LEAF topology. Each Leaf provide local L2 connectivity + L3 gateways to hosts. Flows between hosts from different leaf are L3 routed, in a "node path" LEAF-SPINE-LEAF.

Nothing complicated, but it allows to put a first foot into SONiC world :)

For the moment, as far as we know, MC-LAG + EVPN/VXLAN features are only available on entreprise SONiC versions. Tests to come once physic switches in enterprise version will be available in our lab.

The diagram of the architecture is into the zip file (SONIC_LAB.jpg)

To run the lab you need :

- to have installed docker by using ./install_docker_ovs.sh - to run the ./load_image.sh in order to get the Sonic image

Note : don't change file / folders names of network devices (leafX, spineX)

Once you're ready, please run the lab by launching "./start.sh"

The script :<div></div>
* launches the containers without Docket network features
* create network namespaces
* build linux bridges to interconnect the containers
* configure host IPs
* launch startup sonic scripts into sonic devices
* once devices started, checks + ping tests are run. If all BGP sessions are ok on spine + all ping are fine, the setup works well.
The "start" script has several comments, perhaps without the right vocabulary, so please be tolerant :)

If you want to stop the lab, please launch "./stop.sh"

Below some useful commands to run for sonic containers :

=== <span>run ping</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> ping <ip>
</syntaxhighlight>
This command triggers ping into the targetted container :
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it leaf1 ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.052 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.034 ms
^C--- 192.168.1.1 ping statistics ---
</syntaxhighlight>

=== <span>check bgp sessions</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip bgp sum"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print the bgp status.
Below, we run it on one spine, you should have 4 sessiosn UP (for the 4 leaves)
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip bgp sum"
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 6409 6411 0 0 0 01:46:46 3
10.1.0.2 4 65102 6406 6410 0 0 0 01:46:44 3
10.1.0.4 4 65103 6407 6410 0 0 0 01:46:44 4
10.1.0.6 4 65104 6406 6412 0 0 0 01:46:46 1
Total number of neighbors 4
</syntaxhighlight>

=== <span>check ip route</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip route"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print its routing table?

<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,
> - selected route, * - FIB route
C>* 10.1.0.0/31 is directly connected, Vlan11
C>* 10.1.0.2/31 is directly connected, Vlan21
C>* 10.1.0.4/31 is directly connected, Vlan31
C>* 10.1.0.6/31 is directly connected, Vlan41
C>* 127.0.0.0/8 is directly connected, lo
B>* 192.168.1.0/24 [20/0] via 10.1.0.0, Vlan11, 01:47:23
B>* 192.168.2.0/24 [20/0] via 10.1.0.2, Vlan21, 01:47:21
B>* 192.168.3.0/24 [20/0] via 10.1.0.4, Vlan31, 01:47:21
B>* 192.168.4.0/24 [20/0] via 10.1.0.6, Vlan41, 01:47:23
</syntaxhighlight>
For more commands check SONiC docs : https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md

To finish, below the output of the start.sh when all goes well<div></div>
plancastre@dockerhost:~/00.Projects/p4-version-pla$ ./start.sh
---------------------------------------------------
Creating the containers: starting
---------------------------------------------------
0b4bbb3c6cfc9895466ea13a9e940d1a176d66f45deee9752e9b4732d735bb03
5ca9486baab3e8a5259062167132a944eee717875d3cc1ef29fe490cba3184c1
46c9c09dba27a32715b9fbe0bc434a9284b9d59317d278df693f425cb669f037
898e6afbcb2977115acbba2a69d069796cfd70fb90228b6c18e09ba4584365f7
eb9b3ef6d14784c9ffe4d9d5df3e921a24559889cb8770d226263aad6761182a
ae5f3b2c628da687eaabb331dc2cd989e71bcb386854e7a0be6ec587df99d49a
122197bffd7566a01ce3b6565fdfb6eb893aab7d5236f14e5a9bbbe95037ced2
a604520a080e2a2184598ad1d915c0c182d16e5185033373e95a4772d8978a67
0c49e18d78b2aa7c8ef8e937b60285aa56ec0bf4a8530226285270a085e11d5c
eb66336b8c6bd8333f1f4f685369e6cac973b817c35bb7126ac4d7ba72ef71bf
---------------------------------------------------
Retreiving PID from docker containers
---------------------------------------------------
---------------------------------------------------
Creating network namespaces for containers
---------------------------------------------------
List of created namespaces
ns_host4
ns_host3
ns_host2
ns_host1
ns_spine2
ns_spine1
ns_leaf4
ns_leaf3
ns_leaf2
ns_leaf1
---------------------------------------------------
Creating network connectivity between containers
---------------------------------------------------
---------------------------------------------------
Now assigning IPs to hosts (host1 to host4)
---------------------------------------------------
---------------------------------------------------
Configuration of containers is now finished
---------------------------------------------------
---------------------------------------------------
Now booting switches, please wait 2 minutes for switches to load
---------------------------------------------------
---------------------------------------------------
2 minutes period is finished
---------------------------------------------------
---------------------------------------------------
Now fixing the iptables
---------------------------------------------------
---------------------------------------------------
Backend actions are now finished
---------------------------------------------------
---------------------------------------------------
Now we'll check in 15 sec that BGP peers are all Up
---------------------------------------------------
---------------------------------------------------
SPINE1 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 94 97 0 0 0 00:01:28 4
10.1.0.2 4 65102 95 97 0 0 0 00:01:28 5
10.1.0.4 4 65103 95 97 0 0 0 00:01:28 5
10.1.0.6 4 65104 95 97 0 0 0 00:01:28 5
Total number of neighbors 4
---------------------------------------------------
SPINE2 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.2.0.1, local AS number 65002
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.2.0.0 4 65101 93 96 0 0 0 00:01:29 2
10.2.0.2 4 65102 94 100 0 0 0 00:01:32 1
10.2.0.4 4 65103 94 100 0 0 0 00:01:32 1
10.2.0.6 4 65104 94 100 0 0 0 00:01:32 1
Total number of neighbors 4
---------------------------------------------------
Now if all is ok, we'll do some ping tests
---------------------------------------------------
---------------------------------------------------
host1 ping its gateway
---------------------------------------------------
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=17.6 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=9.14 ms
--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 9.145/13.390/17.636/4.247 ms
---------------------------------------------------
host2 ping its gateway
---------------------------------------------------
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=23.6 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=6.21 ms
--- 192.168.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 6.218/14.950/23.683/8.733 ms
---------------------------------------------------
host3 ping its gateway
---------------------------------------------------
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=18.8 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=12.7 ms
--- 192.168.3.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.716/15.764/18.812/3.048 ms
---------------------------------------------------
host4 ping its gateway
---------------------------------------------------
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.
64 bytes from 192.168.4.1: icmp_seq=1 ttl=64 time=17.7 ms
64 bytes from 192.168.4.1: icmp_seq=2 ttl=64 time=4.46 ms
--- 192.168.4.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.464/11.107/17.750/6.643 ms
---------------------------------------------------
host1 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=16.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.049/17.253/19.532/1.615 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=15.5 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=14.3 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.312/15.592/16.939/1.083 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=20.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=16.8 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.048/17.871/20.739/2.058 ms
---------------------------------------------------
host2 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=16.6 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.222/16.583/16.924/0.286 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.6 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=18.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=26.8 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.620/20.549/26.887/4.524 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=33.8 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=27.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=20.2 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 20.216/27.259/33.818/5.566 ms
---------------------------------------------------
host3 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=14.5 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=15.5 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.535/16.549/19.541/2.162 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.4 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=14.6 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=18.2 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.643/16.458/18.278/1.491 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=15.3 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=14.1 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=13.1 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 13.171/14.234/15.334/0.883 ms
---------------------------------------------------
host4 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=21.9 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=21.3 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=14.6 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 14.619/19.284/21.932/3.310 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=14.8 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=18.4 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=14.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.148/15.786/18.409/1.879 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=15.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=18.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 15.117/16.788/18.962/1.612 ms
---------------------------------------------------
If bot spine have all BGP sessions and all ping succeeded, all works well
Congrats ! please check readme file to find some details
---------------------------------------------------

Nesevo Virtual Labs

2021-10-29T08:15:14Z

PierreLancastre: /* Lab diagram */

[[File:nesevo-labs-v2.png]]

= Nokia/Cumulus Multi-Vendor Lab =

The targets of this lab are to :

* mix Nokia SR-Linux and Cumulus VXLAN nodes, in spine or leaf roles
* make this environment work for different kinds of setup : cross-DC L2, differentiated L3 gateways per leaf per VRF, anycast gateway cross or intra DC, etc.
* spot compatibility issues if existing

To reach those goals, we'll use the Nokia lab tool "containerlab" https://containerlab.srlinux.dev/, labs from https://clabs.netdevops.me/ helped a lot. Many thanks to the contributors.

We'll need a Linux host with Docker installed on it + kvm virtualization enhancements activated (important if your host is a VM). If virtualization optimization is not ok, you won't be able to launch the most of devices.

Installation details are referenced there : https://containerlab.srlinux.dev/install/

Once installed, you'll need to register the different NOS images you'll use. Cumulus image is present on a public registrery, but Nokia/Arista/Juniper are not, you'll need a vendor support account to retreive them. Once downloaded (.tgz file typically) you can "docker load" the image :

For private images :<syntaxhighlight lang="bash">
sudo docker load -i 21.6.1-235.tar.xz /* (Nokia) */
sudo docker load -i cEOS64-lab-4.26.1F.tar /* (Arista) */
</syntaxhighlight>For public images :<syntaxhighlight lang="bash">
sudo docker pull networkop/cx:4.3.0 /* (Cumulus) */
</syntaxhighlight>

To verify your installed images, you can run "sudo docker images"<syntaxhighlight lang="bash">
plancastre@containerlab:~/00.LAB/senss$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
networkop/cx 4.3.0 a40541ed15ba 2 weeks ago 733MB
srlinux 21.6.1-235 e29aed30e83f 3 weeks ago 1.14GB
networkop/ignite dev dcc5eff29355 4 weeks ago 37.9MB
networkop/kernel 4.19 e794d8875f83 3 months ago 88MB
frrouting/frr v7.5.1 c3e13a4c5918 3 months ago 123MB

</syntaxhighlight>Once you've got all your images, to run the lab, you'll need a .yml file defining the topology to run.

In our lab, we've started with the 5-clos lab file from Nokia site, which we've forked.

We can see there is a branch "kinds" where we can define templates of nodes, in order to not rewrite whole parameters when creating nodes. For the images, it's important to take the repository name + tag.

'''Note:''' If the tag is "latest" it is not needed.

All supported "kinds" are referenced there : https://containerlab.srlinux.dev/manual/kinds/kinds/

'''Note:''' yml is indentation sensitive, so please respect the "double space" used to define hierarchy. Below the lab diagram + the yml file associated

== Infos on nodes ==

=== L3 infos ===
{| class="wikitable"
|+
!Device
!OS
!ASN
!VTEP IP
!swp1 or eth1/1
!swp2 or eth1/2
!swp3 or eth1/3
!SVI
|-
|client1
|Linux
|<none>
|<none>
|vlan100: 192.168.1.1/24
vlan200: 192.168.2.1/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client2
|Linux
|<none>
|<none>
|vlan100: 192.168.1.2/24
vlan200: 192.168.2.2/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client3
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.1/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|client4
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.2/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|leaf1
|CVX
|65011
|10.255.255.5/32
|10.254.254.8/31
|10.254.254.10/31
|trunk vlan 100,200
|vlan200 : 192.168.2.254/24
|-
|leaf2
|SRLinux
|65012
|10.255.255.6/32
|10.254.254.12/31
|10.254.254.14/31
|trunk vlan 100,200
|irb0.200 : 192.168.2.254/24
|-
|leaf3
|CVX
|65013
|10.255.255.7/32
|10.254.254.22/31
|10.254.254.24/31
|trunk vlan 100,200
|vlan200 : 192.168.200.254/24
|-
|leaf4
|SRLinux
|65014
|10.255.255.8/32
|10.254.254.26/31
|10.254.254.28/31
|trunk vlan 100,200
|irb0.200 : 192.168.200.254/24
|-
|spine1
|CVX
|65001
|10.255.255.1/32
|10.254.254.9/31
|10.254.254.13/31
|10.254.254.0/31
|<none>
|-
|spine2
|CVX
|65001
|10.255.255.2/32
|10.254.254.11/31
|10.254.254.15/31
|10.254.254.16/31
|<none>
|-
|spine3
|SRLinux
|65002
|10.255.255.3/32
|10.254.254.23/31
|10.254.254.27/31
|10.254.254.18/31
|<none>
|-
|spine4
|SRLinux
|65002
|10.255.255.4/32
|10.254.254.25/31
|10.254.254.29/31
|10.254.254.20/31
|<none>
|-
|superspine1
|CVX
|65003
|10.255.255.9/32
|10.254.254.1/31
|10.254.254.19/31
|<none>
|<none>
|-
|superspine2
|SRLinux
|65003
|10.255.255.10/32
|10.254.254.17/31
|10.254.254.21/31
|<none>
|<none>
|}

=== VXLAN/BGP infos ===
Note : in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>
{| class="wikitable"
|+
!service
!vlan id
!route-target
!VNI
!Commentaires
|-
|L2 cross-DC
|100
|target:65000:100
|100
|the 4 clients are in the same broadcast domain / subnet
|-
|L3VPN cros-DC
|200
|target:65000:200
target:65000:10200
|200 (for L2)
10200 (for routing)
|the 4 clients are separated in 2 different subnets. Traffic is routed within dedicated VNI
To be noted that in the current setup VMTO is not implemented, so cross DC traffic is not optimized.
|}

== Lab diagram ==
[[File:Lab-srl-diagram.png|ohne|mini|600x600px]]

== yml lab file ==
<syntaxhighlight lang="yaml">
# topology documentation: http://containerlab.srlinux.dev/lab-examples/min-5clos/
name: lab-srl-cum

topology:
kinds:
srl:
image: srlinux:21.6.1-235
linux:
image: ghcr.io/hellt/network-multitool
cvx:
image: networkop/cx:4.3.0
runtime: docker
nodes:
leaf1:
kind: cvx
leaf2:
kind: srl
type: ixrd2
leaf3:
kind: cvx
leaf4:
kind: srl
type: ixrd2

spine1:
kind: cvx
spine2:
kind: cvx
spine3:
kind: srl
type: ixr6
spine4:
kind: srl
type: ixr6

superspine1:
kind: cvx
superspine2:
kind: srl
type: ixr6

client1:
kind: linux
client2:
kind: linux
client3:
kind: linux
client4:
kind: linux

links:
# leaf to spine links POD1
- endpoints: ["leaf1:swp1", "spine1:swp1"]
- endpoints: ["leaf1:swp2", "spine2:swp1"]
- endpoints: ["leaf2:e1-1", "spine1:swp2"]
- endpoints: ["leaf2:e1-2", "spine2:swp2"]

# spine to superspine links POD1
- endpoints: ["spine1:swp3", "superspine1:swp1"]
- endpoints: ["spine2:swp3", "superspine2:e1-1"]

# leaf to spine links POD2
- endpoints: ["leaf3:swp1", "spine3:e1-1"]
- endpoints: ["leaf3:swp2", "spine4:e1-1"]
- endpoints: ["leaf4:e1-1", "spine3:e1-2"]
- endpoints: ["leaf4:e1-2", "spine4:e1-2"]

# spine to superspine links POD2
- endpoints: ["spine3:e1-3", "superspine1:swp2"]
- endpoints: ["spine4:e1-3", "superspine2:e1-2"]

# client connection links
- endpoints: ["client1:eth1", "leaf1:swp3"]
- endpoints: ["client2:eth1", "leaf2:e1-3"]
- endpoints: ["client3:eth1", "leaf3:swp3"]
- endpoints: ["client4:eth1", "leaf4:e1-3"]

</syntaxhighlight>

Once the nodes are defined, we have to define the needed connections between the nodes.

Each endpoint is defined as '<nowiki/>'''<nowiki/>'<node_name>:<interface_name><nowiki>''</nowiki>'''

When all is ok, you can launch in suduoer the topology.

For example :<syntaxhighlight lang="bash">
sudo containerlab deploy --topo topo.clab.yml --reconfigure

Note: The option "reconfigure" is used there in order to not have to destroy instances when updating topology/relaunching.
</syntaxhighlight>Another info regarding the yml file, it's possible to quickly generate files with the '''"containerlab gen -n 3tier --nodes 4,2,1"''' command for example. https://containerlab.srlinux.dev/cmd/generate/

To be noted that the "tier-x" notion in containerlab is inversed regarding the rfc 7938 (tier1 is superspine level into the rfc, tier1 is leaf level in nokia tool)

At the end of the topology launch, you can retreive the details :<syntaxhighlight lang="yaml">
INFO[0011] Writing /etc/hosts file
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| # | Name | Container ID | Image | Kind | Group | State | IPv4 Address | IPv6 Address |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| 1 | clab-lab-srl-cum-client1 | 25e2bbf6b13f | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
| 2 | clab-lab-srl-cum-client2 | 17d82d5a6e21 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.4/24 | 2001:172:20:20::4/64 |
| 3 | clab-lab-srl-cum-client3 | f8d78527c1b1 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.6/24 | 2001:172:20:20::6/64 |
| 4 | clab-lab-srl-cum-client4 | 913a0172d0c5 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 5 | clab-lab-srl-cum-leaf1 | 9ffc334af2b1 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.9/24 | 2001:172:20:20::9/64 |
| 6 | clab-lab-srl-cum-leaf2 | 7d3eb855d749 | srlinux:21.6.1-235 | srl | | running | 172.20.20.14/24 | 2001:172:20:20::e/64 |
| 7 | clab-lab-srl-cum-leaf3 | e0d36ffc606a | networkop/cx:4.3.0 | cvx | | running | 172.20.20.7/24 | 2001:172:20:20::7/64 |
| 8 | clab-lab-srl-cum-leaf4 | e6753b1d099e | srlinux:21.6.1-235 | srl | | running | 172.20.20.12/24 | 2001:172:20:20::c/64 |
| 9 | clab-lab-srl-cum-spine1 | a7e63993d052 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.5/24 | 2001:172:20:20::5/64 |
| 10 | clab-lab-srl-cum-spine2 | 3124250d5b75 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
| 11 | clab-lab-srl-cum-spine3 | 5e300ca3f66f | srlinux:21.6.1-235 | srl | | running | 172.20.20.11/24 | 2001:172:20:20::b/64 |
| 12 | clab-lab-srl-cum-spine4 | 0a59e60a596b | srlinux:21.6.1-235 | srl | | running | 172.20.20.13/24 | 2001:172:20:20::d/64 |
| 13 | clab-lab-srl-cum-superspine1 | 3d9935413fc2 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.8/24 | 2001:172:20:20::8/64 |
| 14 | clab-lab-srl-cum-superspine2 | 5bc58d423e2c | srlinux:21.6.1-235 | srl | | running | 172.20.20.15/24 | 2001:172:20:20::f/64 |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
</syntaxhighlight>The containerlab program creates the updates into the host file + generates an ansible inventory file.

There are different ways to connect onto the docker instances, within docker exec commands or SSH directly the IP address related above.

For example :<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-spine1 bash
</syntaxhighlight>-->>> allows to launch a bash on the cumulus device (then you run the different net commands)<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-leaf2 sr_cli
</syntaxhighlight>-->>> allows to launch the cli on a SRLinux device

To end this setup preparation, the containerlab tool integrates an option to launch a webserver showing the lab topology in graph + node details

By default, it's listening on port 50080 of the host

Then, you can access to the following web page showing the topology + lab infos

[[File:Lab graph tool.jpg|ohne|mini|900x900px]]

Finally, if you want to stop a lab, please run '''<nowiki>''</nowiki>containerlab destroy --topo <topology>.yml<nowiki>''</nowiki>'''

At this step, nodes are running, connections are done, but configurations are empty.

'''So, it's time to configure them :)'''

Configuration files are attached to this blog. Once validated, they are exportable and can be called by the tool when launching the lab. Below how to do it :

Firstable, once you ve finished your "hand made config", you can run the following command to save it :<syntaxhighlight lang="bash">
containerlab save -t quickstart.clab.yml
</syntaxhighlight>'''To be noted it's only working for Nokia nodes in this lab'''. CVX configurations will have to be called by another method (or pushed manually)

It will create one folder per node into the lab folder which gives the following tree :
[[File:Topology-srl.jpg|ohne|mini]]

'''Note''': For Cumulus, you can apparently retrieve configurations files within adding the following parameters into lab config file. '''To be tested.''' some lab samples here : https://github.com/hellt/clabs/blob/main/labs/cvx04/symm-mh.clab.yml

Now we have the lab running, below the different commands to check that all is working as expected : <syntaxhighlight lang="yaml">
-------------------------------------------
Check L2 tables ///
-------------------------------------------

Nokia :

A:leaf2# show network-instance mac-vrf-100 bridge-table mac-table all
----------------------------------------------------------------------------------------------------------------------------------------------------------
Mac-table of network instance mac-vrf-100
----------------------------------------------------------------------------------------------------------------------------------------------------------
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
| Address | Destination | Dest Index | Type | Active | Aging | Last Update |
+====================+=========================================+============+=============+=========+========+=========================================+
| AA:C1:AB:0C:60:D6 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T13:06:17.000Z |
| | vtep:10.255.255.8 vni:100 | 67 | | | | |
| AA:C1:AB:23:31:04 | ethernet-1/3.100 | 2 | learnt | true | 300 | 2021-10-08T13:06:10.000Z |
| AA:C1:AB:4B:FF:71 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:55:28.000Z |
| | vtep:10.255.255.5 vni:100 | 60 | | | | |
| AA:C1:AB:B5:79:50 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:43:37.000Z |
| | vtep:10.255.255.7 vni:100 | 64 | | | | |
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 0 Total 0 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 1 Total 1 Active
Total Evpn Macs : 3 Total 3 Active
Total Evpn static Macs : 0 Total 0 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 4 Total 4 Active
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : type learnt matches the local learnt mac, evpn are learnt within VXLAN

Cumulus :

root@leaf1-cum-dc1:/# net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
100 bridge swp3 aa:c1:ab:4b:ff:71 00:00:02
100 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:17:21
100 bridge vni100 22:2d:67:f1:bb:26 permanent 02:17:21
100 bridge vni100 aa:c1:ab:0c:60:d6 extern_learn <1 sec
100 bridge vni100 aa:c1:ab:23:31:04 extern_learn 00:00:07
100 bridge vni100 aa:c1:ab:b5:79:50 extern_learn 00:13:28

root@leaf1-cum-dc1:/# net show bridge macs vlan 200

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
200 bridge swp3 aa:c1:ab:4b:ff:71 00:08:46
200 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:16:36
200 bridge vni200 00:00:5e:00:01:01 static sticky 00:14:26
200 bridge vni200 1e:0d:aa:46:53:b1 permanent 02:16:36
200 bridge vni200 02:77:6b:ff:00:41 static sticky 01:00:52
200 bridge vni200 02:f8:b8:ff:00:41 static sticky 00:14:26
200 bridge vni200 aa:c1:ab:b5:79:50 extern_learn 00:11:51
200 bridge vni10200 02:77:6b:ff:00:00 extern_learn 00:53:02
200 bridge vni10200 02:f8:b8:ff:00:00 extern_learn 00:14:26
200 bridge vni10200 86:5e:aa:ca:50:be permanent 02:16:36

Note : the macs with permanent flag are local interfaces, physical ou logic (vtep)
Note2 : extern_learn mac are the one learned via the VXLAN infra (and mapped to the vnixxx interface)

-------------------------------------------
Check arp tables ///
-------------------------------------------

Nokia :

:leaf2# show arpnd arp-entries
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| ethernet-1/1 | 0 | 10.254.254.13 | dynamic | AA:C1:AB:59:65:EA | 3 hours from now |
| ethernet-1/2 | 0 | 10.254.254.15 | dynamic | AA:C1:AB:CB:F8:8A | 3 hours from now |
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
| mgmt0 | 0 | 172.20.20.1 | dynamic | 02:42:63:9E:74:03 | 2 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 5 (0 static, 5 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : on Nokia you have to sort the entries basing on subinterface and the retrieve the L3VPN attached

A:leaf2# show arpnd arp-entries subinterface 200
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 2 (0 static, 2 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Cumulus :

Here we retrieve all arp infos linked to evpn

root@leaf1-cum-dc1:/# net show evpn arp-cache vni all
VNI 100 #ARP (IPv4 and IPv6, local and remote) 2

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.1.3 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0
192.168.1.1 local active aa:c1:ab:4b:ff:71 0/0

VNI 200 #ARP (IPv4 and IPv6, local and remote) 4

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.2.1 local active aa:c1:ab:4b:ff:71 0/0
192.168.200.254 remote active 00:00:5e:00:01:01 10.255.255.8 0/0
192.168.2.254 remote active 00:00:5e:00:01:01 10.255.255.6 0/0
192.168.200.1 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0

VNI 400 #ARP (IPv4 and IPv6, local and remote) 0

-------------------------------------------
Check bgp sessions' status - underlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep ipv4
| default | 10.254.254.19 | udl-superspine | S | 65003 | established | 0d:0h:22m:44s | ipv4-unicast | [13/4/4] |
| default | 10.254.254.22 | udl-leaf | S | 65013 | established | 0d:0h:22m:52s | ipv4-unicast | [13/1/4] |

Cumulus :

root@leaf3:/# net show bgp ipv4 unicast summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 13
RIB entries 25, using 4800 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.254.254.23 4 65002 466 403 0 0 0 00:19:34 4 13
10.254.254.25 4 65002 434 355 0 0 0 00:17:10 8 13

-------------------------------------------
Check bgp sessions' status - overlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep evpn
| default | 10.255.255.7 | ovl-leaf | S | 65013 | established | 0d:0h:24m:14s| evpn | [15/0/11] |
| default | 10.255.255.9 | ovl-superspine | S | 65003 | established | 0d:0h:24m:7s | evpn | [11/0/4] |

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.255.255.3 4 65002 5618 4742 0 0 0 03:53:35 19 27
10.255.255.4 4 65002 5565 4694 0 0 0 03:51:11 19 27

-------------------------------------------
Check EVPN L2 tables///
-------------------------------------------

Nokia :

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn route type 2
BGP table version is 17, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:100
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
Route Distinguisher: 10.255.255.5:200
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
Route Distinguisher: 10.255.255.6:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
Route Distinguisher: 10.255.255.7:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100
Route Distinguisher: 10.255.255.7:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.200.1]
10.255.255.7 32768 i
ET:8 RT:65000:200
Route Distinguisher: 10.255.255.8:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC

and if we focus on our vni 100

root@leaf3-cum-dc1:/# net show bgp evpn route vni 100 type 2
BGP table version is 48, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100

The additional route type "MAC+IP" is optional and is useful/mandatory when you do anycast irb.

-------------------------------------------
Check tables EVPN L3 ///
-------------------------------------------

Nokia :

--{ + running }--[ ]--
A:leaf2# show network-instance ip-vrf-200 route-table all
-----------------------------------------------------------------------------------------------------------------------------------------------------------
IPv4 Unicast route table of network instance ip-vrf-200
-----------------------------------------------------------------------------------------------------------------------------------------------------------
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
| Prefix | ID | Route Type | Route Owner | Best/Fib- | Metric | Pref | Next-hop | Next-hop |
| | | | | status(slot) | | | (Type) | Interface |
+=======================+=======+============+======================+======================+==========+=========+==============+==============+
| 192.168.2.0/24 | 9 | local | net_inst_mgr | True/success | 0 | 0 | 192.168.2.25 | irb0.200 |
| | | | | | | | 4 (direct) | |
| 192.168.2.254/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (extract) | |
| 192.168.2.255/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (broadcast) | |
| 192.168.200.0/24 | 0 | bgp-evpn | bgp_evpn_mgr | True/success | 0 | 170 | 10.255.255.8 | None |
| | | | | | | | (indirect/vx | |
| | | | | | | | lan) | |
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
-----------------------------------------------------------------------------------------------------------------------------------------------------------
4 IPv4 routes total
4 IPv4 prefixes with active routes
0 IPv4 prefixes with active ECMP routes
-----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--
A:leaf2#

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf1-cum-dc1:/# net show route vrf ip-vrf-200
show ip route vrf ip-vrf-200
=============================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF ip-vrf-200:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 04:48:42
B 192.168.2.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 04:27:26
C>* 192.168.2.0/24 is directly connected, vlan200, 04:48:42
B>* 192.168.200.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:48:49

root@leaf1-cum-dc1:/# net show bgp evpn route type 5
BGP table version is 3, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.6:10200
* [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
*> [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
Route Distinguisher: 10.255.255.8:10200
*> [5]:[0]:[24]:[192.168.200.0]
10.255.255.8 0 65001 65003 65002 65014 i
RT:65000:10200 ET:8 Rmac:02:f8:b8:ff:00:00

-------------------------------------------
Check arp hosts ///
-------------------------------------------

From client #1 /

bash-5.0# arp
Address HWtype HWaddress Flags Mask Iface
192.168.1.3 ether aa:c1:ab:b5:79:50 C eth1.100
192.168.1.4 ether aa:c1:ab:0c:60:d6 C eth1.100
192.168.2.2 ether aa:c1:ab:23:31:04 C eth1.200
192.168.2.254 ether aa:c1:ab:4f:f2:e7 C eth1.200
192.168.1.2 ether aa:c1:ab:23:31:04 C eth1.100
containerlab.local ether 02:42:63:9e:74:03 C eth0

-------------------------------------------
Ping ///
-------------------------------------------

From client #1 /

L2 service ping (vlan 100) ----------------------

bash-5.0# ping -c 1 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.812 ms

--- 192.168.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.812/0.812/0.812/0.000 MsgRcvd
bash-5.0# ping -c 1 192.168.1.3
PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_seq=1 ttl=64 time=1.40 ms

--- 192.168.1.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.403/1.403/1.403/0.000 ms
bash-5.0# ping -c 1 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=1.62 ms

--- 192.168.1.4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.619/1.619/1.619/0.000 ms

L3 service ping (vlan 200)

bash-5.0# ping -c 1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.053 ms

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms
bash-5.0# ^C
bash-5.0# ping -c 1 192.168.2.2
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.611 ms

--- 192.168.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.611/0.611/0.611/0.000 ms

-------------------------------------------
Traceroute ///
-------------------------------------------

L3 service traceroute (vlan 200)

bash-5.0# traceroute 192.168.200.1
traceroute to 192.168.200.1 (192.168.200.1), 30 hops max, 46 byte packets
1 192.168.2.254 (192.168.2.254) 0.012 ms 0.010 ms 0.007 ms
2 192.168.200.254 (192.168.200.254) 2.751 ms * 2.549 ms
3 192.168.200.1 (192.168.200.1) 2.084 ms 1.805 ms 1.761 ms

</syntaxhighlight>To come back on the setup, we wanted to implement solutions inspired by the <nowiki>RFC 7938</nowiki> rfc7938 (see section 5.2.1)

* eBGP will be used for Underlay and Overlay
* eBGP single-hop sessions are established over direct point-to-point links interconnecting the network nodes, no multi-hop or loopback sessions are used, even in the case of multiple links between the same pair of nodes.
* Private Use ASNs from the range 64512-65534 are used to avoid ASN conflicts.
* A single ASN is allocated to all of the Clos topology's Tier 1 devices.
* A unique ASN is allocated to each set of Tier 2 devices in the same cluster.
* A unique ASN is allocated to every Tier 3 device (e.g., ToR) in this topology.

In add, depending of network nodes, it will be interesting to :

* Add fast-failover option for underlay + add link-debounce on links to limit possible impacts in case of link flaps. BFD could be considered in some cases
* Have the best ECMP hashing algorythm on leaf nodes. On tier-1/tier-2 levels hashing will be based on underlay part (trafic inter-vtep), so L3-L4 decision, but some models can retreive the vni-id if I well remember.
* Implement the "multipath relax" since we could have multiple AS paths for a same destination.
* Implement gshut community on underlay (on tier-1/tier-2) or at least policies to prepend underlay routes to avoid some nodes in case of maintenance.

== Thoughts/Points to deepend ==
On leaf nodes, LACP over multiple nodes is feasible within ESI, but impact on incidents/maintenance can vary depending on the vendors.

'''Tests need to be done to check that.'''

Some sysadmin would like to KISS and have client servers connected in active/passive bonding, with eventually single-leaf ports aggregation.

'''Note :''' NH ESI in FIB management on ingress nodes can have some limitations (numbers of installable next-hop, load-balancing, etc. oftenly linked to the node chipset)

Another important engineering point to define at the beginning is : '''Do we authorize multisite L2 propagation or not ?'''

From my point-of-view, the smaller are the L2 segments the better it is. I would suggest Local L2 segments wit anycast gateways provided by Leaf switches, with L3 routing between DCs within EVPN type5 routes sharing only. Nevertheless, some business requirements for endcustomers can keep extended L2 alive.

We have to be careful when implementing Consistent hashing. ex: <nowiki>https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-44/Layer-3/Routing/Equal-Cost-Multipath-Load-Sharing-Hardware-ECMP/</nowiki>

If possible activate it per VRF, not globally, and restrict on dedicated subnets << depends of the vendor, it can be dangerous regarding memory management

Think at configuring the bgp option to not advertise routes to peers before they are installed in FIB.

We can implement leaf roles in some cases, to be able to do "light traffic engineering " on DCI links in some setup.

These roles will be managed within bgp communities added to VTEP /32. Look at the diagram below. Depending of the colour, the eBGP sessions between SupS will prepend or not the VTEP IPs announced to the remote DC

[[File:Fabric-roles.jpg|ohne|mini|1200x1200px]]

<blockquote>'''To be continued'''</blockquote>

== Lab archives ==
<blockquote>Reminder : following config are for lab, so please don't pay attention on typos/lack of security params/unused config. Thanks \o/</blockquote>

=== Credentials ===
<syntaxhighlight lang="bash">
SRLinux : admin/<no_password>

CVX : cumulus/cumulus
</syntaxhighlight>

=== Lab archives ===
layer 2 (vlan 100) + layer 3 (vlan 200) lab, 10/12/2021
[[File:20211008 lab-nokia-cvx-containerlab.zip|ohne|mini]]
<u>If you want to exchange about that lab, those topics, please don't hesitate to contact us</u>

= SONiC Lab =

<span><to be noted that those informations are included into the readme.md file into the archive></span>

== Lab diagram ==
[[File:SONIC_LAB.jpg]]

Lab archive [https://wiki.nesevo.com/images/0/0a/Sonic-lab-20201030.zip Sonic-lab-20201030.zip]

This lab is a forked lab from many found on @, starting from Microsoft one : https://github.com/Azure/SONiC/wiki/SONiC-P4-Software-Switch

The target of this lab is to simulate a simple IP/Fabric in SPINE/LEAF topology. Each Leaf provide local L2 connectivity + L3 gateways to hosts. Flows between hosts from different leaf are L3 routed, in a "node path" LEAF-SPINE-LEAF.

Nothing complicated, but it allows to put a first foot into SONiC world :)

For the moment, as far as we know, MC-LAG + EVPN/VXLAN features are only available on entreprise SONiC versions. Tests to come once physic switches in enterprise version will be available in our lab.

The diagram of the architecture is into the zip file (SONIC_LAB.jpg)

To run the lab you need :

- to have installed docker by using ./install_docker_ovs.sh - to run the ./load_image.sh in order to get the Sonic image

Note : don't change file / folders names of network devices (leafX, spineX)

Once you're ready, please run the lab by launching "./start.sh"

The script :<div></div>
* launches the containers without Docket network features
* create network namespaces
* build linux bridges to interconnect the containers
* configure host IPs
* launch startup sonic scripts into sonic devices
* once devices started, checks + ping tests are run. If all BGP sessions are ok on spine + all ping are fine, the setup works well.
The "start" script has several comments, perhaps without the right vocabulary, so please be tolerant :)

If you want to stop the lab, please launch "./stop.sh"

Below some useful commands to run for sonic containers :

=== <span>run ping</span> ===
sudo docker exec -it <container> ping <ip>

This command triggers ping into the targetted container :
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it leaf1 ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.052 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.034 ms
^C--- 192.168.1.1 ping statistics ---
</syntaxhighlight>

=== <span>check bgp sessions</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip bgp sum"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print the bgp status.
Below, we run it on one spine, you should have 4 sessiosn UP (for the 4 leaves)
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip bgp sum"
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 6409 6411 0 0 0 01:46:46 3
10.1.0.2 4 65102 6406 6410 0 0 0 01:46:44 3
10.1.0.4 4 65103 6407 6410 0 0 0 01:46:44 4
10.1.0.6 4 65104 6406 6412 0 0 0 01:46:46 1
Total number of neighbors 4
</syntaxhighlight>

=== <span>check ip route</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip route"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print its routing table?

<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,
> - selected route, * - FIB route
C>* 10.1.0.0/31 is directly connected, Vlan11
C>* 10.1.0.2/31 is directly connected, Vlan21
C>* 10.1.0.4/31 is directly connected, Vlan31
C>* 10.1.0.6/31 is directly connected, Vlan41
C>* 127.0.0.0/8 is directly connected, lo
B>* 192.168.1.0/24 [20/0] via 10.1.0.0, Vlan11, 01:47:23
B>* 192.168.2.0/24 [20/0] via 10.1.0.2, Vlan21, 01:47:21
B>* 192.168.3.0/24 [20/0] via 10.1.0.4, Vlan31, 01:47:21
B>* 192.168.4.0/24 [20/0] via 10.1.0.6, Vlan41, 01:47:23
</syntaxhighlight>
For more commands check SONiC docs : https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md

To finish, below the output of the start.sh when all goes well<div></div>
plancastre@dockerhost:~/00.Projects/p4-version-pla$ ./start.sh
---------------------------------------------------
Creating the containers: starting
---------------------------------------------------
0b4bbb3c6cfc9895466ea13a9e940d1a176d66f45deee9752e9b4732d735bb03
5ca9486baab3e8a5259062167132a944eee717875d3cc1ef29fe490cba3184c1
46c9c09dba27a32715b9fbe0bc434a9284b9d59317d278df693f425cb669f037
898e6afbcb2977115acbba2a69d069796cfd70fb90228b6c18e09ba4584365f7
eb9b3ef6d14784c9ffe4d9d5df3e921a24559889cb8770d226263aad6761182a
ae5f3b2c628da687eaabb331dc2cd989e71bcb386854e7a0be6ec587df99d49a
122197bffd7566a01ce3b6565fdfb6eb893aab7d5236f14e5a9bbbe95037ced2
a604520a080e2a2184598ad1d915c0c182d16e5185033373e95a4772d8978a67
0c49e18d78b2aa7c8ef8e937b60285aa56ec0bf4a8530226285270a085e11d5c
eb66336b8c6bd8333f1f4f685369e6cac973b817c35bb7126ac4d7ba72ef71bf
---------------------------------------------------
Retreiving PID from docker containers
---------------------------------------------------
---------------------------------------------------
Creating network namespaces for containers
---------------------------------------------------
List of created namespaces
ns_host4
ns_host3
ns_host2
ns_host1
ns_spine2
ns_spine1
ns_leaf4
ns_leaf3
ns_leaf2
ns_leaf1
---------------------------------------------------
Creating network connectivity between containers
---------------------------------------------------
---------------------------------------------------
Now assigning IPs to hosts (host1 to host4)
---------------------------------------------------
---------------------------------------------------
Configuration of containers is now finished
---------------------------------------------------
---------------------------------------------------
Now booting switches, please wait 2 minutes for switches to load
---------------------------------------------------
---------------------------------------------------
2 minutes period is finished
---------------------------------------------------
---------------------------------------------------
Now fixing the iptables
---------------------------------------------------
---------------------------------------------------
Backend actions are now finished
---------------------------------------------------
---------------------------------------------------
Now we'll check in 15 sec that BGP peers are all Up
---------------------------------------------------
---------------------------------------------------
SPINE1 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 94 97 0 0 0 00:01:28 4
10.1.0.2 4 65102 95 97 0 0 0 00:01:28 5
10.1.0.4 4 65103 95 97 0 0 0 00:01:28 5
10.1.0.6 4 65104 95 97 0 0 0 00:01:28 5
Total number of neighbors 4
---------------------------------------------------
SPINE2 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.2.0.1, local AS number 65002
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.2.0.0 4 65101 93 96 0 0 0 00:01:29 2
10.2.0.2 4 65102 94 100 0 0 0 00:01:32 1
10.2.0.4 4 65103 94 100 0 0 0 00:01:32 1
10.2.0.6 4 65104 94 100 0 0 0 00:01:32 1
Total number of neighbors 4
---------------------------------------------------
Now if all is ok, we'll do some ping tests
---------------------------------------------------
---------------------------------------------------
host1 ping its gateway
---------------------------------------------------
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=17.6 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=9.14 ms
--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 9.145/13.390/17.636/4.247 ms
---------------------------------------------------
host2 ping its gateway
---------------------------------------------------
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=23.6 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=6.21 ms
--- 192.168.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 6.218/14.950/23.683/8.733 ms
---------------------------------------------------
host3 ping its gateway
---------------------------------------------------
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=18.8 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=12.7 ms
--- 192.168.3.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.716/15.764/18.812/3.048 ms
---------------------------------------------------
host4 ping its gateway
---------------------------------------------------
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.
64 bytes from 192.168.4.1: icmp_seq=1 ttl=64 time=17.7 ms
64 bytes from 192.168.4.1: icmp_seq=2 ttl=64 time=4.46 ms
--- 192.168.4.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.464/11.107/17.750/6.643 ms
---------------------------------------------------
host1 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=16.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.049/17.253/19.532/1.615 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=15.5 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=14.3 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.312/15.592/16.939/1.083 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=20.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=16.8 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.048/17.871/20.739/2.058 ms
---------------------------------------------------
host2 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=16.6 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.222/16.583/16.924/0.286 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.6 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=18.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=26.8 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.620/20.549/26.887/4.524 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=33.8 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=27.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=20.2 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 20.216/27.259/33.818/5.566 ms
---------------------------------------------------
host3 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=14.5 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=15.5 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.535/16.549/19.541/2.162 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.4 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=14.6 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=18.2 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.643/16.458/18.278/1.491 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=15.3 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=14.1 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=13.1 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 13.171/14.234/15.334/0.883 ms
---------------------------------------------------
host4 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=21.9 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=21.3 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=14.6 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 14.619/19.284/21.932/3.310 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=14.8 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=18.4 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=14.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.148/15.786/18.409/1.879 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=15.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=18.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 15.117/16.788/18.962/1.612 ms
---------------------------------------------------
If bot spine have all BGP sessions and all ping succeeded, all works well
Congrats ! please check readme file to find some details
---------------------------------------------------

Nesevo Virtual Labs

2021-10-29T08:12:49Z

PierreLancastre: /* Lab diagram */

[[File:nesevo-labs-v2.png]]

= Nokia/Cumulus Multi-Vendor Lab =

The targets of this lab are to :

* mix Nokia SR-Linux and Cumulus VXLAN nodes, in spine or leaf roles
* make this environment work for different kinds of setup : cross-DC L2, differentiated L3 gateways per leaf per VRF, anycast gateway cross or intra DC, etc.
* spot compatibility issues if existing

To reach those goals, we'll use the Nokia lab tool "containerlab" https://containerlab.srlinux.dev/, labs from https://clabs.netdevops.me/ helped a lot. Many thanks to the contributors.

We'll need a Linux host with Docker installed on it + kvm virtualization enhancements activated (important if your host is a VM). If virtualization optimization is not ok, you won't be able to launch the most of devices.

Installation details are referenced there : https://containerlab.srlinux.dev/install/

Once installed, you'll need to register the different NOS images you'll use. Cumulus image is present on a public registrery, but Nokia/Arista/Juniper are not, you'll need a vendor support account to retreive them. Once downloaded (.tgz file typically) you can "docker load" the image :

For private images :<syntaxhighlight lang="bash">
sudo docker load -i 21.6.1-235.tar.xz /* (Nokia) */
sudo docker load -i cEOS64-lab-4.26.1F.tar /* (Arista) */
</syntaxhighlight>For public images :<syntaxhighlight lang="bash">
sudo docker pull networkop/cx:4.3.0 /* (Cumulus) */
</syntaxhighlight>

To verify your installed images, you can run "sudo docker images"<syntaxhighlight lang="bash">
plancastre@containerlab:~/00.LAB/senss$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
networkop/cx 4.3.0 a40541ed15ba 2 weeks ago 733MB
srlinux 21.6.1-235 e29aed30e83f 3 weeks ago 1.14GB
networkop/ignite dev dcc5eff29355 4 weeks ago 37.9MB
networkop/kernel 4.19 e794d8875f83 3 months ago 88MB
frrouting/frr v7.5.1 c3e13a4c5918 3 months ago 123MB

</syntaxhighlight>Once you've got all your images, to run the lab, you'll need a .yml file defining the topology to run.

In our lab, we've started with the 5-clos lab file from Nokia site, which we've forked.

We can see there is a branch "kinds" where we can define templates of nodes, in order to not rewrite whole parameters when creating nodes. For the images, it's important to take the repository name + tag.

'''Note:''' If the tag is "latest" it is not needed.

All supported "kinds" are referenced there : https://containerlab.srlinux.dev/manual/kinds/kinds/

'''Note:''' yml is indentation sensitive, so please respect the "double space" used to define hierarchy. Below the lab diagram + the yml file associated

== Infos on nodes ==

=== L3 infos ===
{| class="wikitable"
|+
!Device
!OS
!ASN
!VTEP IP
!swp1 or eth1/1
!swp2 or eth1/2
!swp3 or eth1/3
!SVI
|-
|client1
|Linux
|<none>
|<none>
|vlan100: 192.168.1.1/24
vlan200: 192.168.2.1/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client2
|Linux
|<none>
|<none>
|vlan100: 192.168.1.2/24
vlan200: 192.168.2.2/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client3
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.1/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|client4
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.2/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|leaf1
|CVX
|65011
|10.255.255.5/32
|10.254.254.8/31
|10.254.254.10/31
|trunk vlan 100,200
|vlan200 : 192.168.2.254/24
|-
|leaf2
|SRLinux
|65012
|10.255.255.6/32
|10.254.254.12/31
|10.254.254.14/31
|trunk vlan 100,200
|irb0.200 : 192.168.2.254/24
|-
|leaf3
|CVX
|65013
|10.255.255.7/32
|10.254.254.22/31
|10.254.254.24/31
|trunk vlan 100,200
|vlan200 : 192.168.200.254/24
|-
|leaf4
|SRLinux
|65014
|10.255.255.8/32
|10.254.254.26/31
|10.254.254.28/31
|trunk vlan 100,200
|irb0.200 : 192.168.200.254/24
|-
|spine1
|CVX
|65001
|10.255.255.1/32
|10.254.254.9/31
|10.254.254.13/31
|10.254.254.0/31
|<none>
|-
|spine2
|CVX
|65001
|10.255.255.2/32
|10.254.254.11/31
|10.254.254.15/31
|10.254.254.16/31
|<none>
|-
|spine3
|SRLinux
|65002
|10.255.255.3/32
|10.254.254.23/31
|10.254.254.27/31
|10.254.254.18/31
|<none>
|-
|spine4
|SRLinux
|65002
|10.255.255.4/32
|10.254.254.25/31
|10.254.254.29/31
|10.254.254.20/31
|<none>
|-
|superspine1
|CVX
|65003
|10.255.255.9/32
|10.254.254.1/31
|10.254.254.19/31
|<none>
|<none>
|-
|superspine2
|SRLinux
|65003
|10.255.255.10/32
|10.254.254.17/31
|10.254.254.21/31
|<none>
|<none>
|}

=== VXLAN/BGP infos ===
Note : in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>
{| class="wikitable"
|+
!service
!vlan id
!route-target
!VNI
!Commentaires
|-
|L2 cross-DC
|100
|target:65000:100
|100
|the 4 clients are in the same broadcast domain / subnet
|-
|L3VPN cros-DC
|200
|target:65000:200
target:65000:10200
|200 (for L2)
10200 (for routing)
|the 4 clients are separated in 2 different subnets. Traffic is routed within dedicated VNI
To be noted that in the current setup VMTO is not implemented, so cross DC traffic is not optimized.
|}

== Lab diagram ==
[[File:Lab-srl-diagram.png|ohne|mini|600x600px]]

== yml lab file ==
<syntaxhighlight lang="yaml">
# topology documentation: http://containerlab.srlinux.dev/lab-examples/min-5clos/
name: lab-srl-cum

topology:
kinds:
srl:
image: srlinux:21.6.1-235
linux:
image: ghcr.io/hellt/network-multitool
cvx:
image: networkop/cx:4.3.0
runtime: docker
nodes:
leaf1:
kind: cvx
leaf2:
kind: srl
type: ixrd2
leaf3:
kind: cvx
leaf4:
kind: srl
type: ixrd2

spine1:
kind: cvx
spine2:
kind: cvx
spine3:
kind: srl
type: ixr6
spine4:
kind: srl
type: ixr6

superspine1:
kind: cvx
superspine2:
kind: srl
type: ixr6

client1:
kind: linux
client2:
kind: linux
client3:
kind: linux
client4:
kind: linux

links:
# leaf to spine links POD1
- endpoints: ["leaf1:swp1", "spine1:swp1"]
- endpoints: ["leaf1:swp2", "spine2:swp1"]
- endpoints: ["leaf2:e1-1", "spine1:swp2"]
- endpoints: ["leaf2:e1-2", "spine2:swp2"]

# spine to superspine links POD1
- endpoints: ["spine1:swp3", "superspine1:swp1"]
- endpoints: ["spine2:swp3", "superspine2:e1-1"]

# leaf to spine links POD2
- endpoints: ["leaf3:swp1", "spine3:e1-1"]
- endpoints: ["leaf3:swp2", "spine4:e1-1"]
- endpoints: ["leaf4:e1-1", "spine3:e1-2"]
- endpoints: ["leaf4:e1-2", "spine4:e1-2"]

# spine to superspine links POD2
- endpoints: ["spine3:e1-3", "superspine1:swp2"]
- endpoints: ["spine4:e1-3", "superspine2:e1-2"]

# client connection links
- endpoints: ["client1:eth1", "leaf1:swp3"]
- endpoints: ["client2:eth1", "leaf2:e1-3"]
- endpoints: ["client3:eth1", "leaf3:swp3"]
- endpoints: ["client4:eth1", "leaf4:e1-3"]

</syntaxhighlight>

Once the nodes are defined, we have to define the needed connections between the nodes.

Each endpoint is defined as '<nowiki/>'''<nowiki/>'<node_name>:<interface_name><nowiki>''</nowiki>'''

When all is ok, you can launch in suduoer the topology.

For example :<syntaxhighlight lang="bash">
sudo containerlab deploy --topo topo.clab.yml --reconfigure

Note: The option "reconfigure" is used there in order to not have to destroy instances when updating topology/relaunching.
</syntaxhighlight>Another info regarding the yml file, it's possible to quickly generate files with the '''"containerlab gen -n 3tier --nodes 4,2,1"''' command for example. https://containerlab.srlinux.dev/cmd/generate/

To be noted that the "tier-x" notion in containerlab is inversed regarding the rfc 7938 (tier1 is superspine level into the rfc, tier1 is leaf level in nokia tool)

At the end of the topology launch, you can retreive the details :<syntaxhighlight lang="yaml">
INFO[0011] Writing /etc/hosts file
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| # | Name | Container ID | Image | Kind | Group | State | IPv4 Address | IPv6 Address |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| 1 | clab-lab-srl-cum-client1 | 25e2bbf6b13f | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
| 2 | clab-lab-srl-cum-client2 | 17d82d5a6e21 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.4/24 | 2001:172:20:20::4/64 |
| 3 | clab-lab-srl-cum-client3 | f8d78527c1b1 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.6/24 | 2001:172:20:20::6/64 |
| 4 | clab-lab-srl-cum-client4 | 913a0172d0c5 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 5 | clab-lab-srl-cum-leaf1 | 9ffc334af2b1 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.9/24 | 2001:172:20:20::9/64 |
| 6 | clab-lab-srl-cum-leaf2 | 7d3eb855d749 | srlinux:21.6.1-235 | srl | | running | 172.20.20.14/24 | 2001:172:20:20::e/64 |
| 7 | clab-lab-srl-cum-leaf3 | e0d36ffc606a | networkop/cx:4.3.0 | cvx | | running | 172.20.20.7/24 | 2001:172:20:20::7/64 |
| 8 | clab-lab-srl-cum-leaf4 | e6753b1d099e | srlinux:21.6.1-235 | srl | | running | 172.20.20.12/24 | 2001:172:20:20::c/64 |
| 9 | clab-lab-srl-cum-spine1 | a7e63993d052 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.5/24 | 2001:172:20:20::5/64 |
| 10 | clab-lab-srl-cum-spine2 | 3124250d5b75 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
| 11 | clab-lab-srl-cum-spine3 | 5e300ca3f66f | srlinux:21.6.1-235 | srl | | running | 172.20.20.11/24 | 2001:172:20:20::b/64 |
| 12 | clab-lab-srl-cum-spine4 | 0a59e60a596b | srlinux:21.6.1-235 | srl | | running | 172.20.20.13/24 | 2001:172:20:20::d/64 |
| 13 | clab-lab-srl-cum-superspine1 | 3d9935413fc2 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.8/24 | 2001:172:20:20::8/64 |
| 14 | clab-lab-srl-cum-superspine2 | 5bc58d423e2c | srlinux:21.6.1-235 | srl | | running | 172.20.20.15/24 | 2001:172:20:20::f/64 |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
</syntaxhighlight>The containerlab program creates the updates into the host file + generates an ansible inventory file.

There are different ways to connect onto the docker instances, within docker exec commands or SSH directly the IP address related above.

For example :<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-spine1 bash
</syntaxhighlight>-->>> allows to launch a bash on the cumulus device (then you run the different net commands)<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-leaf2 sr_cli
</syntaxhighlight>-->>> allows to launch the cli on a SRLinux device

To end this setup preparation, the containerlab tool integrates an option to launch a webserver showing the lab topology in graph + node details

By default, it's listening on port 50080 of the host

Then, you can access to the following web page showing the topology + lab infos

[[File:Lab graph tool.jpg|ohne|mini|900x900px]]

Finally, if you want to stop a lab, please run '''<nowiki>''</nowiki>containerlab destroy --topo <topology>.yml<nowiki>''</nowiki>'''

At this step, nodes are running, connections are done, but configurations are empty.

'''So, it's time to configure them :)'''

Configuration files are attached to this blog. Once validated, they are exportable and can be called by the tool when launching the lab. Below how to do it :

Firstable, once you ve finished your "hand made config", you can run the following command to save it :<syntaxhighlight lang="bash">
containerlab save -t quickstart.clab.yml
</syntaxhighlight>'''To be noted it's only working for Nokia nodes in this lab'''. CVX configurations will have to be called by another method (or pushed manually)

It will create one folder per node into the lab folder which gives the following tree :
[[File:Topology-srl.jpg|ohne|mini]]

'''Note''': For Cumulus, you can apparently retrieve configurations files within adding the following parameters into lab config file. '''To be tested.''' some lab samples here : https://github.com/hellt/clabs/blob/main/labs/cvx04/symm-mh.clab.yml

Now we have the lab running, below the different commands to check that all is working as expected : <syntaxhighlight lang="yaml">
-------------------------------------------
Check L2 tables ///
-------------------------------------------

Nokia :

A:leaf2# show network-instance mac-vrf-100 bridge-table mac-table all
----------------------------------------------------------------------------------------------------------------------------------------------------------
Mac-table of network instance mac-vrf-100
----------------------------------------------------------------------------------------------------------------------------------------------------------
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
| Address | Destination | Dest Index | Type | Active | Aging | Last Update |
+====================+=========================================+============+=============+=========+========+=========================================+
| AA:C1:AB:0C:60:D6 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T13:06:17.000Z |
| | vtep:10.255.255.8 vni:100 | 67 | | | | |
| AA:C1:AB:23:31:04 | ethernet-1/3.100 | 2 | learnt | true | 300 | 2021-10-08T13:06:10.000Z |
| AA:C1:AB:4B:FF:71 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:55:28.000Z |
| | vtep:10.255.255.5 vni:100 | 60 | | | | |
| AA:C1:AB:B5:79:50 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:43:37.000Z |
| | vtep:10.255.255.7 vni:100 | 64 | | | | |
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 0 Total 0 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 1 Total 1 Active
Total Evpn Macs : 3 Total 3 Active
Total Evpn static Macs : 0 Total 0 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 4 Total 4 Active
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : type learnt matches the local learnt mac, evpn are learnt within VXLAN

Cumulus :

root@leaf1-cum-dc1:/# net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
100 bridge swp3 aa:c1:ab:4b:ff:71 00:00:02
100 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:17:21
100 bridge vni100 22:2d:67:f1:bb:26 permanent 02:17:21
100 bridge vni100 aa:c1:ab:0c:60:d6 extern_learn <1 sec
100 bridge vni100 aa:c1:ab:23:31:04 extern_learn 00:00:07
100 bridge vni100 aa:c1:ab:b5:79:50 extern_learn 00:13:28

root@leaf1-cum-dc1:/# net show bridge macs vlan 200

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
200 bridge swp3 aa:c1:ab:4b:ff:71 00:08:46
200 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:16:36
200 bridge vni200 00:00:5e:00:01:01 static sticky 00:14:26
200 bridge vni200 1e:0d:aa:46:53:b1 permanent 02:16:36
200 bridge vni200 02:77:6b:ff:00:41 static sticky 01:00:52
200 bridge vni200 02:f8:b8:ff:00:41 static sticky 00:14:26
200 bridge vni200 aa:c1:ab:b5:79:50 extern_learn 00:11:51
200 bridge vni10200 02:77:6b:ff:00:00 extern_learn 00:53:02
200 bridge vni10200 02:f8:b8:ff:00:00 extern_learn 00:14:26
200 bridge vni10200 86:5e:aa:ca:50:be permanent 02:16:36

Note : the macs with permanent flag are local interfaces, physical ou logic (vtep)
Note2 : extern_learn mac are the one learned via the VXLAN infra (and mapped to the vnixxx interface)

-------------------------------------------
Check arp tables ///
-------------------------------------------

Nokia :

:leaf2# show arpnd arp-entries
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| ethernet-1/1 | 0 | 10.254.254.13 | dynamic | AA:C1:AB:59:65:EA | 3 hours from now |
| ethernet-1/2 | 0 | 10.254.254.15 | dynamic | AA:C1:AB:CB:F8:8A | 3 hours from now |
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
| mgmt0 | 0 | 172.20.20.1 | dynamic | 02:42:63:9E:74:03 | 2 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 5 (0 static, 5 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : on Nokia you have to sort the entries basing on subinterface and the retrieve the L3VPN attached

A:leaf2# show arpnd arp-entries subinterface 200
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 2 (0 static, 2 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Cumulus :

Here we retrieve all arp infos linked to evpn

root@leaf1-cum-dc1:/# net show evpn arp-cache vni all
VNI 100 #ARP (IPv4 and IPv6, local and remote) 2

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.1.3 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0
192.168.1.1 local active aa:c1:ab:4b:ff:71 0/0

VNI 200 #ARP (IPv4 and IPv6, local and remote) 4

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.2.1 local active aa:c1:ab:4b:ff:71 0/0
192.168.200.254 remote active 00:00:5e:00:01:01 10.255.255.8 0/0
192.168.2.254 remote active 00:00:5e:00:01:01 10.255.255.6 0/0
192.168.200.1 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0

VNI 400 #ARP (IPv4 and IPv6, local and remote) 0

-------------------------------------------
Check bgp sessions' status - underlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep ipv4
| default | 10.254.254.19 | udl-superspine | S | 65003 | established | 0d:0h:22m:44s | ipv4-unicast | [13/4/4] |
| default | 10.254.254.22 | udl-leaf | S | 65013 | established | 0d:0h:22m:52s | ipv4-unicast | [13/1/4] |

Cumulus :

root@leaf3:/# net show bgp ipv4 unicast summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 13
RIB entries 25, using 4800 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.254.254.23 4 65002 466 403 0 0 0 00:19:34 4 13
10.254.254.25 4 65002 434 355 0 0 0 00:17:10 8 13

-------------------------------------------
Check bgp sessions' status - overlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep evpn
| default | 10.255.255.7 | ovl-leaf | S | 65013 | established | 0d:0h:24m:14s| evpn | [15/0/11] |
| default | 10.255.255.9 | ovl-superspine | S | 65003 | established | 0d:0h:24m:7s | evpn | [11/0/4] |

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.255.255.3 4 65002 5618 4742 0 0 0 03:53:35 19 27
10.255.255.4 4 65002 5565 4694 0 0 0 03:51:11 19 27

-------------------------------------------
Check EVPN L2 tables///
-------------------------------------------

Nokia :

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn route type 2
BGP table version is 17, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:100
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
Route Distinguisher: 10.255.255.5:200
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
Route Distinguisher: 10.255.255.6:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
Route Distinguisher: 10.255.255.7:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100
Route Distinguisher: 10.255.255.7:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.200.1]
10.255.255.7 32768 i
ET:8 RT:65000:200
Route Distinguisher: 10.255.255.8:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC

and if we focus on our vni 100

root@leaf3-cum-dc1:/# net show bgp evpn route vni 100 type 2
BGP table version is 48, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100

The additional route type "MAC+IP" is optional and is useful/mandatory when you do anycast irb.

-------------------------------------------
Check tables EVPN L3 ///
-------------------------------------------

Nokia :

--{ + running }--[ ]--
A:leaf2# show network-instance ip-vrf-200 route-table all
-----------------------------------------------------------------------------------------------------------------------------------------------------------
IPv4 Unicast route table of network instance ip-vrf-200
-----------------------------------------------------------------------------------------------------------------------------------------------------------
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
| Prefix | ID | Route Type | Route Owner | Best/Fib- | Metric | Pref | Next-hop | Next-hop |
| | | | | status(slot) | | | (Type) | Interface |
+=======================+=======+============+======================+======================+==========+=========+==============+==============+
| 192.168.2.0/24 | 9 | local | net_inst_mgr | True/success | 0 | 0 | 192.168.2.25 | irb0.200 |
| | | | | | | | 4 (direct) | |
| 192.168.2.254/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (extract) | |
| 192.168.2.255/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (broadcast) | |
| 192.168.200.0/24 | 0 | bgp-evpn | bgp_evpn_mgr | True/success | 0 | 170 | 10.255.255.8 | None |
| | | | | | | | (indirect/vx | |
| | | | | | | | lan) | |
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
-----------------------------------------------------------------------------------------------------------------------------------------------------------
4 IPv4 routes total
4 IPv4 prefixes with active routes
0 IPv4 prefixes with active ECMP routes
-----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--
A:leaf2#

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf1-cum-dc1:/# net show route vrf ip-vrf-200
show ip route vrf ip-vrf-200
=============================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF ip-vrf-200:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 04:48:42
B 192.168.2.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 04:27:26
C>* 192.168.2.0/24 is directly connected, vlan200, 04:48:42
B>* 192.168.200.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:48:49

root@leaf1-cum-dc1:/# net show bgp evpn route type 5
BGP table version is 3, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.6:10200
* [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
*> [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
Route Distinguisher: 10.255.255.8:10200
*> [5]:[0]:[24]:[192.168.200.0]
10.255.255.8 0 65001 65003 65002 65014 i
RT:65000:10200 ET:8 Rmac:02:f8:b8:ff:00:00

-------------------------------------------
Check arp hosts ///
-------------------------------------------

From client #1 /

bash-5.0# arp
Address HWtype HWaddress Flags Mask Iface
192.168.1.3 ether aa:c1:ab:b5:79:50 C eth1.100
192.168.1.4 ether aa:c1:ab:0c:60:d6 C eth1.100
192.168.2.2 ether aa:c1:ab:23:31:04 C eth1.200
192.168.2.254 ether aa:c1:ab:4f:f2:e7 C eth1.200
192.168.1.2 ether aa:c1:ab:23:31:04 C eth1.100
containerlab.local ether 02:42:63:9e:74:03 C eth0

-------------------------------------------
Ping ///
-------------------------------------------

From client #1 /

L2 service ping (vlan 100) ----------------------

bash-5.0# ping -c 1 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.812 ms

--- 192.168.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.812/0.812/0.812/0.000 MsgRcvd
bash-5.0# ping -c 1 192.168.1.3
PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_seq=1 ttl=64 time=1.40 ms

--- 192.168.1.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.403/1.403/1.403/0.000 ms
bash-5.0# ping -c 1 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=1.62 ms

--- 192.168.1.4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.619/1.619/1.619/0.000 ms

L3 service ping (vlan 200)

bash-5.0# ping -c 1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.053 ms

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms
bash-5.0# ^C
bash-5.0# ping -c 1 192.168.2.2
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.611 ms

--- 192.168.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.611/0.611/0.611/0.000 ms

-------------------------------------------
Traceroute ///
-------------------------------------------

L3 service traceroute (vlan 200)

bash-5.0# traceroute 192.168.200.1
traceroute to 192.168.200.1 (192.168.200.1), 30 hops max, 46 byte packets
1 192.168.2.254 (192.168.2.254) 0.012 ms 0.010 ms 0.007 ms
2 192.168.200.254 (192.168.200.254) 2.751 ms * 2.549 ms
3 192.168.200.1 (192.168.200.1) 2.084 ms 1.805 ms 1.761 ms

</syntaxhighlight>To come back on the setup, we wanted to implement solutions inspired by the <nowiki>RFC 7938</nowiki> rfc7938 (see section 5.2.1)

* eBGP will be used for Underlay and Overlay
* eBGP single-hop sessions are established over direct point-to-point links interconnecting the network nodes, no multi-hop or loopback sessions are used, even in the case of multiple links between the same pair of nodes.
* Private Use ASNs from the range 64512-65534 are used to avoid ASN conflicts.
* A single ASN is allocated to all of the Clos topology's Tier 1 devices.
* A unique ASN is allocated to each set of Tier 2 devices in the same cluster.
* A unique ASN is allocated to every Tier 3 device (e.g., ToR) in this topology.

In add, depending of network nodes, it will be interesting to :

* Add fast-failover option for underlay + add link-debounce on links to limit possible impacts in case of link flaps. BFD could be considered in some cases
* Have the best ECMP hashing algorythm on leaf nodes. On tier-1/tier-2 levels hashing will be based on underlay part (trafic inter-vtep), so L3-L4 decision, but some models can retreive the vni-id if I well remember.
* Implement the "multipath relax" since we could have multiple AS paths for a same destination.
* Implement gshut community on underlay (on tier-1/tier-2) or at least policies to prepend underlay routes to avoid some nodes in case of maintenance.

== Thoughts/Points to deepend ==
On leaf nodes, LACP over multiple nodes is feasible within ESI, but impact on incidents/maintenance can vary depending on the vendors.

'''Tests need to be done to check that.'''

Some sysadmin would like to KISS and have client servers connected in active/passive bonding, with eventually single-leaf ports aggregation.

'''Note :''' NH ESI in FIB management on ingress nodes can have some limitations (numbers of installable next-hop, load-balancing, etc. oftenly linked to the node chipset)

Another important engineering point to define at the beginning is : '''Do we authorize multisite L2 propagation or not ?'''

From my point-of-view, the smaller are the L2 segments the better it is. I would suggest Local L2 segments wit anycast gateways provided by Leaf switches, with L3 routing between DCs within EVPN type5 routes sharing only. Nevertheless, some business requirements for endcustomers can keep extended L2 alive.

We have to be careful when implementing Consistent hashing. ex: <nowiki>https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-44/Layer-3/Routing/Equal-Cost-Multipath-Load-Sharing-Hardware-ECMP/</nowiki>

If possible activate it per VRF, not globally, and restrict on dedicated subnets << depends of the vendor, it can be dangerous regarding memory management

Think at configuring the bgp option to not advertise routes to peers before they are installed in FIB.

We can implement leaf roles in some cases, to be able to do "light traffic engineering " on DCI links in some setup.

These roles will be managed within bgp communities added to VTEP /32. Look at the diagram below. Depending of the colour, the eBGP sessions between SupS will prepend or not the VTEP IPs announced to the remote DC

[[File:Fabric-roles.jpg|ohne|mini|1200x1200px]]

<blockquote>'''To be continued'''</blockquote>

== Lab archives ==
<blockquote>Reminder : following config are for lab, so please don't pay attention on typos/lack of security params/unused config. Thanks \o/</blockquote>

=== Credentials ===
<syntaxhighlight lang="bash">
SRLinux : admin/<no_password>

CVX : cumulus/cumulus
</syntaxhighlight>

=== Lab archives ===
layer 2 (vlan 100) + layer 3 (vlan 200) lab, 10/12/2021
[[File:20211008 lab-nokia-cvx-containerlab.zip|ohne|mini]]
<u>If you want to exchange about that lab, those topics, please don't hesitate to contact us</u>

= SONiC Lab =

<span><to be noted that those informations are included into the readme.md file into the archive></span>

== Lab diagram ==
[[File:SONIC_LAB.jpg]]

<div></div>Lab archive [Attach:Sonic-lab-20201030.zip]

This lab is a forked lab from many found on @, starting from Microsoft one : https://github.com/Azure/SONiC/wiki/SONiC-P4-Software-Switch

The target of this lab is to simulate a simple IP/Fabric in SPINE/LEAF topology. Each Leaf provide local L2 connectivity + L3 gateways to hosts. Flows between hosts from different leaf are L3 routed, in a "node path" LEAF-SPINE-LEAF.

Nothing complicated, but it allows to put a first foot into SONiC world :)

For the moment, as far as we know, MC-LAG + EVPN/VXLAN features are only available on entreprise SONiC versions. Tests to come once physic switches in enterprise version will be available in our lab.

The diagram of the architecture is into the zip file (SONIC_LAB.jpg)

To run the lab you need :

- to have installed docker by using ./install_docker_ovs.sh - to run the ./load_image.sh in order to get the Sonic image

Note : don't change file / folders names of network devices (leafX, spineX)

Once you're ready, please run the lab by launching "./start.sh"

The script :<div></div>
* launches the containers without Docket network features
* create network namespaces
* build linux bridges to interconnect the containers
* configure host IPs
* launch startup sonic scripts into sonic devices
* once devices started, checks + ping tests are run. If all BGP sessions are ok on spine + all ping are fine, the setup works well.
The "start" script has several comments, perhaps without the right vocabulary, so please be tolerant :)

If you want to stop the lab, please launch "./stop.sh"

Below some useful commands to run for sonic containers :

=== <span>run ping</span> ===
sudo docker exec -it <container> ping <ip>

This command triggers ping into the targetted container :
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it leaf1 ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.052 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.034 ms
^C--- 192.168.1.1 ping statistics ---
</syntaxhighlight>

=== <span>check bgp sessions</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip bgp sum"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print the bgp status.
Below, we run it on one spine, you should have 4 sessiosn UP (for the 4 leaves)
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip bgp sum"
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 6409 6411 0 0 0 01:46:46 3
10.1.0.2 4 65102 6406 6410 0 0 0 01:46:44 3
10.1.0.4 4 65103 6407 6410 0 0 0 01:46:44 4
10.1.0.6 4 65104 6406 6412 0 0 0 01:46:46 1
Total number of neighbors 4
</syntaxhighlight>

=== <span>check ip route</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip route"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print its routing table?

<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,
> - selected route, * - FIB route
C>* 10.1.0.0/31 is directly connected, Vlan11
C>* 10.1.0.2/31 is directly connected, Vlan21
C>* 10.1.0.4/31 is directly connected, Vlan31
C>* 10.1.0.6/31 is directly connected, Vlan41
C>* 127.0.0.0/8 is directly connected, lo
B>* 192.168.1.0/24 [20/0] via 10.1.0.0, Vlan11, 01:47:23
B>* 192.168.2.0/24 [20/0] via 10.1.0.2, Vlan21, 01:47:21
B>* 192.168.3.0/24 [20/0] via 10.1.0.4, Vlan31, 01:47:21
B>* 192.168.4.0/24 [20/0] via 10.1.0.6, Vlan41, 01:47:23
</syntaxhighlight>
For more commands check SONiC docs : https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md

To finish, below the output of the start.sh when all goes well<div></div>
plancastre@dockerhost:~/00.Projects/p4-version-pla$ ./start.sh
---------------------------------------------------
Creating the containers: starting
---------------------------------------------------
0b4bbb3c6cfc9895466ea13a9e940d1a176d66f45deee9752e9b4732d735bb03
5ca9486baab3e8a5259062167132a944eee717875d3cc1ef29fe490cba3184c1
46c9c09dba27a32715b9fbe0bc434a9284b9d59317d278df693f425cb669f037
898e6afbcb2977115acbba2a69d069796cfd70fb90228b6c18e09ba4584365f7
eb9b3ef6d14784c9ffe4d9d5df3e921a24559889cb8770d226263aad6761182a
ae5f3b2c628da687eaabb331dc2cd989e71bcb386854e7a0be6ec587df99d49a
122197bffd7566a01ce3b6565fdfb6eb893aab7d5236f14e5a9bbbe95037ced2
a604520a080e2a2184598ad1d915c0c182d16e5185033373e95a4772d8978a67
0c49e18d78b2aa7c8ef8e937b60285aa56ec0bf4a8530226285270a085e11d5c
eb66336b8c6bd8333f1f4f685369e6cac973b817c35bb7126ac4d7ba72ef71bf
---------------------------------------------------
Retreiving PID from docker containers
---------------------------------------------------
---------------------------------------------------
Creating network namespaces for containers
---------------------------------------------------
List of created namespaces
ns_host4
ns_host3
ns_host2
ns_host1
ns_spine2
ns_spine1
ns_leaf4
ns_leaf3
ns_leaf2
ns_leaf1
---------------------------------------------------
Creating network connectivity between containers
---------------------------------------------------
---------------------------------------------------
Now assigning IPs to hosts (host1 to host4)
---------------------------------------------------
---------------------------------------------------
Configuration of containers is now finished
---------------------------------------------------
---------------------------------------------------
Now booting switches, please wait 2 minutes for switches to load
---------------------------------------------------
---------------------------------------------------
2 minutes period is finished
---------------------------------------------------
---------------------------------------------------
Now fixing the iptables
---------------------------------------------------
---------------------------------------------------
Backend actions are now finished
---------------------------------------------------
---------------------------------------------------
Now we'll check in 15 sec that BGP peers are all Up
---------------------------------------------------
---------------------------------------------------
SPINE1 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 94 97 0 0 0 00:01:28 4
10.1.0.2 4 65102 95 97 0 0 0 00:01:28 5
10.1.0.4 4 65103 95 97 0 0 0 00:01:28 5
10.1.0.6 4 65104 95 97 0 0 0 00:01:28 5
Total number of neighbors 4
---------------------------------------------------
SPINE2 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.2.0.1, local AS number 65002
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.2.0.0 4 65101 93 96 0 0 0 00:01:29 2
10.2.0.2 4 65102 94 100 0 0 0 00:01:32 1
10.2.0.4 4 65103 94 100 0 0 0 00:01:32 1
10.2.0.6 4 65104 94 100 0 0 0 00:01:32 1
Total number of neighbors 4
---------------------------------------------------
Now if all is ok, we'll do some ping tests
---------------------------------------------------
---------------------------------------------------
host1 ping its gateway
---------------------------------------------------
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=17.6 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=9.14 ms
--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 9.145/13.390/17.636/4.247 ms
---------------------------------------------------
host2 ping its gateway
---------------------------------------------------
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=23.6 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=6.21 ms
--- 192.168.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 6.218/14.950/23.683/8.733 ms
---------------------------------------------------
host3 ping its gateway
---------------------------------------------------
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=18.8 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=12.7 ms
--- 192.168.3.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.716/15.764/18.812/3.048 ms
---------------------------------------------------
host4 ping its gateway
---------------------------------------------------
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.
64 bytes from 192.168.4.1: icmp_seq=1 ttl=64 time=17.7 ms
64 bytes from 192.168.4.1: icmp_seq=2 ttl=64 time=4.46 ms
--- 192.168.4.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.464/11.107/17.750/6.643 ms
---------------------------------------------------
host1 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=16.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.049/17.253/19.532/1.615 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=15.5 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=14.3 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.312/15.592/16.939/1.083 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=20.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=16.8 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.048/17.871/20.739/2.058 ms
---------------------------------------------------
host2 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=16.6 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.222/16.583/16.924/0.286 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.6 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=18.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=26.8 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.620/20.549/26.887/4.524 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=33.8 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=27.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=20.2 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 20.216/27.259/33.818/5.566 ms
---------------------------------------------------
host3 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=14.5 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=15.5 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.535/16.549/19.541/2.162 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.4 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=14.6 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=18.2 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.643/16.458/18.278/1.491 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=15.3 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=14.1 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=13.1 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 13.171/14.234/15.334/0.883 ms
---------------------------------------------------
host4 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=21.9 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=21.3 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=14.6 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 14.619/19.284/21.932/3.310 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=14.8 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=18.4 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=14.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.148/15.786/18.409/1.879 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=15.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=18.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 15.117/16.788/18.962/1.612 ms
---------------------------------------------------
If bot spine have all BGP sessions and all ping succeeded, all works well
Congrats ! please check readme file to find some details
---------------------------------------------------

Nesevo Virtual Labs

2021-10-29T08:12:19Z

PierreLancastre: /* Lab diagram */

[[File:nesevo-labs-v2.png]]

= Nokia/Cumulus Multi-Vendor Lab =

The targets of this lab are to :

* mix Nokia SR-Linux and Cumulus VXLAN nodes, in spine or leaf roles
* make this environment work for different kinds of setup : cross-DC L2, differentiated L3 gateways per leaf per VRF, anycast gateway cross or intra DC, etc.
* spot compatibility issues if existing

To reach those goals, we'll use the Nokia lab tool "containerlab" https://containerlab.srlinux.dev/, labs from https://clabs.netdevops.me/ helped a lot. Many thanks to the contributors.

We'll need a Linux host with Docker installed on it + kvm virtualization enhancements activated (important if your host is a VM). If virtualization optimization is not ok, you won't be able to launch the most of devices.

Installation details are referenced there : https://containerlab.srlinux.dev/install/

Once installed, you'll need to register the different NOS images you'll use. Cumulus image is present on a public registrery, but Nokia/Arista/Juniper are not, you'll need a vendor support account to retreive them. Once downloaded (.tgz file typically) you can "docker load" the image :

For private images :<syntaxhighlight lang="bash">
sudo docker load -i 21.6.1-235.tar.xz /* (Nokia) */
sudo docker load -i cEOS64-lab-4.26.1F.tar /* (Arista) */
</syntaxhighlight>For public images :<syntaxhighlight lang="bash">
sudo docker pull networkop/cx:4.3.0 /* (Cumulus) */
</syntaxhighlight>

To verify your installed images, you can run "sudo docker images"<syntaxhighlight lang="bash">
plancastre@containerlab:~/00.LAB/senss$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
networkop/cx 4.3.0 a40541ed15ba 2 weeks ago 733MB
srlinux 21.6.1-235 e29aed30e83f 3 weeks ago 1.14GB
networkop/ignite dev dcc5eff29355 4 weeks ago 37.9MB
networkop/kernel 4.19 e794d8875f83 3 months ago 88MB
frrouting/frr v7.5.1 c3e13a4c5918 3 months ago 123MB

</syntaxhighlight>Once you've got all your images, to run the lab, you'll need a .yml file defining the topology to run.

In our lab, we've started with the 5-clos lab file from Nokia site, which we've forked.

We can see there is a branch "kinds" where we can define templates of nodes, in order to not rewrite whole parameters when creating nodes. For the images, it's important to take the repository name + tag.

'''Note:''' If the tag is "latest" it is not needed.

All supported "kinds" are referenced there : https://containerlab.srlinux.dev/manual/kinds/kinds/

'''Note:''' yml is indentation sensitive, so please respect the "double space" used to define hierarchy. Below the lab diagram + the yml file associated

== Infos on nodes ==

=== L3 infos ===
{| class="wikitable"
|+
!Device
!OS
!ASN
!VTEP IP
!swp1 or eth1/1
!swp2 or eth1/2
!swp3 or eth1/3
!SVI
|-
|client1
|Linux
|<none>
|<none>
|vlan100: 192.168.1.1/24
vlan200: 192.168.2.1/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client2
|Linux
|<none>
|<none>
|vlan100: 192.168.1.2/24
vlan200: 192.168.2.2/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client3
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.1/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|client4
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.2/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|leaf1
|CVX
|65011
|10.255.255.5/32
|10.254.254.8/31
|10.254.254.10/31
|trunk vlan 100,200
|vlan200 : 192.168.2.254/24
|-
|leaf2
|SRLinux
|65012
|10.255.255.6/32
|10.254.254.12/31
|10.254.254.14/31
|trunk vlan 100,200
|irb0.200 : 192.168.2.254/24
|-
|leaf3
|CVX
|65013
|10.255.255.7/32
|10.254.254.22/31
|10.254.254.24/31
|trunk vlan 100,200
|vlan200 : 192.168.200.254/24
|-
|leaf4
|SRLinux
|65014
|10.255.255.8/32
|10.254.254.26/31
|10.254.254.28/31
|trunk vlan 100,200
|irb0.200 : 192.168.200.254/24
|-
|spine1
|CVX
|65001
|10.255.255.1/32
|10.254.254.9/31
|10.254.254.13/31
|10.254.254.0/31
|<none>
|-
|spine2
|CVX
|65001
|10.255.255.2/32
|10.254.254.11/31
|10.254.254.15/31
|10.254.254.16/31
|<none>
|-
|spine3
|SRLinux
|65002
|10.255.255.3/32
|10.254.254.23/31
|10.254.254.27/31
|10.254.254.18/31
|<none>
|-
|spine4
|SRLinux
|65002
|10.255.255.4/32
|10.254.254.25/31
|10.254.254.29/31
|10.254.254.20/31
|<none>
|-
|superspine1
|CVX
|65003
|10.255.255.9/32
|10.254.254.1/31
|10.254.254.19/31
|<none>
|<none>
|-
|superspine2
|SRLinux
|65003
|10.255.255.10/32
|10.254.254.17/31
|10.254.254.21/31
|<none>
|<none>
|}

=== VXLAN/BGP infos ===
Note : in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>
{| class="wikitable"
|+
!service
!vlan id
!route-target
!VNI
!Commentaires
|-
|L2 cross-DC
|100
|target:65000:100
|100
|the 4 clients are in the same broadcast domain / subnet
|-
|L3VPN cros-DC
|200
|target:65000:200
target:65000:10200
|200 (for L2)
10200 (for routing)
|the 4 clients are separated in 2 different subnets. Traffic is routed within dedicated VNI
To be noted that in the current setup VMTO is not implemented, so cross DC traffic is not optimized.
|}

== Lab diagram ==
[[File:Lab-srl-diagram.png|ohne|mini|600x600px]]

== yml lab file ==
<syntaxhighlight lang="yaml">
# topology documentation: http://containerlab.srlinux.dev/lab-examples/min-5clos/
name: lab-srl-cum

topology:
kinds:
srl:
image: srlinux:21.6.1-235
linux:
image: ghcr.io/hellt/network-multitool
cvx:
image: networkop/cx:4.3.0
runtime: docker
nodes:
leaf1:
kind: cvx
leaf2:
kind: srl
type: ixrd2
leaf3:
kind: cvx
leaf4:
kind: srl
type: ixrd2

spine1:
kind: cvx
spine2:
kind: cvx
spine3:
kind: srl
type: ixr6
spine4:
kind: srl
type: ixr6

superspine1:
kind: cvx
superspine2:
kind: srl
type: ixr6

client1:
kind: linux
client2:
kind: linux
client3:
kind: linux
client4:
kind: linux

links:
# leaf to spine links POD1
- endpoints: ["leaf1:swp1", "spine1:swp1"]
- endpoints: ["leaf1:swp2", "spine2:swp1"]
- endpoints: ["leaf2:e1-1", "spine1:swp2"]
- endpoints: ["leaf2:e1-2", "spine2:swp2"]

# spine to superspine links POD1
- endpoints: ["spine1:swp3", "superspine1:swp1"]
- endpoints: ["spine2:swp3", "superspine2:e1-1"]

# leaf to spine links POD2
- endpoints: ["leaf3:swp1", "spine3:e1-1"]
- endpoints: ["leaf3:swp2", "spine4:e1-1"]
- endpoints: ["leaf4:e1-1", "spine3:e1-2"]
- endpoints: ["leaf4:e1-2", "spine4:e1-2"]

# spine to superspine links POD2
- endpoints: ["spine3:e1-3", "superspine1:swp2"]
- endpoints: ["spine4:e1-3", "superspine2:e1-2"]

# client connection links
- endpoints: ["client1:eth1", "leaf1:swp3"]
- endpoints: ["client2:eth1", "leaf2:e1-3"]
- endpoints: ["client3:eth1", "leaf3:swp3"]
- endpoints: ["client4:eth1", "leaf4:e1-3"]

</syntaxhighlight>

Once the nodes are defined, we have to define the needed connections between the nodes.

Each endpoint is defined as '<nowiki/>'''<nowiki/>'<node_name>:<interface_name><nowiki>''</nowiki>'''

When all is ok, you can launch in suduoer the topology.

For example :<syntaxhighlight lang="bash">
sudo containerlab deploy --topo topo.clab.yml --reconfigure

Note: The option "reconfigure" is used there in order to not have to destroy instances when updating topology/relaunching.
</syntaxhighlight>Another info regarding the yml file, it's possible to quickly generate files with the '''"containerlab gen -n 3tier --nodes 4,2,1"''' command for example. https://containerlab.srlinux.dev/cmd/generate/

To be noted that the "tier-x" notion in containerlab is inversed regarding the rfc 7938 (tier1 is superspine level into the rfc, tier1 is leaf level in nokia tool)

At the end of the topology launch, you can retreive the details :<syntaxhighlight lang="yaml">
INFO[0011] Writing /etc/hosts file
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| # | Name | Container ID | Image | Kind | Group | State | IPv4 Address | IPv6 Address |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| 1 | clab-lab-srl-cum-client1 | 25e2bbf6b13f | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
| 2 | clab-lab-srl-cum-client2 | 17d82d5a6e21 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.4/24 | 2001:172:20:20::4/64 |
| 3 | clab-lab-srl-cum-client3 | f8d78527c1b1 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.6/24 | 2001:172:20:20::6/64 |
| 4 | clab-lab-srl-cum-client4 | 913a0172d0c5 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 5 | clab-lab-srl-cum-leaf1 | 9ffc334af2b1 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.9/24 | 2001:172:20:20::9/64 |
| 6 | clab-lab-srl-cum-leaf2 | 7d3eb855d749 | srlinux:21.6.1-235 | srl | | running | 172.20.20.14/24 | 2001:172:20:20::e/64 |
| 7 | clab-lab-srl-cum-leaf3 | e0d36ffc606a | networkop/cx:4.3.0 | cvx | | running | 172.20.20.7/24 | 2001:172:20:20::7/64 |
| 8 | clab-lab-srl-cum-leaf4 | e6753b1d099e | srlinux:21.6.1-235 | srl | | running | 172.20.20.12/24 | 2001:172:20:20::c/64 |
| 9 | clab-lab-srl-cum-spine1 | a7e63993d052 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.5/24 | 2001:172:20:20::5/64 |
| 10 | clab-lab-srl-cum-spine2 | 3124250d5b75 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
| 11 | clab-lab-srl-cum-spine3 | 5e300ca3f66f | srlinux:21.6.1-235 | srl | | running | 172.20.20.11/24 | 2001:172:20:20::b/64 |
| 12 | clab-lab-srl-cum-spine4 | 0a59e60a596b | srlinux:21.6.1-235 | srl | | running | 172.20.20.13/24 | 2001:172:20:20::d/64 |
| 13 | clab-lab-srl-cum-superspine1 | 3d9935413fc2 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.8/24 | 2001:172:20:20::8/64 |
| 14 | clab-lab-srl-cum-superspine2 | 5bc58d423e2c | srlinux:21.6.1-235 | srl | | running | 172.20.20.15/24 | 2001:172:20:20::f/64 |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
</syntaxhighlight>The containerlab program creates the updates into the host file + generates an ansible inventory file.

There are different ways to connect onto the docker instances, within docker exec commands or SSH directly the IP address related above.

For example :<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-spine1 bash
</syntaxhighlight>-->>> allows to launch a bash on the cumulus device (then you run the different net commands)<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-leaf2 sr_cli
</syntaxhighlight>-->>> allows to launch the cli on a SRLinux device

To end this setup preparation, the containerlab tool integrates an option to launch a webserver showing the lab topology in graph + node details

By default, it's listening on port 50080 of the host

Then, you can access to the following web page showing the topology + lab infos

[[File:Lab graph tool.jpg|ohne|mini|900x900px]]

Finally, if you want to stop a lab, please run '''<nowiki>''</nowiki>containerlab destroy --topo <topology>.yml<nowiki>''</nowiki>'''

At this step, nodes are running, connections are done, but configurations are empty.

'''So, it's time to configure them :)'''

Configuration files are attached to this blog. Once validated, they are exportable and can be called by the tool when launching the lab. Below how to do it :

Firstable, once you ve finished your "hand made config", you can run the following command to save it :<syntaxhighlight lang="bash">
containerlab save -t quickstart.clab.yml
</syntaxhighlight>'''To be noted it's only working for Nokia nodes in this lab'''. CVX configurations will have to be called by another method (or pushed manually)

It will create one folder per node into the lab folder which gives the following tree :
[[File:Topology-srl.jpg|ohne|mini]]

'''Note''': For Cumulus, you can apparently retrieve configurations files within adding the following parameters into lab config file. '''To be tested.''' some lab samples here : https://github.com/hellt/clabs/blob/main/labs/cvx04/symm-mh.clab.yml

Now we have the lab running, below the different commands to check that all is working as expected : <syntaxhighlight lang="yaml">
-------------------------------------------
Check L2 tables ///
-------------------------------------------

Nokia :

A:leaf2# show network-instance mac-vrf-100 bridge-table mac-table all
----------------------------------------------------------------------------------------------------------------------------------------------------------
Mac-table of network instance mac-vrf-100
----------------------------------------------------------------------------------------------------------------------------------------------------------
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
| Address | Destination | Dest Index | Type | Active | Aging | Last Update |
+====================+=========================================+============+=============+=========+========+=========================================+
| AA:C1:AB:0C:60:D6 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T13:06:17.000Z |
| | vtep:10.255.255.8 vni:100 | 67 | | | | |
| AA:C1:AB:23:31:04 | ethernet-1/3.100 | 2 | learnt | true | 300 | 2021-10-08T13:06:10.000Z |
| AA:C1:AB:4B:FF:71 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:55:28.000Z |
| | vtep:10.255.255.5 vni:100 | 60 | | | | |
| AA:C1:AB:B5:79:50 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:43:37.000Z |
| | vtep:10.255.255.7 vni:100 | 64 | | | | |
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 0 Total 0 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 1 Total 1 Active
Total Evpn Macs : 3 Total 3 Active
Total Evpn static Macs : 0 Total 0 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 4 Total 4 Active
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : type learnt matches the local learnt mac, evpn are learnt within VXLAN

Cumulus :

root@leaf1-cum-dc1:/# net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
100 bridge swp3 aa:c1:ab:4b:ff:71 00:00:02
100 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:17:21
100 bridge vni100 22:2d:67:f1:bb:26 permanent 02:17:21
100 bridge vni100 aa:c1:ab:0c:60:d6 extern_learn <1 sec
100 bridge vni100 aa:c1:ab:23:31:04 extern_learn 00:00:07
100 bridge vni100 aa:c1:ab:b5:79:50 extern_learn 00:13:28

root@leaf1-cum-dc1:/# net show bridge macs vlan 200

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
200 bridge swp3 aa:c1:ab:4b:ff:71 00:08:46
200 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:16:36
200 bridge vni200 00:00:5e:00:01:01 static sticky 00:14:26
200 bridge vni200 1e:0d:aa:46:53:b1 permanent 02:16:36
200 bridge vni200 02:77:6b:ff:00:41 static sticky 01:00:52
200 bridge vni200 02:f8:b8:ff:00:41 static sticky 00:14:26
200 bridge vni200 aa:c1:ab:b5:79:50 extern_learn 00:11:51
200 bridge vni10200 02:77:6b:ff:00:00 extern_learn 00:53:02
200 bridge vni10200 02:f8:b8:ff:00:00 extern_learn 00:14:26
200 bridge vni10200 86:5e:aa:ca:50:be permanent 02:16:36

Note : the macs with permanent flag are local interfaces, physical ou logic (vtep)
Note2 : extern_learn mac are the one learned via the VXLAN infra (and mapped to the vnixxx interface)

-------------------------------------------
Check arp tables ///
-------------------------------------------

Nokia :

:leaf2# show arpnd arp-entries
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| ethernet-1/1 | 0 | 10.254.254.13 | dynamic | AA:C1:AB:59:65:EA | 3 hours from now |
| ethernet-1/2 | 0 | 10.254.254.15 | dynamic | AA:C1:AB:CB:F8:8A | 3 hours from now |
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
| mgmt0 | 0 | 172.20.20.1 | dynamic | 02:42:63:9E:74:03 | 2 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 5 (0 static, 5 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : on Nokia you have to sort the entries basing on subinterface and the retrieve the L3VPN attached

A:leaf2# show arpnd arp-entries subinterface 200
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 2 (0 static, 2 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Cumulus :

Here we retrieve all arp infos linked to evpn

root@leaf1-cum-dc1:/# net show evpn arp-cache vni all
VNI 100 #ARP (IPv4 and IPv6, local and remote) 2

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.1.3 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0
192.168.1.1 local active aa:c1:ab:4b:ff:71 0/0

VNI 200 #ARP (IPv4 and IPv6, local and remote) 4

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.2.1 local active aa:c1:ab:4b:ff:71 0/0
192.168.200.254 remote active 00:00:5e:00:01:01 10.255.255.8 0/0
192.168.2.254 remote active 00:00:5e:00:01:01 10.255.255.6 0/0
192.168.200.1 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0

VNI 400 #ARP (IPv4 and IPv6, local and remote) 0

-------------------------------------------
Check bgp sessions' status - underlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep ipv4
| default | 10.254.254.19 | udl-superspine | S | 65003 | established | 0d:0h:22m:44s | ipv4-unicast | [13/4/4] |
| default | 10.254.254.22 | udl-leaf | S | 65013 | established | 0d:0h:22m:52s | ipv4-unicast | [13/1/4] |

Cumulus :

root@leaf3:/# net show bgp ipv4 unicast summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 13
RIB entries 25, using 4800 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.254.254.23 4 65002 466 403 0 0 0 00:19:34 4 13
10.254.254.25 4 65002 434 355 0 0 0 00:17:10 8 13

-------------------------------------------
Check bgp sessions' status - overlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep evpn
| default | 10.255.255.7 | ovl-leaf | S | 65013 | established | 0d:0h:24m:14s| evpn | [15/0/11] |
| default | 10.255.255.9 | ovl-superspine | S | 65003 | established | 0d:0h:24m:7s | evpn | [11/0/4] |

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.255.255.3 4 65002 5618 4742 0 0 0 03:53:35 19 27
10.255.255.4 4 65002 5565 4694 0 0 0 03:51:11 19 27

-------------------------------------------
Check EVPN L2 tables///
-------------------------------------------

Nokia :

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn route type 2
BGP table version is 17, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:100
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
Route Distinguisher: 10.255.255.5:200
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
Route Distinguisher: 10.255.255.6:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
Route Distinguisher: 10.255.255.7:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100
Route Distinguisher: 10.255.255.7:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.200.1]
10.255.255.7 32768 i
ET:8 RT:65000:200
Route Distinguisher: 10.255.255.8:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC

and if we focus on our vni 100

root@leaf3-cum-dc1:/# net show bgp evpn route vni 100 type 2
BGP table version is 48, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100

The additional route type "MAC+IP" is optional and is useful/mandatory when you do anycast irb.

-------------------------------------------
Check tables EVPN L3 ///
-------------------------------------------

Nokia :

--{ + running }--[ ]--
A:leaf2# show network-instance ip-vrf-200 route-table all
-----------------------------------------------------------------------------------------------------------------------------------------------------------
IPv4 Unicast route table of network instance ip-vrf-200
-----------------------------------------------------------------------------------------------------------------------------------------------------------
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
| Prefix | ID | Route Type | Route Owner | Best/Fib- | Metric | Pref | Next-hop | Next-hop |
| | | | | status(slot) | | | (Type) | Interface |
+=======================+=======+============+======================+======================+==========+=========+==============+==============+
| 192.168.2.0/24 | 9 | local | net_inst_mgr | True/success | 0 | 0 | 192.168.2.25 | irb0.200 |
| | | | | | | | 4 (direct) | |
| 192.168.2.254/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (extract) | |
| 192.168.2.255/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (broadcast) | |
| 192.168.200.0/24 | 0 | bgp-evpn | bgp_evpn_mgr | True/success | 0 | 170 | 10.255.255.8 | None |
| | | | | | | | (indirect/vx | |
| | | | | | | | lan) | |
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
-----------------------------------------------------------------------------------------------------------------------------------------------------------
4 IPv4 routes total
4 IPv4 prefixes with active routes
0 IPv4 prefixes with active ECMP routes
-----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--
A:leaf2#

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf1-cum-dc1:/# net show route vrf ip-vrf-200
show ip route vrf ip-vrf-200
=============================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF ip-vrf-200:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 04:48:42
B 192.168.2.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 04:27:26
C>* 192.168.2.0/24 is directly connected, vlan200, 04:48:42
B>* 192.168.200.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:48:49

root@leaf1-cum-dc1:/# net show bgp evpn route type 5
BGP table version is 3, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.6:10200
* [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
*> [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
Route Distinguisher: 10.255.255.8:10200
*> [5]:[0]:[24]:[192.168.200.0]
10.255.255.8 0 65001 65003 65002 65014 i
RT:65000:10200 ET:8 Rmac:02:f8:b8:ff:00:00

-------------------------------------------
Check arp hosts ///
-------------------------------------------

From client #1 /

bash-5.0# arp
Address HWtype HWaddress Flags Mask Iface
192.168.1.3 ether aa:c1:ab:b5:79:50 C eth1.100
192.168.1.4 ether aa:c1:ab:0c:60:d6 C eth1.100
192.168.2.2 ether aa:c1:ab:23:31:04 C eth1.200
192.168.2.254 ether aa:c1:ab:4f:f2:e7 C eth1.200
192.168.1.2 ether aa:c1:ab:23:31:04 C eth1.100
containerlab.local ether 02:42:63:9e:74:03 C eth0

-------------------------------------------
Ping ///
-------------------------------------------

From client #1 /

L2 service ping (vlan 100) ----------------------

bash-5.0# ping -c 1 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.812 ms

--- 192.168.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.812/0.812/0.812/0.000 MsgRcvd
bash-5.0# ping -c 1 192.168.1.3
PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_seq=1 ttl=64 time=1.40 ms

--- 192.168.1.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.403/1.403/1.403/0.000 ms
bash-5.0# ping -c 1 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=1.62 ms

--- 192.168.1.4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.619/1.619/1.619/0.000 ms

L3 service ping (vlan 200)

bash-5.0# ping -c 1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.053 ms

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms
bash-5.0# ^C
bash-5.0# ping -c 1 192.168.2.2
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.611 ms

--- 192.168.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.611/0.611/0.611/0.000 ms

-------------------------------------------
Traceroute ///
-------------------------------------------

L3 service traceroute (vlan 200)

bash-5.0# traceroute 192.168.200.1
traceroute to 192.168.200.1 (192.168.200.1), 30 hops max, 46 byte packets
1 192.168.2.254 (192.168.2.254) 0.012 ms 0.010 ms 0.007 ms
2 192.168.200.254 (192.168.200.254) 2.751 ms * 2.549 ms
3 192.168.200.1 (192.168.200.1) 2.084 ms 1.805 ms 1.761 ms

</syntaxhighlight>To come back on the setup, we wanted to implement solutions inspired by the <nowiki>RFC 7938</nowiki> rfc7938 (see section 5.2.1)

* eBGP will be used for Underlay and Overlay
* eBGP single-hop sessions are established over direct point-to-point links interconnecting the network nodes, no multi-hop or loopback sessions are used, even in the case of multiple links between the same pair of nodes.
* Private Use ASNs from the range 64512-65534 are used to avoid ASN conflicts.
* A single ASN is allocated to all of the Clos topology's Tier 1 devices.
* A unique ASN is allocated to each set of Tier 2 devices in the same cluster.
* A unique ASN is allocated to every Tier 3 device (e.g., ToR) in this topology.

In add, depending of network nodes, it will be interesting to :

* Add fast-failover option for underlay + add link-debounce on links to limit possible impacts in case of link flaps. BFD could be considered in some cases
* Have the best ECMP hashing algorythm on leaf nodes. On tier-1/tier-2 levels hashing will be based on underlay part (trafic inter-vtep), so L3-L4 decision, but some models can retreive the vni-id if I well remember.
* Implement the "multipath relax" since we could have multiple AS paths for a same destination.
* Implement gshut community on underlay (on tier-1/tier-2) or at least policies to prepend underlay routes to avoid some nodes in case of maintenance.

== Thoughts/Points to deepend ==
On leaf nodes, LACP over multiple nodes is feasible within ESI, but impact on incidents/maintenance can vary depending on the vendors.

'''Tests need to be done to check that.'''

Some sysadmin would like to KISS and have client servers connected in active/passive bonding, with eventually single-leaf ports aggregation.

'''Note :''' NH ESI in FIB management on ingress nodes can have some limitations (numbers of installable next-hop, load-balancing, etc. oftenly linked to the node chipset)

Another important engineering point to define at the beginning is : '''Do we authorize multisite L2 propagation or not ?'''

From my point-of-view, the smaller are the L2 segments the better it is. I would suggest Local L2 segments wit anycast gateways provided by Leaf switches, with L3 routing between DCs within EVPN type5 routes sharing only. Nevertheless, some business requirements for endcustomers can keep extended L2 alive.

We have to be careful when implementing Consistent hashing. ex: <nowiki>https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-44/Layer-3/Routing/Equal-Cost-Multipath-Load-Sharing-Hardware-ECMP/</nowiki>

If possible activate it per VRF, not globally, and restrict on dedicated subnets << depends of the vendor, it can be dangerous regarding memory management

Think at configuring the bgp option to not advertise routes to peers before they are installed in FIB.

We can implement leaf roles in some cases, to be able to do "light traffic engineering " on DCI links in some setup.

These roles will be managed within bgp communities added to VTEP /32. Look at the diagram below. Depending of the colour, the eBGP sessions between SupS will prepend or not the VTEP IPs announced to the remote DC

[[File:Fabric-roles.jpg|ohne|mini|1200x1200px]]

<blockquote>'''To be continued'''</blockquote>

== Lab archives ==
<blockquote>Reminder : following config are for lab, so please don't pay attention on typos/lack of security params/unused config. Thanks \o/</blockquote>

=== Credentials ===
<syntaxhighlight lang="bash">
SRLinux : admin/<no_password>

CVX : cumulus/cumulus
</syntaxhighlight>

=== Lab archives ===
layer 2 (vlan 100) + layer 3 (vlan 200) lab, 10/12/2021
[[File:20211008 lab-nokia-cvx-containerlab.zip|ohne|mini]]
<u>If you want to exchange about that lab, those topics, please don't hesitate to contact us</u>

= SONiC Lab =

<span><to be noted that those informations are included into the readme.md file into the archive></span>

== Lab diagram ==
[[File:SONIC_LAB.jpg]]

<div></div>Lab archive [File:Sonic-lab-20201030.zip]

This lab is a forked lab from many found on @, starting from Microsoft one : https://github.com/Azure/SONiC/wiki/SONiC-P4-Software-Switch

The target of this lab is to simulate a simple IP/Fabric in SPINE/LEAF topology. Each Leaf provide local L2 connectivity + L3 gateways to hosts. Flows between hosts from different leaf are L3 routed, in a "node path" LEAF-SPINE-LEAF.

Nothing complicated, but it allows to put a first foot into SONiC world :)

For the moment, as far as we know, MC-LAG + EVPN/VXLAN features are only available on entreprise SONiC versions. Tests to come once physic switches in enterprise version will be available in our lab.

The diagram of the architecture is into the zip file (SONIC_LAB.jpg)

To run the lab you need :

- to have installed docker by using ./install_docker_ovs.sh - to run the ./load_image.sh in order to get the Sonic image

Note : don't change file / folders names of network devices (leafX, spineX)

Once you're ready, please run the lab by launching "./start.sh"

The script :<div></div>
* launches the containers without Docket network features
* create network namespaces
* build linux bridges to interconnect the containers
* configure host IPs
* launch startup sonic scripts into sonic devices
* once devices started, checks + ping tests are run. If all BGP sessions are ok on spine + all ping are fine, the setup works well.
The "start" script has several comments, perhaps without the right vocabulary, so please be tolerant :)

If you want to stop the lab, please launch "./stop.sh"

Below some useful commands to run for sonic containers :

=== <span>run ping</span> ===
sudo docker exec -it <container> ping <ip>

This command triggers ping into the targetted container :
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it leaf1 ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.052 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.034 ms
^C--- 192.168.1.1 ping statistics ---
</syntaxhighlight>

=== <span>check bgp sessions</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip bgp sum"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print the bgp status.
Below, we run it on one spine, you should have 4 sessiosn UP (for the 4 leaves)
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip bgp sum"
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 6409 6411 0 0 0 01:46:46 3
10.1.0.2 4 65102 6406 6410 0 0 0 01:46:44 3
10.1.0.4 4 65103 6407 6410 0 0 0 01:46:44 4
10.1.0.6 4 65104 6406 6412 0 0 0 01:46:46 1
Total number of neighbors 4
</syntaxhighlight>

=== <span>check ip route</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip route"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print its routing table?

<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,
> - selected route, * - FIB route
C>* 10.1.0.0/31 is directly connected, Vlan11
C>* 10.1.0.2/31 is directly connected, Vlan21
C>* 10.1.0.4/31 is directly connected, Vlan31
C>* 10.1.0.6/31 is directly connected, Vlan41
C>* 127.0.0.0/8 is directly connected, lo
B>* 192.168.1.0/24 [20/0] via 10.1.0.0, Vlan11, 01:47:23
B>* 192.168.2.0/24 [20/0] via 10.1.0.2, Vlan21, 01:47:21
B>* 192.168.3.0/24 [20/0] via 10.1.0.4, Vlan31, 01:47:21
B>* 192.168.4.0/24 [20/0] via 10.1.0.6, Vlan41, 01:47:23
</syntaxhighlight>
For more commands check SONiC docs : https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md

To finish, below the output of the start.sh when all goes well<div></div>
plancastre@dockerhost:~/00.Projects/p4-version-pla$ ./start.sh
---------------------------------------------------
Creating the containers: starting
---------------------------------------------------
0b4bbb3c6cfc9895466ea13a9e940d1a176d66f45deee9752e9b4732d735bb03
5ca9486baab3e8a5259062167132a944eee717875d3cc1ef29fe490cba3184c1
46c9c09dba27a32715b9fbe0bc434a9284b9d59317d278df693f425cb669f037
898e6afbcb2977115acbba2a69d069796cfd70fb90228b6c18e09ba4584365f7
eb9b3ef6d14784c9ffe4d9d5df3e921a24559889cb8770d226263aad6761182a
ae5f3b2c628da687eaabb331dc2cd989e71bcb386854e7a0be6ec587df99d49a
122197bffd7566a01ce3b6565fdfb6eb893aab7d5236f14e5a9bbbe95037ced2
a604520a080e2a2184598ad1d915c0c182d16e5185033373e95a4772d8978a67
0c49e18d78b2aa7c8ef8e937b60285aa56ec0bf4a8530226285270a085e11d5c
eb66336b8c6bd8333f1f4f685369e6cac973b817c35bb7126ac4d7ba72ef71bf
---------------------------------------------------
Retreiving PID from docker containers
---------------------------------------------------
---------------------------------------------------
Creating network namespaces for containers
---------------------------------------------------
List of created namespaces
ns_host4
ns_host3
ns_host2
ns_host1
ns_spine2
ns_spine1
ns_leaf4
ns_leaf3
ns_leaf2
ns_leaf1
---------------------------------------------------
Creating network connectivity between containers
---------------------------------------------------
---------------------------------------------------
Now assigning IPs to hosts (host1 to host4)
---------------------------------------------------
---------------------------------------------------
Configuration of containers is now finished
---------------------------------------------------
---------------------------------------------------
Now booting switches, please wait 2 minutes for switches to load
---------------------------------------------------
---------------------------------------------------
2 minutes period is finished
---------------------------------------------------
---------------------------------------------------
Now fixing the iptables
---------------------------------------------------
---------------------------------------------------
Backend actions are now finished
---------------------------------------------------
---------------------------------------------------
Now we'll check in 15 sec that BGP peers are all Up
---------------------------------------------------
---------------------------------------------------
SPINE1 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 94 97 0 0 0 00:01:28 4
10.1.0.2 4 65102 95 97 0 0 0 00:01:28 5
10.1.0.4 4 65103 95 97 0 0 0 00:01:28 5
10.1.0.6 4 65104 95 97 0 0 0 00:01:28 5
Total number of neighbors 4
---------------------------------------------------
SPINE2 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.2.0.1, local AS number 65002
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.2.0.0 4 65101 93 96 0 0 0 00:01:29 2
10.2.0.2 4 65102 94 100 0 0 0 00:01:32 1
10.2.0.4 4 65103 94 100 0 0 0 00:01:32 1
10.2.0.6 4 65104 94 100 0 0 0 00:01:32 1
Total number of neighbors 4
---------------------------------------------------
Now if all is ok, we'll do some ping tests
---------------------------------------------------
---------------------------------------------------
host1 ping its gateway
---------------------------------------------------
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=17.6 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=9.14 ms
--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 9.145/13.390/17.636/4.247 ms
---------------------------------------------------
host2 ping its gateway
---------------------------------------------------
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=23.6 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=6.21 ms
--- 192.168.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 6.218/14.950/23.683/8.733 ms
---------------------------------------------------
host3 ping its gateway
---------------------------------------------------
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=18.8 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=12.7 ms
--- 192.168.3.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.716/15.764/18.812/3.048 ms
---------------------------------------------------
host4 ping its gateway
---------------------------------------------------
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.
64 bytes from 192.168.4.1: icmp_seq=1 ttl=64 time=17.7 ms
64 bytes from 192.168.4.1: icmp_seq=2 ttl=64 time=4.46 ms
--- 192.168.4.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.464/11.107/17.750/6.643 ms
---------------------------------------------------
host1 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=16.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.049/17.253/19.532/1.615 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=15.5 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=14.3 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.312/15.592/16.939/1.083 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=20.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=16.8 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.048/17.871/20.739/2.058 ms
---------------------------------------------------
host2 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=16.6 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.222/16.583/16.924/0.286 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.6 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=18.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=26.8 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.620/20.549/26.887/4.524 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=33.8 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=27.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=20.2 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 20.216/27.259/33.818/5.566 ms
---------------------------------------------------
host3 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=14.5 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=15.5 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.535/16.549/19.541/2.162 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.4 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=14.6 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=18.2 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.643/16.458/18.278/1.491 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=15.3 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=14.1 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=13.1 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 13.171/14.234/15.334/0.883 ms
---------------------------------------------------
host4 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=21.9 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=21.3 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=14.6 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 14.619/19.284/21.932/3.310 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=14.8 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=18.4 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=14.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.148/15.786/18.409/1.879 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=15.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=18.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 15.117/16.788/18.962/1.612 ms
---------------------------------------------------
If bot spine have all BGP sessions and all ping succeeded, all works well
Congrats ! please check readme file to find some details
---------------------------------------------------

File:Sonic-lab-20201030.zip

2021-10-29T08:11:18Z

PierreLancastre:

Nesevo Virtual Labs

2021-10-29T08:10:02Z

PierreLancastre: /* check ip route */

[[File:nesevo-labs-v2.png]]

= Nokia/Cumulus Multi-Vendor Lab =

The targets of this lab are to :

* mix Nokia SR-Linux and Cumulus VXLAN nodes, in spine or leaf roles
* make this environment work for different kinds of setup : cross-DC L2, differentiated L3 gateways per leaf per VRF, anycast gateway cross or intra DC, etc.
* spot compatibility issues if existing

To reach those goals, we'll use the Nokia lab tool "containerlab" https://containerlab.srlinux.dev/, labs from https://clabs.netdevops.me/ helped a lot. Many thanks to the contributors.

We'll need a Linux host with Docker installed on it + kvm virtualization enhancements activated (important if your host is a VM). If virtualization optimization is not ok, you won't be able to launch the most of devices.

Installation details are referenced there : https://containerlab.srlinux.dev/install/

Once installed, you'll need to register the different NOS images you'll use. Cumulus image is present on a public registrery, but Nokia/Arista/Juniper are not, you'll need a vendor support account to retreive them. Once downloaded (.tgz file typically) you can "docker load" the image :

For private images :<syntaxhighlight lang="bash">
sudo docker load -i 21.6.1-235.tar.xz /* (Nokia) */
sudo docker load -i cEOS64-lab-4.26.1F.tar /* (Arista) */
</syntaxhighlight>For public images :<syntaxhighlight lang="bash">
sudo docker pull networkop/cx:4.3.0 /* (Cumulus) */
</syntaxhighlight>

To verify your installed images, you can run "sudo docker images"<syntaxhighlight lang="bash">
plancastre@containerlab:~/00.LAB/senss$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
networkop/cx 4.3.0 a40541ed15ba 2 weeks ago 733MB
srlinux 21.6.1-235 e29aed30e83f 3 weeks ago 1.14GB
networkop/ignite dev dcc5eff29355 4 weeks ago 37.9MB
networkop/kernel 4.19 e794d8875f83 3 months ago 88MB
frrouting/frr v7.5.1 c3e13a4c5918 3 months ago 123MB

</syntaxhighlight>Once you've got all your images, to run the lab, you'll need a .yml file defining the topology to run.

In our lab, we've started with the 5-clos lab file from Nokia site, which we've forked.

We can see there is a branch "kinds" where we can define templates of nodes, in order to not rewrite whole parameters when creating nodes. For the images, it's important to take the repository name + tag.

'''Note:''' If the tag is "latest" it is not needed.

All supported "kinds" are referenced there : https://containerlab.srlinux.dev/manual/kinds/kinds/

'''Note:''' yml is indentation sensitive, so please respect the "double space" used to define hierarchy. Below the lab diagram + the yml file associated

== Infos on nodes ==

=== L3 infos ===
{| class="wikitable"
|+
!Device
!OS
!ASN
!VTEP IP
!swp1 or eth1/1
!swp2 or eth1/2
!swp3 or eth1/3
!SVI
|-
|client1
|Linux
|<none>
|<none>
|vlan100: 192.168.1.1/24
vlan200: 192.168.2.1/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client2
|Linux
|<none>
|<none>
|vlan100: 192.168.1.2/24
vlan200: 192.168.2.2/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client3
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.1/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|client4
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.2/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|leaf1
|CVX
|65011
|10.255.255.5/32
|10.254.254.8/31
|10.254.254.10/31
|trunk vlan 100,200
|vlan200 : 192.168.2.254/24
|-
|leaf2
|SRLinux
|65012
|10.255.255.6/32
|10.254.254.12/31
|10.254.254.14/31
|trunk vlan 100,200
|irb0.200 : 192.168.2.254/24
|-
|leaf3
|CVX
|65013
|10.255.255.7/32
|10.254.254.22/31
|10.254.254.24/31
|trunk vlan 100,200
|vlan200 : 192.168.200.254/24
|-
|leaf4
|SRLinux
|65014
|10.255.255.8/32
|10.254.254.26/31
|10.254.254.28/31
|trunk vlan 100,200
|irb0.200 : 192.168.200.254/24
|-
|spine1
|CVX
|65001
|10.255.255.1/32
|10.254.254.9/31
|10.254.254.13/31
|10.254.254.0/31
|<none>
|-
|spine2
|CVX
|65001
|10.255.255.2/32
|10.254.254.11/31
|10.254.254.15/31
|10.254.254.16/31
|<none>
|-
|spine3
|SRLinux
|65002
|10.255.255.3/32
|10.254.254.23/31
|10.254.254.27/31
|10.254.254.18/31
|<none>
|-
|spine4
|SRLinux
|65002
|10.255.255.4/32
|10.254.254.25/31
|10.254.254.29/31
|10.254.254.20/31
|<none>
|-
|superspine1
|CVX
|65003
|10.255.255.9/32
|10.254.254.1/31
|10.254.254.19/31
|<none>
|<none>
|-
|superspine2
|SRLinux
|65003
|10.255.255.10/32
|10.254.254.17/31
|10.254.254.21/31
|<none>
|<none>
|}

=== VXLAN/BGP infos ===
Note : in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>
{| class="wikitable"
|+
!service
!vlan id
!route-target
!VNI
!Commentaires
|-
|L2 cross-DC
|100
|target:65000:100
|100
|the 4 clients are in the same broadcast domain / subnet
|-
|L3VPN cros-DC
|200
|target:65000:200
target:65000:10200
|200 (for L2)
10200 (for routing)
|the 4 clients are separated in 2 different subnets. Traffic is routed within dedicated VNI
To be noted that in the current setup VMTO is not implemented, so cross DC traffic is not optimized.
|}

== Lab diagram ==
[[File:Lab-srl-diagram.png|ohne|mini|600x600px]]

== yml lab file ==
<syntaxhighlight lang="yaml">
# topology documentation: http://containerlab.srlinux.dev/lab-examples/min-5clos/
name: lab-srl-cum

topology:
kinds:
srl:
image: srlinux:21.6.1-235
linux:
image: ghcr.io/hellt/network-multitool
cvx:
image: networkop/cx:4.3.0
runtime: docker
nodes:
leaf1:
kind: cvx
leaf2:
kind: srl
type: ixrd2
leaf3:
kind: cvx
leaf4:
kind: srl
type: ixrd2

spine1:
kind: cvx
spine2:
kind: cvx
spine3:
kind: srl
type: ixr6
spine4:
kind: srl
type: ixr6

superspine1:
kind: cvx
superspine2:
kind: srl
type: ixr6

client1:
kind: linux
client2:
kind: linux
client3:
kind: linux
client4:
kind: linux

links:
# leaf to spine links POD1
- endpoints: ["leaf1:swp1", "spine1:swp1"]
- endpoints: ["leaf1:swp2", "spine2:swp1"]
- endpoints: ["leaf2:e1-1", "spine1:swp2"]
- endpoints: ["leaf2:e1-2", "spine2:swp2"]

# spine to superspine links POD1
- endpoints: ["spine1:swp3", "superspine1:swp1"]
- endpoints: ["spine2:swp3", "superspine2:e1-1"]

# leaf to spine links POD2
- endpoints: ["leaf3:swp1", "spine3:e1-1"]
- endpoints: ["leaf3:swp2", "spine4:e1-1"]
- endpoints: ["leaf4:e1-1", "spine3:e1-2"]
- endpoints: ["leaf4:e1-2", "spine4:e1-2"]

# spine to superspine links POD2
- endpoints: ["spine3:e1-3", "superspine1:swp2"]
- endpoints: ["spine4:e1-3", "superspine2:e1-2"]

# client connection links
- endpoints: ["client1:eth1", "leaf1:swp3"]
- endpoints: ["client2:eth1", "leaf2:e1-3"]
- endpoints: ["client3:eth1", "leaf3:swp3"]
- endpoints: ["client4:eth1", "leaf4:e1-3"]

</syntaxhighlight>

Once the nodes are defined, we have to define the needed connections between the nodes.

Each endpoint is defined as '<nowiki/>'''<nowiki/>'<node_name>:<interface_name><nowiki>''</nowiki>'''

When all is ok, you can launch in suduoer the topology.

For example :<syntaxhighlight lang="bash">
sudo containerlab deploy --topo topo.clab.yml --reconfigure

Note: The option "reconfigure" is used there in order to not have to destroy instances when updating topology/relaunching.
</syntaxhighlight>Another info regarding the yml file, it's possible to quickly generate files with the '''"containerlab gen -n 3tier --nodes 4,2,1"''' command for example. https://containerlab.srlinux.dev/cmd/generate/

To be noted that the "tier-x" notion in containerlab is inversed regarding the rfc 7938 (tier1 is superspine level into the rfc, tier1 is leaf level in nokia tool)

At the end of the topology launch, you can retreive the details :<syntaxhighlight lang="yaml">
INFO[0011] Writing /etc/hosts file
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| # | Name | Container ID | Image | Kind | Group | State | IPv4 Address | IPv6 Address |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| 1 | clab-lab-srl-cum-client1 | 25e2bbf6b13f | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
| 2 | clab-lab-srl-cum-client2 | 17d82d5a6e21 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.4/24 | 2001:172:20:20::4/64 |
| 3 | clab-lab-srl-cum-client3 | f8d78527c1b1 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.6/24 | 2001:172:20:20::6/64 |
| 4 | clab-lab-srl-cum-client4 | 913a0172d0c5 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 5 | clab-lab-srl-cum-leaf1 | 9ffc334af2b1 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.9/24 | 2001:172:20:20::9/64 |
| 6 | clab-lab-srl-cum-leaf2 | 7d3eb855d749 | srlinux:21.6.1-235 | srl | | running | 172.20.20.14/24 | 2001:172:20:20::e/64 |
| 7 | clab-lab-srl-cum-leaf3 | e0d36ffc606a | networkop/cx:4.3.0 | cvx | | running | 172.20.20.7/24 | 2001:172:20:20::7/64 |
| 8 | clab-lab-srl-cum-leaf4 | e6753b1d099e | srlinux:21.6.1-235 | srl | | running | 172.20.20.12/24 | 2001:172:20:20::c/64 |
| 9 | clab-lab-srl-cum-spine1 | a7e63993d052 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.5/24 | 2001:172:20:20::5/64 |
| 10 | clab-lab-srl-cum-spine2 | 3124250d5b75 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
| 11 | clab-lab-srl-cum-spine3 | 5e300ca3f66f | srlinux:21.6.1-235 | srl | | running | 172.20.20.11/24 | 2001:172:20:20::b/64 |
| 12 | clab-lab-srl-cum-spine4 | 0a59e60a596b | srlinux:21.6.1-235 | srl | | running | 172.20.20.13/24 | 2001:172:20:20::d/64 |
| 13 | clab-lab-srl-cum-superspine1 | 3d9935413fc2 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.8/24 | 2001:172:20:20::8/64 |
| 14 | clab-lab-srl-cum-superspine2 | 5bc58d423e2c | srlinux:21.6.1-235 | srl | | running | 172.20.20.15/24 | 2001:172:20:20::f/64 |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
</syntaxhighlight>The containerlab program creates the updates into the host file + generates an ansible inventory file.

There are different ways to connect onto the docker instances, within docker exec commands or SSH directly the IP address related above.

For example :<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-spine1 bash
</syntaxhighlight>-->>> allows to launch a bash on the cumulus device (then you run the different net commands)<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-leaf2 sr_cli
</syntaxhighlight>-->>> allows to launch the cli on a SRLinux device

To end this setup preparation, the containerlab tool integrates an option to launch a webserver showing the lab topology in graph + node details

By default, it's listening on port 50080 of the host

Then, you can access to the following web page showing the topology + lab infos

[[File:Lab graph tool.jpg|ohne|mini|900x900px]]

Finally, if you want to stop a lab, please run '''<nowiki>''</nowiki>containerlab destroy --topo <topology>.yml<nowiki>''</nowiki>'''

At this step, nodes are running, connections are done, but configurations are empty.

'''So, it's time to configure them :)'''

Configuration files are attached to this blog. Once validated, they are exportable and can be called by the tool when launching the lab. Below how to do it :

Firstable, once you ve finished your "hand made config", you can run the following command to save it :<syntaxhighlight lang="bash">
containerlab save -t quickstart.clab.yml
</syntaxhighlight>'''To be noted it's only working for Nokia nodes in this lab'''. CVX configurations will have to be called by another method (or pushed manually)

It will create one folder per node into the lab folder which gives the following tree :
[[File:Topology-srl.jpg|ohne|mini]]

'''Note''': For Cumulus, you can apparently retrieve configurations files within adding the following parameters into lab config file. '''To be tested.''' some lab samples here : https://github.com/hellt/clabs/blob/main/labs/cvx04/symm-mh.clab.yml

Now we have the lab running, below the different commands to check that all is working as expected : <syntaxhighlight lang="yaml">
-------------------------------------------
Check L2 tables ///
-------------------------------------------

Nokia :

A:leaf2# show network-instance mac-vrf-100 bridge-table mac-table all
----------------------------------------------------------------------------------------------------------------------------------------------------------
Mac-table of network instance mac-vrf-100
----------------------------------------------------------------------------------------------------------------------------------------------------------
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
| Address | Destination | Dest Index | Type | Active | Aging | Last Update |
+====================+=========================================+============+=============+=========+========+=========================================+
| AA:C1:AB:0C:60:D6 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T13:06:17.000Z |
| | vtep:10.255.255.8 vni:100 | 67 | | | | |
| AA:C1:AB:23:31:04 | ethernet-1/3.100 | 2 | learnt | true | 300 | 2021-10-08T13:06:10.000Z |
| AA:C1:AB:4B:FF:71 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:55:28.000Z |
| | vtep:10.255.255.5 vni:100 | 60 | | | | |
| AA:C1:AB:B5:79:50 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:43:37.000Z |
| | vtep:10.255.255.7 vni:100 | 64 | | | | |
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 0 Total 0 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 1 Total 1 Active
Total Evpn Macs : 3 Total 3 Active
Total Evpn static Macs : 0 Total 0 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 4 Total 4 Active
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : type learnt matches the local learnt mac, evpn are learnt within VXLAN

Cumulus :

root@leaf1-cum-dc1:/# net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
100 bridge swp3 aa:c1:ab:4b:ff:71 00:00:02
100 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:17:21
100 bridge vni100 22:2d:67:f1:bb:26 permanent 02:17:21
100 bridge vni100 aa:c1:ab:0c:60:d6 extern_learn <1 sec
100 bridge vni100 aa:c1:ab:23:31:04 extern_learn 00:00:07
100 bridge vni100 aa:c1:ab:b5:79:50 extern_learn 00:13:28

root@leaf1-cum-dc1:/# net show bridge macs vlan 200

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
200 bridge swp3 aa:c1:ab:4b:ff:71 00:08:46
200 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:16:36
200 bridge vni200 00:00:5e:00:01:01 static sticky 00:14:26
200 bridge vni200 1e:0d:aa:46:53:b1 permanent 02:16:36
200 bridge vni200 02:77:6b:ff:00:41 static sticky 01:00:52
200 bridge vni200 02:f8:b8:ff:00:41 static sticky 00:14:26
200 bridge vni200 aa:c1:ab:b5:79:50 extern_learn 00:11:51
200 bridge vni10200 02:77:6b:ff:00:00 extern_learn 00:53:02
200 bridge vni10200 02:f8:b8:ff:00:00 extern_learn 00:14:26
200 bridge vni10200 86:5e:aa:ca:50:be permanent 02:16:36

Note : the macs with permanent flag are local interfaces, physical ou logic (vtep)
Note2 : extern_learn mac are the one learned via the VXLAN infra (and mapped to the vnixxx interface)

-------------------------------------------
Check arp tables ///
-------------------------------------------

Nokia :

:leaf2# show arpnd arp-entries
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| ethernet-1/1 | 0 | 10.254.254.13 | dynamic | AA:C1:AB:59:65:EA | 3 hours from now |
| ethernet-1/2 | 0 | 10.254.254.15 | dynamic | AA:C1:AB:CB:F8:8A | 3 hours from now |
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
| mgmt0 | 0 | 172.20.20.1 | dynamic | 02:42:63:9E:74:03 | 2 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 5 (0 static, 5 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : on Nokia you have to sort the entries basing on subinterface and the retrieve the L3VPN attached

A:leaf2# show arpnd arp-entries subinterface 200
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 2 (0 static, 2 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Cumulus :

Here we retrieve all arp infos linked to evpn

root@leaf1-cum-dc1:/# net show evpn arp-cache vni all
VNI 100 #ARP (IPv4 and IPv6, local and remote) 2

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.1.3 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0
192.168.1.1 local active aa:c1:ab:4b:ff:71 0/0

VNI 200 #ARP (IPv4 and IPv6, local and remote) 4

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.2.1 local active aa:c1:ab:4b:ff:71 0/0
192.168.200.254 remote active 00:00:5e:00:01:01 10.255.255.8 0/0
192.168.2.254 remote active 00:00:5e:00:01:01 10.255.255.6 0/0
192.168.200.1 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0

VNI 400 #ARP (IPv4 and IPv6, local and remote) 0

-------------------------------------------
Check bgp sessions' status - underlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep ipv4
| default | 10.254.254.19 | udl-superspine | S | 65003 | established | 0d:0h:22m:44s | ipv4-unicast | [13/4/4] |
| default | 10.254.254.22 | udl-leaf | S | 65013 | established | 0d:0h:22m:52s | ipv4-unicast | [13/1/4] |

Cumulus :

root@leaf3:/# net show bgp ipv4 unicast summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 13
RIB entries 25, using 4800 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.254.254.23 4 65002 466 403 0 0 0 00:19:34 4 13
10.254.254.25 4 65002 434 355 0 0 0 00:17:10 8 13

-------------------------------------------
Check bgp sessions' status - overlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep evpn
| default | 10.255.255.7 | ovl-leaf | S | 65013 | established | 0d:0h:24m:14s| evpn | [15/0/11] |
| default | 10.255.255.9 | ovl-superspine | S | 65003 | established | 0d:0h:24m:7s | evpn | [11/0/4] |

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.255.255.3 4 65002 5618 4742 0 0 0 03:53:35 19 27
10.255.255.4 4 65002 5565 4694 0 0 0 03:51:11 19 27

-------------------------------------------
Check EVPN L2 tables///
-------------------------------------------

Nokia :

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn route type 2
BGP table version is 17, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:100
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
Route Distinguisher: 10.255.255.5:200
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
Route Distinguisher: 10.255.255.6:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
Route Distinguisher: 10.255.255.7:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100
Route Distinguisher: 10.255.255.7:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.200.1]
10.255.255.7 32768 i
ET:8 RT:65000:200
Route Distinguisher: 10.255.255.8:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC

and if we focus on our vni 100

root@leaf3-cum-dc1:/# net show bgp evpn route vni 100 type 2
BGP table version is 48, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100

The additional route type "MAC+IP" is optional and is useful/mandatory when you do anycast irb.

-------------------------------------------
Check tables EVPN L3 ///
-------------------------------------------

Nokia :

--{ + running }--[ ]--
A:leaf2# show network-instance ip-vrf-200 route-table all
-----------------------------------------------------------------------------------------------------------------------------------------------------------
IPv4 Unicast route table of network instance ip-vrf-200
-----------------------------------------------------------------------------------------------------------------------------------------------------------
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
| Prefix | ID | Route Type | Route Owner | Best/Fib- | Metric | Pref | Next-hop | Next-hop |
| | | | | status(slot) | | | (Type) | Interface |
+=======================+=======+============+======================+======================+==========+=========+==============+==============+
| 192.168.2.0/24 | 9 | local | net_inst_mgr | True/success | 0 | 0 | 192.168.2.25 | irb0.200 |
| | | | | | | | 4 (direct) | |
| 192.168.2.254/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (extract) | |
| 192.168.2.255/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (broadcast) | |
| 192.168.200.0/24 | 0 | bgp-evpn | bgp_evpn_mgr | True/success | 0 | 170 | 10.255.255.8 | None |
| | | | | | | | (indirect/vx | |
| | | | | | | | lan) | |
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
-----------------------------------------------------------------------------------------------------------------------------------------------------------
4 IPv4 routes total
4 IPv4 prefixes with active routes
0 IPv4 prefixes with active ECMP routes
-----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--
A:leaf2#

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf1-cum-dc1:/# net show route vrf ip-vrf-200
show ip route vrf ip-vrf-200
=============================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF ip-vrf-200:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 04:48:42
B 192.168.2.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 04:27:26
C>* 192.168.2.0/24 is directly connected, vlan200, 04:48:42
B>* 192.168.200.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:48:49

root@leaf1-cum-dc1:/# net show bgp evpn route type 5
BGP table version is 3, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.6:10200
* [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
*> [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
Route Distinguisher: 10.255.255.8:10200
*> [5]:[0]:[24]:[192.168.200.0]
10.255.255.8 0 65001 65003 65002 65014 i
RT:65000:10200 ET:8 Rmac:02:f8:b8:ff:00:00

-------------------------------------------
Check arp hosts ///
-------------------------------------------

From client #1 /

bash-5.0# arp
Address HWtype HWaddress Flags Mask Iface
192.168.1.3 ether aa:c1:ab:b5:79:50 C eth1.100
192.168.1.4 ether aa:c1:ab:0c:60:d6 C eth1.100
192.168.2.2 ether aa:c1:ab:23:31:04 C eth1.200
192.168.2.254 ether aa:c1:ab:4f:f2:e7 C eth1.200
192.168.1.2 ether aa:c1:ab:23:31:04 C eth1.100
containerlab.local ether 02:42:63:9e:74:03 C eth0

-------------------------------------------
Ping ///
-------------------------------------------

From client #1 /

L2 service ping (vlan 100) ----------------------

bash-5.0# ping -c 1 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.812 ms

--- 192.168.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.812/0.812/0.812/0.000 MsgRcvd
bash-5.0# ping -c 1 192.168.1.3
PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_seq=1 ttl=64 time=1.40 ms

--- 192.168.1.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.403/1.403/1.403/0.000 ms
bash-5.0# ping -c 1 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=1.62 ms

--- 192.168.1.4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.619/1.619/1.619/0.000 ms

L3 service ping (vlan 200)

bash-5.0# ping -c 1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.053 ms

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms
bash-5.0# ^C
bash-5.0# ping -c 1 192.168.2.2
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.611 ms

--- 192.168.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.611/0.611/0.611/0.000 ms

-------------------------------------------
Traceroute ///
-------------------------------------------

L3 service traceroute (vlan 200)

bash-5.0# traceroute 192.168.200.1
traceroute to 192.168.200.1 (192.168.200.1), 30 hops max, 46 byte packets
1 192.168.2.254 (192.168.2.254) 0.012 ms 0.010 ms 0.007 ms
2 192.168.200.254 (192.168.200.254) 2.751 ms * 2.549 ms
3 192.168.200.1 (192.168.200.1) 2.084 ms 1.805 ms 1.761 ms

</syntaxhighlight>To come back on the setup, we wanted to implement solutions inspired by the <nowiki>RFC 7938</nowiki> rfc7938 (see section 5.2.1)

* eBGP will be used for Underlay and Overlay
* eBGP single-hop sessions are established over direct point-to-point links interconnecting the network nodes, no multi-hop or loopback sessions are used, even in the case of multiple links between the same pair of nodes.
* Private Use ASNs from the range 64512-65534 are used to avoid ASN conflicts.
* A single ASN is allocated to all of the Clos topology's Tier 1 devices.
* A unique ASN is allocated to each set of Tier 2 devices in the same cluster.
* A unique ASN is allocated to every Tier 3 device (e.g., ToR) in this topology.

In add, depending of network nodes, it will be interesting to :

* Add fast-failover option for underlay + add link-debounce on links to limit possible impacts in case of link flaps. BFD could be considered in some cases
* Have the best ECMP hashing algorythm on leaf nodes. On tier-1/tier-2 levels hashing will be based on underlay part (trafic inter-vtep), so L3-L4 decision, but some models can retreive the vni-id if I well remember.
* Implement the "multipath relax" since we could have multiple AS paths for a same destination.
* Implement gshut community on underlay (on tier-1/tier-2) or at least policies to prepend underlay routes to avoid some nodes in case of maintenance.

== Thoughts/Points to deepend ==
On leaf nodes, LACP over multiple nodes is feasible within ESI, but impact on incidents/maintenance can vary depending on the vendors.

'''Tests need to be done to check that.'''

Some sysadmin would like to KISS and have client servers connected in active/passive bonding, with eventually single-leaf ports aggregation.

'''Note :''' NH ESI in FIB management on ingress nodes can have some limitations (numbers of installable next-hop, load-balancing, etc. oftenly linked to the node chipset)

Another important engineering point to define at the beginning is : '''Do we authorize multisite L2 propagation or not ?'''

From my point-of-view, the smaller are the L2 segments the better it is. I would suggest Local L2 segments wit anycast gateways provided by Leaf switches, with L3 routing between DCs within EVPN type5 routes sharing only. Nevertheless, some business requirements for endcustomers can keep extended L2 alive.

We have to be careful when implementing Consistent hashing. ex: <nowiki>https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-44/Layer-3/Routing/Equal-Cost-Multipath-Load-Sharing-Hardware-ECMP/</nowiki>

If possible activate it per VRF, not globally, and restrict on dedicated subnets << depends of the vendor, it can be dangerous regarding memory management

Think at configuring the bgp option to not advertise routes to peers before they are installed in FIB.

We can implement leaf roles in some cases, to be able to do "light traffic engineering " on DCI links in some setup.

These roles will be managed within bgp communities added to VTEP /32. Look at the diagram below. Depending of the colour, the eBGP sessions between SupS will prepend or not the VTEP IPs announced to the remote DC

[[File:Fabric-roles.jpg|ohne|mini|1200x1200px]]

<blockquote>'''To be continued'''</blockquote>

== Lab archives ==
<blockquote>Reminder : following config are for lab, so please don't pay attention on typos/lack of security params/unused config. Thanks \o/</blockquote>

=== Credentials ===
<syntaxhighlight lang="bash">
SRLinux : admin/<no_password>

CVX : cumulus/cumulus
</syntaxhighlight>

=== Lab archives ===
layer 2 (vlan 100) + layer 3 (vlan 200) lab, 10/12/2021
[[File:20211008 lab-nokia-cvx-containerlab.zip|ohne|mini]]
<u>If you want to exchange about that lab, those topics, please don't hesitate to contact us</u>

= SONiC Lab =

<span><to be noted that those informations are included into the readme.md file into the archive></span>

== Lab diagram ==
[[File:SONIC_LAB.jpg]]

<div></div>Lab archive [http://wiki.intra.nesevo.com/uploads/LAB/sonic-lab-20201030.zip Attach:sonic-lab-20201030.zip]

This lab is a forked lab from many found on @, starting from Microsoft one : https://github.com/Azure/SONiC/wiki/SONiC-P4-Software-Switch

The target of this lab is to simulate a simple IP/Fabric in SPINE/LEAF topology. Each Leaf provide local L2 connectivity + L3 gateways to hosts. Flows between hosts from different leaf are L3 routed, in a "node path" LEAF-SPINE-LEAF.

Nothing complicated, but it allows to put a first foot into SONiC world :)

For the moment, as far as we know, MC-LAG + EVPN/VXLAN features are only available on entreprise SONiC versions. Tests to come once physic switches in enterprise version will be available in our lab.

The diagram of the architecture is into the zip file (SONIC_LAB.jpg)

To run the lab you need :

- to have installed docker by using ./install_docker_ovs.sh - to run the ./load_image.sh in order to get the Sonic image

Note : don't change file / folders names of network devices (leafX, spineX)

Once you're ready, please run the lab by launching "./start.sh"

The script :<div></div>
* launches the containers without Docket network features
* create network namespaces
* build linux bridges to interconnect the containers
* configure host IPs
* launch startup sonic scripts into sonic devices
* once devices started, checks + ping tests are run. If all BGP sessions are ok on spine + all ping are fine, the setup works well.
The "start" script has several comments, perhaps without the right vocabulary, so please be tolerant :)

If you want to stop the lab, please launch "./stop.sh"

Below some useful commands to run for sonic containers :

=== <span>run ping</span> ===
sudo docker exec -it <container> ping <ip>

This command triggers ping into the targetted container :
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it leaf1 ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.052 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.034 ms
^C--- 192.168.1.1 ping statistics ---
</syntaxhighlight>

=== <span>check bgp sessions</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip bgp sum"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print the bgp status.
Below, we run it on one spine, you should have 4 sessiosn UP (for the 4 leaves)
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip bgp sum"
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 6409 6411 0 0 0 01:46:46 3
10.1.0.2 4 65102 6406 6410 0 0 0 01:46:44 3
10.1.0.4 4 65103 6407 6410 0 0 0 01:46:44 4
10.1.0.6 4 65104 6406 6412 0 0 0 01:46:46 1
Total number of neighbors 4
</syntaxhighlight>

=== <span>check ip route</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip route"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print its routing table?

<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,
> - selected route, * - FIB route
C>* 10.1.0.0/31 is directly connected, Vlan11
C>* 10.1.0.2/31 is directly connected, Vlan21
C>* 10.1.0.4/31 is directly connected, Vlan31
C>* 10.1.0.6/31 is directly connected, Vlan41
C>* 127.0.0.0/8 is directly connected, lo
B>* 192.168.1.0/24 [20/0] via 10.1.0.0, Vlan11, 01:47:23
B>* 192.168.2.0/24 [20/0] via 10.1.0.2, Vlan21, 01:47:21
B>* 192.168.3.0/24 [20/0] via 10.1.0.4, Vlan31, 01:47:21
B>* 192.168.4.0/24 [20/0] via 10.1.0.6, Vlan41, 01:47:23
</syntaxhighlight>
For more commands check SONiC docs : https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md

To finish, below the output of the start.sh when all goes well<div></div>
plancastre@dockerhost:~/00.Projects/p4-version-pla$ ./start.sh
---------------------------------------------------
Creating the containers: starting
---------------------------------------------------
0b4bbb3c6cfc9895466ea13a9e940d1a176d66f45deee9752e9b4732d735bb03
5ca9486baab3e8a5259062167132a944eee717875d3cc1ef29fe490cba3184c1
46c9c09dba27a32715b9fbe0bc434a9284b9d59317d278df693f425cb669f037
898e6afbcb2977115acbba2a69d069796cfd70fb90228b6c18e09ba4584365f7
eb9b3ef6d14784c9ffe4d9d5df3e921a24559889cb8770d226263aad6761182a
ae5f3b2c628da687eaabb331dc2cd989e71bcb386854e7a0be6ec587df99d49a
122197bffd7566a01ce3b6565fdfb6eb893aab7d5236f14e5a9bbbe95037ced2
a604520a080e2a2184598ad1d915c0c182d16e5185033373e95a4772d8978a67
0c49e18d78b2aa7c8ef8e937b60285aa56ec0bf4a8530226285270a085e11d5c
eb66336b8c6bd8333f1f4f685369e6cac973b817c35bb7126ac4d7ba72ef71bf
---------------------------------------------------
Retreiving PID from docker containers
---------------------------------------------------
---------------------------------------------------
Creating network namespaces for containers
---------------------------------------------------
List of created namespaces
ns_host4
ns_host3
ns_host2
ns_host1
ns_spine2
ns_spine1
ns_leaf4
ns_leaf3
ns_leaf2
ns_leaf1
---------------------------------------------------
Creating network connectivity between containers
---------------------------------------------------
---------------------------------------------------
Now assigning IPs to hosts (host1 to host4)
---------------------------------------------------
---------------------------------------------------
Configuration of containers is now finished
---------------------------------------------------
---------------------------------------------------
Now booting switches, please wait 2 minutes for switches to load
---------------------------------------------------
---------------------------------------------------
2 minutes period is finished
---------------------------------------------------
---------------------------------------------------
Now fixing the iptables
---------------------------------------------------
---------------------------------------------------
Backend actions are now finished
---------------------------------------------------
---------------------------------------------------
Now we'll check in 15 sec that BGP peers are all Up
---------------------------------------------------
---------------------------------------------------
SPINE1 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 94 97 0 0 0 00:01:28 4
10.1.0.2 4 65102 95 97 0 0 0 00:01:28 5
10.1.0.4 4 65103 95 97 0 0 0 00:01:28 5
10.1.0.6 4 65104 95 97 0 0 0 00:01:28 5
Total number of neighbors 4
---------------------------------------------------
SPINE2 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.2.0.1, local AS number 65002
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.2.0.0 4 65101 93 96 0 0 0 00:01:29 2
10.2.0.2 4 65102 94 100 0 0 0 00:01:32 1
10.2.0.4 4 65103 94 100 0 0 0 00:01:32 1
10.2.0.6 4 65104 94 100 0 0 0 00:01:32 1
Total number of neighbors 4
---------------------------------------------------
Now if all is ok, we'll do some ping tests
---------------------------------------------------
---------------------------------------------------
host1 ping its gateway
---------------------------------------------------
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=17.6 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=9.14 ms
--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 9.145/13.390/17.636/4.247 ms
---------------------------------------------------
host2 ping its gateway
---------------------------------------------------
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=23.6 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=6.21 ms
--- 192.168.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 6.218/14.950/23.683/8.733 ms
---------------------------------------------------
host3 ping its gateway
---------------------------------------------------
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=18.8 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=12.7 ms
--- 192.168.3.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.716/15.764/18.812/3.048 ms
---------------------------------------------------
host4 ping its gateway
---------------------------------------------------
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.
64 bytes from 192.168.4.1: icmp_seq=1 ttl=64 time=17.7 ms
64 bytes from 192.168.4.1: icmp_seq=2 ttl=64 time=4.46 ms
--- 192.168.4.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.464/11.107/17.750/6.643 ms
---------------------------------------------------
host1 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=16.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.049/17.253/19.532/1.615 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=15.5 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=14.3 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.312/15.592/16.939/1.083 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=20.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=16.8 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.048/17.871/20.739/2.058 ms
---------------------------------------------------
host2 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=16.6 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.222/16.583/16.924/0.286 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.6 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=18.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=26.8 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.620/20.549/26.887/4.524 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=33.8 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=27.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=20.2 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 20.216/27.259/33.818/5.566 ms
---------------------------------------------------
host3 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=14.5 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=15.5 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.535/16.549/19.541/2.162 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.4 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=14.6 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=18.2 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.643/16.458/18.278/1.491 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=15.3 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=14.1 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=13.1 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 13.171/14.234/15.334/0.883 ms
---------------------------------------------------
host4 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=21.9 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=21.3 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=14.6 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 14.619/19.284/21.932/3.310 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=14.8 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=18.4 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=14.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.148/15.786/18.409/1.879 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=15.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=18.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 15.117/16.788/18.962/1.612 ms
---------------------------------------------------
If bot spine have all BGP sessions and all ping succeeded, all works well
Congrats ! please check readme file to find some details
---------------------------------------------------

Nesevo Virtual Labs

2021-10-29T08:09:13Z

PierreLancastre: /* check bgp sessions */

[[File:nesevo-labs-v2.png]]

= Nokia/Cumulus Multi-Vendor Lab =

The targets of this lab are to :

* mix Nokia SR-Linux and Cumulus VXLAN nodes, in spine or leaf roles
* make this environment work for different kinds of setup : cross-DC L2, differentiated L3 gateways per leaf per VRF, anycast gateway cross or intra DC, etc.
* spot compatibility issues if existing

To reach those goals, we'll use the Nokia lab tool "containerlab" https://containerlab.srlinux.dev/, labs from https://clabs.netdevops.me/ helped a lot. Many thanks to the contributors.

We'll need a Linux host with Docker installed on it + kvm virtualization enhancements activated (important if your host is a VM). If virtualization optimization is not ok, you won't be able to launch the most of devices.

Installation details are referenced there : https://containerlab.srlinux.dev/install/

Once installed, you'll need to register the different NOS images you'll use. Cumulus image is present on a public registrery, but Nokia/Arista/Juniper are not, you'll need a vendor support account to retreive them. Once downloaded (.tgz file typically) you can "docker load" the image :

For private images :<syntaxhighlight lang="bash">
sudo docker load -i 21.6.1-235.tar.xz /* (Nokia) */
sudo docker load -i cEOS64-lab-4.26.1F.tar /* (Arista) */
</syntaxhighlight>For public images :<syntaxhighlight lang="bash">
sudo docker pull networkop/cx:4.3.0 /* (Cumulus) */
</syntaxhighlight>

To verify your installed images, you can run "sudo docker images"<syntaxhighlight lang="bash">
plancastre@containerlab:~/00.LAB/senss$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
networkop/cx 4.3.0 a40541ed15ba 2 weeks ago 733MB
srlinux 21.6.1-235 e29aed30e83f 3 weeks ago 1.14GB
networkop/ignite dev dcc5eff29355 4 weeks ago 37.9MB
networkop/kernel 4.19 e794d8875f83 3 months ago 88MB
frrouting/frr v7.5.1 c3e13a4c5918 3 months ago 123MB

</syntaxhighlight>Once you've got all your images, to run the lab, you'll need a .yml file defining the topology to run.

In our lab, we've started with the 5-clos lab file from Nokia site, which we've forked.

We can see there is a branch "kinds" where we can define templates of nodes, in order to not rewrite whole parameters when creating nodes. For the images, it's important to take the repository name + tag.

'''Note:''' If the tag is "latest" it is not needed.

All supported "kinds" are referenced there : https://containerlab.srlinux.dev/manual/kinds/kinds/

'''Note:''' yml is indentation sensitive, so please respect the "double space" used to define hierarchy. Below the lab diagram + the yml file associated

== Infos on nodes ==

=== L3 infos ===
{| class="wikitable"
|+
!Device
!OS
!ASN
!VTEP IP
!swp1 or eth1/1
!swp2 or eth1/2
!swp3 or eth1/3
!SVI
|-
|client1
|Linux
|<none>
|<none>
|vlan100: 192.168.1.1/24
vlan200: 192.168.2.1/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client2
|Linux
|<none>
|<none>
|vlan100: 192.168.1.2/24
vlan200: 192.168.2.2/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client3
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.1/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|client4
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.2/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|leaf1
|CVX
|65011
|10.255.255.5/32
|10.254.254.8/31
|10.254.254.10/31
|trunk vlan 100,200
|vlan200 : 192.168.2.254/24
|-
|leaf2
|SRLinux
|65012
|10.255.255.6/32
|10.254.254.12/31
|10.254.254.14/31
|trunk vlan 100,200
|irb0.200 : 192.168.2.254/24
|-
|leaf3
|CVX
|65013
|10.255.255.7/32
|10.254.254.22/31
|10.254.254.24/31
|trunk vlan 100,200
|vlan200 : 192.168.200.254/24
|-
|leaf4
|SRLinux
|65014
|10.255.255.8/32
|10.254.254.26/31
|10.254.254.28/31
|trunk vlan 100,200
|irb0.200 : 192.168.200.254/24
|-
|spine1
|CVX
|65001
|10.255.255.1/32
|10.254.254.9/31
|10.254.254.13/31
|10.254.254.0/31
|<none>
|-
|spine2
|CVX
|65001
|10.255.255.2/32
|10.254.254.11/31
|10.254.254.15/31
|10.254.254.16/31
|<none>
|-
|spine3
|SRLinux
|65002
|10.255.255.3/32
|10.254.254.23/31
|10.254.254.27/31
|10.254.254.18/31
|<none>
|-
|spine4
|SRLinux
|65002
|10.255.255.4/32
|10.254.254.25/31
|10.254.254.29/31
|10.254.254.20/31
|<none>
|-
|superspine1
|CVX
|65003
|10.255.255.9/32
|10.254.254.1/31
|10.254.254.19/31
|<none>
|<none>
|-
|superspine2
|SRLinux
|65003
|10.255.255.10/32
|10.254.254.17/31
|10.254.254.21/31
|<none>
|<none>
|}

=== VXLAN/BGP infos ===
Note : in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>
{| class="wikitable"
|+
!service
!vlan id
!route-target
!VNI
!Commentaires
|-
|L2 cross-DC
|100
|target:65000:100
|100
|the 4 clients are in the same broadcast domain / subnet
|-
|L3VPN cros-DC
|200
|target:65000:200
target:65000:10200
|200 (for L2)
10200 (for routing)
|the 4 clients are separated in 2 different subnets. Traffic is routed within dedicated VNI
To be noted that in the current setup VMTO is not implemented, so cross DC traffic is not optimized.
|}

== Lab diagram ==
[[File:Lab-srl-diagram.png|ohne|mini|600x600px]]

== yml lab file ==
<syntaxhighlight lang="yaml">
# topology documentation: http://containerlab.srlinux.dev/lab-examples/min-5clos/
name: lab-srl-cum

topology:
kinds:
srl:
image: srlinux:21.6.1-235
linux:
image: ghcr.io/hellt/network-multitool
cvx:
image: networkop/cx:4.3.0
runtime: docker
nodes:
leaf1:
kind: cvx
leaf2:
kind: srl
type: ixrd2
leaf3:
kind: cvx
leaf4:
kind: srl
type: ixrd2

spine1:
kind: cvx
spine2:
kind: cvx
spine3:
kind: srl
type: ixr6
spine4:
kind: srl
type: ixr6

superspine1:
kind: cvx
superspine2:
kind: srl
type: ixr6

client1:
kind: linux
client2:
kind: linux
client3:
kind: linux
client4:
kind: linux

links:
# leaf to spine links POD1
- endpoints: ["leaf1:swp1", "spine1:swp1"]
- endpoints: ["leaf1:swp2", "spine2:swp1"]
- endpoints: ["leaf2:e1-1", "spine1:swp2"]
- endpoints: ["leaf2:e1-2", "spine2:swp2"]

# spine to superspine links POD1
- endpoints: ["spine1:swp3", "superspine1:swp1"]
- endpoints: ["spine2:swp3", "superspine2:e1-1"]

# leaf to spine links POD2
- endpoints: ["leaf3:swp1", "spine3:e1-1"]
- endpoints: ["leaf3:swp2", "spine4:e1-1"]
- endpoints: ["leaf4:e1-1", "spine3:e1-2"]
- endpoints: ["leaf4:e1-2", "spine4:e1-2"]

# spine to superspine links POD2
- endpoints: ["spine3:e1-3", "superspine1:swp2"]
- endpoints: ["spine4:e1-3", "superspine2:e1-2"]

# client connection links
- endpoints: ["client1:eth1", "leaf1:swp3"]
- endpoints: ["client2:eth1", "leaf2:e1-3"]
- endpoints: ["client3:eth1", "leaf3:swp3"]
- endpoints: ["client4:eth1", "leaf4:e1-3"]

</syntaxhighlight>

Once the nodes are defined, we have to define the needed connections between the nodes.

Each endpoint is defined as '<nowiki/>'''<nowiki/>'<node_name>:<interface_name><nowiki>''</nowiki>'''

When all is ok, you can launch in suduoer the topology.

For example :<syntaxhighlight lang="bash">
sudo containerlab deploy --topo topo.clab.yml --reconfigure

Note: The option "reconfigure" is used there in order to not have to destroy instances when updating topology/relaunching.
</syntaxhighlight>Another info regarding the yml file, it's possible to quickly generate files with the '''"containerlab gen -n 3tier --nodes 4,2,1"''' command for example. https://containerlab.srlinux.dev/cmd/generate/

To be noted that the "tier-x" notion in containerlab is inversed regarding the rfc 7938 (tier1 is superspine level into the rfc, tier1 is leaf level in nokia tool)

At the end of the topology launch, you can retreive the details :<syntaxhighlight lang="yaml">
INFO[0011] Writing /etc/hosts file
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| # | Name | Container ID | Image | Kind | Group | State | IPv4 Address | IPv6 Address |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| 1 | clab-lab-srl-cum-client1 | 25e2bbf6b13f | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
| 2 | clab-lab-srl-cum-client2 | 17d82d5a6e21 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.4/24 | 2001:172:20:20::4/64 |
| 3 | clab-lab-srl-cum-client3 | f8d78527c1b1 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.6/24 | 2001:172:20:20::6/64 |
| 4 | clab-lab-srl-cum-client4 | 913a0172d0c5 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 5 | clab-lab-srl-cum-leaf1 | 9ffc334af2b1 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.9/24 | 2001:172:20:20::9/64 |
| 6 | clab-lab-srl-cum-leaf2 | 7d3eb855d749 | srlinux:21.6.1-235 | srl | | running | 172.20.20.14/24 | 2001:172:20:20::e/64 |
| 7 | clab-lab-srl-cum-leaf3 | e0d36ffc606a | networkop/cx:4.3.0 | cvx | | running | 172.20.20.7/24 | 2001:172:20:20::7/64 |
| 8 | clab-lab-srl-cum-leaf4 | e6753b1d099e | srlinux:21.6.1-235 | srl | | running | 172.20.20.12/24 | 2001:172:20:20::c/64 |
| 9 | clab-lab-srl-cum-spine1 | a7e63993d052 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.5/24 | 2001:172:20:20::5/64 |
| 10 | clab-lab-srl-cum-spine2 | 3124250d5b75 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
| 11 | clab-lab-srl-cum-spine3 | 5e300ca3f66f | srlinux:21.6.1-235 | srl | | running | 172.20.20.11/24 | 2001:172:20:20::b/64 |
| 12 | clab-lab-srl-cum-spine4 | 0a59e60a596b | srlinux:21.6.1-235 | srl | | running | 172.20.20.13/24 | 2001:172:20:20::d/64 |
| 13 | clab-lab-srl-cum-superspine1 | 3d9935413fc2 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.8/24 | 2001:172:20:20::8/64 |
| 14 | clab-lab-srl-cum-superspine2 | 5bc58d423e2c | srlinux:21.6.1-235 | srl | | running | 172.20.20.15/24 | 2001:172:20:20::f/64 |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
</syntaxhighlight>The containerlab program creates the updates into the host file + generates an ansible inventory file.

There are different ways to connect onto the docker instances, within docker exec commands or SSH directly the IP address related above.

For example :<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-spine1 bash
</syntaxhighlight>-->>> allows to launch a bash on the cumulus device (then you run the different net commands)<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-leaf2 sr_cli
</syntaxhighlight>-->>> allows to launch the cli on a SRLinux device

To end this setup preparation, the containerlab tool integrates an option to launch a webserver showing the lab topology in graph + node details

By default, it's listening on port 50080 of the host

Then, you can access to the following web page showing the topology + lab infos

[[File:Lab graph tool.jpg|ohne|mini|900x900px]]

Finally, if you want to stop a lab, please run '''<nowiki>''</nowiki>containerlab destroy --topo <topology>.yml<nowiki>''</nowiki>'''

At this step, nodes are running, connections are done, but configurations are empty.

'''So, it's time to configure them :)'''

Configuration files are attached to this blog. Once validated, they are exportable and can be called by the tool when launching the lab. Below how to do it :

Firstable, once you ve finished your "hand made config", you can run the following command to save it :<syntaxhighlight lang="bash">
containerlab save -t quickstart.clab.yml
</syntaxhighlight>'''To be noted it's only working for Nokia nodes in this lab'''. CVX configurations will have to be called by another method (or pushed manually)

It will create one folder per node into the lab folder which gives the following tree :
[[File:Topology-srl.jpg|ohne|mini]]

'''Note''': For Cumulus, you can apparently retrieve configurations files within adding the following parameters into lab config file. '''To be tested.''' some lab samples here : https://github.com/hellt/clabs/blob/main/labs/cvx04/symm-mh.clab.yml

Now we have the lab running, below the different commands to check that all is working as expected : <syntaxhighlight lang="yaml">
-------------------------------------------
Check L2 tables ///
-------------------------------------------

Nokia :

A:leaf2# show network-instance mac-vrf-100 bridge-table mac-table all
----------------------------------------------------------------------------------------------------------------------------------------------------------
Mac-table of network instance mac-vrf-100
----------------------------------------------------------------------------------------------------------------------------------------------------------
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
| Address | Destination | Dest Index | Type | Active | Aging | Last Update |
+====================+=========================================+============+=============+=========+========+=========================================+
| AA:C1:AB:0C:60:D6 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T13:06:17.000Z |
| | vtep:10.255.255.8 vni:100 | 67 | | | | |
| AA:C1:AB:23:31:04 | ethernet-1/3.100 | 2 | learnt | true | 300 | 2021-10-08T13:06:10.000Z |
| AA:C1:AB:4B:FF:71 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:55:28.000Z |
| | vtep:10.255.255.5 vni:100 | 60 | | | | |
| AA:C1:AB:B5:79:50 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:43:37.000Z |
| | vtep:10.255.255.7 vni:100 | 64 | | | | |
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 0 Total 0 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 1 Total 1 Active
Total Evpn Macs : 3 Total 3 Active
Total Evpn static Macs : 0 Total 0 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 4 Total 4 Active
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : type learnt matches the local learnt mac, evpn are learnt within VXLAN

Cumulus :

root@leaf1-cum-dc1:/# net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
100 bridge swp3 aa:c1:ab:4b:ff:71 00:00:02
100 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:17:21
100 bridge vni100 22:2d:67:f1:bb:26 permanent 02:17:21
100 bridge vni100 aa:c1:ab:0c:60:d6 extern_learn <1 sec
100 bridge vni100 aa:c1:ab:23:31:04 extern_learn 00:00:07
100 bridge vni100 aa:c1:ab:b5:79:50 extern_learn 00:13:28

root@leaf1-cum-dc1:/# net show bridge macs vlan 200

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
200 bridge swp3 aa:c1:ab:4b:ff:71 00:08:46
200 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:16:36
200 bridge vni200 00:00:5e:00:01:01 static sticky 00:14:26
200 bridge vni200 1e:0d:aa:46:53:b1 permanent 02:16:36
200 bridge vni200 02:77:6b:ff:00:41 static sticky 01:00:52
200 bridge vni200 02:f8:b8:ff:00:41 static sticky 00:14:26
200 bridge vni200 aa:c1:ab:b5:79:50 extern_learn 00:11:51
200 bridge vni10200 02:77:6b:ff:00:00 extern_learn 00:53:02
200 bridge vni10200 02:f8:b8:ff:00:00 extern_learn 00:14:26
200 bridge vni10200 86:5e:aa:ca:50:be permanent 02:16:36

Note : the macs with permanent flag are local interfaces, physical ou logic (vtep)
Note2 : extern_learn mac are the one learned via the VXLAN infra (and mapped to the vnixxx interface)

-------------------------------------------
Check arp tables ///
-------------------------------------------

Nokia :

:leaf2# show arpnd arp-entries
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| ethernet-1/1 | 0 | 10.254.254.13 | dynamic | AA:C1:AB:59:65:EA | 3 hours from now |
| ethernet-1/2 | 0 | 10.254.254.15 | dynamic | AA:C1:AB:CB:F8:8A | 3 hours from now |
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
| mgmt0 | 0 | 172.20.20.1 | dynamic | 02:42:63:9E:74:03 | 2 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 5 (0 static, 5 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : on Nokia you have to sort the entries basing on subinterface and the retrieve the L3VPN attached

A:leaf2# show arpnd arp-entries subinterface 200
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 2 (0 static, 2 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Cumulus :

Here we retrieve all arp infos linked to evpn

root@leaf1-cum-dc1:/# net show evpn arp-cache vni all
VNI 100 #ARP (IPv4 and IPv6, local and remote) 2

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.1.3 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0
192.168.1.1 local active aa:c1:ab:4b:ff:71 0/0

VNI 200 #ARP (IPv4 and IPv6, local and remote) 4

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.2.1 local active aa:c1:ab:4b:ff:71 0/0
192.168.200.254 remote active 00:00:5e:00:01:01 10.255.255.8 0/0
192.168.2.254 remote active 00:00:5e:00:01:01 10.255.255.6 0/0
192.168.200.1 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0

VNI 400 #ARP (IPv4 and IPv6, local and remote) 0

-------------------------------------------
Check bgp sessions' status - underlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep ipv4
| default | 10.254.254.19 | udl-superspine | S | 65003 | established | 0d:0h:22m:44s | ipv4-unicast | [13/4/4] |
| default | 10.254.254.22 | udl-leaf | S | 65013 | established | 0d:0h:22m:52s | ipv4-unicast | [13/1/4] |

Cumulus :

root@leaf3:/# net show bgp ipv4 unicast summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 13
RIB entries 25, using 4800 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.254.254.23 4 65002 466 403 0 0 0 00:19:34 4 13
10.254.254.25 4 65002 434 355 0 0 0 00:17:10 8 13

-------------------------------------------
Check bgp sessions' status - overlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep evpn
| default | 10.255.255.7 | ovl-leaf | S | 65013 | established | 0d:0h:24m:14s| evpn | [15/0/11] |
| default | 10.255.255.9 | ovl-superspine | S | 65003 | established | 0d:0h:24m:7s | evpn | [11/0/4] |

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.255.255.3 4 65002 5618 4742 0 0 0 03:53:35 19 27
10.255.255.4 4 65002 5565 4694 0 0 0 03:51:11 19 27

-------------------------------------------
Check EVPN L2 tables///
-------------------------------------------

Nokia :

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn route type 2
BGP table version is 17, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:100
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
Route Distinguisher: 10.255.255.5:200
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
Route Distinguisher: 10.255.255.6:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
Route Distinguisher: 10.255.255.7:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100
Route Distinguisher: 10.255.255.7:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.200.1]
10.255.255.7 32768 i
ET:8 RT:65000:200
Route Distinguisher: 10.255.255.8:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC

and if we focus on our vni 100

root@leaf3-cum-dc1:/# net show bgp evpn route vni 100 type 2
BGP table version is 48, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100

The additional route type "MAC+IP" is optional and is useful/mandatory when you do anycast irb.

-------------------------------------------
Check tables EVPN L3 ///
-------------------------------------------

Nokia :

--{ + running }--[ ]--
A:leaf2# show network-instance ip-vrf-200 route-table all
-----------------------------------------------------------------------------------------------------------------------------------------------------------
IPv4 Unicast route table of network instance ip-vrf-200
-----------------------------------------------------------------------------------------------------------------------------------------------------------
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
| Prefix | ID | Route Type | Route Owner | Best/Fib- | Metric | Pref | Next-hop | Next-hop |
| | | | | status(slot) | | | (Type) | Interface |
+=======================+=======+============+======================+======================+==========+=========+==============+==============+
| 192.168.2.0/24 | 9 | local | net_inst_mgr | True/success | 0 | 0 | 192.168.2.25 | irb0.200 |
| | | | | | | | 4 (direct) | |
| 192.168.2.254/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (extract) | |
| 192.168.2.255/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (broadcast) | |
| 192.168.200.0/24 | 0 | bgp-evpn | bgp_evpn_mgr | True/success | 0 | 170 | 10.255.255.8 | None |
| | | | | | | | (indirect/vx | |
| | | | | | | | lan) | |
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
-----------------------------------------------------------------------------------------------------------------------------------------------------------
4 IPv4 routes total
4 IPv4 prefixes with active routes
0 IPv4 prefixes with active ECMP routes
-----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--
A:leaf2#

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf1-cum-dc1:/# net show route vrf ip-vrf-200
show ip route vrf ip-vrf-200
=============================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF ip-vrf-200:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 04:48:42
B 192.168.2.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 04:27:26
C>* 192.168.2.0/24 is directly connected, vlan200, 04:48:42
B>* 192.168.200.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:48:49

root@leaf1-cum-dc1:/# net show bgp evpn route type 5
BGP table version is 3, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.6:10200
* [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
*> [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
Route Distinguisher: 10.255.255.8:10200
*> [5]:[0]:[24]:[192.168.200.0]
10.255.255.8 0 65001 65003 65002 65014 i
RT:65000:10200 ET:8 Rmac:02:f8:b8:ff:00:00

-------------------------------------------
Check arp hosts ///
-------------------------------------------

From client #1 /

bash-5.0# arp
Address HWtype HWaddress Flags Mask Iface
192.168.1.3 ether aa:c1:ab:b5:79:50 C eth1.100
192.168.1.4 ether aa:c1:ab:0c:60:d6 C eth1.100
192.168.2.2 ether aa:c1:ab:23:31:04 C eth1.200
192.168.2.254 ether aa:c1:ab:4f:f2:e7 C eth1.200
192.168.1.2 ether aa:c1:ab:23:31:04 C eth1.100
containerlab.local ether 02:42:63:9e:74:03 C eth0

-------------------------------------------
Ping ///
-------------------------------------------

From client #1 /

L2 service ping (vlan 100) ----------------------

bash-5.0# ping -c 1 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.812 ms

--- 192.168.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.812/0.812/0.812/0.000 MsgRcvd
bash-5.0# ping -c 1 192.168.1.3
PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_seq=1 ttl=64 time=1.40 ms

--- 192.168.1.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.403/1.403/1.403/0.000 ms
bash-5.0# ping -c 1 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=1.62 ms

--- 192.168.1.4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.619/1.619/1.619/0.000 ms

L3 service ping (vlan 200)

bash-5.0# ping -c 1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.053 ms

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms
bash-5.0# ^C
bash-5.0# ping -c 1 192.168.2.2
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.611 ms

--- 192.168.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.611/0.611/0.611/0.000 ms

-------------------------------------------
Traceroute ///
-------------------------------------------

L3 service traceroute (vlan 200)

bash-5.0# traceroute 192.168.200.1
traceroute to 192.168.200.1 (192.168.200.1), 30 hops max, 46 byte packets
1 192.168.2.254 (192.168.2.254) 0.012 ms 0.010 ms 0.007 ms
2 192.168.200.254 (192.168.200.254) 2.751 ms * 2.549 ms
3 192.168.200.1 (192.168.200.1) 2.084 ms 1.805 ms 1.761 ms

</syntaxhighlight>To come back on the setup, we wanted to implement solutions inspired by the <nowiki>RFC 7938</nowiki> rfc7938 (see section 5.2.1)

* eBGP will be used for Underlay and Overlay
* eBGP single-hop sessions are established over direct point-to-point links interconnecting the network nodes, no multi-hop or loopback sessions are used, even in the case of multiple links between the same pair of nodes.
* Private Use ASNs from the range 64512-65534 are used to avoid ASN conflicts.
* A single ASN is allocated to all of the Clos topology's Tier 1 devices.
* A unique ASN is allocated to each set of Tier 2 devices in the same cluster.
* A unique ASN is allocated to every Tier 3 device (e.g., ToR) in this topology.

In add, depending of network nodes, it will be interesting to :

* Add fast-failover option for underlay + add link-debounce on links to limit possible impacts in case of link flaps. BFD could be considered in some cases
* Have the best ECMP hashing algorythm on leaf nodes. On tier-1/tier-2 levels hashing will be based on underlay part (trafic inter-vtep), so L3-L4 decision, but some models can retreive the vni-id if I well remember.
* Implement the "multipath relax" since we could have multiple AS paths for a same destination.
* Implement gshut community on underlay (on tier-1/tier-2) or at least policies to prepend underlay routes to avoid some nodes in case of maintenance.

== Thoughts/Points to deepend ==
On leaf nodes, LACP over multiple nodes is feasible within ESI, but impact on incidents/maintenance can vary depending on the vendors.

'''Tests need to be done to check that.'''

Some sysadmin would like to KISS and have client servers connected in active/passive bonding, with eventually single-leaf ports aggregation.

'''Note :''' NH ESI in FIB management on ingress nodes can have some limitations (numbers of installable next-hop, load-balancing, etc. oftenly linked to the node chipset)

Another important engineering point to define at the beginning is : '''Do we authorize multisite L2 propagation or not ?'''

From my point-of-view, the smaller are the L2 segments the better it is. I would suggest Local L2 segments wit anycast gateways provided by Leaf switches, with L3 routing between DCs within EVPN type5 routes sharing only. Nevertheless, some business requirements for endcustomers can keep extended L2 alive.

We have to be careful when implementing Consistent hashing. ex: <nowiki>https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-44/Layer-3/Routing/Equal-Cost-Multipath-Load-Sharing-Hardware-ECMP/</nowiki>

If possible activate it per VRF, not globally, and restrict on dedicated subnets << depends of the vendor, it can be dangerous regarding memory management

Think at configuring the bgp option to not advertise routes to peers before they are installed in FIB.

We can implement leaf roles in some cases, to be able to do "light traffic engineering " on DCI links in some setup.

These roles will be managed within bgp communities added to VTEP /32. Look at the diagram below. Depending of the colour, the eBGP sessions between SupS will prepend or not the VTEP IPs announced to the remote DC

[[File:Fabric-roles.jpg|ohne|mini|1200x1200px]]

<blockquote>'''To be continued'''</blockquote>

== Lab archives ==
<blockquote>Reminder : following config are for lab, so please don't pay attention on typos/lack of security params/unused config. Thanks \o/</blockquote>

=== Credentials ===
<syntaxhighlight lang="bash">
SRLinux : admin/<no_password>

CVX : cumulus/cumulus
</syntaxhighlight>

=== Lab archives ===
layer 2 (vlan 100) + layer 3 (vlan 200) lab, 10/12/2021
[[File:20211008 lab-nokia-cvx-containerlab.zip|ohne|mini]]
<u>If you want to exchange about that lab, those topics, please don't hesitate to contact us</u>

= SONiC Lab =

<span><to be noted that those informations are included into the readme.md file into the archive></span>

== Lab diagram ==
[[File:SONIC_LAB.jpg]]

<div></div>Lab archive [http://wiki.intra.nesevo.com/uploads/LAB/sonic-lab-20201030.zip Attach:sonic-lab-20201030.zip]

This lab is a forked lab from many found on @, starting from Microsoft one : https://github.com/Azure/SONiC/wiki/SONiC-P4-Software-Switch

The target of this lab is to simulate a simple IP/Fabric in SPINE/LEAF topology. Each Leaf provide local L2 connectivity + L3 gateways to hosts. Flows between hosts from different leaf are L3 routed, in a "node path" LEAF-SPINE-LEAF.

Nothing complicated, but it allows to put a first foot into SONiC world :)

For the moment, as far as we know, MC-LAG + EVPN/VXLAN features are only available on entreprise SONiC versions. Tests to come once physic switches in enterprise version will be available in our lab.

The diagram of the architecture is into the zip file (SONIC_LAB.jpg)

To run the lab you need :

- to have installed docker by using ./install_docker_ovs.sh - to run the ./load_image.sh in order to get the Sonic image

Note : don't change file / folders names of network devices (leafX, spineX)

Once you're ready, please run the lab by launching "./start.sh"

The script :<div></div>
* launches the containers without Docket network features
* create network namespaces
* build linux bridges to interconnect the containers
* configure host IPs
* launch startup sonic scripts into sonic devices
* once devices started, checks + ping tests are run. If all BGP sessions are ok on spine + all ping are fine, the setup works well.
The "start" script has several comments, perhaps without the right vocabulary, so please be tolerant :)

If you want to stop the lab, please launch "./stop.sh"

Below some useful commands to run for sonic containers :

=== <span>run ping</span> ===
sudo docker exec -it <container> ping <ip>

This command triggers ping into the targetted container :
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it leaf1 ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.052 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.034 ms
^C--- 192.168.1.1 ping statistics ---
</syntaxhighlight>

=== <span>check bgp sessions</span> ===
<syntaxhighlight lang="bash">
sudo docker exec -it <container> vtysh -c "show ip bgp sum"
</syntaxhighlight>
This command sollicitates the SONiC shell of the targeted container, in order to print the bgp status.
Below, we run it on one spine, you should have 4 sessiosn UP (for the 4 leaves)
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip bgp sum"
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 6409 6411 0 0 0 01:46:46 3
10.1.0.2 4 65102 6406 6410 0 0 0 01:46:44 3
10.1.0.4 4 65103 6407 6410 0 0 0 01:46:44 4
10.1.0.6 4 65104 6406 6412 0 0 0 01:46:46 1
Total number of neighbors 4
</syntaxhighlight>

=== <span>check ip route</span> ===
<div></div>
sudo docker exec -it <container> vtysh -c "show ip route"
This command sollicitates the SONiC shell of the targeted container, in order to print its routing table?
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,
> - selected route, * - FIB route
C>* 10.1.0.0/31 is directly connected, Vlan11
C>* 10.1.0.2/31 is directly connected, Vlan21
C>* 10.1.0.4/31 is directly connected, Vlan31
C>* 10.1.0.6/31 is directly connected, Vlan41
C>* 127.0.0.0/8 is directly connected, lo
B>* 192.168.1.0/24 [20/0] via 10.1.0.0, Vlan11, 01:47:23
B>* 192.168.2.0/24 [20/0] via 10.1.0.2, Vlan21, 01:47:21
B>* 192.168.3.0/24 [20/0] via 10.1.0.4, Vlan31, 01:47:21
B>* 192.168.4.0/24 [20/0] via 10.1.0.6, Vlan41, 01:47:23

For more commands check SONiC docs : https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md

To finish, below the output of the start.sh when all goes well<div></div>
plancastre@dockerhost:~/00.Projects/p4-version-pla$ ./start.sh
---------------------------------------------------
Creating the containers: starting
---------------------------------------------------
0b4bbb3c6cfc9895466ea13a9e940d1a176d66f45deee9752e9b4732d735bb03
5ca9486baab3e8a5259062167132a944eee717875d3cc1ef29fe490cba3184c1
46c9c09dba27a32715b9fbe0bc434a9284b9d59317d278df693f425cb669f037
898e6afbcb2977115acbba2a69d069796cfd70fb90228b6c18e09ba4584365f7
eb9b3ef6d14784c9ffe4d9d5df3e921a24559889cb8770d226263aad6761182a
ae5f3b2c628da687eaabb331dc2cd989e71bcb386854e7a0be6ec587df99d49a
122197bffd7566a01ce3b6565fdfb6eb893aab7d5236f14e5a9bbbe95037ced2
a604520a080e2a2184598ad1d915c0c182d16e5185033373e95a4772d8978a67
0c49e18d78b2aa7c8ef8e937b60285aa56ec0bf4a8530226285270a085e11d5c
eb66336b8c6bd8333f1f4f685369e6cac973b817c35bb7126ac4d7ba72ef71bf
---------------------------------------------------
Retreiving PID from docker containers
---------------------------------------------------
---------------------------------------------------
Creating network namespaces for containers
---------------------------------------------------
List of created namespaces
ns_host4
ns_host3
ns_host2
ns_host1
ns_spine2
ns_spine1
ns_leaf4
ns_leaf3
ns_leaf2
ns_leaf1
---------------------------------------------------
Creating network connectivity between containers
---------------------------------------------------
---------------------------------------------------
Now assigning IPs to hosts (host1 to host4)
---------------------------------------------------
---------------------------------------------------
Configuration of containers is now finished
---------------------------------------------------
---------------------------------------------------
Now booting switches, please wait 2 minutes for switches to load
---------------------------------------------------
---------------------------------------------------
2 minutes period is finished
---------------------------------------------------
---------------------------------------------------
Now fixing the iptables
---------------------------------------------------
---------------------------------------------------
Backend actions are now finished
---------------------------------------------------
---------------------------------------------------
Now we'll check in 15 sec that BGP peers are all Up
---------------------------------------------------
---------------------------------------------------
SPINE1 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 94 97 0 0 0 00:01:28 4
10.1.0.2 4 65102 95 97 0 0 0 00:01:28 5
10.1.0.4 4 65103 95 97 0 0 0 00:01:28 5
10.1.0.6 4 65104 95 97 0 0 0 00:01:28 5
Total number of neighbors 4
---------------------------------------------------
SPINE2 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.2.0.1, local AS number 65002
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.2.0.0 4 65101 93 96 0 0 0 00:01:29 2
10.2.0.2 4 65102 94 100 0 0 0 00:01:32 1
10.2.0.4 4 65103 94 100 0 0 0 00:01:32 1
10.2.0.6 4 65104 94 100 0 0 0 00:01:32 1
Total number of neighbors 4
---------------------------------------------------
Now if all is ok, we'll do some ping tests
---------------------------------------------------
---------------------------------------------------
host1 ping its gateway
---------------------------------------------------
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=17.6 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=9.14 ms
--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 9.145/13.390/17.636/4.247 ms
---------------------------------------------------
host2 ping its gateway
---------------------------------------------------
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=23.6 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=6.21 ms
--- 192.168.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 6.218/14.950/23.683/8.733 ms
---------------------------------------------------
host3 ping its gateway
---------------------------------------------------
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=18.8 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=12.7 ms
--- 192.168.3.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.716/15.764/18.812/3.048 ms
---------------------------------------------------
host4 ping its gateway
---------------------------------------------------
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.
64 bytes from 192.168.4.1: icmp_seq=1 ttl=64 time=17.7 ms
64 bytes from 192.168.4.1: icmp_seq=2 ttl=64 time=4.46 ms
--- 192.168.4.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.464/11.107/17.750/6.643 ms
---------------------------------------------------
host1 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=16.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.049/17.253/19.532/1.615 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=15.5 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=14.3 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.312/15.592/16.939/1.083 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=20.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=16.8 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.048/17.871/20.739/2.058 ms
---------------------------------------------------
host2 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=16.6 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.222/16.583/16.924/0.286 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.6 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=18.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=26.8 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.620/20.549/26.887/4.524 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=33.8 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=27.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=20.2 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 20.216/27.259/33.818/5.566 ms
---------------------------------------------------
host3 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=14.5 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=15.5 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.535/16.549/19.541/2.162 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.4 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=14.6 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=18.2 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.643/16.458/18.278/1.491 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=15.3 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=14.1 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=13.1 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 13.171/14.234/15.334/0.883 ms
---------------------------------------------------
host4 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=21.9 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=21.3 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=14.6 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 14.619/19.284/21.932/3.310 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=14.8 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=18.4 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=14.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.148/15.786/18.409/1.879 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=15.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=18.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 15.117/16.788/18.962/1.612 ms
---------------------------------------------------
If bot spine have all BGP sessions and all ping succeeded, all works well
Congrats ! please check readme file to find some details
---------------------------------------------------

Nesevo Virtual Labs

2021-10-29T08:08:25Z

PierreLancastre: /* run ping */

[[File:nesevo-labs-v2.png]]

= Nokia/Cumulus Multi-Vendor Lab =

The targets of this lab are to :

* mix Nokia SR-Linux and Cumulus VXLAN nodes, in spine or leaf roles
* make this environment work for different kinds of setup : cross-DC L2, differentiated L3 gateways per leaf per VRF, anycast gateway cross or intra DC, etc.
* spot compatibility issues if existing

To reach those goals, we'll use the Nokia lab tool "containerlab" https://containerlab.srlinux.dev/, labs from https://clabs.netdevops.me/ helped a lot. Many thanks to the contributors.

We'll need a Linux host with Docker installed on it + kvm virtualization enhancements activated (important if your host is a VM). If virtualization optimization is not ok, you won't be able to launch the most of devices.

Installation details are referenced there : https://containerlab.srlinux.dev/install/

Once installed, you'll need to register the different NOS images you'll use. Cumulus image is present on a public registrery, but Nokia/Arista/Juniper are not, you'll need a vendor support account to retreive them. Once downloaded (.tgz file typically) you can "docker load" the image :

For private images :<syntaxhighlight lang="bash">
sudo docker load -i 21.6.1-235.tar.xz /* (Nokia) */
sudo docker load -i cEOS64-lab-4.26.1F.tar /* (Arista) */
</syntaxhighlight>For public images :<syntaxhighlight lang="bash">
sudo docker pull networkop/cx:4.3.0 /* (Cumulus) */
</syntaxhighlight>

To verify your installed images, you can run "sudo docker images"<syntaxhighlight lang="bash">
plancastre@containerlab:~/00.LAB/senss$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
networkop/cx 4.3.0 a40541ed15ba 2 weeks ago 733MB
srlinux 21.6.1-235 e29aed30e83f 3 weeks ago 1.14GB
networkop/ignite dev dcc5eff29355 4 weeks ago 37.9MB
networkop/kernel 4.19 e794d8875f83 3 months ago 88MB
frrouting/frr v7.5.1 c3e13a4c5918 3 months ago 123MB

</syntaxhighlight>Once you've got all your images, to run the lab, you'll need a .yml file defining the topology to run.

In our lab, we've started with the 5-clos lab file from Nokia site, which we've forked.

We can see there is a branch "kinds" where we can define templates of nodes, in order to not rewrite whole parameters when creating nodes. For the images, it's important to take the repository name + tag.

'''Note:''' If the tag is "latest" it is not needed.

All supported "kinds" are referenced there : https://containerlab.srlinux.dev/manual/kinds/kinds/

'''Note:''' yml is indentation sensitive, so please respect the "double space" used to define hierarchy. Below the lab diagram + the yml file associated

== Infos on nodes ==

=== L3 infos ===
{| class="wikitable"
|+
!Device
!OS
!ASN
!VTEP IP
!swp1 or eth1/1
!swp2 or eth1/2
!swp3 or eth1/3
!SVI
|-
|client1
|Linux
|<none>
|<none>
|vlan100: 192.168.1.1/24
vlan200: 192.168.2.1/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client2
|Linux
|<none>
|<none>
|vlan100: 192.168.1.2/24
vlan200: 192.168.2.2/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client3
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.1/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|client4
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.2/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|leaf1
|CVX
|65011
|10.255.255.5/32
|10.254.254.8/31
|10.254.254.10/31
|trunk vlan 100,200
|vlan200 : 192.168.2.254/24
|-
|leaf2
|SRLinux
|65012
|10.255.255.6/32
|10.254.254.12/31
|10.254.254.14/31
|trunk vlan 100,200
|irb0.200 : 192.168.2.254/24
|-
|leaf3
|CVX
|65013
|10.255.255.7/32
|10.254.254.22/31
|10.254.254.24/31
|trunk vlan 100,200
|vlan200 : 192.168.200.254/24
|-
|leaf4
|SRLinux
|65014
|10.255.255.8/32
|10.254.254.26/31
|10.254.254.28/31
|trunk vlan 100,200
|irb0.200 : 192.168.200.254/24
|-
|spine1
|CVX
|65001
|10.255.255.1/32
|10.254.254.9/31
|10.254.254.13/31
|10.254.254.0/31
|<none>
|-
|spine2
|CVX
|65001
|10.255.255.2/32
|10.254.254.11/31
|10.254.254.15/31
|10.254.254.16/31
|<none>
|-
|spine3
|SRLinux
|65002
|10.255.255.3/32
|10.254.254.23/31
|10.254.254.27/31
|10.254.254.18/31
|<none>
|-
|spine4
|SRLinux
|65002
|10.255.255.4/32
|10.254.254.25/31
|10.254.254.29/31
|10.254.254.20/31
|<none>
|-
|superspine1
|CVX
|65003
|10.255.255.9/32
|10.254.254.1/31
|10.254.254.19/31
|<none>
|<none>
|-
|superspine2
|SRLinux
|65003
|10.255.255.10/32
|10.254.254.17/31
|10.254.254.21/31
|<none>
|<none>
|}

=== VXLAN/BGP infos ===
Note : in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>
{| class="wikitable"
|+
!service
!vlan id
!route-target
!VNI
!Commentaires
|-
|L2 cross-DC
|100
|target:65000:100
|100
|the 4 clients are in the same broadcast domain / subnet
|-
|L3VPN cros-DC
|200
|target:65000:200
target:65000:10200
|200 (for L2)
10200 (for routing)
|the 4 clients are separated in 2 different subnets. Traffic is routed within dedicated VNI
To be noted that in the current setup VMTO is not implemented, so cross DC traffic is not optimized.
|}

== Lab diagram ==
[[File:Lab-srl-diagram.png|ohne|mini|600x600px]]

== yml lab file ==
<syntaxhighlight lang="yaml">
# topology documentation: http://containerlab.srlinux.dev/lab-examples/min-5clos/
name: lab-srl-cum

topology:
kinds:
srl:
image: srlinux:21.6.1-235
linux:
image: ghcr.io/hellt/network-multitool
cvx:
image: networkop/cx:4.3.0
runtime: docker
nodes:
leaf1:
kind: cvx
leaf2:
kind: srl
type: ixrd2
leaf3:
kind: cvx
leaf4:
kind: srl
type: ixrd2

spine1:
kind: cvx
spine2:
kind: cvx
spine3:
kind: srl
type: ixr6
spine4:
kind: srl
type: ixr6

superspine1:
kind: cvx
superspine2:
kind: srl
type: ixr6

client1:
kind: linux
client2:
kind: linux
client3:
kind: linux
client4:
kind: linux

links:
# leaf to spine links POD1
- endpoints: ["leaf1:swp1", "spine1:swp1"]
- endpoints: ["leaf1:swp2", "spine2:swp1"]
- endpoints: ["leaf2:e1-1", "spine1:swp2"]
- endpoints: ["leaf2:e1-2", "spine2:swp2"]

# spine to superspine links POD1
- endpoints: ["spine1:swp3", "superspine1:swp1"]
- endpoints: ["spine2:swp3", "superspine2:e1-1"]

# leaf to spine links POD2
- endpoints: ["leaf3:swp1", "spine3:e1-1"]
- endpoints: ["leaf3:swp2", "spine4:e1-1"]
- endpoints: ["leaf4:e1-1", "spine3:e1-2"]
- endpoints: ["leaf4:e1-2", "spine4:e1-2"]

# spine to superspine links POD2
- endpoints: ["spine3:e1-3", "superspine1:swp2"]
- endpoints: ["spine4:e1-3", "superspine2:e1-2"]

# client connection links
- endpoints: ["client1:eth1", "leaf1:swp3"]
- endpoints: ["client2:eth1", "leaf2:e1-3"]
- endpoints: ["client3:eth1", "leaf3:swp3"]
- endpoints: ["client4:eth1", "leaf4:e1-3"]

</syntaxhighlight>

Once the nodes are defined, we have to define the needed connections between the nodes.

Each endpoint is defined as '<nowiki/>'''<nowiki/>'<node_name>:<interface_name><nowiki>''</nowiki>'''

When all is ok, you can launch in suduoer the topology.

For example :<syntaxhighlight lang="bash">
sudo containerlab deploy --topo topo.clab.yml --reconfigure

Note: The option "reconfigure" is used there in order to not have to destroy instances when updating topology/relaunching.
</syntaxhighlight>Another info regarding the yml file, it's possible to quickly generate files with the '''"containerlab gen -n 3tier --nodes 4,2,1"''' command for example. https://containerlab.srlinux.dev/cmd/generate/

To be noted that the "tier-x" notion in containerlab is inversed regarding the rfc 7938 (tier1 is superspine level into the rfc, tier1 is leaf level in nokia tool)

At the end of the topology launch, you can retreive the details :<syntaxhighlight lang="yaml">
INFO[0011] Writing /etc/hosts file
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| # | Name | Container ID | Image | Kind | Group | State | IPv4 Address | IPv6 Address |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| 1 | clab-lab-srl-cum-client1 | 25e2bbf6b13f | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
| 2 | clab-lab-srl-cum-client2 | 17d82d5a6e21 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.4/24 | 2001:172:20:20::4/64 |
| 3 | clab-lab-srl-cum-client3 | f8d78527c1b1 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.6/24 | 2001:172:20:20::6/64 |
| 4 | clab-lab-srl-cum-client4 | 913a0172d0c5 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 5 | clab-lab-srl-cum-leaf1 | 9ffc334af2b1 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.9/24 | 2001:172:20:20::9/64 |
| 6 | clab-lab-srl-cum-leaf2 | 7d3eb855d749 | srlinux:21.6.1-235 | srl | | running | 172.20.20.14/24 | 2001:172:20:20::e/64 |
| 7 | clab-lab-srl-cum-leaf3 | e0d36ffc606a | networkop/cx:4.3.0 | cvx | | running | 172.20.20.7/24 | 2001:172:20:20::7/64 |
| 8 | clab-lab-srl-cum-leaf4 | e6753b1d099e | srlinux:21.6.1-235 | srl | | running | 172.20.20.12/24 | 2001:172:20:20::c/64 |
| 9 | clab-lab-srl-cum-spine1 | a7e63993d052 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.5/24 | 2001:172:20:20::5/64 |
| 10 | clab-lab-srl-cum-spine2 | 3124250d5b75 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
| 11 | clab-lab-srl-cum-spine3 | 5e300ca3f66f | srlinux:21.6.1-235 | srl | | running | 172.20.20.11/24 | 2001:172:20:20::b/64 |
| 12 | clab-lab-srl-cum-spine4 | 0a59e60a596b | srlinux:21.6.1-235 | srl | | running | 172.20.20.13/24 | 2001:172:20:20::d/64 |
| 13 | clab-lab-srl-cum-superspine1 | 3d9935413fc2 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.8/24 | 2001:172:20:20::8/64 |
| 14 | clab-lab-srl-cum-superspine2 | 5bc58d423e2c | srlinux:21.6.1-235 | srl | | running | 172.20.20.15/24 | 2001:172:20:20::f/64 |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
</syntaxhighlight>The containerlab program creates the updates into the host file + generates an ansible inventory file.

There are different ways to connect onto the docker instances, within docker exec commands or SSH directly the IP address related above.

For example :<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-spine1 bash
</syntaxhighlight>-->>> allows to launch a bash on the cumulus device (then you run the different net commands)<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-leaf2 sr_cli
</syntaxhighlight>-->>> allows to launch the cli on a SRLinux device

To end this setup preparation, the containerlab tool integrates an option to launch a webserver showing the lab topology in graph + node details

By default, it's listening on port 50080 of the host

Then, you can access to the following web page showing the topology + lab infos

[[File:Lab graph tool.jpg|ohne|mini|900x900px]]

Finally, if you want to stop a lab, please run '''<nowiki>''</nowiki>containerlab destroy --topo <topology>.yml<nowiki>''</nowiki>'''

At this step, nodes are running, connections are done, but configurations are empty.

'''So, it's time to configure them :)'''

Configuration files are attached to this blog. Once validated, they are exportable and can be called by the tool when launching the lab. Below how to do it :

Firstable, once you ve finished your "hand made config", you can run the following command to save it :<syntaxhighlight lang="bash">
containerlab save -t quickstart.clab.yml
</syntaxhighlight>'''To be noted it's only working for Nokia nodes in this lab'''. CVX configurations will have to be called by another method (or pushed manually)

It will create one folder per node into the lab folder which gives the following tree :
[[File:Topology-srl.jpg|ohne|mini]]

'''Note''': For Cumulus, you can apparently retrieve configurations files within adding the following parameters into lab config file. '''To be tested.''' some lab samples here : https://github.com/hellt/clabs/blob/main/labs/cvx04/symm-mh.clab.yml

Now we have the lab running, below the different commands to check that all is working as expected : <syntaxhighlight lang="yaml">
-------------------------------------------
Check L2 tables ///
-------------------------------------------

Nokia :

A:leaf2# show network-instance mac-vrf-100 bridge-table mac-table all
----------------------------------------------------------------------------------------------------------------------------------------------------------
Mac-table of network instance mac-vrf-100
----------------------------------------------------------------------------------------------------------------------------------------------------------
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
| Address | Destination | Dest Index | Type | Active | Aging | Last Update |
+====================+=========================================+============+=============+=========+========+=========================================+
| AA:C1:AB:0C:60:D6 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T13:06:17.000Z |
| | vtep:10.255.255.8 vni:100 | 67 | | | | |
| AA:C1:AB:23:31:04 | ethernet-1/3.100 | 2 | learnt | true | 300 | 2021-10-08T13:06:10.000Z |
| AA:C1:AB:4B:FF:71 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:55:28.000Z |
| | vtep:10.255.255.5 vni:100 | 60 | | | | |
| AA:C1:AB:B5:79:50 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:43:37.000Z |
| | vtep:10.255.255.7 vni:100 | 64 | | | | |
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 0 Total 0 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 1 Total 1 Active
Total Evpn Macs : 3 Total 3 Active
Total Evpn static Macs : 0 Total 0 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 4 Total 4 Active
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : type learnt matches the local learnt mac, evpn are learnt within VXLAN

Cumulus :

root@leaf1-cum-dc1:/# net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
100 bridge swp3 aa:c1:ab:4b:ff:71 00:00:02
100 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:17:21
100 bridge vni100 22:2d:67:f1:bb:26 permanent 02:17:21
100 bridge vni100 aa:c1:ab:0c:60:d6 extern_learn <1 sec
100 bridge vni100 aa:c1:ab:23:31:04 extern_learn 00:00:07
100 bridge vni100 aa:c1:ab:b5:79:50 extern_learn 00:13:28

root@leaf1-cum-dc1:/# net show bridge macs vlan 200

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
200 bridge swp3 aa:c1:ab:4b:ff:71 00:08:46
200 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:16:36
200 bridge vni200 00:00:5e:00:01:01 static sticky 00:14:26
200 bridge vni200 1e:0d:aa:46:53:b1 permanent 02:16:36
200 bridge vni200 02:77:6b:ff:00:41 static sticky 01:00:52
200 bridge vni200 02:f8:b8:ff:00:41 static sticky 00:14:26
200 bridge vni200 aa:c1:ab:b5:79:50 extern_learn 00:11:51
200 bridge vni10200 02:77:6b:ff:00:00 extern_learn 00:53:02
200 bridge vni10200 02:f8:b8:ff:00:00 extern_learn 00:14:26
200 bridge vni10200 86:5e:aa:ca:50:be permanent 02:16:36

Note : the macs with permanent flag are local interfaces, physical ou logic (vtep)
Note2 : extern_learn mac are the one learned via the VXLAN infra (and mapped to the vnixxx interface)

-------------------------------------------
Check arp tables ///
-------------------------------------------

Nokia :

:leaf2# show arpnd arp-entries
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| ethernet-1/1 | 0 | 10.254.254.13 | dynamic | AA:C1:AB:59:65:EA | 3 hours from now |
| ethernet-1/2 | 0 | 10.254.254.15 | dynamic | AA:C1:AB:CB:F8:8A | 3 hours from now |
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
| mgmt0 | 0 | 172.20.20.1 | dynamic | 02:42:63:9E:74:03 | 2 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 5 (0 static, 5 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : on Nokia you have to sort the entries basing on subinterface and the retrieve the L3VPN attached

A:leaf2# show arpnd arp-entries subinterface 200
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 2 (0 static, 2 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Cumulus :

Here we retrieve all arp infos linked to evpn

root@leaf1-cum-dc1:/# net show evpn arp-cache vni all
VNI 100 #ARP (IPv4 and IPv6, local and remote) 2

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.1.3 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0
192.168.1.1 local active aa:c1:ab:4b:ff:71 0/0

VNI 200 #ARP (IPv4 and IPv6, local and remote) 4

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.2.1 local active aa:c1:ab:4b:ff:71 0/0
192.168.200.254 remote active 00:00:5e:00:01:01 10.255.255.8 0/0
192.168.2.254 remote active 00:00:5e:00:01:01 10.255.255.6 0/0
192.168.200.1 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0

VNI 400 #ARP (IPv4 and IPv6, local and remote) 0

-------------------------------------------
Check bgp sessions' status - underlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep ipv4
| default | 10.254.254.19 | udl-superspine | S | 65003 | established | 0d:0h:22m:44s | ipv4-unicast | [13/4/4] |
| default | 10.254.254.22 | udl-leaf | S | 65013 | established | 0d:0h:22m:52s | ipv4-unicast | [13/1/4] |

Cumulus :

root@leaf3:/# net show bgp ipv4 unicast summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 13
RIB entries 25, using 4800 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.254.254.23 4 65002 466 403 0 0 0 00:19:34 4 13
10.254.254.25 4 65002 434 355 0 0 0 00:17:10 8 13

-------------------------------------------
Check bgp sessions' status - overlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep evpn
| default | 10.255.255.7 | ovl-leaf | S | 65013 | established | 0d:0h:24m:14s| evpn | [15/0/11] |
| default | 10.255.255.9 | ovl-superspine | S | 65003 | established | 0d:0h:24m:7s | evpn | [11/0/4] |

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.255.255.3 4 65002 5618 4742 0 0 0 03:53:35 19 27
10.255.255.4 4 65002 5565 4694 0 0 0 03:51:11 19 27

-------------------------------------------
Check EVPN L2 tables///
-------------------------------------------

Nokia :

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn route type 2
BGP table version is 17, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:100
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
Route Distinguisher: 10.255.255.5:200
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
Route Distinguisher: 10.255.255.6:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
Route Distinguisher: 10.255.255.7:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100
Route Distinguisher: 10.255.255.7:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.200.1]
10.255.255.7 32768 i
ET:8 RT:65000:200
Route Distinguisher: 10.255.255.8:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC

and if we focus on our vni 100

root@leaf3-cum-dc1:/# net show bgp evpn route vni 100 type 2
BGP table version is 48, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100

The additional route type "MAC+IP" is optional and is useful/mandatory when you do anycast irb.

-------------------------------------------
Check tables EVPN L3 ///
-------------------------------------------

Nokia :

--{ + running }--[ ]--
A:leaf2# show network-instance ip-vrf-200 route-table all
-----------------------------------------------------------------------------------------------------------------------------------------------------------
IPv4 Unicast route table of network instance ip-vrf-200
-----------------------------------------------------------------------------------------------------------------------------------------------------------
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
| Prefix | ID | Route Type | Route Owner | Best/Fib- | Metric | Pref | Next-hop | Next-hop |
| | | | | status(slot) | | | (Type) | Interface |
+=======================+=======+============+======================+======================+==========+=========+==============+==============+
| 192.168.2.0/24 | 9 | local | net_inst_mgr | True/success | 0 | 0 | 192.168.2.25 | irb0.200 |
| | | | | | | | 4 (direct) | |
| 192.168.2.254/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (extract) | |
| 192.168.2.255/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (broadcast) | |
| 192.168.200.0/24 | 0 | bgp-evpn | bgp_evpn_mgr | True/success | 0 | 170 | 10.255.255.8 | None |
| | | | | | | | (indirect/vx | |
| | | | | | | | lan) | |
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
-----------------------------------------------------------------------------------------------------------------------------------------------------------
4 IPv4 routes total
4 IPv4 prefixes with active routes
0 IPv4 prefixes with active ECMP routes
-----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--
A:leaf2#

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf1-cum-dc1:/# net show route vrf ip-vrf-200
show ip route vrf ip-vrf-200
=============================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF ip-vrf-200:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 04:48:42
B 192.168.2.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 04:27:26
C>* 192.168.2.0/24 is directly connected, vlan200, 04:48:42
B>* 192.168.200.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:48:49

root@leaf1-cum-dc1:/# net show bgp evpn route type 5
BGP table version is 3, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.6:10200
* [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
*> [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
Route Distinguisher: 10.255.255.8:10200
*> [5]:[0]:[24]:[192.168.200.0]
10.255.255.8 0 65001 65003 65002 65014 i
RT:65000:10200 ET:8 Rmac:02:f8:b8:ff:00:00

-------------------------------------------
Check arp hosts ///
-------------------------------------------

From client #1 /

bash-5.0# arp
Address HWtype HWaddress Flags Mask Iface
192.168.1.3 ether aa:c1:ab:b5:79:50 C eth1.100
192.168.1.4 ether aa:c1:ab:0c:60:d6 C eth1.100
192.168.2.2 ether aa:c1:ab:23:31:04 C eth1.200
192.168.2.254 ether aa:c1:ab:4f:f2:e7 C eth1.200
192.168.1.2 ether aa:c1:ab:23:31:04 C eth1.100
containerlab.local ether 02:42:63:9e:74:03 C eth0

-------------------------------------------
Ping ///
-------------------------------------------

From client #1 /

L2 service ping (vlan 100) ----------------------

bash-5.0# ping -c 1 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.812 ms

--- 192.168.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.812/0.812/0.812/0.000 MsgRcvd
bash-5.0# ping -c 1 192.168.1.3
PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_seq=1 ttl=64 time=1.40 ms

--- 192.168.1.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.403/1.403/1.403/0.000 ms
bash-5.0# ping -c 1 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=1.62 ms

--- 192.168.1.4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.619/1.619/1.619/0.000 ms

L3 service ping (vlan 200)

bash-5.0# ping -c 1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.053 ms

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms
bash-5.0# ^C
bash-5.0# ping -c 1 192.168.2.2
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.611 ms

--- 192.168.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.611/0.611/0.611/0.000 ms

-------------------------------------------
Traceroute ///
-------------------------------------------

L3 service traceroute (vlan 200)

bash-5.0# traceroute 192.168.200.1
traceroute to 192.168.200.1 (192.168.200.1), 30 hops max, 46 byte packets
1 192.168.2.254 (192.168.2.254) 0.012 ms 0.010 ms 0.007 ms
2 192.168.200.254 (192.168.200.254) 2.751 ms * 2.549 ms
3 192.168.200.1 (192.168.200.1) 2.084 ms 1.805 ms 1.761 ms

</syntaxhighlight>To come back on the setup, we wanted to implement solutions inspired by the <nowiki>RFC 7938</nowiki> rfc7938 (see section 5.2.1)

* eBGP will be used for Underlay and Overlay
* eBGP single-hop sessions are established over direct point-to-point links interconnecting the network nodes, no multi-hop or loopback sessions are used, even in the case of multiple links between the same pair of nodes.
* Private Use ASNs from the range 64512-65534 are used to avoid ASN conflicts.
* A single ASN is allocated to all of the Clos topology's Tier 1 devices.
* A unique ASN is allocated to each set of Tier 2 devices in the same cluster.
* A unique ASN is allocated to every Tier 3 device (e.g., ToR) in this topology.

In add, depending of network nodes, it will be interesting to :

* Add fast-failover option for underlay + add link-debounce on links to limit possible impacts in case of link flaps. BFD could be considered in some cases
* Have the best ECMP hashing algorythm on leaf nodes. On tier-1/tier-2 levels hashing will be based on underlay part (trafic inter-vtep), so L3-L4 decision, but some models can retreive the vni-id if I well remember.
* Implement the "multipath relax" since we could have multiple AS paths for a same destination.
* Implement gshut community on underlay (on tier-1/tier-2) or at least policies to prepend underlay routes to avoid some nodes in case of maintenance.

== Thoughts/Points to deepend ==
On leaf nodes, LACP over multiple nodes is feasible within ESI, but impact on incidents/maintenance can vary depending on the vendors.

'''Tests need to be done to check that.'''

Some sysadmin would like to KISS and have client servers connected in active/passive bonding, with eventually single-leaf ports aggregation.

'''Note :''' NH ESI in FIB management on ingress nodes can have some limitations (numbers of installable next-hop, load-balancing, etc. oftenly linked to the node chipset)

Another important engineering point to define at the beginning is : '''Do we authorize multisite L2 propagation or not ?'''

From my point-of-view, the smaller are the L2 segments the better it is. I would suggest Local L2 segments wit anycast gateways provided by Leaf switches, with L3 routing between DCs within EVPN type5 routes sharing only. Nevertheless, some business requirements for endcustomers can keep extended L2 alive.

We have to be careful when implementing Consistent hashing. ex: <nowiki>https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-44/Layer-3/Routing/Equal-Cost-Multipath-Load-Sharing-Hardware-ECMP/</nowiki>

If possible activate it per VRF, not globally, and restrict on dedicated subnets << depends of the vendor, it can be dangerous regarding memory management

Think at configuring the bgp option to not advertise routes to peers before they are installed in FIB.

We can implement leaf roles in some cases, to be able to do "light traffic engineering " on DCI links in some setup.

These roles will be managed within bgp communities added to VTEP /32. Look at the diagram below. Depending of the colour, the eBGP sessions between SupS will prepend or not the VTEP IPs announced to the remote DC

[[File:Fabric-roles.jpg|ohne|mini|1200x1200px]]

<blockquote>'''To be continued'''</blockquote>

== Lab archives ==
<blockquote>Reminder : following config are for lab, so please don't pay attention on typos/lack of security params/unused config. Thanks \o/</blockquote>

=== Credentials ===
<syntaxhighlight lang="bash">
SRLinux : admin/<no_password>

CVX : cumulus/cumulus
</syntaxhighlight>

=== Lab archives ===
layer 2 (vlan 100) + layer 3 (vlan 200) lab, 10/12/2021
[[File:20211008 lab-nokia-cvx-containerlab.zip|ohne|mini]]
<u>If you want to exchange about that lab, those topics, please don't hesitate to contact us</u>

= SONiC Lab =

<span><to be noted that those informations are included into the readme.md file into the archive></span>

== Lab diagram ==
[[File:SONIC_LAB.jpg]]

<div></div>Lab archive [http://wiki.intra.nesevo.com/uploads/LAB/sonic-lab-20201030.zip Attach:sonic-lab-20201030.zip]

This lab is a forked lab from many found on @, starting from Microsoft one : https://github.com/Azure/SONiC/wiki/SONiC-P4-Software-Switch

The target of this lab is to simulate a simple IP/Fabric in SPINE/LEAF topology. Each Leaf provide local L2 connectivity + L3 gateways to hosts. Flows between hosts from different leaf are L3 routed, in a "node path" LEAF-SPINE-LEAF.

Nothing complicated, but it allows to put a first foot into SONiC world :)

For the moment, as far as we know, MC-LAG + EVPN/VXLAN features are only available on entreprise SONiC versions. Tests to come once physic switches in enterprise version will be available in our lab.

The diagram of the architecture is into the zip file (SONIC_LAB.jpg)

To run the lab you need :

- to have installed docker by using ./install_docker_ovs.sh - to run the ./load_image.sh in order to get the Sonic image

Note : don't change file / folders names of network devices (leafX, spineX)

Once you're ready, please run the lab by launching "./start.sh"

The script :<div></div>
* launches the containers without Docket network features
* create network namespaces
* build linux bridges to interconnect the containers
* configure host IPs
* launch startup sonic scripts into sonic devices
* once devices started, checks + ping tests are run. If all BGP sessions are ok on spine + all ping are fine, the setup works well.
The "start" script has several comments, perhaps without the right vocabulary, so please be tolerant :)

If you want to stop the lab, please launch "./stop.sh"

Below some useful commands to run for sonic containers :

=== <span>run ping</span> ===
sudo docker exec -it <container> ping <ip>

This command triggers ping into the targetted container :
<syntaxhighlight lang="bash">
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it leaf1 ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.052 ms
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.034 ms
^C--- 192.168.1.1 ping statistics ---
</syntaxhighlight>

=== <span>check bgp sessions</span> ===
<div></div>
sudo docker exec -it <container> vtysh -c "show ip bgp sum"
This command sollicitates the SONiC shell of the targeted container, in order to print the bgp status.
Below, we run it on one spine, you should have 4 sessiosn UP (for the 4 leaves)
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip bgp sum"
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 6409 6411 0 0 0 01:46:46 3
10.1.0.2 4 65102 6406 6410 0 0 0 01:46:44 3
10.1.0.4 4 65103 6407 6410 0 0 0 01:46:44 4
10.1.0.6 4 65104 6406 6412 0 0 0 01:46:46 1
Total number of neighbors 4

=== <span>check ip route</span> ===
<div></div>
sudo docker exec -it <container> vtysh -c "show ip route"
This command sollicitates the SONiC shell of the targeted container, in order to print its routing table?
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,
> - selected route, * - FIB route
C>* 10.1.0.0/31 is directly connected, Vlan11
C>* 10.1.0.2/31 is directly connected, Vlan21
C>* 10.1.0.4/31 is directly connected, Vlan31
C>* 10.1.0.6/31 is directly connected, Vlan41
C>* 127.0.0.0/8 is directly connected, lo
B>* 192.168.1.0/24 [20/0] via 10.1.0.0, Vlan11, 01:47:23
B>* 192.168.2.0/24 [20/0] via 10.1.0.2, Vlan21, 01:47:21
B>* 192.168.3.0/24 [20/0] via 10.1.0.4, Vlan31, 01:47:21
B>* 192.168.4.0/24 [20/0] via 10.1.0.6, Vlan41, 01:47:23

For more commands check SONiC docs : https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md

To finish, below the output of the start.sh when all goes well<div></div>
plancastre@dockerhost:~/00.Projects/p4-version-pla$ ./start.sh
---------------------------------------------------
Creating the containers: starting
---------------------------------------------------
0b4bbb3c6cfc9895466ea13a9e940d1a176d66f45deee9752e9b4732d735bb03
5ca9486baab3e8a5259062167132a944eee717875d3cc1ef29fe490cba3184c1
46c9c09dba27a32715b9fbe0bc434a9284b9d59317d278df693f425cb669f037
898e6afbcb2977115acbba2a69d069796cfd70fb90228b6c18e09ba4584365f7
eb9b3ef6d14784c9ffe4d9d5df3e921a24559889cb8770d226263aad6761182a
ae5f3b2c628da687eaabb331dc2cd989e71bcb386854e7a0be6ec587df99d49a
122197bffd7566a01ce3b6565fdfb6eb893aab7d5236f14e5a9bbbe95037ced2
a604520a080e2a2184598ad1d915c0c182d16e5185033373e95a4772d8978a67
0c49e18d78b2aa7c8ef8e937b60285aa56ec0bf4a8530226285270a085e11d5c
eb66336b8c6bd8333f1f4f685369e6cac973b817c35bb7126ac4d7ba72ef71bf
---------------------------------------------------
Retreiving PID from docker containers
---------------------------------------------------
---------------------------------------------------
Creating network namespaces for containers
---------------------------------------------------
List of created namespaces
ns_host4
ns_host3
ns_host2
ns_host1
ns_spine2
ns_spine1
ns_leaf4
ns_leaf3
ns_leaf2
ns_leaf1
---------------------------------------------------
Creating network connectivity between containers
---------------------------------------------------
---------------------------------------------------
Now assigning IPs to hosts (host1 to host4)
---------------------------------------------------
---------------------------------------------------
Configuration of containers is now finished
---------------------------------------------------
---------------------------------------------------
Now booting switches, please wait 2 minutes for switches to load
---------------------------------------------------
---------------------------------------------------
2 minutes period is finished
---------------------------------------------------
---------------------------------------------------
Now fixing the iptables
---------------------------------------------------
---------------------------------------------------
Backend actions are now finished
---------------------------------------------------
---------------------------------------------------
Now we'll check in 15 sec that BGP peers are all Up
---------------------------------------------------
---------------------------------------------------
SPINE1 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 94 97 0 0 0 00:01:28 4
10.1.0.2 4 65102 95 97 0 0 0 00:01:28 5
10.1.0.4 4 65103 95 97 0 0 0 00:01:28 5
10.1.0.6 4 65104 95 97 0 0 0 00:01:28 5
Total number of neighbors 4
---------------------------------------------------
SPINE2 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.2.0.1, local AS number 65002
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.2.0.0 4 65101 93 96 0 0 0 00:01:29 2
10.2.0.2 4 65102 94 100 0 0 0 00:01:32 1
10.2.0.4 4 65103 94 100 0 0 0 00:01:32 1
10.2.0.6 4 65104 94 100 0 0 0 00:01:32 1
Total number of neighbors 4
---------------------------------------------------
Now if all is ok, we'll do some ping tests
---------------------------------------------------
---------------------------------------------------
host1 ping its gateway
---------------------------------------------------
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=17.6 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=9.14 ms
--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 9.145/13.390/17.636/4.247 ms
---------------------------------------------------
host2 ping its gateway
---------------------------------------------------
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=23.6 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=6.21 ms
--- 192.168.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 6.218/14.950/23.683/8.733 ms
---------------------------------------------------
host3 ping its gateway
---------------------------------------------------
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=18.8 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=12.7 ms
--- 192.168.3.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.716/15.764/18.812/3.048 ms
---------------------------------------------------
host4 ping its gateway
---------------------------------------------------
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.
64 bytes from 192.168.4.1: icmp_seq=1 ttl=64 time=17.7 ms
64 bytes from 192.168.4.1: icmp_seq=2 ttl=64 time=4.46 ms
--- 192.168.4.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.464/11.107/17.750/6.643 ms
---------------------------------------------------
host1 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=16.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.049/17.253/19.532/1.615 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=15.5 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=14.3 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.312/15.592/16.939/1.083 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=20.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=16.8 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.048/17.871/20.739/2.058 ms
---------------------------------------------------
host2 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=16.6 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.222/16.583/16.924/0.286 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.6 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=18.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=26.8 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.620/20.549/26.887/4.524 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=33.8 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=27.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=20.2 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 20.216/27.259/33.818/5.566 ms
---------------------------------------------------
host3 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=14.5 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=15.5 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.535/16.549/19.541/2.162 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.4 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=14.6 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=18.2 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.643/16.458/18.278/1.491 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=15.3 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=14.1 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=13.1 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 13.171/14.234/15.334/0.883 ms
---------------------------------------------------
host4 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=21.9 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=21.3 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=14.6 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 14.619/19.284/21.932/3.310 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=14.8 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=18.4 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=14.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.148/15.786/18.409/1.879 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=15.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=18.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 15.117/16.788/18.962/1.612 ms
---------------------------------------------------
If bot spine have all BGP sessions and all ping succeeded, all works well
Congrats ! please check readme file to find some details
---------------------------------------------------

Nesevo Virtual Labs

2021-10-29T08:07:10Z

PierreLancastre: /* check bgp sessions */

[[File:nesevo-labs-v2.png]]

= Nokia/Cumulus Multi-Vendor Lab =

The targets of this lab are to :

* mix Nokia SR-Linux and Cumulus VXLAN nodes, in spine or leaf roles
* make this environment work for different kinds of setup : cross-DC L2, differentiated L3 gateways per leaf per VRF, anycast gateway cross or intra DC, etc.
* spot compatibility issues if existing

To reach those goals, we'll use the Nokia lab tool "containerlab" https://containerlab.srlinux.dev/, labs from https://clabs.netdevops.me/ helped a lot. Many thanks to the contributors.

We'll need a Linux host with Docker installed on it + kvm virtualization enhancements activated (important if your host is a VM). If virtualization optimization is not ok, you won't be able to launch the most of devices.

Installation details are referenced there : https://containerlab.srlinux.dev/install/

Once installed, you'll need to register the different NOS images you'll use. Cumulus image is present on a public registrery, but Nokia/Arista/Juniper are not, you'll need a vendor support account to retreive them. Once downloaded (.tgz file typically) you can "docker load" the image :

For private images :<syntaxhighlight lang="bash">
sudo docker load -i 21.6.1-235.tar.xz /* (Nokia) */
sudo docker load -i cEOS64-lab-4.26.1F.tar /* (Arista) */
</syntaxhighlight>For public images :<syntaxhighlight lang="bash">
sudo docker pull networkop/cx:4.3.0 /* (Cumulus) */
</syntaxhighlight>

To verify your installed images, you can run "sudo docker images"<syntaxhighlight lang="bash">
plancastre@containerlab:~/00.LAB/senss$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
networkop/cx 4.3.0 a40541ed15ba 2 weeks ago 733MB
srlinux 21.6.1-235 e29aed30e83f 3 weeks ago 1.14GB
networkop/ignite dev dcc5eff29355 4 weeks ago 37.9MB
networkop/kernel 4.19 e794d8875f83 3 months ago 88MB
frrouting/frr v7.5.1 c3e13a4c5918 3 months ago 123MB

</syntaxhighlight>Once you've got all your images, to run the lab, you'll need a .yml file defining the topology to run.

In our lab, we've started with the 5-clos lab file from Nokia site, which we've forked.

We can see there is a branch "kinds" where we can define templates of nodes, in order to not rewrite whole parameters when creating nodes. For the images, it's important to take the repository name + tag.

'''Note:''' If the tag is "latest" it is not needed.

All supported "kinds" are referenced there : https://containerlab.srlinux.dev/manual/kinds/kinds/

'''Note:''' yml is indentation sensitive, so please respect the "double space" used to define hierarchy. Below the lab diagram + the yml file associated

== Infos on nodes ==

=== L3 infos ===
{| class="wikitable"
|+
!Device
!OS
!ASN
!VTEP IP
!swp1 or eth1/1
!swp2 or eth1/2
!swp3 or eth1/3
!SVI
|-
|client1
|Linux
|<none>
|<none>
|vlan100: 192.168.1.1/24
vlan200: 192.168.2.1/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client2
|Linux
|<none>
|<none>
|vlan100: 192.168.1.2/24
vlan200: 192.168.2.2/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client3
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.1/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|client4
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.2/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|leaf1
|CVX
|65011
|10.255.255.5/32
|10.254.254.8/31
|10.254.254.10/31
|trunk vlan 100,200
|vlan200 : 192.168.2.254/24
|-
|leaf2
|SRLinux
|65012
|10.255.255.6/32
|10.254.254.12/31
|10.254.254.14/31
|trunk vlan 100,200
|irb0.200 : 192.168.2.254/24
|-
|leaf3
|CVX
|65013
|10.255.255.7/32
|10.254.254.22/31
|10.254.254.24/31
|trunk vlan 100,200
|vlan200 : 192.168.200.254/24
|-
|leaf4
|SRLinux
|65014
|10.255.255.8/32
|10.254.254.26/31
|10.254.254.28/31
|trunk vlan 100,200
|irb0.200 : 192.168.200.254/24
|-
|spine1
|CVX
|65001
|10.255.255.1/32
|10.254.254.9/31
|10.254.254.13/31
|10.254.254.0/31
|<none>
|-
|spine2
|CVX
|65001
|10.255.255.2/32
|10.254.254.11/31
|10.254.254.15/31
|10.254.254.16/31
|<none>
|-
|spine3
|SRLinux
|65002
|10.255.255.3/32
|10.254.254.23/31
|10.254.254.27/31
|10.254.254.18/31
|<none>
|-
|spine4
|SRLinux
|65002
|10.255.255.4/32
|10.254.254.25/31
|10.254.254.29/31
|10.254.254.20/31
|<none>
|-
|superspine1
|CVX
|65003
|10.255.255.9/32
|10.254.254.1/31
|10.254.254.19/31
|<none>
|<none>
|-
|superspine2
|SRLinux
|65003
|10.255.255.10/32
|10.254.254.17/31
|10.254.254.21/31
|<none>
|<none>
|}

=== VXLAN/BGP infos ===
Note : in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>
{| class="wikitable"
|+
!service
!vlan id
!route-target
!VNI
!Commentaires
|-
|L2 cross-DC
|100
|target:65000:100
|100
|the 4 clients are in the same broadcast domain / subnet
|-
|L3VPN cros-DC
|200
|target:65000:200
target:65000:10200
|200 (for L2)
10200 (for routing)
|the 4 clients are separated in 2 different subnets. Traffic is routed within dedicated VNI
To be noted that in the current setup VMTO is not implemented, so cross DC traffic is not optimized.
|}

== Lab diagram ==
[[File:Lab-srl-diagram.png|ohne|mini|600x600px]]

== yml lab file ==
<syntaxhighlight lang="yaml">
# topology documentation: http://containerlab.srlinux.dev/lab-examples/min-5clos/
name: lab-srl-cum

topology:
kinds:
srl:
image: srlinux:21.6.1-235
linux:
image: ghcr.io/hellt/network-multitool
cvx:
image: networkop/cx:4.3.0
runtime: docker
nodes:
leaf1:
kind: cvx
leaf2:
kind: srl
type: ixrd2
leaf3:
kind: cvx
leaf4:
kind: srl
type: ixrd2

spine1:
kind: cvx
spine2:
kind: cvx
spine3:
kind: srl
type: ixr6
spine4:
kind: srl
type: ixr6

superspine1:
kind: cvx
superspine2:
kind: srl
type: ixr6

client1:
kind: linux
client2:
kind: linux
client3:
kind: linux
client4:
kind: linux

links:
# leaf to spine links POD1
- endpoints: ["leaf1:swp1", "spine1:swp1"]
- endpoints: ["leaf1:swp2", "spine2:swp1"]
- endpoints: ["leaf2:e1-1", "spine1:swp2"]
- endpoints: ["leaf2:e1-2", "spine2:swp2"]

# spine to superspine links POD1
- endpoints: ["spine1:swp3", "superspine1:swp1"]
- endpoints: ["spine2:swp3", "superspine2:e1-1"]

# leaf to spine links POD2
- endpoints: ["leaf3:swp1", "spine3:e1-1"]
- endpoints: ["leaf3:swp2", "spine4:e1-1"]
- endpoints: ["leaf4:e1-1", "spine3:e1-2"]
- endpoints: ["leaf4:e1-2", "spine4:e1-2"]

# spine to superspine links POD2
- endpoints: ["spine3:e1-3", "superspine1:swp2"]
- endpoints: ["spine4:e1-3", "superspine2:e1-2"]

# client connection links
- endpoints: ["client1:eth1", "leaf1:swp3"]
- endpoints: ["client2:eth1", "leaf2:e1-3"]
- endpoints: ["client3:eth1", "leaf3:swp3"]
- endpoints: ["client4:eth1", "leaf4:e1-3"]

</syntaxhighlight>

Once the nodes are defined, we have to define the needed connections between the nodes.

Each endpoint is defined as '<nowiki/>'''<nowiki/>'<node_name>:<interface_name><nowiki>''</nowiki>'''

When all is ok, you can launch in suduoer the topology.

For example :<syntaxhighlight lang="bash">
sudo containerlab deploy --topo topo.clab.yml --reconfigure

Note: The option "reconfigure" is used there in order to not have to destroy instances when updating topology/relaunching.
</syntaxhighlight>Another info regarding the yml file, it's possible to quickly generate files with the '''"containerlab gen -n 3tier --nodes 4,2,1"''' command for example. https://containerlab.srlinux.dev/cmd/generate/

To be noted that the "tier-x" notion in containerlab is inversed regarding the rfc 7938 (tier1 is superspine level into the rfc, tier1 is leaf level in nokia tool)

At the end of the topology launch, you can retreive the details :<syntaxhighlight lang="yaml">
INFO[0011] Writing /etc/hosts file
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| # | Name | Container ID | Image | Kind | Group | State | IPv4 Address | IPv6 Address |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| 1 | clab-lab-srl-cum-client1 | 25e2bbf6b13f | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
| 2 | clab-lab-srl-cum-client2 | 17d82d5a6e21 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.4/24 | 2001:172:20:20::4/64 |
| 3 | clab-lab-srl-cum-client3 | f8d78527c1b1 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.6/24 | 2001:172:20:20::6/64 |
| 4 | clab-lab-srl-cum-client4 | 913a0172d0c5 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 5 | clab-lab-srl-cum-leaf1 | 9ffc334af2b1 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.9/24 | 2001:172:20:20::9/64 |
| 6 | clab-lab-srl-cum-leaf2 | 7d3eb855d749 | srlinux:21.6.1-235 | srl | | running | 172.20.20.14/24 | 2001:172:20:20::e/64 |
| 7 | clab-lab-srl-cum-leaf3 | e0d36ffc606a | networkop/cx:4.3.0 | cvx | | running | 172.20.20.7/24 | 2001:172:20:20::7/64 |
| 8 | clab-lab-srl-cum-leaf4 | e6753b1d099e | srlinux:21.6.1-235 | srl | | running | 172.20.20.12/24 | 2001:172:20:20::c/64 |
| 9 | clab-lab-srl-cum-spine1 | a7e63993d052 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.5/24 | 2001:172:20:20::5/64 |
| 10 | clab-lab-srl-cum-spine2 | 3124250d5b75 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
| 11 | clab-lab-srl-cum-spine3 | 5e300ca3f66f | srlinux:21.6.1-235 | srl | | running | 172.20.20.11/24 | 2001:172:20:20::b/64 |
| 12 | clab-lab-srl-cum-spine4 | 0a59e60a596b | srlinux:21.6.1-235 | srl | | running | 172.20.20.13/24 | 2001:172:20:20::d/64 |
| 13 | clab-lab-srl-cum-superspine1 | 3d9935413fc2 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.8/24 | 2001:172:20:20::8/64 |
| 14 | clab-lab-srl-cum-superspine2 | 5bc58d423e2c | srlinux:21.6.1-235 | srl | | running | 172.20.20.15/24 | 2001:172:20:20::f/64 |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
</syntaxhighlight>The containerlab program creates the updates into the host file + generates an ansible inventory file.

There are different ways to connect onto the docker instances, within docker exec commands or SSH directly the IP address related above.

For example :<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-spine1 bash
</syntaxhighlight>-->>> allows to launch a bash on the cumulus device (then you run the different net commands)<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-leaf2 sr_cli
</syntaxhighlight>-->>> allows to launch the cli on a SRLinux device

To end this setup preparation, the containerlab tool integrates an option to launch a webserver showing the lab topology in graph + node details

By default, it's listening on port 50080 of the host

Then, you can access to the following web page showing the topology + lab infos

[[File:Lab graph tool.jpg|ohne|mini|900x900px]]

Finally, if you want to stop a lab, please run '''<nowiki>''</nowiki>containerlab destroy --topo <topology>.yml<nowiki>''</nowiki>'''

At this step, nodes are running, connections are done, but configurations are empty.

'''So, it's time to configure them :)'''

Configuration files are attached to this blog. Once validated, they are exportable and can be called by the tool when launching the lab. Below how to do it :

Firstable, once you ve finished your "hand made config", you can run the following command to save it :<syntaxhighlight lang="bash">
containerlab save -t quickstart.clab.yml
</syntaxhighlight>'''To be noted it's only working for Nokia nodes in this lab'''. CVX configurations will have to be called by another method (or pushed manually)

It will create one folder per node into the lab folder which gives the following tree :
[[File:Topology-srl.jpg|ohne|mini]]

'''Note''': For Cumulus, you can apparently retrieve configurations files within adding the following parameters into lab config file. '''To be tested.''' some lab samples here : https://github.com/hellt/clabs/blob/main/labs/cvx04/symm-mh.clab.yml

Now we have the lab running, below the different commands to check that all is working as expected : <syntaxhighlight lang="yaml">
-------------------------------------------
Check L2 tables ///
-------------------------------------------

Nokia :

A:leaf2# show network-instance mac-vrf-100 bridge-table mac-table all
----------------------------------------------------------------------------------------------------------------------------------------------------------
Mac-table of network instance mac-vrf-100
----------------------------------------------------------------------------------------------------------------------------------------------------------
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
| Address | Destination | Dest Index | Type | Active | Aging | Last Update |
+====================+=========================================+============+=============+=========+========+=========================================+
| AA:C1:AB:0C:60:D6 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T13:06:17.000Z |
| | vtep:10.255.255.8 vni:100 | 67 | | | | |
| AA:C1:AB:23:31:04 | ethernet-1/3.100 | 2 | learnt | true | 300 | 2021-10-08T13:06:10.000Z |
| AA:C1:AB:4B:FF:71 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:55:28.000Z |
| | vtep:10.255.255.5 vni:100 | 60 | | | | |
| AA:C1:AB:B5:79:50 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:43:37.000Z |
| | vtep:10.255.255.7 vni:100 | 64 | | | | |
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 0 Total 0 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 1 Total 1 Active
Total Evpn Macs : 3 Total 3 Active
Total Evpn static Macs : 0 Total 0 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 4 Total 4 Active
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : type learnt matches the local learnt mac, evpn are learnt within VXLAN

Cumulus :

root@leaf1-cum-dc1:/# net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
100 bridge swp3 aa:c1:ab:4b:ff:71 00:00:02
100 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:17:21
100 bridge vni100 22:2d:67:f1:bb:26 permanent 02:17:21
100 bridge vni100 aa:c1:ab:0c:60:d6 extern_learn <1 sec
100 bridge vni100 aa:c1:ab:23:31:04 extern_learn 00:00:07
100 bridge vni100 aa:c1:ab:b5:79:50 extern_learn 00:13:28

root@leaf1-cum-dc1:/# net show bridge macs vlan 200

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
200 bridge swp3 aa:c1:ab:4b:ff:71 00:08:46
200 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:16:36
200 bridge vni200 00:00:5e:00:01:01 static sticky 00:14:26
200 bridge vni200 1e:0d:aa:46:53:b1 permanent 02:16:36
200 bridge vni200 02:77:6b:ff:00:41 static sticky 01:00:52
200 bridge vni200 02:f8:b8:ff:00:41 static sticky 00:14:26
200 bridge vni200 aa:c1:ab:b5:79:50 extern_learn 00:11:51
200 bridge vni10200 02:77:6b:ff:00:00 extern_learn 00:53:02
200 bridge vni10200 02:f8:b8:ff:00:00 extern_learn 00:14:26
200 bridge vni10200 86:5e:aa:ca:50:be permanent 02:16:36

Note : the macs with permanent flag are local interfaces, physical ou logic (vtep)
Note2 : extern_learn mac are the one learned via the VXLAN infra (and mapped to the vnixxx interface)

-------------------------------------------
Check arp tables ///
-------------------------------------------

Nokia :

:leaf2# show arpnd arp-entries
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| ethernet-1/1 | 0 | 10.254.254.13 | dynamic | AA:C1:AB:59:65:EA | 3 hours from now |
| ethernet-1/2 | 0 | 10.254.254.15 | dynamic | AA:C1:AB:CB:F8:8A | 3 hours from now |
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
| mgmt0 | 0 | 172.20.20.1 | dynamic | 02:42:63:9E:74:03 | 2 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 5 (0 static, 5 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : on Nokia you have to sort the entries basing on subinterface and the retrieve the L3VPN attached

A:leaf2# show arpnd arp-entries subinterface 200
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 2 (0 static, 2 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Cumulus :

Here we retrieve all arp infos linked to evpn

root@leaf1-cum-dc1:/# net show evpn arp-cache vni all
VNI 100 #ARP (IPv4 and IPv6, local and remote) 2

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.1.3 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0
192.168.1.1 local active aa:c1:ab:4b:ff:71 0/0

VNI 200 #ARP (IPv4 and IPv6, local and remote) 4

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.2.1 local active aa:c1:ab:4b:ff:71 0/0
192.168.200.254 remote active 00:00:5e:00:01:01 10.255.255.8 0/0
192.168.2.254 remote active 00:00:5e:00:01:01 10.255.255.6 0/0
192.168.200.1 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0

VNI 400 #ARP (IPv4 and IPv6, local and remote) 0

-------------------------------------------
Check bgp sessions' status - underlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep ipv4
| default | 10.254.254.19 | udl-superspine | S | 65003 | established | 0d:0h:22m:44s | ipv4-unicast | [13/4/4] |
| default | 10.254.254.22 | udl-leaf | S | 65013 | established | 0d:0h:22m:52s | ipv4-unicast | [13/1/4] |

Cumulus :

root@leaf3:/# net show bgp ipv4 unicast summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 13
RIB entries 25, using 4800 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.254.254.23 4 65002 466 403 0 0 0 00:19:34 4 13
10.254.254.25 4 65002 434 355 0 0 0 00:17:10 8 13

-------------------------------------------
Check bgp sessions' status - overlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep evpn
| default | 10.255.255.7 | ovl-leaf | S | 65013 | established | 0d:0h:24m:14s| evpn | [15/0/11] |
| default | 10.255.255.9 | ovl-superspine | S | 65003 | established | 0d:0h:24m:7s | evpn | [11/0/4] |

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.255.255.3 4 65002 5618 4742 0 0 0 03:53:35 19 27
10.255.255.4 4 65002 5565 4694 0 0 0 03:51:11 19 27

-------------------------------------------
Check EVPN L2 tables///
-------------------------------------------

Nokia :

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn route type 2
BGP table version is 17, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:100
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
Route Distinguisher: 10.255.255.5:200
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
Route Distinguisher: 10.255.255.6:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
Route Distinguisher: 10.255.255.7:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100
Route Distinguisher: 10.255.255.7:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.200.1]
10.255.255.7 32768 i
ET:8 RT:65000:200
Route Distinguisher: 10.255.255.8:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC

and if we focus on our vni 100

root@leaf3-cum-dc1:/# net show bgp evpn route vni 100 type 2
BGP table version is 48, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100

The additional route type "MAC+IP" is optional and is useful/mandatory when you do anycast irb.

-------------------------------------------
Check tables EVPN L3 ///
-------------------------------------------

Nokia :

--{ + running }--[ ]--
A:leaf2# show network-instance ip-vrf-200 route-table all
-----------------------------------------------------------------------------------------------------------------------------------------------------------
IPv4 Unicast route table of network instance ip-vrf-200
-----------------------------------------------------------------------------------------------------------------------------------------------------------
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
| Prefix | ID | Route Type | Route Owner | Best/Fib- | Metric | Pref | Next-hop | Next-hop |
| | | | | status(slot) | | | (Type) | Interface |
+=======================+=======+============+======================+======================+==========+=========+==============+==============+
| 192.168.2.0/24 | 9 | local | net_inst_mgr | True/success | 0 | 0 | 192.168.2.25 | irb0.200 |
| | | | | | | | 4 (direct) | |
| 192.168.2.254/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (extract) | |
| 192.168.2.255/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (broadcast) | |
| 192.168.200.0/24 | 0 | bgp-evpn | bgp_evpn_mgr | True/success | 0 | 170 | 10.255.255.8 | None |
| | | | | | | | (indirect/vx | |
| | | | | | | | lan) | |
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
-----------------------------------------------------------------------------------------------------------------------------------------------------------
4 IPv4 routes total
4 IPv4 prefixes with active routes
0 IPv4 prefixes with active ECMP routes
-----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--
A:leaf2#

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf1-cum-dc1:/# net show route vrf ip-vrf-200
show ip route vrf ip-vrf-200
=============================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF ip-vrf-200:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 04:48:42
B 192.168.2.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 04:27:26
C>* 192.168.2.0/24 is directly connected, vlan200, 04:48:42
B>* 192.168.200.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:48:49

root@leaf1-cum-dc1:/# net show bgp evpn route type 5
BGP table version is 3, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.6:10200
* [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
*> [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
Route Distinguisher: 10.255.255.8:10200
*> [5]:[0]:[24]:[192.168.200.0]
10.255.255.8 0 65001 65003 65002 65014 i
RT:65000:10200 ET:8 Rmac:02:f8:b8:ff:00:00

-------------------------------------------
Check arp hosts ///
-------------------------------------------

From client #1 /

bash-5.0# arp
Address HWtype HWaddress Flags Mask Iface
192.168.1.3 ether aa:c1:ab:b5:79:50 C eth1.100
192.168.1.4 ether aa:c1:ab:0c:60:d6 C eth1.100
192.168.2.2 ether aa:c1:ab:23:31:04 C eth1.200
192.168.2.254 ether aa:c1:ab:4f:f2:e7 C eth1.200
192.168.1.2 ether aa:c1:ab:23:31:04 C eth1.100
containerlab.local ether 02:42:63:9e:74:03 C eth0

-------------------------------------------
Ping ///
-------------------------------------------

From client #1 /

L2 service ping (vlan 100) ----------------------

bash-5.0# ping -c 1 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.812 ms

--- 192.168.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.812/0.812/0.812/0.000 MsgRcvd
bash-5.0# ping -c 1 192.168.1.3
PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_seq=1 ttl=64 time=1.40 ms

--- 192.168.1.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.403/1.403/1.403/0.000 ms
bash-5.0# ping -c 1 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=1.62 ms

--- 192.168.1.4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.619/1.619/1.619/0.000 ms

L3 service ping (vlan 200)

bash-5.0# ping -c 1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.053 ms

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms
bash-5.0# ^C
bash-5.0# ping -c 1 192.168.2.2
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.611 ms

--- 192.168.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.611/0.611/0.611/0.000 ms

-------------------------------------------
Traceroute ///
-------------------------------------------

L3 service traceroute (vlan 200)

bash-5.0# traceroute 192.168.200.1
traceroute to 192.168.200.1 (192.168.200.1), 30 hops max, 46 byte packets
1 192.168.2.254 (192.168.2.254) 0.012 ms 0.010 ms 0.007 ms
2 192.168.200.254 (192.168.200.254) 2.751 ms * 2.549 ms
3 192.168.200.1 (192.168.200.1) 2.084 ms 1.805 ms 1.761 ms

</syntaxhighlight>To come back on the setup, we wanted to implement solutions inspired by the <nowiki>RFC 7938</nowiki> rfc7938 (see section 5.2.1)

* eBGP will be used for Underlay and Overlay
* eBGP single-hop sessions are established over direct point-to-point links interconnecting the network nodes, no multi-hop or loopback sessions are used, even in the case of multiple links between the same pair of nodes.
* Private Use ASNs from the range 64512-65534 are used to avoid ASN conflicts.
* A single ASN is allocated to all of the Clos topology's Tier 1 devices.
* A unique ASN is allocated to each set of Tier 2 devices in the same cluster.
* A unique ASN is allocated to every Tier 3 device (e.g., ToR) in this topology.

In add, depending of network nodes, it will be interesting to :

* Add fast-failover option for underlay + add link-debounce on links to limit possible impacts in case of link flaps. BFD could be considered in some cases
* Have the best ECMP hashing algorythm on leaf nodes. On tier-1/tier-2 levels hashing will be based on underlay part (trafic inter-vtep), so L3-L4 decision, but some models can retreive the vni-id if I well remember.
* Implement the "multipath relax" since we could have multiple AS paths for a same destination.
* Implement gshut community on underlay (on tier-1/tier-2) or at least policies to prepend underlay routes to avoid some nodes in case of maintenance.

== Thoughts/Points to deepend ==
On leaf nodes, LACP over multiple nodes is feasible within ESI, but impact on incidents/maintenance can vary depending on the vendors.

'''Tests need to be done to check that.'''

Some sysadmin would like to KISS and have client servers connected in active/passive bonding, with eventually single-leaf ports aggregation.

'''Note :''' NH ESI in FIB management on ingress nodes can have some limitations (numbers of installable next-hop, load-balancing, etc. oftenly linked to the node chipset)

Another important engineering point to define at the beginning is : '''Do we authorize multisite L2 propagation or not ?'''

From my point-of-view, the smaller are the L2 segments the better it is. I would suggest Local L2 segments wit anycast gateways provided by Leaf switches, with L3 routing between DCs within EVPN type5 routes sharing only. Nevertheless, some business requirements for endcustomers can keep extended L2 alive.

We have to be careful when implementing Consistent hashing. ex: <nowiki>https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-44/Layer-3/Routing/Equal-Cost-Multipath-Load-Sharing-Hardware-ECMP/</nowiki>

If possible activate it per VRF, not globally, and restrict on dedicated subnets << depends of the vendor, it can be dangerous regarding memory management

Think at configuring the bgp option to not advertise routes to peers before they are installed in FIB.

We can implement leaf roles in some cases, to be able to do "light traffic engineering " on DCI links in some setup.

These roles will be managed within bgp communities added to VTEP /32. Look at the diagram below. Depending of the colour, the eBGP sessions between SupS will prepend or not the VTEP IPs announced to the remote DC

[[File:Fabric-roles.jpg|ohne|mini|1200x1200px]]

<blockquote>'''To be continued'''</blockquote>

== Lab archives ==
<blockquote>Reminder : following config are for lab, so please don't pay attention on typos/lack of security params/unused config. Thanks \o/</blockquote>

=== Credentials ===
<syntaxhighlight lang="bash">
SRLinux : admin/<no_password>

CVX : cumulus/cumulus
</syntaxhighlight>

=== Lab archives ===
layer 2 (vlan 100) + layer 3 (vlan 200) lab, 10/12/2021
[[File:20211008 lab-nokia-cvx-containerlab.zip|ohne|mini]]
<u>If you want to exchange about that lab, those topics, please don't hesitate to contact us</u>

= SONiC Lab =

<span><to be noted that those informations are included into the readme.md file into the archive></span>

== Lab diagram ==
[[File:SONIC_LAB.jpg]]

<div></div>Lab archive [http://wiki.intra.nesevo.com/uploads/LAB/sonic-lab-20201030.zip Attach:sonic-lab-20201030.zip]

This lab is a forked lab from many found on @, starting from Microsoft one : https://github.com/Azure/SONiC/wiki/SONiC-P4-Software-Switch

The target of this lab is to simulate a simple IP/Fabric in SPINE/LEAF topology. Each Leaf provide local L2 connectivity + L3 gateways to hosts. Flows between hosts from different leaf are L3 routed, in a "node path" LEAF-SPINE-LEAF.

Nothing complicated, but it allows to put a first foot into SONiC world :)

For the moment, as far as we know, MC-LAG + EVPN/VXLAN features are only available on entreprise SONiC versions. Tests to come once physic switches in enterprise version will be available in our lab.

The diagram of the architecture is into the zip file (SONIC_LAB.jpg)

To run the lab you need :

- to have installed docker by using ./install_docker_ovs.sh - to run the ./load_image.sh in order to get the Sonic image

Note : don't change file / folders names of network devices (leafX, spineX)

Once you're ready, please run the lab by launching "./start.sh"

The script :<div></div>
* launches the containers without Docket network features
* create network namespaces
* build linux bridges to interconnect the containers
* configure host IPs
* launch startup sonic scripts into sonic devices
* once devices started, checks + ping tests are run. If all BGP sessions are ok on spine + all ping are fine, the setup works well.
The "start" script has several comments, perhaps without the right vocabulary, so please be tolerant :)

If you want to stop the lab, please launch "./stop.sh"

Below some useful commands to run for sonic containers :

=== <span>run ping</span> ===
sudo docker exec -it <container> ping <ip>

This command triggers ping into the targetted container :
<div></div>
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it leaf1 ping 192.168.1.1

PING 192.168.1.1 (192.168.1.1): 56 data bytes

64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.052 ms

64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.034 ms

^C--- 192.168.1.1 ping statistics ---

=== <span>check bgp sessions</span> ===
<div></div>
sudo docker exec -it <container> vtysh -c "show ip bgp sum"
This command sollicitates the SONiC shell of the targeted container, in order to print the bgp status.
Below, we run it on one spine, you should have 4 sessiosn UP (for the 4 leaves)
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip bgp sum"
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 6409 6411 0 0 0 01:46:46 3
10.1.0.2 4 65102 6406 6410 0 0 0 01:46:44 3
10.1.0.4 4 65103 6407 6410 0 0 0 01:46:44 4
10.1.0.6 4 65104 6406 6412 0 0 0 01:46:46 1
Total number of neighbors 4

=== <span>check ip route</span> ===
<div></div>
sudo docker exec -it <container> vtysh -c "show ip route"
This command sollicitates the SONiC shell of the targeted container, in order to print its routing table?
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,
> - selected route, * - FIB route
C>* 10.1.0.0/31 is directly connected, Vlan11
C>* 10.1.0.2/31 is directly connected, Vlan21
C>* 10.1.0.4/31 is directly connected, Vlan31
C>* 10.1.0.6/31 is directly connected, Vlan41
C>* 127.0.0.0/8 is directly connected, lo
B>* 192.168.1.0/24 [20/0] via 10.1.0.0, Vlan11, 01:47:23
B>* 192.168.2.0/24 [20/0] via 10.1.0.2, Vlan21, 01:47:21
B>* 192.168.3.0/24 [20/0] via 10.1.0.4, Vlan31, 01:47:21
B>* 192.168.4.0/24 [20/0] via 10.1.0.6, Vlan41, 01:47:23

For more commands check SONiC docs : https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md

To finish, below the output of the start.sh when all goes well<div></div>
plancastre@dockerhost:~/00.Projects/p4-version-pla$ ./start.sh
---------------------------------------------------
Creating the containers: starting
---------------------------------------------------
0b4bbb3c6cfc9895466ea13a9e940d1a176d66f45deee9752e9b4732d735bb03
5ca9486baab3e8a5259062167132a944eee717875d3cc1ef29fe490cba3184c1
46c9c09dba27a32715b9fbe0bc434a9284b9d59317d278df693f425cb669f037
898e6afbcb2977115acbba2a69d069796cfd70fb90228b6c18e09ba4584365f7
eb9b3ef6d14784c9ffe4d9d5df3e921a24559889cb8770d226263aad6761182a
ae5f3b2c628da687eaabb331dc2cd989e71bcb386854e7a0be6ec587df99d49a
122197bffd7566a01ce3b6565fdfb6eb893aab7d5236f14e5a9bbbe95037ced2
a604520a080e2a2184598ad1d915c0c182d16e5185033373e95a4772d8978a67
0c49e18d78b2aa7c8ef8e937b60285aa56ec0bf4a8530226285270a085e11d5c
eb66336b8c6bd8333f1f4f685369e6cac973b817c35bb7126ac4d7ba72ef71bf
---------------------------------------------------
Retreiving PID from docker containers
---------------------------------------------------
---------------------------------------------------
Creating network namespaces for containers
---------------------------------------------------
List of created namespaces
ns_host4
ns_host3
ns_host2
ns_host1
ns_spine2
ns_spine1
ns_leaf4
ns_leaf3
ns_leaf2
ns_leaf1
---------------------------------------------------
Creating network connectivity between containers
---------------------------------------------------
---------------------------------------------------
Now assigning IPs to hosts (host1 to host4)
---------------------------------------------------
---------------------------------------------------
Configuration of containers is now finished
---------------------------------------------------
---------------------------------------------------
Now booting switches, please wait 2 minutes for switches to load
---------------------------------------------------
---------------------------------------------------
2 minutes period is finished
---------------------------------------------------
---------------------------------------------------
Now fixing the iptables
---------------------------------------------------
---------------------------------------------------
Backend actions are now finished
---------------------------------------------------
---------------------------------------------------
Now we'll check in 15 sec that BGP peers are all Up
---------------------------------------------------
---------------------------------------------------
SPINE1 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 94 97 0 0 0 00:01:28 4
10.1.0.2 4 65102 95 97 0 0 0 00:01:28 5
10.1.0.4 4 65103 95 97 0 0 0 00:01:28 5
10.1.0.6 4 65104 95 97 0 0 0 00:01:28 5
Total number of neighbors 4
---------------------------------------------------
SPINE2 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.2.0.1, local AS number 65002
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.2.0.0 4 65101 93 96 0 0 0 00:01:29 2
10.2.0.2 4 65102 94 100 0 0 0 00:01:32 1
10.2.0.4 4 65103 94 100 0 0 0 00:01:32 1
10.2.0.6 4 65104 94 100 0 0 0 00:01:32 1
Total number of neighbors 4
---------------------------------------------------
Now if all is ok, we'll do some ping tests
---------------------------------------------------
---------------------------------------------------
host1 ping its gateway
---------------------------------------------------
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=17.6 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=9.14 ms
--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 9.145/13.390/17.636/4.247 ms
---------------------------------------------------
host2 ping its gateway
---------------------------------------------------
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=23.6 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=6.21 ms
--- 192.168.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 6.218/14.950/23.683/8.733 ms
---------------------------------------------------
host3 ping its gateway
---------------------------------------------------
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=18.8 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=12.7 ms
--- 192.168.3.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.716/15.764/18.812/3.048 ms
---------------------------------------------------
host4 ping its gateway
---------------------------------------------------
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.
64 bytes from 192.168.4.1: icmp_seq=1 ttl=64 time=17.7 ms
64 bytes from 192.168.4.1: icmp_seq=2 ttl=64 time=4.46 ms
--- 192.168.4.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.464/11.107/17.750/6.643 ms
---------------------------------------------------
host1 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=16.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.049/17.253/19.532/1.615 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=15.5 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=14.3 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.312/15.592/16.939/1.083 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=20.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=16.8 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.048/17.871/20.739/2.058 ms
---------------------------------------------------
host2 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=16.6 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.222/16.583/16.924/0.286 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.6 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=18.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=26.8 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.620/20.549/26.887/4.524 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=33.8 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=27.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=20.2 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 20.216/27.259/33.818/5.566 ms
---------------------------------------------------
host3 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=14.5 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=15.5 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.535/16.549/19.541/2.162 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.4 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=14.6 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=18.2 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.643/16.458/18.278/1.491 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=15.3 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=14.1 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=13.1 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 13.171/14.234/15.334/0.883 ms
---------------------------------------------------
host4 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=21.9 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=21.3 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=14.6 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 14.619/19.284/21.932/3.310 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=14.8 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=18.4 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=14.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.148/15.786/18.409/1.879 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=15.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=18.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 15.117/16.788/18.962/1.612 ms
---------------------------------------------------
If bot spine have all BGP sessions and all ping succeeded, all works well
Congrats ! please check readme file to find some details
---------------------------------------------------

Nesevo Virtual Labs

2021-10-29T08:05:47Z

PierreLancastre: /* check ip route */

[[File:nesevo-labs-v2.png]]

= Nokia/Cumulus Multi-Vendor Lab =

The targets of this lab are to :

* mix Nokia SR-Linux and Cumulus VXLAN nodes, in spine or leaf roles
* make this environment work for different kinds of setup : cross-DC L2, differentiated L3 gateways per leaf per VRF, anycast gateway cross or intra DC, etc.
* spot compatibility issues if existing

To reach those goals, we'll use the Nokia lab tool "containerlab" https://containerlab.srlinux.dev/, labs from https://clabs.netdevops.me/ helped a lot. Many thanks to the contributors.

We'll need a Linux host with Docker installed on it + kvm virtualization enhancements activated (important if your host is a VM). If virtualization optimization is not ok, you won't be able to launch the most of devices.

Installation details are referenced there : https://containerlab.srlinux.dev/install/

Once installed, you'll need to register the different NOS images you'll use. Cumulus image is present on a public registrery, but Nokia/Arista/Juniper are not, you'll need a vendor support account to retreive them. Once downloaded (.tgz file typically) you can "docker load" the image :

For private images :<syntaxhighlight lang="bash">
sudo docker load -i 21.6.1-235.tar.xz /* (Nokia) */
sudo docker load -i cEOS64-lab-4.26.1F.tar /* (Arista) */
</syntaxhighlight>For public images :<syntaxhighlight lang="bash">
sudo docker pull networkop/cx:4.3.0 /* (Cumulus) */
</syntaxhighlight>

To verify your installed images, you can run "sudo docker images"<syntaxhighlight lang="bash">
plancastre@containerlab:~/00.LAB/senss$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
networkop/cx 4.3.0 a40541ed15ba 2 weeks ago 733MB
srlinux 21.6.1-235 e29aed30e83f 3 weeks ago 1.14GB
networkop/ignite dev dcc5eff29355 4 weeks ago 37.9MB
networkop/kernel 4.19 e794d8875f83 3 months ago 88MB
frrouting/frr v7.5.1 c3e13a4c5918 3 months ago 123MB

</syntaxhighlight>Once you've got all your images, to run the lab, you'll need a .yml file defining the topology to run.

In our lab, we've started with the 5-clos lab file from Nokia site, which we've forked.

We can see there is a branch "kinds" where we can define templates of nodes, in order to not rewrite whole parameters when creating nodes. For the images, it's important to take the repository name + tag.

'''Note:''' If the tag is "latest" it is not needed.

All supported "kinds" are referenced there : https://containerlab.srlinux.dev/manual/kinds/kinds/

'''Note:''' yml is indentation sensitive, so please respect the "double space" used to define hierarchy. Below the lab diagram + the yml file associated

== Infos on nodes ==

=== L3 infos ===
{| class="wikitable"
|+
!Device
!OS
!ASN
!VTEP IP
!swp1 or eth1/1
!swp2 or eth1/2
!swp3 or eth1/3
!SVI
|-
|client1
|Linux
|<none>
|<none>
|vlan100: 192.168.1.1/24
vlan200: 192.168.2.1/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client2
|Linux
|<none>
|<none>
|vlan100: 192.168.1.2/24
vlan200: 192.168.2.2/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client3
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.1/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|client4
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.2/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|leaf1
|CVX
|65011
|10.255.255.5/32
|10.254.254.8/31
|10.254.254.10/31
|trunk vlan 100,200
|vlan200 : 192.168.2.254/24
|-
|leaf2
|SRLinux
|65012
|10.255.255.6/32
|10.254.254.12/31
|10.254.254.14/31
|trunk vlan 100,200
|irb0.200 : 192.168.2.254/24
|-
|leaf3
|CVX
|65013
|10.255.255.7/32
|10.254.254.22/31
|10.254.254.24/31
|trunk vlan 100,200
|vlan200 : 192.168.200.254/24
|-
|leaf4
|SRLinux
|65014
|10.255.255.8/32
|10.254.254.26/31
|10.254.254.28/31
|trunk vlan 100,200
|irb0.200 : 192.168.200.254/24
|-
|spine1
|CVX
|65001
|10.255.255.1/32
|10.254.254.9/31
|10.254.254.13/31
|10.254.254.0/31
|<none>
|-
|spine2
|CVX
|65001
|10.255.255.2/32
|10.254.254.11/31
|10.254.254.15/31
|10.254.254.16/31
|<none>
|-
|spine3
|SRLinux
|65002
|10.255.255.3/32
|10.254.254.23/31
|10.254.254.27/31
|10.254.254.18/31
|<none>
|-
|spine4
|SRLinux
|65002
|10.255.255.4/32
|10.254.254.25/31
|10.254.254.29/31
|10.254.254.20/31
|<none>
|-
|superspine1
|CVX
|65003
|10.255.255.9/32
|10.254.254.1/31
|10.254.254.19/31
|<none>
|<none>
|-
|superspine2
|SRLinux
|65003
|10.255.255.10/32
|10.254.254.17/31
|10.254.254.21/31
|<none>
|<none>
|}

=== VXLAN/BGP infos ===
Note : in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>
{| class="wikitable"
|+
!service
!vlan id
!route-target
!VNI
!Commentaires
|-
|L2 cross-DC
|100
|target:65000:100
|100
|the 4 clients are in the same broadcast domain / subnet
|-
|L3VPN cros-DC
|200
|target:65000:200
target:65000:10200
|200 (for L2)
10200 (for routing)
|the 4 clients are separated in 2 different subnets. Traffic is routed within dedicated VNI
To be noted that in the current setup VMTO is not implemented, so cross DC traffic is not optimized.
|}

== Lab diagram ==
[[File:Lab-srl-diagram.png|ohne|mini|600x600px]]

== yml lab file ==
<syntaxhighlight lang="yaml">
# topology documentation: http://containerlab.srlinux.dev/lab-examples/min-5clos/
name: lab-srl-cum

topology:
kinds:
srl:
image: srlinux:21.6.1-235
linux:
image: ghcr.io/hellt/network-multitool
cvx:
image: networkop/cx:4.3.0
runtime: docker
nodes:
leaf1:
kind: cvx
leaf2:
kind: srl
type: ixrd2
leaf3:
kind: cvx
leaf4:
kind: srl
type: ixrd2

spine1:
kind: cvx
spine2:
kind: cvx
spine3:
kind: srl
type: ixr6
spine4:
kind: srl
type: ixr6

superspine1:
kind: cvx
superspine2:
kind: srl
type: ixr6

client1:
kind: linux
client2:
kind: linux
client3:
kind: linux
client4:
kind: linux

links:
# leaf to spine links POD1
- endpoints: ["leaf1:swp1", "spine1:swp1"]
- endpoints: ["leaf1:swp2", "spine2:swp1"]
- endpoints: ["leaf2:e1-1", "spine1:swp2"]
- endpoints: ["leaf2:e1-2", "spine2:swp2"]

# spine to superspine links POD1
- endpoints: ["spine1:swp3", "superspine1:swp1"]
- endpoints: ["spine2:swp3", "superspine2:e1-1"]

# leaf to spine links POD2
- endpoints: ["leaf3:swp1", "spine3:e1-1"]
- endpoints: ["leaf3:swp2", "spine4:e1-1"]
- endpoints: ["leaf4:e1-1", "spine3:e1-2"]
- endpoints: ["leaf4:e1-2", "spine4:e1-2"]

# spine to superspine links POD2
- endpoints: ["spine3:e1-3", "superspine1:swp2"]
- endpoints: ["spine4:e1-3", "superspine2:e1-2"]

# client connection links
- endpoints: ["client1:eth1", "leaf1:swp3"]
- endpoints: ["client2:eth1", "leaf2:e1-3"]
- endpoints: ["client3:eth1", "leaf3:swp3"]
- endpoints: ["client4:eth1", "leaf4:e1-3"]

</syntaxhighlight>

Once the nodes are defined, we have to define the needed connections between the nodes.

Each endpoint is defined as '<nowiki/>'''<nowiki/>'<node_name>:<interface_name><nowiki>''</nowiki>'''

When all is ok, you can launch in suduoer the topology.

For example :<syntaxhighlight lang="bash">
sudo containerlab deploy --topo topo.clab.yml --reconfigure

Note: The option "reconfigure" is used there in order to not have to destroy instances when updating topology/relaunching.
</syntaxhighlight>Another info regarding the yml file, it's possible to quickly generate files with the '''"containerlab gen -n 3tier --nodes 4,2,1"''' command for example. https://containerlab.srlinux.dev/cmd/generate/

To be noted that the "tier-x" notion in containerlab is inversed regarding the rfc 7938 (tier1 is superspine level into the rfc, tier1 is leaf level in nokia tool)

At the end of the topology launch, you can retreive the details :<syntaxhighlight lang="yaml">
INFO[0011] Writing /etc/hosts file
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| # | Name | Container ID | Image | Kind | Group | State | IPv4 Address | IPv6 Address |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| 1 | clab-lab-srl-cum-client1 | 25e2bbf6b13f | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
| 2 | clab-lab-srl-cum-client2 | 17d82d5a6e21 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.4/24 | 2001:172:20:20::4/64 |
| 3 | clab-lab-srl-cum-client3 | f8d78527c1b1 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.6/24 | 2001:172:20:20::6/64 |
| 4 | clab-lab-srl-cum-client4 | 913a0172d0c5 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 5 | clab-lab-srl-cum-leaf1 | 9ffc334af2b1 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.9/24 | 2001:172:20:20::9/64 |
| 6 | clab-lab-srl-cum-leaf2 | 7d3eb855d749 | srlinux:21.6.1-235 | srl | | running | 172.20.20.14/24 | 2001:172:20:20::e/64 |
| 7 | clab-lab-srl-cum-leaf3 | e0d36ffc606a | networkop/cx:4.3.0 | cvx | | running | 172.20.20.7/24 | 2001:172:20:20::7/64 |
| 8 | clab-lab-srl-cum-leaf4 | e6753b1d099e | srlinux:21.6.1-235 | srl | | running | 172.20.20.12/24 | 2001:172:20:20::c/64 |
| 9 | clab-lab-srl-cum-spine1 | a7e63993d052 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.5/24 | 2001:172:20:20::5/64 |
| 10 | clab-lab-srl-cum-spine2 | 3124250d5b75 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
| 11 | clab-lab-srl-cum-spine3 | 5e300ca3f66f | srlinux:21.6.1-235 | srl | | running | 172.20.20.11/24 | 2001:172:20:20::b/64 |
| 12 | clab-lab-srl-cum-spine4 | 0a59e60a596b | srlinux:21.6.1-235 | srl | | running | 172.20.20.13/24 | 2001:172:20:20::d/64 |
| 13 | clab-lab-srl-cum-superspine1 | 3d9935413fc2 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.8/24 | 2001:172:20:20::8/64 |
| 14 | clab-lab-srl-cum-superspine2 | 5bc58d423e2c | srlinux:21.6.1-235 | srl | | running | 172.20.20.15/24 | 2001:172:20:20::f/64 |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
</syntaxhighlight>The containerlab program creates the updates into the host file + generates an ansible inventory file.

There are different ways to connect onto the docker instances, within docker exec commands or SSH directly the IP address related above.

For example :<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-spine1 bash
</syntaxhighlight>-->>> allows to launch a bash on the cumulus device (then you run the different net commands)<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-leaf2 sr_cli
</syntaxhighlight>-->>> allows to launch the cli on a SRLinux device

To end this setup preparation, the containerlab tool integrates an option to launch a webserver showing the lab topology in graph + node details

By default, it's listening on port 50080 of the host

Then, you can access to the following web page showing the topology + lab infos

[[File:Lab graph tool.jpg|ohne|mini|900x900px]]

Finally, if you want to stop a lab, please run '''<nowiki>''</nowiki>containerlab destroy --topo <topology>.yml<nowiki>''</nowiki>'''

At this step, nodes are running, connections are done, but configurations are empty.

'''So, it's time to configure them :)'''

Configuration files are attached to this blog. Once validated, they are exportable and can be called by the tool when launching the lab. Below how to do it :

Firstable, once you ve finished your "hand made config", you can run the following command to save it :<syntaxhighlight lang="bash">
containerlab save -t quickstart.clab.yml
</syntaxhighlight>'''To be noted it's only working for Nokia nodes in this lab'''. CVX configurations will have to be called by another method (or pushed manually)

It will create one folder per node into the lab folder which gives the following tree :
[[File:Topology-srl.jpg|ohne|mini]]

'''Note''': For Cumulus, you can apparently retrieve configurations files within adding the following parameters into lab config file. '''To be tested.''' some lab samples here : https://github.com/hellt/clabs/blob/main/labs/cvx04/symm-mh.clab.yml

Now we have the lab running, below the different commands to check that all is working as expected : <syntaxhighlight lang="yaml">
-------------------------------------------
Check L2 tables ///
-------------------------------------------

Nokia :

A:leaf2# show network-instance mac-vrf-100 bridge-table mac-table all
----------------------------------------------------------------------------------------------------------------------------------------------------------
Mac-table of network instance mac-vrf-100
----------------------------------------------------------------------------------------------------------------------------------------------------------
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
| Address | Destination | Dest Index | Type | Active | Aging | Last Update |
+====================+=========================================+============+=============+=========+========+=========================================+
| AA:C1:AB:0C:60:D6 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T13:06:17.000Z |
| | vtep:10.255.255.8 vni:100 | 67 | | | | |
| AA:C1:AB:23:31:04 | ethernet-1/3.100 | 2 | learnt | true | 300 | 2021-10-08T13:06:10.000Z |
| AA:C1:AB:4B:FF:71 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:55:28.000Z |
| | vtep:10.255.255.5 vni:100 | 60 | | | | |
| AA:C1:AB:B5:79:50 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:43:37.000Z |
| | vtep:10.255.255.7 vni:100 | 64 | | | | |
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 0 Total 0 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 1 Total 1 Active
Total Evpn Macs : 3 Total 3 Active
Total Evpn static Macs : 0 Total 0 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 4 Total 4 Active
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : type learnt matches the local learnt mac, evpn are learnt within VXLAN

Cumulus :

root@leaf1-cum-dc1:/# net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
100 bridge swp3 aa:c1:ab:4b:ff:71 00:00:02
100 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:17:21
100 bridge vni100 22:2d:67:f1:bb:26 permanent 02:17:21
100 bridge vni100 aa:c1:ab:0c:60:d6 extern_learn <1 sec
100 bridge vni100 aa:c1:ab:23:31:04 extern_learn 00:00:07
100 bridge vni100 aa:c1:ab:b5:79:50 extern_learn 00:13:28

root@leaf1-cum-dc1:/# net show bridge macs vlan 200

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
200 bridge swp3 aa:c1:ab:4b:ff:71 00:08:46
200 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:16:36
200 bridge vni200 00:00:5e:00:01:01 static sticky 00:14:26
200 bridge vni200 1e:0d:aa:46:53:b1 permanent 02:16:36
200 bridge vni200 02:77:6b:ff:00:41 static sticky 01:00:52
200 bridge vni200 02:f8:b8:ff:00:41 static sticky 00:14:26
200 bridge vni200 aa:c1:ab:b5:79:50 extern_learn 00:11:51
200 bridge vni10200 02:77:6b:ff:00:00 extern_learn 00:53:02
200 bridge vni10200 02:f8:b8:ff:00:00 extern_learn 00:14:26
200 bridge vni10200 86:5e:aa:ca:50:be permanent 02:16:36

Note : the macs with permanent flag are local interfaces, physical ou logic (vtep)
Note2 : extern_learn mac are the one learned via the VXLAN infra (and mapped to the vnixxx interface)

-------------------------------------------
Check arp tables ///
-------------------------------------------

Nokia :

:leaf2# show arpnd arp-entries
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| ethernet-1/1 | 0 | 10.254.254.13 | dynamic | AA:C1:AB:59:65:EA | 3 hours from now |
| ethernet-1/2 | 0 | 10.254.254.15 | dynamic | AA:C1:AB:CB:F8:8A | 3 hours from now |
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
| mgmt0 | 0 | 172.20.20.1 | dynamic | 02:42:63:9E:74:03 | 2 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 5 (0 static, 5 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : on Nokia you have to sort the entries basing on subinterface and the retrieve the L3VPN attached

A:leaf2# show arpnd arp-entries subinterface 200
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 2 (0 static, 2 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Cumulus :

Here we retrieve all arp infos linked to evpn

root@leaf1-cum-dc1:/# net show evpn arp-cache vni all
VNI 100 #ARP (IPv4 and IPv6, local and remote) 2

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.1.3 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0
192.168.1.1 local active aa:c1:ab:4b:ff:71 0/0

VNI 200 #ARP (IPv4 and IPv6, local and remote) 4

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.2.1 local active aa:c1:ab:4b:ff:71 0/0
192.168.200.254 remote active 00:00:5e:00:01:01 10.255.255.8 0/0
192.168.2.254 remote active 00:00:5e:00:01:01 10.255.255.6 0/0
192.168.200.1 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0

VNI 400 #ARP (IPv4 and IPv6, local and remote) 0

-------------------------------------------
Check bgp sessions' status - underlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep ipv4
| default | 10.254.254.19 | udl-superspine | S | 65003 | established | 0d:0h:22m:44s | ipv4-unicast | [13/4/4] |
| default | 10.254.254.22 | udl-leaf | S | 65013 | established | 0d:0h:22m:52s | ipv4-unicast | [13/1/4] |

Cumulus :

root@leaf3:/# net show bgp ipv4 unicast summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 13
RIB entries 25, using 4800 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.254.254.23 4 65002 466 403 0 0 0 00:19:34 4 13
10.254.254.25 4 65002 434 355 0 0 0 00:17:10 8 13

-------------------------------------------
Check bgp sessions' status - overlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep evpn
| default | 10.255.255.7 | ovl-leaf | S | 65013 | established | 0d:0h:24m:14s| evpn | [15/0/11] |
| default | 10.255.255.9 | ovl-superspine | S | 65003 | established | 0d:0h:24m:7s | evpn | [11/0/4] |

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.255.255.3 4 65002 5618 4742 0 0 0 03:53:35 19 27
10.255.255.4 4 65002 5565 4694 0 0 0 03:51:11 19 27

-------------------------------------------
Check EVPN L2 tables///
-------------------------------------------

Nokia :

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn route type 2
BGP table version is 17, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:100
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
Route Distinguisher: 10.255.255.5:200
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
Route Distinguisher: 10.255.255.6:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
Route Distinguisher: 10.255.255.7:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100
Route Distinguisher: 10.255.255.7:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.200.1]
10.255.255.7 32768 i
ET:8 RT:65000:200
Route Distinguisher: 10.255.255.8:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC

and if we focus on our vni 100

root@leaf3-cum-dc1:/# net show bgp evpn route vni 100 type 2
BGP table version is 48, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100

The additional route type "MAC+IP" is optional and is useful/mandatory when you do anycast irb.

-------------------------------------------
Check tables EVPN L3 ///
-------------------------------------------

Nokia :

--{ + running }--[ ]--
A:leaf2# show network-instance ip-vrf-200 route-table all
-----------------------------------------------------------------------------------------------------------------------------------------------------------
IPv4 Unicast route table of network instance ip-vrf-200
-----------------------------------------------------------------------------------------------------------------------------------------------------------
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
| Prefix | ID | Route Type | Route Owner | Best/Fib- | Metric | Pref | Next-hop | Next-hop |
| | | | | status(slot) | | | (Type) | Interface |
+=======================+=======+============+======================+======================+==========+=========+==============+==============+
| 192.168.2.0/24 | 9 | local | net_inst_mgr | True/success | 0 | 0 | 192.168.2.25 | irb0.200 |
| | | | | | | | 4 (direct) | |
| 192.168.2.254/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (extract) | |
| 192.168.2.255/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (broadcast) | |
| 192.168.200.0/24 | 0 | bgp-evpn | bgp_evpn_mgr | True/success | 0 | 170 | 10.255.255.8 | None |
| | | | | | | | (indirect/vx | |
| | | | | | | | lan) | |
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
-----------------------------------------------------------------------------------------------------------------------------------------------------------
4 IPv4 routes total
4 IPv4 prefixes with active routes
0 IPv4 prefixes with active ECMP routes
-----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--
A:leaf2#

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf1-cum-dc1:/# net show route vrf ip-vrf-200
show ip route vrf ip-vrf-200
=============================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF ip-vrf-200:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 04:48:42
B 192.168.2.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 04:27:26
C>* 192.168.2.0/24 is directly connected, vlan200, 04:48:42
B>* 192.168.200.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:48:49

root@leaf1-cum-dc1:/# net show bgp evpn route type 5
BGP table version is 3, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.6:10200
* [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
*> [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
Route Distinguisher: 10.255.255.8:10200
*> [5]:[0]:[24]:[192.168.200.0]
10.255.255.8 0 65001 65003 65002 65014 i
RT:65000:10200 ET:8 Rmac:02:f8:b8:ff:00:00

-------------------------------------------
Check arp hosts ///
-------------------------------------------

From client #1 /

bash-5.0# arp
Address HWtype HWaddress Flags Mask Iface
192.168.1.3 ether aa:c1:ab:b5:79:50 C eth1.100
192.168.1.4 ether aa:c1:ab:0c:60:d6 C eth1.100
192.168.2.2 ether aa:c1:ab:23:31:04 C eth1.200
192.168.2.254 ether aa:c1:ab:4f:f2:e7 C eth1.200
192.168.1.2 ether aa:c1:ab:23:31:04 C eth1.100
containerlab.local ether 02:42:63:9e:74:03 C eth0

-------------------------------------------
Ping ///
-------------------------------------------

From client #1 /

L2 service ping (vlan 100) ----------------------

bash-5.0# ping -c 1 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.812 ms

--- 192.168.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.812/0.812/0.812/0.000 MsgRcvd
bash-5.0# ping -c 1 192.168.1.3
PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_seq=1 ttl=64 time=1.40 ms

--- 192.168.1.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.403/1.403/1.403/0.000 ms
bash-5.0# ping -c 1 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=1.62 ms

--- 192.168.1.4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.619/1.619/1.619/0.000 ms

L3 service ping (vlan 200)

bash-5.0# ping -c 1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.053 ms

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms
bash-5.0# ^C
bash-5.0# ping -c 1 192.168.2.2
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.611 ms

--- 192.168.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.611/0.611/0.611/0.000 ms

-------------------------------------------
Traceroute ///
-------------------------------------------

L3 service traceroute (vlan 200)

bash-5.0# traceroute 192.168.200.1
traceroute to 192.168.200.1 (192.168.200.1), 30 hops max, 46 byte packets
1 192.168.2.254 (192.168.2.254) 0.012 ms 0.010 ms 0.007 ms
2 192.168.200.254 (192.168.200.254) 2.751 ms * 2.549 ms
3 192.168.200.1 (192.168.200.1) 2.084 ms 1.805 ms 1.761 ms

</syntaxhighlight>To come back on the setup, we wanted to implement solutions inspired by the <nowiki>RFC 7938</nowiki> rfc7938 (see section 5.2.1)

* eBGP will be used for Underlay and Overlay
* eBGP single-hop sessions are established over direct point-to-point links interconnecting the network nodes, no multi-hop or loopback sessions are used, even in the case of multiple links between the same pair of nodes.
* Private Use ASNs from the range 64512-65534 are used to avoid ASN conflicts.
* A single ASN is allocated to all of the Clos topology's Tier 1 devices.
* A unique ASN is allocated to each set of Tier 2 devices in the same cluster.
* A unique ASN is allocated to every Tier 3 device (e.g., ToR) in this topology.

In add, depending of network nodes, it will be interesting to :

* Add fast-failover option for underlay + add link-debounce on links to limit possible impacts in case of link flaps. BFD could be considered in some cases
* Have the best ECMP hashing algorythm on leaf nodes. On tier-1/tier-2 levels hashing will be based on underlay part (trafic inter-vtep), so L3-L4 decision, but some models can retreive the vni-id if I well remember.
* Implement the "multipath relax" since we could have multiple AS paths for a same destination.
* Implement gshut community on underlay (on tier-1/tier-2) or at least policies to prepend underlay routes to avoid some nodes in case of maintenance.

== Thoughts/Points to deepend ==
On leaf nodes, LACP over multiple nodes is feasible within ESI, but impact on incidents/maintenance can vary depending on the vendors.

'''Tests need to be done to check that.'''

Some sysadmin would like to KISS and have client servers connected in active/passive bonding, with eventually single-leaf ports aggregation.

'''Note :''' NH ESI in FIB management on ingress nodes can have some limitations (numbers of installable next-hop, load-balancing, etc. oftenly linked to the node chipset)

Another important engineering point to define at the beginning is : '''Do we authorize multisite L2 propagation or not ?'''

From my point-of-view, the smaller are the L2 segments the better it is. I would suggest Local L2 segments wit anycast gateways provided by Leaf switches, with L3 routing between DCs within EVPN type5 routes sharing only. Nevertheless, some business requirements for endcustomers can keep extended L2 alive.

We have to be careful when implementing Consistent hashing. ex: <nowiki>https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-44/Layer-3/Routing/Equal-Cost-Multipath-Load-Sharing-Hardware-ECMP/</nowiki>

If possible activate it per VRF, not globally, and restrict on dedicated subnets << depends of the vendor, it can be dangerous regarding memory management

Think at configuring the bgp option to not advertise routes to peers before they are installed in FIB.

We can implement leaf roles in some cases, to be able to do "light traffic engineering " on DCI links in some setup.

These roles will be managed within bgp communities added to VTEP /32. Look at the diagram below. Depending of the colour, the eBGP sessions between SupS will prepend or not the VTEP IPs announced to the remote DC

[[File:Fabric-roles.jpg|ohne|mini|1200x1200px]]

<blockquote>'''To be continued'''</blockquote>

== Lab archives ==
<blockquote>Reminder : following config are for lab, so please don't pay attention on typos/lack of security params/unused config. Thanks \o/</blockquote>

=== Credentials ===
<syntaxhighlight lang="bash">
SRLinux : admin/<no_password>

CVX : cumulus/cumulus
</syntaxhighlight>

=== Lab archives ===
layer 2 (vlan 100) + layer 3 (vlan 200) lab, 10/12/2021
[[File:20211008 lab-nokia-cvx-containerlab.zip|ohne|mini]]
<u>If you want to exchange about that lab, those topics, please don't hesitate to contact us</u>

= SONiC Lab =

<span><to be noted that those informations are included into the readme.md file into the archive></span>

== Lab diagram ==
[[File:SONIC_LAB.jpg]]

<div></div>Lab archive [http://wiki.intra.nesevo.com/uploads/LAB/sonic-lab-20201030.zip Attach:sonic-lab-20201030.zip]

This lab is a forked lab from many found on @, starting from Microsoft one : https://github.com/Azure/SONiC/wiki/SONiC-P4-Software-Switch

The target of this lab is to simulate a simple IP/Fabric in SPINE/LEAF topology. Each Leaf provide local L2 connectivity + L3 gateways to hosts. Flows between hosts from different leaf are L3 routed, in a "node path" LEAF-SPINE-LEAF.

Nothing complicated, but it allows to put a first foot into SONiC world :)

For the moment, as far as we know, MC-LAG + EVPN/VXLAN features are only available on entreprise SONiC versions. Tests to come once physic switches in enterprise version will be available in our lab.

The diagram of the architecture is into the zip file (SONIC_LAB.jpg)

To run the lab you need :

- to have installed docker by using ./install_docker_ovs.sh - to run the ./load_image.sh in order to get the Sonic image

Note : don't change file / folders names of network devices (leafX, spineX)

Once you're ready, please run the lab by launching "./start.sh"

The script :<div></div>
* launches the containers without Docket network features
* create network namespaces
* build linux bridges to interconnect the containers
* configure host IPs
* launch startup sonic scripts into sonic devices
* once devices started, checks + ping tests are run. If all BGP sessions are ok on spine + all ping are fine, the setup works well.
The "start" script has several comments, perhaps without the right vocabulary, so please be tolerant :)

If you want to stop the lab, please launch "./stop.sh"

Below some useful commands to run for sonic containers :

=== <span>run ping</span> ===
sudo docker exec -it <container> ping <ip>

This command triggers ping into the targetted container :
<div></div>
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it leaf1 ping 192.168.1.1

PING 192.168.1.1 (192.168.1.1): 56 data bytes

64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.052 ms

64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.034 ms

^C--- 192.168.1.1 ping statistics ---

=== <span>check bgp sessions</span> ===
sudo docker exec -it <container> vtysh -c "show ip bgp sum"

This command sollicitates the SONiC shell of the targeted container, in order to print the bgp status.

Below, we run it on one spine, you should have 4 sessiosn UP (for the 4 leaves)

plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip bgp sum"

BGP router identifier 10.1.0.1, local AS number 65001

RIB entries 8, using 896 bytes of memory

Peers 4, using 18 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

10.1.0.0 4 65101 6409 6411 0 0 0 01:46:46 3

10.1.0.2 4 65102 6406 6410 0 0 0 01:46:44 3

10.1.0.4 4 65103 6407 6410 0 0 0 01:46:44 4

10.1.0.6 4 65104 6406 6412 0 0 0 01:46:46 1

Total number of neighbors 4

=== <span>check ip route</span> ===<div></div>
sudo docker exec -it <container> vtysh -c "show ip route"
This command sollicitates the SONiC shell of the targeted container, in order to print its routing table?
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,
> - selected route, * - FIB route
C>* 10.1.0.0/31 is directly connected, Vlan11
C>* 10.1.0.2/31 is directly connected, Vlan21
C>* 10.1.0.4/31 is directly connected, Vlan31
C>* 10.1.0.6/31 is directly connected, Vlan41
C>* 127.0.0.0/8 is directly connected, lo
B>* 192.168.1.0/24 [20/0] via 10.1.0.0, Vlan11, 01:47:23
B>* 192.168.2.0/24 [20/0] via 10.1.0.2, Vlan21, 01:47:21
B>* 192.168.3.0/24 [20/0] via 10.1.0.4, Vlan31, 01:47:21
B>* 192.168.4.0/24 [20/0] via 10.1.0.6, Vlan41, 01:47:23

For more commands check SONiC docs : https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md

To finish, below the output of the start.sh when all goes well<div></div>
plancastre@dockerhost:~/00.Projects/p4-version-pla$ ./start.sh
---------------------------------------------------
Creating the containers: starting
---------------------------------------------------
0b4bbb3c6cfc9895466ea13a9e940d1a176d66f45deee9752e9b4732d735bb03
5ca9486baab3e8a5259062167132a944eee717875d3cc1ef29fe490cba3184c1
46c9c09dba27a32715b9fbe0bc434a9284b9d59317d278df693f425cb669f037
898e6afbcb2977115acbba2a69d069796cfd70fb90228b6c18e09ba4584365f7
eb9b3ef6d14784c9ffe4d9d5df3e921a24559889cb8770d226263aad6761182a
ae5f3b2c628da687eaabb331dc2cd989e71bcb386854e7a0be6ec587df99d49a
122197bffd7566a01ce3b6565fdfb6eb893aab7d5236f14e5a9bbbe95037ced2
a604520a080e2a2184598ad1d915c0c182d16e5185033373e95a4772d8978a67
0c49e18d78b2aa7c8ef8e937b60285aa56ec0bf4a8530226285270a085e11d5c
eb66336b8c6bd8333f1f4f685369e6cac973b817c35bb7126ac4d7ba72ef71bf
---------------------------------------------------
Retreiving PID from docker containers
---------------------------------------------------
---------------------------------------------------
Creating network namespaces for containers
---------------------------------------------------
List of created namespaces
ns_host4
ns_host3
ns_host2
ns_host1
ns_spine2
ns_spine1
ns_leaf4
ns_leaf3
ns_leaf2
ns_leaf1
---------------------------------------------------
Creating network connectivity between containers
---------------------------------------------------
---------------------------------------------------
Now assigning IPs to hosts (host1 to host4)
---------------------------------------------------
---------------------------------------------------
Configuration of containers is now finished
---------------------------------------------------
---------------------------------------------------
Now booting switches, please wait 2 minutes for switches to load
---------------------------------------------------
---------------------------------------------------
2 minutes period is finished
---------------------------------------------------
---------------------------------------------------
Now fixing the iptables
---------------------------------------------------
---------------------------------------------------
Backend actions are now finished
---------------------------------------------------
---------------------------------------------------
Now we'll check in 15 sec that BGP peers are all Up
---------------------------------------------------
---------------------------------------------------
SPINE1 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 94 97 0 0 0 00:01:28 4
10.1.0.2 4 65102 95 97 0 0 0 00:01:28 5
10.1.0.4 4 65103 95 97 0 0 0 00:01:28 5
10.1.0.6 4 65104 95 97 0 0 0 00:01:28 5
Total number of neighbors 4
---------------------------------------------------
SPINE2 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.2.0.1, local AS number 65002
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.2.0.0 4 65101 93 96 0 0 0 00:01:29 2
10.2.0.2 4 65102 94 100 0 0 0 00:01:32 1
10.2.0.4 4 65103 94 100 0 0 0 00:01:32 1
10.2.0.6 4 65104 94 100 0 0 0 00:01:32 1
Total number of neighbors 4
---------------------------------------------------
Now if all is ok, we'll do some ping tests
---------------------------------------------------
---------------------------------------------------
host1 ping its gateway
---------------------------------------------------
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=17.6 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=9.14 ms
--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 9.145/13.390/17.636/4.247 ms
---------------------------------------------------
host2 ping its gateway
---------------------------------------------------
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=23.6 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=6.21 ms
--- 192.168.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 6.218/14.950/23.683/8.733 ms
---------------------------------------------------
host3 ping its gateway
---------------------------------------------------
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=18.8 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=12.7 ms
--- 192.168.3.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.716/15.764/18.812/3.048 ms
---------------------------------------------------
host4 ping its gateway
---------------------------------------------------
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.
64 bytes from 192.168.4.1: icmp_seq=1 ttl=64 time=17.7 ms
64 bytes from 192.168.4.1: icmp_seq=2 ttl=64 time=4.46 ms
--- 192.168.4.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.464/11.107/17.750/6.643 ms
---------------------------------------------------
host1 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=16.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.049/17.253/19.532/1.615 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=15.5 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=14.3 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.312/15.592/16.939/1.083 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=20.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=16.8 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.048/17.871/20.739/2.058 ms
---------------------------------------------------
host2 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=16.6 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.222/16.583/16.924/0.286 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.6 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=18.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=26.8 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.620/20.549/26.887/4.524 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=33.8 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=27.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=20.2 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 20.216/27.259/33.818/5.566 ms
---------------------------------------------------
host3 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=14.5 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=15.5 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.535/16.549/19.541/2.162 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.4 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=14.6 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=18.2 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.643/16.458/18.278/1.491 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=15.3 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=14.1 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=13.1 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 13.171/14.234/15.334/0.883 ms
---------------------------------------------------
host4 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=21.9 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=21.3 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=14.6 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 14.619/19.284/21.932/3.310 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=14.8 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=18.4 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=14.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.148/15.786/18.409/1.879 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=15.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=18.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 15.117/16.788/18.962/1.612 ms
---------------------------------------------------
If bot spine have all BGP sessions and all ping succeeded, all works well
Congrats ! please check readme file to find some details
---------------------------------------------------

Nesevo Virtual Labs

2021-10-29T08:04:33Z

PierreLancastre: /* SONiC Lab */

[[File:nesevo-labs-v2.png]]

= Nokia/Cumulus Multi-Vendor Lab =

The targets of this lab are to :

* mix Nokia SR-Linux and Cumulus VXLAN nodes, in spine or leaf roles
* make this environment work for different kinds of setup : cross-DC L2, differentiated L3 gateways per leaf per VRF, anycast gateway cross or intra DC, etc.
* spot compatibility issues if existing

To reach those goals, we'll use the Nokia lab tool "containerlab" https://containerlab.srlinux.dev/, labs from https://clabs.netdevops.me/ helped a lot. Many thanks to the contributors.

We'll need a Linux host with Docker installed on it + kvm virtualization enhancements activated (important if your host is a VM). If virtualization optimization is not ok, you won't be able to launch the most of devices.

Installation details are referenced there : https://containerlab.srlinux.dev/install/

Once installed, you'll need to register the different NOS images you'll use. Cumulus image is present on a public registrery, but Nokia/Arista/Juniper are not, you'll need a vendor support account to retreive them. Once downloaded (.tgz file typically) you can "docker load" the image :

For private images :<syntaxhighlight lang="bash">
sudo docker load -i 21.6.1-235.tar.xz /* (Nokia) */
sudo docker load -i cEOS64-lab-4.26.1F.tar /* (Arista) */
</syntaxhighlight>For public images :<syntaxhighlight lang="bash">
sudo docker pull networkop/cx:4.3.0 /* (Cumulus) */
</syntaxhighlight>

To verify your installed images, you can run "sudo docker images"<syntaxhighlight lang="bash">
plancastre@containerlab:~/00.LAB/senss$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
networkop/cx 4.3.0 a40541ed15ba 2 weeks ago 733MB
srlinux 21.6.1-235 e29aed30e83f 3 weeks ago 1.14GB
networkop/ignite dev dcc5eff29355 4 weeks ago 37.9MB
networkop/kernel 4.19 e794d8875f83 3 months ago 88MB
frrouting/frr v7.5.1 c3e13a4c5918 3 months ago 123MB

</syntaxhighlight>Once you've got all your images, to run the lab, you'll need a .yml file defining the topology to run.

In our lab, we've started with the 5-clos lab file from Nokia site, which we've forked.

We can see there is a branch "kinds" where we can define templates of nodes, in order to not rewrite whole parameters when creating nodes. For the images, it's important to take the repository name + tag.

'''Note:''' If the tag is "latest" it is not needed.

All supported "kinds" are referenced there : https://containerlab.srlinux.dev/manual/kinds/kinds/

'''Note:''' yml is indentation sensitive, so please respect the "double space" used to define hierarchy. Below the lab diagram + the yml file associated

== Infos on nodes ==

=== L3 infos ===
{| class="wikitable"
|+
!Device
!OS
!ASN
!VTEP IP
!swp1 or eth1/1
!swp2 or eth1/2
!swp3 or eth1/3
!SVI
|-
|client1
|Linux
|<none>
|<none>
|vlan100: 192.168.1.1/24
vlan200: 192.168.2.1/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client2
|Linux
|<none>
|<none>
|vlan100: 192.168.1.2/24
vlan200: 192.168.2.2/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client3
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.1/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|client4
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.2/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|leaf1
|CVX
|65011
|10.255.255.5/32
|10.254.254.8/31
|10.254.254.10/31
|trunk vlan 100,200
|vlan200 : 192.168.2.254/24
|-
|leaf2
|SRLinux
|65012
|10.255.255.6/32
|10.254.254.12/31
|10.254.254.14/31
|trunk vlan 100,200
|irb0.200 : 192.168.2.254/24
|-
|leaf3
|CVX
|65013
|10.255.255.7/32
|10.254.254.22/31
|10.254.254.24/31
|trunk vlan 100,200
|vlan200 : 192.168.200.254/24
|-
|leaf4
|SRLinux
|65014
|10.255.255.8/32
|10.254.254.26/31
|10.254.254.28/31
|trunk vlan 100,200
|irb0.200 : 192.168.200.254/24
|-
|spine1
|CVX
|65001
|10.255.255.1/32
|10.254.254.9/31
|10.254.254.13/31
|10.254.254.0/31
|<none>
|-
|spine2
|CVX
|65001
|10.255.255.2/32
|10.254.254.11/31
|10.254.254.15/31
|10.254.254.16/31
|<none>
|-
|spine3
|SRLinux
|65002
|10.255.255.3/32
|10.254.254.23/31
|10.254.254.27/31
|10.254.254.18/31
|<none>
|-
|spine4
|SRLinux
|65002
|10.255.255.4/32
|10.254.254.25/31
|10.254.254.29/31
|10.254.254.20/31
|<none>
|-
|superspine1
|CVX
|65003
|10.255.255.9/32
|10.254.254.1/31
|10.254.254.19/31
|<none>
|<none>
|-
|superspine2
|SRLinux
|65003
|10.255.255.10/32
|10.254.254.17/31
|10.254.254.21/31
|<none>
|<none>
|}

=== VXLAN/BGP infos ===
Note : in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>
{| class="wikitable"
|+
!service
!vlan id
!route-target
!VNI
!Commentaires
|-
|L2 cross-DC
|100
|target:65000:100
|100
|the 4 clients are in the same broadcast domain / subnet
|-
|L3VPN cros-DC
|200
|target:65000:200
target:65000:10200
|200 (for L2)
10200 (for routing)
|the 4 clients are separated in 2 different subnets. Traffic is routed within dedicated VNI
To be noted that in the current setup VMTO is not implemented, so cross DC traffic is not optimized.
|}

== Lab diagram ==
[[File:Lab-srl-diagram.png|ohne|mini|600x600px]]

== yml lab file ==
<syntaxhighlight lang="yaml">
# topology documentation: http://containerlab.srlinux.dev/lab-examples/min-5clos/
name: lab-srl-cum

topology:
kinds:
srl:
image: srlinux:21.6.1-235
linux:
image: ghcr.io/hellt/network-multitool
cvx:
image: networkop/cx:4.3.0
runtime: docker
nodes:
leaf1:
kind: cvx
leaf2:
kind: srl
type: ixrd2
leaf3:
kind: cvx
leaf4:
kind: srl
type: ixrd2

spine1:
kind: cvx
spine2:
kind: cvx
spine3:
kind: srl
type: ixr6
spine4:
kind: srl
type: ixr6

superspine1:
kind: cvx
superspine2:
kind: srl
type: ixr6

client1:
kind: linux
client2:
kind: linux
client3:
kind: linux
client4:
kind: linux

links:
# leaf to spine links POD1
- endpoints: ["leaf1:swp1", "spine1:swp1"]
- endpoints: ["leaf1:swp2", "spine2:swp1"]
- endpoints: ["leaf2:e1-1", "spine1:swp2"]
- endpoints: ["leaf2:e1-2", "spine2:swp2"]

# spine to superspine links POD1
- endpoints: ["spine1:swp3", "superspine1:swp1"]
- endpoints: ["spine2:swp3", "superspine2:e1-1"]

# leaf to spine links POD2
- endpoints: ["leaf3:swp1", "spine3:e1-1"]
- endpoints: ["leaf3:swp2", "spine4:e1-1"]
- endpoints: ["leaf4:e1-1", "spine3:e1-2"]
- endpoints: ["leaf4:e1-2", "spine4:e1-2"]

# spine to superspine links POD2
- endpoints: ["spine3:e1-3", "superspine1:swp2"]
- endpoints: ["spine4:e1-3", "superspine2:e1-2"]

# client connection links
- endpoints: ["client1:eth1", "leaf1:swp3"]
- endpoints: ["client2:eth1", "leaf2:e1-3"]
- endpoints: ["client3:eth1", "leaf3:swp3"]
- endpoints: ["client4:eth1", "leaf4:e1-3"]

</syntaxhighlight>

Once the nodes are defined, we have to define the needed connections between the nodes.

Each endpoint is defined as '<nowiki/>'''<nowiki/>'<node_name>:<interface_name><nowiki>''</nowiki>'''

When all is ok, you can launch in suduoer the topology.

For example :<syntaxhighlight lang="bash">
sudo containerlab deploy --topo topo.clab.yml --reconfigure

Note: The option "reconfigure" is used there in order to not have to destroy instances when updating topology/relaunching.
</syntaxhighlight>Another info regarding the yml file, it's possible to quickly generate files with the '''"containerlab gen -n 3tier --nodes 4,2,1"''' command for example. https://containerlab.srlinux.dev/cmd/generate/

To be noted that the "tier-x" notion in containerlab is inversed regarding the rfc 7938 (tier1 is superspine level into the rfc, tier1 is leaf level in nokia tool)

At the end of the topology launch, you can retreive the details :<syntaxhighlight lang="yaml">
INFO[0011] Writing /etc/hosts file
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| # | Name | Container ID | Image | Kind | Group | State | IPv4 Address | IPv6 Address |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| 1 | clab-lab-srl-cum-client1 | 25e2bbf6b13f | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
| 2 | clab-lab-srl-cum-client2 | 17d82d5a6e21 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.4/24 | 2001:172:20:20::4/64 |
| 3 | clab-lab-srl-cum-client3 | f8d78527c1b1 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.6/24 | 2001:172:20:20::6/64 |
| 4 | clab-lab-srl-cum-client4 | 913a0172d0c5 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 5 | clab-lab-srl-cum-leaf1 | 9ffc334af2b1 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.9/24 | 2001:172:20:20::9/64 |
| 6 | clab-lab-srl-cum-leaf2 | 7d3eb855d749 | srlinux:21.6.1-235 | srl | | running | 172.20.20.14/24 | 2001:172:20:20::e/64 |
| 7 | clab-lab-srl-cum-leaf3 | e0d36ffc606a | networkop/cx:4.3.0 | cvx | | running | 172.20.20.7/24 | 2001:172:20:20::7/64 |
| 8 | clab-lab-srl-cum-leaf4 | e6753b1d099e | srlinux:21.6.1-235 | srl | | running | 172.20.20.12/24 | 2001:172:20:20::c/64 |
| 9 | clab-lab-srl-cum-spine1 | a7e63993d052 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.5/24 | 2001:172:20:20::5/64 |
| 10 | clab-lab-srl-cum-spine2 | 3124250d5b75 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
| 11 | clab-lab-srl-cum-spine3 | 5e300ca3f66f | srlinux:21.6.1-235 | srl | | running | 172.20.20.11/24 | 2001:172:20:20::b/64 |
| 12 | clab-lab-srl-cum-spine4 | 0a59e60a596b | srlinux:21.6.1-235 | srl | | running | 172.20.20.13/24 | 2001:172:20:20::d/64 |
| 13 | clab-lab-srl-cum-superspine1 | 3d9935413fc2 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.8/24 | 2001:172:20:20::8/64 |
| 14 | clab-lab-srl-cum-superspine2 | 5bc58d423e2c | srlinux:21.6.1-235 | srl | | running | 172.20.20.15/24 | 2001:172:20:20::f/64 |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
</syntaxhighlight>The containerlab program creates the updates into the host file + generates an ansible inventory file.

There are different ways to connect onto the docker instances, within docker exec commands or SSH directly the IP address related above.

For example :<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-spine1 bash
</syntaxhighlight>-->>> allows to launch a bash on the cumulus device (then you run the different net commands)<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-leaf2 sr_cli
</syntaxhighlight>-->>> allows to launch the cli on a SRLinux device

To end this setup preparation, the containerlab tool integrates an option to launch a webserver showing the lab topology in graph + node details

By default, it's listening on port 50080 of the host

Then, you can access to the following web page showing the topology + lab infos

[[File:Lab graph tool.jpg|ohne|mini|900x900px]]

Finally, if you want to stop a lab, please run '''<nowiki>''</nowiki>containerlab destroy --topo <topology>.yml<nowiki>''</nowiki>'''

At this step, nodes are running, connections are done, but configurations are empty.

'''So, it's time to configure them :)'''

Configuration files are attached to this blog. Once validated, they are exportable and can be called by the tool when launching the lab. Below how to do it :

Firstable, once you ve finished your "hand made config", you can run the following command to save it :<syntaxhighlight lang="bash">
containerlab save -t quickstart.clab.yml
</syntaxhighlight>'''To be noted it's only working for Nokia nodes in this lab'''. CVX configurations will have to be called by another method (or pushed manually)

It will create one folder per node into the lab folder which gives the following tree :
[[File:Topology-srl.jpg|ohne|mini]]

'''Note''': For Cumulus, you can apparently retrieve configurations files within adding the following parameters into lab config file. '''To be tested.''' some lab samples here : https://github.com/hellt/clabs/blob/main/labs/cvx04/symm-mh.clab.yml

Now we have the lab running, below the different commands to check that all is working as expected : <syntaxhighlight lang="yaml">
-------------------------------------------
Check L2 tables ///
-------------------------------------------

Nokia :

A:leaf2# show network-instance mac-vrf-100 bridge-table mac-table all
----------------------------------------------------------------------------------------------------------------------------------------------------------
Mac-table of network instance mac-vrf-100
----------------------------------------------------------------------------------------------------------------------------------------------------------
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
| Address | Destination | Dest Index | Type | Active | Aging | Last Update |
+====================+=========================================+============+=============+=========+========+=========================================+
| AA:C1:AB:0C:60:D6 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T13:06:17.000Z |
| | vtep:10.255.255.8 vni:100 | 67 | | | | |
| AA:C1:AB:23:31:04 | ethernet-1/3.100 | 2 | learnt | true | 300 | 2021-10-08T13:06:10.000Z |
| AA:C1:AB:4B:FF:71 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:55:28.000Z |
| | vtep:10.255.255.5 vni:100 | 60 | | | | |
| AA:C1:AB:B5:79:50 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:43:37.000Z |
| | vtep:10.255.255.7 vni:100 | 64 | | | | |
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 0 Total 0 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 1 Total 1 Active
Total Evpn Macs : 3 Total 3 Active
Total Evpn static Macs : 0 Total 0 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 4 Total 4 Active
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : type learnt matches the local learnt mac, evpn are learnt within VXLAN

Cumulus :

root@leaf1-cum-dc1:/# net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
100 bridge swp3 aa:c1:ab:4b:ff:71 00:00:02
100 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:17:21
100 bridge vni100 22:2d:67:f1:bb:26 permanent 02:17:21
100 bridge vni100 aa:c1:ab:0c:60:d6 extern_learn <1 sec
100 bridge vni100 aa:c1:ab:23:31:04 extern_learn 00:00:07
100 bridge vni100 aa:c1:ab:b5:79:50 extern_learn 00:13:28

root@leaf1-cum-dc1:/# net show bridge macs vlan 200

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
200 bridge swp3 aa:c1:ab:4b:ff:71 00:08:46
200 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:16:36
200 bridge vni200 00:00:5e:00:01:01 static sticky 00:14:26
200 bridge vni200 1e:0d:aa:46:53:b1 permanent 02:16:36
200 bridge vni200 02:77:6b:ff:00:41 static sticky 01:00:52
200 bridge vni200 02:f8:b8:ff:00:41 static sticky 00:14:26
200 bridge vni200 aa:c1:ab:b5:79:50 extern_learn 00:11:51
200 bridge vni10200 02:77:6b:ff:00:00 extern_learn 00:53:02
200 bridge vni10200 02:f8:b8:ff:00:00 extern_learn 00:14:26
200 bridge vni10200 86:5e:aa:ca:50:be permanent 02:16:36

Note : the macs with permanent flag are local interfaces, physical ou logic (vtep)
Note2 : extern_learn mac are the one learned via the VXLAN infra (and mapped to the vnixxx interface)

-------------------------------------------
Check arp tables ///
-------------------------------------------

Nokia :

:leaf2# show arpnd arp-entries
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| ethernet-1/1 | 0 | 10.254.254.13 | dynamic | AA:C1:AB:59:65:EA | 3 hours from now |
| ethernet-1/2 | 0 | 10.254.254.15 | dynamic | AA:C1:AB:CB:F8:8A | 3 hours from now |
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
| mgmt0 | 0 | 172.20.20.1 | dynamic | 02:42:63:9E:74:03 | 2 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 5 (0 static, 5 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : on Nokia you have to sort the entries basing on subinterface and the retrieve the L3VPN attached

A:leaf2# show arpnd arp-entries subinterface 200
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 2 (0 static, 2 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Cumulus :

Here we retrieve all arp infos linked to evpn

root@leaf1-cum-dc1:/# net show evpn arp-cache vni all
VNI 100 #ARP (IPv4 and IPv6, local and remote) 2

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.1.3 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0
192.168.1.1 local active aa:c1:ab:4b:ff:71 0/0

VNI 200 #ARP (IPv4 and IPv6, local and remote) 4

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.2.1 local active aa:c1:ab:4b:ff:71 0/0
192.168.200.254 remote active 00:00:5e:00:01:01 10.255.255.8 0/0
192.168.2.254 remote active 00:00:5e:00:01:01 10.255.255.6 0/0
192.168.200.1 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0

VNI 400 #ARP (IPv4 and IPv6, local and remote) 0

-------------------------------------------
Check bgp sessions' status - underlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep ipv4
| default | 10.254.254.19 | udl-superspine | S | 65003 | established | 0d:0h:22m:44s | ipv4-unicast | [13/4/4] |
| default | 10.254.254.22 | udl-leaf | S | 65013 | established | 0d:0h:22m:52s | ipv4-unicast | [13/1/4] |

Cumulus :

root@leaf3:/# net show bgp ipv4 unicast summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 13
RIB entries 25, using 4800 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.254.254.23 4 65002 466 403 0 0 0 00:19:34 4 13
10.254.254.25 4 65002 434 355 0 0 0 00:17:10 8 13

-------------------------------------------
Check bgp sessions' status - overlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep evpn
| default | 10.255.255.7 | ovl-leaf | S | 65013 | established | 0d:0h:24m:14s| evpn | [15/0/11] |
| default | 10.255.255.9 | ovl-superspine | S | 65003 | established | 0d:0h:24m:7s | evpn | [11/0/4] |

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.255.255.3 4 65002 5618 4742 0 0 0 03:53:35 19 27
10.255.255.4 4 65002 5565 4694 0 0 0 03:51:11 19 27

-------------------------------------------
Check EVPN L2 tables///
-------------------------------------------

Nokia :

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn route type 2
BGP table version is 17, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:100
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
Route Distinguisher: 10.255.255.5:200
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
Route Distinguisher: 10.255.255.6:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
Route Distinguisher: 10.255.255.7:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100
Route Distinguisher: 10.255.255.7:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.200.1]
10.255.255.7 32768 i
ET:8 RT:65000:200
Route Distinguisher: 10.255.255.8:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC

and if we focus on our vni 100

root@leaf3-cum-dc1:/# net show bgp evpn route vni 100 type 2
BGP table version is 48, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100

The additional route type "MAC+IP" is optional and is useful/mandatory when you do anycast irb.

-------------------------------------------
Check tables EVPN L3 ///
-------------------------------------------

Nokia :

--{ + running }--[ ]--
A:leaf2# show network-instance ip-vrf-200 route-table all
-----------------------------------------------------------------------------------------------------------------------------------------------------------
IPv4 Unicast route table of network instance ip-vrf-200
-----------------------------------------------------------------------------------------------------------------------------------------------------------
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
| Prefix | ID | Route Type | Route Owner | Best/Fib- | Metric | Pref | Next-hop | Next-hop |
| | | | | status(slot) | | | (Type) | Interface |
+=======================+=======+============+======================+======================+==========+=========+==============+==============+
| 192.168.2.0/24 | 9 | local | net_inst_mgr | True/success | 0 | 0 | 192.168.2.25 | irb0.200 |
| | | | | | | | 4 (direct) | |
| 192.168.2.254/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (extract) | |
| 192.168.2.255/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (broadcast) | |
| 192.168.200.0/24 | 0 | bgp-evpn | bgp_evpn_mgr | True/success | 0 | 170 | 10.255.255.8 | None |
| | | | | | | | (indirect/vx | |
| | | | | | | | lan) | |
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
-----------------------------------------------------------------------------------------------------------------------------------------------------------
4 IPv4 routes total
4 IPv4 prefixes with active routes
0 IPv4 prefixes with active ECMP routes
-----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--
A:leaf2#

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf1-cum-dc1:/# net show route vrf ip-vrf-200
show ip route vrf ip-vrf-200
=============================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF ip-vrf-200:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 04:48:42
B 192.168.2.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 04:27:26
C>* 192.168.2.0/24 is directly connected, vlan200, 04:48:42
B>* 192.168.200.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:48:49

root@leaf1-cum-dc1:/# net show bgp evpn route type 5
BGP table version is 3, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.6:10200
* [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
*> [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
Route Distinguisher: 10.255.255.8:10200
*> [5]:[0]:[24]:[192.168.200.0]
10.255.255.8 0 65001 65003 65002 65014 i
RT:65000:10200 ET:8 Rmac:02:f8:b8:ff:00:00

-------------------------------------------
Check arp hosts ///
-------------------------------------------

From client #1 /

bash-5.0# arp
Address HWtype HWaddress Flags Mask Iface
192.168.1.3 ether aa:c1:ab:b5:79:50 C eth1.100
192.168.1.4 ether aa:c1:ab:0c:60:d6 C eth1.100
192.168.2.2 ether aa:c1:ab:23:31:04 C eth1.200
192.168.2.254 ether aa:c1:ab:4f:f2:e7 C eth1.200
192.168.1.2 ether aa:c1:ab:23:31:04 C eth1.100
containerlab.local ether 02:42:63:9e:74:03 C eth0

-------------------------------------------
Ping ///
-------------------------------------------

From client #1 /

L2 service ping (vlan 100) ----------------------

bash-5.0# ping -c 1 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.812 ms

--- 192.168.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.812/0.812/0.812/0.000 MsgRcvd
bash-5.0# ping -c 1 192.168.1.3
PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_seq=1 ttl=64 time=1.40 ms

--- 192.168.1.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.403/1.403/1.403/0.000 ms
bash-5.0# ping -c 1 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=1.62 ms

--- 192.168.1.4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.619/1.619/1.619/0.000 ms

L3 service ping (vlan 200)

bash-5.0# ping -c 1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.053 ms

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms
bash-5.0# ^C
bash-5.0# ping -c 1 192.168.2.2
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.611 ms

--- 192.168.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.611/0.611/0.611/0.000 ms

-------------------------------------------
Traceroute ///
-------------------------------------------

L3 service traceroute (vlan 200)

bash-5.0# traceroute 192.168.200.1
traceroute to 192.168.200.1 (192.168.200.1), 30 hops max, 46 byte packets
1 192.168.2.254 (192.168.2.254) 0.012 ms 0.010 ms 0.007 ms
2 192.168.200.254 (192.168.200.254) 2.751 ms * 2.549 ms
3 192.168.200.1 (192.168.200.1) 2.084 ms 1.805 ms 1.761 ms

</syntaxhighlight>To come back on the setup, we wanted to implement solutions inspired by the <nowiki>RFC 7938</nowiki> rfc7938 (see section 5.2.1)

* eBGP will be used for Underlay and Overlay
* eBGP single-hop sessions are established over direct point-to-point links interconnecting the network nodes, no multi-hop or loopback sessions are used, even in the case of multiple links between the same pair of nodes.
* Private Use ASNs from the range 64512-65534 are used to avoid ASN conflicts.
* A single ASN is allocated to all of the Clos topology's Tier 1 devices.
* A unique ASN is allocated to each set of Tier 2 devices in the same cluster.
* A unique ASN is allocated to every Tier 3 device (e.g., ToR) in this topology.

In add, depending of network nodes, it will be interesting to :

* Add fast-failover option for underlay + add link-debounce on links to limit possible impacts in case of link flaps. BFD could be considered in some cases
* Have the best ECMP hashing algorythm on leaf nodes. On tier-1/tier-2 levels hashing will be based on underlay part (trafic inter-vtep), so L3-L4 decision, but some models can retreive the vni-id if I well remember.
* Implement the "multipath relax" since we could have multiple AS paths for a same destination.
* Implement gshut community on underlay (on tier-1/tier-2) or at least policies to prepend underlay routes to avoid some nodes in case of maintenance.

== Thoughts/Points to deepend ==
On leaf nodes, LACP over multiple nodes is feasible within ESI, but impact on incidents/maintenance can vary depending on the vendors.

'''Tests need to be done to check that.'''

Some sysadmin would like to KISS and have client servers connected in active/passive bonding, with eventually single-leaf ports aggregation.

'''Note :''' NH ESI in FIB management on ingress nodes can have some limitations (numbers of installable next-hop, load-balancing, etc. oftenly linked to the node chipset)

Another important engineering point to define at the beginning is : '''Do we authorize multisite L2 propagation or not ?'''

From my point-of-view, the smaller are the L2 segments the better it is. I would suggest Local L2 segments wit anycast gateways provided by Leaf switches, with L3 routing between DCs within EVPN type5 routes sharing only. Nevertheless, some business requirements for endcustomers can keep extended L2 alive.

We have to be careful when implementing Consistent hashing. ex: <nowiki>https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-44/Layer-3/Routing/Equal-Cost-Multipath-Load-Sharing-Hardware-ECMP/</nowiki>

If possible activate it per VRF, not globally, and restrict on dedicated subnets << depends of the vendor, it can be dangerous regarding memory management

Think at configuring the bgp option to not advertise routes to peers before they are installed in FIB.

We can implement leaf roles in some cases, to be able to do "light traffic engineering " on DCI links in some setup.

These roles will be managed within bgp communities added to VTEP /32. Look at the diagram below. Depending of the colour, the eBGP sessions between SupS will prepend or not the VTEP IPs announced to the remote DC

[[File:Fabric-roles.jpg|ohne|mini|1200x1200px]]

<blockquote>'''To be continued'''</blockquote>

== Lab archives ==
<blockquote>Reminder : following config are for lab, so please don't pay attention on typos/lack of security params/unused config. Thanks \o/</blockquote>

=== Credentials ===
<syntaxhighlight lang="bash">
SRLinux : admin/<no_password>

CVX : cumulus/cumulus
</syntaxhighlight>

=== Lab archives ===
layer 2 (vlan 100) + layer 3 (vlan 200) lab, 10/12/2021
[[File:20211008 lab-nokia-cvx-containerlab.zip|ohne|mini]]
<u>If you want to exchange about that lab, those topics, please don't hesitate to contact us</u>

= SONiC Lab =

<span><to be noted that those informations are included into the readme.md file into the archive></span>

== Lab diagram ==
[[File:SONIC_LAB.jpg]]

<div></div>Lab archive [http://wiki.intra.nesevo.com/uploads/LAB/sonic-lab-20201030.zip Attach:sonic-lab-20201030.zip]

This lab is a forked lab from many found on @, starting from Microsoft one : https://github.com/Azure/SONiC/wiki/SONiC-P4-Software-Switch

The target of this lab is to simulate a simple IP/Fabric in SPINE/LEAF topology. Each Leaf provide local L2 connectivity + L3 gateways to hosts. Flows between hosts from different leaf are L3 routed, in a "node path" LEAF-SPINE-LEAF.

Nothing complicated, but it allows to put a first foot into SONiC world :)

For the moment, as far as we know, MC-LAG + EVPN/VXLAN features are only available on entreprise SONiC versions. Tests to come once physic switches in enterprise version will be available in our lab.

The diagram of the architecture is into the zip file (SONIC_LAB.jpg)

To run the lab you need :

- to have installed docker by using ./install_docker_ovs.sh - to run the ./load_image.sh in order to get the Sonic image

Note : don't change file / folders names of network devices (leafX, spineX)

Once you're ready, please run the lab by launching "./start.sh"

The script :<div></div>
* launches the containers without Docket network features
* create network namespaces
* build linux bridges to interconnect the containers
* configure host IPs
* launch startup sonic scripts into sonic devices
* once devices started, checks + ping tests are run. If all BGP sessions are ok on spine + all ping are fine, the setup works well.
The "start" script has several comments, perhaps without the right vocabulary, so please be tolerant :)

If you want to stop the lab, please launch "./stop.sh"

Below some useful commands to run for sonic containers :

=== <span>run ping</span> ===
sudo docker exec -it <container> ping <ip>

This command triggers ping into the targetted container :
<div></div>
plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it leaf1 ping 192.168.1.1

PING 192.168.1.1 (192.168.1.1): 56 data bytes

64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.052 ms

64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.034 ms

^C--- 192.168.1.1 ping statistics ---

=== <span>check bgp sessions</span> ===
sudo docker exec -it <container> vtysh -c "show ip bgp sum"

This command sollicitates the SONiC shell of the targeted container, in order to print the bgp status.

Below, we run it on one spine, you should have 4 sessiosn UP (for the 4 leaves)

plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip bgp sum"

BGP router identifier 10.1.0.1, local AS number 65001

RIB entries 8, using 896 bytes of memory

Peers 4, using 18 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

10.1.0.0 4 65101 6409 6411 0 0 0 01:46:46 3

10.1.0.2 4 65102 6406 6410 0 0 0 01:46:44 3

10.1.0.4 4 65103 6407 6410 0 0 0 01:46:44 4

10.1.0.6 4 65104 6406 6412 0 0 0 01:46:46 1

Total number of neighbors 4

=== <span>check ip route</span> ===
sudo docker exec -it <container> vtysh -c "show ip route"

This command sollicitates the SONiC shell of the targeted container, in order to print its routing table?

plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip route"

Codes: K - kernel route, C - connected, S - static, R - RIP,

O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,

> - selected route, * - FIB route

C>* 10.1.0.0/31 is directly connected, Vlan11

C>* 10.1.0.2/31 is directly connected, Vlan21

C>* 10.1.0.4/31 is directly connected, Vlan31

C>* 10.1.0.6/31 is directly connected, Vlan41

C>* 127.0.0.0/8 is directly connected, lo

B>* 192.168.1.0/24 [20/0] via 10.1.0.0, Vlan11, 01:47:23

B>* 192.168.2.0/24 [20/0] via 10.1.0.2, Vlan21, 01:47:21

B>* 192.168.3.0/24 [20/0] via 10.1.0.4, Vlan31, 01:47:21

B>* 192.168.4.0/24 [20/0] via 10.1.0.6, Vlan41, 01:47:23

For more commands check SONiC docs : https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md

To finish, below the output of the start.sh when all goes well<div></div>
plancastre@dockerhost:~/00.Projects/p4-version-pla$ ./start.sh
---------------------------------------------------
Creating the containers: starting
---------------------------------------------------
0b4bbb3c6cfc9895466ea13a9e940d1a176d66f45deee9752e9b4732d735bb03
5ca9486baab3e8a5259062167132a944eee717875d3cc1ef29fe490cba3184c1
46c9c09dba27a32715b9fbe0bc434a9284b9d59317d278df693f425cb669f037
898e6afbcb2977115acbba2a69d069796cfd70fb90228b6c18e09ba4584365f7
eb9b3ef6d14784c9ffe4d9d5df3e921a24559889cb8770d226263aad6761182a
ae5f3b2c628da687eaabb331dc2cd989e71bcb386854e7a0be6ec587df99d49a
122197bffd7566a01ce3b6565fdfb6eb893aab7d5236f14e5a9bbbe95037ced2
a604520a080e2a2184598ad1d915c0c182d16e5185033373e95a4772d8978a67
0c49e18d78b2aa7c8ef8e937b60285aa56ec0bf4a8530226285270a085e11d5c
eb66336b8c6bd8333f1f4f685369e6cac973b817c35bb7126ac4d7ba72ef71bf
---------------------------------------------------
Retreiving PID from docker containers
---------------------------------------------------
---------------------------------------------------
Creating network namespaces for containers
---------------------------------------------------
List of created namespaces
ns_host4
ns_host3
ns_host2
ns_host1
ns_spine2
ns_spine1
ns_leaf4
ns_leaf3
ns_leaf2
ns_leaf1
---------------------------------------------------
Creating network connectivity between containers
---------------------------------------------------
---------------------------------------------------
Now assigning IPs to hosts (host1 to host4)
---------------------------------------------------
---------------------------------------------------
Configuration of containers is now finished
---------------------------------------------------
---------------------------------------------------
Now booting switches, please wait 2 minutes for switches to load
---------------------------------------------------
---------------------------------------------------
2 minutes period is finished
---------------------------------------------------
---------------------------------------------------
Now fixing the iptables
---------------------------------------------------
---------------------------------------------------
Backend actions are now finished
---------------------------------------------------
---------------------------------------------------
Now we'll check in 15 sec that BGP peers are all Up
---------------------------------------------------
---------------------------------------------------
SPINE1 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 94 97 0 0 0 00:01:28 4
10.1.0.2 4 65102 95 97 0 0 0 00:01:28 5
10.1.0.4 4 65103 95 97 0 0 0 00:01:28 5
10.1.0.6 4 65104 95 97 0 0 0 00:01:28 5
Total number of neighbors 4
---------------------------------------------------
SPINE2 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.2.0.1, local AS number 65002
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.2.0.0 4 65101 93 96 0 0 0 00:01:29 2
10.2.0.2 4 65102 94 100 0 0 0 00:01:32 1
10.2.0.4 4 65103 94 100 0 0 0 00:01:32 1
10.2.0.6 4 65104 94 100 0 0 0 00:01:32 1
Total number of neighbors 4
---------------------------------------------------
Now if all is ok, we'll do some ping tests
---------------------------------------------------
---------------------------------------------------
host1 ping its gateway
---------------------------------------------------
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=17.6 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=9.14 ms
--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 9.145/13.390/17.636/4.247 ms
---------------------------------------------------
host2 ping its gateway
---------------------------------------------------
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=23.6 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=6.21 ms
--- 192.168.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 6.218/14.950/23.683/8.733 ms
---------------------------------------------------
host3 ping its gateway
---------------------------------------------------
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=18.8 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=12.7 ms
--- 192.168.3.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.716/15.764/18.812/3.048 ms
---------------------------------------------------
host4 ping its gateway
---------------------------------------------------
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.
64 bytes from 192.168.4.1: icmp_seq=1 ttl=64 time=17.7 ms
64 bytes from 192.168.4.1: icmp_seq=2 ttl=64 time=4.46 ms
--- 192.168.4.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.464/11.107/17.750/6.643 ms
---------------------------------------------------
host1 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=16.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.049/17.253/19.532/1.615 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=15.5 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=14.3 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.312/15.592/16.939/1.083 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=20.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=16.8 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.048/17.871/20.739/2.058 ms
---------------------------------------------------
host2 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=16.6 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.222/16.583/16.924/0.286 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.6 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=18.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=26.8 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.620/20.549/26.887/4.524 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=33.8 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=27.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=20.2 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 20.216/27.259/33.818/5.566 ms
---------------------------------------------------
host3 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=14.5 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=15.5 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.535/16.549/19.541/2.162 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.4 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=14.6 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=18.2 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.643/16.458/18.278/1.491 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=15.3 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=14.1 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=13.1 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 13.171/14.234/15.334/0.883 ms
---------------------------------------------------
host4 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=21.9 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=21.3 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=14.6 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 14.619/19.284/21.932/3.310 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=14.8 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=18.4 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=14.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.148/15.786/18.409/1.879 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=15.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=18.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 15.117/16.788/18.962/1.612 ms
---------------------------------------------------
If bot spine have all BGP sessions and all ping succeeded, all works well
Congrats ! please check readme file to find some details
---------------------------------------------------

Nesevo Virtual Labs

2021-10-29T08:02:24Z

PierreLancastre: /* Lab diagram */

[[File:nesevo-labs-v2.png]]

= Nokia/Cumulus Multi-Vendor Lab =

The targets of this lab are to :

* mix Nokia SR-Linux and Cumulus VXLAN nodes, in spine or leaf roles
* make this environment work for different kinds of setup : cross-DC L2, differentiated L3 gateways per leaf per VRF, anycast gateway cross or intra DC, etc.
* spot compatibility issues if existing

To reach those goals, we'll use the Nokia lab tool "containerlab" https://containerlab.srlinux.dev/, labs from https://clabs.netdevops.me/ helped a lot. Many thanks to the contributors.

We'll need a Linux host with Docker installed on it + kvm virtualization enhancements activated (important if your host is a VM). If virtualization optimization is not ok, you won't be able to launch the most of devices.

Installation details are referenced there : https://containerlab.srlinux.dev/install/

Once installed, you'll need to register the different NOS images you'll use. Cumulus image is present on a public registrery, but Nokia/Arista/Juniper are not, you'll need a vendor support account to retreive them. Once downloaded (.tgz file typically) you can "docker load" the image :

For private images :<syntaxhighlight lang="bash">
sudo docker load -i 21.6.1-235.tar.xz /* (Nokia) */
sudo docker load -i cEOS64-lab-4.26.1F.tar /* (Arista) */
</syntaxhighlight>For public images :<syntaxhighlight lang="bash">
sudo docker pull networkop/cx:4.3.0 /* (Cumulus) */
</syntaxhighlight>

To verify your installed images, you can run "sudo docker images"<syntaxhighlight lang="bash">
plancastre@containerlab:~/00.LAB/senss$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
networkop/cx 4.3.0 a40541ed15ba 2 weeks ago 733MB
srlinux 21.6.1-235 e29aed30e83f 3 weeks ago 1.14GB
networkop/ignite dev dcc5eff29355 4 weeks ago 37.9MB
networkop/kernel 4.19 e794d8875f83 3 months ago 88MB
frrouting/frr v7.5.1 c3e13a4c5918 3 months ago 123MB

</syntaxhighlight>Once you've got all your images, to run the lab, you'll need a .yml file defining the topology to run.

In our lab, we've started with the 5-clos lab file from Nokia site, which we've forked.

We can see there is a branch "kinds" where we can define templates of nodes, in order to not rewrite whole parameters when creating nodes. For the images, it's important to take the repository name + tag.

'''Note:''' If the tag is "latest" it is not needed.

All supported "kinds" are referenced there : https://containerlab.srlinux.dev/manual/kinds/kinds/

'''Note:''' yml is indentation sensitive, so please respect the "double space" used to define hierarchy. Below the lab diagram + the yml file associated

== Infos on nodes ==

=== L3 infos ===
{| class="wikitable"
|+
!Device
!OS
!ASN
!VTEP IP
!swp1 or eth1/1
!swp2 or eth1/2
!swp3 or eth1/3
!SVI
|-
|client1
|Linux
|<none>
|<none>
|vlan100: 192.168.1.1/24
vlan200: 192.168.2.1/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client2
|Linux
|<none>
|<none>
|vlan100: 192.168.1.2/24
vlan200: 192.168.2.2/24

route 192.168.200.0/24 via 192.168.2.254
|<none>
|<none>
|<none>
|-
|client3
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.1/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|client4
|Linux
|<none>
|<none>
|vlan100: 192.168.1.3/24
vlan200: 192.168.200.2/24

route 192.168.2.0/24 via 192.168.200.254
|<none>
|<none>
|<none>
|-
|leaf1
|CVX
|65011
|10.255.255.5/32
|10.254.254.8/31
|10.254.254.10/31
|trunk vlan 100,200
|vlan200 : 192.168.2.254/24
|-
|leaf2
|SRLinux
|65012
|10.255.255.6/32
|10.254.254.12/31
|10.254.254.14/31
|trunk vlan 100,200
|irb0.200 : 192.168.2.254/24
|-
|leaf3
|CVX
|65013
|10.255.255.7/32
|10.254.254.22/31
|10.254.254.24/31
|trunk vlan 100,200
|vlan200 : 192.168.200.254/24
|-
|leaf4
|SRLinux
|65014
|10.255.255.8/32
|10.254.254.26/31
|10.254.254.28/31
|trunk vlan 100,200
|irb0.200 : 192.168.200.254/24
|-
|spine1
|CVX
|65001
|10.255.255.1/32
|10.254.254.9/31
|10.254.254.13/31
|10.254.254.0/31
|<none>
|-
|spine2
|CVX
|65001
|10.255.255.2/32
|10.254.254.11/31
|10.254.254.15/31
|10.254.254.16/31
|<none>
|-
|spine3
|SRLinux
|65002
|10.255.255.3/32
|10.254.254.23/31
|10.254.254.27/31
|10.254.254.18/31
|<none>
|-
|spine4
|SRLinux
|65002
|10.255.255.4/32
|10.254.254.25/31
|10.254.254.29/31
|10.254.254.20/31
|<none>
|-
|superspine1
|CVX
|65003
|10.255.255.9/32
|10.254.254.1/31
|10.254.254.19/31
|<none>
|<none>
|-
|superspine2
|SRLinux
|65003
|10.255.255.10/32
|10.254.254.17/31
|10.254.254.21/31
|<none>
|<none>
|}

=== VXLAN/BGP infos ===
Note : in our lab, route-distinguisher is constructed on the model <loopback_ip>:<vlan_id>
{| class="wikitable"
|+
!service
!vlan id
!route-target
!VNI
!Commentaires
|-
|L2 cross-DC
|100
|target:65000:100
|100
|the 4 clients are in the same broadcast domain / subnet
|-
|L3VPN cros-DC
|200
|target:65000:200
target:65000:10200
|200 (for L2)
10200 (for routing)
|the 4 clients are separated in 2 different subnets. Traffic is routed within dedicated VNI
To be noted that in the current setup VMTO is not implemented, so cross DC traffic is not optimized.
|}

== Lab diagram ==
[[File:Lab-srl-diagram.png|ohne|mini|600x600px]]

== yml lab file ==
<syntaxhighlight lang="yaml">
# topology documentation: http://containerlab.srlinux.dev/lab-examples/min-5clos/
name: lab-srl-cum

topology:
kinds:
srl:
image: srlinux:21.6.1-235
linux:
image: ghcr.io/hellt/network-multitool
cvx:
image: networkop/cx:4.3.0
runtime: docker
nodes:
leaf1:
kind: cvx
leaf2:
kind: srl
type: ixrd2
leaf3:
kind: cvx
leaf4:
kind: srl
type: ixrd2

spine1:
kind: cvx
spine2:
kind: cvx
spine3:
kind: srl
type: ixr6
spine4:
kind: srl
type: ixr6

superspine1:
kind: cvx
superspine2:
kind: srl
type: ixr6

client1:
kind: linux
client2:
kind: linux
client3:
kind: linux
client4:
kind: linux

links:
# leaf to spine links POD1
- endpoints: ["leaf1:swp1", "spine1:swp1"]
- endpoints: ["leaf1:swp2", "spine2:swp1"]
- endpoints: ["leaf2:e1-1", "spine1:swp2"]
- endpoints: ["leaf2:e1-2", "spine2:swp2"]

# spine to superspine links POD1
- endpoints: ["spine1:swp3", "superspine1:swp1"]
- endpoints: ["spine2:swp3", "superspine2:e1-1"]

# leaf to spine links POD2
- endpoints: ["leaf3:swp1", "spine3:e1-1"]
- endpoints: ["leaf3:swp2", "spine4:e1-1"]
- endpoints: ["leaf4:e1-1", "spine3:e1-2"]
- endpoints: ["leaf4:e1-2", "spine4:e1-2"]

# spine to superspine links POD2
- endpoints: ["spine3:e1-3", "superspine1:swp2"]
- endpoints: ["spine4:e1-3", "superspine2:e1-2"]

# client connection links
- endpoints: ["client1:eth1", "leaf1:swp3"]
- endpoints: ["client2:eth1", "leaf2:e1-3"]
- endpoints: ["client3:eth1", "leaf3:swp3"]
- endpoints: ["client4:eth1", "leaf4:e1-3"]

</syntaxhighlight>

Once the nodes are defined, we have to define the needed connections between the nodes.

Each endpoint is defined as '<nowiki/>'''<nowiki/>'<node_name>:<interface_name><nowiki>''</nowiki>'''

When all is ok, you can launch in suduoer the topology.

For example :<syntaxhighlight lang="bash">
sudo containerlab deploy --topo topo.clab.yml --reconfigure

Note: The option "reconfigure" is used there in order to not have to destroy instances when updating topology/relaunching.
</syntaxhighlight>Another info regarding the yml file, it's possible to quickly generate files with the '''"containerlab gen -n 3tier --nodes 4,2,1"''' command for example. https://containerlab.srlinux.dev/cmd/generate/

To be noted that the "tier-x" notion in containerlab is inversed regarding the rfc 7938 (tier1 is superspine level into the rfc, tier1 is leaf level in nokia tool)

At the end of the topology launch, you can retreive the details :<syntaxhighlight lang="yaml">
INFO[0011] Writing /etc/hosts file
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| # | Name | Container ID | Image | Kind | Group | State | IPv4 Address | IPv6 Address |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
| 1 | clab-lab-srl-cum-client1 | 25e2bbf6b13f | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
| 2 | clab-lab-srl-cum-client2 | 17d82d5a6e21 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.4/24 | 2001:172:20:20::4/64 |
| 3 | clab-lab-srl-cum-client3 | f8d78527c1b1 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.6/24 | 2001:172:20:20::6/64 |
| 4 | clab-lab-srl-cum-client4 | 913a0172d0c5 | ghcr.io/hellt/network-multitool | linux | | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 5 | clab-lab-srl-cum-leaf1 | 9ffc334af2b1 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.9/24 | 2001:172:20:20::9/64 |
| 6 | clab-lab-srl-cum-leaf2 | 7d3eb855d749 | srlinux:21.6.1-235 | srl | | running | 172.20.20.14/24 | 2001:172:20:20::e/64 |
| 7 | clab-lab-srl-cum-leaf3 | e0d36ffc606a | networkop/cx:4.3.0 | cvx | | running | 172.20.20.7/24 | 2001:172:20:20::7/64 |
| 8 | clab-lab-srl-cum-leaf4 | e6753b1d099e | srlinux:21.6.1-235 | srl | | running | 172.20.20.12/24 | 2001:172:20:20::c/64 |
| 9 | clab-lab-srl-cum-spine1 | a7e63993d052 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.5/24 | 2001:172:20:20::5/64 |
| 10 | clab-lab-srl-cum-spine2 | 3124250d5b75 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
| 11 | clab-lab-srl-cum-spine3 | 5e300ca3f66f | srlinux:21.6.1-235 | srl | | running | 172.20.20.11/24 | 2001:172:20:20::b/64 |
| 12 | clab-lab-srl-cum-spine4 | 0a59e60a596b | srlinux:21.6.1-235 | srl | | running | 172.20.20.13/24 | 2001:172:20:20::d/64 |
| 13 | clab-lab-srl-cum-superspine1 | 3d9935413fc2 | networkop/cx:4.3.0 | cvx | | running | 172.20.20.8/24 | 2001:172:20:20::8/64 |
| 14 | clab-lab-srl-cum-superspine2 | 5bc58d423e2c | srlinux:21.6.1-235 | srl | | running | 172.20.20.15/24 | 2001:172:20:20::f/64 |
+----+------------------------------+--------------+---------------------------------+-------+-------+---------+-----------------+----------------------+
</syntaxhighlight>The containerlab program creates the updates into the host file + generates an ansible inventory file.

There are different ways to connect onto the docker instances, within docker exec commands or SSH directly the IP address related above.

For example :<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-spine1 bash
</syntaxhighlight>-->>> allows to launch a bash on the cumulus device (then you run the different net commands)<syntaxhighlight lang="bash">
sudo docker exec -it clab-lab-srl-cum-leaf2 sr_cli
</syntaxhighlight>-->>> allows to launch the cli on a SRLinux device

To end this setup preparation, the containerlab tool integrates an option to launch a webserver showing the lab topology in graph + node details

By default, it's listening on port 50080 of the host

Then, you can access to the following web page showing the topology + lab infos

[[File:Lab graph tool.jpg|ohne|mini|900x900px]]

Finally, if you want to stop a lab, please run '''<nowiki>''</nowiki>containerlab destroy --topo <topology>.yml<nowiki>''</nowiki>'''

At this step, nodes are running, connections are done, but configurations are empty.

'''So, it's time to configure them :)'''

Configuration files are attached to this blog. Once validated, they are exportable and can be called by the tool when launching the lab. Below how to do it :

Firstable, once you ve finished your "hand made config", you can run the following command to save it :<syntaxhighlight lang="bash">
containerlab save -t quickstart.clab.yml
</syntaxhighlight>'''To be noted it's only working for Nokia nodes in this lab'''. CVX configurations will have to be called by another method (or pushed manually)

It will create one folder per node into the lab folder which gives the following tree :
[[File:Topology-srl.jpg|ohne|mini]]

'''Note''': For Cumulus, you can apparently retrieve configurations files within adding the following parameters into lab config file. '''To be tested.''' some lab samples here : https://github.com/hellt/clabs/blob/main/labs/cvx04/symm-mh.clab.yml

Now we have the lab running, below the different commands to check that all is working as expected : <syntaxhighlight lang="yaml">
-------------------------------------------
Check L2 tables ///
-------------------------------------------

Nokia :

A:leaf2# show network-instance mac-vrf-100 bridge-table mac-table all
----------------------------------------------------------------------------------------------------------------------------------------------------------
Mac-table of network instance mac-vrf-100
----------------------------------------------------------------------------------------------------------------------------------------------------------
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
| Address | Destination | Dest Index | Type | Active | Aging | Last Update |
+====================+=========================================+============+=============+=========+========+=========================================+
| AA:C1:AB:0C:60:D6 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T13:06:17.000Z |
| | vtep:10.255.255.8 vni:100 | 67 | | | | |
| AA:C1:AB:23:31:04 | ethernet-1/3.100 | 2 | learnt | true | 300 | 2021-10-08T13:06:10.000Z |
| AA:C1:AB:4B:FF:71 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:55:28.000Z |
| | vtep:10.255.255.5 vni:100 | 60 | | | | |
| AA:C1:AB:B5:79:50 | vxlan-interface:vxlan1.100 | 6379518792 | evpn | true | N/A | 2021-10-08T12:43:37.000Z |
| | vtep:10.255.255.7 vni:100 | 64 | | | | |
+--------------------+-----------------------------------------+------------+-------------+---------+--------+-----------------------------------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 0 Total 0 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 1 Total 1 Active
Total Evpn Macs : 3 Total 3 Active
Total Evpn static Macs : 0 Total 0 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 4 Total 4 Active
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : type learnt matches the local learnt mac, evpn are learnt within VXLAN

Cumulus :

root@leaf1-cum-dc1:/# net show bridge macs vlan 100

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
100 bridge swp3 aa:c1:ab:4b:ff:71 00:00:02
100 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:17:21
100 bridge vni100 22:2d:67:f1:bb:26 permanent 02:17:21
100 bridge vni100 aa:c1:ab:0c:60:d6 extern_learn <1 sec
100 bridge vni100 aa:c1:ab:23:31:04 extern_learn 00:00:07
100 bridge vni100 aa:c1:ab:b5:79:50 extern_learn 00:13:28

root@leaf1-cum-dc1:/# net show bridge macs vlan 200

VLAN Master Interface MAC TunnelDest State Flags LastSeen
---- ------ --------- ----------------- ---------- --------- ------------ --------
200 bridge swp3 aa:c1:ab:4b:ff:71 00:08:46
200 bridge swp3 aa:c1:ab:4f:f2:e7 permanent 02:16:36
200 bridge vni200 00:00:5e:00:01:01 static sticky 00:14:26
200 bridge vni200 1e:0d:aa:46:53:b1 permanent 02:16:36
200 bridge vni200 02:77:6b:ff:00:41 static sticky 01:00:52
200 bridge vni200 02:f8:b8:ff:00:41 static sticky 00:14:26
200 bridge vni200 aa:c1:ab:b5:79:50 extern_learn 00:11:51
200 bridge vni10200 02:77:6b:ff:00:00 extern_learn 00:53:02
200 bridge vni10200 02:f8:b8:ff:00:00 extern_learn 00:14:26
200 bridge vni10200 86:5e:aa:ca:50:be permanent 02:16:36

Note : the macs with permanent flag are local interfaces, physical ou logic (vtep)
Note2 : extern_learn mac are the one learned via the VXLAN infra (and mapped to the vnixxx interface)

-------------------------------------------
Check arp tables ///
-------------------------------------------

Nokia :

:leaf2# show arpnd arp-entries
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| ethernet-1/1 | 0 | 10.254.254.13 | dynamic | AA:C1:AB:59:65:EA | 3 hours from now |
| ethernet-1/2 | 0 | 10.254.254.15 | dynamic | AA:C1:AB:CB:F8:8A | 3 hours from now |
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
| mgmt0 | 0 | 172.20.20.1 | dynamic | 02:42:63:9E:74:03 | 2 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 5 (0 static, 5 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Note : on Nokia you have to sort the entries basing on subinterface and the retrieve the L3VPN attached

A:leaf2# show arpnd arp-entries subinterface 200
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
| Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry |
+===============+===============+=================+===============+=============================+=========================================================+
| irb0 | 200 | 192.168.2.1 | evpn | AA:C1:AB:4B:FF:71 | |
| irb0 | 200 | 192.168.2.2 | dynamic | AA:C1:AB:23:31:04 | 3 hours from now |
+---------------+---------------+-----------------+---------------+-----------------------------+---------------------------------------------------------+
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total entries : 2 (0 static, 2 dynamic)
----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--

Cumulus :

Here we retrieve all arp infos linked to evpn

root@leaf1-cum-dc1:/# net show evpn arp-cache vni all
VNI 100 #ARP (IPv4 and IPv6, local and remote) 2

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.1.3 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0
192.168.1.1 local active aa:c1:ab:4b:ff:71 0/0

VNI 200 #ARP (IPv4 and IPv6, local and remote) 4

Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
192.168.2.1 local active aa:c1:ab:4b:ff:71 0/0
192.168.200.254 remote active 00:00:5e:00:01:01 10.255.255.8 0/0
192.168.2.254 remote active 00:00:5e:00:01:01 10.255.255.6 0/0
192.168.200.1 remote active aa:c1:ab:b5:79:50 10.255.255.7 0/0

VNI 400 #ARP (IPv4 and IPv6, local and remote) 0

-------------------------------------------
Check bgp sessions' status - underlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep ipv4
| default | 10.254.254.19 | udl-superspine | S | 65003 | established | 0d:0h:22m:44s | ipv4-unicast | [13/4/4] |
| default | 10.254.254.22 | udl-leaf | S | 65013 | established | 0d:0h:22m:52s | ipv4-unicast | [13/1/4] |

Cumulus :

root@leaf3:/# net show bgp ipv4 unicast summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 13
RIB entries 25, using 4800 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.254.254.23 4 65002 466 403 0 0 0 00:19:34 4 13
10.254.254.25 4 65002 434 355 0 0 0 00:17:10 8 13

-------------------------------------------
Check bgp sessions' status - overlay ///
-------------------------------------------

Nokia :

A:spine3# show network-instance default protocols bgp neighbor | grep evpn
| default | 10.255.255.7 | ovl-leaf | S | 65013 | established | 0d:0h:24m:14s| evpn | [15/0/11] |
| default | 10.255.255.9 | ovl-superspine | S | 65003 | established | 0d:0h:24m:7s | evpn | [11/0/4] |

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn summary
BGP router identifier 10.255.255.7, local AS number 65013 vrf-id 0
BGP table version 0
RIB entries 29, using 5568 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
10.255.255.3 4 65002 5618 4742 0 0 0 03:53:35 19 27
10.255.255.4 4 65002 5565 4694 0 0 0 03:51:11 19 27

-------------------------------------------
Check EVPN L2 tables///
-------------------------------------------

Nokia :

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf3-cum-dc1:/# net show bgp evpn route type 2
BGP table version is 17, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.5:100
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
Route Distinguisher: 10.255.255.5:200
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.2.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:200 RT:65011:10200 ET:8 Rmac:aa:c1:ab:4f:f2:e7
Route Distinguisher: 10.255.255.6:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.2.254]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:77:6b:ff:00:41]
10.255.255.6 0 65002 65003 65001 65012 i
RT:65000:200 ET:8 MM:0, sticky MAC
Route Distinguisher: 10.255.255.7:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100
Route Distinguisher: 10.255.255.7:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:200
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.200.1]
10.255.255.7 32768 i
ET:8 RT:65000:200
Route Distinguisher: 10.255.255.8:200
* [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[00:00:5e:00:01:01]:[32]:[192.168.200.254]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
* [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC
*> [2]:[0]:[48]:[02:f8:b8:ff:00:41]
10.255.255.8 0 65002 65014 i
RT:65000:200 ET:8 MM:0, sticky MAC

and if we focus on our vni 100

root@leaf3-cum-dc1:/# net show bgp evpn route vni 100 type 2
BGP table version is 48, local router ID is 10.255.255.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
* [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:4b:ff:71]:[32]:[192.168.1.1]
10.255.255.5 0 65002 65003 65001 65011 i
RT:65000:100 ET:8
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]
10.255.255.7 32768 i
ET:8 RT:65000:100
*> [2]:[0]:[48]:[aa:c1:ab:b5:79:50]:[32]:[192.168.1.3]
10.255.255.7 32768 i
ET:8 RT:65000:100

The additional route type "MAC+IP" is optional and is useful/mandatory when you do anycast irb.

-------------------------------------------
Check tables EVPN L3 ///
-------------------------------------------

Nokia :

--{ + running }--[ ]--
A:leaf2# show network-instance ip-vrf-200 route-table all
-----------------------------------------------------------------------------------------------------------------------------------------------------------
IPv4 Unicast route table of network instance ip-vrf-200
-----------------------------------------------------------------------------------------------------------------------------------------------------------
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
| Prefix | ID | Route Type | Route Owner | Best/Fib- | Metric | Pref | Next-hop | Next-hop |
| | | | | status(slot) | | | (Type) | Interface |
+=======================+=======+============+======================+======================+==========+=========+==============+==============+
| 192.168.2.0/24 | 9 | local | net_inst_mgr | True/success | 0 | 0 | 192.168.2.25 | irb0.200 |
| | | | | | | | 4 (direct) | |
| 192.168.2.254/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (extract) | |
| 192.168.2.255/32 | 9 | host | net_inst_mgr | True/success | 0 | 0 | None | None |
| | | | | | | | (broadcast) | |
| 192.168.200.0/24 | 0 | bgp-evpn | bgp_evpn_mgr | True/success | 0 | 170 | 10.255.255.8 | None |
| | | | | | | | (indirect/vx | |
| | | | | | | | lan) | |
+-----------------------+-------+------------+----------------------+----------------------+----------+---------+--------------+--------------+
-----------------------------------------------------------------------------------------------------------------------------------------------------------
4 IPv4 routes total
4 IPv4 prefixes with active routes
0 IPv4 prefixes with active ECMP routes
-----------------------------------------------------------------------------------------------------------------------------------------------------------
--{ + running }--[ ]--
A:leaf2#

<didn't spotted how to show evpn routes in a bgp fashion>

Cumulus :

root@leaf1-cum-dc1:/# net show route vrf ip-vrf-200
show ip route vrf ip-vrf-200
=============================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF ip-vrf-200:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 04:48:42
B 192.168.2.0/24 [20/0] via 10.255.255.6, vlan200 onlink, weight 1, 04:27:26
C>* 192.168.2.0/24 is directly connected, vlan200, 04:48:42
B>* 192.168.200.0/24 [20/0] via 10.255.255.8, vlan200 onlink, weight 1, 03:48:49

root@leaf1-cum-dc1:/# net show bgp evpn route type 5
BGP table version is 3, local router ID is 10.255.255.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.255.255.6:10200
* [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
*> [5]:[0]:[24]:[192.168.2.0]
10.255.255.6 0 65001 65012 i
RT:65000:10200 ET:8 Rmac:02:77:6b:ff:00:00
Route Distinguisher: 10.255.255.8:10200
*> [5]:[0]:[24]:[192.168.200.0]
10.255.255.8 0 65001 65003 65002 65014 i
RT:65000:10200 ET:8 Rmac:02:f8:b8:ff:00:00

-------------------------------------------
Check arp hosts ///
-------------------------------------------

From client #1 /

bash-5.0# arp
Address HWtype HWaddress Flags Mask Iface
192.168.1.3 ether aa:c1:ab:b5:79:50 C eth1.100
192.168.1.4 ether aa:c1:ab:0c:60:d6 C eth1.100
192.168.2.2 ether aa:c1:ab:23:31:04 C eth1.200
192.168.2.254 ether aa:c1:ab:4f:f2:e7 C eth1.200
192.168.1.2 ether aa:c1:ab:23:31:04 C eth1.100
containerlab.local ether 02:42:63:9e:74:03 C eth0

-------------------------------------------
Ping ///
-------------------------------------------

From client #1 /

L2 service ping (vlan 100) ----------------------

bash-5.0# ping -c 1 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.812 ms

--- 192.168.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.812/0.812/0.812/0.000 MsgRcvd
bash-5.0# ping -c 1 192.168.1.3
PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data.
64 bytes from 192.168.1.3: icmp_seq=1 ttl=64 time=1.40 ms

--- 192.168.1.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.403/1.403/1.403/0.000 ms
bash-5.0# ping -c 1 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=1.62 ms

--- 192.168.1.4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.619/1.619/1.619/0.000 ms

L3 service ping (vlan 200)

bash-5.0# ping -c 1 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.053 ms

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms
bash-5.0# ^C
bash-5.0# ping -c 1 192.168.2.2
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.611 ms

--- 192.168.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.611/0.611/0.611/0.000 ms

-------------------------------------------
Traceroute ///
-------------------------------------------

L3 service traceroute (vlan 200)

bash-5.0# traceroute 192.168.200.1
traceroute to 192.168.200.1 (192.168.200.1), 30 hops max, 46 byte packets
1 192.168.2.254 (192.168.2.254) 0.012 ms 0.010 ms 0.007 ms
2 192.168.200.254 (192.168.200.254) 2.751 ms * 2.549 ms
3 192.168.200.1 (192.168.200.1) 2.084 ms 1.805 ms 1.761 ms

</syntaxhighlight>To come back on the setup, we wanted to implement solutions inspired by the <nowiki>RFC 7938</nowiki> rfc7938 (see section 5.2.1)

* eBGP will be used for Underlay and Overlay
* eBGP single-hop sessions are established over direct point-to-point links interconnecting the network nodes, no multi-hop or loopback sessions are used, even in the case of multiple links between the same pair of nodes.
* Private Use ASNs from the range 64512-65534 are used to avoid ASN conflicts.
* A single ASN is allocated to all of the Clos topology's Tier 1 devices.
* A unique ASN is allocated to each set of Tier 2 devices in the same cluster.
* A unique ASN is allocated to every Tier 3 device (e.g., ToR) in this topology.

In add, depending of network nodes, it will be interesting to :

* Add fast-failover option for underlay + add link-debounce on links to limit possible impacts in case of link flaps. BFD could be considered in some cases
* Have the best ECMP hashing algorythm on leaf nodes. On tier-1/tier-2 levels hashing will be based on underlay part (trafic inter-vtep), so L3-L4 decision, but some models can retreive the vni-id if I well remember.
* Implement the "multipath relax" since we could have multiple AS paths for a same destination.
* Implement gshut community on underlay (on tier-1/tier-2) or at least policies to prepend underlay routes to avoid some nodes in case of maintenance.

== Thoughts/Points to deepend ==
On leaf nodes, LACP over multiple nodes is feasible within ESI, but impact on incidents/maintenance can vary depending on the vendors.

'''Tests need to be done to check that.'''

Some sysadmin would like to KISS and have client servers connected in active/passive bonding, with eventually single-leaf ports aggregation.

'''Note :''' NH ESI in FIB management on ingress nodes can have some limitations (numbers of installable next-hop, load-balancing, etc. oftenly linked to the node chipset)

Another important engineering point to define at the beginning is : '''Do we authorize multisite L2 propagation or not ?'''

From my point-of-view, the smaller are the L2 segments the better it is. I would suggest Local L2 segments wit anycast gateways provided by Leaf switches, with L3 routing between DCs within EVPN type5 routes sharing only. Nevertheless, some business requirements for endcustomers can keep extended L2 alive.

We have to be careful when implementing Consistent hashing. ex: <nowiki>https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-44/Layer-3/Routing/Equal-Cost-Multipath-Load-Sharing-Hardware-ECMP/</nowiki>

If possible activate it per VRF, not globally, and restrict on dedicated subnets << depends of the vendor, it can be dangerous regarding memory management

Think at configuring the bgp option to not advertise routes to peers before they are installed in FIB.

We can implement leaf roles in some cases, to be able to do "light traffic engineering " on DCI links in some setup.

These roles will be managed within bgp communities added to VTEP /32. Look at the diagram below. Depending of the colour, the eBGP sessions between SupS will prepend or not the VTEP IPs announced to the remote DC

[[File:Fabric-roles.jpg|ohne|mini|1200x1200px]]

<blockquote>'''To be continued'''</blockquote>

== Lab archives ==
<blockquote>Reminder : following config are for lab, so please don't pay attention on typos/lack of security params/unused config. Thanks \o/</blockquote>

=== Credentials ===
<syntaxhighlight lang="bash">
SRLinux : admin/<no_password>

CVX : cumulus/cumulus
</syntaxhighlight>

=== Lab archives ===
layer 2 (vlan 100) + layer 3 (vlan 200) lab, 10/12/2021
[[File:20211008 lab-nokia-cvx-containerlab.zip|ohne|mini]]
<u>If you want to exchange about that lab, those topics, please don't hesitate to contact us</u>

= SONiC Lab =

<span><to be noted that those informations are included into the readme.md file into the archive></span>

== Lab diagram ==
[[File:SONIC_LAB.jpg]]

<div></div>Lab archive [http://wiki.intra.nesevo.com/uploads/LAB/sonic-lab-20201030.zip Attach:sonic-lab-20201030.zip]

This lab is a forked lab from many found on @, starting from Microsoft one : https://github.com/Azure/SONiC/wiki/SONiC-P4-Software-Switch

The target of this lab is to simulate a simple IP/Fabric in SPINE/LEAF topology. Each Leaf provide local L2 connectivity + L3 gateways to hosts. Flows between hosts from different leaf are L3 routed, in a "node path" LEAF-SPINE-LEAF.

Nothing complicated, but it allows to put a first foot into SONiC world :)

For the moment, as far as we know, MC-LAG + EVPN/VXLAN features are only available on entreprise SONiC versions. Tests to come once physic switches in enterprise version will be available in our lab.

The diagram of the architecture is into the zip file (SONIC_LAB.jpg)

To run the lab you need :

- to have installed docker by using ./install_docker_ovs.sh - to run the ./load_image.sh in order to get the Sonic image

Note : don't change file / folders names of network devices (leafX, spineX)

Once you're ready, please run the lab by launching "./start.sh"

The script :<div></div>
* launches the containers without Docket network features
* create network namespaces
* build linux bridges to interconnect the containers
* configure host IPs
* launch startup sonic scripts into sonic devices
* once devices started, checks + ping tests are run. If all BGP sessions are ok on spine + all ping are fine, the setup works well.
The "start" script has several comments, perhaps without the right vocabulary, so please be tolerant :)

If you want to stop the lab, please launch "./stop.sh"

Below some useful commands to run for sonic containers :

=== <span>run ping</span> ===
sudo docker exec -it <container> ping <ip>

This command triggers ping into the targetted container :

plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it leaf1 ping 192.168.1.1

PING 192.168.1.1 (192.168.1.1): 56 data bytes

64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.052 ms

64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.034 ms

^C--- 192.168.1.1 ping statistics ---

=== <span>check bgp sessions</span> ===
sudo docker exec -it <container> vtysh -c "show ip bgp sum"

This command sollicitates the SONiC shell of the targeted container, in order to print the bgp status.

Below, we run it on one spine, you should have 4 sessiosn UP (for the 4 leaves)

plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip bgp sum"

BGP router identifier 10.1.0.1, local AS number 65001

RIB entries 8, using 896 bytes of memory

Peers 4, using 18 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

10.1.0.0 4 65101 6409 6411 0 0 0 01:46:46 3

10.1.0.2 4 65102 6406 6410 0 0 0 01:46:44 3

10.1.0.4 4 65103 6407 6410 0 0 0 01:46:44 4

10.1.0.6 4 65104 6406 6412 0 0 0 01:46:46 1

Total number of neighbors 4

=== <span>check ip route</span> ===
sudo docker exec -it <container> vtysh -c "show ip route"

This command sollicitates the SONiC shell of the targeted container, in order to print its routing table?

plancastre@dockerhost:~/00.Projects/p4-version-pla$ sudo docker exec -it spine1 vtysh -c "show ip route"

Codes: K - kernel route, C - connected, S - static, R - RIP,

O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,

> - selected route, * - FIB route

C>* 10.1.0.0/31 is directly connected, Vlan11

C>* 10.1.0.2/31 is directly connected, Vlan21

C>* 10.1.0.4/31 is directly connected, Vlan31

C>* 10.1.0.6/31 is directly connected, Vlan41

C>* 127.0.0.0/8 is directly connected, lo

B>* 192.168.1.0/24 [20/0] via 10.1.0.0, Vlan11, 01:47:23

B>* 192.168.2.0/24 [20/0] via 10.1.0.2, Vlan21, 01:47:21

B>* 192.168.3.0/24 [20/0] via 10.1.0.4, Vlan31, 01:47:21

B>* 192.168.4.0/24 [20/0] via 10.1.0.6, Vlan41, 01:47:23

For more commands check SONiC docs : https://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md

To finish, below the output of the start.sh when all goes well<div></div>
plancastre@dockerhost:~/00.Projects/p4-version-pla$ ./start.sh
---------------------------------------------------
Creating the containers: starting
---------------------------------------------------
0b4bbb3c6cfc9895466ea13a9e940d1a176d66f45deee9752e9b4732d735bb03
5ca9486baab3e8a5259062167132a944eee717875d3cc1ef29fe490cba3184c1
46c9c09dba27a32715b9fbe0bc434a9284b9d59317d278df693f425cb669f037
898e6afbcb2977115acbba2a69d069796cfd70fb90228b6c18e09ba4584365f7
eb9b3ef6d14784c9ffe4d9d5df3e921a24559889cb8770d226263aad6761182a
ae5f3b2c628da687eaabb331dc2cd989e71bcb386854e7a0be6ec587df99d49a
122197bffd7566a01ce3b6565fdfb6eb893aab7d5236f14e5a9bbbe95037ced2
a604520a080e2a2184598ad1d915c0c182d16e5185033373e95a4772d8978a67
0c49e18d78b2aa7c8ef8e937b60285aa56ec0bf4a8530226285270a085e11d5c
eb66336b8c6bd8333f1f4f685369e6cac973b817c35bb7126ac4d7ba72ef71bf
---------------------------------------------------
Retreiving PID from docker containers
---------------------------------------------------
---------------------------------------------------
Creating network namespaces for containers
---------------------------------------------------
List of created namespaces
ns_host4
ns_host3
ns_host2
ns_host1
ns_spine2
ns_spine1
ns_leaf4
ns_leaf3
ns_leaf2
ns_leaf1
---------------------------------------------------
Creating network connectivity between containers
---------------------------------------------------
---------------------------------------------------
Now assigning IPs to hosts (host1 to host4)
---------------------------------------------------
---------------------------------------------------
Configuration of containers is now finished
---------------------------------------------------
---------------------------------------------------
Now booting switches, please wait 2 minutes for switches to load
---------------------------------------------------
---------------------------------------------------
2 minutes period is finished
---------------------------------------------------
---------------------------------------------------
Now fixing the iptables
---------------------------------------------------
---------------------------------------------------
Backend actions are now finished
---------------------------------------------------
---------------------------------------------------
Now we'll check in 15 sec that BGP peers are all Up
---------------------------------------------------
---------------------------------------------------
SPINE1 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.1.0.1, local AS number 65001
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.0.0 4 65101 94 97 0 0 0 00:01:28 4
10.1.0.2 4 65102 95 97 0 0 0 00:01:28 5
10.1.0.4 4 65103 95 97 0 0 0 00:01:28 5
10.1.0.6 4 65104 95 97 0 0 0 00:01:28 5
Total number of neighbors 4
---------------------------------------------------
SPINE2 check BGP Sessions
---------------------------------------------------
BGP router identifier 10.2.0.1, local AS number 65002
RIB entries 8, using 896 bytes of memory
Peers 4, using 18 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.2.0.0 4 65101 93 96 0 0 0 00:01:29 2
10.2.0.2 4 65102 94 100 0 0 0 00:01:32 1
10.2.0.4 4 65103 94 100 0 0 0 00:01:32 1
10.2.0.6 4 65104 94 100 0 0 0 00:01:32 1
Total number of neighbors 4
---------------------------------------------------
Now if all is ok, we'll do some ping tests
---------------------------------------------------
---------------------------------------------------
host1 ping its gateway
---------------------------------------------------
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=17.6 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=9.14 ms
--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 9.145/13.390/17.636/4.247 ms
---------------------------------------------------
host2 ping its gateway
---------------------------------------------------
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=23.6 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=6.21 ms
--- 192.168.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 6.218/14.950/23.683/8.733 ms
---------------------------------------------------
host3 ping its gateway
---------------------------------------------------
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=18.8 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=12.7 ms
--- 192.168.3.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.716/15.764/18.812/3.048 ms
---------------------------------------------------
host4 ping its gateway
---------------------------------------------------
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.
64 bytes from 192.168.4.1: icmp_seq=1 ttl=64 time=17.7 ms
64 bytes from 192.168.4.1: icmp_seq=2 ttl=64 time=4.46 ms
--- 192.168.4.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.464/11.107/17.750/6.643 ms
---------------------------------------------------
host1 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=16.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.049/17.253/19.532/1.615 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=15.5 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=14.3 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.312/15.592/16.939/1.083 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=16.0 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=20.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=16.8 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.048/17.871/20.739/2.058 ms
---------------------------------------------------
host2 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=16.6 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=16.9 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.222/16.583/16.924/0.286 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.6 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=18.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=26.8 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.620/20.549/26.887/4.524 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=33.8 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=27.7 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=20.2 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 20.216/27.259/33.818/5.566 ms
---------------------------------------------------
host3 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=14.5 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=19.5 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=15.5 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.535/16.549/19.541/2.162 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=16.4 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=14.6 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=18.2 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.643/16.458/18.278/1.491 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_seq=1 ttl=61 time=15.3 ms
64 bytes from 192.168.4.2: icmp_seq=2 ttl=61 time=14.1 ms
64 bytes from 192.168.4.2: icmp_seq=3 ttl=61 time=13.1 ms
--- 192.168.4.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 13.171/14.234/15.334/0.883 ms
---------------------------------------------------
host4 sends 3 ping the 3 other hosts
---------------------------------------------------
PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=61 time=21.9 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=61 time=21.3 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=61 time=14.6 ms
--- 192.168.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 14.619/19.284/21.932/3.310 ms
PING 192.168.2.2 (192.168.2.2) 56(84) bytes of data.
64 bytes from 192.168.2.2: icmp_seq=1 ttl=61 time=14.8 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=61 time=18.4 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=61 time=14.1 ms
--- 192.168.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.148/15.786/18.409/1.879 ms
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_seq=1 ttl=61 time=16.2 ms
64 bytes from 192.168.3.2: icmp_seq=2 ttl=61 time=15.1 ms
64 bytes from 192.168.3.2: icmp_seq=3 ttl=61 time=18.9 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 15.117/16.788/18.962/1.612 ms
---------------------------------------------------
If bot spine have all BGP sessions and all ping succeeded, all works well
Congrats ! please check readme file to find some details
---------------------------------------------------