Difference between revisions of "Mvendor-VXLAN-lab"

From NesevoWiki
Jump to navigationJump to search
(Created page with "<div> = Useful Infos = == <span>Lab Target</span> == The target of the lab is to build a multivendor lab, with an architecture reflecting those deployed in customers' environ...")
 
Line 1: Line 1:
−
<div>
+
 
−
= Useful Infos =
+
= Introduction =
 +
 
 +
This lab was realized early 2021. So different behaviors could be observed today with last releases, in a good or a bad way.
  
 
== <span>Lab Target</span> ==
 
== <span>Lab Target</span> ==
−
The target of the lab is to build a multivendor lab, with an architecture reflecting those deployed in customers' environment?.
+
 
 +
The target of the lab is to build a multivendor lab, with an architecture reflecting those deployed in customers' environment.
  
 
This lab will be constructed in several steps, starting in a virtual version, followed by a physical one.
 
This lab will be constructed in several steps, starting in a virtual version, followed by a physical one.
  
−
For the moment, we are working on typical "modern" datacenters implementations :<div></div>
+
For the moment, we are working on typical "modern" datacenters implementations :
  
 
* EVPN/VXLAN layer 2 only
 
* EVPN/VXLAN layer 2 only
 
* EVPN/VXLAN layer 2 + layer 3 per rack (ou layer 3 anycast gw)
 
* EVPN/VXLAN layer 2 + layer 3 per rack (ou layer 3 anycast gw)
−
* EVPN en MPLS en L2/L3
+
* EVPN within MPLS en L2/L3
 
* "standard" IP Fabric
 
* "standard" IP Fabric
−
 
−
but there are also labs for SP customers running LDP/OSPF or SR/ISIS for EVPN/L3VPN services.
 
−
 
−
One goal is to spot/identify the (in)compatibilities between the different vendors, and build the different configurations to be able to provide "ready-to-deploy" setup. In add, working permits to securize the project of vendors migration / NOS migration (ex : EOS to Sonic for Arista).
 
−
 
−
Once the different setup consolidated, we can begin an automation work, within tools like ansible/saltstack/other, in parallel with the implementation of propietary vendor tools (like Arista Cloudvision).
 
−
 
−
The idea behing of all of that is to gain a good knwoledge of the products, wihtou necessarily being expert, but ideally with front-desk-setup, reflecting "the real life". In add, for me a work has to be led to implement a ZTP/ material validation setup, in order to improve/securize at most the deployment/assets management.
 
−
 
−
Now, the more vendors we have the best it is. We have to sollicitate the different vendors and communicate about the different setup in a "high level manner", without taking publicly position. The idea is to stay neutral whatever happens.
 
  
 
== <span>Host specifications</span> ==
 
== <span>Host specifications</span> ==
−
For the moment, I'm working on an OVH dedicated server.
 
−
 
−
<span>Commercial reference :</span> RISE-LE-1 - Intel Xeon E5-1660v3 - 64GB DDR4 ECC 2133MHz - 2x HDD SATA 4TB Datacenter Class Soft RA
 
  
−
Système (OS) VMware ESXi 6.7 U3
+
Server : - Intel Xeon E5-1660v3 - 64GB DDR4 ECC 2133MHz - 2x HDD SATA 4TB Datacenter Class Soft RA
 +
System : VMware ESXi 6.7 U3
  
−
<span>Software versions :</span>
+
Software versions :<
  
 
All network devices instances run within EVE-NG software , here EVE-NG 2.0.3-110 The software has been installed with an OVF available on the website https://www.eve-ng.net/
 
All network devices instances run within EVE-NG software , here EVE-NG 2.0.3-110 The software has been installed with an OVF available on the website https://www.eve-ng.net/
  
−
Regarding the vendors images, they have been downloaded on the vendors portals (or provided by them directly). For Juniper you need a "special" account to be allowed to download the images, on Arista it's free even you are a simple user. For Nokia I don't know, but I suppose it's just in deal contexts since the licences are mandatory and need to be generated by Nokia guys.<div></div>
+
Regarding the vendors images, they have been downloaded on the vendors portals (or provided by them directly).  
  
 
* Arista : veos-4.24.1F
 
* Arista : veos-4.24.1F
Line 49: Line 40:
 
* Juniper : root:Juniper
 
* Juniper : root:Juniper
 
* Cumulus : cumulus:CumulusLinux!
 
* Cumulus : cumulus:CumulusLinux!
−
</div>
+
 
 +
 
 +
= GOAL =
 +
The target of this lab is to put in place a setup providing a "cross-dc" switch, multi-vendor, based on EVPN/VXLAN technology.
 +
 
 +
In this lab, we have :
 +
 
 +
* Arista devices as Spine nodes
 +
* Spine nodes carry directly DCI connections
 +
* Local Spine share the same autonomous system
 +
* Leaf devices are from different vendors : Juniper, Arista, Nokia and Cumulus
 +
* attached host are single-homed and are simulated by Arista instances. Thus allowing to test N vlans at the same time
 +
* EVPN uses VXLAN encapsulation, the control plane is BGP base. All nodes have dedicated private ASN (except for local SPINE)
 +
* Underlay is managed by eBGP sessions, where only Loopback IPs are distributed. Those IPs are used as VTEP IPs by LEAF nodes
 +
* Overlay is managed by eBGP multihop sessions between loopback IPs. Well think to only activate EVPN family for those sessions + define the upsate-source IP (loopback IP)
 +
 
 +
= LAB DIAGRAM =
 +
[[Datei:Lab-mvendor-vxlan-l2-nok-ars-jun-cum.jpeg|klasse=ve-pasteProtect]]
 +
 
 +
= Configurations =
 +
[[:Datei:Config-lab-mvendor-vxlan-l2-nok-ars-jun-cum.zip|Datei:config-lab-mvendor-vxlan-l2-nok-ars-jun-cum.zip]]
 +
 
 +
== Note ==
 +
Juniper vQFX supports only vlan-aware-bundle service in VXLAN (one EVPN instance for N VLAN)
 +
 
 +
On Arista you can choose either use VLAN-BASED or VLAN-AWARE-BUNDLE
 +
 
 +
<syntaxhighlight lang="bash">
 +
    vlan 100
 +
      rd 10.255.255.5:1
 +
      route-target both 65000:100
 +
      redistribute learned
 +
</syntaxhighlight>
 +
or
 +
<syntaxhighlight lang="bash">
 +
    vlan-aware-bundle LAB
 +
      rd 10.255.255.5:1
 +
      route-target both 65000:100
 +
      redistribute learned
 +
      vlan 100
 +
</syntaxhighlight>
 +
 
 +
On Nokia I've only found the way to do vlan-based.
 +
 
 +
It can be seen in type-routes (ether tag id part in yellow bellow) :
 +
 
 +
Arista vlan-based :
 +
 
 +
<syntaxhighlight lang="bash">
 +
{master:0}
 +
root@QFX_DC1> show route | match 2:10.255.255.5
 +
2:10.255.255.5:1::0::50:01:00:3e:d9:dc/304 MAC/IP
 +
2:10.255.255.5:1::0::50:01:00:3e:d9:dc/304 MAC/IP
 +
</syntaxhighlight>
 +
 
 +
Arista vlan-bundle (the ::0:: is replaced by ::100::):
 +
<syntaxhighlight lang="bash">
 +
{master:0}
 +
root@QFX_DC1> show route | match 2:10.255.255.8
 +
2:10.255.255.8:1::100::50:01:00:cd:ff:68/304 MAC/IP
 +
2:10.255.255.8:1::100::50:01:00:cd:ff:68/304 MAC/IP
 +
</syntaxhighlight>
 +
 
 +
Juniper vlan-bundle :
 +
 
 +
<syntaxhighlight lang="bash">
 +
{master:0}
 +
root@QFX_DC1> show route | match 2:10.255.255.10
 +
2:10.255.255.10:1::100::50:01:00:36:5f:a1/304 MAC/IP
 +
2:10.255.255.10:1::100::50:01:00:36:5f:a1/304 MAC/IP
 +
</syntaxhighlight>
 +
 
 +
Nokia vlan based :
 +
 
 +
<syntaxhighlight lang="bash">
 +
{master:0}
 +
root@QFX_DC1> show route | match 2:10.255.255.9
 +
 
 +
2:10.255.255.9:100::0::50:01:00:06:09:e7/304 MAC/IP<div></div>
 +
</syntaxhighlight>
 +
 
 +
So, the border effect is that on Arista devices, fdb install routes basing on ether tag id. Juniper doesn't do that (as Nokia)
 +
 
 +
Juniper DC1 output
 +
 
 +
<syntaxhighlight lang="bash">
 +
 
 +
{master:0}
 +
root@QFX_DC1> show ethernet-switching table
 +
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static<div></div>
 +
            SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
 +
Ethernet switching table : 6 entries, 6 learned
 +
Routing instance : default-switch<div></div>
 +
    Vlan                MAC                MAC      Logical                Active
 +
    name                address            flags    interface              source
 +
    VL_100              50:01:00:06:09:e7  D        vtep.32770            10.255.255.9                  NOKIA-DC2
 +
    VL_100              50:01:00:36:5f:a1  D        vtep.32773            10.255.255.10                JUNIPER-DC2
 +
    VL_100              50:01:00:3e:d9:dc  D        vtep.32769            10.255.255.5                  ARISTA-DC1
 +
    VL_100              50:01:00:65:ae:e8  D        vtep.32772            10.255.255.6                  NOKIA-DC1
 +
    VL_100              50:01:00:96:6b:29  D        xe-0/0/3.0         
 +
    VL_100              50:01:00:cd:ff:68  D        vtep.32771            10.255.255.8                    ARISTA-DC2
 +
 
 +
Nokia DC1 output
 +
 
 +
<syntaxhighlight lang="bash">
 +
Forwarding Database, Service 100 ServId    MAC              Source-Identifier      Type    Last Change
 +
            Transport:Tnl-Id                        Age     
 +
 
 +
----100        50:01:00:06:09:e7 vxlan-1:                Evpn    10/15/20 08:50:03                          NOKIA-DC2<div></div>
 +
                              10.255.255.9:100
 +
100        50:01:00:36:5f:a1 vxlan-1:                Evpn    10/15/20 08:50:03                              JUNIPER-DC2<div></div>
 +
                              10.255.255.10:100
 +
100        50:01:00:3e:d9:dc vxlan-1:                Evpn    10/15/20 09:16:27                          ARISTA-DC1<div></div>
 +
                              10.255.255.5:100
 +
100        50:01:00:65:ae:e8 sap:1/1/c2/1:100        L/150    10/13/20 16:51:07
 +
 
 +
100        50:01:00:96:6b:29 vxlan-1:                Evpn    10/15/20 08:50:03                            JUNIPER-DC1<div></div>
 +
                              10.255.255.7:100
 +
100        50:01:00:cd:ff:68 vxlan-1:                Evpn    10/15/20 08:51:55                              ARISTA-DC2<div></div>
 +
                              10.255.255.8:100
 +
 
 +
</syntaxhighlight>
 +
 
 +
Arista DC2 output
 +
 
 +
vlan-aware bundle :
 +
 
 +
<syntaxhighlight lang="bash">
 +
LEAF-ARS-DC2#show mac address-table
 +
          Mac Address Table
 +
<div></div>
 +
----Vlan    Mac Address      Type        Ports      Moves  Last Move<div></div>
 +
---------------      ----        -----      -----  ---------<div></div>
 +
  100    5001.0036.5fa1    DYNAMIC    Vx1        1      0:37:45 ago    JUNIPER-DC2
 +
 +
  100    5001.003e.d9dc    DYNAMIC    Vx1        1      0:13:14 ago    ARISTA-DC1
 +
 +
  100    5001.0096.6b29    DYNAMIC    Vx1        1      0:37:45 ago    JUNIPER-DC1
 +
 +
  100    5001.00cd.ff68    DYNAMIC    Et2        1      1 day, 19:22:18 ago
 +
</syntaxhighlight>
 +
 
 +
Only mac @ for Arista / Juniper are installed even the Nokia routes are well received
 +
 
 +
<syntaxhighlight lang="bash">
 +
LEAF-ARS-DC2#  show bgp evpn rd 10.255.255.6:100
 +
 
 +
BGP routing table information for VRF default
 +
 
 +
Router identifier 10.255.255.8, local AS number 65021
 +
 
 +
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP<div></div>
 +
                    S - Stale, c - Contributing to ECMP, b - backup
 +
 +
                    % - Pending BGP convergence
 +
Origin codes: i - IGP, e - EGP, ? - incomplete
 +
 
 +
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop<div></div>
 +
          Network                Next Hop              Metric  AIGP      LocPref Weight  Path
 +
 +
  * >Ec  RD: 10.255.255.6:100 mac-ip 5001.0065.aee8
 +
 +
                                10.255.255.6          -      100    0      65002 65001 65012 i
 +
 +
  *  ec  RD: 10.255.255.6:100 mac-ip 5001.0065.aee8
 +
 +
                                10.255.255.6          -      100    0      65002 65001 65012 i
 +
 +
  * >Ec  RD: 10.255.255.6:100 imet 10.255.255.6
 +
 +
                                10.255.255.6          -      100    0      65002 65001 65012 i
 +
 +
  *  ec  RD: 10.255.255.6:100 imet 10.255.255.6
 +
 +
                                10.255.255.6          -      100    0      65002 65001 65012 i
 +
</syntaxhighlight>
 +
 
 +
compared to Juniper routes for example :
 +
 
 +
<syntaxhighlight lang="bash">
 +
LEAF-ARS-DC2#  show bgp evpn rd 10.255.255.7:1
 +
 
 +
BGP routing table information for VRF default
 +
 
 +
Router identifier 10.255.255.8, local AS number 65021
 +
 
 +
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP<div></div>
 +
                    S - Stale, c - Contributing to ECMP, b - backup
 +
 +
                    % - Pending BGP convergence
 +
Origin codes: i - IGP, e - EGP, ? - incomplete
 +
 
 +
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop<div></div>
 +
          Network                Next Hop              Metric  AIGP      LocPref Weight  Path
 +
 +
  * >Ec  RD: 10.255.255.7:1 mac-ip 100 5001.0096.6b29
 +
 +
                                10.255.255.7          -      100    0      65002 65001 65013 i
 +
[...]
 +
</syntaxhighlight>
 +
 
 +
Ether tag id makes the difference for me.</div>
 +
 
 +
So, since no type 2 routes is received, the packets from Arista to Nokia are processed as BUM (so sent to all VTEP from the VXLAN instance).
 +
 
 +
Below the sources of different versions of the lab :

Revision as of 15:21, 29 October 2021

Introduction

This lab was realized early 2021. So different behaviors could be observed today with last releases, in a good or a bad way.

Lab Target

The target of the lab is to build a multivendor lab, with an architecture reflecting those deployed in customers' environment.

This lab will be constructed in several steps, starting in a virtual version, followed by a physical one.

For the moment, we are working on typical "modern" datacenters implementations :

  • EVPN/VXLAN layer 2 only
  • EVPN/VXLAN layer 2 + layer 3 per rack (ou layer 3 anycast gw)
  • EVPN within MPLS en L2/L3
  • "standard" IP Fabric

Host specifications

Server : - Intel Xeon E5-1660v3 - 64GB DDR4 ECC 2133MHz - 2x HDD SATA 4TB Datacenter Class Soft RA System : VMware ESXi 6.7 U3

Software versions :<

All network devices instances run within EVE-NG software , here EVE-NG 2.0.3-110 The software has been installed with an OVF available on the website https://www.eve-ng.net/

Regarding the vendors images, they have been downloaded on the vendors portals (or provided by them directly).

  • Arista : veos-4.24.1F
  • Juniper MX : 20.1R1.11-limited-VCP
  • Juniper QFX : 18.4R1.8
  • Juniper SRX : next-gen 20.1R1.11
  • Nokia : 7750 VSR-I timos-20.7R1

Password used :

  • Nokia : admin:nokia2020
  • Arista : admin: <-- no password
  • Juniper : root:Juniper
  • Cumulus : cumulus:CumulusLinux!


GOAL

The target of this lab is to put in place a setup providing a "cross-dc" switch, multi-vendor, based on EVPN/VXLAN technology.

In this lab, we have :

  • Arista devices as Spine nodes
  • Spine nodes carry directly DCI connections
  • Local Spine share the same autonomous system
  • Leaf devices are from different vendors : Juniper, Arista, Nokia and Cumulus
  • attached host are single-homed and are simulated by Arista instances. Thus allowing to test N vlans at the same time
  • EVPN uses VXLAN encapsulation, the control plane is BGP base. All nodes have dedicated private ASN (except for local SPINE)
  • Underlay is managed by eBGP sessions, where only Loopback IPs are distributed. Those IPs are used as VTEP IPs by LEAF nodes
  • Overlay is managed by eBGP multihop sessions between loopback IPs. Well think to only activate EVPN family for those sessions + define the upsate-source IP (loopback IP)

LAB DIAGRAM

klasse=ve-pasteProtect

Configurations

Datei:config-lab-mvendor-vxlan-l2-nok-ars-jun-cum.zip

Note

Juniper vQFX supports only vlan-aware-bundle service in VXLAN (one EVPN instance for N VLAN)

On Arista you can choose either use VLAN-BASED or VLAN-AWARE-BUNDLE 

    vlan 100
       rd 10.255.255.5:1
       route-target both 65000:100
       redistribute learned

or 

    vlan-aware-bundle LAB
       rd 10.255.255.5:1
       route-target both 65000:100
       redistribute learned
       vlan 100

On Nokia I've only found the way to do vlan-based.

It can be seen in type-routes (ether tag id part in yellow bellow) :

Arista vlan-based : 

{master:0}
root@QFX_DC1> show route | match 2:10.255.255.5
2:10.255.255.5:1::0::50:01:00:3e:d9:dc/304 MAC/IP
2:10.255.255.5:1::0::50:01:00:3e:d9:dc/304 MAC/IP

Arista vlan-bundle (the ::0:: is replaced by ::100::): 

{master:0}
root@QFX_DC1> show route | match 2:10.255.255.8
2:10.255.255.8:1::100::50:01:00:cd:ff:68/304 MAC/IP
2:10.255.255.8:1::100::50:01:00:cd:ff:68/304 MAC/IP

Juniper vlan-bundle : 

{master:0}
root@QFX_DC1> show route | match 2:10.255.255.10
2:10.255.255.10:1::100::50:01:00:36:5f:a1/304 MAC/IP
2:10.255.255.10:1::100::50:01:00:36:5f:a1/304 MAC/IP

Nokia vlan based : 

{master:0}
root@QFX_DC1> show route | match 2:10.255.255.9

2:10.255.255.9:100::0::50:01:00:06:09:e7/304 MAC/IP<div></div>

So, the border effect is that on Arista devices, fdb install routes basing on ether tag id. Juniper doesn't do that (as Nokia)

Juniper DC1 output 

{master:0}
root@QFX_DC1> show ethernet-switching table
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static<div></div>
            SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
Ethernet switching table : 6 entries, 6 learned
Routing instance : default-switch<div></div>
    Vlan                MAC                 MAC      Logical                Active
    name                address             flags    interface              source
    VL_100              50:01:00:06:09:e7   D        vtep.32770             10.255.255.9                  NOKIA-DC2
    VL_100              50:01:00:36:5f:a1   D        vtep.32773             10.255.255.10                JUNIPER-DC2
    VL_100              50:01:00:3e:d9:dc   D        vtep.32769             10.255.255.5                  ARISTA-DC1
    VL_100              50:01:00:65:ae:e8   D        vtep.32772             10.255.255.6                  NOKIA-DC1
    VL_100              50:01:00:96:6b:29   D        xe-0/0/3.0          
    VL_100              50:01:00:cd:ff:68   D        vtep.32771             10.255.255.8                    ARISTA-DC2

Nokia DC1 output

<syntaxhighlight lang="bash">
Forwarding Database, Service 100 ServId     MAC               Source-Identifier       Type     Last Change
             Transport:Tnl-Id                         Age      

----100        50:01:00:06:09:e7 vxlan-1:                Evpn     10/15/20 08:50:03                           NOKIA-DC2<div></div>
                              10.255.255.9:100
100        50:01:00:36:5f:a1 vxlan-1:                Evpn     10/15/20 08:50:03                               JUNIPER-DC2<div></div>
                              10.255.255.10:100
100        50:01:00:3e:d9:dc vxlan-1:                Evpn     10/15/20 09:16:27                           ARISTA-DC1<div></div>
                              10.255.255.5:100
100        50:01:00:65:ae:e8 sap:1/1/c2/1:100        L/150    10/13/20 16:51:07

100        50:01:00:96:6b:29 vxlan-1:                Evpn     10/15/20 08:50:03                             JUNIPER-DC1<div></div>
                              10.255.255.7:100
100        50:01:00:cd:ff:68 vxlan-1:                Evpn     10/15/20 08:51:55                               ARISTA-DC2<div></div>
                              10.255.255.8:100

Arista DC2 output

vlan-aware bundle : 

LEAF-ARS-DC2#show mac address-table
           Mac Address Table
<div></div>
----Vlan    Mac Address       Type        Ports      Moves   Last Move<div></div>
---------------       ----        -----      -----   ---------<div></div>
  100    5001.0036.5fa1    DYNAMIC     Vx1        1       0:37:45 ago     JUNIPER-DC2
 
  100    5001.003e.d9dc    DYNAMIC     Vx1        1       0:13:14 ago    ARISTA-DC1
 
  100    5001.0096.6b29    DYNAMIC     Vx1        1       0:37:45 ago     JUNIPER-DC1
 
  100    5001.00cd.ff68    DYNAMIC     Et2        1       1 day, 19:22:18 ago

Only mac @ for Arista / Juniper are installed even the Nokia routes are well received 

LEAF-ARS-DC2#   show bgp evpn rd 10.255.255.6:100

BGP routing table information for VRF default

Router identifier 10.255.255.8, local AS number 65021

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP<div></div>
                     S - Stale, c - Contributing to ECMP, b - backup
 
                     % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop<div></div>
           Network                Next Hop              Metric  AIGP       LocPref Weight  Path
 
  * >Ec   RD: 10.255.255.6:100 mac-ip 5001.0065.aee8
 
                                 10.255.255.6          -       100     0       65002 65001 65012 i
 
  *  ec   RD: 10.255.255.6:100 mac-ip 5001.0065.aee8
 
                                 10.255.255.6          -       100     0       65002 65001 65012 i
 
  * >Ec   RD: 10.255.255.6:100 imet 10.255.255.6
 
                                 10.255.255.6          -       100     0       65002 65001 65012 i
 
  *  ec   RD: 10.255.255.6:100 imet 10.255.255.6
 
                                 10.255.255.6          -       100     0       65002 65001 65012 i

compared to Juniper routes for example : 

LEAF-ARS-DC2#   show bgp evpn rd 10.255.255.7:1

BGP routing table information for VRF default

Router identifier 10.255.255.8, local AS number 65021

Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP<div></div>
                     S - Stale, c - Contributing to ECMP, b - backup
 
                     % - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete

AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop<div></div>
           Network                Next Hop              Metric  AIGP       LocPref Weight  Path
 
  * >Ec   RD: 10.255.255.7:1 mac-ip 100 5001.0096.6b29
 
                                 10.255.255.7          -       100     0       65002 65001 65013 i
[...]

Ether tag id makes the difference for me.

So, since no type 2 routes is received, the packets from Arista to Nokia are processed as BUM (so sent to all VTEP from the VXLAN instance).

Below the sources of different versions of the lab :

Retrieved from "http://wiki.nesevo.com/index.php?title=Mvendor-VXLAN-lab&oldid=2988"