Troubleshoot Control Plane for FabricPath Environments

Introduction

This document describes basic steps to troubleshoot FabricPath.

Prerequisites

Requirements

Cisco NXOS® recommends that you have knowledge of these topics:

• FabricPath
• Intermediate System to Intermediate System (IS-IS)
• Spanning Tree Protocol (STP)
• Embedded Logic Analyzer Module (ELAM)

Components Used

This document is restricted to specific hardware such as Nexus 7000.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

FabricPath is a Cisco technology that aims to enhance the Ethernet networking capabilities, particularly in large scale data center environments.

Here are key features and benefits of FabricPath on the Cisco Nexus 7000 series:

1. Scalability: FabricPath is designed to allow for a large number of virtual port channels (vPCs) and to provide a highly scalable Layer 2 network that can handle a large number of hosts without the limitations typically associated with Spanning Tree Protocol (STP).

2. Loop-Free Topology: FabricPath eliminates the need for STP within the FabricPath network domain. This is done by using a routing-like technology to forward Ethernet frames, called Transparent Interconnection of Lots of Links (TRILL), which prevents loops and allows all paths to be active.

3. High Availability: With FabricPath, network topology changes are handled more efficiently, reducing the convergence time. This enhances the overall network stability and provides better availability of the network.

4. Ease of Use: The technology simplifies the network design by allowing for a flexible and scalable layer 2 architecture. This makes the network easier to manage and reduces operational complexity.

5. Equal-Cost Multipath (ECMP): FabricPath supports ECMP, enabling the use of multiple parallel paths between any two points in the network. This optimizes bandwidth utilization by load balancing traffic across all available paths.

6. Virtualization Support: FabricPath provides an ideal infrastructure for virtualized data centers and private cloud deployments. Its ability to handle a large number of virtual environments makes it a good fit for these types of applications.

It is also important to note that while FabricPath provides many benefits, it is best used in environments where its specific advantages align with the network design goals, such as data centers that require large, dynamic, and scalable layer 2 domains.

Topology

For simplicity, only one Spine and two leaves are presented in this topology.

Emulated Switch ID Leaf A: 3101

Emulated Switch ID Leaf D :3102

Troubleshoot

Host 1 cannot communicate with Host 2.

Leaf_A# ping 192.168.10.17
PING 192.168.10.17 (192.168.10.17): 56 data bytes
ping: sendto 192.168.10.17 64 chars, No route to host
Request 0 timed out
ping: sendto 192.168.10.17 64 chars, No route to host
^C
--- 192.168.10.17 ping statistics ---
2 packets transmitted, 0 packets received, 100.00% packet loss
Leaf_A# 

1) Verify the MAC address table is correctly populated for the two hosts.

Leaf_A# show mac address-table vlan 409
Note: MAC table entries displayed are getting read from software.
Use the 'hardware-age' keyword to get information related to 'Age'

Legend: 
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False , ~~~ - use 'hardware-age' keyword to retrieve age info 
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 409 aaaa.aaaa.aaaa dynamic ~~~ F F Po3
                                                 <------ Leaf A is not learning the mac address of Host 2
Leaf_A#

Leaf_D# show mac address-table vlan 409
Note: MAC table entries displayed are getting read from software.
Use the 'hardware-age' keyword to get information related to 'Age'

Legend: 
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False , ~~~ - use 'hardware-age' keyword to retrieve age info 
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 409 bbbb.bbbb.bbbb dynamic ~~~ F F Po4
409 aaaa.aaaa.aaaa dynamic ~~~ F F 3101.1.65535    <------ Leaf D is correctly learning both Mac Addresses.

Leaf_D#

2) Verify the configuration for every interface and vlan involved in the path. FabricPath must be enabled.

Leaf_A# show run fabricpath

!Command: show running-config fabricpath
!Time: Mon Apr 22 23:12:40 2024

version 6.2(12)
install feature-set fabricpath
feature-set fabricpath

vlan 409
mode fabricpath
fabricpath domain default

fabricpath switch-id 301

vpc domain 301
fabricpath switch-id 3101

interface port-channel1
switchport mode fabricpath

interface port-channel2
switchport mode fabricpath

interface Ethernet1/1
switchport mode fabricpath

interface Ethernet1/2
switchport mode fabricpath

Leaf_A#

Leaf_D# show run fabricpath

!Command: show running-config fabricpath
!Time: Mon Apr 22 23:12:40 2024

version 6.2(12)
install feature-set fabricpath
feature-set fabricpath

                         <--------- FabricPath is not enabled for VLAN 409


fabricpath switch-id 101

vpc domain 302
fabricpath switch-id 3102

interface port-channel1
switchport mode fabricpath

interface port-channel2
switchport mode fabricpath 

interface Ethernet1/1
switchport mode fabricpath

interface Ethernet1/2
switchport mode fabricpath


Leaf_D#
Leaf_D#

Spine_A# show run fabricpath

!Command: show running-config fabricpath
!Time: Mon Apr 22 23:12:40 2024

version 6.2(12)
install feature-set fabricpath
feature-set fabricpath

vlan 409
mode fabricpath
fabricpath domain default

fabricpath switch-id 31

vpc domain 101
fabricpath switch-id 1003

interface port-channel1
switchport mode fabricpath

interface port-channel2
switchport mode fabricpath 

interface Ethernet1/1
switchport mode fabricpath

interface Ethernet1/2
switchport mode fabricpath

interface Ethernet1/3
switchport mode fabricpath

interface Ethernet1/4
switchport mode fabricpath 

Spine_A#

3) Verify the Switch IDs for every device participating in FabricPath.

Leaf_A# show fabricpath switch-id local
Switch-Id: 301
System-Id: aaaa.aaaa.b341
Leaf_A#

Leaf_D# show fabricpath switch-id local
Switch-Id: 101
System-Id: bbbb.bbbb.b342
Leaf_D#

Spine_A# show fabricpath switch-id local
Switch-Id: 31
System-Id: cccc.cccc.b343
Spine_A#

4) Verify the right routes are configured with the switch id of every device.

Leaf_A# show fabricpath route switchid 101        
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id


FabricPath Unicast Route Table for Topology-Default

1/101/0, number of next-hops: 1
via Po1, [115/5], 1 day/s 12:21:29, isis_fabricpath-default <------ The route from Leaf A to Leaf D is correctly configured.
Leaf_A

Leaf_D# show fabricpath route switchid 301       
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id


FabricPath Unicast Route Table for Topology-Default

1/301/0, number of next-hops: 1
via Po2, [115/5], 1 day/s 12:21:29, isis_fabricpath-default  <------ The route from Leaf D to Leaf A is correctly configured.
Leaf_D

Spine_A# show fabricpath route switchid 301       
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id


FabricPath Unicast Route Table for Topology-Default

1/301/0, number of next-hops: 1
via Po1, [115/20], 1 day/s 06:13:21, isis_fabricpath-default <------ The route from Spine A to Leaf A is correctly configured.
Spine_A#


Spine_A# show fabricpath route switchid 101
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id


FabricPath Unicast Route Table for Topology-Default

1/101/0, number of next-hops: 1
via Po2, [115/20], 1 day/s 06:13:21, isis_fabricpath-default <------ The route from Spine A to Leaf D is correctly configured.
Spine_A#

5) Verify IS-IS adjacency between leaves and spines.

Leaf_A# show fabricpath isis adjacency 
Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:
System ID SNPA Level State Hold Time Interface
cccc.cccc.b343 N/A 1 UP 00:00:27 port-channel1

Leaf_A#

Leaf_D# show fabricpath isis adjacency 
Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:
System ID SNPA Level State Hold Time Interface
cccc.cccc.b343 N/A 1 UP 00:00:27 port-channel2

Leaf_D#

6) Verify that conflicts are not present in the current deployment.

Leaf_A# show fabricpath conflict all
No Fabricpath ports in a state of resource conflict.


No Switch id Conflicts

No transitions in progress

Leaf_A#

Leaf_D# show fabricpath conflict all
No Fabricpath ports in a state of resource conflict.


No Switch id Conflicts

No transitions in progress

Leaf_D#

Spine_A# show fabricpath conflict all
No Fabricpath ports in a state of resource conflict.


No Switch id Conflicts

No transitions in progress

Spine_A#

7) Verify VLANs are added to IS-IS VLAN range.

Leaf_A# show fabricpath isis vlan-range
Fabricpath IS-IS domain: default
MT-0
Vlans configured:1,409
Leaf_A#

Leaf_D# show fabricpath isis vlan-range
Fabricpath IS-IS domain: default
MT-0
Vlans configured:1             <------ VLAN 409 is not present
Leaf_D

Spine_A# show fabricpath isis vlan-range
Fabricpath IS-IS domain: default
MT-0
Vlans configured:1, 409
Spine_A#

8)Verify if an ELAM is triggered in Spine A.

module-1# show hardware internal dev-port-map      <------ Determine the F4 ASIC that is used for the FE on port Eth1/2. Enter this command in order to verify this.
--------------------------------------------------------------
CARD_TYPE: 48 port 10G 
>Front Panel ports:48
--------------------------------------------------------------
Device name Dev role Abbr num_inst:
--------------------------------------------------------------
> Flanker Eth Mac Driver DEV_ETHERNET_MAC MAC_0 6
> Flanker Fwd Driver DEV_LAYER_2_LOOKUP L2LKP 6
> Flanker Xbar Driver DEV_XBAR_INTF XBAR_INTF 6
> Flanker Queue Driver DEV_QUEUEING QUEUE 6
> Sacramento Xbar ASIC DEV_SWITCH_FABRIC SWICHF 2
> Flanker L3 Driver DEV_LAYER_3_LOOKUP L3LKP 6
> EDC DEV_PHY PHYS 7
+-----------------------------------------------------------------------+
+----------------+++FRONT PANEL PORT TO ASIC INSTANCE MAP+++------------+
+-----------------------------------------------------------------------+
FP port | PHYS | MAC_0 | L2LKP | L3LKP | QUEUE |SWICHF 
    1       0      0       0       0       0     0,1 
    2       0      0       0       0       0     0,1 
...
module-1#
module-1# 
module-1# elam asic flanker instance 0
module-1(fln-elam)# 
module-1(fln-elam)# elam asic flanker instance 0
module-1(fln-elam)# layer3
module-1(fln-l2-elam)# 
module-1(fln-l2-elam)# trigger dbus ipv4 ingress if source-ipv4-address 192.168.10.17
module-1(fln-l2-elam)# 
module-1(fln-l2-elam)# trigger rbus ingress if trig
module-1(fln-l2-elam)# start
module-1(fln-l2-elam)# 
module-1(fln-l2-elam)# 
module-1(fln-l2-elam)# status
ELAM Slot 1 instance 0: L2 DBUS Configuration: trigger dbus ipv4 ingress if source-ipv4-address 192.168.10.17 
L2 DBUS: Armed
ELAM Slot 1 instance 0: L2 RBUS Configuration: trigger rbus ingress if trig 
L2 RBUS: Armed
module-1(fln-l2-elam)# status
ELAM Slot 1 instance 0: L2 DBUS Configuration: trigger dbus ipv4 ingress if source-ipv4-address 192.168.10.17 
L2 DBUS: Armed
ELAM Slot 1 instance 0: L2 RBUS Configuration: trigger rbus ingress if trig 
L2 RBUS: Armed                                                                                                      <------ ELAM is not triggered. 
module-1(fln-l2-elam)#

9) Add VLAN 409 to FabricPath.

Leaf_D(config)# vlan 409
Leaf_D(config-vlan)# mode fabricpath
Leaf_D(config-vlan)# show run vlan

!Command: show running-config vlan
!Time: Wed Apr 24 20:27:29 2024

version 6.2(12)
vlan 1,409
vlan 409
mode fabricpath


Leaf_D(config-vlan)#

Verify

1) Verify the mac address table.

Leaf_A# show mac address-table vlan 409
Note: MAC table entries displayed are getting read from software.
Use the 'hardware-age' keyword to get information related to 'Age'

Legend: 
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False , ~~~ - use 'hardware-age' keyword to retrieve age info 
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 409 aaaa.aaaa.aaaa dynamic ~~~ F F Po3
409 bbbb.bbbb.bbbb dynamic ~~~ F F 3102.1.65535

Leaf_A#

Leaf_D# show mac address-table vlan 409
Note: MAC table entries displayed are getting read from software.
Use the 'hardware-age' keyword to get information related to 'Age'

Legend: 
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False , ~~~ - use 'hardware-age' keyword to retrieve age info 
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 409 bbbb.bbbb.bbbb dynamic ~~~ F F Po4
409 aaaa.aaaa.aaaa dynamic ~~~ F F 3101.1.65535

Leaf_D#

2) Verify if an ELAM is triggered in Spine A.

module-1# elam asic flanker instance 0
module-1(fln-elam)# 
module-1(fln-elam)# elam asic flanker instance 0
module-1(fln-elam)# layer2
module-1(fln-l2-elam)# 
module-1(fln-l2-elam)# trigger dbus ipv4 ingress if source-ipv4-address 192.168.10.17
module-1(fln-l2-elam)# 
module-1(fln-l2-elam)# trigger rbus ingress if trig
module-1(fln-l2-elam)# start
module-1(fln-l2-elam)# 
module-1(fln-l2-elam)# status
ELAM Slot 1 instance 0: L2 DBUS Configuration: trigger dbus ipv4 ingress if source-ipv4-address 192.168.10.17 
L2 DBUS: Armed
ELAM Slot 1 instance 0: L2 RBUS Configuration: trigger rbus ingress if trig 
L2 RBUS: Armed
module-1(fln-l2-elam)# status
ELAM Slot 1 instance 0: L2 DBUS Configuration: trigger dbus ipv4 ingress if source-ipv4-address 192.168.10.17 
L2 DBUS: Triggered             <-------- ELAM triggered
ELAM Slot 1 instance 0: L2 RBUS Configuration: trigger rbus ingress if trig 
L2 RBUS: Triggered             <-------- ELAM triggered
module-1(fln-l2-elam)# 

3) Verify the connectivity from Leaf A to host A.

Leaf_A# ping 192.168.10.17 
PING 192.168.10.17 (192.168.10.17): 56 data bytes
64 bytes from 192.168.10.17: icmp_seq=0 ttl=254 time=1.703 ms
64 bytes from 192.168.10.17: icmp_seq=1 ttl=254 time=1.235 ms
64 bytes from 192.168.10.17: icmp_seq=2 ttl=254 time=1.197 ms
64 bytes from 192.168.10.17: icmp_seq=3 ttl=254 time=3.442 ms
64 bytes from 192.168.10.17: icmp_seq=4 ttl=254 time=1.331 ms

--- 192.168.10.17 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 1.197/1.781/3.442 ms
Leaf_A#

Related Information

Cisco FabricPath

Cisco Nexus 7000 Series NX-OS FabricPath Command Reference

Nexus 7000 M3 Module ELAM Procedure