Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Topology

Troubleshoot

Verify

Related Information

Introduction

This document describes basic steps to troubleshoot FabricPath.

Prerequisites

Requirements

Cisco NXOS® recommends that you have knowledge of these topics:

- FabricPath

- Intermediate System to Intermediate System (IS-IS)

- Spanning Tree Protocol (STP)

- Embedded Logic Analyzer Module (ELAM)

Components Used

This document is restricted to specific hardware such as Nexus 7000.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

FabricPath is a Cisco technology that aims to enhance the Ethernet networking capabilities, particularly in large scale data center environments.

Here are key features and benefits of FabricPath on the Cisco Nexus 7000 series:

1. Scalability: FabricPath is designed to allow for a large number of virtual port channels (vPCs) and to provide a highly scalable Layer 2 network that can handle a large number of hosts without the limitations typically associated with Spanning Tree Protocol (STP).

2. Loop-Free Topology: FabricPath eliminates the need for STP within the FabricPath network domain. This is done by using a routing-like technology to forward Ethernet frames, called Transparent Interconnection of Lots of Links (TRILL), which prevents loops and allows all paths to be active.

3. High Availability: With FabricPath, network topology changes are handled more efficiently, reducing the convergence time. This enhances the overall network stability and provides better availability of the network.

4. Ease of Use: The technology simplifies the network design by allowing for a flexible and scalable layer 2 architecture. This makes the network easier to manage and reduces operational complexity.

5. Equal-Cost Multipath (ECMP): FabricPath supports ECMP, enabling the use of multiple parallel paths between any two points in the network. This optimizes bandwidth utilization by load balancing traffic across all available paths.

6. Virtualization Support: FabricPath provides an ideal infrastructure for virtualized data centers and private cloud deployments. Its ability to handle a large number of virtual environments makes it a good fit for these types of applications.

It is also important to note that while FabricPath provides many benefits, it is best used in environments where its specific advantages align with the network design goals, such as data centers that require large, dynamic, and scalable layer 2 domains.

Topology

For simplicity, only one Spine and two leaves are presented in this topology.

Emulated Switch ID Leaf A: 3101

Emulated Switch ID Leaf D :3102

Troubleshoot

Host 1 cannot communicate with Host 2.

Leaf_A# ping 192.168.10.17

PING 192.168.10.17 (192.168.10.17): 56 data bytes

ping: sendto 192.168.10.17 64 chars, No route to host

Request 0 timed out

ping: sendto 192.168.10.17 64 chars, No route to host

^C

--- 192.168.10.17 ping statistics ---

2 packets transmitted, 0 packets received, 100.00% packet loss

Leaf_A#

1) Verify the MAC address table is correctly populated for the two hosts.

Leaf_A# show mac address-table vlan 409

Note: MAC table entries displayed are getting read from software.

Use the 'hardware-age' keyword to get information related to 'Age'

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link,

(T) - True, (F) - False , ~~~ - use 'hardware-age' keyword to retrieve age info

VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

* 409 aaaa.aaaa.aaaa dynamic ~~~ F F Po3

<------ Leaf A is not learning the mac address of Host 2

Leaf_A#

Leaf_D# show mac address-table vlan 409

Note: MAC table entries displayed are getting read from software.

Use the 'hardware-age' keyword to get information related to 'Age'

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link,

(T) - True, (F) - False , ~~~ - use 'hardware-age' keyword to retrieve age info

VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

* 409 bbbb.bbbb.bbbb dynamic ~~~ F F Po4

409 aaaa.aaaa.aaaa dynamic ~~~ F F 3101.1.65535    <------ Leaf D is correctly learning both Mac Addresses.

Leaf_D#

2) Verify the configuration for every interface and vlan involved in the path. FabricPath must be enabled.

Leaf_A# show run fabricpath

!Command: show running-config fabricpath

!Time: Mon Apr 22 23:12:40 2024

version 6.2(12)

install feature-set fabricpath

feature-set fabricpath

vlan 409

mode fabricpath

fabricpath domain default

fabricpath switch-id 301

vpc domain 301

fabricpath switch-id 3101

interface port-channel1

switchport mode fabricpath

interface port-channel2

switchport mode fabricpath

interface Ethernet1/1

switchport mode fabricpath

interface Ethernet1/2

switchport mode fabricpath

Leaf_A#

Leaf_D# show run fabricpath

!Command: show running-config fabricpath

!Time: Mon Apr 22 23:12:40 2024

version 6.2(12)

install feature-set fabricpath

feature-set fabricpath

<--------- FabricPath is not enabled for VLAN 409

fabricpath switch-id 101

vpc domain 302

fabricpath switch-id 3102

interface port-channel1

switchport mode fabricpath

interface port-channel2

switchport mode fabricpath

interface Ethernet1/1

switchport mode fabricpath

interface Ethernet1/2

switchport mode fabricpath

Leaf_D#

Leaf_D#

Spine_A# show run fabricpath

!Command: show running-config fabricpath

!Time: Mon Apr 22 23:12:40 2024

version 6.2(12)

install feature-set fabricpath

feature-set fabricpath

vlan 409

mode fabricpath

fabricpath domain default

fabricpath switch-id 31

vpc domain 101

fabricpath switch-id 1003

interface port-channel1

switchport mode fabricpath

interface port-channel2

switchport mode fabricpath

interface Ethernet1/1

switchport mode fabricpath

interface Ethernet1/2

switchport mode fabricpath

interface Ethernet1/3

switchport mode fabricpath

interface Ethernet1/4

switchport mode fabricpath

Spine_A#

3) Verify the Switch IDs for every device participating in FabricPath.

Leaf_A# show fabricpath switch-id local

Switch-Id: 301

System-Id: aaaa.aaaa.b341

Leaf_A#

Leaf_D# show fabricpath switch-id local

Switch-Id: 101

System-Id: bbbb.bbbb.b342

Leaf_D#

Spine_A# show fabricpath switch-id local

Switch-Id: 31

System-Id: cccc.cccc.b343

Spine_A#

4) Verify the right routes are configured with the switch id of every device.

Leaf_A# show fabricpath route switchid 101

FabricPath Unicast Route Table

'a/b/c' denotes ftag/switch-id/subswitch-id

'[x/y]' denotes [admin distance/metric]

ftag 0 is local ftag

subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default

1/101/0, number of next-hops: 1

via Po1, [115/5], 1 day/s 12:21:29, isis_fabricpath-default <------ The route from Leaf A to Leaf D is correctly configured.

Leaf_A

Leaf_D# show fabricpath route switchid 301

FabricPath Unicast Route Table

'a/b/c' denotes ftag/switch-id/subswitch-id

'[x/y]' denotes [admin distance/metric]

ftag 0 is local ftag

subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default

1/301/0, number of next-hops: 1

via Po2, [115/5], 1 day/s 12:21:29, isis_fabricpath-default  <------ The route from Leaf D to Leaf A is correctly configured.

Leaf_D

Spine_A# show fabricpath route switchid 301

FabricPath Unicast Route Table

'a/b/c' denotes ftag/switch-id/subswitch-id

'[x/y]' denotes [admin distance/metric]

ftag 0 is local ftag

subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default

1/301/0, number of next-hops: 1

via Po1, [115/20], 1 day/s 06:13:21, isis_fabricpath-default <------ The route from Spine A to Leaf A is correctly configured.

Spine_A#

Spine_A# show fabricpath route switchid 101

FabricPath Unicast Route Table

'a/b/c' denotes ftag/switch-id/subswitch-id

'[x/y]' denotes [admin distance/metric]

ftag 0 is local ftag

subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default

1/101/0, number of next-hops: 1

via Po2, [115/20], 1 day/s 06:13:21, isis_fabricpath-default <------ The route from Spine A to Leaf D is correctly configured.

Spine_A#

5) Verify IS-IS adjacency between leaves and spines.

Leaf_A# show fabricpath isis adjacency

Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:

System ID SNPA Level State Hold Time Interface

cccc.cccc.b343 N/A 1 UP 00:00:27 port-channel1

Leaf_A#

Leaf_D# show fabricpath isis adjacency

Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:

System ID SNPA Level State Hold Time Interface

cccc.cccc.b343 N/A 1 UP 00:00:27 port-channel2

Leaf_D#

6) Verify that conflicts are not present in the current deployment.

Leaf_A# show fabricpath conflict all

No Fabricpath ports in a state of resource conflict.

No Switch id Conflicts

No transitions in progress

Leaf_A#

Leaf_D# show fabricpath conflict all

No Fabricpath ports in a state of resource conflict.

No Switch id Conflicts

No transitions in progress

Leaf_D#

Spine_A# show fabricpath conflict all

No Fabricpath ports in a state of resource conflict.

No Switch id Conflicts

No transitions in progress

Spine_A#

7) Verify VLANs are added to IS-IS VLAN range.

Leaf_A# show fabricpath isis vlan-range

Fabricpath IS-IS domain: default

MT-0

Vlans configured:1,409

Leaf_A#

Leaf_D# show fabricpath isis vlan-range

Fabricpath IS-IS domain: default

MT-0

Vlans configured:1             <------ VLAN 409 is not present

Leaf_D

Spine_A# show fabricpath isis vlan-range

Fabricpath IS-IS domain: default

MT-0

Vlans configured:1, 409

Spine_A#

8)Verify if an ELAM is triggered in Spine A.

module-1# show hardware internal dev-port-map      <------ Determine the F4 ASIC that is used for the FE on port Eth1/2. Enter this command in order to verify this.

--------------------------------------------------------------

CARD_TYPE: 48 port 10G

>Front Panel ports:48

--------------------------------------------------------------

Device name Dev role Abbr num_inst:

--------------------------------------------------------------

> Flanker Eth Mac Driver DEV_ETHERNET_MAC MAC_0 6

> Flanker Fwd Driver DEV_LAYER_2_LOOKUP L2LKP 6

> Flanker Xbar Driver DEV_XBAR_INTF XBAR_INTF 6

> Flanker Queue Driver DEV_QUEUEING QUEUE 6

> Sacramento Xbar ASIC DEV_SWITCH_FABRIC SWICHF 2

> Flanker L3 Driver DEV_LAYER_3_LOOKUP L3LKP 6

> EDC DEV_PHY PHYS 7

+-----------------------------------------------------------------------+

+----------------+++FRONT PANEL PORT TO ASIC INSTANCE MAP+++------------+

+-----------------------------------------------------------------------+

FP port | PHYS | MAC_0 | L2LKP | L3LKP | QUEUE |SWICHF

1       0      0       0       0       0     0,1

2       0      0       0       0       0     0,1

...

module-1#

module-1#

module-1# elam asic flanker instance 0

module-1(fln-elam)#

module-1(fln-elam)# elam asic flanker instance 0

module-1(fln-elam)# layer3

module-1(fln-l2-elam)#

module-1(fln-l2-elam)# trigger dbus ipv4 ingress if source-ipv4-address 192.168.10.17

module-1(fln-l2-elam)#

module-1(fln-l2-elam)# trigger rbus ingress if trig

module-1(fln-l2-elam)# start

module-1(fln-l2-elam)#

module-1(fln-l2-elam)#

module-1(fln-l2-elam)# status

ELAM Slot 1 instance 0: L2 DBUS Configuration: trigger dbus ipv4 ingress if source-ipv4-address 192.168.10.17

L2 DBUS: Armed

ELAM Slot 1 instance 0: L2 RBUS Configuration: trigger rbus ingress if trig

L2 RBUS: Armed

module-1(fln-l2-elam)# status

ELAM Slot 1 instance 0: L2 DBUS Configuration: trigger dbus ipv4 ingress if source-ipv4-address 192.168.10.17

L2 DBUS: Armed

ELAM Slot 1 instance 0: L2 RBUS Configuration: trigger rbus ingress if trig

L2 RBUS: Armed                                                                                                      <------ ELAM is not triggered.

module-1(fln-l2-elam)#

9) Add VLAN 409 to FabricPath.

Leaf_D(config)# vlan 409

Leaf_D(config-vlan)# mode fabricpath

Leaf_D(config-vlan)# show run vlan

!Command: show running-config vlan

!Time: Wed Apr 24 20:27:29 2024

version 6.2(12)

vlan 1,409

vlan 409

mode fabricpath

Leaf_D(config-vlan)#

Verify

1) Verify the mac address table.

Leaf_A# show mac address-table vlan 409

Note: MAC table entries displayed are getting read from software.

Use the 'hardware-age' keyword to get information related to 'Age'

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link,

(T) - True, (F) - False , ~~~ - use 'hardware-age' keyword to retrieve age info

VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

* 409 aaaa.aaaa.aaaa dynamic ~~~ F F Po3

409 bbbb.bbbb.bbbb dynamic ~~~ F F 3102.1.65535

Leaf_A#

Leaf_D# show mac address-table vlan 409

Note: MAC table entries displayed are getting read from software.

Use the 'hardware-age' keyword to get information related to 'Age'

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link,

(T) - True, (F) - False , ~~~ - use 'hardware-age' keyword to retrieve age info

VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

* 409 bbbb.bbbb.bbbb dynamic ~~~ F F Po4

409 aaaa.aaaa.aaaa dynamic ~~~ F F 3101.1.65535

Leaf_D#

2) Verify if an ELAM is triggered in Spine A.

module-1# elam asic flanker instance 0

module-1(fln-elam)#

module-1(fln-elam)# elam asic flanker instance 0

module-1(fln-elam)# layer2

module-1(fln-l2-elam)#

module-1(fln-l2-elam)# trigger dbus ipv4 ingress if source-ipv4-address 192.168.10.17

module-1(fln-l2-elam)#

module-1(fln-l2-elam)# trigger rbus ingress if trig

module-1(fln-l2-elam)# start

module-1(fln-l2-elam)#

module-1(fln-l2-elam)# status

ELAM Slot 1 instance 0: L2 DBUS Configuration: trigger dbus ipv4 ingress if source-ipv4-address 192.168.10.17

L2 DBUS: Armed

ELAM Slot 1 instance 0: L2 RBUS Configuration: trigger rbus ingress if trig

L2 RBUS: Armed

module-1(fln-l2-elam)# status

ELAM Slot 1 instance 0: L2 DBUS Configuration: trigger dbus ipv4 ingress if source-ipv4-address 192.168.10.17

L2 DBUS: Triggered             <-------- ELAM triggered

ELAM Slot 1 instance 0: L2 RBUS Configuration: trigger rbus ingress if trig

L2 RBUS: Triggered             <-------- ELAM triggered

module-1(fln-l2-elam)#

3) Verify the connectivity from Leaf A to host A.

Leaf_A# ping 192.168.10.17

PING 192.168.10.17 (192.168.10.17): 56 data bytes

64 bytes from 192.168.10.17: icmp_seq=0 ttl=254 time=1.703 ms

64 bytes from 192.168.10.17: icmp_seq=1 ttl=254 time=1.235 ms

64 bytes from 192.168.10.17: icmp_seq=2 ttl=254 time=1.197 ms

64 bytes from 192.168.10.17: icmp_seq=3 ttl=254 time=3.442 ms

64 bytes from 192.168.10.17: icmp_seq=4 ttl=254 time=1.331 ms

--- 192.168.10.17 ping statistics ---

5 packets transmitted, 5 packets received, 0.00% packet loss

round-trip min/avg/max = 1.197/1.781/3.442 ms

Leaf_A#

Related Information

Cisco FabricPath

Cisco Nexus 7000 Series NX-OS FabricPath Command Reference

Nexus 7000 M3 Module ELAM Procedure