This section provides examples of topologies and applications that demonstrate the benefits of FSPF.

Note	The FSPF feature can be used on any topology.

The SPF computational hold time sets the minimum time between two consecutive SPF computations on the VSAN. Setting this to a small value means that FSPF reacts faster to any fabric changes by recomputing paths on the VSAN. A small SPF computational hold time uses more switch CPU time.

Each time a new switch enters the fabric, a link state record (LSR) is sent to the neighboring switches, and then flooded throughout the fabric. Table 1 displays the default settings for switch responses.

The LSR minimum arrival time is the period between receiving LSR updates on this VSAN. Any LSR updates that arrive before the LSR minimum arrival time are discarded.

The LSR minimum interval time is the frequency at which this switch sends LSR updates on a VSAN.

FSPF tracks the state of links on all switches in the fabric, associates a cost with each link in its database, and then chooses the path with a minimal cost. The cost associated with an interface can be administratively changed to implement the FSPF route selection. The integer value to specify cost can range from 1 to 30000. The default cost for 1 Gbps is 1000 and for 2 Gbps is 500.

FSPF uses link costs to determine the shortest path between devices in a fabric. The default link costs become inefficient when calculating the cost of larger capacity port channels. Such paths may appear to have the same cost although they have different bandwidths leading to poor path selection by FSPF. The FSPF cost multiplier feature allows reassigning of links costs so that FSPF can calculate and select optimal high-speed paths.

Path cost calculation inefficiencies can occur when the total link bandwidth is over 128 Gbps. This feature should be configured when parallel paths above this threshold exist in a fabric so that FSPF selects paths as expected. A port channel can have a maximum of 16 member links so path inefficiencies may occur when port channels with as low as 9 x 16-Gbps members are present.

All switches in a fabric must use the same FSPF cost multiplier so that they all use the same basis for path cost calculations. This feature automatically distributes the configured FSPF cost multiplier to all Cisco MDS switches in the fabric with Cisco NX-OS versions that support the feature. If any switches are present in the fabric that do not support the feature, then the configuration fails and not applied to any switches. After the cost multiplier is accepted by all switches, a delay of 20 seconds occurs before being applied to ensure that all switches apply the update simultaneously. If the link costs do not change, there will not be any traffic disruption. However, if the update results in a different path selection by FSPF there may be a brief, one-time interruption to traffic as the new path is applied.

The link cost of an interface may also be manually changed in the default value. For more information, see About FSPF Link Cost section.

You can set the FSPF Hello time interval to specify the interval between the periodic hello messages sent to verify the health of the link. The integer value can range from 1 to 65,535 seconds.

Note	This value must be the same in the ports at both ends of the ISL.

You can set the FSPF dead time interval to specify the maximum interval for which a hello message must be received before the neighbor is considered lost and removed from the database. The integer value can range from 1 to 65,535 seconds.

Note	This value must be the same in the ports at both ends of the ISL.

• An error is reported at the command prompt if the configured dead time interval is less than the hello time interval
• During a software upgrade, ensure that the fspf dead-interval is greater than the ISSU downtime (80 seconds). If the fspf dead-interval is lesser than the ISSU downtime, the software upgrade fails and the following error is displayed:

Service "fspf" returned error: Dead interval for interface is less than ISSU upgrade time. 

You can specify the time after which an unacknowledged link state update should be transmitted on the interface. The integer value to specify retransmit intervals can range from 1 to 65,535 seconds.

Note	This value must be the same on the switches on both ends of the interface.

You can disable the FSPF protocol for selected interfaces. By default, FSPF is enabled on all E ports and TE ports. This default can be disabled by setting the interface as passive.

Note	FSPF must be enabled at both ends of the interface for the protocol to work.

Each port implements forwarding logic, which forwards frames based on its FC ID. Using the FC ID for the specified interface and domain, you can configure the specified route (for example FC ID 111211 and domain ID 3) in the switch with domain ID 1 (see Fibre Channel Routes).

Note	Other than in VSANs, runtime checks are not performed on configured and suspended static routes.

Broadcast and multicast in a Fibre Channel fabric uses the concept of a distribution tree to reach all switches in the fabric.

FSPF provides the topology information to compute the distribution tree. Fibre Channel defines 256 multicast groups and one broadcast address for each VSAN. Switches in the Cisco MDS 9000 Family only use broadcast routing. By default, they use the principal switch as the root node to derive a loop-free distribution tree for multicast and broadcast routing in a VSAN.

Caution

All switches in the fabric should run the same multicast and broadcast distribution tree algorithm to ensure the same distribution tree.

To interoperate with other vendor switches (following FC-SW3 guidelines), the SAN-OS and NX-OS 4.1(1b) and later software uses the lowest domain switch as the root to compute the multicast tree in interop mode.

By default, the native (non-interop) mode uses the principal switch as the root. If you change the default, be sure to configure the same mode in all switches in the fabric. Otherwise, multicast traffic could encounter potential loop and frame-drop problems.

Note	The operational mode can be different from the configured interop mode. The interop mode always uses the lowest domain switch as the root.

Use the mcast root lowest vsan command to change the multicast root from the principal switch to lowest domain switch.

The following types of load balancing schemes are supported:

• Flow based—All frames between a given source FCID and destination FCID are transmitted on the same link. That is, whichever link is selected for the first exchange between the source-destination pair is used for all subsequent exchanges.

• Exchange based—The first frame in an exchange between a given source FCID and destination FCID is used to select an egress link and subsequent frames in the exchange are transmitted on the same link. However, subsequent exchanges between the source-destination pair will likely be transmitted on a different link. This provides more granular load balancing while preserving the order of frames within each exchange.

Flow Based Load Balancing illustrates how flow based load balancing works. In this example, when the first frame with a source FCID of sid1 and destination FCID of did1 is received for forwarding, port channel 2 is selected. Each subsequent frame in that flow is sent over the same port channel. No frame from sid1 to did1 utilizes port channel 1. Similarly, all frames with sid2 and did2 are sent over port channel 1. Exchange ID is not used with this type of load balancing.

Exchange Based Load Balancing illustrates how exchange based load balancing works. In this example, when the first frame in an exchange between a source FCID sid1 and destination FCID did1 is received for forwarding, port channel 2 is selected. All remaining frames in that particular exchange are sent on the same port channel and none are sent on port channel 1. For the next exchange, the hash algorithm chooses port channel 1. So all frames in exchange 2 between the same source-destination pair are sent on port channel 1.

Load balancing is applied to an ingress frame at two levels—At the first level, an ECMP hash is used to select an egress ECMP interface (this can be either a physical interface or logical interface such as a port channel interface) and at the second level, a port channel hash is used to select an egress port channel member.

By default, the hash method that is used depends on the ingress hardware type. If either level of hash does not apply to the egress route, then no hash method is applied.

The following types of hashing methods are supported:

• ECMP Hashing Method—If multiple paths to a destination with equal cost exist in the switch, the FIB for the ingress port is updated with these paths for that destination. This hashing method is used to select one of such paths to send frames to.

• Port Channel Hashing Method—This hashing method is used to select an operational interface of an egress port channel.

ECMP Hashing Method illustrates how the ECMP hash method works. There are two port channels each including two equal speed links. Since the FSPF costs of the port channels are the same, both port channels are used for hashing. In this example, ECMP level hashing method selects port channel 2 as the egress port.

Depending on the type of ingress port, the following subtypes of ECMP hashing methods are supported:

• Type 1a

• Type 1b

For information on which hashing method is selected for a given ingress port, see Table 1.

Port Channel Hashing Method illustrates how port channel hashing method works. Continuing the ECMP Hashing Method example where port channel 2 was selected as the egress port, a port channel hash is subsequently applied to select an egress port within the port channel. In this example, the frames are transmitted by interface 1/4 of the selected port channel.

Depending on the type of ingress port, the following types of port channel hashing methods are supported:

• Type 2a

• Type 2b

For information on which hashing method is selected for a given ingress port, see Table 1.

When you experience a route change in the network, the new selected path may be faster or less congested than the old route.

In Route Change Delivery, the new path from Switch 1 to Switch 4 is faster. In this scenario, Frame 3 and Frame 4 may be delivered before Frame 1 and Frame 2.

If the in-order guarantee feature is enabled, the frames within the network are treated as follows:

• Frames in the network are delivered in the order in which they are transmitted.

• Frames that cannot be delivered in order within the network latency drop period are dropped inside the network.

When a link change occurs in a PortChannel, the frames for the same exchange flow or the same initiator-target flow can switch from one path to another faster path.

In Link Congestion Delivery , the port of the old path (black dot) is congested. In this scenario, Frame 3 and Frame 4 can be delivered before Frame 1 and Frame 2.

The in-order delivery feature attempts to minimize the number of frames dropped during PortChannel link changes when the in-order delivery is enabled by sending a request to the remote switch on the PortChannel to flush all frames for this PortChannel.

Note	Both switches on the PortChannel must be running Cisco SAN-OS Release 3.0(1) for this IOD enhancement, known as Lossless IOD. For earlier releases, IOD waits for the switch latency period before sending new frames.

When the in-order delivery guarantee feature is enabled and a PortChannel link change occurs, the frames crossing the PortChannel are treated as follows:

• Frames using the old path are delivered before new frames are accepted.

• The new frames are delivered through the new path after the switch latency drop period has elapsed and all old frames are flushed.

Frames that cannot be delivered in order through the old path within the switch latency drop period are dropped. See the Configuring the Drop Latency Time.

You can enable the in-order delivery feature for a specific VSAN or for the entire switch. By default, in-order delivery is disabled on switches in the Cisco MDS 9000 Series.

Note	Enabling or disabling the IOD feature does not disrupt traffic.

Tip

We recommend that you only enable this feature when devices that cannot handle any out-of-order frames are connected to the fabric. Load-balancing algorithms within the Cisco MDS 9000 Series ensure that frames are delivered in order during normal fabric operation. The load-balancing algorithms based on source FC ID, destination FC ID, and exchange ID are enforced in hardware without any performance degradation. However, if the fabric encounters a failure and this feature is enabled, the recovery will be delayed because of an intentional pausing of fabric forwarding to purge the fabric of resident frames that could potentially be forwarded out-of-order.

Use the show in-order-guarantee command to display the present configuration status:

switch# show in-order-guarantee
global inorder delivery configuration:guaranteed
VSAN specific settings
vsan 1 inorder delivery:guaranteed
vsan 101 inorder delivery:not guaranteed
vsan 1000 inorder delivery:guaranteed
vsan 1001 inorder delivery:guaranteed
vsan 1682 inorder delivery:guaranteed
vsan 2001 inorder delivery:guaranteed
vsan 2009 inorder delivery:guaranteed
vsan 2456 inorder delivery:guaranteed
vsan 3277 inorder delivery:guaranteed
vsan 3451 inorder delivery:guaranteed
vsan 3452 inorder delivery:guaranteed

You can view the configured latency parameters using the show fcdroplatency command (see Displays Administrative Distance).

switch# show fcdroplatency
 
switch latency value:500 milliseconds
global network latency value:2000 milliseconds
VSAN specific network latency settings
vsan 1 network latency:5000 milliseconds
vsan 2 network latency:2000 milliseconds
vsan 103 network latency:2000 milliseconds
vsan 460 network latency:500 milliseconds

If you enable flow counters, you can enable a maximum of 1 K entries for aggregate flow and flow statistics for Generation 1 modules, and 2 K entries for Generation 2 modules. Be sure to assign an unused flow index to a module for each new flow. Flow indexes can be repeated across modules. The number space for flow index is shared between the aggregate flow statistics and the flow statistics.

Generation 1 modules allow a maximum of 1024 flow statements per module. Generation 2 modules allow a maximum of 2048-128 flow statements per module.

Note	For each session, fcflow counter will increment only on locally connected devices and should be configured on the switch where the initiator is connected.

Use the clear fcflow stats command to clear the aggregated flow counter (see Examples Clears Aggregated Flow Counters and Clears Flow Counters for Source and Destination FC IDs).

switch# clear fcflow stats aggregated module 2 index 1

switch# clear fcflow stats module 2 index 1

Use the show fcflow stats commands to view flow statistics (see Example Displays Aggregated Flow Details for the Specified Module to Displays Flow Index Usage for the Specified Module).

switch# show fcflow stats aggregated module 6
Idx  VSAN frames 			 		 bytes
---- ---- -------- 		 -------
1	 800 	 20185860	 		 1211151600

switch# show fcflow stats module 6
Idx 	 VSAN 		DID 			SID 		 Mask     		 frames 		 	 bytes
---- ----- 	------- 			 	 ------		 	 ----- 		 	-----		 ------ 
2 	 800   0x520400 		 0x530260 	 0xffffff 		 		20337793 1220267580

switch# show fcflow stats usage module 6
Configured flows for module 6: 1-2

Displays FSPF Information for a Specified VSAN displays global FSPF information for a specific VSAN:

• Domain number of the switch.
• Autonomous region for the switch.
• Min_LS_arrival: minimum time that must elapse before the switch accepts LSR updates.
• Min_LS_interval: minimum time that must elapse before the switch can transmit an LSR.

Tip	If the Min_LS_interval is higher than 10 seconds, the graceful shutdown feature is not implemented.

• LS_refresh_time: interval time lapse between refresh LSR transmissions.
• Max_age: maximum time aa LSR can stay before being deleted.

switch# show fspf vsan 1
FSPF routing for VSAN 1
FSPF routing administration status is enabled
FSPF routing operational status is UP
It is an intra-domain router
Autonomous region is 0
SPF hold time is 0 msec
MinLsArrival = 1000 msec , MinLsInterval = 5000 msec
Local Domain is 0x65(101)
Number of LSRs = 3, Total Checksum = 0x0001288b
Protocol constants :
   LS_REFRESH_TIME = 1800 sec
   MAX_AGE         = 3600 sec
Statistics counters :
   Number of LSR that reached MaxAge = 0
   Number of SPF computations        = 7
   Number of Checksum Errors         = 0
   Number of Transmitted packets :  LSU 65 LSA 55 Hello 474 Retranmsitted LSU 0
   Number of received packets :  LSU 55 LSA 60 Hello 464 Error packets 10

Displays FSPF Database Information (Prior to Cisco MDS NX-OS Release 9.4(1)) displays a summary of the FSPF database for a specified VSAN. If other parameters are not specified, all LSRs in the database are displayed:

• LSR type

• Domain ID of the LSR owner

• Domain ID of the advertising router

• LSR age

• LSR incarnation member

• Number of links

You could narrow the display to obtain specific information by issuing additional parameters for the domain ID of the LSR owner. For each interface, the following information is also available:

• Domain ID of the neighboring switch

• E port index

• Port index of the neighboring switch

• Prior to Cisco MDS NX-OS Release 9.4(1), the Link type is numerical.

• From Cisco MDS NX-OS Release 9.4(1), the Link type is alphnumercial and the following types.

  Table 5. Link Tyoe

  Link Type	Description
  P2P	Peer-to-peer interfaces connections
  FCIP PC	Fibre Channel over IP Protocol (FCIP) connection
  FC PC	Fibre Channel connections
  VFC PC	virtual Fibre Channel connections

• Cost

switch# show fspf database vsan 1
FSPF Link State Database for VSAN 1 Domain 0x0c(12)
LSR Type                = 1
Advertising domain ID   = 0x0c(12)
LSR Age                 = 1686
LSR Incarnation number  = 0x80000024
LSR Checksum            = 0x3caf
Number of links         = 2
 NbrDomainId      IfIndex   NbrIfIndex    Link Type         Cost
-----------------------------------------------------------------------------
   0x65(101) 0x0000100e     0x00001081               1          500
   0x65(101) 0x0000100f     0x00001080               1          500
FSPF Link State Database for VSAN 1 Domain 0x65(101)
LSR Type                = 1
Advertising domain ID   = 0x65(101)
LSR Age                 = 1685
LSR Incarnation number  = 0x80000028
LSR Checksum            = 0x8443
Number of links         = 6
 NbrDomainId      IfIndex   NbrIfIndex    Link Type         Cost
-----------------------------------------------------------------------------
   0xc3(195) 0x00001085     0x00001095               1          500
   0xc3(195) 0x00001086     0x00001096               1          500
   0xc3(195) 0x00001087     0x00001097               1          500
   0xc3(195) 0x00001084     0x00001094               1          500
    0x0c(12) 0x00001081     0x0000100e               1          500
    0x0c(12) 0x00001080     0x0000100f               1          500
FSPF Link State Database for VSAN 1 Domain 0xc3(195)
LSR Type                = 1
Advertising domain ID   = 0xc3(195)
LSR Age                 = 1686
LSR Incarnation number  = 0x80000033
LSR Checksum            = 0x6799
Number of links         = 4
 NbrDomainId      IfIndex   NbrIfIndex    Link Type         Cost
-----------------------------------------------------------------------------
   0x65(101) 0x00001095     0x00001085               1          500
   0x65(101) 0x00001096     0x00001086               1          500
   0x65(101) 0x00001097     0x00001087               1          500
   0x65(101) 0x00001094     0x00001084               1          500

switch# show fspf database vsan 1
FSPF Link State Database for VSAN 1 Domain 0xd8(216)
LSR Type                = 1
Advertising domain ID   = 0xd8(216)
LSR Age                 = 646
LSR Incarnation number  = 0x80001c06
LSR Checksum            = 0x0e03
Number of links         = 5
    NbrDomainId           IfIndex(Interface Name)  NbrIfIndex  Link Type   Cost
    ---------------------------------------------------------------------------
      0xe3(227)  0x00010312(              fc4/19)  0x00010011        P2P     62
      0xe3(227)  0x00010313(              fc4/20)  0x0001000e        P2P     62
      0xdb(219)  0x0004003b(      port-channel60)  0x0004003b    FCIP PC    100
      0xdb(219)  0x000400ff(     port-channel256)  0x000400ff      FC PC     31
      0x59(89)   0x00fb0200(           vfc-po513)  0x00fb0200     VFC PC     50

Displays FSPF Interface Information displays the following information for each selected interface.

• Link cost
• Timer values
• Neighbor’s domain ID (if known)
• Local interface number
• Remote interface number (if known)
• FSPF state of the interface
• Interface counters

switch# show fspf vsan 1 interface fc1/1
FSPF interface fc1/1 in VSAN 1
FSPF routing administrative state is active
Interface cost is 500
Timer intervals configured, Hello 20 s, Dead 80 s, Retransmit 5 s
FSPF State is FULL
Neighbor Domain Id is 0x0c(12), Neighbor Interface index is 0x0f100000
Statistics counters :
   Number of packets received : LSU  8  LSA  8  Hello 118  Error packets 0
   Number of packets transmitted : LSU  8  LSA  8  Hello 119  Retransmitted LSU 0
   Number of times inactivity timer expired for the interface = 0