-
Cisco MDS 9000 Family Configuration Guide, Release 2.x
-
New and Changed Information
-
Index
-
Preface
- Part 1 - Getting Started
- Part 2 - Cisco MDS SAN-OS Installation and Switch Management
- Part 3 - Switch Configuration
-
Part 4 - Fabric Configuration
-
Configuring and Managing VSANs
-
Creating Dynamic VSANs
-
Configuring Inter-VSAN Routing
-
Configuring Zones
-
Distributing Device Alias Services
-
Configuring Fibre Channel Routing Services and Protocols
-
Managing FLOGI, Name Server, FDMI, and RSCN Databases
-
Discovering SCSI Targets
-
Configuring FICON
-
Advanced Features and Concepts
-
- Part 5 - Security
- Part 6 - IP Services
- Part 7 - Intelligent Storage Services
- Part 8 - Network and Switch Monitoring
- Part 9 - Traffic Management
- Part 10 - Troubleshooting
-
Feedback
Table Of Contents
Configuring Fibre Channel Routing Services and Protocols
Fail-Over Scenarios for PortChannels and FSPF Links
Disabling FSPF for Specific Interfaces
Configuring Fibre Channel Routes
Broadcast and Multicast Routing
Configuring the Drop Latency Time
Displaying Latency Information
Displaying Routing and Forwarding Information
Displaying Global FSPF Information
Configuring Fibre Channel Routing Services and Protocols
Fabric Shortest Path First (FSPF) is the standard path selection protocol used by Fibre Channel fabrics. The FSPF feature is enabled by default on all Fibre Channel switches. Except in configurations that require special consideration, you do not need to configure any FSPF services. FSPF automatically calculates the best path between any two switches in a fabric. Specifically, FSPF is used to:
•
Dynamically compute routes throughout a fabric by establishing the shortest and quickest path between any two switches.
•
This chapter provides details on Fibre Channel routing services and protocols. It includes the following sections:
•
•
•
•
•
FSPF Features
FSPF is the protocol currently standardized by the T11 committee for routing in Fibre Channel networks. The FSPF protocol has the following characteristics and features:
•
•
•
•
•
•
•
FSPF Examples
Note
Fault Tolerant Fabric
Figure 21-1 Fault Tolerant Fabric
For example, if all links are of equal speed, the FSPF calculates two equal paths from A to C: A-D-C (green) and A-E-C (blue).
Redundant Links
To further improve on the topology in Figure 21-1, each connection between any pair of switches can be replicated; two or more links can be present between a pair of switches. Figure 21-2 shows this arrangement. Because switches in the Cisco MDS 9000 Family support PortChanneling, each pair of physical links can appear to the FSPF protocol as one single logical link.
By bundling pairs of physical links, FSPF efficiency is considerably improved by the reduced database size and the frequency of link updates. Once physical links are aggregated, failures are not attached to a single link but to the entire PortChannel. This configuration also improves the resiliency of the network. The failure of a link in a PortChannel does not trigger a route change, thereby reducing the risks of routing loops, traffic loss, or fabric downtime for route reconfiguration.
Figure 21-2 Fault Tolerant Fabric with Redundant Links
For example, if all links are of equal speed and no PortChannels exist, the FSPF calculates four equal paths from A to C: A1-E-C, A2-E-C, A3-D-C, and A4-D-C. If PortChannels exist, these paths are reduced to two.
Fail-Over Scenarios for PortChannels and FSPF Links
The SmartBits traffic generator was used to evaluate the scenarios displayed in Figure 21-3. Two links between switch 1 and switch 2 exist as either equal-cost ISLs or PortChannels. There is one flow from traffic generator 1 to traffic generator 2. The traffic was tested at 100% utilization of 1 Gbps in two scenarios:
•
•
Figure 21-3 Fail-Over Scenario Using Traffic Generators
FSPF Global Configuration
By default, FSPF is enabled on switches in the Cisco MDS 9000 Family.
Some FSPF features can be globally configured in each VSAN. By configuring a feature for the entire VSAN, you do not have to specify the VSAN number for every command. This global configuration feature also reduces the chance of typing errors or other minor configuration errors.
Note
Caution
Global FSPF Configuration
To configure a FSPF feature for the entire VSAN, follow these steps:
FSPF Configuration Deletion
To delete the FSPF configuration for the entire VSAN, follow these steps:
Step 1
switch# config t
Enters configuration mode.
Step 2
switch(config)# no fspf config vsan 3
Deletes the FSPF configuration for VSAN 3.
FSPF Routing Protocol Usage
To enable or disable FSPF routing protocols, follow these steps:
Link State Record Defaults
Each time a new switch enters the fabric, a link state record (LSR) is sent to the neighboring switches, and then flooded throughout the fabric. Table 21-3 displays the default settings for switch responses.
FSPF Interface Configuration
Several FSPF commands are available on a per interface basis. These configuration procedures apply to an interface in a specific VSAN.
FSPF Link Cost
FSPF tracks the state of links on all switches in the fabric, associates a cost with each link in its database, and then chooses the path with a minimal cost. The cost associated with an interface can be administratively changed to implement the FSPF route selection. The integer value to specify cost can range from 1 to 65,535. The default cost for 1 Gbps is 1000 and for 2Gbps is 500.
To configure FSPF link cost, follow these steps:
Hello Time Intervals
You can set the FSPF Hello time interval to specify the interval between the periodic hello messages sent to verify the health of the link. The integer value can range from 1 to 65,535 seconds.
Note
To configure the FSPF Hello time interval, follow these steps:
Dead Time Intervals
You can set the FSPF dead time interval to specify the maximum interval for which a hello message must be received before the neighbor is considered lost and removed from the database. The integer value can range from 1 to 65,535 seconds.
Note
Caution
To configure the FSPF dead time interval, follow these steps:
Disabling FSPF for Specific Interfaces
You can disable the FSPF protocol for selected interfaces. By default, FSPF is enabled on all E ports and TE ports. This default can be disabled by setting the interface as passive.
Note
To disable FSPF for a specific interface, follow these steps:
Retransmitting Intervals
You can specify the time after which an unacknowledged link state update should be transmitted on the interface. The integer value to specify retransmit intervals can range from 1 to 65,535 seconds.
Note
To configure the FSPF retransmit time interval, follow these steps:
Configuring Fibre Channel Routes
Each port implements forwarding logic, which forwards frames based on its FC ID. To configure the FC ID for the specified interface and domain, you can configure the specified route (for example FC ID 111211 and domain ID 3) in the switch with domain ID 1 (see Figure 21-4).
Figure 21-4 Fibre Channel Routes
Note
To configure an FC route, follow these steps:
Step 1
switch# config t
Enters configuration mode.
Step 2
switch(config)# fcroute 0x111211 interface fc1/1 domain 3 vsan 2
switch(config)#
Configures the route for the specified Fibre Channel interface and domain. In this example, interface fc1/1 is assigned an FC ID (0x111211) and a domain ID (3) to the next hop switch.
switch(config)# fcroute 0x111211 interface port-channel 1 domain 3 vsan 4
switch(config)#
Configures the route for the specified PortChannel interface and domain. In this example, interface port-channel 1 is assigned an FC ID (0x111211) and a domain ID (3) to the next hop switch.
switch(config)# fcroute 0x031211 interface fc1/1 domain 3 metric 1 vsan 1
switch(config-if)#
Configures the static route for a specific FC ID and next hop domain ID and also assigns the cost of the route.
If the remote destination option is not specified, the default is direct.
switch(config)# fcroute 0x111112 interface fc1/1 domain 3 metric 3 remote vsan 3
Adds a static route to the RIB. If this is an active route and the FIB1 records are free, it is also added to the FIB.
If the cost (metric) of the route is not specified, the default is 10.
Step 3
switch(config)# fcroute 0x610000 0xff0000 interface fc 1/1 domain 1 vsan 2
switch(config)#
Configures the netmask for the specified route in interface fc1/1 (or PortChannel). You can specify one of three routes: ff0000 matches only the domain, ffff00 matches the domain and the area, ffffff matches the domain, area, and port.
1 FIB = Forwarding Information Base
Clearing FSPF Counters
To clear the FSPF statistics counters for one interface or for the entire VSAN, follow this step:
Broadcast and Multicast Routing
Broadcast and Multicast in a Fibre Channel fabric uses the concept of a distribution tree to reach all switches in the fabric.
FSPF provides the topology information to compute the distribution tree. Fibre Channel defines 256 multicast groups and one broadcast address for each VSAN. Switches in the Cisco MDS 9000 Family only use broadcast routing. By default, they use the principal switch as the root node to derive a loop-free distribution tree for multicast and broadcast routing in a VSAN.
Caution
To interoperate with other vendor switches (following FC-SW3 guidelines), the SAN-OS software uses the lowest domain switch as the root to compute the multicast tree in interop mode.
By default, the native (non-interop) mode uses the principal switch as the root. If you change the default, be sure to configure the same mode in all switches in the fabric. Otherwise, multicast traffic could face potential loop and frame-drop problems.
Note
Use the mcast root lowest vsan command to change the multicast root from the principal switch to lowest domain switch.
To use the lowest domain switch for the multicast tree computation, follow these steps.
To display the configured and operational multicast mode and the selected root domain, use the show mcast command.
switch# show mcast vsan 1Multicast root for VSAN 1Configured root mode : Principal switchOperational root mode : Principal switchRoot Domain ID : 0xef(239)In-Order Delivery
In-Order Delivery (IOD) of data frames guarantees frame delivery to a destination in the same order that they were sent by the originator.
Some Fibre Channel protocols or applications cannot handle out-of-order frame delivery. In these cases, switches in the Cisco MDS 9000 Family preserve frame ordering in the frame flow. The source ID (SID), destination ID (DID), and optionally the originator exchange ID (OX ID) identify the flow of the frame.
In case of a single switch, all frames received by a specific ingress port and destined to a certain egress port are always delivered in the same order in which they were received.
Tip
Reordering Network Frames
When you experience a route change in the network, the new selected path may be faster or less congested than the old route (see Figure 21-5).
Figure 21-5 Route Change Delivery
In Figure 21-5, the new path from Switch 1 to Switch 4 is faster. Hence, Frame 3 and Frame 4 may be delivered before Frame 1 and Frame 2.
If the in-order guarantee feature is enabled, the frames within the network are treated as follows:
•
•
Reordering PortChannel Frames
When a link change occurs in a PortChannel, the frames for the same exchange or the same flow can switch from one path to another faster path (see Figure 21-6).
Figure 21-6 Link Congestion Delivery
In Figure 21-6, the port of the old path (red dot) is congested. Hence Frame 3 and Frame 4 can be delivered before Frame 1 and Frame 2.
When the in-order guarantee feature is enabled, the frames crossing a PortChannel are treated as follows:
•
•
•
Enabling In-Order Delivery
By default, in-order delivery is disabled on switches in the Cisco MDS 9000 Family.
You can enable the in-order delivery feature for a specific VSAN or for the entire switch.
Tip
Enabling IOD Globally
To verify that the IOD parameters are uniform across all VSANs, enable IOD globally before performing a downgrade to a Cisco MDS SAN-OS Release1.3(3) or earlier.
To enable in-order delivery for the switch, follow these steps.
Enabling IOD for a VSAN
When you create a new VSAN, that VSAN automatically inherits the global in-order-guarantee value. You can subsequently change the in-order-guarantee for the new VSAN.
To enable in-order delivery for a specific VSAN, follow these steps.
Displaying the IOD Status
Use the show in-order-guarantee command to display the present configuration status:
switch# show in-order-guaranteeglobal inorder delivery configuration:guaranteedVSAN specific settingsvsan 1 inorder delivery:guaranteedvsan 101 inorder delivery:not guaranteedvsan 1000 inorder delivery:guaranteedvsan 1001 inorder delivery:guaranteedvsan 1682 inorder delivery:guaranteedvsan 2001 inorder delivery:guaranteedvsan 2009 inorder delivery:guaranteedvsan 2456 inorder delivery:guaranteedvsan 3277 inorder delivery:guaranteedvsan 3451 inorder delivery:guaranteedvsan 3452 inorder delivery:guaranteedvsan 3453 inorder delivery:guaranteedConfiguring the Drop Latency Time
You can change the default latency time for either a network, a specified VSAN in a network, or for the entire switch.
To configure the network and the switch drop latency time, follow these steps.
Displaying Latency Information
You can view the configured latency parameters using the show fcdroplatency command (see Example 21-1).
Example 21-1 Displays Administrative Distance
switch# show fcdroplatencyswitch latency value:500 millisecondsglobal network latency value:2000 millisecondsVSAN specific network latency settingsvsan 1 network latency:5000 millisecondsvsan 2 network latency:2000 millisecondsvsan 103 network latency:2000 millisecondsvsan 460 network latency:500 millisecondsFlow Statistics Configuration
Flow statistics count the ingress traffic in the aggregated statistics table. You can collect two kinds of statistics:
•
•
If you enable flow counters, you can enable a maximum of 1K entries for aggregate flow and flow statistics. Be sure to assign an unused flow index to a module for each new flow. Flow indexes can be repeated across modules. The number space for flow index is shared between the aggregate flow statistics and the flow statistics.
Configuring Flow Statistics
To count the aggregated flow statistics for a VSAN, follow these steps:
Counting Flow Statistics
To count the flow statistics for a source and destination FC ID in a VSAN, follow these steps:
Clearing FIB Statistics
Use the clear fcflow stats command to clear the aggregated flow counter (see Examples 21-2 and 21-3).
Example 21-2 Clears Aggregated Flow Counters
switch# clear fcflow stats aggregated module 2 index 1Example 21-3 Clears Flow Counters for Source and Destination FC IDs
switch# clear fcflow stats module 2 index 1Displaying Flow Statistics
Use the show fcflow stats commands to view flow statistics (see Example 21-4 to 21-6).
Example 21-4 Displays Aggregated fcflow Details for the Specified Module
switch# show fcflow stats aggregated module 2Idx VSAN # frames # bytes---- ---- -------- -------0000 4 387,653 674,235,8750001 6 34,402 2,896,628Example 21-5 Displays fcflow Details for the Specified Module
switch# show fcflow stats module 2Idx VSAN D ID S ID mask # frames # bytes---- ---- ----------- ----------- ----- -------- -------0000 4 032.001.002 007.081.012 ff.ff.ff 387,653 674,235,8750001 6 004.002.001 019.002.004 ff.00.00 34,402 2,896,628Example 21-6 Displays fcflow Index Usage for the Specified Module
switch# show fcflow stats usage module 22 flows configuredconfigured flow : 3,7Displaying Routing and Forwarding Information
You can view specific information about existing Fibre Channel and FSPF configurations (see Examples 21-7 to 21-15).
Example 21-7 Displays Administrative Distance
switch# show fcroute distanceRouteUUID Distance Name---- -------- ----10 20 RIB22 40 FCDOMAIN39 80 RIB-CONFIG12 100 FSPF17 120 FLOGI21 140 TLPM14 180 MCAST64 200 RIB-TEST
Note
Example 21-8 Displays Multicast Routing Information
switch# show fcroute multicast VSAN FC ID # Interfaces---- -------- ------------1 0xffffff 02 0xffffff 13 0xffffff 14 0xffffff 05 0xffffff 06 0xffffff 07 0xffffff 08 0xffffff 09 0xffffff 010 0xffffff 0Example 21-9 Displays FCID Information for a Specified VSAN
switch# show fcroute multicast vsan 3VSAN FC ID # Interfaces---- -------- ------------3 0xffffff 1Example 21-10 Displays FCID and interface Information for a Specified VSAN
switch# show fcroute multicast 0xffffff vsan 2VSAN FC ID # Interfaces---- -------- ------------2 0xffffff 1fc1/1Example 21-11 Displays Unicast Routing Information
switch# show fcroute unicastD:direct R:remote P:permanent V:volatile A:active N:non-active# NextProtocol VSAN FC ID/Mask RCtl/Mask Flags Hops Cost-------- ---- -------- -------- ---- ---- ----- ------ ----static 1 0x010101 0xffffff 0x00 0x00 D P A 1 10static 2 0x111211 0xffffff 0x00 0x00 R P A 1 10fspf 3 0x610000 0xff0000 0x00 0x00 D P A 4 500static 4 0x040101 0xffffff 0x00 0x00 R P A 1 103static 4 0x040102 0xffffff 0x00 0x00 R P A 1 103static 4 0x040103 0xffffff 0x00 0x00 R P A 1 103static 4 0x040104 0xffffff 0x00 0x00 R P A 1 103static 4 0x111211 0xffffff 0x00 0x00 D P A 1 10Example 21-12 Displays Unicast Routing Information for a Specified VSAN
switch# show fcroute unicast vsan 4D:direct R:remote P:permanent V:volatile A:active N:non-active# NextProtocol VSAN FC ID/Mask RCtl/Mask Flags Hops Cost-------- ---- -------- -------- ---- ---- ----- ------ ----static 4 0x040101 0xffffff 0x00 0x00 R P A 1 103static 4 0x040102 0xffffff 0x00 0x00 R P A 1 103static 4 0x040103 0xffffff 0x00 0x00 R P A 1 103static 4 0x040104 0xffffff 0x00 0x00 R P A 1 103static 4 0x111211 0xffffff 0x00 0x00 D P A 1 10Example 21-13 Displays Unicast Routing Information for a Specified FCID
switch# show fcroute unicast 0x040101 0xffffff vsan 4D:direct R:remote P:permanent V:volatile A:active N:non-active# NextProtocol VSAN FC ID/Mask RCtl/Mask Flags Hops Cost-------- ---- -------- -------- ---- ---- ----- ------ ----static 4 0x040101 0xffffff 0x00 0x00 R P A 1 103fc1/2 Domain 0xa6(166)Example 21-14 Displays Route Database Information
switch# show fcroute summaryFC Route Database Created Thu Feb 13 07:21:52 2003VSAN Ucast Mcast Label Last Modified Time----- -------- -------- -------- ------------------------1 5 1 0 Thu Feb 13 10:21:06 20032 4 1 0 Thu Feb 13 10:21:07 20033 4 1 0 Thu Feb 13 10:21:08 20034 4 1 0 Thu Feb 13 10:21:09 20035 4 1 0 Thu Feb 13 10:21:10 20036 4 1 0 Thu Feb 13 10:21:11 20037 4 1 0 Thu Feb 13 10:21:12 20038 4 1 0 Thu Feb 13 10:21:13 20039 4 1 0 Thu Feb 13 10:21:14 200310 4 1 0 Thu Feb 13 10:21:15 200311 4 1 0 Thu Feb 13 10:21:16 200312 4 1 0 Thu Feb 13 10:21:17 200313 4 1 0 Thu Feb 13 10:21:18 200314 4 1 0 Thu Feb 13 10:21:18 200315 4 1 0 Thu Feb 13 10:21:19 2003----- -------- -------- -------- ------------------------Total 61 15 0Example 21-15 Displays Route Database Information for a Specified VSAN
switch# show fcroute summary vsan 5FC Route Database Created Thu Feb 13 07:21:52 2003VSAN Ucast Mcast Label Last Modified Time----- -------- -------- -------- ------------------------5 4 1 0 Thu Feb 13 10:21:10 2003----- -------- -------- -------- ------------------------Total 4 1 0Displaying Global FSPF Information
Example 21-16 displays global FSPF information for a specific VSAN:
•
•
•
•
Tip
•
•
Example 21-16 Displays FSPF Information for a Specified VSAN
switch# show fspf vsan 1FSPF routing for VSAN 1FSPF routing administration status is enabledFSPF routing operational status is UPIt is an intra-domain routerAutonomous region is 0SPF hold time is 0 msecMinLsArrival = 1000 msec , MinLsInterval = 5000 msecLocal Domain is 0x65(101)Number of LSRs = 3, Total Checksum = 0x0001288bProtocol constants :LS_REFRESH_TIME = 1800 secMAX_AGE = 3600 secStatistics counters :Number of LSR that reached MaxAge = 0Number of SPF computations = 7Number of Checksum Errors = 0Number of Transmitted packets : LSU 65 LSA 55 Hello 474 Retranmsitted LSU 0Number of received packets : LSU 55 LSA 60 Hello 464 Error packets 10Displaying the FSPF Database
Example 21-17 displays a summary of the FSPF database for a specified VSAN. If other parameters are not specified, all LSRs in the database are displayed:
•
•
•
•
•
•
You could narrow the display to obtain specific information by issuing additional parameters for the domain ID of the LSR owner. For each interface, the following information is also available:
•
•
•
•
Example 21-17 Displays FSPF Database Information
switch# show fspf database vsan 1FSPF Link State Database for VSAN 1 Domain 0x0c(12)LSR Type = 1Advertising domain ID = 0x0c(12)LSR Age = 1686LSR Incarnation number = 0x80000024LSR Checksum = 0x3cafNumber of links = 2NbrDomainId IfIndex NbrIfIndex Link Type Cost-----------------------------------------------------------------------------0x65(101) 0x0000100e 0x00001081 1 5000x65(101) 0x0000100f 0x00001080 1 500FSPF Link State Database for VSAN 1 Domain 0x65(101)LSR Type = 1Advertising domain ID = 0x65(101)LSR Age = 1685LSR Incarnation number = 0x80000028LSR Checksum = 0x8443Number of links = 6NbrDomainId IfIndex NbrIfIndex Link Type Cost-----------------------------------------------------------------------------0xc3(195) 0x00001085 0x00001095 1 5000xc3(195) 0x00001086 0x00001096 1 5000xc3(195) 0x00001087 0x00001097 1 5000xc3(195) 0x00001084 0x00001094 1 5000x0c(12) 0x00001081 0x0000100e 1 5000x0c(12) 0x00001080 0x0000100f 1 500FSPF Link State Database for VSAN 1 Domain 0xc3(195)LSR Type = 1Advertising domain ID = 0xc3(195)LSR Age = 1686LSR Incarnation number = 0x80000033LSR Checksum = 0x6799Number of links = 4NbrDomainId IfIndex NbrIfIndex Link Type Cost-----------------------------------------------------------------------------0x65(101) 0x00001095 0x00001085 1 5000x65(101) 0x00001096 0x00001086 1 5000x65(101) 0x00001097 0x00001087 1 5000x65(101) 0x00001094 0x00001084 1 500Displaying FSPF Interfaces
Example 21-18 displays the following information for each selected interface.
•
•
•
•
•
•
•
Example 21-18 Displays FSPF Interface Information
switch# show fspf vsan 1 interface fc1/1FSPF interface fc1/1 in VSAN 1FSPF routing administrative state is activeInterface cost is 500Timer intervals configured, Hello 20 s, Dead 80 s, Retransmit 5 sFSPF State is FULLNeighbor Domain Id is 0x0c(12), Neighbor Interface index is 0x0f100000Statistics counters :Number of packets received : LSU 8 LSA 8 Hello 118 Error packets 0Number of packets transmitted : LSU 8 LSA 8 Hello 119 Retransmitted LSU 0Number of times inactivity timer expired for the interface = 0Default Settings
Table 21-4 lists the default settings for FSPF features.
