Cisco MDS 9000 Family Troubleshooting Guide, Release 2.x
Troubleshooting VSANs, Domains, and FSPF
Downloads: This chapterpdf (PDF - 456.0KB) The complete bookPDF (PDF - 3.92MB) | Feedback

Troubleshooting VSANs, Domains, and FSPF

Table Of Contents

Troubleshooting VSANs, Domains, and FSPF

Best Practices for VSAN Implementation

Best Practices for Domain ID Assignment

Best Practices for FSPF

Initial Troubleshooting Checklist

Common Troubleshooting Tools in Fabric Manager

Common Troubleshooting Commands in the CLI

VSAN Issues

Host Cannot Communicate with Storage

Verifying VSAN Membership Using Fabric Manager

Verifying VSAN Membership Using the CLI

xE Port Is Isolated in a VSAN

Resolving an Isolated E Port Using Fabric Manager

Resolving an Isolated E Port Using the CLI

Resolving an Isolated ISL Using Fabric Manager

Resolving an Isolated ISL Using the CLI

Resolving Fabric Timer Issues Using Fabric Manager

Resolving Fabric Timer Issues Using the CLI

Troubleshooting Interop Mode Issues

Dynamic Port VSAN Membership Issues

Troubleshooting DPVM Using Fabric Manager

Troubleshooting DPVM Using the CLI

DPVM Configuration Not Available

DPVM Database Not Distributed

DPVM Autolearn Not Working

No Autolearn Entries in Active Database.

VSAN Membership not Added to Database.

DPVM Config Database Not Activating

Cannot Copy Active to Config DPVM Database

Port Suspended or Disabled after DPVM Activation

DPVM Merge Failed

Domain Issues

Domain ID Conflict Troubleshooting

Switch Cannot See Other Switches in a VSAN

FC Domain ID Overlap

Assigning a New Domain ID Using Fabric Manager

Assigning a New Domain ID Using the CLI

Using Fabric Reconfiguration for Domain ID Assignments

FSPF Issues

Troubleshooting FSPF

Troubleshooting FSPF Using Device Manager

Troubleshooting FSPF Using the CLI

Loss of Two-Way Communication

Resolving a Wrong Hello Interval on an ISL Using Device Manager

Resolving a Wrong Hello Interval on an ISL Using the CLI

Resolving a Mismatched Retransmit Interval on an ISL Using Device Manager

Resolving a Mismatched Retransmit Interval on an ISL Using the CLI

Resolving a Mismatch in Dead Intervals on an ISL Using Fabric Manager

Resolving a Mismatch in Dead Intervals on an ISL Using the CLI

Resolving a Region Mismatch Using Fabric Manager

Resolving a Region Mismatch Using the CLI


Troubleshooting VSANs, Domains, and FSPF


This chapter describes how to identify and resolve problems that might occur when implementing VSANs, domains, and FSPF. This chapter includes the following sections:

Best Practices for VSAN Implementation

Best Practices for Domain ID Assignment

Best Practices for FSPF

Initial Troubleshooting Checklist

VSAN Issues

Dynamic Port VSAN Membership Issues

Domain Issues

FSPF Issues

Best Practices for VSAN Implementation

Virtual SANs (VSANs) provide a method of isolating devices that are physically connected to the same storage network, but are logically considered to be part of different SAN fabrics that do not need to be aware of one another. VSANs provide a way to:

Isolate devices physically connected to the same fabric.

Reduce the size of a Fibre Channel distributed database.

Enable more scalable and secure fabrics.

This section provides the best practices for implementing VSANs.

Avoid using VSAN 1 (the default VSAN) for production network traffic. Create at least one VSAN to carry your network traffic.

Isolate devices in VSANs whenever practical.

Leave fabric timers and FSPF timers at their default settings.

Avoid modifying fabric timers and FSPF timers unless changes are required because of interoperability with an existing fabric, or long-haul links are being deployed.

Use Inter-VSAN routing (IVR) only when necessary to selectively connect devices across VSANs.

If IVR is used without NAT, ensure that domain IDs are are statically configured and unique across all VSANs.

Place FCIP gateways in their own native VSAN.

Placing FCIP gateways in their own VSAN isolates disturbances when problems in the IP cloud (such as flapping links) occur.

Use VSAN-based roles to control and limit management access to your switches.

We recommend using only following characters in a VSAN name:

- a-z or A-Z

- 0 - 9

- - (hyphen) or _ (underscore)

Best Practices for Domain ID Assignment

This section provides best practices for domain ID assignments.

Use static domains in most environments. To use static domains, choose Fabricxx > All VSANs > Domain Manager and select static from the Config Type drop-down menu in Fabric Manager or use the fcdomain domain n static vsan x CLI command. You must then issue a disruptive restart so that the configured domain ID matches the running domain ID. Select the Configuration tab and select disruptive from the Restart drop-down menu in Fabric Manager and click Apply Changes. In the CLI, use the fcdomain restart disruptive CLI command.


Note You cannot issue a disruptive restart for VSANs that are in any of the interop modes. Use a nondisruptive restart as needed.


To disable the Domain manager, choose Fabricxx > All VSANs > Domain Manager and uncheck the Enable check box in Fabric Manager or use the no fcdomain vsan x CLI command.

Disable the Domain Manager to disable the principal switch selection process. This is possible if all domains are statically assigned. Disabling principal switch selection can reduce disruption when switches are rebooted or added to the fabric. This must be done on each switch that should not participate in principal switch selection. A disruptive restart of the fabric is required to apply this change.

Keep domain ID allowed lists the same on all switches in a fabric for consistency. If the principal switch changes, the allowed domain lists will remain the same.

Assign domain IDs between decimal 97 and 127 if the domain may be used for standards-based interop mode.

Do not perform frequent changes to the Domain Manager on production fabrics. Experienced administrators familiar with switch operations should be responsible for Domain Manager changes. Plan your domain configuration carefully so that you avoid the need to make disruptive changes at a later time.

Save Domain Manager changes. When you change the configuration, be sure to save the running configuration by choosing Switches > Copy Configuration in Fabric Manager or using the copy running-config startup-config CLI command. The next time you reboot the switch, the saved configuration is used. If you do not save the configuration, the previously saved startup configuration is used.

Enable reconfigure fabric (RCF) rejection on every ISL port if high availability is mandatory. Choose Switches > Interfaces > FC Physical in Fabric Manager and select the Domain Manager tab in the Information pane and then check the RcfReject check box on all ISL ports to enable rcf-rejects. Or use the interface CLI command on a TE or E port and then use the fcdomain rcf-reject vsan CLI command in interface configuration mode to enable the rcf-reject option. RCF reject prevents other switches from sending an RCF and potentially causing a disruption in your production traffic.

Best Practices for FSPF

This section provides best practices for implementing FSPF.

Use the default FSPF link cost, which can be configured on a per-VSAN basis for the same physical link, provides preferred and alternate paths. If you must alter the FSPF link cost, use caution to avoid the potential for asymmetric Fibre Channel routing.

Use the default FSPF load-balancing configuration unless you are required to load balance based on your unique fabric, for example, if you have FICON VSANs.

Use the default FSPF timer configuration. If FSPF timers are misconfigured, then the switches will not reach the "two-way" state and FSPF will not operate properly.

Initial Troubleshooting Checklist

Most VSAN problems can be avoided by following the best practices for VSAN implementation. In addition to Fabric Manager and the CLI, another tool that may be used to verify different categories of problems (VSANs, zoning, FCdomain, admin issues, or other switch-specific or fabric-specific issues) is the Fabric Analysis tool provided by Fabric Manager.

The configuration consistency check tool is also provided by Fabric Manager. Refer to the Cisco MDS 9000 Fabric Manager Configuration Guide for more information about this tool.

Troubleshooting a SAN problem involves gathering information about the configuration and connectivity of individual devices and the entire SAN fabric. In the case of VSANs, begin your troubleshooting activity as follows:

Checklist 
Checkoff

Verify the FSPF parameters for switches in the VSAN.

Verify the domain parameters for switches in the VSAN.

Verify the physical connectivity for any problem ports or VSANs.

Verify that you have both devices in the name server.

Verify that you have both end devices in the same VSAN.

Verify that you have both end devices in the same zone.

Verify that the zone is part of the active zone set.


Common Troubleshooting Tools in Fabric Manager

The following Fabric Manager procedures are used to verify the VSAN, domain, FSPF, and zone configuration:

Choose Fabricxx > VSANxx to view the VSAN configuration in the Information pane.

Choose Fabricxx > VSANxx and select the Host or Storage tab in the Information pane to view the VSAN members.

Choose Fabricxx > VSANxx > Domain Manager to view the FCdomain configuration in the Information pane.

Choose Fabricxx > VSANxx > FSPF to view the FSPF configuration in the Information pane.

Choose Fabricxx > VSANxx > zonesetname to view the zone configuration for this VSAN. Zone configuration problems may appear to be a VSAN problem.

Common Troubleshooting Commands in the CLI

The following CLI commands are used to display VSAN, FCdomain, and FSPF information:

show vsan

show vsan vsan-id

show vsan membership

show interface fc slot/port trunk vsan-id

show vsan-id membership

show vsan membership interface fc slot/port

show fcdomain

show fspf

show fspf internal route vsan vsan-id

show fcns database vsan vsan-id

The following zone CLI commands may be useful to validate your configuration:

show zoneset name zonesetName vsan-id

show zoneset active vsan-id


Note An asterix (*) near the device listed by the show zoneset active CLI command indicates that the device is logged into the name server.


show zone vsan-id

show zone status show vsan-id


Note For more information on zoning issues, see "Troubleshooting Zones and Zone Sets."


VSAN Issues

This section covers the following VSAN issues:

Host Cannot Communicate with Storage

xE Port Is Isolated in a VSAN

Troubleshooting Interop Mode Issues

Host Cannot Communicate with Storage

Communication problems between a host and storage devices can be caused by port, VSAN, or zone issues.

Symptom    Host cannot communicate with storage.

Table 7-1 Host Cannot Communicate with Storage

Symptom
Possible Cause
Solution

Host cannot communicate with storage.

Host and storage are not in the same VSAN.

Verify the VSAN membership. See the "Verifying VSAN Membership Using Fabric Manager" section or the "Verifying VSAN Membership Using the CLI" section.

xE port connecting to the remote switch is isolated.

See the "xE Port Is Isolated in a VSAN" section.

Host and storage are not in the same zone.

See the "Zone and Zone Set Issues" section.


Verifying VSAN Membership Using Fabric Manager

To verify VSAN membership for host and storage devices using Fabric Manager, follow these steps:


Step 1 Choose Fabricxx > VSANxx and select the Host or Storage tab in the Information pane. Verify that both devices are in the same VSAN.

Step 2 If the host and storage are in different VSANs, verify which port is not in the correct VSAN and then follow these steps to change the port VSAN:

a. Highlight the host or storage in the Information pane. You see the link to that end device highlighted in blue in the map pane.

b. Right-click on the highlighted link and select Interface Attributes from the pop-up menu.

c. Set the PortVSAN field to the VSAN that holds the other end device and click Apply Changes.

Step 3 Right-click any ISL between the switches and select Interface Attributes. Select the Trunk Config tab and verify that the allowed VSAN list includes the VSAN found in Step 1.

Step 4 If the trunk is not configured for the VSAN, set the Allowed VSANs field to include the VSAN that the host and storage devices are on and click Apply Changes.


Verifying VSAN Membership Using the CLI

To verify VSAN membership for host and storage devices using the CLI, follow these steps:


Step 1 Use the show vsan membership command to see all the ports connected to your host and storage, and verify that both devices are in the same VSAN. Use this command on the switches that connect to your host or storage devices.

switch# show vsan membership 
vsan 1 interfaces:
        fc2/7   fc2/8   fc2/9   fc2/10  fc2/11  fc2/12  fc2/13  fc2/14
        fc2/15  fc2/16  fc7/1   fc7/2   fc7/3   fc7/4   fc7/5   fc7/6
        fc7/7   fc7/8   fc7/9   fc7/10  fc7/11  fc7/12  fc7/13  fc7/14
        fc7/15  fc7/16  fc7/17  fc7/18  fc7/19  fc7/20  fc7/21  fc7/22
        fc7/25  fc7/26  fc7/27  fc7/28  fc7/29  fc7/30  fc7/31  fc7/32

vsan 2 interfaces:
        fc2/6   fc7/23  fc7/24

vsan 3 interfaces:
        fc2/1   fc2/2   fc2/5

vsan 4 interfaces:
        fc2/3   fc2/4

Step 2 If the host and storage are in different VSANs, use the vsan database vsan vsan-id interface CLI command to move the interface connected to the host and storage devices into the same VSAN.

Step 3 Use the show interface command to verify that the trunks connecting the end switches are configured to transport the VSAN found in Step 1.

 switch# show interface fc2/14
 fc2/14 is trunking
    Hardware is Fibre Channel, WWN is 20:4e:00:05:30:00:63:9e
    Port mode is TE
    Speed is 2 Gbps
    vsan is 2
    Beacon is turned off
    Trunk vsans (allowed active) (1-3,5)
    Trunk vsans (operational)    (1-3,5)
    Trunk vsans (up)             (2-3,5)
    Trunk vsans (isolated)       (1)
    Trunk vsans (initializing)   ()
      475 frames input, 8982 bytes, 0 discards
      0 runts, 0 jabber, 0 too long, 0 too short
      0 input errors, 0 CRC, 3 invalid transmission words
      0 address id, 0 delimiter
      0 EOF abort, 0 fragmented, 0 unknown class
      514 frames output, 7509 bytes, 16777216 discards
      Received 30 OLS, 21 LRR, 18 NOS, 53 loop inits
      Transmitted 68 OLS, 25 LRR, 28 NOS, 32 loop inits

Step 4 If the trunk is not configured for the VSAN, use the interface CLI command and then the switchport trunk allowed vsan CLI command in interface mode to add the VSAN to the allowed VSAN list for the interface that connects the host and storage devices.


xE Port Is Isolated in a VSAN

Symptom    xE port is isolated in a VSAN.

Table 7-2 xE Port is Isolated in a VSAN 

Symptom
Possible Cause
Solution

xE port is isolated in a VSAN.

E port connecting to the remote switch is isolated.

Verify the VSAN. See the "Resolving an Isolated E Port Using Fabric Manager" section or the "Resolving an Isolated E Port Using Fabric Manager" section.

TE port connecting to the remote switch is isolated.

See the "Resolving an Isolated ISL Using Fabric Manager" section or the "Resolving an Isolated ISL Using the CLI" section

Fabric timers misconfigured.

Use caution when changing fabric timers. See the "Resolving Fabric Timer Issues Using Fabric Manager" section or the "Resolving Fabric Timer Issues Using the CLI" section.

Port parameters misconfigured.

See the "Common Problems with Port Interfaces" section.

Zoning mismatch.

See "Troubleshooting Zones and Zone Sets."


Resolving an Isolated E Port Using Fabric Manager

To resolve VSAN isolation on an E port using Fabric Manager, follow these steps:


Step 1 Choose Switches > Interfaces > FC Physical and check the FailureCause column on the E port to verify that you have a VSAN mismatch problem.

Step 2 Choose Switches > Interfaces > FC Physical and set the PortVSAN field to correct a VSAN mismatch.


Resolving an Isolated E Port Using the CLI

To resolve VSAN isolation on an E port using the CLI, follow these steps:


Step 1 Use the show interface command to verify that the port is isolated because of a VSAN mismatch.

switch# show interface fc2/4
fc2/4 is down fc2/4 is down (isolation due to port vsan mismatch) 

    Hardware is Fibre Channel, WWN is 20:44:00:05:30:00:63:5e
    vsan is 4
    Beacon is turned off
      30 frames input, 682 bytes, 0 discards
      0 runts, 0 jabber, 0 too long, 0 too short
      0 input errors, 0 CRC, 0 invalid transmission words
      0 address id, 0 delimiter
      0 EOF abort, 0 fragmented, 0 unknown class
      30 frames output, 583 bytes, 0 discards
      Received 2 OLS, 2 LRR, 2 NOS, 5 loop inits
      Transmitted 5 OLS, 3 LRR, 2 NOS, 4 loop inits

Step 2 Use the show vsan membership CLI command to verify that the ports are in separate VSANs.

switch# show vsan membership
vsan 3 interfaces:
        fc2/1   fc2/2   fc2/3   fc2/4   fc2/6   fc2/7   fc2/8   fc2/9
        fc2/10  fc2/11  fc2/12  fc2/14  fc2/15  fc2/16  fc7/1   fc7/2
        fc7/3   fc7/4   fc7/5   fc7/6   fc7/7   fc7/8   fc7/9   fc7/10
        fc7/11  fc7/12  fc7/13  fc7/14  fc7/15  fc7/16  fc7/17  fc7/18
        fc7/19  fc7/20  fc7/21  fc7/22  fc7/23  fc7/24  fc7/25  fc7/26
        fc7/27  fc7/28  fc7/29  fc7/30  fc7/31  fc7/32

vsan 4 interfaces:
        fc2/5   fc2/13

vsan 4094(isolated_vsan) interfaces:

This sample output shows that all the interfaces on the switch belong to VSAN 3, with the exception of interface fc2/5 and fc2/13, which are part of VSAN 4.

Step 3 Use the vsan database vsan vsan-id interface CLI command to move the ports into the same VSAN.


Resolving an Isolated ISL Using Fabric Manager

Trunking E ports (TE ports) are similar to E ports except that they carry traffic for multiple VSANs. E ports carry traffic for a single VSAN. Because TE ports carry traffic for multiple VSANs, ISL isolation can affect one or more VSANs. For this reason, on a TE port you must troubleshoot for ISL isolation on each VSAN.

To resolve VSAN isolation on a TE port using Fabric Manager, follow these steps:


Step 1 Choose Switches > Interfaces > FC Physical and check the FailureCause column on the TE port to verify that you have trunk problems.

Step 2 Choose Switches > Interfaces > FC Physical and select the Trunk Failures tab to determine the reason for the trunk problem.

Step 3 Correct the problem listed in the FailureCause column. See the "DPVM Config Database Not Activating" section for domain misconfiguration problems. Choose Switches > Interfaces > FC Physical and set the PortVSAN field to to correct the VSAN misconfiguration problems.

Step 4 Repeat this procedure for all isolated VSANs on this TE port.


Resolving an Isolated ISL Using the CLI

Trunking E ports (TE ports) are similar to E ports except that they carry traffic for multiple VSANs. E ports carry traffic for a single VSAN. Because TE ports carry traffic for multiple VSANs, ISL isolation can affect one or more VSANs. For this reason, on a TE port you must troubleshoot for ISL isolation on each VSAN.

To resolve VSAN isolation on a TE port using the CLI, follow these steps:


Step 1 Use the show interface command on the TE port to verify that you have an isolated VSAN.

switch# show interface fc2/14
fc2/14 is trunking
    Hardware is Fibre Channel, WWN is 20:4e:00:05:30:00:63:9e
    Port mode is TE
    Speed is 2 Gbps
    vsan is 2
    Beacon is turned off
    Trunk vsans (allowed active) (1-3,5)
    Trunk vsans (operational)    (1-3,5)
    Trunk vsans (up)             (2-3,5)
    Trunk vsans (isolated)       (1)
    Trunk vsans (initializing)   ()
      475 frames input, 8982 bytes, 0 discards
      0 runts, 0 jabber, 0 too long, 0 too short
      0 input errors, 0 CRC, 3 invalid transmission words
      0 address id, 0 delimiter
      0 EOF abort, 0 fragmented, 0 unknown class
      514 frames output, 7509 bytes, 16777216 discards
      Received 30 OLS, 21 LRR, 18 NOS, 53 loop inits

The example shows the output of the show interface command with one or more isolated VSANs. Here, the TE port has one VSAN isolated.

Step 2 Use the show interface fc slot/port trunk vsan vsan-id command to verify the reason for VSAN isolation.

switch# show interface fc2/14 trunk vsan 1
fc2/15 is trunking
    Vsan 1 is down (Isolation due to zone merge failure)

This output shows that VSAN 1 is isolated because of a zone merge error.

Step 3 Use the show port internal info interface fc slot/port command to determine the root cause of the VSAN isolation.


Note To issue commands with the internal keyword, you must have an account that is a member of the network-admin group.


switch# show port internal info interface fc2/14

fc2/14 - if_index: 0x0109C000, phy_port_index: 0x3c
  Admin Config - state(up), mode(TE), speed(auto), trunk(on)
    beacon(off), snmp trap(on), tem(false)
    rx bb_credit(default), rx bb_credit multiplier(default)
    rxbufsize(2112), encap(default), user_cfg_flag(0x3)
    description()
    Hw Capabilities: 0xb
    trunk vsans (up) (7)
    .
    .
    .
    trunk vsans (isolated) (1,8)
  TE port per vsan information
  fc2/29, Vsan 1 - state(down), state reason(Isolation due to domain other side eport 
isolated), fcid(0x000000)
    port init flag(0x10000), current state [TE_FSM_ST_ISOLATED_DM_ZS]
  fc2/29, Vsan 7 - state(up), state reason(None), fcid(0x690202)
    port init flag(0x38000), current state [TE_FSM_ST_E_PORT_UP]
  fc2/29, Vsan 8 - state(down), state reason(Isolation due to vsan not configured on 
peer), fcid(0x000000)
    port init flag(0x0), current state [TE_FSM_ST_ISOLATED_VSAN_MISMATCH]

The last few lines of the command output provide a description of the reason for VSAN isolation for every isolated VSAN.

In this example, VSAN 7 is up, while two VSANs are isolated. VSAN 1 is isolated because of domain ID misconfiguration, and VSAN 8 is isolated because of VSAN misconfiguration.

Step 4 Correct the root cause. See the "DPVM Config Database Not Activating" section for domain misconfiguration problems. Use the the vsan vsan-id interface CLI command to correct the VSAN misconfiguration problems.

Step 5 Repeat this procedure for all isolated VSANs on this TE port.


Resolving Fabric Timer Issues Using Fabric Manager

Use caution when changing fabric timers.

To resolve FC timer issues between VSANs using Fabric Manager, follow these steps:


Step 1 Choose Fabricxx > VSANxx > VSAN Attributes to verify that the fabric timers are inconsistent across the VSANs.

Step 2 Choose Switches > FC Services > Timers and Policies. You see the fabric timers in the Information pane.

Step 3 Click Change Timeout Values and set the timers and click Apply.


Resolving Fabric Timer Issues Using the CLI

Use caution when changing fabric timers.

To resolve fabric timer issues between VSANs using the CLI, follow these steps:


Step 1 Use the show fctimer CLI command to verify that the fabric timers are inconsistent across the VSANs.

Step 2 Use the fctimer distribute CLI command to enable CFS distribution for the fabric timers. Repeat this on all switches in this VSAN.

Step 3 Use the fctimer CLI command to set each timer.

Step 4 Use the fctimer commit command to save these changes and distribute them to all switches in the VSAN.


Troubleshooting Interop Mode Issues

To troubleshoot interop modes, refer to the switch to switch interop guide at the following website:

http://www.cisco.com/univercd/cc/td/doc/product/sn5000/mds9000/mdsint/intgd.pdf

Dynamic Port VSAN Membership Issues

Dynamically assigning VSAN membership to ports is achieved by assigning VSANs based on the device WWN. This method is referred to as the Dynamic Port VSAN Membership (DPVM) feature. DPVM offers flexibility and eliminates the need to reconfigure the VSAN to maintain fabric topology when a host or storage device connection is moved between two switches or between ports on the same switch. It retains the configured VSAN regardless of where a device is connected or moved.

Verify the following requirements when using DPVM:

The interface through which the dynamic device connects to the Cisco MDS switch must be configured as an F port. FL ports do not support DPVM and no entries will be learned through an FL port.

The static port VSAN of the F port should be valid (not isolated, not suspended, and in existence).

The dynamic VSAN configured for the device in the DPVM database should be valid (not isolated, not suspended, and in existence).


Note The DPVM feature overrides any existing static port VSAN membership configuration. If the VSAN corresponding to the dynamic port is deleted or suspended, the port is shut down.



Note If you copy the DPVM database and fabric distribution is enabled, you must commit the changes.


To begin configuring the DPVM feature, you must explicitly enable DPVM on the required switches in the fabric. By default, this feature is disabled in all switches in the Cisco MDS 9000 Family.

For more information on enabling DPVM, see one of the following guides:

Cisco MDS 9000 Family Fabric Manager Configuration Guide

Cisco MDS 9000 Family Configuration Guide

This section contains the following topics:

Troubleshooting DPVM Using Fabric Manager

Troubleshooting DPVM Using the CLI

DPVM Configuration Not Available

DPVM Database Not Distributed

DPVM Autolearn Not Working

No Autolearn Entries in Active Database.

VSAN Membership not Added to Database.

DPVM Config Database Not Activating

Cannot Copy Active to Config DPVM Database

Port Suspended or Disabled after DPVM Activation

DPVM Merge Failed

Troubleshooting DPVM Using Fabric Manager

To troubleshoot DPVM using Fabric Manager, follow these steps:


Step 1 Choose Fabricxx > All VSANs > DPVM and select the CFS tab.

Step 2 Verify that the Oper and Global columns are enabled. If not, set the Admin drop-down menu to enable and the Global drop-down menu to enable. Then click Apply Changes.

Step 3 Select the Actions tab. Uncheck AutoLearn Enable if it is checked and click Apply Changes.

Step 4 Select the Active Database tab.

Step 5 Select Pending from the Compare To drop-down menu. You see a dialog box listing any differences between the active DPVM database and the pending database.

Step 6 Select the CFS tab and set Config Action to commit if there are any pending changes that you want to save. Click Apply Changes.

Step 7 Select the Actions tab and select activate from the Actions drop-down menu to activate the database. Click Apply Changes.


Troubleshooting DPVM Using the CLI

To troubleshoot DPVM using the CLI, follow these steps:


Step 1 Use the show dpvm CLI command in EXEC mode to verify that CFS distribution is enabled for DPVM.
Optionally, use the dpvm distribute CLI command in config mode to enable CFS distribution if required.

Step 2 Use the show dpvm status CLI command in EXEC mode to verify that autolearning is disabled.
Optionally, use the no dpvm auto-learn command in config mode if you need to disable autolearning before activating the database.

Step 3 Use the show dpvm pending-diff CLI in EXEC mode command to compare the active and pending databases.
Optionally use the dpvm commit CLI command in config mode to commit any pending entries to the config database.

Step 4 Use the dpvm activate CLI in config mode command to activate the database.


DPVM Configuration Not Available

Symptom    DPVM configuration is not available on Fabric Manager or CLI.

Table 7-3 DPVM Configuration not Available

Symptom
Possible Cause
Solution

DPVM configuration is not available on Fabric Manager or CLI.

DPVM has not been enabled.

DPVM must be enabled before it can be configured. Choose Fabricxx > All VSANs > DPVM and check the Status field in Fabric Manager or use the show dpvm status CLI command to verify that DPVM is not enabled. Set the Status field to enable in Fabric Manager and then click Apply Changes or use the dpvm enable CLI command to enable DPVM.


DPVM Database Not Distributed

Symptom    DPVM databases are not distributed.

Table 7-4 DPVM Database not Distributed

Symptom
Possible Cause
Solution

DPVM databases are not distributed.

DPVM distribution has not been enabled on the local switch.

Choose Fabricxx > All VSANs > DPVM and select the CFS tab. Check the Global field in Fabric Manager or use the show dpvm status CLI command to verify that DPVM distribution is not enabled. Set the Global field to enable in Fabric Manager and then click Apply Changes or use the dpvm distribute CLI command to enable DPVM.

DPVM distribution has not been enabled on one or more remote switches.


DPVM Autolearn Not Working

The DPVM autolearn feature allows you to automatically populate the DPVM configuration database with all devices currently in the fabric. This feature is best used when you first turn on DPVM on a stable fabric. Once the devices are learned, you disable autolearning to populate the configuration database with these autolearned entries.

When you add a new device, it is best practices to manually add that device to the DPVM configuration database. If you turn on autolearning for a new device, you may add other devices that you did not intend to add.

Symptom    DPVM autolearn does not work or is not getting enabled.

Table 7-5 DPVM Autolearn not Working

Symptom
Possible Cause
Solution

DPVM autolearn does not work or is not getting enabled.

DPVM active database may be absent.

Choose Fabricxx > All VSANs > DPVM and select the Active Database tab in Fabric Manager or use the show dpvm database CLI command to verify that DPVM is not enabled. Select the Actions tab and set the Action field to activate in Fabric Manager and then click Apply Changes or use the dpvm activate and dpvm commit CLI commands to create the DPVM active database.



Note When DPVM distribution is enabled, you must do an explicit commit for DPVM activate and autolearn to take effect.


No Autolearn Entries in Active Database.

Symptom    There are no autolearn entries in the active database.

Table 7-6 No Autolearn Entries in Active Database.

Symptom
Possible Cause
Solution

There are no autolearn entries in the active database.

Autolearn is not enabled.

Choose Fabricxx > All VSANs > DPVM and select the Actions tab in Fabric Manager or use the show dpvm status CLI command to determine if autolearn is enabled. Check the Auto Learn Enable check box in Fabric Manager and click Apply Changes or use the dpvm auto-learn enable and dpvm commit CLI commands to enable autolearning.

Port type is not supported.

Verify that the device you want to autolearn is connected to an F port. DPVM does not support FL, TE, FCIP, or PortChannels.


VSAN Membership not Added to Database.

Symptom    The VSAN membership of the port is not added to the database.

Table 7-7 VSAN Membership not Added to Database.

Symptom
Possible Cause
Solution

The VSAN membership of the port is not added to the database.

Entry may be present in the config database.

Choose Fabricxx > All VSANs > DPVM and select the Config Database tab in Fabric Manager or use the show dpvm database CLI command to determine if the entry is present in the config database.

DPVM distribution is enabled but a database change was not committed.

Choose Fabricxx > All VSANs > DPVM and select the CFS tab in Fabric Manager. Set the Config Action drop-down menu to commit.

Or

Use the show dpvm pending CLI command to determine if there are uncommitted changes. Use the dpvm database and dpvm commit CLI commands to commit any pending changes.


DPVM Config Database Not Activating

Symptom    DPVM config database is not getting activated.

Table 7-8 DPVM Config Database not Activating

Symptom
Possible Cause
Solution

DPVM config database is not getting activated.

Conflicting entries may be present between the DPVM config and active databases.

Use the dpvm database diff active conf CLI command to determine if there are conflicting entries between the active and config databases.

Choose Fabricxx > All VSANs > DPVM and select the Actions tab in Fabric Manager. Set the Actions drop-down menu to forceActivate and Click Apply Changes or use the dpvm activate force and dpvm commit CLI commands to override the active database with the config database.


Cannot Copy Active to Config DPVM Database

Symptom    Cannot copy the active DPVM database to the config database.

Table 7-9 DPVM Merge Failed

Symptom
Possible Cause
Solution

Cannot copy the active DPVM database to the config database.

Active database may be absent.

Choose Fabricxx > All VSANs > DPVM and select the Active Database tab in Fabric Manager or use the show dpvm database CLI command to verify that DPVM is not enabled. Select the Actions tab and set the Action field to activate in Fabric Manager and then click Apply Changes or use the dpvm activate and dpvm commit CLI commands to create the DPVM active database. Then copy the active database again.


Port Suspended or Disabled after DPVM Activation

Symptom    A port in a static VSAN that was operational goes into suspend or disabled state after DPVM database activation.

Table 7-10 DPVM Merge Failed

Symptom
Possible Cause
Solution

A port in a static VSAN that was operational goes into suspend or disabled state after DPVM database activation.

DPVM database maps a connected device to a nonexistent VSAN.

Choose Switches > Interfaces > FC Physical in Fabric Manager or use the show interface CLI command to check the interface status for a dynamic VSAN related failure. Create the VSAN or map the device to another VSAN.


DPVM Merge Failed

Symptom    DPVM merge failed.

Table 7-11 DPVM Merge Failed

Symptom
Possible Cause
Solution

DPVM merge failed.

DPVM operational parameters in the two merging fabrics are different.

Choose Fabricxx > All VSANs > DPVM and check the or use the show dpvm CLI command to verify the DPVM configuration in both fabrics. Manually reconcile any differences before attempting to merge the fabrics. Use the show cfs merge status name dpvm CLI command to show the merge status.


Domain Issues

This section includes the following topics:

Domain ID Conflict Troubleshooting

Switch Cannot See Other Switches in a VSAN

FC Domain ID Overlap

Domain ID Conflict Troubleshooting

In a Fibre Channel network, the principal switch assigns domain IDs when a new switch is added to an existing fabric. However, when two fabrics merge, the principal switch selection process determines which one of the preexisting switches becomes the principal switch for the merged fabric.

The election of the new principal switch is characterized by the following rules:

A switch with a populated domain ID list has priority over a switch that has an empty domain ID list, and the principal switch will be the principal switch of the first fabric.

If both fabrics have a domain ID list, the priority between the two principal switches is determined by the configured switch priority. This is a user-settable parameter. The lower the value is, the higher the priority.

If the principal switch cannot be determined by the two previous criteria, the principal switch is then determined by the WWNs of the two switches. The lower value WWN has the higher priority.

When merging two fabrics, the administrator can expect the following behavior:

In Cisco SAN-OS Release 2.1(1a) and later releases, when connecting a single-switch fabric to a multi-switch fabric, a BF occurs and the switch with the better priority becomes the principal switch. In earlier releases, when connecting a single-switch fabric to a multi-switch fabric, the multi-switch fabric always retains its principal switch regardless of the principal switch priority setting on the single switch fabric.

In Cisco SAN-OS Release 2.1(1a) and later releases, when powering up a new switch in a multi-switch fabric, a BF occurs and the switch with the better priority becomes the principal switch. In earlier releases, when powering up a new switch in a multi-switch fabric, the multi-switch fabric always retains its principal switch regardless of the principal switch priority setting on the single switch fabric.

When powering up a new switch that is connected to a standalone switch, the new principal switch is determined by the administratively assigned priority if both switches are running Cisco SAN-OS Release 2.0(x) or earlier. If no priority is assigned (where the default priority is used in every switch), the principal switch is determined by the WWN. This also applies to connecting to two single-switch fabrics.

When connecting a multi-switch fabric to another multi-switch fabric, the principal switch is determined by the administratively assigned priority. If no priority is assigned (where the default value is used by every switch), the principal switch is determined by the WWN of the existing principal switches of the two fabrics.

Two switch fabrics might not merge. If two fabrics with two or more switches are connected, and they have at least one assigned domain ID in common, and the auto-reconfigure option is disabled (this option is disabled by default), then the E ports that are used to connect the two fabrics will be isolated due to domain ID overlap.

Switch Cannot See Other Switches in a VSAN

Symptom    Switch cannot see other switches in a VSAN.

Table 7-12 Switch Cannot See Other Switches in a VSAN

Symptom
Possible Cause
Solution

Switch cannot see other switches in a VSAN.

Switch is isolated because of a domain ID overlap.

To resolve the problem, you can either change the overlapping static domain ID by manually configuring a new static domain ID for the isolated switch, or disable the static domain assignment and allow the switch to request a new domain ID after a fabric reconfiguration.

See the "FC Domain ID Overlap" section.

Fabric timers are misconfigured.

See the "Resolving Fabric Timer Issues Using Fabric Manager" section or the "Resolving Fabric Timer Issues Using the CLI" section.


FC Domain ID Overlap

To resolve an FC domain ID overlap, you can either change the overlapping static domain ID by manually configuring a new static domain ID for the isolated switch, or disable the static domain assignment and allow the switch to request a new domain ID after a fabric reconfiguration.

To assign a static domain ID, see the "Assigning a New Domain ID Using Fabric Manager" section or the "Assigning a New Domain ID Using the CLI" section.

To assign a dynamic domain ID after a fabric reconfiguration, see the "Using Fabric Reconfiguration for Domain ID Assignments" section.

You may see the following system message in the message log when a domain ID overlap occurs:

Error Message    PORT-5-IF_DOWN_DOMAIN_OVERLAP_ISOLATION: Interface [chars] is down 
(Isolation due to domain overlap). 

Explanation    The interface is isolated because of a domain overlap.

Recommended Action    Use the show fcdomain domain-list to determine which domain IDs are overlapping. Us the fcdomain Use the fcdomain domain domain-id [static | preferred] vsan vsan-id CLI command or similar Fabric Manager procedure to change the domain ID for one of the overlapping domain IDs.

Assigning a New Domain ID Using Fabric Manager

All devices attached to the switch in the VSAN get a new FC ID when a new domain ID is assigned. Some hosts or storage devices may not function as expected if the FC ID of the host or storage device changes.

To verify FC domain ID overlap and reassign a new Domain ID using Fabric Manager, follow these steps:


Step 1 Choose Switches > Interfaces > FC Physical and check the FailureCause column for an isolation or domain overlap status.

Step 2 Choose Fabricxx > VSANxx > Domain Manger to view which domains are currently in the VSAN.

Step 3 Repeat Step 2 on the other switch to determine which domain IDs overlap.

Step 4 Select the Configuration tab and set Config Domain and Config Type to change the domain ID for one of the overlapping domain IDs.

The static option tells the switch to request that particular domain ID. If it does not get that particular address, it will isolate itself from the fabric.

The preferred option has the switch request a specified domain ID. If that ID is unavailable, it will accept another ID.

Step 5 Set the Restart drop-down menu to disruptive and click Apply Changes to restart the Domain Manager.


Note While the static option can be applied to runtime after a disruptive or nondisruptive restart, the preferred option is applied to runtime only after a disruptive restart.



Assigning a New Domain ID Using the CLI

All devices attached to the switch in the VSAN get a new FC ID when a new domain ID is assigned. Some hosts or storage devices may not function as expected if the FC ID of the host or storage device changes.

To verify FC domain ID overlap and reassign a new Domain ID using the CLI, follow these steps:


Step 1 Issue the show interface command. The following example output shows the isolation error message.

switch# show interface fc2/14
fc2/14 is down (Isolation due to domain overlap)
    Hardware is Fibre Channel, WWN is 20:4e:00:05:30:00:63:9e
    vsan is 2
    Beacon is turned off
      192 frames input, 3986 bytes, 0 discards
      0 runts, 0 jabber, 0 too long, 0 too short
      0 input errors, 0 CRC, 3 invalid transmission words
      0 address id, 0 delimiter
      0 EOF abort, 0 fragmented, 0 unknown class
      231 frames output, 3709 bytes, 16777216 discards
      Received 28 OLS, 19 LRR, 16 NOS, 48 loop inits
      Transmitted 62 OLS, 22 LRR, 25 NOS, 30 loop inits

Step 2 Use the show fcdomain domain-list vsan vsan-id command to view which domains are currently in your fabric.

switch1# show fcdomain domain-list vsan 2

Number of domains: 2
Domain ID              WWN
---------    -----------------------
 0x4a(74)    20:01:00:05:30:00:13:9f [Local] 
 0x4b(75)    20:01:00:05:30:00:13:9e [Principal]
---------    -----------------------

Step 3 Repeat Step 2 on the other switch to determine which domain IDs overlap.

switch2# show fcdomain domain-list vsan 2

Number of domains: 1
Domain ID              WWN
---------    -----------------------
0x4b(75)    20:01:00:05:30:00:13:9e [Local][Principal]
---------    -----------------------

In this example, switch 2 is isolated because of a domain ID 75 overlap.

Step 4 Use the fcdomain domain domain-id [static | preferred] vsan vsan-id CLI command to change the domain ID for one of the overlapping domain IDs.

The static option tells the switch to request that particular domain ID. If it does not get that particular address, it will isolate itself from the fabric.

The preferred option has the switch request a specified domain ID. If that ID is unavailable, it will accept another ID.

Step 5 Use the fcdomain restart disruptive vsan CLI command to restart the Domain Manager.


Note While the static option can be applied to runtime after a disruptive or nondisruptive restart, the preferred option is applied to runtime only after a disruptive restart.



Using Fabric Reconfiguration for Domain ID Assignments

You can use a fabric reconfiguration to reassign domain IDs and resolve any overlapping domain IDs. If you enable the auto-reconfigure option on both switches before connecting the fabric, a disruptive reconfiguration (RCF) occurs. The RCF functionality would automatically force a new principal switch selection and cause a new domain IDs to be assigned to the different switches.


Caution A disruptive reconfiguration might affect data traffic.

Using Fabric Reconfiguration for Domain ID Assignments with Fabric Manager

To use fabric reconfiguration to reassign domain IDs for a particular VSAN using Fabric Manager, follow these steps:


Step 1 Choose Switches > Interfaces > FC Physical and select the Domain Manager tab in the Information pane.

Step 2 Uncheck the RcfReject check box and click Apply Changes to disable RCF rejection.

Step 3 Choose Fabricxx > VSANxx > Domain Manager in the Logical Domain pane.

Step 4 Click the Configuration tab in the Information pane and set the Config Type drop-down menu to preferred to remove any static domain ID assignments.

Step 5 Check the AutoReconfigure check box to enable the auto-reconfiguration option.

Step 6 Set the Restart drop-down menu to disruptive and click Apply Changes to restart the Domain Manager.


Using Fabric Reconfiguration for Domain ID Assignments with the CLI

To use fabric reconfiguration to reassign domain IDs for a particular VSAN using the CLI, follow these steps:


Step 1 Use the show fcdomain domain-list CLI command to determine if you have statically assigned domain IDs on the switches.

Step 2 If you have statically assigned domain IDs, use the no fcdomain domain CLI command to remove the static assignments.

Step 3 Use the show fcdomain vsan CLI command to determine if you have rcf-reject option enabled.

switch# show fcdomain vsan 1
The local switch is a Subordinated Switch

Local switch run time information:
      State: Stable
      Local switch WWN:    20:01:00:05:30:00:51:1f
      Running fabric name: 10:00:00:60:69:22:32:91
      Running priority: 128
      Current domain ID: 0x64(100) ß verify domain id

Local switch configuration information:
        State: Enabled
        Auto-reconfiguration: Disabled
        Contiguous-allocation: Disabled
        Configured fabric name: 41:6e:64:69:61:6d:6f:21
        Configured priority: 128
        Configured domain ID: 0x64(100) (preferred)

Principal switch run time information:
       Running priority: 2

Interface               Role          RCF-reject
----------------    -------------    ------------
fc2/1               Downstream       Enabled
fc2/2               Downstream       Disabled
fc2/7               Upstream         Disabled
----------------    -------------    ------------

Step 4 If you have the rcf-reject option enabled, use the interface CLI command and then the no fcdomain rcf-reject vsan CLI command in interface mode.

Step 5 Use the fcdomain auto-reconfigure vsan CLI command in the EXEC mode on both switches to enable auto-reconfiguration after a Domain Manager restart.

Step 6 Use the fcdomain restart disruptive vsan CLI command to restart the Domain Manager.


FSPF Issues

The implementation of VSANs dictates that each configured VSAN support a separate set of fabric services. One such service is the FSPF routing protocol, which can be independently configured per VSAN. Therefore, within each VSAN topology, FSPF can be configured to provide a unique routing configuration and resulting traffic flow. Using the traffic engineering capabilities offered by VSANs allows a greater control over traffic within the fabric and a higher utilization of the deployed fabric resources.

This section describes how to identify and resolve Fabric Shortest Path First (FSFP) problems. It includes the following topics:

Troubleshooting FSPF

Loss of Two-Way Communication

Troubleshooting FSPF

Figure 7-1 shows a single-VSAN topology.

Figure 7-1 Single VSAN Topology

For the purpose of this example, assume that all interfaces are located in VSAN 1.

Troubleshooting FSPF Using Device Manager

To troubleshoot FSPF using Device Manager, follow these steps:


Step 1 Choose FC > Advanced > FSPF and select the LSDB LSRs tab to verify the link state records in the FSPF database.

The VSANId/ DomainId column shows the domain's view of the fabric topology.

The AdvDomainId column shows which domain is the owner of the LSR (link state record).

The Age value is a 16-bit counter starting at 0x0000, incremented by one for each switch during flooding and by one for each second held in the database. This field is used as a tie-breaker if Incarnation numbers are the same.

The IncarnationNumber is a 32-bit value between 0x80000001 and 0x7FFFFFFF that is incremented by one each time the originating switch transmits an LSR. This is used first before the Age value.

Step 2 Choose FC > Advanced > FSPF and select the LSDB Links tab to verify that each path is in the FSPF database.

Step 3 Choose FC > Advanced > FSPF and select the Interfaces tab to verify that the FSPF parameters are correct for each interface and verify that the AdminStatus is up.

The Cost column shows the cost of the path out of the interface.

The Intervals column shows the configured FSPF timers for this interface, which must match on both sides.

The State column shows the full or adjacent state if the interface has sent and received all database exchanges and required Acks. The port is now ready to route frames.

The Neighbors column shows FSPF neighbor information.

Step 4 Choose FC > Advanced > FSPF and select the Statistics or InterfaceStats tab to verify that there are no excessive errors present.


Troubleshooting FSPF Using the CLI

To troubleshoot FSPF using the CLI, follow these steps:


Step 1 Use the show fspf database vsan CLI command to verify that each path is in the FSPF database.

switch1# show fspf database
FSPF Link State Database for VSAN 2 Domain 1 -----1
LSR Type                = 1
Advertising domain ID   = 1  -----2
LSR Age                 = 81 -----3
LSR Incarnation number  = 0x80000098 -----4
LSR Checksum            = 0x2cd3
Number of links         = 2
 NbrDomainId          IfIndex          NbrIfIndex          Link Type          Cost
--------------------------------------------------------------------------------------
237          0x00010002          0x00010001          1          1000 -----5
238          0x00010003          0x00010002          1          1000 -----6

FSPF Link State Database for VSAN 2 Domain 237 <-----------LSR for another switch
LSR Type                = 1
Advertising domain ID   = 237 -----7
LSR Age                 = 185
LSR Incarnation number  = 0x8000000c
LSR Checksum            = 0xe0a2
Number of links         = 2
 NbrDomainId          IfIndex          NbrIfIndex          Link Type          Cost
--------------------------------------------------------------------------------------
239          0x00010000          0x00010003          1          1000 -----8
  1          0x00010001          0x00010002          1          1000 -----9

FSPF Link State Database for VSAN 2 Domain 238  <-----------LSR for another switch
LSR Type                = 1
Advertising domain ID   = 238
LSR Age                 = 1052
LSR Incarnation number  = 0x80000013
LSR Checksum            = 0xe294
Number of links         = 2
 NbrDomainId          IfIndex          NbrIfIndex          Link Type         Cost
--------------------------------------------------------------------------------------
239          0x00010003          0x00010001          1          1000
 1           0x00010002          0x00010003          1          1000

FSPF Link State Database for VSAN 2 Domain 239 <-----------LSR for another switch
LSR Type                = 1
Advertising domain ID   = 239
LSR Age                 = 1061
LSR Incarnation number  = 0x80000086
LSR Checksum            = 0x66ac
Number of links         = 4
 NbrDomainId          IfIndex          NbrIfIndex          Link Type          Cost
--------------------------------------------------------------------------------------
237          0x00010003          0x00010000          1          1000
238          0x00010001          0x00010003          1          1000

1. The domain 1 view of the fabric topology.

2. Domain 1 is owner of the LSR (link state record).

3. This is a 16-bit counter starting at 0x0000, incremented by one for each switch during flooding and by one for each second held in database. This field is used as a tie-breaker if Incarnation numbers are the same.

4. This is a 32-bit value between 0x80000001 and 0x7FFFFFFF. which is incremented by one each time the originating switch transmits an LSR. This is used first before LSR Age.

5. The path to domain 237, switch 1.

6. The path to domain 238, switch 5.

7. Switch 1, domain ID 237 is the owner.

8. The path to domain 239, switch 3.

9. The path to domain 1, switch 2.

Step 2 Use the show fspf vsan vsan-id interface CLI command to verify that the FSPF parameters are correct for each interface and verify that the interface is in the FSPF active state.

switch1# show fspf vsan 2 interface fc1/2 
FSPF interface fc1/2 in VSAN 2
FSPF routing administrative state is active -----1
Interface cost is 1000  -----2
Timer intervals configured, Hello 20 s, Dead 80 s, Retransmit 5 s -----3
FSPF State is FULL -----4
Neighbor Domain Id is 1, Neighbor Interface index is 0x00010002 -----5
Statistics counters : 
   Number of packets received : LSU  46  LSA  24  Hello 103  Error packets 0   
   Number of packets transmitted : LSU  24  LSA  45  Hello 104  Retransmitted LSU  0 
   Number of times inactivity timer expired for the interface = 0 

This displays the number of packets; Hellos should be received every 20 seconds.

10. The cost of the path out this interface.

11. The configured FSPF timers for this interface, which must match on both sides.

12. Either Full State or Adjacent. Sent and received all database exchanges and required Acks. Port is now ready to route frames.

13. FSPF neighbor information.

Step 3 Use the show fspf internal route vsan CLI command to verify that all Fibre Channel routes are available.


Note To issue commands with the internal keyword, you must have an account that is a member of the network-admin group.


switch1# show fspf internal route vsan 2
FSPF Unicast Routes 
---------------------------
 VSAN     Number          Dest Domain          Route Cost          Next hops
----------------------------------------------------------------------------------------
1                   0x01(1)                    1000                  fc1/2
1                   0xEF(239)                  1000                  fc1/1
1                   0xED(238)                  2000                  fc1/1
                                                                     fc1/2

This shows the total cost of all links.

The next hop to (238) has two interfaces. This indicates that both paths will be used during load sharing. Up to sixteen paths can be used by FSPF with a Cisco MDS 9000 Family switch.


With the implementation of VSANs used with Cisco MDS 9000 Family switches, a separate instance of FSPF runs within each VSAN, and each instance is independent of the others. For this reason, FSPF issues affecting one VSAN have no effect on FSPF running in other VSANs.


Note For all FSPF configuration statements and diagnostic commands, if the vsan keyword is not specified, VSAN 1 is used by default. When making configuration changes or issuing diagnostic commands in a multi-VSAN environment, be sure to explicitly specify the target VSAN by including the vsan keyword in the statement or command


Loss of Two-Way Communication

If FSPF is misconfigured, then the switches will not reach the "two-way" state.

The following events occur when two-way communication is lost:

The port enters Init state and removes its neighbor's domain ID from the Recipient Domain ID field and inserts 0xFFFFFFFF.

FSPF removes the Inter-Switch Link (ISL) from the topology database.

New link state records (LSRs) are flooded to adjacent switches to notify them that the FSPF database has changed.

Symptom    Traffic is not being routed through the fabric.

Table 7-13 Traffic is not Being Routed Through the Fabric

Symptom
Possible Cause
Solution

Traffic is not being routed through the fabric.

FSPF hello interval misconfigured.

See the "Resolving a Wrong Hello Interval on an ISL Using Device Manager" section or the "Resolving a Wrong Hello Interval on an ISL Using the CLI" section.

FSPF retransmit time misconfigured.

See the "Resolving a Mismatched Retransmit Interval on an ISL Using Device Manager" section or the "Resolving a Mismatched Retransmit Interval on an ISL Using the CLI" section.

FSPF dead interval misconfigured.

See the "Resolving a Mismatch in Dead Intervals on an ISL Using Fabric Manager" section or the "Resolving a Mismatch in Dead Intervals on an ISL Using the CLI" section.

There is a region mismatch on the switch.

See the "Resolving a Region Mismatch Using Fabric Manager" section or the "Resolving a Region Mismatch Using the CLI" section.


Resolving a Wrong Hello Interval on an ISL Using Device Manager

To resolve a wrong hello interval on an ISL using Device Manager, follow these steps:


Step 1 Choose FC > Advanced > FSPF and select the Interfaces tab to verify that the FSPF parameters are correct for each interface and check the Hello interval column and the State column.

The Intervals column shows the configured FSPF timers for this interface, which must match on both sides.

The State column shows the full or adjacent state if the interface has sent and received all database exchanges and required Acks. The port is now ready to route frames.

Step 2 Repeat Step 1 to determine the value of the hello interval on the adjacent switch.

Step 3 Fill in the Hello field to change the hello interval and click Apply.


Resolving a Wrong Hello Interval on an ISL Using the CLI

To resolve a wrong hello interval on an ISL using the CLI, follow these steps:


Step 1 Use the debug fspf all CLI command and look for wrong hello interval messages.

switch1# debug fspf all
Jan  5 00:28:14 fspf: Wrong hello interval for packet on interface 100f000 in VSAN 1 
Jan  5 00:28:14 fspf: Error in processing hello packet ,  error code = 4  

Tip We recommend that you open a second Telnet or SSH session before entering any debug commands. If the debug output overwhelms the current session, you can use the second session to enter the undebug all command to stop the debug message output.


Step 2 Use the undebug all command to turn off debugging.

Step 3 Use the show fspf internal route vsan to show FSPF information.


Note To issue commands with the internal keyword, you must have an account that is a member of the network-admin group.


switch1# show fspf internal route vsan 1
FSPF Unicast Routes 
---------------------------
 VSAN Number   Dest Domain   Route Cost    Next hops
----------------------------------------------------------------------
1              0xEF(239)      1000          fc1/1 -----1
1              0xED(238)      2000          fc1/1 
1              0x01(1)        3000          fc1/1 -----2

1. There is no second path to domain 238, through domain 1 switch 2.

2. There is no direct path to domain 1 switch 2; traffic must travel through three ISLs. This is based on the route cost column.


Step 4 Use the show fspf vsan vsan-id interface CLI command to view the FSFP configuration.

switch1# show fspf vsan 1 interface fc1/16
FSPF interface fc1/16 in VSAN 1
FSPF routing administrative state is active
Interface cost is 500
Timer intervals configured, Hello 5 s, Dead 80 s, Retransmit 5 s -----1
FSPF State is INIT -----2
Statistics counters :
   Number of packets received : LSU  0  LSA  0  Hello 2  Error packets 1 
   Number of packets transmitted : LSU  0  LSA  0  Hello 4  Retransmitted LSU  0
Number of times inactivity timer expired for the interface = 0

1. The Hello timer is not set to the default, so you should check the neighbor configuration to make sure it matches.

2. FSPF is not in FULL state, indicating a problem.

Step 5 Repeat Step 4 to determine the value of the Hello timer on the adjacent switch.

switch2# show fspf v 1 interface fc2/16
FSPF interface fc2/16 in VSAN 1
FSPF routing administrative state is active
Interface cost is 500
Timer intervals configured, Hello 20 s, Dead 80 s, Retransmit 5 s -----1
FSPF State is INIT -----2
Statistics counters :
   Number of packets received : LSU  0  LSA  0  Hello 2  Error packets 1 
   Number of packets transmitted : LSU  0  LSA  0  Hello 4  Retransmitted LSU  0
   Number of times inactivity timer expired for the interface = 0

1. The neighbor FSPF Hello interval is set to the default (20 seconds).

2. FSPF is not in full state, indicating a problem.

Step 6 Use the interface CLI command and then the fspf hello-interval CLI command in interface mode to change the default Hello interval.


Resolving a Mismatched Retransmit Interval on an ISL Using Device Manager

To resolve a mismatched retransmit interval on an ISL using Device Manager, follow these steps:


Step 1 Choose FC > Advanced > FSPF and select the Interfaces tab to verify that the FSPF parameters are correct for each interface and check the Retransmit interval column and the State column.

The Intervals column shows the configured FSPF timers for this interface, which must match on both sides.

The State column shows the full or adjacent state if the interface has sent and received all database exchanges and required Acks. The port is now ready to route frames.

Step 2 Repeat Step 1 to determine the value of the retransmit interval on the adjacent switch.

Step 3 Fill in the Retransmit field to change the retransmit interval and click Apply.


Resolving a Mismatched Retransmit Interval on an ISL Using the CLI

To resolve a mismatched retransmit interval on an ISL using the CLI, follow these steps:


Step 1 Use the show fspf vsan vsan-id interface CLI command to view the FSFP configuration.

switch1# show fspf vsan 1 interface fc1/16
FSPF interface fc1/16 in VSAN 1
FSPF routing administrative state is active
Interface cost is 500
Timer intervals configured, Hello 5 s, Dead 80 s, Retransmit 10 s -----1
FSPF State is INIT -----2
Statistics counters :
   Number of packets received : LSU  0  LSA  0  Hello 2  Error packets 1 
   Number of packets transmitted : LSU  0  LSA  0  Hello 4  Retransmitted LSU  0
Number of times inactivity timer expired for the interface = 0

1. The retransmit interval is not set to the default, so you should check the neighbor configuration to make sure it matches.

2. FSPF is not in FULL state, indicating a problem.

Step 2 Repeat Step 1 to determine the value of the retransmit interval on the adjacent switch.

switch2# show fspf v 1 interface fc2/16
FSPF interface fc2/16 in VSAN 1
FSPF routing administrative state is active
Interface cost is 500
Timer intervals configured, Hello 20 s, Dead 80 s, Retransmit 5 s -----1
FSPF State is INIT -----2
Statistics counters :
   Number of packets received : LSU  0  LSA  0  Hello 2  Error packets 1 
   Number of packets transmitted : LSU  0  LSA  0  Hello 4  Retransmitted LSU  0
   Number of times inactivity timer expired for the interface = 0

1. The neighbor retransmit interval interval is set to the default (5 seconds).

2. FSPF is not in FULL state, indicating a problem.

Step 3 Use the interface CLI command and then the fspf retransmit-interval CLI command in interface mode to change the retransmit interval.


Resolving a Mismatch in Dead Intervals on an ISL Using Fabric Manager

To resolve a mismatch of dead intervals on an ISL using Fabric Manager, follow these steps:


Step 1 Choose FC > Advanced > FSPF and select the Interfaces tab to verify that the FSPF parameters are correct for each interface and check the Dead interval column and the State column.

The Intervals column shows the configured FSPF timers for this interface, which must match on both sides.

The State column shows the full or adjacent state if the interface has sent and received all database exchanges and required Acks. The port is now ready to route frames.

Step 2 Repeat Step 1 to determine the value of the dead interval on the adjacent switch.

Step 3 Fill in the Dead field to change the dead interval and click Apply.


Resolving a Mismatch in Dead Intervals on an ISL Using the CLI

To identify a mismatch in dead intervals on an ISL, follow these steps:


Step 1 Use the debug fspf all CLI command and look for wrong dead interval messages.

switch1# debug fspf all
Jan  5 00:28:14 fspf: Wrong dead interval for packet on interface 100f000 in VSAN 1 
Jan  5 00:28:14 fspf: Error in processing hello packet ,  error code = 4  

Tip We recommend that you open a second Telnet or SSH session before entering any debug commands. If the debug output overwhelms the current session, you can use the second session to enter the undebug all command to stop the debug message output.


Step 2 Use the undebug all command to turn off debugging.

Step 3 Use the show fspf vsan <vsan-id> interface to show FSPF information.


Note To issue commands with the internal keyword, you must have an account that is a member of the network-admin group.


switch1# show fspf vsan 1 interface fc1/16
FSPF interface fc1/16 in VSAN 1
FSPF routing administrative state is active
Interface cost is 500
Timer intervals configured, Hello 20 s, Dead 95 s, Retransmit 5 s -----1
FSPF State is INIT -----2
xStatistics counters :
   Number of packets received : LSU  0  LSA  0  Hello 2  Error packets 1 
   Number of packets transmitted : LSU  0  LSA  0  Hello 4  Retransmitted LSU  0
   Number of times inactivity timer expired for the interface = 0

1. The dead timer is not set to the default, so you should check the neighbor configuration.

2. FSPF is not in full state, which indicates a problem.

Step 4 Use the interface CLI comma nd and then the fspf dead-interval CLI command in interface mode to change the dead interval.


Resolving a Region Mismatch Using Fabric Manager

To identify a region mismatch problem on a switch using Fabric Manager, follow these steps:


Step 1 Choose FC > Advanced > FSPF and select the General tab to verify the RegionId.

Step 2 Repeat Step 1 to determine the value of the region on the adjacent switch.

Step 3 Fill in the RegionId field to change the region and click Apply.


Resolving a Region Mismatch Using the CLI

To identify a region mismatch problem on a switch using the CLI, follow these steps:


Step 1 Use the show fspf vsan CLI command to display the currently configured region in a VSAN.

switch# show fspf vsan 99

FSPF routing for VSAN 99 
FSPF routing administration status is enabled 
FSPF routing operational status is UP 
It is an intra-domain router 
Autonomous region is 0    /* This is the region */ 
SPF hold time is 0 msec 
MinLsArrival = 1000 msec , MinLsInterval = 5000 msec 
Local Domain is 0x78(120) 
Number of LSRs = 2, Total Checksum = 0x000133de

Step 2 Use the debug fspf all CLI command and look for nonexistent region messages.

switch1# debug fspf all
Jan  5 00:39:31 fspf: FC2 packet received for non existent  region 0 in VSAN 1 -----1
Jan  5 00:39:33 fspf: FC2 packet received for non existent  region 0 in VSAN 1  
Jan  5 00:39:45 fspf: Interface fc1/1 in VSAN 1 : Event INACTIVITY , State change INIT -> 
INIT  
Jan  5 00:39:45 fspf: Interface fc1/2 in VSAN 1 : Event INACTIVITY , State change INIT -> 
INIT -----2

1. The neighbor switch advertising region is 0.

2. FSPF is in init state for each ISL.


Tip We recommend that you open a second Telnet or SSH session before entering any debug commands. If the debug output overwhelms the current session, you can use the second session to enter the undebug all command to stop the debug message output.


Step 3 Use the undebug all command to turn off debugging.

Step 4 Use the show fspf CLI command to show FSPF configuration and check the autonomous region.

Step 5 Use the fspf config vsan CLI command to enter the fspf configuration mode and use the region CLI command to change the region.


The region must match on all switches in the VSAN.