Cisco MDS 9000 Family Storage Media Encryption Configuration Guide
Cisco SME Troubleshooting
Downloads: This chapterpdf (PDF - 165.0KB) The complete bookPDF (PDF - 10.16MB) | Feedback

Cisco SME Troubleshooting

Table Of Contents

Cisco SME Troubleshooting

Troubleshooting Resources

Cluster Recovery Scenarios

Deleting an Offline Switch from a Cisco SME Cluster

Deleting a Cisco SME Cluster with One or More Offline Switches while the Master Switch is Online

Deleting a Cisco SME Cluster when All Switches Are Offline

Reviving a Cisco SME Cluster

Reassigning the Cisco SME Cluster Master Switch

Troubleshooting General Issues

Troubleshooting Scenarios


Cisco SME Troubleshooting


This chapter describes basic troubleshooting methods used to resolve issues with Cisco Storage Media Encryption.

This chapter includes the following sections:

Cluster Recovery Scenarios

Troubleshooting General Issues

Troubleshooting Scenarios

Troubleshooting Resources

For additional information on troubleshooting, the Cisco MDS 9000 Family Troubleshooting Guide provides guidance for troubleshooting issues that may appear when deploying a storage area network (SAN) using the Cisco MDS 9000 Family of switches. The Cisco MDS 9000 Family Troubleshooting Guide introduces tools and methodologies that are used to recognize a problem, determine its cause, and find possible solutions.

Cluster Recovery Scenarios

This section includes information on recovery procedures used when one or more switches in a Cisco SME cluster are offline or when you want to change the master switch assignment from one switch to another switch. It includes the following procedures:

Deleting an Offline Switch from a Cisco SME Cluster

Deleting a Cisco SME Cluster with One or More Offline Switches while the Master Switch is Online

Deleting a Cisco SME Cluster when All Switches Are Offline

Reviving a Cisco SME Cluster

Reassigning the Cisco SME Cluster Master Switch


Note The procedures in this section describe troubleshooting solutions that use the CLI.



Note The Cisco SME cluster configuration for an offline switch must be done using the CLI. Cisco SME cluster configuration for an online switch can be done using Fabric Manager or the CLI.


Deleting an Offline Switch from a Cisco SME Cluster

To delete an offline switch when one or more switches are offline and the master switch is online, use these procedures.

On the offline switch (for example, switch2), shut down the cluster by performing this task:

 
Command
Purpose

Step 1 

switch# config t

Enters configuration mode.

Step 2 

switch(config)# sme cluster ABC

switch(config-sme-cl)# shutdown

Shuts down the ABC cluster on the offline switch.


Note Repeat the procedure for every offline switch.


On the cluster master switch, delete the offline switch (for example, switch2) by performing this task:

 
Command
Purpose

Step 1 

switch# config t

Enters configuration mode.

Step 2 

switch(config)# sme cluster ABC

switch(config-sme-cl)# no node switch2

Deletes switch2 from the ABC cluster configuration.

Note Repeat this step for every offline switch that was shut down in Step 1.

On the offline switch (switch2), delete the cluster by performing this task:

 
Command
Purpose

Step 1 

switch# config t

Enters configuration mode.

Step 2 

switch(config)#no sme cluster ABC

Deletes the ABC cluster configuration.


Note Delete the cluster on every offline switch that was shut down in the first procedure.


Deleting a Cisco SME Cluster with One or More Offline Switches while the Master Switch is Online

To delete a Cisco SME cluster that includes one or more offline switches and online master switch, use these procedures.


Caution Do not remove a cluster master switch from a cluster and then try to revive the cluster on an offline switch. Since the offline switch was not part of the operational cluster, the cluster master may have progressed beyond what is in the offline switch's state. Deleting the cluster master and reviving the cluster on an offline switch can lead to data corruption.

On the offline switch (switch2), shut down the cluster by performing this task:

 
Command
Purpose

Step 1 

switch# config t

Enters configuration mode.

Step 2 

switch(config)# sme cluster ABC

switch(config-sme-cl)# shutdown

Shuts down the ABC cluster on the offline switch


Note Repeat the procedure for every offline switch.


On the cluster master switch, delete the offline switch (switch2) and then delete the cluster by performing this task:

 
Command
Purpose

Step 1 

switch# config t

Enters configuration mode.

Step 2 

switch(config)# sme cluster ABC

switch(config-sme-cl)# no node switch2

Deletes switch2 from the ABC cluster configuration.

Note Repeat this step for every offline switch that was shut down in the first procedure.

Step 3 

switch(config)# no sme cluster ABC

Deletes the ABC cluster configuration.

On the offline switch (switch2), delete the cluster by performing this task:

 
Command
Purpose

Step 1 

switch# config t

Enters configuration mode.

Step 2 

switch(config)# no sme cluster ABC

Deletes the ABC cluster configuration.


Note Delete the cluster on every offline switch that was shut down in the first procedure.


Deleting a Cisco SME Cluster when All Switches Are Offline

To delete a Cisco SME cluster when the master switch and all other switches are offline, use these procedures.


Note When all switches are offline, the cluster is offline.


On the offline switch (for example, switch2), shut down the cluster by performing this task:

 
Command
Purpose

Step 1 

switch# config t

Enters configuration mode.

Step 2 

switch(config)#sme cluster ABC

switch(config-sme-cl)#shutdown

Shuts down the ABC cluster on the offline switch.


Note Repeat this procedure for every offline switch.


On the cluster master switch, shut down the cluster and then delete the cluster by performing this task:

 
Command
Purpose

Step 1 

switch# config t

Enters configuration mode.

Step 2 

switch(config)#sme cluster ABC

switch(config-sme-cl)#shutdown

Shuts down the ABC cluster.

Step 3 

switch(config)#no sme cluster ABC

Deletes the ABC cluster configuration.

On the offline switch (switch2), delete the cluster by performing this task:

 
Command
Purpose

Step 1 

switch# config t

Enters configuration mode.

Step 2 

switch(config)# no sme cluster ABC

Deletes the ABC cluster configuration.


Note Delete the cluster on every offline switch that was shut down in the first procedure.


Reviving a Cisco SME Cluster

To revive a cluster on the switch that has the latest Cisco SME configuration version, use these procedures.

This procedure is used to revive a cluster when one or more switches are offline and the cluster is nonoperational (for example, due to a quorum loss). The recovery procedure includes deleting one or more offline switches and then reviving the cluster on the remaining switches.


Caution A Cisco SME cluster must only be revived on the switch with the latest SME configuration version as displayed by the show sme cluster detail command. Reviving the cluster on a switch that does not have the highest configuration version can lead to data corruption.


Note The following procedure assumes that switch1 has the latest SME configuration version. The steps shown for switch2 should be carried out for every switch that needs to be removed before reviving the cluster.


On switch2, shut down the cluster by performing this task:

 
Command
Purpose

Step 1 

switch# config t

Enters configuration mode.

Step 2 

switch(config)# sme cluster ABC

switch(config-sme-cl)# shutdown

Shuts down the ABC cluster on the offline switch.


Note Repeat this procedure for every offline switch.


On switch1, shut down the cluster by performing this task:

 
Command
Purpose

Step 1 

switch# config t

Enters configuration mode.

Step 2 

switch(config)# sme cluster ABC

switch(config-sme-cl)# shutdown

Shuts down the ABC cluster.

On switch2, delete the cluster by performing this task:

 
Command
Purpose

Step 1 

switch# config t

Enters configuration mode.

Step 2 

switch(config)# no sme cluster ABC

Deletes the ABC cluster configuration.


Note Repeat the preceding procedure for every offline switch that was shut down in the first procedure.


On switch1, delete switch2 from the configuration and restart the cluster by performing this task:

 
Command
Purpose

Step 1 

switch# config t

Enters configuration mode.

Step 2 

switch(config)# sme cluster ABC

switch(config-sme-cl# no node switch2

Deletes switch2 from the configuration.

Note Repeat for every switch that needs to be deleted.

Step 3 

switch(config)# sme cluster ABC

switch(config-sme-cl)# no shutdown

Restarts the cluster on switch2.

Reassigning the Cisco SME Cluster Master Switch

To reassign the master switch assignment to another switch, use these procedures. This will change the master switch assignment from one switch to another switch. For information on the master switch election, see the "Cluster Quorum and Master Switch Election Overview" section.


Caution This procedure is disruptive and it will shut down Cisco SME during this change.


Note The following procedure assumes that switch1 is the cluster master switch and switch2 will become the new master switch.


On all switches other than switch1 and switch2, shut down the cluster by entering the following commands. Perform this step for every switch in the cluster other than the current master switch and the desired new master switch.

 
Command
Purpose

Step 1 

switch# config t

Enters configuration mode.

Step 2 

switch(config)# sme cluster ABC

switch(config-sme-cl)# shutdown

Shuts down the ABC cluster on all switches other than the current master switch and the desired new master switch.

On switch2, shut down the cluster by performing this task:

 
Command
Purpose

Step 1 

switch# config t

Enters configuration mode.

Step 2 

switch(config)# sme cluster ABC

switch(config-sme-cl)#shutdown

Shuts down the ABC cluster on the switch2.

On the switch1, the current cluster master switch, shut down the cluster and then delete the cluster by performing this task:

 
Command
Purpose

Step 1 

switch# config t

Enters configuration mode.

Step 2 

switch(config)# sme cluster ABC

switch(config-sme-cl)# shutdown

Shuts down the ABC cluster.

Step 3 

switch(config)# no sme cluster ABC

Deletes the ABC cluster configuration.

On switch2, delete switch1 (the original master switch) from the configuration and restart the cluster by performing this task:

 
Command
Purpose

Step 1 

switch# config t

Enters configuration mode.

Step 2 

switch(config)# sme cluster ABC

switch(config-sme-cl)# no node switch1

switch(config-sme-cl)# no shutdown

Deletes switch1 from the configuration and restarts the cluster on switch2.

Switch2 now becomes the cluster master for this cluster.

On all remaining switches in the cluster, restart the cluster by performing this task:

 
Command
Purpose

Step 1 

switch# config t

Enters configuration mode.

Step 2 

switch(config)# sme cluster ABC

switch(config-sme-cl)# no shutdown

Restarts the cluster on all remaining switches and synchronizes the configuration from the new master switch: switch2.

Troubleshooting General Issues

The Cisco SME naming convention includes alphanumeric, dash, and underscore characters. Other types of characters will cause problems in the cluster configuration.

Troubleshooting Scenarios

The following scenarios are described in this section:

If DNS is not configured on all switches in a cluster

If you need to replace an MSM-18/4 module with another MSM-18/4 module

If a Cisco SME cluster is not successfully created

A Cisco SME interface does not come up in a cluster

When selecting paths, a "no paths found" message is displayed

Newly added tape drives are not showing in a cluster

If you need to contact your customer support representative or Cisco TAC

A syslog message is displayed when a Cisco MDS switch configured with Cisco SME in the startup configuration boots up

Importing a volume group file causes a 'wrap key object not found' error message

Accounting log file shows the replication of keys failed.

If DNS is not configured on all switches in a cluster

You can use sme.useIP for IP address or name selection when DNS is not configured on all switches in a cluster.

sme.useIP can be used in smeserver.properties to enable the use of IP addresses instead of switch names. By default sme.useIP is set to false and DNS names will be used. When DNS is not configured, Fabric Manager Server cannot resolve the switch names.

When sme.useIP is set to true, Fabric Manager Server uses an IP address to communicate with switch in the cluster using SSH. All switches are added to the cluster with an IP address. When you add a local switch, the switch name is used if the name server is configured on the switch otherwise the IP address is used.

When sme.useIP is false, Fabric Manager Server will use the switch name to select interfaces. All the switches added to the clusters will be identified with names. A name server is required for this type of configuration. Otherwise, switches will not be able to communicate with other switches to form the cluster and Fabric Manager Server will not be able to resolve the switch name.

If you need to replace an MSM-18/4 module with another MSM-18/4 module

In the existing MDS 9000 Family platform, a module can be replaced with another module and there is no change in configuration. In Cisco SME, due to security reasons, when an MSM-18/4 module is configured as part of a cluster, it cannot be replaced with another MSM-18/4 module otherwise the Cisco SME interface will come up in an inactive state. The correct procedure is to remove the Cisco SME interface from the cluster and re-add the interface back into the cluster.

If a Cisco SME cluster is not successfully created

There are three main reasons that a Cisco SME cluster may not be successfully created:

SSH must be enabled on every switch that is part of a Cisco SME cluster.


Note Only SSH/dsa or SSH/rsa are supported for Cisco SME cluster configurations using Fabric Manager Web Client. SSH/rsa1 is not supported for SME cluster config via FM web client in 3.2.2 (release with SME feature). It may (or may not) be supported in future releases.


If the Cisco SME switches are managed using their IP addresses (instead of host names or FQDN), the entry "sme.useIP=true" must be set in the smeserver.properties file. Be sure to restart the Fabric Manager Server after modifying the smeserver.properties file.

The DNS server must be configured.

Sometimes, improperly configured personal firewall software (running on Cisco Fabric Manager Server) may also cause a created SME cluster to stay in the "pending" state. Be sure to create proper firewall rules to allow necessary traffic between Fabric Manager Server/Fabric Manager web client/switches.

A Cisco SME interface does not come up in a cluster

If a Cisco SME interface does not come up, this can be due to the following:

A Cisco SME license is not installed or the license has expired.

An MSM-18/4 module has been replaced after the Cisco SME interface has been configured.

The copy running-config startup-config command was not entered after adding or deleting a Cisco SME interface from a cluster or before rebooting the switch.

For the second and third scenarios, you must first remove and re-add the interface to the cluster and then enter the copy running-config startup-config command.

When selecting paths, a "no paths found" message is displayed

A tape library controller or robot can be shown as a target in the Select Tape Drives wizard. If you select the controller or robot as a target, a "no paths found" message is displayed. You will need to verify whether or not the selected target is a controller or robot.

When the "no paths found" message is displayed, enter the show tech and show tech-support sme command.

Newly added tape drives are not showing in a cluster

If you add new tape drives as LUNs to a tape library after Cisco SME has already discovered available tape drives, a rescan is required from the host to discover the new LUNs.

If you need to contact your customer support representative or Cisco TAC

At some point, you may need to contact your customer support representative or Cisco TAC for some additional assistance. Before doing so, enter the show tech details and the show tech sme commands and collect all logs from the C:\Program Files\Cisco Systems\MDS 9000\logs directory before contacting your support organization.

A syslog message is displayed when a Cisco MDS switch configured with Cisco SME in the startup configuration boots up

When you reboot a Cisco MDS switch that has the cluster configuration stored in the startup-config file, the following syslog message may be displayed:

<timestamp> <switch name> %CLUSTER-2-CLUSTER_DB_SYNC_FAIL: Cluster <cluster-id> 
application 3 dataset 1 database synchronization failed, reason="Invalid cluster API 
registration"
 
   

This error message is expected and can be ignored.

Importing a volume group file causes a 'wrap key object not found' error message

A tape volume group was created and the volume group was exported to a file. The tape volume group was deleted and a new tape volume group was created. When the same volume group was imported, the import operation fails and the error message "wrap key object not found" is displayed.

This error occurs because there is another volume group key active in the Key Management Center with the same index (but different versions) as the current volume group into which the import operation is performed.

Accounting log file shows the replication of keys failed.

The replication of a key for a cluster fails when the transaction context is invalid or is expired. The key entry will be moved to Sme_repl_error_key table. You should manually remove this record from the Sme_repl_error_key table to the Sme_repl_pending_key table and retry the replication process.