Table Of Contents

Cisco SME Troubleshooting

Troubleshooting Resources

Cluster Recovery Scenarios

Deleting an Offline Switch from a Cisco SME Cluster

Deleting a Cisco SME Cluster with One or More Offline Switches while the Master Switch is Online

Deleting a Cisco SME Cluster when All Switches Are Offline

Reviving a Cisco SME Cluster

Reassigning the Cisco SME Cluster Master Switch

Troubleshooting General Issues

Cisco SME Naming Conventions

Troubleshooting Scenarios

Cisco SME Troubleshooting

---

This chapter describes basic troubleshooting methods used to resolve issues with Cisco Storage Media Encryption.

This chapter includes the following sections:

•Cluster Recovery Scenarios

•Troubleshooting General Issues

•Troubleshooting Scenarios

Troubleshooting Resources

For additional information on troubleshooting, the Cisco MDS 9000 Family Troubleshooting Guide provides guidance for troubleshooting issues that may appear when deploying a storage area network (SAN) using the Cisco MDS 9000 Family of switches. The Cisco MDS 9000 Family Troubleshooting Guide introduces tools and methodologies that are used to recognize a problem, determine its cause, and find possible solutions.

Cluster Recovery Scenarios

This section includes information on recovery procedures used when one or more switches in a Cisco SME cluster are offline or when you want to change the master switch assignment from one switch to another switch. It includes the following:

•Deleting an Offline Switch from a Cisco SME Cluster

•Deleting a Cisco SME Cluster with One or More Offline Switches while the Master Switch is Online

•Deleting a Cisco SME Cluster when All Switches Are Offline

•Reviving a Cisco SME Cluster

•Reassigning the Cisco SME Cluster Master Switch

---

Note The procedures in this section describe troubleshooting solutions that use the CLI.

---

---

Note The Cisco SME cluster configuration for an offline switch must be done using the CLI. Cisco SME cluster configuration for an online switch can be done using Fabric Manager or the CLI.

---

Deleting an Offline Switch from a Cisco SME Cluster

Use this procedure when one or more switches are offline and the master switch is online.

To delete an offline switch from a Cisco SME cluster, follow these steps:

---

Step 1 On the offline switch (for example, switch 2), shut down the cluster by entering the following commands:

	Command	Purpose
Step 1	switch# config t	Enters configuration mode.
Step 2	switch(config)# sme cluster ABC switch(config-sme-cl)# shutdown	Shuts down the ABC cluster on the offline switch

---

Note Repeat Step 1 for every offline switch.

---

Step 2 On the cluster master switch, delete the offline switch (for example, switch2) by entering the following commands:

	Command	Purpose
Step 1	switch# config t	Enters configuration mode.
Step 2	switch(config)# sme cluster ABC switch(config-sme-cl)# no node switch2	Deletes switch2 from the ABC cluster configuration. Note Repeat this step for every offline switch that was shut down in Step 1.

Step 3 On the offline switch (switch2), delete the cluster by entering the following commands:

	Command	Purpose
Step 1	switch# config t	Enters configuration mode.
Step 2	switch(config)#no sme cluster ABC	Deletes the ABC cluster configuration.

---

Note Delete the cluster on every offline switch that was shut down in Step 1.

---

Deleting a Cisco SME Cluster with One or More Offline Switches while the Master Switch is Online

Use this procedure to delete a cluster that includes one or more offline switches and the master switch is online.

---

Caution Do not remove a cluster master switch from a cluster and then try to revive the cluster on an offline switch. Since the offline switch was not part of the operational cluster, the cluster master may have progressed beyond what is in the offline switch's state. Deleting the cluster master and reviving the cluster on an offline switch can lead to data corruption.

---

To delete a Cisco SME cluster, follow these steps:

---

Step 1 On the offline switch (switch2), shut down the cluster by entering the following commands:

	Command	Purpose
Step 1	switch# config t	Enters configuration mode.
Step 2	switch(config)# sme cluster ABC switch(config-sme-cl)# shutdown	Shuts down the ABC cluster on the offline switch

---

Note Repeat Step 1 for every offline switch.

---

Step 2 On the cluster master switch, delete the offline switch (switch2) and then delete the cluster by entering the following commands:

	Command	Purpose
Step 1	switch# config t	Enters configuration mode.
Step 2	switch(config)# sme cluster ABC switch(config-sme-cl)# no node switch2	Deletes switch2 from the ABC cluster configuration. Note Repeat this step for every offline switch that was shut down in Step 1.
Step 3	switch(config)# no sme cluster ABC	Deletes the ABC cluster configuration.

Step 3 On the offline switch (switch2), delete the cluster by entering the following commands:

	Command	Purpose
Step 1	switch# config t	Enters configuration mode.
Step 2	switch(config)# no sme cluster ABC	Deletes the ABC cluster configuration.

---

Note Delete the cluster on every offline switch that was shut down in Step 1.

---

Deleting a Cisco SME Cluster when All Switches Are Offline

Use this procedure to delete a cluster when the master switch and all other switches are offline.

---

Note When all switches are offline, the cluster is offline.

---

To delete a Cisco SME cluster, follow these steps:

---

Step 1 On the offline switch (for example, switch2), shut down the cluster by entering the following commands:

	Command	Purpose
Step 1	switch# config t	Enters configuration mode.
Step 2	switch(config)#sme cluster ABC switch(config-sme-cl)#shutdown	Shuts down the ABC cluster on the offline switch.

---

Note Repeat Step 1 for every offline switch.

---

Step 2 On the cluster master switch, shut down the cluster and then delete the cluster by entering the following commands:

	Command	Purpose
Step 1	switch# config t	Enters configuration mode.
Step 2	switch(config)#sme cluster ABC switch(config-sme-cl)#shutdown	Shuts down the ABC cluster.
Step 3	switch(config)#no sme cluster ABC	Deletes the ABC cluster configuration.

Step 3 On the offline switch (switch2), delete the cluster by entering the following commands:

	Command	Purpose
Step 1	switch# config t	Enters configuration mode.
Step 2	switch(config)# no sme cluster ABC	Deletes the ABC cluster configuration.

---

Note Delete the cluster on every offline switch that was shut down in Step 1.

---

Reviving a Cisco SME Cluster

Use this procedure to revive a cluster on the switch that has the latest Cisco SME configuration version.

This procedure is used to revive a cluster when one or more switches are offline and the cluster is nonoperational (for example, due to a quorum loss). The recovery procedure includes deleting one or more offline switches and then reviving the cluster on the remaining switches.

---

Caution A Cisco SME cluster must only be revived on the switch with the latest SME configuration version as displayed by the show sme cluster detail command. Reviving the cluster on a switch that does not have the highest configuration version can lead to data corruption.

---

---

Note The following procedure assumes that switch1 has the latest SME configuration version. The steps shown for switch2 should be carried out for every switch that needs to be removed before reviving the cluster.

---

To revive a Cisco SME cluster, follow these steps:

---

Step 1 On switch2, shut down the cluster by entering the following commands:

	Command	Purpose
Step 1	switch# config t	Enters configuration mode.
Step 2	switch(config)# sme cluster ABC switch(config-sme-cl)# shutdown	Shuts down the ABC cluster on the offline switch.

---

Note Repeat Step 1 for every offline switch.

---

Step 2 On switch1, shut down the cluster by entering the following commands:

	Command	Purpose
Step 1	switch# config t	Enters configuration mode.
Step 2	switch(config)# sme cluster ABC switch(config-sme-cl)# shutdown	Shuts down the ABC cluster.

Step 3 On switch2, delete the cluster by entering the following commands:

	Command	Purpose
Step 1	switch# config t	Enters configuration mode.
Step 2	switch(config)# no sme cluster ABC	Deletes the ABC cluster configuration.

---

Note Repeat Step 3 for every offline switch that was shut down in Step 1.

---

Step 4 On switch1, delete switch2 from the configuration and restart the cluster by entering the following commands:

	Command	Purpose
Step 1	switch# config t	Enters configuration mode.
Step 2	switch(config)# sme cluster ABC switch(config-sme-cl# no node switch2	Deletes switch2 from the configuration. Note Repeat for every switch that needs to be deleted.
Step 3	switch(config)# sme cluster ABC switch(config-sme-cl)# no shutdown	Restarts the cluster on switch2.

Reassigning the Cisco SME Cluster Master Switch

Use this procedure to delete the master switch. This will change the master switch assignment from one switch to another switch. For information on the master switch election, see Cluster Quorum and Master Switch Election Overview, page 3-30.

---

Caution This procedure is disruptive and it will shut down Cisco SME during this change.

---

---

Note The following procedure assumes that switch1 is the cluster master switch and switch2 will become the new master switch.

---

To reassign the master switch assignment to another switch, follow these steps:

---

Step 1 On all switches other than switch1 and switch2, shut down the cluster by entering the following commands. Perform this step for every switch in the cluster other than the current master switch and the desired new master switch.

	Command	Purpose
Step 1	switch# config t	Enters configuration mode.
Step 2	switch(config)# sme cluster ABC switch(config-sme-cl)# shutdown	Shuts down the ABC cluster on all switches other than the current master switch and the desired new master switch.

Step 2 On switch2, shut down the cluster by entering the following commands:.

	Command	Purpose
Step 1	switch# config t	Enters configuration mode.
Step 2	switch(config)# sme cluster ABC switch(config-sme-cl)#shutdown	Shuts down the ABC cluster on the switch2.

Step 3 On the switch1, the current cluster master switch, shut down the cluster and then delete the cluster by entering the following commands:

	Command	Purpose
Step 1	switch# config t	Enters configuration mode.
Step 2	switch(config)# sme cluster ABC switch(config-sme-cl)# shutdown	Shuts down the ABC cluster.
Step 3	switch(config)# no sme cluster ABC	Deletes the ABC cluster configuration.

Step 4 On switch2, delete switch1 (the original master switch) from the configuration and restart the cluster by entering the following commands.

	Command	Purpose
Step 1	switch# config t	Enters configuration mode.
Step 2	switch(config)# sme cluster ABC switch(config-sme-cl)# no node switch1 switch(config-sme-cl)# no shutdown	Deletes switch 1 from the configuration and restarts the cluster on switch2. Switch2 now becomes the cluster master for this cluster.

Step 5 On all remaining switches in the cluster, restart the cluster by entering the following commands:

	Command	Purpose
Step 1	switch# config t	Enters configuration mode.
Step 2	switch(config)# sme cluster ABC switch(config-sme-cl)# no shutdown	Restarts the cluster on all remaining switches and synchronizes the configuration from the new master switch: switch2.

Troubleshooting General Issues

The following general issues are described in this section:

•Cisco SME Naming Conventions

Cisco SME Naming Conventions

The Cisco SME naming convention includes alphanumeric, dash, and underscore characters. Other types of characters will cause problems in the cluster configuration.

Troubleshooting Scenarios

The following scenarios are described in this section:

•If DNS is not configured on all switches in a cluster

•If you need to replace an MSM-18/4 module with another MSM-18/4 module

•If a Cisco SME cluster is not successfully created

•A Cisco SME interface does not come up in a cluster

•When selecting paths, a "no paths found" message is displayed

•Newly added tape drives are not showing in a cluster

•If you need to contact your customer support representative or Cisco TAC

•A syslog message is displayed when a Cisco MDS switch configured with Cisco SME in the start-up configuration boots up

If DNS is not configured on all switches in a cluster

You can use sme.useIP for IP Address or name selection when DNS is not configured on all switches in a cluster.

sme.useIP can be used in smeserver.properties to enable the use of IP addresses instead of switch names. By default sme.useIP is set to false and DNS names will be used. When DNS is not configured, Fabric Manager Server cannot resolve the switch names.

When sme.useIP is set to true, Fabric Manager Server uses an IP address to communicate with switch in the cluster using SSH. All switches are added to the cluster with an IP address. When you add a local switch, the switch name is used if the nameserver is configured on the switch otherwise the IP address is used.

When sme.useIP is false, Fabric Manager Server will use the switch name to select interfaces. All the switches added to the clusters will be identified with names. Nameserver is required for this type of configuration. Otherwise, switches will not be able to communicate with other switches to form the cluster and Fabric Manager Server will not be able to resolve the switch name.

If you need to replace an MSM-18/4 module with another MSM-18/4 module

In the existing MDS 9000 Family platform, a module can be replaced with another module and there is no change in configuration. In Cisco SME, due to security reasons, when an MSM-18/4 module is configured as part of a cluster, it cannot be replaced with another MSM-18/4 module otherwise the Cisco SME interface will come up in an inactive state. The correct procedure is to remove the Cisco SME interface from the cluster and re-add the interface back into the cluster.

If a Cisco SME cluster is not successfully created

There are three main reasons that a Cisco SME cluster may not be successfully created:

•SSH must be enabled on every switch that is part of a Cisco SME cluster.

---

Note Only SSH/dsa or SSH/rsa are supported for Cisco SME cluster configurations using Fabric Manager Web Client. SSH/rsa1 is not supported for SME cluster config via FM web client in 3.2.2 (release with SME feature). It may (or may not) be supported in future releases.

---

•If the Cisco SME switches are managed using their IP addresses (instead of host names or FQDN), the entry "sme.useIP=true" must be set in the smeserver.properties file. Be sure to restart the Fabric Manager Server after modifying the smeserver.properties file.

•The DNS server must be configured.

•Sometimes, improperly configured personal firewall software (running on Cisco Fabric Manager Server) may also cause a created SME cluster to stay in the "pending" state. Be sure to create proper firewall rules to allow necessary traffic between Fabric Manager Server/Fabric Manager web client/switches.

A Cisco SME interface does not come up in a cluster

If a Cisco SME interface does not come up, this can be due to the following:

•A Cisco SME license is not installed or the license has expired.

•An MSM-18/4 module has been replaced after the Cisco SME interface has been configured.

•The copy running-config startup-config command was not entered after adding or deleting a Cisco SME interface from a cluster or before rebooting the switch.

For the second and third scenarios, you must first remove and re-add the interface to the cluster and then enter the copy running-config startup-config command.

When selecting paths, a "no paths found" message is displayed

A tape library controller or robot can be shown as a target in the Select Tape Drives wizard. If you select the controller or robot as a target, a "no paths found" message is displayed. You will need to verify whether or not the selected target is a controller or robot.

When the "no paths found" message is displayed, enter the show tech and show tech-support sme command.

Newly added tape drives are not showing in a cluster

If you add new tape drives as LUNs to a tape library after Cisco SME has already discovered available tape drives, a rescan is required from the host to discover the new LUNs.

If you need to contact your customer support representative or Cisco TAC

At some point, you may need to contact your customer support representative or Cisco TAC for some additional assistance. Before doing so, enter the show tech details and the show tech sme commands and collect all logs from the C:\Program Files\Cisco Systems\MDS 9000\logs directory before contacting your support organization.

A syslog message is displayed when a Cisco MDS switch configured with Cisco SME in the start-up configuration boots up

When you reboot a Cisco MDS switch that has the cluster configuration stored in the startup-config file, the following syslog message may be displayed:

<timestamp> <switch name> %CLUSTER-2-CLUSTER_DB_SYNC_FAIL: Cluster <cluster-id> application 3 dataset 1 database synchronization failed, reason="Invalid cluster API registration"

This error message is expected and can be ignored.