Cisco Virtual Security Gateway for Nexus 1000V Series Switch Troubleshooting Guide, Release 4.2(1)VSG1(2)
Chapter 7 - Troubleshooting High Availability Issues
Downloads: This chapterpdf (PDF - 160.0KB) The complete bookPDF (PDF - 1.84MB) | Feedback

Troubleshooting High Availability Issues

Table Of Contents

Troubleshooting High Availability Issues

Information About Cisco VSG High Availability

Redundancy

Isolation of Processes

Cisco VSG Failovers

Problems with High Availability

High Availability Troubleshooting Commands

Checking the Domain ID of the Cisco VSG

Checking Redundancy

Checking the System Redundancy Status

Checking the System Internal Redundancy Status

Checking the System Manager State

Reloading a Module

Attaching to the Standby Cisco VSG Console

Checking for the Event History Errors

Standby Synchronization

Synchronization Fails


Troubleshooting High Availability Issues


This chapter describes how to troubleshoot issues related to high availability (HA).

This chapter includes the following sections:

Information About Cisco VSG High Availability

Problems with High Availability

High Availability Troubleshooting Commands

Standby Synchronization

Standby Synchronization

Information About Cisco VSG High Availability

Cisco VSG high availability (HA) is a subset of the Cisco NX-OS HA. The following HA features minimize or prevent traffic disruption in the event of a failure:

Redundancy

Isolation of Processes

Cisco VSG Failovers

Redundancy

Cisco VSG redundancy is equivalent to HA pairing. The two possible redundancy states are active and standby. An active Cisco VSG is always paired with a standby Cisco VSG. HA pairing is based on the Cisco VSG ID. Two Cisco VSGs that are assigned an identical ID are automatically paired. All processes running in the Cisco VSG are data path critical. If one process fails in an active Cisco VSG, a failover to the standby Cisco VSG occurs instantly and automatically.

Isolation of Processes

The Cisco VSG software contains independent processes known as services. These services perform a function or set of functions for a subsystem or feature set of the Cisco VSG. Each service and service instance runs as an independent, protected process. This operational process ensures a highly fault-tolerant software infrastructure and fault isolation between services. Any failure in a service instance does not affect any other services running at that time. Additionally, each instance of a service runs as an independent process. For example, two instances of a routing protocol run as separate processes.

Cisco VSG Failovers

The Cisco VSG HA pair configuration allows uninterrupted traffic forwarding by using stateful failovers when a failure occurs.

Problems with High Availability

The following key problems are found in Cisco VSG HA. In addition to these issues, some of the common Cisco NX-OS HA symptoms are listed in Table 7-1. The symptoms that are related to high availability, their possible causes, and recommended solutions are as follows.

Cisco VSG pair communication problems

Cisco VSGs do not reach an active/standby status

Sometimes, the Cisco VSG reboots continuously when tenants share the management network (for example, collisions of the domain ID):

In a multitenant environment, if there is a shared management network and if a collision occurs in the domain ID (two or more tenants using the same domain ID), that triggers spontaneous reboots of the Cisco VSGs that are involved in the collision. There is no workaround for this issue. When domain IDs are provisioned, they must be unique across all tenants

Cisco VSGs in the HA pair get stuck in bash# prompt mode during reboots/upgrades/switchovers

Cisco VSGs in the HA pair get stuck in boot# prompt mode during reboots/upgrades/switchovers

Table 7-1 Problems with High Availability  

Symptom
Possible Cause
Solution

The active Cisco VSG does not see the standby Cisco VSG.

Roles are not configured properly:

primary

secondary

Do the following:

1. Verify the role of each Cisco VSG by entering the show system redundancy status command.

2. Update an incorrect role by entering the system redundancy role command.

3. Save the configuration by entering the copy run start command.

Network connectivity problems are occurring between the Cisco VSG and the upstream and virtual switches. The problem could be in the control or management VLAN.

Restore connectivity as follows:

1. From the vSphere client, shut down the Cisco VSG, which should be in standby mode.

2. From the vSphere client, bring up the standby Cisco VSG after network connectivity is restored.

The active Cisco VSG does not complete synchronization with the standby Cisco VSG.

A version mismatch between Cisco VSGs might be occurring.

Do the following:

1. Verify the software version on both Cisco VSGs by entering the show version command.

2. Reinstall the secondary Cisco VSG with the same version used in the primary.

Fatal errors occur during the gsync process.

Check the gsyncctrl log by entering the show system internal log sysmgr gsyncctrl command and look for fatal errors.

Reload the standby Cisco VSG by entering the reload module standby_module_number command.

See the "Reloading a Module" section.

The standby Cisco VSG reboots periodically.

The Cisco VSG has connectivity only through the management interface.

When a Cisco VSG is able to communicate through the management interface, but not through the control interface, the active Cisco VSG resets the standby to prevent the two Cisco VSGs from being in HA mode and out of sync.

Check control VLAN connectivity between the primary and secondary Cisco VSG by entering the show system internal redundancy info command.

In the output, degraded_mode flag = true.

If there is no connectivity, restore it through the control interface.

Both Cisco VSGs are in active mode.

The following network connectivity problems might be occurring:

Check for control and management VLAN connectivity between the Cisco VSG at the upstream and virtual switches.

When the Cisco VSG cannot communicate through any of these two interfaces, they both try to become active.

If network problems exist, do the following:

1. From the vSphere client, shut down the Cisco VSG, which should be in standby mode.

2. From the vSphere client, bring up the standby Cisco VSG after network connectivity is restored.

Active and standby Cisco VSGs are not synchronized.

Incompatible versions

The boot variables for active and standby Cisco VSGs are set to different image names, or if image names are the same, the files are not the correct files.

When active and standby Cisco VSGs are running different versions that are not HA compatible, they are unable to synchronize.

Update the software version or the boot variables as follows:

1. From each Cisco VSG (active and standby), verify the software version by entering the show version command.

2. Reload the standby Cisco VSG with the version that is running in the active by doing one of the following:

Correcting the boot variable names

Replacing the incorrect software files

See the "Reloading a Module" section.

Broadcast traffic problem

The broadcast traffic from the standby to the active Cisco VSG might prevent the Cisco VSGs from synchronizing. The standby Cisco VSG tries to contact the active Cisco VSG periodically, but if broadcast traffic problems persist for over a minute when the standby is booting up, the system cannot synchronize.

Fix the traffic problem and reload the standby Cisco VSG as follows.

1. From the standby Cisco VSG, verify the broadcast traffic problem by entering the show system internal log sysmgr verctrl command.

The following message appears:

standby_verctrl: no response from the active System Manager

2. Fix network connectivity.

3. Reload the standby Cisco VSG by using the reload module standby_module_number command.

See the "Reloading a Module" section.

False standby removal

The active Cisco VSG falsely detects a disconnect with the standby. The standby is removed and reinserted and synchronization does not occur.

Verify redundancy states and reload the standby Cisco VSG as follows:

1. Verify active Cisco VSG redundancy by using the show system internal redundancy status command. The output is as follows:

RDN_DRV_ST_AC_NP 

2. Verify the standby Cisco VSG redundancy by using the show system internal redundancy status command. The output is as follows:

RDN_DRV_ST_SB_AC 

3. Reload the standby Cisco VSG by using the reload module standby_module_number command.

See the "Reloading a Module" section.


High Availability Troubleshooting Commands

This section lists commands that you can use to troubleshoot problems related to high availability.

This section includes the following topics:

Checking the Domain ID of the Cisco VSG

Checking Redundancy

Checking the System Manager State

Reloading a Module

Attaching to the Standby Cisco VSG Console

Checking the Domain ID of the Cisco VSG

You can display the domain ID information by entering the show vsg command.

This example shows how to display the domain ID information:

vsg# show vsg
Model: VSG
HA ID: 3000
VSG Software Version: 4.2(1)VSG1(1) build [4.2(1)VSG1(1)]
VNMC IP: 10.193.75.145
vsg#

Checking Redundancy

This section includes the following topics:

Checking the System Redundancy Status

Checking the System Internal Redundancy Status

Checking the System Redundancy Status

You can check the system redundancy status by entering the show system redundancy status command.

This example shows how to display the system redundancy status:

vsg# show system redundancy status 
Redundancy role
---------------
      administrative:   primary <-- Configured redundancy role
         operational:   primary <-- Current operational redundancy role
 
   
Redundancy mode
---------------
      administrative:   HA
         operational:   HA
 
   
This supervisor (sup-1)
-----------------------
    Redundancy state:   Active <-- Redundancy state of this VSG
    Supervisor state:   Active
      Internal state:   Active with HA standby                  
 
   
Other supervisor (sup-2)
------------------------
    Redundancy state:   Standby <-- Redundancy state of the other VSG
    Supervisor state:   HA standby
      Internal state:   HA standby <-- The standby VSG is in HA mode and in sync
vsg#

Checking the System Internal Redundancy Status

You can check the system internal redundancy status by entering the show system internal redundancy info command.

This example shows how to display the system internal redundancy status information:

vsg# show system internal redundancy info 
My CP:
  slot: 0
  domain: 184 <-- Domain id used by this VSG
  role:   primary <-- Redundancy role of this VSG
  status: RDN_ST_AC <-- Indicates redundancy state (RDN_ST) of the this VSG is Active (AC)
  state:  RDN_DRV_ST_AC_SB
  intr:   enabled
  power_off_reqs: 0
  reset_reqs:     0
Other CP:
  slot: 1
  status: RDN_ST_SB <-- Indicates redundancy state (RDN_ST) of the other VSG is Standby 
(SB)
  active: true
  ver_rcvd: true
  degraded_mode: false <-- When true, it indicates that communication through the control 
interface is faulty
Redun Device 0: <-- This device maps to the control interface
  name: ha0
  pdev: ad7b6c60
  alarm: false
  mac: 00:50:56:b7:4b:59
  tx_set_ver_req_pkts:   11590
  tx_set_ver_rsp_pkts:   4
  tx_heartbeat_req_pkts: 442571
  tx_heartbeat_rsp_pkts: 6
  rx_set_ver_req_pkts:   4
  rx_set_ver_rsp_pkts:   1
  rx_heartbeat_req_pkts: 6
  rx_heartbeat_rsp_pkts: 442546 <-- Counter should be increasing, as this indicates that 
communication between VSG is working properly.
  rx_drops_wrong_domain: 0
  rx_drops_wrong_slot:   0
  rx_drops_short_pkt:    0
  rx_drops_queue_full:   0
  rx_drops_inactive_cp:  0
  rx_drops_bad_src:      0
  rx_drops_not_ready:    0
  rx_unknown_pkts:       0
Redun Device 1: <-- This device maps to the mgmt interface
  name: ha1
  pdev: ad7b6860
  alarm: true
  mac: ff:ff:ff:ff:ff:ff
  tx_set_ver_req_pkts:   11589
  tx_set_ver_rsp_pkts:   0
  tx_heartbeat_req_pkts: 12
  tx_heartbeat_rsp_pkts: 0
  rx_set_ver_req_pkts:   0
  rx_set_ver_rsp_pkts:   0
  rx_heartbeat_req_pkts: 0
  rx_heartbeat_rsp_pkts: 0 <-- When communication between VSG through the control 
interface is interrupted but continues through the mgmt interface, the 
rx_heartbeat_rsp_pkts will increase.
  rx_drops_wrong_domain: 0
  rx_drops_wrong_slot:   0
  rx_drops_short_pkt:    0
  rx_drops_queue_full:   0
  rx_drops_inactive_cp:  0
  rx_drops_bad_src:      0
  rx_drops_not_ready:    0
  rx_unknown_pkts:       0
vsg#

Checking the System Manager State

You can check the system internal sysmgr state by entering the show system internal sysmgr state command.

This example shows how to display the system internal sysmgr state information:

vsg# show system internal sysmgr state 
 
   
The master System Manager has PID 1988 and UUID 0x1.
Last time System Manager was gracefully shutdown.
The state is SRV_STATE_MASTER_ACTIVE_HOTSTDBY entered at time Tue Apr 28 13:09:13 2009.
 
   
The '-b' option (disable heartbeat) is currently disabled.
 
   
The '-n' (don't use rlimit) option is currently disabled.
 
   
Hap-reset is currently enabled.
 
   
Watchdog checking is currently disabled.
 
   
Watchdog kgdb setting is currently enabled.
 
   
 
   
        Debugging info:
 
   
The trace mask is 0x00000000, the syslog priority enabled is 3.
The '-d' option is currently disabled.
The statistics generation is currently enabled.
 
   
 
   
        HA info:
 
   
slotid = 1    supid = 0
cardstate = SYSMGR_CARDSTATE_ACTIVE.
cardstate = SYSMGR_CARDSTATE_ACTIVE (hot switchover is configured enabled).
Configured to use the real platform manager.
Configured to use the real redundancy driver.
Redundancy register: this_sup = RDN_ST_AC, other_sup = RDN_ST_SB.
EOBC device name: eth0.
Remote addresses:  MTS - 0x00000201/3      IP - 127.1.1.2
MSYNC done.
Remote MSYNC not done.
Module online notification received.
Local super-state is: SYSMGR_SUPERSTATE_STABLE
Standby super-state is: SYSMGR_SUPERSTATE_STABLE
Swover Reason : SYSMGR_SUP_REMOVED_SWOVER <-- Reason for the last switchover
Total number of Switchovers: 0 <-- Number of switchovers 
								>> Duration of the switchover would be listed, if any.
 
   
        Statistics:
 
   
Message count:           0
Total latency:           0              Max latency:             0
Total exec:              0              Max exec:                0
vsg#

Reloading a Module

You can reload a module by entering the reload module command.


Note Using the reload command without specifying a module reloads the whole system.


This example shows the output for the command:

vsg# reload module 2
This command will reboot the system (y/n)? y
vsg# 

Attaching to the Standby Cisco VSG Console

The standby Cisco VSG console is not accessible externally. You can access the standby Cisco VSG console through the active Cisco VSG by entering the attach module module-number command.

This example shows how to access the standby Cisco VSG through the active Cisco VSG:

vsg# attach module 2
Attaching to module 2...
To exit type 'exit', to abort type '$.' 
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
vsg#

Checking for the Event History Errors

You can check for errors in the event history by entering the show system internal sysmgr event-history errors command.

This example shows how to display errors that have been logged in the event-history:

vsg# show system internal sysmgr event-history errors 
Event:E_DEBUG, length:122, at 370850 usecs after Thu Feb  3 09:45:28 2011
[101] sysmgr_sdb_set_standby_state: Setting standby super state in sdb for vdc 1 to 
SYSMGR_SUPERSTATE_STABLE, returned
0x0
 
   
Event:E_DEBUG, length:73, at 408277 usecs after Thu Feb  3 09:44:52 2011
[101] active_gsyncctrl_info_parse: UUID 0xB6 in vdc 1 service not running
 
   
Event:E_DEBUG, length:73, at 593428 usecs after Thu Feb  3 09:44:49 2011
[101] active_gsyncctrl_info_parse: UUID 0xE0 in vdc 1 service not running
 
   
Event:E_DEBUG, length:80, at 624613 usecs after Thu Feb  3 09:44:40 2011
[101] process_plugin_load_complete_msg: Start done rcvd for all plugins in vdc 1
 
   
Event:E_DEBUG, length:89, at 624611 usecs after Thu Feb  3 09:44:40 2011
[101] process_plugin_load_complete_msg: Received plugin start done for plugin 1 for vdc 1
 
   
Event:E_DEBUG, length:99, at 518152 usecs after Thu Feb  3 09:44:39 2011
[101] perform_bootup_plugin_manager_interactions: all bootup plugins have now been loaded 
in vdc: 1
 
   
Event:E_DEBUG, length:79, at 518150 usecs after Thu Feb  3 09:44:39 2011
[101] perform_bootup_plugin_manager_interactions:incrementing number of plugins
 
   
Event:E_DEBUG, length:118, at 518020 usecs after Thu Feb  3 09:44:39 2011
[101] perform_bootup_plugin_manager_interactions: plugin has been loaded in vdc 1 - 
sending response to Plugin Manager
 
   
Event:E_DEBUG, length:58, at 631599 usecs after Thu Feb  3 09:44:38 2011
[101] process_reparse_request: on vdc 1, plugin start rcvd
vsg#

Standby Synchronization

This section includes the following topic:

Synchronization Fails

Synchronization Fails

If the standby Cisco VSG is stuck in the synchronization stage, follow these steps on the active Cisco VSG:


Step 1 Enter the show system internal sysmgr state command and check for a line similar to the following:

Gsync in progress for uuid: xxxx 

If this entry is present and shows the same xxxx value for a long time, the system has trouble synchronizing the state for one of the processes.

Step 2 Identify the process by entering the show system internal sysmgr service running | grep xxxx command.

This message appears in the first few lines of the output:
 
BL-bash# show system internal sysmgr state

The master System Manager has PID 1350 and UUID 0x1.
Last time System Manager was gracefully shutdown.
Gsync in progress for uuid: 0x18
The state is SRV_STATE_MASTER_ACTIVE_HOTSTDBY entered at time Mon Feb 21 17:56:3
9 2011.

The '-b' option (disable heartbeat) is currently disabled.
...

If the synchronization for each process occurs quickly, you might not have the chance to see the line (you might have to enter the command repeatedly as the standby Cisco VSG). If gsync for a particular process gets stuck, the line stays in the output for a while.

 
   

Step 3 If you are able to login to the console of the standby Cisco VSG (you might need to press Ctrl-C after giving the password), check the output of these two commands:

show system internal sysmgr state

show system internal sysmgr service running | grep xxxx
where xxxx is from the line "Gsync in progress for uuid: xxxx" (found by using the show system internal sysmgr state command)

Step 4 If access to the system is available only after the standby server has booted up and synchronized with the active server, use the following commands:

Enter the show system internal sysmgr bootupstats command and look for processes that took much longer than the rest, in the time that the system took to boot.

On the standby console, enter the show system internal sysmgr gsyncstats command and look for processes with large Gsync time values.