Cisco Virtual Security Gateway for Nexus 1000V Series Switch Troubleshooting Guide, Release 4.2(1)VSG1(4.1)

Identifying the Running Processes and Their States

The show processes command identifies the running processes and the status of each process as follows:

• PID—Process ID.
• State—Process state.
• PC—Current program counter in hex format.
• Start_cnt—How many times a process has been started (or restarted).
• TTY—Terminal that controls the process. A dash (-) usually means a daemon that is not running on any particular TTY.
• Process—Name of the process.

Process states are as follows:

• D—Uninterruptible sleep (usually I/O).
• R—Runnable (on run queue).
• S—Sleeping.
• T—Traced or stopped.
• Z—Defunct zombie process.
• NR—Not-running.
• ER—Should be running but is currently not running. The ER state typically designates a process that has been restarted too many times, which causes the system to classify it as faulty and disable it.

This example shows how to identify the available options for the show processes command:

This example shows how to display the complete output from the Cisco VSG:

Displaying CPU Usage

You can use the show processes cpu command to display CPU usage. The command output includes the following information:

• Runtime(ms)—CPU time that the process has used, expressed in milliseconds
• Invoked—Number of times that the process has been invoked
• uSecs—Microseconds of CPU time as an average for each process invocation
• 1Sec—CPU usage as a percentage for the last one second

This example shows how to display all of the CPU processes:

Displaying CPU and Memory Information

You can use the show system resources command to display system-related CPU and memory statistics as follows:

• The load is defined as the number of running processes. The average reflects the system load over the past 1, 5, and 15 minutes.
• Processes display the number of processes in the system and how many processes are running when the command is issued.
• The CPU states show the CPU usage percentage in the user mode, kernel mode, and idle time in the last one second.
• The memory usage provides the total memory, used memory, free memory, memory used for buffers, and memory used for the cache in kilobytes. Buffers and cache are also included in the used memory statistics.

This example shows how to display how to display the results of available system resources:

Logging Levels

The Cisco VSG supports the following logging levels:

• 0—Emergency
• 1—Alert
• 2—Critical
• 3—Error
• 4—Warning
• 5—Notification
• 6—Informational
• 7—Debugging

By default, the switch logs normal but significant system messages to a log file and sends these messages to the system console. Users can specify which system messages are saved, based on the type of facility and the severity level. Messages are time stamped to enhance real-time debugging and management.

Enabling Logging for Telnet or SSH

System logging messages are sent to the console based on the default or configured logging facility and severity values.

Users can disable logging to the console or enable logging to a given Telnet or Secure Shell (SSH) session.

• To disable console logging, use the no logging console command in interface configuration mode.
• To enable logging for Telnet or SSH, use the terminal monitor command in EXEC mode.

Note When logging to a console session is disabled or enabled, that state is applied to all future console sessions. If you exit and log in again to a new session, the state is preserved. When logging to a Telnet or SSH session that is enabled or disabled, that state applies only to that session. The state is not preserved after you exit the session.

The no logging console command is enabled by default. Use this command to disable console logging.

The terminal monitor command is disabled by default. Use this command to enable logging for Telnet or SSH.

For more information about configuring syslog, see the Cisco Virtual Network Management Center GUI Configuration Guide.

Event Log

This section describes event logs.

This section includes the following topics:

• Event Log Configuration Format
• Viewing the Event Log Configuration
• Viewing Event Logs
• Event Log Configuration Persistence

Event Log Configuration Format

The configuration is displayed using this format:

You can configure event logs for either the inspection process or one of its modules. For example, you can use the event-log inspect error terminal command to enable error events for the inspection process and to display these messages on the terminal where the command was entered.

Viewing the Event Log Configuration

You can display the event log configuration by using the show event-log all command. This example shows how to display the event logs for all the processes and their modules:

Viewing Event Logs

Event logs are always logged in a process that is specific to the message buffer. Process logging in the event log buffer does not incur any overhead. In addition to using the show event-log command, you can display messages on a terminal where the event logs are enabled by using the terminal option, which is useful for reproducing a certain behavior.

The show command shows all the processes that are integrated with the event log Cisco VSG infrastructure. You can display inspection event logs using the show system internal event-log inspect command. The Cisco VSG event log infrastructure is a layer on top of the Cisco NX-OS event log infrastructure. Event logs can be redirected to a file and exported.

To display event logs on the terminal, use the terminal option while configuring the event. Different terminals can view different event logs. For example, use the event-log inspect ftp info terminal command to enable the information event logs for the inspection FTP module and to display the logs on the terminal. Use the event-log inspect rsh error terminal command to display only the error logs that are related to the RSH module. This command helps to debug various modules at the same time.

Event Log Configuration Persistence

You can save the event log configuration by using the event-log save config command. This command allows you to save all of the currently enabled event logs in a file. This file is read at the time of the module/process initialization with the event log infrastructure. The event log configuration that is relevant to the process is reapplied during initialization, which makes the event log configuration persistent across the process/system reboot. Some important things about the event log configuration are as follows:

• Terminal information is not reapplied for process or system restarts because that information might not be applicable.
• The event log configuration is independent of the other Cisco NX-OS configurations. The copy running-config startup-config and show running-config commands do not save and display the event log configuration.
• The event log configuration is specific to the individual system. In a high-availability setup, the configuration must be set up on both systems.

Configuration and Restrictions

Event logs CLIs for the Cisco VSG are classified based on the process and its modules. This section descrubes event log commands.

This section includes the following topics:

• VNS Agent
• Inspection Process
• Service Path Process
• Policy Engine Process
• Restrictions

VNS Agent

Virtual Network Service (VNS) agent-related event logs are maintained on the Virtual Supervisor Module (VSM), not on the Cisco VSG.

This section includes the following topics:

• Core Module
• VPath Module
• License Module

Core Module

Core events are those events that are related to port attach, port detach, Internet Protocol Database (IPDB), and to port-profile CLI.

This example shows how to enable/disable error messages for the vns_agent core module:

This example shows how to enable/disable informational messages for the vns_agent core module:

VPath Module

Because the vPath module works based on core-module events, you should always enable core module event logs before you enable the vPath module events.

This example shows how to enable/disable error messages for the vns_agent vPath module:

This example shows how to enable/disable informational messages for the vns_agent vPath module:

License Module

Because the license module works based on core-module events, you should always enable the core module event logs before enabling the license module.

This example shows how to enable/disable error messages for the vns_agent license module:

This example shows how to enable/disable informational messages for the vns_agent license module:

Inspection Process

The inspection process uses event log commands for the inspection process and the File Transfer Protocol (FTP), Remote Shell (RSH), and Trivial File Transfer Protocol (TFTP) modules. These processes are all available on the Cisco VSG.

Use the event-log inspect error command to display configuration errors, process initialization errors, and so forth. This example shows how to enable/disable error messages for the inspection process:

This example shows how to enable/disable informational messages for the inspection process:

Use the event-log inspect ftp error command to display FTP packet processing errors. This example shows how to enable/disable error messages for the inspection FTP module:

The command output is as follows:

This example shows how to enable/disable informational event log messages for the inspection FTP module:

The command output is as follows:

This example shows how to enable/disable warning messages for the inspection FTP module:

The command output is as follows:

This example shows how to enable/disable packet trace messages for the inspection FTP module:

The command output is as follows:

This example shows how to enable/disable error messages for the inspection RSH module:

This example shows how to enable/disable informational messages for the inspection RSH module:

The command output is as follows:

This example shows how to enable/disable packet trace messages for the inspection RSH module:

The command output is as follows:

This example shows how to enable/disable error messages for the inspection TFTP module:

This example shows how to enable/disable informational messages for the inspection TFTP module:

The command output is as follows:

Service Path Process

The service path processes are available on the Cisco VSG.

This section includes the following topics:

• Service Path Module
• Service Path Flow Manager
• AC Module

The service path process exposes event log output for the VSN service path, flow manager, AC infrastructure modules.

Service Path Module

The event-log service-path sp error command can display a failure to initialize the FE, and so forth. This example shows how to enable/disable error messages for the service path module:

Use the event-log service-path sp info command to display FE initialization messages, control path messages, and so forth. This example shows how to enable/disable informational messages for the service path module:

The event-log service-path sp pkt-error command can display failures to read or write a packet, a corrupted packet, and so forth.

This example shows how to enable/disable packet error messages for the service path module:

The event-log service-path sp pkt-info command can display the field description of a packet, where the packet arrived from or going to, decisions taken on the packet, and so forth.

This example shows how to enable/disable packet informational messages for the service path module:

The event-log service-path sp pkt-detail command can display the first few 100 bytes of the incoming packets.

This example shows how to enable/disable detailed packet messages for the service path module:

Service Path Flow Manager

This example shows how to enable/disable the packet messages for the service path flow manager module:

AC Module

The event-log service-path ac error command can display failures to initialize the AC, timer, FD, pending queue, and so forth.

This example shows how to enable/disable error messages for the AC module:

The event-log service-path ac info command can display AC initialization messages, control path messages, and so forth.

This example shows how to enable/disable informational messages for the AC module:

Policy Engine Process

The policy engine processes are available on the Cisco VSG.

This section includes the following topic:

• Attribute Manager Module

Attribute Manager Module

This section describes the attribute manager-related errors.

You can use the event-log policy-engine attr-mgr error command to display the policy ID for a PE evaluation lookup based on the VNSP ID, IP address, zone name resolution, attribute fetched, and so forth.

This example shows how to enable/disable error messages for the attribute manager module:

This example shows how to enable/disable informational messages for the attribute manager module:

Restrictions

The event log configuration has the following restrictions:

• Terminal information is not reapplied in case of process restart/system restart bacause it may or may not be applicable.
• Event log configuration is independent of the other Cisco NX-OS configurations. The Cisco NX-OS copy running-config startup-config and show running-config commands do not save and display event log configuration.
• Event log configuration is specific to the individual system. In the high availability (HA) setup, this configuration must be done on both of the systems.

VSM Show Commands

This section includes the following topics:

• show vnm-pa status
• show vservice node mac brief
• show vservice node detail
• show vservice port brief
• show vservice connection
• show vservice statistics [vlan vlan-num ip ip-addr] [module module-num]
• clear vservice statistics [vlan vlan-num ip ip-addr] [module module-num]

show vnm-pa status

You can display the VNM policy agent status by entering the show vnm-pa status command.

This example shows how to display the VNM policy agent installation status:

show vservice node mac brief

You can display a consolidated view of all VSNs in use by using the show vservice node mac brief command.

This example shows how to display all VSNs in use:

The MAC-ADDR column lists the MAC address of the data0 interface that corresponds to that Cisco VSG (if the VEM can resolve it). If the VEM does not resolve the MAC address, it cannot redirect packets to the Cisco VSG. If a valid MAC address is not shown, check if the Cisco VSG data0 is reachable from the VEM. If there is no valid MAC-ADDR, the possible reasons are as follows:

• The data0 interface on the Cisco VSG is not configured
• The VLAN is not up
• A mismatch has occurred in the VLAN specified in the vn-service command and the port profile used for the Cisco VSG VM.

FAIL specifies the behavior when the Virtual Ethernet Module (VEM) has no connectivity to the Cisco VSG. The default is Close (packets are dropped). Open means packets are forwarded.

The STATE can be Alive, Unreach, or No Licenses. If Unreach, the MAC-ADDR is not resolved or the module is not up. If multiple VEMs inherit the same VM port profile, those interfaces must pass all checks before the state can be Alive. If No Licenses appears, install the Cisco VSG license on the VSM.

The MODULE column lists the VEM numbers whose interfaces have inherited this configuration.

show vservice node detail

You can display detailed information for all VSNs in use by using the show vservice node detail command. Information is displayed for each of the associated VEMs. The command output displays the port profile, security profile, organization, and list of Cisco Nexus 1000V ports that have inherited this configuration. Also displayed are any configuration mismatches between the VSM and VEM missing ports for a given port profile, all ports of a port profile that are not configured with the same security profile, and so forth.

This example shows how to display all VSNs in use:

show vservice port brief

You can display information for each virtual Ethernet (vEth) interface by using the show vservice port brief command. By default, all attached vEths are listed. Use the vethernet option for output of a specific vEth interface.

This example shows how to display the vEth interfaces:

Ensure that the VM Name value matches the name of the VM associated with this vNIC. For VSN Data IP in the brackets in the Node, Profile Name, and Org values, ensure that correct values for this VM are displayed. The Profile ID value should never be zero. For IP addresses, ensure that the list of IP addresses matches the IP addresses configured for the specific vNIC for that VM. If not, use the vemcmd show learnt command on all VEMs to display the Internet Protocol Database (IPDB) table.

show vservice connection

You can display VSN connections by using the show vservice connection command.

This example shows how to display VSN connections:

show vservice statistics [vlan vlan-num ip ip-addr] [module module-num]

You can display VSN statistics by using the show vservice statistics command.

This example shows how to display the VSN statistics:

clear vservice statistics [vlan vlan-num ip ip-addr] [module module-num]

You can clear the VSN statistics by using the clear vservice statistics command.

This example shows how to clear VSN statistics:

Cisco VSG show Commands

The attribute manager maintains a set of tables and does a lookup that is based on the fields in the packet. There are three main tables: DV port table, VM table, and VNSP table. Use the show vsg dvport command to display runtime information for the DV port table. For the other two tables, use the show vsg vm and show vsg vnsp commands.

Hash tables are maintained based on IP addresses (IP address to DV port entry) and VNSP ID (VNSP ID to VNSP entry). An IP address is used when fetching attributes (custom and VM attributes) that are based on the source or destination IP address. It is also used to determine which policy set to evaluate for a given traffic type. The VNSP ID is used (valid VNSP ID in the packet header) to determine which policy set to evaluate. Custom attributes can also be fetched.

This section includes the following topics:

• show vnm-pa status
• show service-path statistics
• clear service-path statistics
• show service-path connection
• clear service-path connection
• show vsg ip-binding
• show vsg dvport {dvport_id}
• show vsg vm name {name}
• show vsg security-profile {[vnsp-name ]| detail | table}
• show vsg zone
• clear policy-engine
• show ac-driver statistics
• clear ac-driver statistics
• show system internal ac ipc-stats fe [process-name]
• clear system internal ac ipc-stats fe [process-name]
• show inspect ftp statistics
• clear inspect ftp statistics

show vnm-pa status

Enter the show vnm-pa status command to display the Cisco VNMC policy agent status.

This example shows how to display the Cisco VNMC policy agent status:

show service-path statistics

You can display the following statistics that pertain to one vPath by using the show service-path statistics command:

• The packets seen by the service path from the vPath.
• Flows created by the service path due to these packets.
• Packets dropped in the service path due to various errors.

Note If no module is given, the command displays the aggregate statistics of all the modules in the given SVS domain.

This command provides the following keyword filters:

• svs-domain-id domain-id—Displays only the Cisco VSG connections that are associated to the svs-domain specified in the domain-id.
• module module-num—Displays only the Cisco VSG connections that are associated to the svs-domain and VEM module specified in the domain-id and the module-num. Use this keyword filter only with the svs-domain-id filter.

This example shows how to display the statistics using the svs-domain-id keyword filter:

clear service-path statistics

You can clear the service path statistics globally by using the clear service-path statistics command when no option is given. When the SVS domain ID and the module are provided, entering the command clears the statistics of the specified module.

This command provides the following keyword filters:

• svs-domain-id domain-id—Displays only the Cisco VSG connections that are associated to the svs-domain specified in the domain-id.
• module module-num—Displays only the Cisco VSG connections that are associated to the svs-domain and VEM module specified in the domain-id and the module-num. Use this only with the svs-domain-id filter.

This example shows how to clear the service path statistics:

show service-path connection

You can display the connections (flow-table) maintained in the Cisco VSG by using the show service-path connection command. These connections are provided per VEM module per SVS domain.

This command provides the following keyword filters:

• svs-domain-id domain-id—Displays only the Cisco VSG connections that are associated to the svs-domain specified in the domain-id.
• module module-num—Displays only the Cisco VSG connections that are associated to the svs-domain and VEM module specified in the domain-id and the module-num. Use this keyword filter only with the svs-domain-id keyword filter.

This example shows how to display the connections in the Cisco VSG:

clear service-path connection

You can clear the connections (flow-table) maintained in the Cisco VSG by using the clear service-path connection command.

This example shows how to clear the flow-table connection output:

show vsg ip-binding

You can display a list of VM IP addresses and associated Virtual Network Service Profiles (VNSPs) with the associated policy set by using the show vsg ip-binding command. This information helps you to troubleshoot data path issues. The attribute manager determines which policy set to evaluate for a given packet (source IP address is the key for the lookup).

When debugging issues (for example, the wrong policy set or no policy), use this command to ensure that IP bindings (IP address to VNSP association) are correct. This association can also affect VNSP and VM attributes fetched by the attribute manager.

This example shows how to display the list of VM IP addresses and associated VNSPs:

You can use the show vsg ip-binding vm command to show all IP VM bindings.

This example shows how to display all IP VM bindings:

You can use the show vsg ip-binding vm detail command to see more details about the IP VM bindings.

This example shows how to display the details of the IP VM bindings:

show vsg dvport {dvport_id}

You can display relevant information for a DV port by using the show vsg dvport command. A DV port is a logical representation of a vNIC. By default, this displays information for all DV ports. Specify a particular DV port with the dvport id parameter.

This example shows how to display the DV port information:

show vsg vm

You can display information for all VMs on a VSG by using the show vsg vm command. In addition to the existing VM information, zone names are displayed in the command output. DV port information is not display to limit verbosity.

This example shows how to display all VMs on a VSG:

show vsg vm name {name}

You can display information of one or more VMs by entering the show vsg vm name command. The VM name should be specified as a parameter and can be the name or the first few characters of the name. The information for the VM includes the details of each DV port used by the VM and zones that the VM belongs to.

This example shows how to display information for the VM that has a name that starts with linux-204:

show vsg vm uuid {vm_uuid}

You can display relevant information for a particular VM by using the show vsg vmuuid command. The attribute manager looks up the VM attributes for the VM based on this association before doing a policy evaluation.

When debugging issues, such as when the wrong VM attributes are fetched, check the output of this command as well as the IP address to DV port mapping.

This example shows how to display the relevant information for a VM:

show vsg security-profile {[vnsp-name ]| detail | table}

You can display information for a specific VNSP or all VNSPs by using the show vsg security-profile command. The attribute manager looks up custom attributes for a particular VNSP that is based on this association before doing a policy evaluation. By default, information is displayed for all VNSPs. You can specify a particular VNSP by using the vnsp-name argument.

When debugging issues such as the wrong policy set are evaluated, check if the correct policy set is associated with the VNSP. If custom attribute values are not correct, this command displays some details.

The detail version of this command includes names of the VMs that are using the security-profile in addition to their security-profile information. A VNSP name can be specified to get details of a specific security-profile.

This example shows how to display detailed information about a specific Cisco VSG security profile with the name sp_deny@root:

You can display the associated VNSP ID and policy for all VNSPs by using the show vsg security-profile command. The attribute manager uses this association when looking up a VNSP and associated policy from the packet that reaches the data0 interface of the Cisco VSG. When VPath redirects the packets to the Cisco VSG, the VNSP ID is added in the packet header.

This example shows how to display brief tabular information for the Cisco VSG security profile:

show vsg zone

You can display VM to zone mappings on a Cisco VSG by using the show vsg zone command.

This example shows how to display the VM to zone mappings on a Cisco VSG:

show policy-engine stats

You can display statistics on the policy engine by using the show policy-engine stats command.

This example shows how to display the statistics for the Cisco VSG policy engine:

This example shows how to use the help (?) feature of the command to display command options:

clear policy-engine

You can clear the policy-engine statistics by using the clear policy-engine command.

This example shows how to see the options for clearing the policy-engine statistics:

When the stats argument is used, the statistics are cleared and the only response for a successful action is a return to the prompt. This example shows how to clear the policy engine statistics:

show ac-driver statistics

You can display statistics that are collected in the AC driver module by using the show ac-driver statistics command. These statistics indicate how many packets are received, how many of those received packets are from vPath, how many packets are passed up to the service path, how many packets are passed as a response to the vPath and any error statistics, and so on.

This example shows how to display the AC driver module statistics:

clear ac-driver statistics

You can clear the statistics that are collected in the AC driver module by using the clear ac-driver statistics command.

This example shows how to clear the statistics collected in the AC driver module:

show system internal ac ipc-stats fe [process-name]

You can display internal statistics of the following processes by using the show system internal ac ipc-stats fe command:

• attribute-manager
• inspection-ftp
• inspection-rsh
• inspection-tftp
• service-path

This example shows how to display the statistics for the inspection-ftp process:

clear system internal ac ipc-stats fe [process-name]

You can clear the internal statistics for the following processes by using the clear system internal ac ipc-stats fe command:

• attribute-manager
• inspection-ftp
• inspection-rsh
• inspection-tftp
• service-path

This example shows how to clear the statistics for the inspection-ftp process:

show inspect ftp statistics

You can display the following inspect FTP statistics pertaining to one vPath by using the show inspect ftp statistics command:

• The packets seen by the inspect FTP path from the vPath.
• Flows created by the inspect FTP path due to these packets.
• Packets dropped in the inspect FTP path due to various errors.

This example shows how to display the FTP statistics:

clear inspect ftp statistics

Use the clear inspect ftp statistics command to clear the inspect FTP statistics globally when no option is given. When the SVS domain ID and the module are provided, the command clears the statistics of the specified module.

This command provides the following keyword filters:

• svs-domain-id domain-id—Displays only the Cisco VSG connections that are associated to the svs-domain specified in the domain-id.
• module module-num—Displays only the Cisco VSG connections that are associated to the SVS domain and VEM module specified in the domain-id and the module-num. Use this keyword filter only with the svs-domain-id filter.

This example shows how to clear the inspect FTP statistics: