The following are the basic steps for troubleshooting:
Step 1 Gather information that defines the specific symptoms.
Step 2 Identify all potential problems that could be causing the symptoms.
Step 3 Systematically eliminate each potential problem (from most likely to least likely) until the symptoms disappear.
To identify the possible problems, you need to use a variety of tools and understand the overall configuration.The following chapters in this guide describe many approaches and specific solutions to potential problems.
Troubleshooting a Switch Crash
When a switch crashes, the cause might be from the failure of a process, and results in a reload of the switch.
A crash is usually recorded with a core file on the switch and includes the reason for the crash, such as a failed process. The following can help you determine the cause of the crash:
show system reset-reason
commands to display the reason for the crash.
switch# show system reset-reason
Please look at Note Details
1) At 4054 usecs after Sat Nov 6 15:15:01 2010
Reason: Reset triggered due to HA policy of Reset
Service: clis hap reset
2) At 841383 usecs after Sat Nov 6 14:56:25 2010
Reason: Reset triggered due to HA policy of Reset
Service: clis hap reset
command to determine if a core file was recorded. You also can use the
show process log
command to display the processes and if a core was created.
switch#show process log
Process PID Normal-exit Stack Core Log-create-time
Redirecting output of the show tech-support details command
command to redirect the output of the
show tech-support details
command to a file and then gzip the file.
The file is stored on bootflash://
provided that there is enough memory available. If you do not specify a filename, NX-OS creates the file as volatile:show_tech_out.gz. Copy the file from the device using the procedure in the copy command section.
switch# dir volatile:
374382 Aug 16 17:15:55 2010 show_tech_out.gz
From volatile, copy the file to the bootflash, FTP, or TFTP server.
switch# copy volatile:show_tech_out.gz ?
bootflash: Select destination filesystem
debug: Select destination filesystem
ftp: Select destination filesystem
log: Select destination filesystem
modflash: Select destination filesystem
nvram: Select destination filesystem
running-config Copy from source to running configuration
scp: Select destination filesystem
sftp: Select destination filesystem
startup-config Copy from source to startup configuration
system: Select destination filesystem
tftp: Select destination filesystem
volatile: Select destination filesystem
NX-OS command listing
switch# show cli list | include ?
-i Ignore case difference when comparing strings
-x Print only lines where the match is a whole line
WORD Search for the expression
switch# show cli list | include debug | include interface
Narrowing scope of keywords
You can use many commands like
to narrow the scope of a keyword.
switch(config-if)# show interface | grep fc
fc2/1 is trunking
fc2/2 is trunking
fc2/3 is up
fc2/4 is down (Administratively down)
vfc29 is up
You can use logging through the CLI or Device Manager. In the following examples, the
command and the Device Manager display severity information:
Name - ciscolive09: Severity - debugging Size - 4194304
Viewing Severity Levels in the Device Manager
Ethanalyzer and SPAN
Ethanalyzer is a tool that collects frames that are destined to, or originate from, the Nexus 5000 control plane. Node to switch or switch to switch traffic can be seen with this tool.
SPAN is a feature whereby frames that are transient to the switch are copied to a second port for analysis. Node to switch or node to node traffic can be seen via this method.
Ethanalyzer is a Cisco NX-OS protocol analyzer tool based on the Wireshark open source code. This tool is a command-line version of Wireshark that captures and decodes packets. You can use Ethanalyzer to troubleshoot your network and analyze the control-plane traffic.
ethanalyzer local sniff-interface
Captures packets sent or received by the supervisor and provides detailed protocol information.
ethanalyzer local sniff-interface brief
Captures packets sent or received by the supervisor and provides a summary of protocol information.
ethanalyzer local sniff-interface limit-captured-frames
Limits the number of frames to capture.
ethanalyzer local sniff-interface limit-frame-size
Limits the length of the frame to capture.
ethanalyzer local sniff-interface capture-filter
Filters the types of packets to capture.
ethanalyzer local sniff-interface display-filter
Filters the types of captured packets to display.
ethanalyzer local sniff-interface decode-internal
Decodes the internal frame header for Cisco NX-OS.
Note Do not use this option if you plan to analyze the data using Wireshark instead of NX-OS Ethanalyzer.
ethanalyzer local sniff-interface write
Saves the captured data to a file.
ethanalyzer local sniff-interface read
Opens the captured data file and analyzes it.
switch# ethanalyzer local sniff-interface
No matches in current mode, matching in (exec) mode
inbound-hi Inbound(high priority) interface
inbound-low Inbound(low priority) interface
mgmt Management interface
switch# ethanalyzer local sniff-interface mgmt brief
The following example is for viewing the Spanning Tree Protocol (STP) and Fibre Channel: Using 0 in the command captures output until you press
. The FCID is a well-known name for switch domain controller.
switch# ethanalyzer local sniff-interface inbound-hi brief limit-captured-frames 0
2008-08-13 01:37:23.229927 00:0d:ec:6b:cd:40 -> fc:fc:fc:ff:ff:fd 4 0 ff.ff.fd -> ff.ff.fd 0x5384 0x2b3f FC Link Ctl, ACK1
switch# ethanalyzer local sniff-interface inbound-hi limit-captured-frames 0
Capturing on eth4
Frame 1 (57 bytes on wire, 57 bytes captured)
Arrival Time: Aug 13, 2008 01:41:32.631969000
[Time delta from previous captured frame: 1218591692.631969000 seconds]
[Time delta from previous displayed frame: 1218591692.631969000 seconds]
[Time since reference or first frame: 1218591692.631969000 seconds]
Frame Number: 1
Frame Length: 57 bytes
Capture Length: 57 bytes
[Frame is marked: False]
[Protocols in frame: eth:vlan:llc:stp]
DSAP: Spanning Tree BPDU (0x42)
IG Bit: Individual
SSAP: Spanning Tree BPDU (0x42)
CR Bit: Command
Control field: U, func=UI (0x03)
000. 00.. = Command: Unnumbered Information (0x00)
.... ..11 = Frame type: Unnumbered frame (0x03)
The Switched Port Analyzer (SPAN) feature—sometimes called port mirroring or port monitoring—selects network traffic for analysis by a network analyzer. The network analyzer can be a Cisco SwitchProbe, a Fibre Channel Analyzer, or other Remote Monitoring (RMON) probes.
SPAN sources refer to the interfaces from which traffic can be monitored. The Cisco Nexus 5000 Series switch supports Ethernet, virtual Ethernet, Fibre Channel, virtual Fibre Channel, port channels, SAN port channels, VLANs, and VSANs as SPAN sources. With VLANs or VSANs, all supported interfaces in the specified VLAN or VSAN are included as SPAN sources. You can choose the SPAN traffic in the ingress direction, the egress direction, or both directions for Ethernet, virtual Ethernet, Fibre Channel, and virtual Fibre Channel source interfaces:
Ingress source (Rx)—Traffic entering the switch through this source port is copied to the SPAN destination port.
Egress source (Tx)—Traffic exiting the switch through this source port is copied to the SPAN destination port.
Note For the Cisco Nexus 5548 Switch, Fibre Channel ports cannot be configured as ingress source ports in a SPAN session.
A source port, also called a monitored port, is a switched interface that you monitor for network traffic analysis. The switch supports any number of ingress source ports (up to the maximum number of available ports on the switch) and any number of source VLANs or VSANs.
A source port has these characteristics:
Can be of any port type: Ethernet, virtual Ethernet, Fibre Channel, virtual Fibre Channel, port channel, SAN port channel, VLAN, and VSAN.
Cannot be monitored in multiple SPAN sessions.
Cannot be a destination port.
Each source port can be configured with a direction (ingress, egress, or both) to monitor. For VLAN, VSAN, port channel, and SAN port channel sources, the monitored direction can only be ingress and applies to all physical ports in the group. The rx/tx option is not available for VLAN or VSAN SPAN sessions.
Beginning with Cisco NX-OS Release 5.0(2)N1(1). Port channel and SAN port channel interfaces can be configured as ingress or egress source ports.
Source ports can be in the same or different VLANs or VSANs.
For VLAN or VSAN SPAN sources, all active ports in the source VLAN or VSAN are included as source ports.
The Cisco Nexus 5010 switch supports a maximum of two egress SPAN source ports. This limit does not apply to the Cisco Nexus 5020 Switch and the Cisco Nexus 5548 switch.
SPAN destinations refer to the interfaces that monitors source ports. The Cisco Nexus 5000 Series switch supports Ethernet and Fibre Channel interfaces as SPAN destinations.
Virtual Fibre Channel
Virtual Fibre Channel
Characteristics of Destination Ports
Each local SPAN session must have a destination port (also called a monitoring port) that receives a copy of traffic from the source ports, VLANs, or VSANs. A destination port has these characteristics:
Can be any physical port, Ethernet, Ethernet (FCoE), or Fibre Channel. Virtual Ethernet and virtual Fibre Channel ports cannot be destination ports.
Cannot be a source port.
Cannot be a port channel or SAN port channel group.
Does not participate in spanning tree while the SPAN session is active.
Is excluded from the source list and is not monitored if it belongs to a source VLAN of any SPAN session.
Receives copies of sent and received traffic for all monitored source ports. If a destination port is oversubscribed, it can become congested. This congestion can affect traffic forwarding on one or more of the source ports.
Limitations of Nexus 5000 SPAN CoS values are not preserved at the monitor (span) destination.
Packets coming in on the monitor source with an unknown VLAN tag are spanned ouf with a 0 VLAN tag (priority tag).
For Ethernet destination, the monitor session is up only if the destination port is configured as switch port monitor.
Out of 18 configurable sessions, only two are active (up state). The rest are in down state (hardware resource unavailable).
Configuration limitations: VLAN or port-channel cannot be configured as egress source
VLAN or port channel cannot be a monitor destination.
Only two egress sources supported.
Only one destination port can be configured for a session.
When the Nexus 5000 experiences loss of fabric connectivity, it brings down all the affected vFC interfaces.
The following methods are used to signal the host of loss of connectivity to the FC fabric
FIP Clear Link Virtual Link to the CNA will be signaled to indicate the ‘shut’ state of vFC. Throughout the ‘shut’ period FCF Advertisements indicate ‘not available for login’.
In case the loss of connectivity is over the FCoE network, FIP keep-alives are used by the FCF and the CNA to timeout the login sessions. The keep-alive timers are configurable.
Under certain failure scenarios where the access switch has lost all uplink connectivity to the aggregation layer, the CNA needs to be signaled of the loss of LAN connectivity. This helps the CNA failover the host traffic to the standby port. Traditionally, such a failure is signaled by bringing down the host facing link. Bringing down the link achieves two purposes:
Host is signaled of loss of connectivity.
The access switch stops forwarding traffic to and from the host-facing link.
However, in the converged network, even though the LAN connectivity is lost at the access switch, the SAN connectivity might still be intact. Bringing down the entire host-facing link is not desirable. Instead, the loss of connectivity is signaled over protocols. Loss of SAN connectivity is signaled using the FIP Clear Virtual Link message. Loss of LAN connectivity is signaled using logical link status TLVs defined in DCBX and VIC protocols.
When LAN connectivity is lost for a particular VLAN on the uplinks, the VLAN is also brought down on the host-facing link.
Dedicating a VLAN solely for FCoE traffic helps with shutting down non-FCoE traffic to and from the host-facing link without disrupting FCoE traffic from the same host.