Guest

Cisco Nexus 5000 Series Switches

Nexus 3000/5000/7000 Use of the Ethanalyzer Tool

Techzone Article content

Document ID: 116201

Updated: Jul 01, 2013

Contributed by Prashanth Krishnappa, Vinayak Sudame, and Shelley Bhalla, Cisco TAC Engineers.

   Print

Introduction

This document describes how to use the built-in packet capture tool, Ethanalyzer, on the Nexus 3000/5000/7000 switches.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on Nexus 3000, Nexus 5000, and Nexus 7000 switches.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Ethanalyzer

Ethanalyzer is a useful tool to troubleshoot control plane and traffic destined to switch CPU. Mgmt is the interface to troubleshoot packets that hit the mgmt0 interface. Inbound-low (eth3) is for low priority (ping, telnet, Secure Shell) CPU-bound traffic, and inbound-hi (eth4) is for high priority (Spanning Tree Protocol (STP), Bridge Protocol Data Units, FIP) CPU-bound traffic.

Note: You can use the Display Filter or the Capture Filter as an option. The Display filter option is preferred on the Nexus 5000, and the Capture Filter is preferred on the Nexus 3000 and Nexus 7000.

Commonly used Display Filters can be found at Wireshark 

Commonly used Capture Filters can be found at Wireshark 

Note: Since Nexus 5000 uses internal VLANs to forward frames, Ethanlyzer has internal VLANs. Nexus 5000 forwards frames based on internal VLANs and Ethanalyzer displays the internal VLAN. When you troubleshoot with Ethanalyzer, the VLAN ID can cause difficulties. However, you can use the command show system internal fcfwd fwcvidmap cvid in order to determine the mapping. Here is an example.

Nexus# ethanalyzer local interface inbound-low detail display-filter icmp
Capturing on eth3
Frame 16 (102 bytes on wire, 102 bytes captured)
     Arrival Time: Sep 7, 2011 15:42:37.081178000
     [Time delta from previous captured frame: 0.642560000 seconds]
     [Time delta from previous displayed frame: 1315424557.081178000 seconds]
     [Time since reference or first frame: 1315424557.081178000 seconds]
     Frame Number: 16
     Frame Length: 102 bytes
     Capture Length: 102 bytes
     [Frame is marked: False]
     [Protocols in frame: eth:vlan:ip:icmp:data]
Ethernet II, Src: 00:0d:ec:a3:81:bc (00:0d:ec:a3:81:bc),
Dst: 00:05:73:ce:3c:7c (00:05:73:ce:3c:7c)
     Destination: 00:05:73:ce:3c:7c (00:05:73:ce:3c:7c)
        Address: 00:05:73:ce:3c:7c (00:05:73:ce:3c:7c)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address(factory default)
     Source: 00:0d:ec:a3:81:bc (00:0d:ec:a3:81:bc)
        Address: 00:0d:ec:a3:81:bc (00:0d:ec:a3:81:bc)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address(factory default)
     Type: 802.1Q Virtual LAN (0x8100)
802.1Q Virtual LAN
     000. .... .... .... = Priority: 0
     ...0 .... .... .... = CFI: 0
     .... 0000 0011 1001 = ID: 57 <<----------
     Type: IP (0x0800)
Internet Protocol, Src: 144.1.1.63 (144.1.1.63), Dst: 144.1.1.41 (144.1.1.41)
     Version: 4
     Header length: 20 bytes
     Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
     Total Length: 84
     Identification: 0x1118 (4376)
<snip>

As you can see, Ethanalyzer indicates that the packet was received on VLAN 57, which is the internal VLAN. However, VLAN 57 is not the actual VLAN, because 57 is not in hex. 57 in hex is 0x0039. This command determines the actual VLAN in hex.

Nexus# show system internal fcfwd fwcvidmap cvid | grep 0x0039
0x0039 enet 0x01 0x0090 0100.0000.080a 0100.0000.0809
0x0039 fc 0x01 0x0090 0100.0000.0007 0100.0000.0006

0x0090 is the actual VLAN in hex. You must then convert the number to decimal, which is 144. This calculation illustrates that the actual VLAN in the previous frame was VLAN 144, although the Ethanalyzer indicates it was 57.

Here is an example that captures FIP frames with the Display Filter of VLAN.(etype==0x8914)

Nexus# ethanalyzer local interface inbound-hi display-filter vlan.etype==0x8914
Capturing on eth4
2011-10-18 13:36:47.047492 00:c0:dd:15:d4:41 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 56
2011-10-18 13:36:48.313531 00:c0:dd:15:d0:95 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 56
2011-10-18 13:36:49.373483 00:0d:ec:a3:81:80 -> 01:10:18:01:00:01 0x8914
PRI: 3 CFI: 0 ID: 56
2011-10-18 13:36:49.373868 00:0d:ec:a3:81:80 -> 01:10:18:01:00:01 0x8914
PRI: 3 CFI: 0 ID: 56
2011-10-18 13:36:49.374131 00:0d:ec:a3:81:80 -> 01:10:18:01:00:01 0x8914
PRI: 3 CFI: 0 ID: 56
2011-10-18 13:36:49.374378 00:0d:ec:a3:81:80 -> 01:10:18:01:00:01 0x8914
PRI: 3 CFI: 0 ID: 56
2011-10-18 13:36:49.374618 00:0d:ec:a3:81:80 -> 01:10:18:01:00:01 0x8914
PRI: 3 CFI: 0 ID: 56
2011-10-18 13:36:49.374859 00:0d:ec:a3:81:80 -> 01:10:18:01:00:01 0x8914
PRI: 3 CFI: 0 ID: 56
2011-10-18 13:36:49.375098 00:0d:ec:a3:81:80 -> 01:10:18:01:00:01 0x8914
PRI: 3 CFI: 0 ID: 56
2011-10-18 13:36:49.375338 00:0d:ec:a3:81:80 -> 01:10:18:01:00:01 0x8914
PRI: 3 CFI: 0 ID: 56
10 packets captured
Program exited with status 0.
Nexus#

Here is an example that captures FKA frames from a particular CNA (vFC1311 tied to Po1311). This configuration causes Ethanalyzer to see FKA from the host every eight seconds, which is the FKA timer.

Nexus# show flogi database 
--------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------------
vfc15 200 0x1e0000 50:0a:09:81:89:4b:84:32 50:0a:09:80:89:4b:84:32
vfc16 200 0x1e0003 50:0a:09:81:99:4b:84:32 50:0a:09:80:89:4b:84:32
vfc17 200 0x1e0002 21:00:00:c0:dd:12:b9:b7 20:00:00:c0:dd:12:b9:b7
vfc18 200 0x1e0006 21:00:00:c0:dd:14:6a:73 20:00:00:c0:dd:14:6a:73
vfc19 200 0x1e0001 21:00:00:c0:dd:11:00:49 20:00:00:c0:dd:11:00:49
vfc20 200 0x1e0007 21:00:00:c0:dd:12:0e:37 20:00:00:c0:dd:12:0e:37
vfc23 200 0x1e0004 10:00:00:00:c9:85:2d:e5 20:00:00:00:c9:85:2d:e5
vfc1311 200 0x1e0008 10:00:00:00:c9:9d:23:73 20:00:00:00:c9:9d:23:73

Total number of flogi = 8.

Nexus# ethanalyzer local interface inbound-hi display-filter "eth.addr==
00:00:c9:9d:23:73 && vlan.etype==0x8914 && frame.len==60"limit-captured-frames 0
Capturing on eth4

2011-10-22 11:06:11.352329 00:00:c9:9d:23:73 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 24
2011-10-22 11:06:19.352116 00:00:c9:9d:23:73 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 24
2011-10-22 11:06:27.351897 00:00:c9:9d:23:73 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 24
2011-10-22 11:06:35.351674 00:00:c9:9d:23:73 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 24
2011-10-22 11:06:43.351455 00:00:c9:9d:23:73 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 24
2011-10-22 11:06:51.351238 00:00:c9:9d:23:73 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 24
2011-10-22 11:06:59.351016 00:00:c9:9d:23:73 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 24
2011-10-22 11:07:07.350790 00:00:c9:9d:23:73 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 24
2011-10-22 11:07:15.350571 00:00:c9:9d:23:73 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 24
2011-10-22 11:07:23.350345 00:00:c9:9d:23:73 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 24
2011-10-22 11:07:31.350116 00:00:c9:9d:23:73 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 24
2011-10-22 11:07:39.349899 00:00:c9:9d:23:73 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 24
2011-10-22 11:07:47.349674 00:00:c9:9d:23:73 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 24
2011-10-22 11:07:55.349481 00:00:c9:9d:23:73 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 24
2011-10-22 11:08:03.349181 00:00:c9:9d:23:73 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 24
2011-10-22 11:08:11.348965 00:00:c9:9d:23:73 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 24
2011-10-22 11:08:19.348706 00:00:c9:9d:23:73 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 24
2011-10-22 11:08:27.348451 00:00:c9:9d:23:73 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 24
2011-10-22 11:08:35.348188 00:00:c9:9d:23:73 -> 00:0d:ec:a3:81:80 0x8914
PRI: 3 CFI: 0 ID: 24
52 packets dropped

Nexus# 19 packets captured

The previous capture only displays headers. You could also print a detail packet; but, when you use the detail option, it is best to write the capture to a file and then open the file with Wireshark.

Nexus# ethanalyzer local interface inbound-hi detail display-filter 
vlan.etype==0x8914 write bootflash:flogi.pcap ?

     <CR>
     > Redirect it to a file
     >> Redirect it to a file in append mode
     display Display packets even when writing to a file
     | Pipe command output to filter

Here is an example to capture LACP frames:

Nexus# ethanalyzer local interface inbound-hi display-filter slow
Capturing on eth42011-12-05 12:00:08.472289 00:0d:ec:a3:81:92 -> 01:80:c2:00:00:02 LACP Link
Aggregation Control ProtocolVersion 1. Actor Port = 16651 Partner Port = 283
2011-12-05 12:00:16.944912 00:1d:a2:00:02:99 -> 01:80:c2:00:00:02 LACP Link
Aggregation Control ProtocolVersion 1. Actor Port = 283 Partner Port = 16651
2011-12-05 12:00:25.038588 00:22:55:77:e3:ad -> 01:80:c2:00:00:02 LACP Link
Aggregation Control ProtocolVersion 1. Actor Port = 16666 Partner Port = 16643
2011-12-05 12:00:25.394222 00:1b:54:c1:94:99 -> 01:80:c2:00:00:02 LACP Link
Aggregation Control ProtocolVersion 1. Actor Port = 282 Partner Port = 16644
2011-12-05 12:00:26.613525 00:0d:ec:8f:c9:ee -> 01:80:c2:00:00:02 LACP Link
Aggregation Control ProtocolVersion 1. Actor Port = 295 Partner Port = 295
2011-12-05 12:00:26.613623 00:0d:ec:8f:c9:ef -> 01:80:c2:00:00:02 LACP Link
Aggregation Control ProtocolVersion 1. Actor Port = 296 Partner Port = 296

Here is an example to capture all frames sourced with a MAC address of 00:26:f0 (a wild card filter).

Nexus# ethanalyzer local interface inbound-hi display-filter 
"eth.src[0:3]==00:26:f0" limit-captured-frames 0

Capturing on eth4
2012-06-20 16:37:22.721291 00:26:f0:05:00:00 -> 01:80:c2:00:00:00 STP Conf.
Root = 8192/d0:57:4c:b7:dc:00 Cost = 200 Port = 0x9004
2012-06-20 16:37:22.721340 00:26:f0:05:00:00 -> 01:00:0c:cc:cc:cd STP Conf.
Root = 8192/d0:57:4c:b7:dc:00 Cost = 200 Port = 0x9004
2012-06-20 16:37:22.721344 00:26:f0:05:00:00 -> 01:00:0c:cc:cc:cd STP Conf.
Root = 8192/d0:57:4c:b7:dc:00 Cost = 200 Port = 0x9004
2012-06-20 16:37:22.721348 00:26:f0:05:00:00 -> 01:00:0c:cc:cc:cd STP Conf.
Root = 8192/d0:57:4c:b7:dc:00 Cost = 200 Port = 0x9004
19 packets dropped
Nexus# 4 packets captured

Note: In the previous output, you see "19 Packets dropped." These packets are not actually dropped, but are not captured by Ethanalyzer.

Make sure you select the appropriate CPU Queue (Inbound-hi, inbound-lo, or mgmt).

Here are common traffic types and queues:

  • Inbound-low - SUP-low (eth3) (Address Resolution Protocol (ARP)/IP over switch virtual interface, Internet Group Management Protocol Snooping)
  • Inbound-hi - SUP-high (eth4) (STP, FIP, Fibre Channel over Ethernet (FCoE), FC, Cisco Discovery Protocol, Link Layer Discovery Protocol/Data Center Bridging Capabilities Exchange Protocol, Link Aggregation Control Protocol, Unidirectional Link Detection)
  • Mgmt - Out-Of-Band (anything through mgmt0 interface)
  • FIP (Fabric Login, Clear Virtual Link, FKA): VLAN.etype==0x8914
  • FCoE (Port Login, Domain Name System): VLAN.etype==0x8906

Here is an example of a capture FIP and FCoE:

ethanalyzer local interface inbound-hi display-filter "vlan.etype==0x8914
|| vlan.etype==0x8906"

Here are some ARP filters:

Nexus# ethanalyzer local interface inbound-low display-filter 
arp.src.hw_mac==0013.8066.8ac2

Capturing on eth3
2012-07-12 21:23:54.643346 00:13:80:66:8a:c2 ->
ff:ff:ff:ff:ff:ff ARP Who has 172.18.121.59?  Tell 172.18.121.1

NexusF340.24.10-5548-2# 1 packets captured

Nexus# ethanalyzer local interface inbound-low display-filter
arp.src.proto_ipv4==172.18.121.4

Capturing on eth3
2012-07-12 21:25:38.767772 00:05:73:ab:29:fc ->
ff:ff:ff:ff:ff:ff ARP Who has 172.18.121.1?  Tell 172.18.121.4
Updated: Jul 01, 2013
Document ID: 116201