Use ELAM on Nexus 7000 F3 Module

Introduction

This document describes the steps used to perform an ELAM (Embedded Logic Analyzer Module) on a Cisco Nexus 7000/7700 F3 module.

Prerequisites

Requirements

Cisco recommends that you have a familiarity with the Cisco Nexus Operating System (NX-OS) and basic Nexus architecture before you proceed with the information that is described in this document.

ELAM can only be done by the network-admin role. Please ensure to log in as a user with network-admin privilege.

Components Used

The information in this document is based on these software and hardware versions:

• Cisco Nexus 7700 Series switches
• Cisco N7700 F3 Series modules (N77-F324FQ-25, 24-Port 10/40 Gigabit Ethernet modules)
• Cisco NX-OS Versions 8.4.9

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

What is ELAM?

The ELAM assists in troubleshooting network forwarding issues by capturing real-time packets without disruptions, and it does not impact performance or control-plane resources. The ELAM is a powerful, granular, and non-intrusive tool most commonly used by Cisco Technical Assistance Center (TAC) engineers. However, it is crucial to know that the ELAM tool captures only one packet at a time - the first packet received after ELAM starts. If you need to capture all packets of a flow, please use SPAN or ERSPAN.

The ELAM can answer questions such as:

• Does the frame of interest enter the switch?
• From which port and VLAN is the packet received?
• What are the source and destination MAC addresses of the incoming packet?
• How is the packet rewritten, and to which port is it sent?

Topology

In this article, Host1 connected to N77-F3 port E5/23, sends traffic to Host2. ELAM is used to capture the individual frame from Host1 to Host2.

To run an ELAM on the N7K, you first need to log in as a user with network-admin, and then you need to connect to the module.

N77-F3# attach module 5
Attaching to module 5 ...
To exit type 'exit', to abort type '$.' 
Last login: Thu Jan 18 05:31:04 pst 2024 from 127.1.1.3 on pts/0

Identifying the Ingress Forwarding Engine

The Nexus 7000 operates as a fully distributed switch, with forwarding decisions made by the forwarding engine of ingress linecard.

In this article, the traffic of interest is expected to ingress the switch via port 5/23. In the example of the N7K, the module 5 is an F3 module.

N77-F3# show module 5 
Mod Ports            Module-Type                     Model         Status
--- ----- ----------------------------------- ------------------ ----------
5     24      10/40 Gbps Ethernet Module        N77-F324FQ-25       ok
Mod Sw Hw
--- --------------- ------
5 8.4(9) 1.3

For F3 modules, perform the ELAM on the Layer 2 (L2) Forwarding Engine (FE) with the internal codename Flanker. 

module-5# show hardware internal dev-port-map 
--------------------------------------------------------------
CARD_TYPE: 24 port 40G 
>Front Panel ports:24
--------------------------------------------------------------
Device name                  Dev role           Abbr num_inst:
--------------------------------------------------------------
> Flanker Fwd Driver     DEV_LAYER_2_LOOKUP      L2LKP 12
FP port | PHYS | MAC_0 | L2LKP | L3LKP | QUEUE |SWICHF 
22                 10     10      10      10      0,1
23                 11     11      11      11      0,1        >>>Port 23 belongs to FE instance 11
24                 11     11      11      11      0,1        
+-----------------------------------------------------------------------+

In this output, it is evident that port E5/23 belongs to FE instance 11.

The Layer 2 FE Data Bus (DBUS) carries the original header information prior to the Layer 2 (L2) and Layer 3 (L3) lookups, whereas the Result Bus (RBUS) contains the results of both L3 and L2 lookups. For most troubleshooting scenarios, a Layer 2 ELAM capture is sufficient.

N77-F3# attach module 5
Attaching to module 5 ...
To exit type 'exit', to abort type '$.' 
Last login: Thu Jan 18 05:31:04 pst 2024 from 127.1.1.3 on pts/0
module-5# elam asic flanker instance 11
module-5(fln-elam)# ?
layer2 ELAMs for layer 2
layer3 ELAMs for layer 3
module-5(fln-elam)# layer2

Example: ARP ELAM Capture

In this example, Host1 on VLAN 100 (with an IP address of 192.168.1.1 and MAC address of 8c60.4fc7.c5bc), linked to port E5/23, sends an Address Resolution Protocol (ARP) request. This request is intended to resolve the MAC address for another host on the same VLAN 100, which has an IP address of 192.168.1.2.

Configuring the Trigger

The Flanker ASIC supports ELAM triggers for various frame types. The ELAM trigger must correspond with the frame type. If the frame is an ARP frame, the trigger must also be set to the ARP option. An ARP frame cannot be captured by the Other L2 trigger. If you use ELAM to capture an MPLS frame, select IPv4 or IPv6 instead of MPLS. For more details, please refer to the bug section.

module-5(fln-l2-elam)# trigger dbus ?
arp ARP Frame Format              >>>capture ARP packet. Other L2 does not work for ARP
fc Fc hdr Frame Format           
ipv4 IPV4 Frame Format            >>>capture IPv4 frame
ipv6 IPV6 Frame Format            >>>capture IPv6 frame
mpls MPLS                          
other L2 hdr Frame Format         >>>capture non-ip l2 frame
rarp RARP Frame Format

In this example, the frame is captured based on the target IP address field of the ARP frame, so only this value is specified.

Flanker necessitates that triggers are set for both the DBUS and the RBUS. The RBUS trigger is simplified,trig matches the same criteria as the DBUS trigger.

module-5(fln-l2-elam)# trigger dbus arp ingress if target-ip-address 192.168.1.2
module-5(fln-l2-elam)# trigger rbus ingress if trig

Now you have configured the trigger, you can start the capture.

module-5(fln-l2-elam)# start

To verify if ELAM has captured any packets, you can run the status command. The term Armed indicates that no matching packets have been captured yet.

module-5(fln-l2-elam)# status
ELAM Slot 5 instance 11: L2 DBUS Configuration: trigger dbus arp ingress if target-ip-address 192.168.1.2 
L2 DBUS: Armed                  >>>no matched packet
ELAM Slot 5 instance 11: L2 RBUS Configuration: trigger rbus ingress if trig 
L2 RBUS: Armed                  >>>no matched packet

Ping from Host1 (192.168.1.1) to 192.168.1.2. Since there is no ARP entry on Host1, Host1 sends out the ARP request in a broadcast packet. Once the FE receives the ARP frame, it checks the trigger. If there is a match, ELAM captures the forwarding decision of this frame, and the ELAM status is then displayed as Triggered.

module-5(fln-l2-elam)# status
ELAM Slot 5 instance 11: L2 DBUS Configuration: trigger dbus arp ingress if target-ip-address 192.168.1.2 
L2 DBUS: Triggered              >>Packet hit
ELAM Slot 5 instance 11: L2 RBUS Configuration: trigger rbus ingress if trig 
L2 RBUS: Triggered              >>Packet hit

Interpreting the Result

The result is valid only if the DBUS and RBUS have captured the same packet. Therefore, it is necessary to check the sequence number in both DBUS and RBUS results. If they do not match, you can restart and capture them again until they align.

module-5(fln-l2-elam)# show dbus | in seq
sequence-number : 0x7 vl : 0x0 
module-5(fln-l2-elam)# show rbus | in seq
l2-rbus-trigger : 0x1 sequence-number : 0x7 

It is advisable to first check the DBUS output, as it contains data prior to any rewriting. Here is an example of an ARP ELAM capture. Note that some output has been omitted.

module-5(fln-l2-elam)# show dbus                   
cp = 0x20c6ad1c, buf = 0x20c6ad1c, end = 0x20c7706c
--------------------------------------------------------------------
Flanker Instance 11 - Capture Buffer On L2 DBUS:
<snip>
--------------------------------------------------------------------
L2 DBUS PRS MLH ARP/RARP 
--------------------------------------------------------------------
valid : 0x1                                    requst-response : 0x1    >>>ARP request
(1:for ARP request,2: for ARP reply, 3:for RARP request, 4:for RARP reply)
port-id : 0x0                                 last-ethertype : 0x806    >>>Ethernet type, 0x0806 means ARP
packet-type : 0x0                              l2-length-check : 0x0    >>>0 for ingress, 1 for egress
vqi : 0x0                                       packet-length : 0x40    >>>L2 ethernet frame totally length 64 byte
vlan : 0x64                                  destination-index : 0x0    >>>VLAN100 
source-index : 0xb79                               bundle-port : 0x0    >>>source port ltl index
status-is-1q : 0x0                                 trill-encap : 0x0    >>>0 means frame without vlan tag 
sender-ip-address: 192.168.1.1                                          >>>sender-ip-address in ARP header
target-ip-address: 192.168.1.2                                          >>>target-ip-address in ARP header
sender-mac-address : 8c60.4fc7.c5bc                                     >>>sender-mac-address in ARP header
target-mac-address : ffff.ffff.ffff                                     >>>target-mac-address in ARP header
destination-mac-address : ffff.ffff.ffff                                >>>sestination mac in ethernet header
source-mac-address : 8c60.4fc7.c5bc                                     >>>source mac in ethernet header

To verify the port on which the frame is received, use the PIXM (Port Index Manager) command. This command displays the mapping of a Local Target Logic (LTL) to a front port or a group of front ports.

N77-F3# show system internal pixm info ltl 0xb79             
------------------
Type LTL
---------------------------------
PHY_PORT Eth5/23 
FLOOD_W_FPOE 0xc031

The output reveals that a source-index of 0xb79 maps to port E5/23. This verifies that the frame is received on port E5/23.

After confirming that ELAM has captured the frame of interest, you can verify the result of the forwarding decision using the RBUS data (note that some output has been omitted).

module-5(fln-l2-elam)# show rbus  
--------------------------------------------------------------------
L2 RBUS INGRESS CONTENT 
--------------------------------------------------------------------
di-ltl-index : 0xc031                        l3-multicast-di : 0xc00    >>> destination ltl index 
source-index : 0xb79                         vlan : 0x64                >>> vlan id after rewritten
vqi : 0x0                                    di2-valid : 0x0            >>> use l3-multicast-di as di if this is 1
routed-frame : 0x0                           copy-cause : 0x0           >>> 0x0 means N7K performs layer 2 switching

With the RBUS data, you can confirm that the frame is switched on VLAN 100 (0x64). To determine the egress port(s) from the di-ltl-index, use the pixm command again.

N77-F3# show system internal pixm info ltl 0xc031      
Member info
------------------
IFIDX LTL
---------------------------------
Eth5/24 0x0b78
Eth5/23 0x0b79

The output shows that ports E5/23 and E5/24 both belong to LTL 0xc031. The ARP packet switches to both of these ports. As it is received from E5/23, it is only sent out from E5/24.

Example: IPv4 ELAM Capture

ipv4 l3 elam

In this example, Host1 on VLAN 100 (with an IP address of 192.168.1.1/24 and MAC address of 8c60.4fc7.c5bc), is connected to port E5/23 and sends an Internet Control Message Protocol (ICMP) request to Host2. Host2 has an IP address of 192.168.2.2/24 and is on a different VLAN, VLAN200.

Configuring the Trigger

In this example, since Host1 and Host2 are in different VLANs, the ICMP packet from Host1 to Host2 is routed through Layer 3 on N77-F3. A Layer 2 ELAM is used to capture the ICMP request packet.

Both the source IP (192.168.1.2) and the destination IP (192.168.2.2) are combined as DBUS triggers. ELAM only captures the packet that matches all triggers.

N77-F3# attach module 5
Attaching to module 5 ...
To exit type 'exit', to abort type '$.' 
Last login: Thu Jan 18 11:19:46 pst 2024 from 127.1.1.3 on pts/0
module-5# elam asic flanker instance 11
module-5(fln-elam)# layer2                    
module-5(fln-l2-elam)# trigger dbus ipv4 ingress if destination-ipv4-address 192.168.2.2 source-ipv4-address 192.168.1.2 
module-5(fln-l2-elam)# trigger rbus ingress if trig
module-5(fln-l2-elam)# start
module-5(fln-l2-elam)# status                 
ELAM Slot 5 instance 11: L2 DBUS Configuration: trigger dbus ipv4 ingress if destination-ipv4-address 192.168.2.2 source-ipv4-address 192.168.1.2 
L2 DBUS: Armed
ELAM Slot 5 instance 11: L2 RBUS Configuration: trigger rbus ingress if trig 
L2 RBUS: Armed

Start a ping from Host1 (192.168.1.2) to Host2 (192.168.2.2). Once the packet that matches the triggers is received by FE instance 11, the ELAM status displays as Triggered.

module-5(fln-l2-elam)# status
ELAM Slot 5 instance 11: L2 DBUS Configuration: trigger dbus ipv4 ingress if destination-ipv4-address 192.168.2.2 source-ipv4-address 192.168.1.2 
L2 DBUS: Triggered
ELAM Slot 5 instance 11: L2 RBUS Configuration: trigger rbus ingress if trig 
L2 RBUS: Triggered

Interpreting the Result

Ensure that the RBUS and DBUS have the same sequence number. This step is necessary for every capture.

module-5(fln-l2-elam)# show dbus | in seq
sequence-number : 0x74 vl : 0x0 
module-5(fln-l2-elam)# show rbus | in seq
l2-rbus-trigger : 0x1 sequence-number : 0x74      >>same sequence number, valid elam result

Here is an example of an IPv4 ICMP ELAM capture. Note that some output has been omitted.

module-5(fln-l2-elam)# show dbus 
--------------------------------------------------------------------
L2 DBUS PRS MLH IPV4 
--------------------------------------------------------------------
l4-protocol : 0x1                                           df : 0x0    >>>L4 protocol id, 1 means icmp packet
ttl : 0xff                                   l3-packet-length : 0x54    >>>ip total length is 84 in this packet, ttl is 255
port-id : 0x0                                 last-ethertype : 0x800    >>>Ethernet type, 0x0800 means IPv4
vqi : 0x0                                       packet-length : 0x66    >>>L2 frame length field
vlan : 0x64                                  destination-index : 0x0    >>>vlan id 100
source-index : 0xb79                               bundle-port : 0x0    >>>source port ltl index
status-is-1q : 0x1                                 trill-encap : 0x0    >>>1 means frame with vlan tag 
source-ipv4-address: 192.168.1.2                                        >>>Packet source IP
destination-ipv4-address: 192.168.2.2                                   >>>Packet destination IP
destination-mac-address : 003a.9c40.8ac3                                >>>Packet destination mac
source-mac-address : 8c60.4fc7.c5bc                                     >>>Packet source mac 

With the DBUS data, you can confirm that the packet is received on VLAN100 (vlan:0x64) with a source IP of 192.168.1.2 and a destination IP of 192.168.2.2. You can also identify that this is an IPv4 ICMP packet.

To verify the port on which the frame is received, run the PIXM  (Port Index Manager) command. This command displays the mapping of a Local Target Logic (LTL) to a front port or a group of front ports.

N77-F3# show system internal pixm info ltl 0xb79             
------------------
Type LTL
---------------------------------
PHY_PORT Eth5/23 
FLOOD_W_FPOE 0xc032
FLOOD_W_FPOE 0xc031
FLOOD_W_FPOE 0xc029

The output shows that a source-index of 0xb79 maps to port E5/23. This confirms that the frame is received on port E5/23.

After confirming that ELAM has captured the ICMP packet of interest, you can verify the result of the forwarding decision using the RBUS data (note that some output has been omitted). From the RBUS data, you can see that the frame is routed from VLAN 100 (0x64) to VLAN200.

module-5(fln-l2-elam)# show rbus
--------------------------------------------------------------------
L2 RBUS INGRESS CONTENT 
--------------------------------------------------------------------
segment-id-valid : 0x0                                 ttl-out : 0xfe      >>>TTL is 254
di-ltl-index : 0xb78                            l3-multicast-di : 0x0      >>>destination port ltl 
source-index : 0xb79                                      vlan : 0xc8      >>>vlan id is 200
routed-frame : 0x1                                   copy-cause : 0x0      >>>routed on N7K

To determine the egress port(s) from the di-ltl-index, run the PIXM command. The output shows that the egress port is E5/24.

N77-F3# show system internal pixm info ltl 0xb78
Member info
------------------
Type LTL
---------------------------------
PHY_PORT Eth5/24 
FLOOD_W_FPOE 0xc032
FLOOD_W_FPOE 0xc031
FLOOD_W_FPOE 0xc029

Additional Verification (F3 ltl-region)

The output from this command is helpful in understanding the purpose of an LTL if it does not match a physical port. Examples include Drop LTL and Inband LTL:

N77-F3# show system internal pixm info ltl-region 
===========================================================
PIXM VDC 1 LTL MAP Version: 3
Description: LTL Map for Crossbow 
===========================================================
LTL_TYPE SIZE START END 
========================================================================
LIBLTLMAP_LTL_TYPE_SUP_ETH_INBAND 64 0xc00 0xc3f 
-------------------------------------------------------------------
LIBLTLMAP_LTL_TYPE_DROP_DI_WO_HW_BITSET 0xcae 
LIBLTLMAP_LTL_TYPE_DROP_DI 0xcad 

ELAM bugs