Troubleshoot Hardware Forwarding Issues on Nexus 7000 Series Switches
Available Languages
Download Options
Updated:October 6, 2016
Document ID:200300
Bias-Free Language
The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
Contents
Introduction
This document describes how to troubleshoot hardware forwarding issues on F3 Series modules for Cisco Nexus 7000 Series switches.
Prerequisites
Requirements
Cisco recommends that you have a familiarity with the Cisco Nexus Operating System (NX-OS) and basic Nexus architecture before you proceed with the information that is described in this document.
Components Used
The information in this document is based on these software and hardware versions:
- Cisco Nexus 7000 Series switches (N7K)
- Cisco N7K F3 Series modules (N7K-F312FQ-25, 12-Port 10/40 Gigabit Ethernet modules)
- Cisco NX-OS Versions 6.2.8a and later
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Background Information
This document focuses primarily on some of the built-in tools that are used for hardware troubleshooting when you have exhausted your software part of the forwarding table or control plane. One such tool is the Embedded Logic Analyzer Module (ELAM), which is an Application-Specific Integrated Circuit (ASIC) that captures a single packet and shows how the ingress packet appears on the Data BUS (DBUS) and the Result BUS (RBUS) after forwarding.
The ASIC is embedded within the forwarding pipeline, and it can capture a packet in real-time without disruptions to performance or control-plane resources. This helps to answer questions such as:
- Did the packet reach the Forwarding Engine (FE)?
- On what port and VLAN is the packet received?
- How does the packet appear (Layer 2 (L2) or Layer 4 (L4) data)?
- How is the packet altered, and where is it sent?
The ELAM is a powerful, granular, and non-intrusive tool that is most commonly used by the Cisco Technical Assistance Center (TAC) engineers who work on hardware switching platforms. However, it is important to know that the ELAM tool only captures one packet at time. That is, the first packet that is received after the ELAM is triggered.
Troubleshoot
This section describes how to troubleshoot ELAM on an F3 Series module in deployments that do not involve the use of a breakout cable, as well as the deployments that do use breakout cables.
Troubleshoot ELAM on F3 Series Modules without Breakout Cables
This is the topology that is used for the examples throughout this section:
+-------------+ +-------------+ +------------+
| | | | | |
| | | | | |
| | e3/3-4 e3/1-2 | | e3/5 e3/7 | |
| N7K2 +-------------------+ N7K1 +---------------------+ N7K3 |
| +-------------------+ | .1 L3 .3 | |
| | Po1(Trunk) | | 192.168.13.0/24 | |
| | | | | |
+-------------+ +-------------+ +------------+
Vlan 10 SVI Vlan10 SVI
IP add 192.168.12.2/24 IP add 192.168.12.1/24
Here are some notes about this topology:
- The N7Ks run NX-OS Version 6.2.8a.
- Pings are sent from the N7K2 VLAN 10 interface to a remote IP address of 192.168.12.1.
- The ELAM captures packets on the N7K1.
- An N7K-F312FQ-25 is used, which is a 12-Port 10/40 Gigabit Ethernet module inserted into Slot 3.
Before you begin to troubleshoot your system, you should confirm the basic connectivity:
N7K2# ping 192.168.13.3
PING 192.168.13.3 (192.168.13.3): 56 data bytes
64 bytes from 192.168.13.3: icmp_seq=0 ttl=253 time=1.513 ms
64 bytes from 192.168.13.3: icmp_seq=1 ttl=253 time=1.062 ms
64 bytes from 192.168.13.3: icmp_seq=2 ttl=253 time=0.822 ms
64 bytes from 192.168.13.3: icmp_seq=3 ttl=253 time=0.830 ms
64 bytes from 192.168.13.3: icmp_seq=4 ttl=253 time=0.845 ms
--- 192.168.13.3 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.822/1.014/1.513 ms
N7K2# show ip route 192.168.13.3
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
192.168.13.0/24, ubest/mbest: 1/0
*via 192.168.12.1, [1/0], 01:20:36, static
!--- The next command verifies the Address Resolution Protocol (ARP) for the next hop.
N7K2# show ip arp 192.168.12.1
----SNIP----
IP ARP Table
Total number of entries: 1
Address Age MAC Address Interface
192.168.12.1 00:10:29 e4c7.2210.a142 Vlan10
You should also verify the Media Access Control (MAC) address learning on the Supervisor engine (Sup) and the module for the next hop:
N7K2# show mac address-table address e4c7.2210.a142
!--- This command output shows the MAC learning on the Sup (software).
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 10 e4c7.2210.a142 dynamic 120 F F Po1
This output shows the MAC learning on the module/hardware; however, in order to know the interface, you must convert the index:
N7K2# show hardware mac address-table 3 address e4c7.2210.a142
FE | Valid| PI| BD | MAC | Index | Stat| SW | Modi| Age| Tmr| GM| Sec|
| | | | | | ic | | fied|Byte| Sel| | ure|
---|------|---|------|---------------|-------|-----|-----|-----|----|----|---|----|
1 1 1 41 e4c7.2210.a142 0x00a2a 0 0x089 1 185 1 0 0
| TR| NT| RM| RMA| Cap| Fld|Always| PV | RD| NN| UC|PI_E8| VIF | SWID| SSWID| LID |
| AP| FY| | |ture| | Learn| | | | | | | | | |
|---|---|---|----|----|----|------|----|---|---|---|-----|-----|-----|------|-------|
0 0 0 0 0 0 0 0x00 0 0 1 0 0x000 0x000 0x000 0x00a2a
N7K2# show system internal pixm info ltl 0x00a2a
!--- This is the index that was received in the previous output.
---SNIP---
PC_TYPE PORT LTL RES_ID LTL_FLAG CB_FLAG MEMB_CNT
------------------------------------------------------------------------------
Normal Po1 0x0a2a 0x16000000 0x00000000 0x00000002 2
Member rbh rbh_cnt
Eth3/4 0x000000f0 0x04
Eth3/3 0x0000000f 0x04
---SNIP---
Enter these commands in order to obtain the Virtual Device Context (VDC) number (in this example, it is 3) and check the MAC directly on the module:
N7K2# show vdc
---SNIP---
vdc_id vdc_name state mac type lc
------ -------- ----- ---------- --------- ---
3 N7K2 active e4:c7:22:10:a1:43 Ethernet f3
module-3#attach module 3
module-3# vdc 3
!--- This data is obtained from the previous command output.
module-3# show mac address-table address e4c7.2210.a142
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, (d) - dec
Age - seconds since last seen,,+ - primary entry using vPC Peer-Link
(T) - True, (F) - False, h - hex, d - decimal
VDC = 3
FE VLAN/BD MAC Address Type Age Secure NTFY Ports/SWID.SSID.LID(d)
-----+------+-------------+--------+------+------+------+-----------------------
* 1 10 e4c7.2210.a142 dynamic 360 F F Po1
Determine the link on port channel 1 that is used in order to forward traffic on the Sup from N7K2, as well as the link that is used in order to send a reply from N7K3 when port channel 1 is used from N7K1 to N7K2:
N7K2# show port-channel load-balance forwarding-path interface port-channel 1 src-ip
192.168.12.2 dst-ip 192.168.13.3 module 3
Module 3: Missing params will be substituted by 0's.
Load-balance Algorithm: src-dst ip
RBH: 0xd2 Outgoing port id: Ethernet3/3
N7K1# show port-channel load-balance forwarding-path interface port-channel 1 src-ip
192.168.13.3 dst-ip 192.168.12.2 module 3
Module 3: Missing params will be substituted by 0's.
Load-balance Algorithm: src-dst ip
RBH: 0xd2 Outgoing port id: Ethernet3/1
Send a ping from N7K2 (IP address 192.168.12.2) and capture the packets on N7K1 in the ingress direction in order to confirm that the packets are forwarded to N7K3 (IP address 192.168.13.3).
Before you send the ping, you should have knowledge of the hardware buildup. Complete these steps in order to understand the buildup:
- Attach the module:
N7K1# attach module 3
Attaching to module 3 ...
To exit type 'exit', to abort type '$.' - Identify the flanker instance. The flanker is a Switch on Chip (SOC) ASIC for the F3 Series module. Each flanker is mapped to two external ports on the module (the information changes per module type and is specific to the N7K-F312FQ-25).
There are 12 ports on the module, and each ASIC maps to two ports on the front panel, which means that there are 6 (0-5) flanker instances available on the module (the instance count is zero-based).Note: Ensure that you have network administrative privileges before you begin.
As you capture the packet that arrives from N7K2 via port channel 1 on N7K1, look for the ports (e3/1 and e3/2) that are mapped to each instance:
module-3# show hardware internal dev-port-map
--------------------------------------------------------------
CARD_TYPE: 12 port 40G
>Front Panel ports:12
--------------------------------------------------------------
Device name Dev role Abbr num_inst:
--------------------------------------------------------------
> Flanker Eth Mac Driver DEV_ETHERNET_MAC MAC_0 6
> Flanker Fwd Driver DEV_LAYER_2_LOOKUP L2LKP 6
!--- Check for the L2LKP number for ports 1 and 2.
> Flanker Xbar Driver DEV_XBAR_INTF XBAR_INTF 6
> Flanker Queue Driver DEV_QUEUEING QUEUE 6
> Sacramento Xbar ASIC DEV_SWITCH_FABRIC SWICHF 1
> Flanker L3 Driver DEV_LAYER_3_LOOKUP L3LKP 6
> EDC DEV_PHY PHYS 2
+-----------------------------------------------------------------------+
+----------------+++FRONT PANEL PORT TO ASIC INSTANCE MAP+++------------+
+-----------------------------------------------------------------------+
FP port | PHYS | MAC_0 | L2LKP | L3LKP | QUEUE |SWICHF
1 0 0 0 0 0
!--- The L2KLP for both ports is 0, so both belong to instance 0.
2 0 0 0 0 0
3 1 1 1 1 0
4 1 1 1 1 0
5 0 2 2 2 2 0
6 0 2 2 2 2 0
7 1 3 3 3 3 0
8 1 3 3 3 3 0
9 4 4 4 4 0
10 4 4 4 4 0
11 5 5 5 5 0
12 5 5 5 5 0
+-----------------------------------------------------------------------+
+-----------------------------------------------------------------------+ - Select the instance, set the trigger, and begin the capture. It is important to understand, however, that there are many options that can be used with the ELAM trigger:
module-3# elam asic flanker instance 0
These two options are important if you want to include the DBUS in the capture (the packet that is received by the switch). This is the raw packet that is not subjected to a lookup. The RBUS shows the lookup results in the hardware for a DBUS. For a complete ELAM and analysis, you must capture both the RBUS and the DBUS.
module-3(fln-elam)# layer2
module-3(fln-l2-elam)# trigger ?
dbus Pre L2 BUS
rbus Post L2 BUS
-----SNIP-----
The next output shows the types of packets that you can capture with the DBUS option. In this example, the Internet Protocol Version 4 (IPv4) packet is selected:
module-3(fln-l2-elam)# trigger dbus ?
Here are some additional options that you can choose to use:
arp ARP Frame Format
fc Fc hdr Frame Format
ipv4 IPV4 Frame Format
ipv6 IPV6 Frame Format
mpls MPLS
other L2 hdr Frame Format
pup PUP Frame Format
rarp RARP Frame Format
valid On valid packet
module-3(fln-l2-elam)# trigger dbus ipv4 ?
In this example, the if handle is used in order to select a condition for the capture. Most of the options that shown in the next output are based on L2, L3, and L4 headers. The source and destination IP addresses are also used for the capture.
egress Egress packets
!--- Capture packets in egress (outbound from the port).
if If Trigger Condition
ingress Ingress packets
!--- Capture packets in ingress (inbound to the port).
multicast Multicast packet
multicast-replication Multicast replication
module-3(fln-l2-elam)# trigger dbus ipv4 ingress if ?
This output shows the final trigger option:
<CR>
acos Acos
block-capture Capture l2 blocks
bpdu Bpdu
bundle-port Bundle-port
ccc Ccc
copp Copp
da-type Da-type
de-cfi De cfi
destination-index Destination-index
destination-ipv4-address destination ipv4 address
destination-mac-address Destination-mac-address
destination-vif Destination-vif
df df
dfst Dfst
dft Dft
disable-index-learn Disable-index-learn
disable-new-learn Disable-new-learn
dont-forward Dont-forward
dont-learn Dont-learn
dtag-ftag Dtag-ftag
dtag-ttl Dtag-ttl
dti-type-vpnid Dti type vpnid
error Error
erspan-kpa-valid Erspan kpa valid
ff Ff
frag frag
header-type Header type
ib-length-bundle Ib length bundle
ids-check-fail Ids-check-fail
ignore-acli Ignore-acli
ignore-aclo Ignore-aclo
ignore-qosi Ignore-qosi
ignore-qoso Ignore-qoso
inband-flow-creation-deletion Inband-flow-creation-deletion
index-direct Index-direct
inner-cos Inner-cos
inner-de-valid Inner de valid
inner-drop-eligibility Inner-drop-eligibility
ip-da-multicast Ip-da-multicast
ip-multicast Ip-multicast
ip-multicast-control Ip-multicast-control
ipv6 Ipv6
l2 L2
l2-frame-type L2-frame-type
l2-length-check L2 length check
l2lu-mode L2lu-mode
l3-packet-length l3 packet length
l4-protocol l4 protocol
label-count Label count
last-ethertype Last-ethertype
lbl0-eos Lbl0 eos
lbl0-exp Lbl0 exp
lbl0-lbl Lbl0 lbl
lbl0-ttl Lbl0 ttl
lbl0-valid Lbl0 valid
lbl1-exp Lbl1 exp
lbl1-ttl Lbl1 ttl
mac-in-mac-valid Mac-in-mac-valid
mc Mc
md-acos Md acos
md-destination-table-index Md destination table index
md-fwd-only Md fwd only
md-lif Md lif
md-mark-enable Md mark enable
md-multicast-bridge-disable Md multicast bridge disable
md-preserve-acos Md preserve acos
md-qos-group-id Md qos group id
md-replication-packet Md replication packet
md-router-mac Md router mac
md-ttl-err Md-ttl-err
md-version Md version
mf mf
mim-destination-mac-address Mim-destination-mac-address
mim-source-mac-address Mim-source-mac-address
mlh-type Mlh-type
no-stats No-stats
notify-index-learn Notify-index-learn
notify-new-learn Notify-new-learn
null-label-exp Null label exp
null-label-ttl Null label ttl
null-label-valid Null label valid
option option
outer-cos Outer-cos
outer-drop-eligibility Outer-drop-eligibility
ovl-mlh-bndl Ovl mlh bndl
ovl-ulh-bndl Ovl ulh bndl
ovl-ulh-bndl-1 Ovl-ulh-bndl-1
ovl-ulh-bndl-2 Ovl-ulh-bndl-2
packet-length Packet-length
packet-type Packet type
pdt-tag-gt-2 Pdt-tag-gt-2
pdt-tag0 Pdt-tag0
pdt-tag1 Pdt-tag1
pdt-valid Pdt-valid
pdt-value Pdt-value
port-id Port-id
rbh Rbh
rdt Rdt
recir-shim-vxlan-src-peer-id Recir shim vxlan src peer id
recirc-acos Recirc acos
recirc-bypass-ife Recirc bypass ife
recirc-bypass-l2 Recirc bypass l2
recirc-destination-table-index Recirc destination table index
recirc-forward-only Recirc forward only
recirc-l2-tunnel-encap Recirc l2 tunnel encap
recirc-lif Recirc lif
recirc-ls-hash Recirc ls hash
recirc-mark-enable Recirc mark enable
recirc-multicast-bridge-disable Recirc multicast bridge disable
recirc-preserve-acos Recirc preserve acos
recirc-preserve-ls-hash Recirc preserve ls hash
recirc-preserve-rbh Recirc preserve rbh
recirc-qos-group-id Recirc qos group id
recirc-replication-packet Recirc replication packet
recirc-router-mac Recirc router mac
recirc-ttl-err Recirc ttl err
recirc-valid Recirc-valid
recirc-version Recirc version
redirect Redirect
repl-bypass-ife Repl bypass ife
repl-bypass-l2 Repl bypass l2
repl-disable-local-bridge Repl disable local bridge
repl-fwd-only Repl fwd only
repl-l2-tunnel-encap Repl l2 tunnel encap
repl-l2-tunnel-info Repl l2 tunnel info
repl-lif Repl lif
repl-mark-enable Repl mark enable
repl-met-lif Repl met lif
repl-ml3 Repl ml3
repl-preserve-acos Repl preserve acos
repl-preserve-rbh Repl preserve rbh
repl-qos-group-id Repl qos group id
repl-replication-packet Repl replication packet
repl-router-mac Repl router mac
repl-ttl-err Repl ttl err
repl-version Repl version
rf Rf
second-inner-cos Second inner cos
segment-id Segment id
segment-id-valid Segment id valid
sequence-number Sequence-number
sg-tag Sg-tag
shim-valid Shim valid
source-index Source-index
source-ipv4-address source ipv4 address
source-mac-address Source-mac-address
source-vif Source-vif
status-ce-1q Status-ce-1q
status-is-1q Status-is-1q
sup-eid Sup-eid
tos tos
traceroute Traceroute
trig Any of previous elam triggered
trill-encap Trill-encap
ttl ttl
tunnel-bundle Tunnel bundle
tunnel-type Tunnel type
ulh-type Ulh-type
valid VALID
vl Vl
vlan Vlan
vn-p Vn p
vn-valid Vn-valid
vqi Vqi
vqi-valid Vqi-valid
vsl-num Vsl-num
module-3# elam asic flanker instance 0
module-3(fln-elam)# layer2
module-3(fln-l2-elam)# trigger dbus ipv4 ingress if source-ipv4-address 192.168.12.2
destination-ipv4-address 192.168.13.3
module-3(fln-l2-elam)# trigger rbus ingress if trigNote: The RBUS configuration is usually not complex and kept simple.
- In order to check the trigger, enter the status command, start the capture process, and initiate a ping from N7K2 to N7K3 (192.168.12.1 to 192.168.13.3):
module-3(fln-l2-elam)# stat
ELAM Slot 3 instance 0: L2 DBUS Configuration: trigger dbus ipv4 ingress if
source-ipv4-address 192.168.12.2 destination-ipv4-address 192.168.13.3
L2 DBUS: Configured
ELAM Slot 3 instance 0: L2 RBUS Configuration: trigger rbus ingress if trig
L2 RBUS: Configured
module-3(fln-l2-elam)# start
module-3(fln-l2-elam)# status
!--- The status shows as Armed because the process has begun.
ELAM Slot 3 instance 0: L2 DBUS Configuration: trigger dbus ipv4 ingress if
source-ipv4-address 192.168.12.2 destination-ipv4-address 192.168.13.1
L2 DBUS: Armed
ELAM Slot 3 instance 0: L2 RBUS Configuration: trigger rbus ingress if trig
L2 RBUS: Armed
module-3(fln-l2-elam)#
module-3(fln-l2-elam)# status
!--- If the packet is captured, the status shows Triggered.
ELAM Slot 3 instance 0: L2 DBUS Configuration: trigger dbus ipv4 ingress if
source-ipv4-address 192.168.12.2 destination-ipv4-address 192.168.13.3
L2 DBUS: Triggered
ELAM Slot 3 instance 0: L2 RBUS Configuration: trigger rbus ingress if trig
L2 RBUS: Triggered
module-3(fln-l2-elam)# - If the status shows Triggered, then verify whether both the RBUS and DBUS have the same sequence number in order to confirm that they are for the same packet. In this example, 0x55 is used, but the column that shows the sequence number is different:
module-3(fln-l2-elam)# show dbus | in seq
sequence-number : 0x6b vl : 0x0
!--- The sequence number is the same (0x6b).
module-3(fln-l2-elam)# show rbus | in seq
l2-rbus-trigger : 0x1 sequence-number : 0x6b - Enter the show dbus and show rbus commands in order to verify the DBUS and RBUS. Look for the source index in the DBUS command output and the destination index in the RBUS command output:
module-3(fln-l2-elam)# show dbus
cp = 0x1007db4c, buf = 0x1007db4c, end = 0x10089e9c
--------------------------------------------------------------------
Flanker Instance 00 - Capture Buffer On L2 DBUS:
Status(0x0102), TriggerWord(0x000), SampleStored(0x005),CaptureBufferPointer(0x005)
is_l2_egress: 0x0000, data_size: 0x023
[000]: 5902a000 08010000 00000000 0cc01400 00145800 00000000 01800100 00000000
00000000 00000000 003931c8 842850b9 31c88428 50c00000 01ac0000 00000000 00000000
00000000 00000000 00000000 00000005 80005000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00605406 01605406 8180008f f0054608 00000000
Printing packet 0
--------------------------------------------------------------------
L2 DBUS PRS MLH IPV4
--------------------------------------------------------------------
label-count : 0x0 mc : 0x0
null-label-valid : 0x0 null-label-exp : 0x0
null-label-ttl : 0x0 lbl0-vld : 0x0
lbl0-eos : 0x0 lbl0-lbl : 0x0
lbl0-exp : 0x0 lbl0-ttl : 0x0
lbl1-exp : 0x0 lbl1-ttl : 0x0
ipv4 : 0x0 ipv6 : 0x0
l4-protocol : 0x1 df : 0x0
mf : 0x0 frag : 0x0
ttl : 0xff l3-packet-length : 0x54
option : 0x0 tos : 0x0
sup-eid : 0x0 header-type : 0x1
error : 0x0 redirect : 0x0
port-id : 0x0 last-ethertype : 0x800
l2-frame-type : 0x0 da-type : 0x0
packet-type : 0x0 l2-length-check : 0x0
ip-da-multicast : 0x0 ip-multicast : 0x0
ip-multicast-control: 0x0 ids-check-fail : 0x0
traceroute : 0x0 outer-cos : 0x0
inner-cos : 0x0 vqi-valid : 0x0
vqi : 0x0 packet-length : 0x66
vlan : 0xa destination-index : 0x0
source-index : 0xa2c bundle-port : 0x0
acos : 0x0 outer-drop-eligibility: 0x0
inner-drop-eligibility: 0x0 sg-tag : 0x0
rbh : 0x0 vsl-num : 0x0
inband-flow-creation-deletion: 0x0 ignore-qoso : 0x0
ignore-qosi : 0x0 ignore-aclo : 0x0
ignore-acli : 0x0 index-direct : 0x0
no-stats : 0x0 dont-forward : 0x0
notify-index-learn : 0x1 notify-new-learn : 0x1
disable-new-learn : 0x0 disable-index-learn : 0x0
dont-learn : 0x0 bpdu : 0x0
ff : 0x0 rf : 0x0
ccc : 0x0 l2 : 0x0
rdt : 0x0 dft : 0x0
dfst : 0x0 status-ce-1q : 0x0
status-is-1q : 0x1 trill-encap : 0x0
mim-valid : 0x0 dtag-ttl : 0x0
dtag-ftag : 0x0 valid : 0x1
erspan-kpa-valid : 0x0 recir-shim-vxlan-src-peer-id: 0x0
vn-valid : 0x0 source-vif : 0x0
destination-vif : 0x0 vn-p : 0x0
sequence-number : 0x6b vl : 0x0
inner-de-valid : 0x0 de-cfi : 0x0
second-inner-cos : 0x0 tunnel-type : 0x0
shim-valid : 0x0
segment-id-valid : 0x0 copp : 0x0
dti-type-vpnid : 0x0 segment-id : 0x0
ib-length-bundle : 0x58000 mlh-type : 0x5
ulh-type : 0x6
source-ipv4-address: 192.168.12.2
destination-ipv4-address: 192.168.13.3
mim-destination-mac-address : 0000.0000.0000
mim-source-mac-address : 0000.0000.0000
destination-mac-address : e4c7.2210.a142
source-mac-address : e4c7.2210.a143
module-3(fln-l2-elam)# show rbus
cp = 0x100a2548, buf = 0x100a2548, end = 0x100ae898
--------------------------------------------------------------------
Flanker Instance 00 - Capture Buffer On L2 RBUS:
Status(0x0102), TriggerWord(0x000), SampleStored(0x005),CaptureBufferPointer(0x005)
is_l2_egress: 0x0000, data_size: 0x018
[000]: 0059d930 0000000c c0000000 03580000 00000000 00000000 0000001f 57b00021
fdfc0000 00000000 02000000 14001402 8b000105 00000000 68200000 00000000 00000000
00000400 00008000 005b0000 00fe0e4c 7220850a 210000a0 000000b6
Printing packet 0
--------------------------------------------------------------------
L2 RBUS INGRESS CONTENT
--------------------------------------------------------------------
pad : 0x16764 valid : 0x1
l2-rbus-trigger : 0x1 sequence-number : 0x6b
rit-ipv4-id : 0x0 ipv4-tunnel-encap : 0x0
rit-mpls-rw : 0x0 ml2-ptr : 0x0
ml3-ptr : 0x0 mark : 0x0
result-cap3 : 0x0 di1-v5-delta-length : 0x0
di1-v5-delta-length-plus: 0x0 di1-v4-delta-length : 0x0
di1-v4-delta-length-plus: 0x0 di2-delta-length : 0x0
di2-delta-length-plus: 0x0 ml2-delta-length : 0x0
ml2-delta-length-plus: 0x0 ml3-delta-length : 0x0
ml3-delta-length-plus: 0x0 s-vector : 0x0
lcpu-ff-valid : 0x0 sup-di-vqi : 0x0
erspan-term-index-dir: 0x0 erspan-buffer-check : 0x0
l2-tunnel-decapped : 0x0 l3-delta-length : 0x0
rit-crc16-valid : 0x1 rit-crc16 : 0xf57b
vntag-p : 0x0 frr-recirc : 0x0
ingress-lif : 0x1 earl-proxy-vld : 0x0
md-di-vld : 0x0 rc : 0x0
segment-id-valid : 0x0 ttl-out : 0xfe
ttl-mid : 0xfe tos-out : 0x0
tos-in : 0x0 orig-vlan1 : 0x0
vlan1 : 0x0 source-peer-id : 0x0
final-ignore-qoso : 0x0 port-id : 0x0
cr-type : 0x1 pup-packet : 0x0
bpdu : 0x0 vdc : 0x0
traceroute : 0x0 de : 0x0
cos : 0x0 inner-drop-eligibility: 0x0
inner-cos : 0x0 acos : 0x0
di-ltl-index : 0x50 l3-multicast-di : 0x50
source-index : 0xa2c vlan : 0x0
index-direct : 0x0 di1-valid : 0x1
vqi : 0x50 di2-valid : 0x0
v5-fpoe-idx : 0x0 di2-fpoe-idx : 0x0
l3-multicast-v5 : 0x0 dft : 0x0
dfst : 0x0 l3-learning-ff : 0x0
result-rbh : 0xd0 di2-cr-type : 0x0
result-2 : 0x1 dtag-ftag : 0x0
dtag-ttl : 0x20 mac-in-mac-op : 0x0
dvif : 0x0 result-cap1 : 0x0
result-cap2 : 0x0 erspan-term : 0x0
erspan-decap : 0x0 dont-learn : 0x0
routed-frame : 0x1 copy-cause : 0x0
l2-copy-cause : 0x0 l3-rit-ptr : 0x5b
sg-tag : 0x0 trill-nh-id : 0x0
ttl-in : 0xfe fc-up : 0x0
up-did : 0x0 did : 0xe4c722
up-sid : 0x0 sid : 0x10a144
shim-l2-tunnel-encap: 0x0 shim-ls-hash : 0x8
shim-rc : 0x0 shim-lif : 0x1
shim-replication-pkt: 0x0 shim-router-mac : 0x1
shim-mark-enable : 0x0 shim-qos-group-id : 0x0
shim-destination-table-index: 0x5b shim-acos-preserve : 0x0
mim-destination-mac-address : 0000.0000.0000
mim-source-mac-address : 0000.0000.0000 - Check the destination index and source index on the Sup:
N7K1# show system internal pixm info ltl 0xa2c
This output confirms that the packet was received on port channel 1 (Po1) and was forwarded via Eth3/5.
PC_TYPE PORT LTL RES_ID LTL_FLAG CB_FLAG MEMB_CNT
------------------------------------------------------------------------------
Normal Po1 0x0a2c 0x16000000 0x00000000 0x00000002 2
Member rbh rbh_cnt
Eth3/2 0x000000f0 0x04
Eth3/1 0x0000000f 0x04
CBL Check States: Ingress: Enabled; Egress: Enabled
VLAN| BD| BD-St | CBL St & Direction:
--------------------------------------------------
1 | 0x15 | INCLUDE_IF_IN_BD | FORWARDING (Both)
10 | 0x19 | INCLUDE_IF_IN_BD | FORWARDING (Both)
Member info
------------------
Type LTL
---------------------------------
PORT_CHANNEL Po1
FLOOD_W_FPOE 0x8019
FLOOD_W_FPOE 0x8015
N7K1# show system internal pixm info ltl 0x50
0x0050 is in DCE/FC pool
Member info
------------------
Type LTL
---------------------------------
PHY_PORT Eth3/5 - Verify the Local Target Logic (LTL) on the module for correct programming:
module-3# show system internal pixmc info ltl-cb ltl 0xa2c
ltl |ltl_type|if_index|lc_type| vdc |v4_fpoe|v5_fpoe| base_fpoe_idx | flag
0x0a2c | 4 | Po1 | 2 | 2 | 0x00 | 0x00 | 0x0000 | 0x0
, local ports:
VDCs the entry is part of:
LTL HW programming info
.......................
----------------------------------------------------
|Index | ec |drop|span_vec|SOM|ucr_fab|
|--------------------------------------------------
|[ a2c]| 1| 0| 0| 0| 0|
| RBH | VQI | PS(INST:LPOE)
|----------------------------------
0, 40 0 : 1
1, 40 0 : 1
2, 40 0 : 1
3, 40 0 : 1
4, 44 0 : 10
5, 44 0 : 10
6, 44 0 : 10
7, 44 0 : 10
8, 0 0 : 1
9, 0 0 : 1
a, 0 0 : 1
b, 0 0 : 1
c, 0 0 : 10
d, 0 0 : 10
e, 0 0 : 10
f, 0 0 : 10
module-3# show system internal pixmc info ltl-cb ltl 0x50
ltl |ltl_type|if_index|lc_type| vdc |v4_fpoe | v5_fpoe| base_fpoe_idx | flag
0x0050 | 5 |Eth3/5 | 2 | 2 | 0x00 | 0x00 | 0x0000 | 0x0
, local ports:
VDCs the entry is part of:
LTL HW programming info
.......................
----------------------------------------------------
|Index | ec |drop|span_vec|SOM|ucr_fab|
|--------------------------------------------------
|[ 50]| 1| 0| 0| 0| 0|
| RBH | VQI | PS
|----------------------------
ALL RBH| 50 | 2 : 1 - Capture the ELAM packet upon egress. In order to capture the packet, send a ping reply from IP address192.168.13.3 to 192.168.12.2. You must set the capture with the egress keyword on the port channel 1 interfaces (e3/1-2). The interfaces belong to instance 0, as previously described.
N7K1# att mo 3
As shown, both the source and destination indexes are a part of the DBUS (unlike that shown in the ingress capture).
Attaching to module 3 ...
To exit type 'exit', to abort type '$.'
module-3# el asic flanker instance 0
module-3(fln-elam)# layer2
module-3(fln-l2-elam)# trigger dbus ipv4 egress if source-ipv4-address 192.168.13.3
destination-ipv4-address 192.168.12.2
module-3(fln-l2-elam)# trigger rbus egress if trig
module-3(fln-l2-elam)# status
ELAM Slot 3 instance 0: L2 DBUS Configuration: trigger dbus ipv4 egress if
source-ipv4-address 192.168.13.3 destination-ipv4-address 192.168.12.2
L2 DBUS: Configured
ELAM Slot 3 instance 0: L2 RBUS Configuration: trigger rbus egress if trig
L2 RBUS: Configured
module-3(fln-l2-elam)# start
module-3(fln-l2-elam)# status
ELAM Slot 3 instance 0: L2 DBUS Configuration: trigger dbus ipv4 egress if
source-ipv4-address 192.168.13.3 destination-ipv4-address 192.168.12.2
L2 DBUS: Armed
ELAM Slot 3 instance 0: L2 RBUS Configuration: trigger rbus egress if trig
L2 RBUS: Armed
module-3(fln-l2-elam)# status
ELAM Slot 3 instance 0: L2 DBUS Configuration: trigger dbus ipv4 egress if
source-ipv4-address 192.168.13.3 destination-ipv4-address 192.168.12.2
L2 DBUS: Triggered
ELAM Slot 3 instance 0: L2 RBUS Configuration: trigger rbus egress if trig
L2 RBUS: Triggered
module-3(fln-l2-elam)#
module-3(fln-l2-elam)# show dbus | in seq
sequence-number : 0x8d vl : 0x3
!--- The sequence number is the same.
module-3(fln-l2-elam)# show rbus | in seq
vl : 0x0 sequence-number : 0x8d
module-3(fln-l2-elam)# show dbus
cp = 0x1007db4c, buf = 0x1007db4c, end = 0x10089e9c
--------------------------------------------------------------------
Flanker Instance 00 - Capture Buffer On L2 DBUS:
Status(0x0102), TriggerWord(0x000), SampleStored(0x005),CaptureBufferPointer(0x005)
is_l2_egress: 0x0000, data_size: 0x023
[000]: 48c22000 08210000 40020800 0cc01414 5800a000 00001a40 01030000 00000000
00000000 00000000 003931c8 842850f9 31c88428 50800000 02358000 00000000 00000000
00000000 00000000 00000000 00000000 00005000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00605406 81e05406 0100008f e0054600 00000000
Printing packet 0
--------------------------------------------------------------------
L2 DBUS PRS MLH IPV4
--------------------------------------------------------------------
label-count : 0x0 mc : 0x0
null-label-valid : 0x0 null-label-exp : 0x0
null-label-ttl : 0x0 lbl0-vld : 0x0
lbl0-eos : 0x0 lbl0-lbl : 0x0
lbl0-exp : 0x0 lbl0-ttl : 0x0
lbl1-exp : 0x0 lbl1-ttl : 0x0
ipv4 : 0x0 ipv6 : 0x0
l4-protocol : 0x1 df : 0x0
mf : 0x0 frag : 0x0
ttl : 0xfe l3-packet-length : 0x54
option : 0x0 tos : 0x0
sup-eid : 0x0 header-type : 0x1
error : 0x0 redirect : 0x0
port-id : 0x1 last-ethertype : 0x800
l2-frame-type : 0x0 da-type : 0x0
packet-type : 0x1 l2-length-check : 0x0
ip-da-multicast : 0x0 ip-multicast : 0x0
ip-multicast-control: 0x0 ids-check-fail : 0x0
traceroute : 0x0 outer-cos : 0x0
inner-cos : 0x0 vqi-valid : 0x1
vqi : 0x40 packet-length : 0x66
vlan : 0xa destination-index : 0xa2c
source-index : 0x50 bundle-port : 0x0
acos : 0x0 outer-drop-eligibility: 0x0
inner-drop-eligibility: 0x0 sg-tag : 0x0
rbh : 0xd2 vsl-num : 0x0
inband-flow-creation-deletion: 0x0 ignore-qoso : 0x0
ignore-qosi : 0x0 ignore-aclo : 0x0
ignore-acli : 0x0 index-direct : 0x0
no-stats : 0x0 dont-forward : 0x0
notify-index-learn : 0x1 notify-new-learn : 0x0
disable-new-learn : 0x0 disable-index-learn : 0x0
dont-learn : 0x0 bpdu : 0x0
ff : 0x0 rf : 0x1
ccc : 0x4 l2 : 0x0
rdt : 0x0 dft : 0x0
dfst : 0x0 status-ce-1q : 0x0
status-is-1q : 0x0 trill-encap : 0x0
mim-valid : 0x0 dtag-ttl : 0x0
dtag-ftag : 0x0 valid : 0x1
erspan-kpa-valid : 0x0 recir-shim-vxlan-src-peer-id: 0x0
vn-valid : 0x0 source-vif : 0x0
destination-vif : 0x0 vn-p : 0x0
sequence-number : 0x8d vl : 0x3
inner-de-valid : 0x0 de-cfi : 0x0
second-inner-cos : 0x0 tunnel-type : 0x0
shim-valid : 0x0
segment-id-valid : 0x0 copp : 0x0
dti-type-vpnid : 0x0 segment-id : 0x0
ib-length-bundle : 0x0 mlh-type : 0x5
ulh-type : 0x6
source-ipv4-address: 192.168.13.3
destination-ipv4-address: 192.168.12.2
mim-destination-mac-address : 0000.0000.0000
mim-source-mac-address : 0000.0000.0000
destination-mac-address : e4c7.2210.a143
source-mac-address : e4c7.2210.a142
module-3(fln-l2-elam)# show rbus
The source and destination IP addresses are correct, as decoded after the ingress ELAM packet capture; however, the direction is definitely opposite when compared to the ingress ELAM, as the return traffic is captured.
cp = 0x100a2548, buf = 0x100a2548, end = 0x100ae898
--------------------------------------------------------------------
Flanker Instance 00 - Capture Buffer On L2 RBUS:
Status(0x1102), TriggerWord(0x000), SampleStored(0x008),CaptureBufferPointer(0x000)
is_l2_egress: 0x0001, data_size: 0x018
[000]: 0048ea00 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 0c000000 00000000 04014008 00005000 00000000
00000726 3910850a 1b931c88 42850800 00000000 00000000 0000008d
Printing packet 0
--------------------------------------------------------------------
L2 RBUS EGRESS CONTENT
--------------------------------------------------------------------
pad : 0x0 valid : 0x1
trig : 0x1 reserved : 0x0
vn-tag-p : 0x0 cbl-vlan-valid : 0x0
vft-hop-count : 0x0 vft-vsan : 0x0
vft-up : 0x0 vft-valid : 0x0
copp : 0x0 segment-id-valid : 0x0
segment-id-23 : 0x0 vsl-num : 0x0
inner-cos : 0x0 inner-drop-eligibility: 0x0
cos : 0x0 drop-eligibility : 0x0
dce-mode : 0x0 flood-to-bd : 0x0
pt-bit-en : 0x1 cpu-port : 0x0
vlan-id : 0xa ip-tos : 0x0
result-rbh : 0x1 met-ptr : 0x2000
packet-type : 0x1 sg-tag : 0x0
dtag-ftag : 0x0 vdc : 0x0
vn-tag-src-vif : 0x0 vn-tag-dst-vif : 0x0
vn-tag-l : 0x0 dc3-tr : 0x0
vl : 0x0 sequence-number : 0x8d
destination-mac-valid: 0x0
source-mac-valid: 0x0
mim-destination-mac-address : 0000.0000.0000
destination-mac-address : e4c7.2210.a143
source-mac-address : e4c7.2210.a142
mim-source-mac-address : 0000.0000.0000 - Check the Color-Based Logic (CBL) for module 3 of port channel 1 on N7K1 in order to determine whether VLAN 10 forwards traffic across it. The CBL is a per-physical interface based logic, so you should enter the member interface number of port channel 1 on N7K1, not the port channel number. In the next output, you can see that VLAN 10 forwards it as expected.
The CBL is used in order to determine the Spanning Tree Protocol (STP) state of a port within the hardware. It is possible that the interface shows the forwarding when you check the STP for a VLAN on the Sup, but the module blocks the traffic.Note: You must check the CBL individually for both of the member interfaces (e3/1 and e3/2).
module-3# show hardware internal mac port 1 table cbl vlan
--------------------------------------------------------------------------------
| INGRESS |
| Disabled State | 0,2-9,11-4031,4036-4095 |
| Forwarding State | 1,10,4032-4035 |
| Blocked State | |
| Learning State | |
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
| EGRESS |
| Disabled State | 0,2-9,11-4031,4036-4095 |
| Forwarding State | 1,10,4032-4035 |
| Blocked State | |
| Learning State | |
--------------------------------------------------------------------------------Note: The previous command is for port channel 1 (module 3 is on e3/1).
module-3# show hardware internal mac port 2 table cbl vlan
--------------------------------------------------------------------------------
| INGRESS |
| Disabled State | 0,2-9,11-4031,4036-4095 |
| Forwarding State | 1,10,4032-4035 |
| Blocked State | |
| Learning State | |
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
| EGRESS |
| Disabled State | 0,2-9,11-4031,4036-4095 |
| Forwarding State | 1,10,4032-4035 |
| Blocked State | |
| Learning State | |
--------------------------------------------------------------------------------Note: Similarly, this command checks the CBL for port channel 2 (e3/2).
Troubleshoot ELAM on F3 Series Modules with Breakout Cables
The ELAM procedure for an F3 Series module when a breakout cable is connected does not differ from the ELAM procedures on a regular module port. However, there are some changes in regards to the verification of the Port Index Manager (PIXM) during attempts to convert the index to the front panel number, in which case the interfaces are received from the breakout cable.
Here is the topology that is used for the examples throughout this section:
+------------+ BreakOut Cable +-----------+
| | e3/8/1-4 te1/1| |
| | +-----------+ |
| |e3/8 | 1/2 | 4500-X |
| N7K3 +----------------------+ |
| | Po2 | 1/3 | |
| | +-----------+ |
| | | 1/4 | |
+------------+ +-----------+-----------+
Vlan20 SVI Vlan20 SVI
IP add 192.168.20.3 IP add 192.168.20.1
For this example, a breakout cable is connected to the Ethernet interface 3/8, which breaks the 40-Gigabit port into four 10-Gigabit ports. The required configuration is provided in this section for reference.
N7K3(config)# interface breakout module 3 port 8 map 10g-4x
N7K3(config)# show interface brief
---SNIP---
---------------------------------------------------------------
Ethernet VLAN Type Mode Status Reason Speed Port
Interface Ch #
---------------------------------------------------------------
Eth3/7 -- eth routed up none 40G(D) --
Eth3/8/1 1 eth trunk up none 10G(D) 2
!--- From 3/8/1 to 3/8/4.
Eth3/8/2 1 eth trunk up none 10G(D) 2
Eth3/8/3 1 eth trunk up none 10G(D) 2
Eth3/8/4 1 eth trunk up none 10G(D) 2
In the previous output, you can see that Ethernet interface 3/7 is still a 40-Gigabit port; however, Ethernet interface 3/8 is now broken into four 10-Gigabit ports, which can be configured individually:
N7K3# show run interface e3/8/1 - 4
!Command: show running-config interface Ethernet3/8/1-4
!Time: Mon May 4 01:46:28 2015
version 6.2(8a)
interface Ethernet3/8/1
switchport
switchport mode trunk
switchport trunk allowed vlan 10,20
no shutdown
interface Ethernet3/8/2
switchport
switchport mode trunk
switchport trunk allowed vlan 30,40
no shutdown
interface Ethernet3/8/3
switchport
switchport mode trunk
switchport trunk allowed vlan 50
no shutdown
interface Ethernet3/8/4
switchport
switchport mode trunk
no shutdown
Begin the packet capture from the N7K3 Switched Virtual Interface (SVI) 20 IP address (192.168.20.3) to the 4500 SVI 20 IP address (192.168.20.1). The packet will be captured on N7K3 upon egress to 4500, and the reply is sent from 4500 to N7K3.
As described in the previous section, you must have knowledge of the flanker instance in order to apply the trigger. This output shows the attachment of module 3:
N7K3# attach module 3
Attaching to module 3 ...
To exit type 'exit', to abort type '$.'
module-3# show hardware internal dev
dev-port-map dev-version
module-3# show hardware internal dev-port-map
--------------------------------------------------------------
CARD_TYPE: 12 port 40G
>Front Panel ports:12
--------------------------------------------------------------
Device name Dev role Abbr num_inst:
--------------------------------------------------------------
> Flanker Eth Mac Driver DEV_ETHERNET_MAC MAC_0 6
> Flanker Fwd Driver DEV_LAYER_2_LOOKUP L2LKP 6
> Flanker Xbar Driver DEV_XBAR_INTF XBAR_INTF 6
> Flanker Queue Driver DEV_QUEUEING QUEUE 6
> Sacramento Xbar ASIC DEV_SWITCH_FABRIC SWICHF 1
> Flanker L3 Driver DEV_LAYER_3_LOOKUP L3LKP 6
> EDC DEV_PHY PHYS 2
+-----------------------------------------------------------------------+
+----------------+++FRONT PANEL PORT TO ASIC INSTANCE MAP+++------------+
+-----------------------------------------------------------------------+
FP port | PHYS | MAC_0 | L2LKP | L3LKP | QUEUE |SWICHF
1 0 0 0 0 0
2 0 0 0 0 0
3 1 1 1 1 0
4 1 1 1 1 0
5 0 2 2 2 2 0
6 0 2 2 2 2 0
7 1 3 3 3 3 0
8 1 3 3 3 3 0
!--- The port 8 L2LKP column shows a value of 3.
9 4 4 4 4 0
10 4 4 4 4 0
11 5 5 5 5 0
12 5 5 5 5 0
+-----------------------------------------------------------------------+
+-----------------------------------------------------------------------+
In this output, port 8 is on flanker instance 3. Now that you know the instance, you can place the trigger via the source and destination IP addresses. Because you will capture the ping request from N7K3 to 4500, it will be an egress ELAM.
module-3# elam asic flanker instance 3
module-3(fln-elam)# layer2
module-3(fln-l2-elam)# trigger dbus ipv4 egress if source-ipv4-address 192.168.20.3
destination-ipv4-address 192.168.20.1
module-3(fln-l2-elam)# trigger rbus egress if trig
module-3(fln-l2-elam)# status
ELAM Slot 3 instance 3: L2 DBUS Configuration: trigger dbus ipv4 egress if
source-ipv4-address 192.168.20.3 destination-ipv4-address 192.168.20.1
L2 DBUS: Configured
ELAM Slot 3 instance 3: L2 RBUS Configuration: trigger rbus egress if trig
L2 RBUS: Configured
module-3(fln-l2-elam)# start
module-3(fln-l2-elam)# status
ELAM Slot 3 instance 3: L2 DBUS Configuration: trigger dbus ipv4 egress if
source-ipv4-address 192.168.20.3 destination-ipv4-address 192.168.20.1
L2 DBUS: Armed
ELAM Slot 3 instance 3: L2 RBUS Configuration: trigger rbus egress if trig
L2 RBUS: Armed
The ping is initiated from N7K3 to 4500:
N7K3# ping 192.168.20.1
PING 192.168.20.1 (192.168.20.1): 56 data bytes
36 bytes from 192.168.20.3: Destination Host Unreachable
Request 0 timed out
64 bytes from 192.168.20.1: icmp_seq=1 ttl=254 time=6.49 ms
64 bytes from 192.168.20.1: icmp_seq=2 ttl=254 time=6.518 ms
64 bytes from 192.168.20.1: icmp_seq=3 ttl=254 time=7.936 ms
64 bytes from 192.168.20.1: icmp_seq=4 ttl=254 time=7.945 ms
--- 192.168.20.1 ping statistics ---
5 packets transmitted, 4 packets received, 20.00% packet loss
round-trip min/avg/max = 6.49/7.222/7.945 ms
Here is the ELAM status:
module-3(fln-l2-elam)# status
ELAM Slot 3 instance 3: L2 DBUS Configuration: trigger dbus ipv4 egress if
source-ipv4-address 192.168.20.3 destination-ipv4-address 192.168.20.1
L2 DBUS: Triggered
ELAM Slot 3 instance 3: L2 RBUS Configuration: trigger rbus egress if trig
L2 RBUS: Triggered
Verify that the sequence numbers are the same:
module-3(fln-l2-elam)# show dbus | in seq
sequence-number : 0x27 vl : 0x3
module-3(fln-l2-elam)# show rbus | in seq
vl : 0x0 sequence-number : 0x27
The sequence numbers are the same. Now you can check the DBUS and RBUS information:
module-3(fln-l2-elam)# show dbus
cp = 0x1011033c, buf = 0x1011033c, end = 0x1011c68c
--------------------------------------------------------------------
Flanker Instance 03 - Capture Buffer On L2 DBUS:
Status(0x0102), TriggerWord(0x000), SampleStored(0x004),CaptureBufferPointer(0x004)
is_l2_egress: 0x0000, data_size: 0x023
[000]: 4c1ea000 20a10000 40021040 0cc02801 04080000 00000000 08100000 00000000
00000000 00000000 003c1fc1 8732dff9 31c88428 51000000 009d8000 00000000 00000000
00000000 00000000 00000000 00000000 00005000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 0060540a 01e0540a 0080008f f0054608 00000000
Printing packet 0
--------------------------------------------------------------------
L2 DBUS PRS MLH IPV4
--------------------------------------------------------------------
label-count : 0x0 mc : 0x0
null-label-valid : 0x0 null-label-exp : 0x0
null-label-ttl : 0x0 lbl0-vld : 0x0
lbl0-eos : 0x0 lbl0-lbl : 0x0
lbl0-exp : 0x0 lbl0-ttl : 0x0
lbl1-exp : 0x0 lbl1-ttl : 0x0
ipv4 : 0x0 ipv6 : 0x0
l4-protocol : 0x1 df : 0x0
mf : 0x0 frag : 0x0
ttl : 0xff l3-packet-length : 0x54
option : 0x0 tos : 0x0
sup-eid : 0x1 header-type : 0x0
error : 0x0 redirect : 0x0
port-id : 0x5 last-ethertype : 0x800
l2-frame-type : 0x0 da-type : 0x0
packet-type : 0x1 l2-length-check : 0x0
ip-da-multicast : 0x0 ip-multicast : 0x0
ip-multicast-control: 0x0 ids-check-fail : 0x0
traceroute : 0x0 outer-cos : 0x0
inner-cos : 0x0 vqi-valid : 0x1
vqi : 0x82 packet-length : 0x66
vlan : 0x14 destination-index : 0x82
source-index : 0x400 bundle-port : 0x0
acos : 0x0 outer-drop-eligibility: 0x0
inner-drop-eligibility: 0x0 sg-tag : 0x0
rbh : 0x0 vsl-num : 0x0
inband-flow-creation-deletion: 0x0 ignore-qoso : 0x0
ignore-qosi : 0x0 ignore-aclo : 0x0
ignore-acli : 0x0 index-direct : 0x1
no-stats : 0x0 dont-forward : 0x0
notify-index-learn : 0x0 notify-new-learn : 0x0
disable-new-learn : 0x0 disable-index-learn : 0x0
dont-learn : 0x1 bpdu : 0x0
ff : 0x0 rf : 0x0
ccc : 0x0 l2 : 0x0
rdt : 0x0 dft : 0x0
dfst : 0x0 status-ce-1q : 0x0
status-is-1q : 0x0 trill-encap : 0x0
mim-valid : 0x0 dtag-ttl : 0x0
dtag-ftag : 0x0 valid : 0x1
erspan-kpa-valid : 0x0 recir-shim-vxlan-src-peer-id: 0x0
vn-valid : 0x0 source-vif : 0x0
destination-vif : 0x0 vn-p : 0x0
sequence-number : 0x27 vl : 0x3
inner-de-valid : 0x0 de-cfi : 0x0
second-inner-cos : 0x0 tunnel-type : 0x0
shim-valid : 0x0
segment-id-valid : 0x0 copp : 0x0
dti-type-vpnid : 0x0 segment-id : 0x0
ib-length-bundle : 0x0 mlh-type : 0x5
ulh-type : 0x6
source-ipv4-address: 192.168.20.3
destination-ipv4-address: 192.168.20.1
mim-destination-mac-address : 0000.0000.0000
mim-source-mac-address : 0000.0000.0000
destination-mac-address : f07f.061c.cb7f
source-mac-address : e4c7.2210.a144
module-3(fln-l2-elam)#
module-3(fln-l2-elam)#
module-3(fln-l2-elam)#
module-3(fln-l2-elam)# show rbus
cp = 0x10134d38, buf = 0x10134d38, end = 0x10141088
--------------------------------------------------------------------
Flanker Instance 03 - Capture Buffer On L2 RBUS:
Status(0x1102), TriggerWord(0x000), SampleStored(0x008),CaptureBufferPointer(0x000)
is_l2_egress: 0x0001, data_size: 0x018
[000]: 004c4780 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 0c001000 00000000 80028010 00009000 00000000
00000783 f830e65b fb931c88 42851000 00000000 00000000 00000027
Printing packet 0
--------------------------------------------------------------------
L2 RBUS EGRESS CONTENT
--------------------------------------------------------------------
pad : 0x0 valid : 0x1
trig : 0x1 reserved : 0x0
vn-tag-p : 0x1 cbl-vlan-valid : 0x0
vft-hop-count : 0x0 vft-vsan : 0x0
vft-up : 0x0 vft-valid : 0x0
copp : 0x0 segment-id-valid : 0x0
segment-id-23 : 0x0 vsl-num : 0x0
inner-cos : 0x0 inner-drop-eligibility: 0x0
cos : 0x0 drop-eligibility : 0x0
dce-mode : 0x0 flood-to-bd : 0x0
pt-bit-en : 0x20 cpu-port : 0x0
vlan-id : 0x14 ip-tos : 0x0
result-rbh : 0x2 met-ptr : 0x4000
packet-type : 0x1 sg-tag : 0x0
dtag-ftag : 0x0 vdc : 0x0
vn-tag-src-vif : 0x0 vn-tag-dst-vif : 0x0
vn-tag-l : 0x0 dc3-tr : 0x0
vl : 0x0 sequence-number : 0x27
destination-mac-valid: 0x0
source-mac-valid: 0x0
mim-destination-mac-address : 0000.0000.0000
destination-mac-address : f07f.061c.cb7f
source-mac-address : e4c7.2210.a144
mim-source-mac-address : 0000.0000.0000
Convert the destination and source indexes into front panel ports in order to confirm the flow:
N7K3# show system internal pixm info ltl 0x400
0x0400 is in SUP In-band LTL range
This output shows the source index. You know it is correct because of the pings that come to N7K3 from the Sup. The next output shows the egress interface (e3/8/1), which is one of the two interfaces on the N7K that allows VLAN 20. The other interface is e3/8/4, which is blocked on 4500 due to the STP.
N7K3# show system internal pixm info ltl 0x82
0x0082 is in DCE/FC pool
Member info
------------------
Type LTL
---------------------------------
PHY_PORT Eth3/8/1
FLOOD_W_FPOE 0x8039
FLOOD_W_FPOE 0x803f
Verify the CBL for the ports that have been created with the breakout cable on the N7K. In order to check the CBL, you must have the hardware port numbers for all of the newly-formed ports.
Note: The interface e3/8 does not exist on the switch. Only the newly-formed ports appear.
N7K3# show interface e3/8
^
% Incomplete command at '^' marker.
N7K3#
Because the breakout cable is used and the e3/8 interface is non-existent on the switch, the calculation that is used in order to obtain the hardware port number changes. For any module that supports breakout, the hardware port numbering is different. You should first check whether a port supports breakout:
N7K3# show int e3/7 capabilities
Ethernet3/7
Model: N7K-F312FQ-25
Type (SFP capable): QSFP-40G-CR4
Speed: 10000,40000
Duplex: full
---SNIP---
PFC capable: yes
Breakout capable: yes
As shown, port e3/7 supports breakout, which means that its bandwidth can be broken into four 10-Gigabit ports. Similarly, other F3 Series modules that have 100-Gigabit ports can be broken into ten ports each with 10-Gigabits, or three ports with 40-Gigabits with oversubscription. This depends on the module.
Since the F3 Series module in this example has 40-Gigabit ports, and each port can be broken into four ports each, the hardware port numbers for each port is 0-3, 4-7, 8-11...40-43, 44-47 on a zero-based scale. If you have a breakout cable on a port for the first example, then its hardware port numbering will be 0, 1, 2 and 3. If you do not have a breakout cable, then its hardware port numbering will be 0 (1, 2, and 3 will not be active).
As the parent port is e3/8, its hardware port number will be 28 if it is used without the breakout cable, and it will be 28, 29, 30, and 31 if it is used with the breakout cable. This command output shows the active hardware ports (zero based):
N7K3# show system internal ifindex info mod 3
Init DB dump follows:
module_num_bitmask = 0x3ffff
Slot:3, Proc:1, breakout_factor:0, sw_card_id:0, active_cfg_ports:, broken_fp_po
rts:
Slot:3, Proc:2, breakout_factor:4, sw_card_id:155, active_cfg_ports:0,4,8,12,16,
20,24,28-32,36,40,44, broken_fp_ports:28
Lookup DB dump follows:
Slot:3, breakout_factor:4
The broken port hardware port number is 28, which is now split into four (28-32). Now you can attach module 3 and check the CBL in the hardware:
N7K3# attach module 3
Attaching to module 3 ...
To exit type 'exit', to abort type '$.'
module-3#
The F3 Series module expects the port number to be formatted in accordance with a one-based scale. For this reason, you should enter 29, 30, 31, and 32:
module-3# show hardware internal mac port ?
<1-96> Port number (1-based)
!--- This is context sensitive, so it helps to say the port number is 1-based.
Here is the running configuration for Ethernet interface 3/8/1 in order to check and confirm the VLAN forwarding state:
interface Ethernet3/8/1
switchport
switchport mode trunk
switchport trunk allowed vlan 10,20
no shutdown
module-3# show hardware internal mac port 29 table cbl vlan
--------------------------------------------------------------------------------
| INGRESS |
| Disabled State | 0,2-9,11-19,21-4031,4036-4095 |
| Forwarding State | 10,20,4032-4035 |
| Blocked State | 1 |
| Learning State | |
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
| EGRESS |
| Disabled State | 0,2-9,11-19,21-4031,4036-4095 |
| Forwarding State | 10,20,4032-4035 |
| Blocked State | 1 |
| Learning State | |
--------------------------------------------------------------------------------
Here is the running configuration for Ethernet interface 3/8/2 in order to check and confirm the VLAN forwarding state:
interface Ethernet3/8/2
switchport
switchport mode trunk
switchport trunk allowed vlan 30,40
no shutdown
module-3# show hardware internal mac port 30 table cbl vlan
--------------------------------------------------------------------------------
| INGRESS |
| Disabled State | 0,2-29,31-39,41-4031,4036-4095 |
| Forwarding State | 30,40,4032-4035 |
| Blocked State | 1 |
| Learning State | |
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
| EGRESS |
| Disabled State | 0,2-29,31-39,41-4031,4036-4095 |
| Forwarding State | 30,40,4032-4035 |
| Blocked State | 1 |
| Learning State | |
--------------------------------------------------------------------------------
Here is the running configuration for Ethernet interface 3/8/3 in order to check and confirm the VLAN forwarding state:
interface Ethernet3/8/3
switchport
switchport mode trunk
switchport trunk allowed vlan 50
no shutdown
module-3# show hardware internal mac port 31 table cbl vlan
--------------------------------------------------------------------------------
| INGRESS |
| Disabled State | 0,2-49,51-4031,4036-4095 |
| Forwarding State | 50,4032-4035 |
| Blocked State | 1 |
| Learning State | |
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
| EGRESS |
| Disabled State | 0,2-49,51-4031,4036-4095 |
| Forwarding State | 50,4032-4035 |
| Blocked State | 1 |
| Learning State | |
--------------------------------------------------------------------------------
Here is the running configuration for Ethernet interface 3/8/4 in order to check and confirm the VLAN forwarding state (all configured VLANs are allowed):
interface Ethernet3/8/4
switchport
switchport mode trunk
no shutdown
module-3# show hardware internal mac port 32 table cbl vlan
--------------------------------------------------------------------------------
| INGRESS |
| Disabled State | 0,2-9,11-19,21-29,31-39,41-49,51-59,61-669,671-4031 |
| Disabled State | 4036-4095 |
| Forwarding State | 1,20,30,40,50,60,670,4032-4035 |
| Blocked State | 10 |
| Learning State | |
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
| EGRESS |
| Disabled State | 0,2-9,11-19,21-29,31-39,41-49,51-59,61-669,671-4031 |
| Disabled State | 4036-4095 |
| Forwarding State | 1,20,30,40,50,60,670,4032-4035 |
| Blocked State | 10 |
| Learning State | |
--------------------------------------------------------------------------------
The CBL shows that the correct VLANs are forwarded.
You can use the show hardware internal error module <module number> command in order to obtain the hardware port number. This command is useful when you must check any internal drops that do not appear in the show interface x/y command output. Here is an example:
N7K2# show hardware internal errors module 3
---SNIP---
Instance:1
Cntr Name Value Ports
----- ---- ----- -----
3836 igr rx pl: cbl drops 0000000000000001 10 -
4636 igr rx pl: cbl drops 0000000000000001 14 -
Instance:2
Cntr Name Value Ports
----- ---- ----- -----
423 igr in upm: pkts with symbol/sequence error rcvd 0000000000000478 18 -
455 igr in upm: pkts with symbol/sequence error rcvd 0000000000000478 17 -
487 igr in upm: pkts with symbol/sequence error rcvd 0000000000000478 19 -
519 igr in upm: pkts with symbol/sequence error rcvd 0000000000000478 20 -
Instance:3
Cntr Name Value Ports
----- ---- ----- -----
423 igr in upm: pkts with symbol/sequence error rcvd 0000000000000745 26 -
455 igr in upm: pkts with symbol/sequence error rcvd 0000000000000745 25 -
487 igr in upm: pkts with symbol/sequence error rcvd 0000000000000745 27 -
519 igr in upm: pkts with symbol/sequence error rcvd 0000000000000745 28 -
550 igr in upm: pkts rcvd, with RCODE violation 0000359810913821 30 -
551 igr in upm: pkts with symbol/sequence error rcvd 0000425092490108 30 -
552 igr in upm: pkts with error 0000000000176136 30 -
582 igr in upm: pkts rcvd, with RCODE violation 0000000000292641 29 -
583 igr in upm: pkts with symbol/sequence error rcvd 0000000000114014 29 -
614 igr in upm: pkts rcvd, with RCODE violation 0000133362265995 31 -
615 igr in upm: pkts with symbol/sequence error rcvd 0000146701474013 31 -
616 igr in upm: pkts with error 0000000000157479 31 -
646 igr in upm: pkts rcvd, with RCODE violation 0000000002160959 32 -
647 igr in upm: pkts with symbol/sequence error rcvd 0000000003722562 32 -
648 igr in upm: pkts with error 0000000000000002 32 -
Contributed by Cisco Engineers
- Mohammed NijamuddinCisco TAC Engineer
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)