Troubleshoot Packet Forwarding for 6500/6880 Instant Access/FEX Design
Available Languages
Contents
Introduction
This document describes how to troubleshoot packet forwarding in Catalyst 6500/6880 Series Instant Access network.
1. Background Information
It is recommended that the user is fully aware about design, VNTAG format, and packet forwarding logic for Instant Access based network.
Additional information about Instant Access architecture for Catalyst 6500/6880 Instant Access can be found at:
Cisco Catalyst Instant Access Solution White Paper
All further analysis has been done for the this topology:
Warning: Steps described here are run only in a lab setup or under direct supervision of a Cisco TAC engineer.
To successfully run some of the commands presented here, enable 'service internal' (hidden command) on the Cisco Catalyst 6500/6880 controller which unlocks some of the hidden/TAC-specific commands:
6880#configure terminal 6880(config)#service internal
Enable full access to FEX (service internal is required) for the specifed number of minutes:
6880#test platform software console fex <fex-id> enable timeout <minutes>
2. Troubleshoot packet flow (FEX -> controller)
In this section, packet flow from the Instant Access / FEX (Fabric Extender) interface (host connected to Gi1/0/1) to the Cisco Catalyst 6500/6880 series controller where fthe inal packet forwarding decision is described.
2.1. FEX
To determine if packets are received on FEX ingress interface (Gi1/0/1), these methods can be used:
2.1.1. Interface counters
Generate the number of packets and verify that the packets are visible on FEX:
6880#show int gi101/1/0/1 | in packets input
101 packets input, 151894 bytes, 0 no buffer
6880#show counters interface gi101/1/0/1 delta | in InPackets|InUcastPkts|InMcastPkts
1. InPackets = 104
3. InUcastPkts = 102
4. InMcastPkts = 2
Note: Based on the design, MAC address table on FEX are always empty as MAC learning happens only on the controller.
2.1.2. Packet capture
If it is required to capture packets from FEX to troubleshoot further, SPAN session can be configured directly on FEX (so that packet replication is executed on FEX itself). The main requirement is that SPAN destination is connected to the same FEX as SPAN source:
6880#attach fex 101 Attach FEX:101 ip:192.168.1.101 Trying 192.168.1.101 ... Open User Access Verification Password: cisco FEX-101>enable Password: cisco FEX-101#conf t FEX-101(config)#monitor session 1 source interface gi1/0/1 FEX-101(config)#monitor session 1 destination interface gi1/0/2
This configuration is different from a scenario in which packet capture (that is, SPAN session) is configured directly on 6880 in this way:
6880#configure terminal 6880(config)#monitor session 1 source interface gi101/1/0/1 6880(config)#monitor session 1 destination interface gi101/1/0/2
When packet capture is configured on FEX, packet replication happens also on FEX itself and packets are observed on the destination port if there is a communication issue between FEX and the controller. If packet capture is configured directly on 6880 (see second example), then packet replication happens on controller itself, and possibly leads to incorrect analysis if there are communication issues between controller and FEX. Using this approach, SPAN destination can be connected to any port / FEX).
This can be confirmed by checking replication SPAN counters on 6880 in both scenarios.
SPAN session configured directly on FEX - all counters equal to 0:
6880#clear platform hardware capacity rewrite-engine counter Clear rewrite-engine counter for both switches [confirm] 6880#sh platform hardware capacity rewrite-engine performance | in SPAN RX 1/5 0 0x210 SPAN RX 0 0 0 1/5 0 0x211 SPAN RX 1 0 0 (...)
SPAN session configured directly on controller - counters correspond to number of replicated packets. Replication happens on the ingress controller module which does not have to belong to the active VSS chassis. (example: chassis 2 / module 5).
6880#clear platform hardware capacity rewrite-engine counter Clear rewrite-engine counter for both switches [confirm] 6880#sh platform hardware capacity rewrite-engine performance | in SPAN RX (...> 2/5 0 0x211 SPAN RX 1 0 0 2/5 1 0x210 SPAN RX 0 0 0 2/5 1 0x211 SPAN RX 1 0 0 2/5 2 0x210 SPAN RX 0 102 0 (...)
2.1.3. Packet redirect to FEX CPU.
An alternative method to SPAN presented in point 2 is to redirect packets to FEX CPU. Create ACL with log statement and attach it on ingress interface (in our scenario: Gi1/0/1) and run debug to capture these packets. This also allows the collection of additional details about the packet forwarding decision on the FEX.
Note: this debug can cause a high-CPU condition on the controller switch as well in case there is a large number of packets.
Note: in case there is a stack of Instant Access switches, debug needs to be run on the switch which is local to monitored port. Log in to the other switch in a stack by running command 'session where switch-no corresponds to switch number (for example, for interface gi2/0/1, switch-no is 2; for interface 5/0/11, switch-no is 5, and so on).
6880#attach fex 101 Attach FEX:101 ip:192.168.1.101 Trying 192.168.1.101 ... Open User Access Verification Password: cisco FEX-101>en Password: cisco FEX-101#conf t FEX-101(config)#access-list 100 permit ip host 10.0.0.2 host 10.0.0.1 log FEX-101(config)#access-list 100 permit ip any any FEX-101(config)#int gi1/0/1 FEX-101(config-if)#ip access-group 100 in ! FEX-101#debug platform cpu-queues logging-q
When the packet is generated by the host, information is added to the logging buffer which provides packet characteristics (for example, source and destination MAC address, source and destination IP addresses), packet dump, and internal packet characteristics.
Jun 7 14:05:23.059: Pak recvd on LOGGING-Q: Local Port Fwding L3If: L2If:GigabitEthernet1/0/1 DI:0x1E19, LT:7, Vlan:10 SrcGPN:1, SrcGID:1, ACLLogIdx:0x1, MacDA:0000.0000.0001, MacSA: 0000.0000.0002 IP_SA:10.0.0.2 IP_DA:10.0.0.1 IP_Proto:1 (FEX-101) TPFFD:F3800001_000A400A_01A00076-00011E19_5F5F0000_00002001 (FEX-101) (FEX-101) Jun 7 14:05:23.059: Raw FEX packet Dump: (FEX-101) Jun 7 14:05:23.059: 00 00 00 00 00 01 00 00 00 00 (FEX-101) Jun 7 14:05:23.059: 00 02 08 00 45 00 00 64 04 D3 (FEX-101) Jun 7 14:05:23.059: 00 00 FF 01 A2 C3 0A 00 00 02 (FEX-101) Jun 7 14:05:23.059: 0A 00 00 01 08 00 42 9D 00 13 (FEX-101) Jun 7 14:05:23.062: 00 00 00 00 00 00 38 96 03 04 (FEX-101)
To verify the decision forwarded, run this check (based on DI - destination index provided in this log, where St1 and St2 corresponds to Stack Ports and Te1/0/1 and Te1/0/2 are uplinks that belong to port-channel):
FEX-101#show platform port-asic dest-map index 0x1E19 Ports: Te1/0/1 St1 Te1/0/2 St2
To determine which port is used from the port-channel toward the controller (its number is confirmed by running command 'show etherchannel summary'), use this approach. (this example is for IP packet, in case of non-IP, MAC addresses are used):
FEX-101#show etherchannel load-balance
EtherChannel Load-Balancing Configuration:
src-dst-ip
EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source XOR Destination MAC address
IPv4: Source XOR Destination IP address
IPv6: Source XOR Destination IP address
!
FEX-101#test etherchannel load-balance interface po1 ip 10.0.0.2 10.0.0.1
Would select Te1/0/2 of Po1The packet that is sent to the controller has VNTAG added that reflects the ingress interface. In order to determine this value, follow this approach (Use VIF+1 for interface Gi1/0/1):
FEX-101#show platform fex ucast-entries vif sw_idb portname GPN handle res_index ==== ========== ====================== ==== ========= ========= 1 0x5CAC278 GigabitEthernet1/0/1 1 0x5 0x30F0000
2.2. Controller
When the packet reaches the controller, it is further handled by the ingress forwarding engine which takes further decision regarding what needs to be done with the packet.
For our example, based on the analysis provided in point 2.1, the packet leaves FEX via interface Te1/0/2. To determine which controller interface is connected on the other end, use this command:
6880#show fex infra | in FEX Uplink|FLGS
FLGS:
Te1/5/13, FLGS: 0x3F
FEX Uplink: Te1/0/1
Te2/5/13, FLGS: 0x1F
FEX Uplink: Te1/0/2The packets are received on port Te2/5/13, therefore, these checks are possible:
2.2.1. MAC Address table
When the packet is received on controller, MAC address learning happens and the host MAC address is learned on all modules (this shows only one example from the active supervisor):
6880#show mac address-table interface gi101/1/0/1
Legend: * - primary entry
age - seconds since last seen
n/a - not available
S - secure entry
R - router's gateway mac address entry
D - Duplicate mac address entry
Displaying entries from active supervisor:
vlan mac address type learn age ports
----+----+---------------+-------+-----+----------+-----------------------------
* 10 0000.0000.0002 dynamic Yes 10 Gi101/1/0/1
(...) 2.2.2. Packet capture
As described in point 2.1.2, packet capture is also configured on controller in this way:
6880#configure terminal 6880(config)#monitor session 1 source interface gi101/1/0/1 6880(config)#monitor session 1 destination interface gi101/1/0/2
If packets are visible on the destination interface, the packet has been successfully sent from FEX to controller.
In addition to standard SPAN session, internal packet capture can be configured so that packets are captured to the internal buffer:
6880(config)#monitor session 1 type capture 6880(config-mon-capture)#source interface gi101/1/0/1 ! 6880#monitor capture start %SPAN-SW1-5-PKTCAP_START: Packet capture session 1 started 6880#show monitor capture buffer 1 IP: s=10.0.0.2 , d=10.0.0.1, len 100 2 IP: s=10.0.0.2 , d=10.0.0.1, len 100 3 IP: s=10.0.0.2 , d=10.0.0.1, len 100 4 IP: s=10.0.0.2 , d=10.0.0.1, len 100 (...)
For more information about this feature, refer to: Mini-protocol analyzer
2.2.3. ELAM
On Catalyst 6500/6880 series, there is built-in internal packet capture that provides further insights into packet forwarding decision that is taken by controller on the packet.
Note: More information about ELAM that is valid for 6500/SUP2T, as well as for 6880 can be found at: Catalyst 6500 Series Switches with Supervisor Engine 2T ELAM Procedure
When the packet is captured, the source of the packet (that is, if it points correctly to FEX interface which is done based on VIF mapping), and the location of where the packet is forwarded is determined.
In this example, the packet is received on standby VSS-chassis:
At first log in,
6880#remote login standby Trying Switch ... Entering CONSOLE for Switch Type "^C^C^C" to end this session User Access Verification Password: 6880-sdby#^e Standby console enabled
Run ELAM:
6880-sdby#show platform capture elam asic eureka slot 5 Assigned asic_desc=eu50 6880-sdby#show platform capture elam trigger master eu50 dbus dbi ingress ipv4 if ip_sa = 10.0.0.2 6880-sdby#show platform capture elam start cap_commands: Default ELAM RBI PB1 added to list
When packets are sent from the host and as they reach ingress forwarding engine (in our case this is eu50 on chassis-2), capture is triggered and we obtain number of data:
6880-sdby#sh platform capture elam status ELAM Mode: local ID# Role ASIC Slot Inst Ver ELAM Status ----- ---- ------- ---- ---- --- --------- ------ eu50 M EUREKA 5 0 1.3 DBI_ING Capture Completed eu50 s EUREKA 5 0 1.3 RBI_PB1 Capture Completed ID# ELAM Trigger ----- --------- ---------- eu50 DBI_ING FORMAT=IP L3_PROTOCOL=IPV4 IP_SA = 10.0.0.2 eu50 RBI_PB1 TRIG=1 ! 6880-sdby#show platform capture elam data | in IP_SA|IP_DA|SMAC|DMAC|SRC_INDEX|DEST_INDEX|data DBUS data: SRC_INDEX ....................... [19] = 0x2000 [Po101[Te1/5/13,Te2/5/13],Gi101/1/0/1] DEST_INDEX ...................... [19] = 0xC [Te1/1/13] DMAC ............................ = 0000.0000.0001 SMAC ............................ = 0000.0000.0002 IP_SA ........................... = 10.0.0.2 IP_DA ........................... = 10.0.0.1 RBUS data: DEST_INDEX ...................... [19] = 0x380 [Switch/Router]
Based on the capture output, the packet with MAC/IP addresses (as shown) has been received on Gi101/1/0/1 (DBUS SRC_INDEX field) on FEX and passed to the controller via Po101 configured on the controller. The decision is to forward this packet to VSS CPU (RBUS DEST_INDEX field).
Note: What is mapped between the VIF and the source index is done by the VNTAG manager and can be checked (in this case, VIF=1 was used when it passed unicast frame from port Gi1/0/1 on FEX; operational state is always 2, type could be unicast or multicast):
6880#test platform software switch virtual vntag_mgr vif-map vif 1 detail VIF INFO: VIF# 1 Type UNICAST VIF LTL# 2000 OperStatus # 2
Troubleshoot packet flow (controller -> FEX)
In this section, packet flow from controller Cisco Catalyst 6500/6880 series ingress forwarding engine, up to Instant Access / FEX (Fabric Extender) interface is described.
3.1. Controller
3.1.1. Packet capture
To determine if the packet is successfully recevied by Catalyst 6500/6880 switch, standard methods to troubleshoot standalone/VSS switches are used. These methods include packet capture (SPAN session, and so on). More information about these tools are found at:
3.1.2. ELAM
In order to determine if the decision for the packet to be forwarded is to sent it to FEX with a valid VIF identifier, packet capture is executed on the controller for the ingress forwarding engine (that is, the forwarding engine which handles the interface on which the packet has been received).
More information about ELAM that is valid for 6500/SUP2T, as well as for 6880 can be found at:
Catalyst 6500 Series Switches with Supervisor Engine 2T ELAM Procedure
In this example, this is the valid ELAM configuration. ('shim' keyword in trigger is used when packet is originated from CPU; for transit traffic this keyword is not used):
6880#show platform capture elam asic eureka slot 5 6880#show platform capture elam trigger master eu50 dbus dbi ingress shim ipv4 if ip_sa = 10.0.0.1 ip_da=10.0.0.2 6880#sh platform capture elam start 6880#sh platform capture elam status ELAM Mode: local ID# Role ASIC Slot Inst Ver ELAM Status ----- ---- ------- ---- ---- --- --------- ------ eu50 M EUREKA 5 0 1.3 DBI_ING Capture Completed eu50 s EUREKA 5 0 1.3 RBI_PB1 Capture Completed ID# ELAM Trigger ----- --------- ---------- eu50 DBI_ING FORMAT=OTHERS SHIM_ETYPE=E8_SHIM_ETYPE ETYPE=IPV4_ETYPE IP_SA = 10.0.0.1 IP_DA=10.0.0.2 eu50 RBI_PB1 TRIG=1
This is the captured result:
6880#show platform capture elam data | in IP_SA|IP_DA|SMAC|DMAC|SRC_INDEX|DEST_INDEX|data DBUS data: SRC_INDEX ....................... [19] = 0x380 [Switch/Router,Po255[Te2/5/1,Te2/5/5,Te2/5/6]] DEST_INDEX ...................... [19] = 0x0 [Te1/1/1] DMAC ............................ = 0000.0000.0002 SMAC ............................ = 0000.0000.0001 IP_SA ........................... = 10.0.0.1 IP_DA ........................... = 10.0.0.2 IP_DATA [224] RBUS data: DEST_INDEX ...................... [19] = 0x2000 [Po101[Te1/5/13,Te2/5/13],Gi101/1/0/1]
The packet has been received from the CPU (Switch/Router entry for DBUS SRC_INDEX) and the forward decision is to send it over Po101 (which is port-channel that connects controller with the FEX) and then through interface Gi1/0/1 on FEX101.
The VIF identifier that used in this case can be checked via command, where the LTL value is taken from RBUS the DEST_INDEX field:
6880#test platform software switch virtual vntag_mgr vif-map ltl 0x2000 detail VIF INFO: VIF# 1 Type UNICAST VIF LTL# 2000 OperStatus # 2
Because the egress interface on the controller is port-channel, these commands can be executed to determine which physical link is used.
6880#show etherchannel load-balance
EtherChannel Load-Balancing Configuration:
src-dst-ip enhanced
mpls label-ip
EtherChannel Load-Balancing Mode on FEX Modules:
src-dst-ip
EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source XOR Destination MAC address
IPv4: Source XOR Destination IP address
IPv6: Source XOR Destination IP address
MPLS: Label or IP
!
6880#test etherchannel load-balance interface po101 ip 10.0.0.1 10.0.0.2
Computed RBH: 0x3
Would select Te1/5/13 of Po1013.2. FEX
Methods to troubleshoot FEX for a scenario where a packet is received from the controller and is to be sent to the FEX host interface match the ones presented in section 2. The only differences mentioned are related with 2.1.3 - packet redirection to FEX CPU.
3.2.1. Packet redirect to FEX CPU.
Refer to section 2.1.3 for detailed explanation of this approach.
In this example, to capture a packet that is received from the controller, this configuration is done on FEX (ACL is attached to a single interface if the interface was first correctly identified as described in point 3.1.2).
6880#attach fex 101 Attach FEX:101 ip:192.168.1.101 Trying 192.168.1.101 ... Open User Access Verification Password: cisco FEX-101>en Password: cisco FEX-101#conf t Enter configuration commands, one per line. End with CNTL/Z. FEX-101(config)#access-list 100 permit ip host 10.0.0.1 host 10.0.0.2 log FEX-101(config)#access-list 100 permit ip any any FEX-101(config)#int te1/0/1 FEX-101(config-if)#ip access-group 100 in FEX-101(config-if)#int te1/0/2 FEX-101(config-if)#ip access-group 100 in FEX-101#debug platform cpu-queues logging-q debug platform cpu-queue logging-q debugging is on
Note: if there is a stack of Instant Access switches, debug is run on the switch which has an uplink to the controller. Log in to another switch in a stack by running command 'session where switch-no corresponds to switch number (for example, for interface gi2/0/1, switch-no is 2, for interface 5/0/11, switch-no is 5, and so on).
When the packet from the controller is successfully sent to FEX, this information is generated in the syslog:
Pak recvd on LOGGING-Q: Local Port Fwding L3If: L2If:TenGigabitEthernet1/0/1 DI:0x1F2A, LT:0, Vlan:10 SrcGPN:456, SrcGID:456, ACLLogIdx:0x1, MacDA:0000.0000.0002, MacSA: 0000.0000.0001 Non (IPv4/IPv6/ARP/RARP) TPFFD:F38001C8_000A400A_01A00080-00011F2A_F5F50000_00002BFD Jun 7 15:37:24.482: Raw FEX packet Dump: Jun 7 15:37:24.482: 00 00 00 00 00 02 00 00 00 00 Jun 7 15:37:24.482: 00 01 89 26 80 01 0B FD 81 00 Jun 7 15:37:24.482: 00 0A 08 00 45 00 00 64 76 87 Jun 7 15:37:24.482: 00 00 FF 01 31 0F 0A 00 00 01 Jun 7 15:37:24.482: 0A 00 00 02 08 00 A9 FF 00 12
12-bits (with their corresponding hex values, refer to VNTAG packet format in Instant Access white-paper) correspond to the destination VIF value that has been passed in the packet, itself. This value (changed to decimal) is further cross-referenced with the point of exit on the FEX.
This command is used to determine point of exit: This takes the unicast VIF into consideration (refer to the note in point 2.2.3 regarding VNTAG manager).
FEX-101#sh platform fex ucast-entries vif sw_idb portname GPN handle res_index ==== ========== ====================== ==== ========= ========= 1 0x5CAC278 GigabitEthernet1/0/1 1 0x5 0x30F0000 2 0x5CAE2E0 GigabitEthernet1/0/2 2 0x6 0x30F0000
(...)
For multicast traffic the logic:
1) Determine the set of egress interfaces for specific multicast VIF.
FEX-101#sh platform fex mcast-entries (...) Entry : 2 ================= Mcast VIF = 3072 : destid = 0x23DF : handle = 0x37 : result_index = 0x4D DestId 23DF details with GPN list index next flags cmi #GPN GPN 0x23DF 0xFFFF 0x00 0x0000 3 1 2 464 (...)
In this example, for VIF=3072, three (3) egress interfaces have been selected with the internal GPN numbers: 1, 2 and 464. In order to translate these internal GPN port numbers into physical interfaces, this command can be used:
FEX-101#show platform pm if-numbers interface gid gpn lpn port slot unit slun port-type lpn-idb gpn-idb ---------------------------------------------------------------------- Gi1/0/1 1 1 1 0/2 1 1 1 local Yes Yes Gi1/0/2 2 2 2 0/1 1 2 2 local Yes Yes (...) Po2 464 464 0 16/0 9 2 2 local No No
These outputs mean that packets that are received with a multicast destination VIF equal to 3072, are forwarded to interfaces: Gi1/0/1, Gi1/0/2 and Port-channel 2.
4. Summary
In case of any issues related with the fex, these command outputs are collected and added to the TAC case when it is being opened:
6880#show tech-support 6880#show tech-support fex infra 6880#attach fex <fex-id> Password: cisco FEX-101>en Password: cisco FEX-101#show tech-support
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
12-May-2022 |
Initial Release |
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)