Contents

Introduction

1. Background Information

2. Troubleshoot packet flow (FEX -> controller)

2.1. FEX

2.1.1. Interface counters

2.1.2. Packet capture

2.1.3. Packet redirect toFEX CPU.

2.2.Controller

2.2.1. MAC Address table

2.2.2. Packet capture

2.2.3. ELAM

Troubleshoot packet flow (controller -> FEX)

3.1. Controller

3.1.1. Packet capture

3.1.2. ELAM

3.2. FEX

3.2.1.Packet redirect toFEX CPU.

4. Summary

Introduction

This document describes how to troubleshoot packet forwarding in Catalyst 6500/6880 Series Instant Access network.

1. Background Information

It is recommended that the user is fully aware about design, VNTAG format, and packet forwarding logic for Instant Access based network.

Additional information about Instant Access architecture for Catalyst 6500/6880 Instant Access can be found at:

Cisco Catalyst Instant Access Solution White Paper

All further analysis has been done for the this topology:

Warning: Steps described here are run only in a lab setup or under direct supervision of a Cisco TAC engineer.

To successfully run some of the commands presented here, enable 'service internal' (hidden command) on the Cisco Catalyst 6500/6880 controller which unlocks some of the hidden/TAC-specific commands:

6880#configure terminal

6880(config)#service internal

Enable full access to FEX (service internal is required) for the specifed number of minutes:

6880#test platform software console fex <fex-id> enable timeout <minutes>

2. Troubleshoot packet flow (FEX -> controller)

In this section, packet flow from the Instant Access / FEX (Fabric Extender) interface (host connected to Gi1/0/1) to the Cisco Catalyst 6500/6880 series controller where fthe inal packet forwarding decision is described.

2.1. FEX

To determine if packets are received on FEX ingress interface (Gi1/0/1), these methods can be used:

2.1.1. Interface counters

Generate the number of packets and verify that the packets are visible on FEX:

6880#show int gi101/1/0/1 | in packets input

101 packets input, 151894 bytes, 0 no buffer

6880#show counters interface gi101/1/0/1  delta | in InPackets|InUcastPkts|InMcastPkts

1.                          InPackets = 104

3.                        InUcastPkts = 102

4.                        InMcastPkts = 2

Note: Based on the design, MAC address table on FEX are always empty as MAC learning happens only on the controller.

2.1.2. Packet capture

If it is required to capture packets from FEX to troubleshoot further, SPAN session can be configured directly on FEX (so that packet replication is executed on FEX itself). The main requirement is that SPAN destination is connected to the same FEX as SPAN source:

6880#attach fex 101

Attach FEX:101 ip:192.168.1.101

Trying 192.168.1.101 ... Open

User Access Verification

Password: cisco

FEX-101>enable

Password: cisco

FEX-101#conf t

FEX-101(config)#monitor session 1 source interface gi1/0/1

FEX-101(config)#monitor session 1 destination interface gi1/0/2

This configuration is different from a scenario in which packet capture (that is, SPAN session) is configured directly on 6880 in this way:

6880#configure terminal

6880(config)#monitor session 1 source interface gi101/1/0/1

6880(config)#monitor session 1 destination interface gi101/1/0/2

When packet capture is configured on FEX, packet replication happens also on FEX itself and packets are observed on the destination port if there is a communication issue between FEX and the  controller. If packet capture is configured directly on 6880 (see second example), then packet replication happens on controller itself, and possibly leads to incorrect analysis if there are communication issues between controller and FEX. Using this approach, SPAN destination can be connected to any port / FEX).

This can be confirmed by checking replication SPAN counters on 6880 in both scenarios.

SPAN session configured directly on FEX - all counters equal to 0:

6880#clear platform hardware capacity rewrite-engine counter

Clear rewrite-engine counter for both switches [confirm]

6880#sh platform hardware capacity rewrite-engine performance  | in SPAN RX

1/5  0       0x210   SPAN RX 0            0                    0

1/5  0       0x211   SPAN RX 1            0                    0

(...)

SPAN session configured directly on controller -  counters correspond to number of replicated packets. Replication happens on the ingress controller module which does not have to belong to the active VSS chassis. (example: chassis 2 / module 5).

6880#clear platform hardware capacity rewrite-engine counter

Clear rewrite-engine counter for both switches [confirm]

6880#sh platform hardware capacity rewrite-engine performance  | in SPAN RX

(...>

2/5  0       0x211   SPAN RX 1            0                    0

2/5  1       0x210   SPAN RX 0            0                    0

2/5  1       0x211   SPAN RX 1            0                    0

2/5  2       0x210   SPAN RX 0            102                  0

(...)

2.1.3. Packet redirect to FEX CPU.

An alternative method to SPAN presented in point 2 is to redirect packets to FEX CPU. Create ACL with log statement and attach it on ingress interface (in our scenario: Gi1/0/1) and run debug to capture these packets. This also allows the collection of additional details about the packet forwarding decision on the FEX.

Note: this debug can cause a high-CPU condition on the controller switch as well in case there is a large number of packets.

Note: in case there is a stack of Instant Access switches, debug needs to be run on the switch which is local to monitored port. Log in to the other switch in a stack by running command 'session where switch-no corresponds to switch number (for example, for interface gi2/0/1, switch-no is 2; for interface 5/0/11, switch-no is 5, and so on).

6880#attach fex 101

Attach FEX:101 ip:192.168.1.101

Trying 192.168.1.101 ... Open

User Access Verification

Password: cisco

FEX-101>en

Password: cisco

FEX-101#conf t

FEX-101(config)#access-list 100 permit ip host 10.0.0.2 host 10.0.0.1 log

FEX-101(config)#access-list 100 permit ip any any

FEX-101(config)#int gi1/0/1

FEX-101(config-if)#ip access-group 100 in

!

FEX-101#debug platform cpu-queues logging-q

When the packet is generated by the host, information is added to the logging buffer which provides packet characteristics (for example, source and destination MAC address, source and destination IP addresses), packet dump, and internal packet characteristics.

Jun  7 14:05:23.059: Pak recvd on LOGGING-Q: Local Port Fwding L3If: L2If:GigabitEthernet1/0/1 DI:0x1E19, LT:7, Vlan:10   SrcGPN:1, SrcGID:1, ACLLogIdx:0x1, MacDA:0000.0000.0001, MacSA: 0000.0000.0002   IP_SA:10.0.0.2 IP_DA:10.0.0.1 IP_Proto:1  (FEX-101)

TPFFD:F3800001_000A400A_01A00076-00011E19_5F5F0000_00002001 (FEX-101)

(FEX-101)

Jun  7 14:05:23.059: Raw FEX packet Dump: (FEX-101)

Jun  7 14:05:23.059: 00 00 00 00 00 01 00 00 00 00  (FEX-101)

Jun  7 14:05:23.059: 00 02 08 00 45 00 00 64 04 D3  (FEX-101)

Jun  7 14:05:23.059: 00 00 FF 01 A2 C3 0A 00 00 02  (FEX-101)

Jun  7 14:05:23.059: 0A 00 00 01 08 00 42 9D 00 13  (FEX-101)

Jun  7 14:05:23.062: 00 00 00 00 00 00 38 96 03 04  (FEX-101)

To verify the decision forwarded, run this check (based on DI - destination index provided in this log, where St1 and St2 corresponds to Stack Ports and Te1/0/1 and Te1/0/2 are uplinks that belong to port-channel):

FEX-101#show platform port-asic dest-map index 0x1E19

Ports: Te1/0/1 St1 Te1/0/2 St2

To determine which port is used from the port-channel toward the controller (its number is confirmed by running command 'show etherchannel summary'), use this approach. (this example is for IP packet, in case of non-IP, MAC addresses are used):

FEX-101#show etherchannel load-balance

EtherChannel Load-Balancing Configuration:

src-dst-ip

EtherChannel Load-Balancing Addresses Used Per-Protocol:

Non-IP: Source XOR Destination MAC address

IPv4: Source XOR Destination IP address

IPv6: Source XOR Destination IP address

!

FEX-101#test etherchannel load-balance interface po1 ip 10.0.0.2 10.0.0.1

Would select Te1/0/2 of Po1

The packet that is sent to the controller has VNTAG added that reflects the ingress interface. In order to determine this value, follow this approach (Use VIF+1 for interface Gi1/0/1):

FEX-101#show platform fex ucast-entries

vif   sw_idb            portname             GPN   handle    res_index

====  ==========  ======================     ====  ========= =========

1     0x5CAC278   GigabitEthernet1/0/1       1     0x5        0x30F0000

2.2. Controller

When the packet reaches the controller, it is further handled by the ingress forwarding engine which takes further decision regarding what needs to be done with the packet.

For our example, based on the analysis provided in point 2.1, the packet leaves FEX via interface Te1/0/2. To determine which controller interface is connected on the other end, use this command:

6880#show fex infra | in FEX Uplink|FLGS

FLGS:

Te1/5/13, FLGS: 0x3F

FEX Uplink: Te1/0/1

Te2/5/13, FLGS: 0x1F

FEX Uplink: Te1/0/2

The packets are received on port Te2/5/13, therefore, these checks are possible:

2.2.1. MAC Address table

When the packet is received on controller, MAC address learning happens and the host MAC address is learned on all modules (this shows only one example from the active supervisor):

6880#show mac address-table interface gi101/1/0/1

Legend: * - primary entry

age - seconds since last seen

n/a - not available

S - secure entry

R - router's gateway mac address entry

D - Duplicate mac address entry

Displaying entries from active supervisor:

vlan   mac address    type   learn    age                 ports

----+----+---------------+-------+-----+----------+-----------------------------

*      10 0000.0000.0002  dynamic  Yes       10     Gi101/1/0/1

(...)

2.2.2. Packet capture

As described in point 2.1.2, packet capture is also configured on controller in this way:

6880#configure terminal

6880(config)#monitor session 1 source interface gi101/1/0/1

6880(config)#monitor session 1 destination interface gi101/1/0/2

If packets are visible on the destination interface, the packet has been successfully sent from FEX to controller.

In addition to standard SPAN session, internal packet capture can be configured so that packets are captured to the internal buffer:

6880(config)#monitor session 1 type capture

6880(config-mon-capture)#source interface gi101/1/0/1

!

6880#monitor capture start

%SPAN-SW1-5-PKTCAP_START: Packet capture session 1 started

6880#show monitor capture buffer

1    IP: s=10.0.0.2 , d=10.0.0.1, len 100

2    IP: s=10.0.0.2 , d=10.0.0.1, len 100

3    IP: s=10.0.0.2 , d=10.0.0.1, len 100

4    IP: s=10.0.0.2 , d=10.0.0.1, len 100

(...)

For more information about this feature, refer to: Mini-protocol analyzer

2.2.3. ELAM

On Catalyst 6500/6880 series, there is built-in internal packet capture that provides further insights into packet forwarding decision that is taken by controller on the packet.

Note: More information about ELAM that is valid for 6500/SUP2T, as well as for 6880 can be found at: Catalyst 6500 Series Switches with Supervisor Engine 2T ELAM Procedure

When the packet is captured, the source of the packet (that is, if it points correctly to FEX interface which is done based on VIF mapping), and the location of where the packet is forwarded is determined.

In this example, the packet is received on standby VSS-chassis:

At first log in,

6880#remote login standby

Trying Switch ...

Entering CONSOLE for Switch

Type "^C^C^C" to end this session

User Access Verification

Password:

6880-sdby#^e

Standby console enabled

Run ELAM:

6880-sdby#show platform capture elam asic eureka slot 5

Assigned asic_desc=eu50

6880-sdby#show platform capture elam trigger master eu50 dbus dbi ingress ipv4 if ip_sa = 10.0.0.2

6880-sdby#show platform capture elam start

cap_commands: Default ELAM RBI PB1 added to list

When packets are sent from the host and as they reach ingress forwarding engine (in our case this is eu50 on chassis-2), capture is triggered and we obtain number of data&colon;

6880-sdby#sh platform capture elam status

ELAM Mode: local

ID#    Role  ASIC     Slot  Inst  Ver  ELAM       Status

-----  ----  -------  ----  ----  ---  ---------  ------

eu50   M     EUREKA   5     0     1.3  DBI_ING    Capture Completed

eu50   s     EUREKA   5     0     1.3  RBI_PB1    Capture Completed

ID#    ELAM       Trigger

-----  ---------  ----------

eu50   DBI_ING    FORMAT=IP L3_PROTOCOL=IPV4 IP_SA = 10.0.0.2

eu50   RBI_PB1    TRIG=1

!

6880-sdby#show platform capture elam data | in IP_SA|IP_DA|SMAC|DMAC|SRC_INDEX|DEST_INDEX|data

DBUS data&colon;

SRC_INDEX ....................... [19] = 0x2000 [Po101[Te1/5/13,Te2/5/13],Gi101/1/0/1]

DEST_INDEX ...................... [19] = 0xC [Te1/1/13]

DMAC ............................ = 0000.0000.0001

SMAC ............................ = 0000.0000.0002

IP_SA ........................... = 10.0.0.2

IP_DA ........................... = 10.0.0.1

RBUS data&colon;

DEST_INDEX ...................... [19] = 0x380 [Switch/Router]

Based on the capture output, the packet with MAC/IP addresses (as shown) has been received on Gi101/1/0/1 (DBUS SRC_INDEX field) on FEX and passed to the controller via Po101 configured on the controller. The decision is to forward this packet to VSS CPU (RBUS DEST_INDEX field).

Note: What is mapped between the VIF and the source index is done by the VNTAG manager and can be checked (in this case, VIF=1 was used when it passed unicast frame from port Gi1/0/1 on FEX; operational state is always 2, type could be unicast or multicast):

6880#test platform software switch virtual vntag_mgr vif-map vif 1 detail

VIF INFO:

VIF#  1

Type  UNICAST VIF

LTL#  2000

OperStatus  #  2

Troubleshoot packet flow (controller -> FEX)

In this section, packet flow from controller Cisco Catalyst 6500/6880 series ingress forwarding engine, up to Instant Access / FEX (Fabric Extender) interface is described.

3.1. Controller

3.1.1. Packet capture

To determine if the packet is successfully recevied by Catalyst 6500/6880 switch, standard methods to troubleshoot standalone/VSS switches are used. These methods include packet capture (SPAN session, and so on). More information about these tools are found at:

SPAN, RSPAN and ERSPAN

3.1.2. ELAM

In order to determine if the decision for the packet to be forwarded is to sent it to FEX with a valid VIF identifier, packet capture is executed on the controller for the ingress forwarding engine (that is, the forwarding engine which handles the interface on which the packet has been received).

More information about ELAM that is valid for 6500/SUP2T, as well as for 6880 can be found at:

Catalyst 6500 Series Switches with Supervisor Engine 2T ELAM Procedure

In this example, this is the valid ELAM configuration. ('shim' keyword in trigger is used when packet is originated from CPU; for transit traffic this keyword is not used):

6880#show platform capture elam asic eureka slot 5

6880#show platform capture elam trigger master eu50 dbus dbi ingress shim ipv4 if ip_sa = 10.0.0.1 ip_da=10.0.0.2

6880#sh platform capture elam start

6880#sh platform capture elam status

ELAM Mode: local

ID#    Role  ASIC     Slot  Inst  Ver  ELAM       Status

-----  ----  -------  ----  ----  ---  ---------  ------

eu50   M     EUREKA   5     0     1.3  DBI_ING    Capture Completed

eu50   s     EUREKA   5     0     1.3  RBI_PB1    Capture Completed

ID#    ELAM       Trigger

-----  ---------  ----------

eu50   DBI_ING    FORMAT=OTHERS SHIM_ETYPE=E8_SHIM_ETYPE                  ETYPE=IPV4_ETYPE IP_SA = 10.0.0.1 IP_DA=10.0.0.2

eu50   RBI_PB1    TRIG=1

This is the captured result:

6880#show platform capture elam data | in IP_SA|IP_DA|SMAC|DMAC|SRC_INDEX|DEST_INDEX|data

DBUS data&colon;

SRC_INDEX ....................... [19] = 0x380 [Switch/Router,Po255[Te2/5/1,Te2/5/5,Te2/5/6]]

DEST_INDEX ...................... [19] = 0x0 [Te1/1/1]

DMAC ............................ = 0000.0000.0002

SMAC ............................ = 0000.0000.0001

IP_SA ........................... = 10.0.0.1

IP_DA ........................... = 10.0.0.2

IP_DATA [224]

RBUS data&colon;

DEST_INDEX ...................... [19] = 0x2000 [Po101[Te1/5/13,Te2/5/13],Gi101/1/0/1]

The packet has been received from the CPU (Switch/Router entry for DBUS SRC_INDEX) and the forward decision is to send it over Po101 (which is port-channel that connects controller with the FEX) and then through interface Gi1/0/1 on FEX101.

The VIF identifier that used in this case can be checked via command, where the LTL value is taken from RBUS the DEST_INDEX field:

6880#test platform software switch virtual vntag_mgr vif-map ltl 0x2000 detail

VIF INFO:

VIF#  1

Type  UNICAST VIF

LTL#  2000

OperStatus  #  2

Because the egress interface on the controller is port-channel, these commands can be executed to determine which physical link is used.

6880#show etherchannel load-balance

EtherChannel Load-Balancing Configuration:

src-dst-ip enhanced

mpls label-ip

EtherChannel Load-Balancing Mode on FEX Modules:

src-dst-ip

EtherChannel Load-Balancing Addresses Used Per-Protocol:

Non-IP: Source XOR Destination MAC address

IPv4: Source XOR Destination IP address

IPv6: Source XOR Destination IP address

MPLS: Label or IP

!

6880#test etherchannel load-balance interface po101 ip 10.0.0.1 10.0.0.2

Computed RBH: 0x3

Would select Te1/5/13 of Po101

3.2. FEX

Methods to troubleshoot FEX for a scenario where a packet is received from the controller and is to be sent to the FEX host interface match the ones presented in section 2. The only differences mentioned are related with 2.1.3 - packet redirection to FEX CPU.

3.2.1. Packet redirect to FEX CPU.

Refer to section 2.1.3 for detailed explanation of this approach.

In this example, to capture a packet that is received from the controller, this configuration is done on FEX (ACL is attached to a single interface if the interface was first correctly identified as described in point 3.1.2).

6880#attach fex 101

Attach FEX:101 ip:192.168.1.101

Trying 192.168.1.101 ... Open

User Access Verification

Password: cisco

FEX-101>en

Password: cisco

FEX-101#conf t

Enter configuration commands, one per line.  End with CNTL/Z.

FEX-101(config)#access-list 100 permit ip host 10.0.0.1 host 10.0.0.2 log

FEX-101(config)#access-list 100 permit ip any any

FEX-101(config)#int te1/0/1

FEX-101(config-if)#ip access-group 100 in

FEX-101(config-if)#int te1/0/2

FEX-101(config-if)#ip access-group 100 in

FEX-101#debug platform cpu-queues logging-q

debug platform cpu-queue logging-q debugging is on

Note: if there is a stack of Instant Access switches, debug is run on the switch which has an uplink to the controller. Log in to another switch in a stack by running command 'session where switch-no corresponds to switch number (for example, for interface gi2/0/1, switch-no is 2, for interface 5/0/11, switch-no is 5, and so on).

When the packet from the controller is successfully sent to FEX, this information is generated in the syslog:

Pak recvd on LOGGING-Q: Local Port Fwding L3If: L2If:TenGigabitEthernet1/0/1 DI:0x1F2A, LT:0, Vlan:10   SrcGPN:456, SrcGID:456, ACLLogIdx:0x1, MacDA:0000.0000.0002, MacSA: 0000.0000.0001   Non (IPv4/IPv6/ARP/RARP)

TPFFD:F38001C8_000A400A_01A00080-00011F2A_F5F50000_00002BFD

Jun  7 15:37:24.482: Raw FEX packet Dump:

Jun  7 15:37:24.482: 00 00 00 00 00 02 00 00 00 00

Jun  7 15:37:24.482: 00 01 89 26 80 01 0B FD 81 00

Jun  7 15:37:24.482: 00 0A 08 00 45 00 00 64 76 87

Jun  7 15:37:24.482: 00 00 FF 01 31 0F 0A 00 00 01

Jun  7 15:37:24.482: 0A 00 00 02 08 00 A9 FF 00 12

12-bits (with their corresponding hex values, refer to VNTAG packet format in Instant Access white-paper) correspond to the destination VIF value that has been passed in the packet, itself. This value (changed to decimal) is further cross-referenced with the point of exit on the FEX.

This command is used to determine point of exit:  This takes the unicast VIF into consideration (refer to the note in point 2.2.3 regarding VNTAG manager).

FEX-101#sh platform fex ucast-entries

vif   sw_idb            portname             GPN   handle    res_index

====  ==========  ======================     ====  ========= =========

1     0x5CAC278   GigabitEthernet1/0/1       1     0x5        0x30F0000

2     0x5CAE2E0   GigabitEthernet1/0/2       2     0x6        0x30F0000

(...)

For multicast traffic the logic:

1) Determine the set of egress interfaces for specific multicast VIF.

FEX-101#sh platform fex mcast-entries

(...)

Entry : 2 =================

Mcast VIF = 3072 : destid = 0x23DF     : handle = 0x37       : result_index = 0x4D

DestId 23DF details with GPN list

index  next   flags cmi    #GPN GPN

0x23DF 0xFFFF 0x00  0x0000 3    1    2    464

(...)

In this example, for VIF=3072, three (3) egress interfaces have been selected with the internal GPN numbers: 1, 2 and 464. In order to translate these internal GPN port numbers into physical interfaces, this command can be used:

FEX-101#show platform pm if-numbers

interface gid  gpn  lpn  port slot unit slun port-type lpn-idb gpn-idb

----------------------------------------------------------------------

Gi1/0/1   1    1    1    0/2  1    1    1    local     Yes     Yes

Gi1/0/2   2    2    2    0/1  1    2    2    local     Yes     Yes

(...)

Po2       464  464  0    16/0  9    2    2    local     No      No

These outputs mean that packets that are received with a multicast destination VIF equal to 3072, are forwarded to interfaces: Gi1/0/1, Gi1/0/2 and Port-channel 2.

4. Summary

In case of any issues related with the fex, these command outputs are collected and added to the TAC case when it is being opened:

6880#show tech-support

6880#show tech-support fex infra

6880#attach fex <fex-id>

Password: cisco

FEX-101>en

Password: cisco

FEX-101#show tech-support