Cisco IOS XR Troubleshooting Guide, Release 3.5
Troubleshooting Fabric
Downloads: This chapterpdf (PDF - 480.0KB) The complete bookPDF (PDF - 2.2MB) | Feedback

Troubleshooting Router Switch Fabric

Table Of Contents

Troubleshooting Router Switch Fabric

CRS-1 Switch Fabric Overview

Flags

Verifying and Troubleshooting the Fabric Plane State

Examples

Verifying and Troubleshooting Up Fabric Planes

Examples

Troubleshooting Down Fabric Planes

Examples


Troubleshooting Router Switch Fabric


This chapter describes techniques that you can use to troubleshoot router switch fabric. It includes the following sections:

CRS-1 Switch Fabric Overview

Flags

Verifying and Troubleshooting the Fabric Plane State

Verifying and Troubleshooting Up Fabric Planes

Troubleshooting Down Fabric Planes

CRS-1 Switch Fabric Overview

The switch fabric is constructed of eight independent fabric planes. Each plane consists of at least three stage switch element ASICs (SEAs), which are collectively known as known as switch fabric elements (SFE). While all SEAs are identical, the operational behavior of a SEA is defined according to the position of the SEA in the fabric. A SEA set for Stage 1 operation is known as an S1 ASIC. A SEA set for Stage 2 operation is known as an S2 ASIC. A SEA set for Stage 3 operation is known as an S3 ASIC. The complete fabric is composed of the ingress queue ASIC on the ingress modular services card (MSC) or designated route processor (D)RP, the S1, S2, and S3 ASICS on the switch fabric planes and one of more fabric queue ASICs on the destination MSC or (D)RP.


Note For Cisco CRS-1 Multishelf Systems, connect to the route processor (RP) for the owner logical router (LR) when troubleshooting the fabric plane. When you are connected to the owner LR, you have control over the entire system and all cards assigned to the owner LR.



Note Line cards in Cisco CRS-1s are called modular services cards (MSCs).


Between ingress queue, S1, S2, S3, and fabric queue ASICs are communication paths known as fabric links. An individual fabric link has a bandwidth capacity of 2.5 Gbps. A link is unidirectional.

An ingress MSC has multiple 2.5 Gbps links from the ingress queue ASIC on the MSC to an individual fabric plane and to an individual S1 ASIC on that fabric plane. Connections are provided in the same fashion to each of the eight fabric planes that are present in a system in normal operation.

All S1 ASICs have at least one 2.5 Gbps link to every S2 ASIC present on that fabric plane. An S1-to-S2 link does not and cannot span across fabric planes.

All S2 ASICs have at least one 2.5 Gbps link to every S3 ASIC present on that fabric plane. An S2-to-S3 link does not and cannot span across fabric planes.

An egress MSC has multiple 2.5 Gbps links from the S3 ASIC to one or two fabric queue ASICs present on that MSC.

In a 16-slot single-chassis system, the eight switch fabric planes are contained on eight individual fabric cards known as FC/S or S123. Each of the fabric cards contains two S1, two S2, and four S3 ASICs.

In an 8-slot system, the eight switch fabric planes are contained on four individual fabric cards known as FC/S or HS123. Each fabric card contains two planes. Each plane is composed of one S1, one S2, and one S3 ASIC.

In a 4-slot system, there are four fabric planes. The fabric planes are contained on four individual fabric cards known as FC/s or QS123. Each plane is composed of one S1, one S2, and one S3 ASIC.

Each S13 card contains two S1 and four S3 ASICs and corresponds to one switch fabric plane. Each fabric card chassis has one or more switch fabric planes and each fabric plane consists of one or more S2 fabric cards. Each S2 fabric card contains six S2 ASICs.

Data is not transmitted across the fabric in packets. Packets are segmented into cells by the ingress queue ASIC and reassembled back to packets by the fabric queue ASIC. After a cell is placed on a plane, it will remain on that plane until it reaches the egress MSC. While the system is operational, cells are always being sent and received even though they may not contain any data. Cells with no data are termed as idle cells.

Cell forwarding on the S3 ASIC on both the Cisco CRS-1 4-slot, 8-slot, and 16-slot systems involves examination of the cell header. Using the cell header, the S3 is able to determine which fabric queue ASIC the cells should be sent to. The S3 ASIC injects cells in a round-robin fashion across the multiple links connecting the S3 ASIC to the fabric queue ASIC.

As the cell is created, forward error correction (FEC) data is calculated and appended. The FEC code is used on the transmission links to ensure the integrity of the data as it is transmitted and received. The FEC data is removed, checked, recalculated, and reapplied by each SFE in the path until it is delivered to the egress MSC. The FEC value is checked before a cell is processed by the SFE. If an error is detected, the FEC code can be used to recover the cell. Depending on the nature of the error, recovery may not be possible, in which case, the cell is discarded.

A system can operate with seven planes with out performance degradation. As further planes are removed, overall capacity degrades but the system remains operational. A system requires a minimum of two planes to maintain service. One plane must be odd numbered and the other plane must be even numbered.


Note A 4-slot system operates on four planes. The loss of a plane reduces the usable forwarding capacity by approximately 6Gbps.


A plane marked MCAST_DOWN means that as far as the ingress line cards are concerned, not all destination FabricQ ASICs can be reached via the plane. Since the destinations of multicast traffic cannot be predetermined due to their dynamic nature, the plane is termed as MCAST_DOWN. Multicast traffic will use the other fabric planes to deliver date to the appropriate egress line cards. Unicast traffic will continue to use the plane if traffic is destined to a FabricQ ASIC that is still reachable. If traffic is destined for a FabricQ ASIC that is not unreachable, the other available plane is used instead of the plane in the MCAST_DOWN state.

Flags

A Flags field is provided in some fabric-related show commands. The flag is an abbreviated reason for a link being placed in the down state. Table 4-1 provides descriptions of the flags.

Table 4-1 Fabric show Command Flags 

Flag
Description

P - plane admin down—plane has been taken out of service by administrative action

p - plane oper down—plane is not able to operate or has been taken out of service by administrative action

C - card admin down—card has been taken out of service by administrative action (reserved for future use)

c - card oper down—flag is set in response to system notification that a card will soon be powered off. This is a transient condition lasting a few seconds before power is withdrawn. Unusual to see this flag in practice

L - link port admin down—link has been taken out of service by administrative action

l - linkport oper down—flag is set or cleared in response to messages from the ASIC driver code. Flags are enabled when the asic is first discovered. Flag is set if the ASIC driver has signaled that an individual link port is not usable for fabric traffic, or that the driver has confirmed that the link is ready. Two common reasons for seeing an l flag that are initialization is not completed yet or that sufficient errors have been seen at a receive port so that the ASIC driver shuts the port down

A - asic admin down —ASIC has been taken out of service by administrative action. If the A flag is set, it is seen on all links connected to that ASIC

a - asic oper down —flag is set or cleared in response to messages from the ASIC driver code. Flags are enabled when the ASIC is first discovered. Flag is set if the ASIC driver has signaled that an SEA ASIC is not usable for fabric traffic. If the a flag is set, it is seen on all links connected to that ASIC

B - bundle port admin down—bundle port has been taken out of service by administrative action. If the B flag is set, it is seen on all links associated to that bundle

b - bundle port oper down—bundle port cannot operate or has been taken out of service by administrative action. If the b flag is set, it is seen on all links associated to that bundle

I - bundle admin down—bundle has been taken out of service by administrative action

i - bundle oper down—bundle cannot operate or has been taken out of service by administrative action

N - node admin down—Node has been taken out of service by administrative action

n - node down—node cannot operate or has been taken out of service by administrative action

o - other end of link down—only applies to TX links. Indicates that receiver link port is down

d - data down— idle cells are being transmitted or received, but link is not able to transport data cells. Flag is set in response to a message from the ASIC driver. Link ports go into this state during link startup. It is typically transient and should only persist if something goes wrong with the startup process

f - failed component downstream—flag is set by fabric status database if it decides to stop using a link because of some downstream failure

m - plane multicast down— not all fabric queue ASICs present on the plane are reachable; multicast forwarding is not operational on this plane. Unicast traffic is unaffected.


Verifying and Troubleshooting the Fabric Plane State

To verify and troubleshoot the fabric plane state, perform the following procedure.


Note All fabric troubleshooting should be performed in administration executive (admin EXEC) mode while you are logged into the default logical router. This allows you to view system-wide parameters.


SUMMARY STEPS

1. admin

2. show controllers fabric plane all detail

3. show controllers fabric plane all statistics

4. show controllers fabric plane plane_id statistics detail

5. show controllers fabric sfe s1 all | include UP.*DOWN
show controllers fabric sfe s2 all | include UP.*DOWN
show controllers fabric sfe s3 all | include UP.*DOWN
show controllers fabric sfe ingress all | include UP.*DOWN
(for MSC fabric interface ASICs)
show controllers fabric sfe fabricq all | include UP.*DOWN (for MSC fabric interface ASICs)

6. Contact Cisco Technical Support if the problem is not resolved

DETAILED STEPS

 
Command or Action
Purpose

Step 1 

admin

Example:

RP/0/RP0/CPU0:router#  admin

Enters administration executive (admin EXEC) mode.

Step 2 

show controllers fabric plane all detail

Example:

RP/0/RP0/CPU0:router(admin)# show controllers fabric plane all

Displays system fabric plane information from all fabric planes.

Check the number of planes in operation. If there are any planes that are down (either the Admin State or Oper State is DOWN), proceed to the "Troubleshooting Down Fabric Planes" section.

A system can operate with seven planes without performance degradation. As further planes are removed, overall capacity degrades, but the system remains operational. A system must have a minimum of two operational planes to maintain service. One must be an odd-numbered plane, the other must be an even-numbered plane.

Step 3 

show controllers fabric plane all statistics
Example:

RP/0/RP0/CPU0:router(admin)# show controllers fabric plane all statistics

Displays controller fabric statistics for all planes.

Errors are indicated in the last three columns in the output:

CE=correctable errors

UCE=uncorrectable errors

PE=parity errors (in an ASIC memory)

The Out Cells value should be equal or greater than the In Cells value because of multicast copies created in the fabric. If the Out Cells value is equal to or greater than the In Cells value, proceed to Step 4.

If the values in the Out Cells column are much less than the values in the In Cells column, cells are being dropped. If the Out Cells value is less than the In Cells value, proceed to the "Verifying and Troubleshooting Up Fabric Planes" section.

If the CE, UCE or PE values are incrementing, contact Cisco Technical Support. For contact information for Cisco Technical Support, see the "Obtaining Documentation, Obtaining Support, and Security Guidelines" section in the Preface.

Step 4 

show controllers fabric plane plane_id 
statistics detail
Example:

RP/0/RP0/CPU0:router(admin)# show controllers fabric plane 0 statistics detail

Displays controller fabric statistics for a specific plane.

The output is displayed for unicast and multicast. This allows you to determine if a unicast problem was hidden by multicast cells. If the Total received unicast data cells value is much lower than the Total transmitted unicast data cells, there are transmission problems. If there is a large drop in the number of Total transmitted unicast data cells compared to the Total received unicast data cells, proceed to the "Verifying and Troubleshooting Up Fabric Planes" section.

The Total unicast lost cells and Total multicast lost cells displays the number of cells that were received by an ASIC but could not be handled. Dropped multicast cells can be due to congestion in the fabric. Dropped unicast cells indicate a more serious problem. If dropped unicast cells are detected, contact Cisco Technical Support. For Cisco Technical Support contact information, see the "Obtaining Documentation, Obtaining Support, and Security Guidelines" section in the Preface.

Step 5 

show controllers fabric sfe s1 all | include 
UP.*DOWN
show controllers fabric sfe s2 all | include 
UP.*DOWN
show controllers fabric sfe s3 all | include 
UP.*DOWN
show controllers fabric sfe ingress all | 
include UP.*DOWN
show controllers fabric sfe fabricq all | 
include UP.*DOWN 
Example:

RP/0/RP0/CPU0:router(admin)# show controllers fabric sfe s1 all | include UP.*DOWN

RP/0/RP0/CPU0:router(admin)# show controllers fabric sfe s2 all | include UP.*DOWN

RP/0/RP0/CPU0:router(admin)# show controllers fabric sfe s3 all | include UP.*DOWN

RP/0/RP0/CPU0:router(admin)# show controllers fabric sfe ingress all | include UP.*DOWN

RP/0/RP0/CPU0:router(admin)# show controllers fabric sfe fabricq all | include UP.*DOWN

Use these commands on the ASICs in the affected planes to help isolate the problem to a specific board.

Once you have isolated the problem to a specific board, contact Cisco Technical Support. For Cisco Technical Support contact information, see the "Obtaining Documentation, Obtaining Support, and Security Guidelines" section in the Preface.

Step 6 

Contact Cisco Technical Support.

If the problem is not resolved, contact Cisco Technical Support. For Cisco Technical Support contact information, see the "Obtaining Documentation, Obtaining Support, and Security Guidelines" section in the Preface.

Examples

The following example shows how to confirm the number of fabric planes.

Verify the number of fabric planes in operation:

RP/0/RP0/CPU0:router(admin)# show controllers fabric plane all detail
 
   
Flags: P - plane admin down,       p - plane oper down 
         C - card admin down,        c - card  oper down 
         L - link port admin down,   l - linkport oper down 
         A - asic admin down,        a - asic oper down 
         B - bundle port admin Down, b - bundle port oper down 
         I - bundle admin down,      i - bundle oper down 
         N - node admin down,        n - node down 
         o - other end of link down  d - data down
         f - failed component downstream 
         m - plane multicast down 
 
   
Plane  Admin   Oper       Down       Total     Down   
Id     State   State      Flags      Bundles   Bundles
------------------------------------------------------
0      UP      UP                    9         3      
1      UP      UP                    9         3      
2      UP      UP                    9         3      
3      UP      UP                    9         3      
4      UP      UP                    9         3      
5      UP      UP                    9         3      
6      UP      UP                    9         3      
7      UP      UP                    9         3 
 
   

The output above shows all eight fabric planes in the up state.

The following output shows Fabric Plane 1 marked as MCAST_DOWN:

RP/0/RP0/CPU0:router(admin)# show controllers fabric plane all detail 
 
   
Flags: P - plane admin down,       p - plane oper down 
         C - card admin down,        c - card  oper down 
         L - link port admin down,   l - linkport oper down 
         A - asic admin down,        a - asic oper down 
         B - bundle port admin Down, b - bundle port oper down 
         I - bundle admin down,      i - bundle oper down 
         N - node admin down,        n - node down 
         o - other end of link down  d - data down
         f - failed component downstream 
         m - plane multicast down 
 
   
Plane  Admin   Oper       Down       Total     Down   
Id     State   State      Flags      Bundles   Bundles
------------------------------------------------------
0      UP      UP                    0         0     
1      UP      MCAST_DOWN m          0         0      
3      UP      UP                    0         0      
4      UP      UP                    0         0      
5      UP      UP                    0         0      
6      UP      UP                    0         0      
7      UP      UP                    0         0   
 
   

The fabric plane is operational, but not all destination fabric queue application-specific integrated circuit (ASICs) can be reached through this fabric plane, as far as the ingress modular services cards (MSCs) are concerned. Because destinations of multicast traffic cannot be predetermined due to their dynamic nature, the fabric plane is shown as MCAST_DOWN. Multicast traffic uses the other fabric planes to deliver data to the appropriate egress MSCs. Unicast traffic continues to use the fabric plane if traffic is destined to a fabric queue ASIC that is still reachable. If traffic is destined for a fabric queue ASIC that is deemed unreachable, the other available fabric plane is used instead of the plane in the MCAST_DOWN state. To change the operational state of a fabric plane from MCAST_DOWN to up, contact Cisco Technical Support. See the "Obtaining Documentation, Obtaining Support, and Security Guidelines" section section in the Preface.

The following output shows Fabric Plane 0 as administratively shut down:

RP/0/RP0/CPU0:router(admin)# show controllers fabric plane all detail
 
   
Flags: P - plane admin down,       p - plane oper down 
         C - card admin down,        c - card  oper down 
         L - link port admin down,   l - linkport oper down 
         A - asic admin down,        a - asic oper down 
         B - bundle port admin Down, b - bundle port oper down 
         I - bundle admin down,      i - bundle oper down 
         N - node admin down,        n - node down 
         o - other end of link down  d - data down
         f - failed component downstream 
         m - plane multicast down 
 
   
Plane  Admin   Oper       Down       Total     Down 
Id     State   State      Flags      Bundles   Bundles 
------------------------------------------------------ 
0      DOWN    DOWN       P          0         0      
1      UP      UP                    0         0      
2      UP      UP                    0         0      
3      UP      UP                    0         0      
4      UP      UP                    0         0      
5      UP      UP                    0         0      
6      UP      UP                    0         0      
7      UP      UP                    0         0      
 
   

The following output shows Fabric Plane 4 as administratively shutdown and powered off:

RP/0/RP0/CPU0:router(admin)# show controllers fabric plane all detail 
 
   
Flags: P - plane admin down,       p - plane oper down 
         C - card admin down,        c - card  oper down 
         L - link port admin down,   l - linkport oper down 
         A - asic admin down,        a - asic oper down 
         B - bundle port admin Down, b - bundle port oper down 
         I - bundle admin down,      i - bundle oper down 
         N - node admin down,        n - node down 
         o - other end of link down  d - data down
         f - failed component downstream 
         m - plane multicast down 
 
   
Plane  Admin   Oper       Down       Total     Down 
Id     State   State      Flags      Bundles   Bundles 
------------------------------------------------------ 
0      UP      UP                    0         0      
1      UP      UP                    0         0      
2      UP      UP                    0         0      
3      UP      UP                    0         0      
4      DOWN    DOWN       pPm        0         0      
5      UP      UP                    0         0      
6      UP      UP                    0         0      
7      UP      UP                    0         0  
 
   
 
   
 
   

The following example shows how to display statistics for all fabric planes:

RP/0/RP0/CPU0:router(admin)# show controllers fabric plane all statistics 
 
   
                      In                  Out       CE        UCE       PE 
Plane                Cells               Cells     Cells     Cells     Cells
--------------------------------------------------------------------------------
0           736182590951        736183103374         0         0         0
1           736324880091        736325098791         0         0         0
2           736315213586        736315442992         0         0         0
3           736299535716        736299764252         0         0         0
4           736300399513        736300627048         0         0         0
5           736295116556        736295346246         0         0         0
6           736311177372        736311406904         0         0         0
7           736296917574        736297149734         0         0         0
 
   

CE = Errored cell with a correctable error detected using FEC code

UCE = Errored cell with an uncorrectable error detected using FEC code

PE = Parity error present within a cell when processed by SFE

The following example shows to display detailed fabric plane statistics for a specified fabric plane:

RP/0/RP0/CPU0:router(admin)# show controllers fabric plane 0 statistics detail
 
   
The fabric plane number is 0
 Total number of providers for the statistics: 1
 Total received data cells: 749351837428 
 Total received unicast data cells: 749351660427
 Total received multicast data cells: 177001
 Total transmitted data cells: 749352358340
 Total transmitted unicast data cells: 749351671469
 Total transmitted multicast data cells: 686871
 Total received correctable errored cells: 0
 Total received uncorrectable errored cells: 0
 Total received parity error cells: 0
 Total unicast lost cells: 0
 Total multicast lost cells: 0
Last clearing of "show controller fabric plane" counters never

Verifying and Troubleshooting Up Fabric Planes

To verify and troubleshoot the up fabric planes, perform the following procedure.

For Cisco CRS-1 Multishelf Systems the S1 to S2 links and the S2 to S3 links are inter-chassis links.

For Cisco CRS-1 single-shelf systems the links are internal chassis links. The commands used are the same.

SUMMARY STEPS

1. admin

2. show controllers fabric link port s2rx all | include UP.*DOWN.*SM
show controllers fabric link port s3rx all | include UP.*DOWN.*SM
show controllers fabric link port s2tx all | include UP.*DOWN.*SM
show controllers fabric link port s3tx all | include UP.*DOWN.*SM

3. show controllers fabric link port s2rx all stat | exclude \ 0 +0 +0$
show controllers fabric link port s3rx all stat | exclude \ 0 +0 +0$

4. config

5. controllers fabric link port {s1rx | s1tx | s2rx | s2tx | s3rx | s3tx} location link-id shutdown

6. Reload the fabric card.

DETAILED STEPS

 
Command or Action
Purpose

Step 1 

admin

Example:

RP/0/RP0/CPU0:router#  admin

Enters administration executive (admin EXEC) mode.

Step 2 

show controllers fabric link port s2rx all | include UP.*DOWN.*SM
show controllers fabric link port s3rx all | include UP.*DOWN.*SM
show controllers fabric link port s2tx all | include UP.*DOWN.*SM
show controllers fabric link port s3tx all | include UP.*DOWN.*SM

Example:

RP/0/RP0/CPU0:router(admin)# show controllers fabric link port s2rx all | include UP.*DOWN.*SM

RP/0/RP0/CPU0:router(admin)# show controllers fabric link port s3rx all | include UP.*DOWN.*SM

RP/0/RP0/CPU0:router(admin)# show controllers fabric link port s2tx all | include UP.*DOWN.*SM

RP/0/RP0/CPU0:router(admin)# show controllers fabric link port s3tx all | include UP.*DOWN.*SM

Displays the link state of a fiber port. The command output shows interchassis link ports that are operationally down, excluding those that are administratively down and those that are completely disconnected.

Repeat this command for each stage and direction and check if a fabric port link is down (ADMIN UP and OPER DOWN).

The following down flags may be displayed:

l—Link is down

o—Downed link port due to the other end of the link being down

a—ASIC is down

A or L—ASIC (A) or link (L) is administratively shut down

If the link state of a fabric port is down, proceed to Step 5 to shut down the link.

Step 3 

show controllers fabric link port s2rx all stat | exclude \ 0 +0 +0$
show controllers fabric link port s3rx all stat | exclude \ 0 +0 +0$

Example:

RP/0/RP0/CPU0:router(admin)#  show control fabric link port s2rx all stat | exclude \ 0 +0 +0$

RP/0/RP0/CPU0:router(admin)#  show control fabric link port s3rx all stat | exclude \ 0 +0 +0$

Displays any link that has an error count, indicating a problem link even if the link has not been shut down.

If there are UCE errors on a link, shut down the link to avoid traffic loss. Proceed to Step 5 to shut down the link.

Step 4 

config

Example:

RP/0/RP0/CPU0:router(admin)#  config

Enters administration configuration mode.

Step 5 

controllers fabric link port {s1rx | s1tx | s2rx | s2tx | s3rx | s3tx} location link-id shutdown

Example:

RP/0/RP0/CPU0:router(admin-config)# controllers fabric link port s1rx 0/0/cpu0 1 shutdown

Places a link out of service. This command closes both ends of the link.

Note that performing such an operation should be conducted in conjunction with Cisco Technical Support engineers. For Cisco Technical Support contact information, see the "Obtaining Documentation, Obtaining Support, and Security Guidelines" section in the Preface.

If you have placed a link out of service on a Cisco CRS-1 single-shelf system, proceed to Step 6.

If you have placed a link out of service on a Cisco CRS-1 Multishelf System, clean the fiber-optic connectors, reconnect the optical fibers (if cleaning and reconnecting the optical fibers does not resolve the down link, replace the optical fibers.) See Cisco CRS-1 Carrier Routing System Fiber-Optic Cleaning Guide for information on cleaning fiber-optic connectors. If the problem is not resolved, contact Cisco Technical Support. For Cisco Technical Support contact information, see the "Obtaining Documentation, Obtaining Support, and Security Guidelines" section in the Preface.

Step 6 

Reload the fabric card containing the out of service link.

See the Managing the Router Hardware module of Cisco IOS XR System Management Configuration Guide for information on reloading the fabric card.

If the problem is not resolved, Cisco Technical Support. For Cisco Technical Support contact information, see the "Obtaining Documentation, Obtaining Support, and Security Guidelines" section in the Preface.

Examples

The following example shows how to display the link state of a fiber port. The output displays link ports that are operationally down, excluding those that are administratively down and those that are completely disconnected.

RP/0/RP0/CPU0:router(admin)# show controllers fabric link port s2rx all statistics 
 
   
Wed May  2 16:29:49.102 EST EDT
 
   
Total racks: 1
 
   
Rack 0:
 
   
       SFE  Port            In                In         CE        
UCE      PE
       R/S/M/A/P        Data Cells        Idle Cells    Cells     
Cells    Cells
--------------------------------------------------------------------------------
    0/SM0/SP/0/0           9149278     5437189623455        0         
0        0
    0/SM0/SP/0/1           9149418     5438017457030        0         
0        0
    0/SM0/SP/0/2           9149418     5438017379235        0         
0        0
    0/SM0/SP/0/3           9149418     5438017274614        0         
0        0
    0/SM0/SP/0/4           9149418     5438017189053        0         
0        0
    0/SM0/SP/0/5           9149418     5438017086116        0         
0        0
    0/SM0/SP/0/6           9149418     5438016991625        0         
0        0
    0/SM0/SP/0/7           9149418     5438016903486        0         
0        0
    0/SM0/SP/0/8           9149418     5438016817714        0         
0        0
.
.
.
 
   

The following example shows how to display a link with an error count, indicating a problem link even if the link has not been shut down.

RP/0/RP0/CPU0:router(admin)# show controllers fabric link port s2rx all stat | exclude \ 0 
+0 +0$ 
 
   
Total racks: 1
 
   
Rack 0:
 
   
      SFE  Port            In                In         CE       UCE      PE 
      R/S/M/A/P        Data Cells        Idle Cells    Cells    Cells    Cells
--------------------------------------------------------------------------------
 
   

The following example shows how to place a link out of service.


Note Performing this operation should be conducted in conjunction with Cisco Technical Support engineers.


RP/0/RP0/CPU0:router(admin-config)# controllers fabric link port s1rx all brief 
 
   
Wed May  2 16:31:45.560 EST EDT
   Flags: P - plane admin down,       p - plane oper down
          C - card admin down,        c - card  oper down
          L - link port admin down,   l - linkport oper down
          A - asic admin down,        a - asic oper down
          B - bundle port admin Down, b - bundle port oper down
          I - bundle admin down,      i - bundle oper down
          N - node admin down,        n - node down
          o - other end of link down  d - data down
          f - failed component downstream
          m - plane multicast down
Sfe Port         Admin Oper  Down    Other            Near-end    Far- 
end
R/S/M/A/P        State State Flags   End              Bport       Bport
------------------------------------------------------------------------
--------
0/SM0/SP/0/0     UP    UP            0/SM0/SP/0/0
0/SM0/SP/0/1     UP    UP            0/SM0/SP/0/2
0/SM0/SP/0/2     UP    UP            0/SM0/SP/0/4
0/SM0/SP/0/3     UP    UP            0/SM0/SP/0/6
0/SM0/SP/0/4     UP    UP            0/SM0/SP/0/8
0/SM0/SP/0/5     UP    UP            0/SM0/SP/0/10
0/SM0/SP/0/6     UP    UP            0/SM0/SP/0/12
0/SM0/SP/0/7     UP    UP            0/SM0/SP/0/14
0/SM0/SP/0/8     UP    UP            0/SM0/SP/0/16
0/SM0/SP/0/9     UP    UP            0/SM0/SP/0/18
0/SM0/SP/0/10    UP    UP            0/SM0/SP/0/20
0/SM0/SP/0/11    UP    UP            0/SM0/SP/0/22
0/SM0/SP/0/12    UP    UP            0/SM0/SP/0/24
0/SM0/SP/0/13    UP    UP            0/SM0/SP/0/26
.
.
.

Troubleshooting Down Fabric Planes

An individual link placed in a down state can be sustained without impact to the overall operation or forwarding capacity of the Cisco CRS-1 Multishelf System. At each stage of the fabric, there is a greater bandwidth capacity than can actually be generated by the modular services cards (MSCs). The egress side of the switch fabric has at least double the capacity of the ingress.

The loss of a single link reduces the switch fabric capacity on a single plane by 2.5 Gbps or approximately 1.5625 Gbps when considering 8b/10b encoding and cell tax overheads. The actual effective loss of bandwidth varies depending on where the link loss takes place. If a link were to be lost between the ingress queue and S1 stage or S1 and S2 stage, the reduction is against a capacity of 80 Gbps or 50 Gbps effective bandwidth. A link loss between S2 and S3 stages or S3 and fabric queue stages means that the reduction is against a capacity of 160 Gbps or 100 Gbps effective bandwidth.

Consider a situation in which a set of S1 links was lost on a single plane. Six or seven links would need to be taken out of service in order to reduce the effective bandwidth capacity by approximately 10 Gbps. It is, therefore, possible to contend that because the MSC is capable of transmitting at 40 Gbps, there is still sufficient capacity to carry the traffic without loss.

The system is designed to operate with seven out of eight fabric planes in operation and provide full line-rate forwarding capability, because it provides approximately 42 Gbps to each MSC. If another fabric plane is removed, forwarding capacity would be reduced to approximately 34 Gbps.

SUMMARY STEPS

1. admin

2. show controllers fabric plane all detail

3. Bring up the plane:

a. configure

b. no controllers fabric plane plane-id shutdown

c. end

4. show running-config

5. show controllers fabric connectivity all

6. show controllers fabric sfe {s1 | s2 | s3 | ingressq | fabricq} all

7. show controllers fabric bundle port all

DETAILED STEPS

 
Command or Action
Purpose

Step 1 

admin

Example:

RP/0/RP0/CPU0:router#  admin

Enters administration executive (admin EXEC) mode.

Step 2 

show controllers fabric plane all detail

Example:

RP/0/RP0/CPU0:router(admin)# show controllers fabric plane all detail

Displays information about the fabric plane.

Verify that the down plane is not administratively down. If the Admin State is down, the operational state should also be down.

If the plane is administratively down and the only flag is P, bring the plane up. Proceed to Step 3 to bring up the plane. See the "Flags" section for detailed information on flags.

If the plane is administratively up but the flag is m, the plane is multicast down and some destinations (for example, MSCs and route processors) are having connectivity issues to the fabric on that plane. Proceed to Step 5. See the "Flags" section for detailed information on flags.

If the plane is administratively down and the flag is p, the plane is operationally down. Proceed to Step 6. See the "Flags" section for detailed information on flags.

Step 3 

configure

no controllers fabric plane plane-id shutdown

end

Example:

RP/0/RP0/CPU0:router(admin)#configure

RP/0/RP0/CPU0:router(admin_config)# no controllers fabric plane 3 shutdown

RP/0/RP0/CPU0:router(admin_config)# end

The following tasks must be completed to bring up the plane:

configure—Enters administration configuration mode

no controllers fabric plane plane-id shutdown—Performs a graceful shutdown of the fabric plane to ensure that data is no longer flowing through the plane before a fabric reconfiguration or fabric plane migration

end—Saves the configuration change

Step 4 

show running-config

Example:

RP/0/RP0/CPU0:router(admin)# show running-config

For Cisco CRS-1 Multishelf Systems only.

Displays the contents of the administrative running configuration.

Verify that the plane is set. In the Cisco CRS-1 Multishelf System, the command output indicates which fiber module in the fabric card chassis (FCC) is connected from the plane.

The following is an example of the output:

controllers fabric plane 0 topology single-module location F0/SM4/FM

Step 5 

show controllers fabric connectivity all

Example:

RP/0/RP0/CPU0:router(admin)# show controllers fabric connectivity all

Displays controller fabric connectivity information for all fabric ports.

For the down plane, check if the MSCs and route processors have receive and transmit connectivity to the fabric.

If there is no transmit and receive connectivity to the fabric, contact Cisco Technical Support. For Cisco Technical Support contact information, see the "Obtaining Documentation, Obtaining Support, and Security Guidelines" section in the Preface.

If there is no transmit and receive connectivity to a MSC that is not required, remove it from the system or power it down. This will bring the plane back into the up state. See Installing the Cisco CRS-1 Carrier Routing System 16-Slot Line Card Chassis at the following URL for information on removing MSCs:

http://www.cisco.com/en/US/products/ps5763/tsd_products_support_series_home.html

Step 6 

show controllers fabric sfe {s1 | s2 | s3 | ingressq | fabricq} all

Example:

RP/0/RP0/CPU0:router(admin)# show controllers fabric sfe s1 all

Displays ASIC information.

Verify that all the ASICs are in the administrative and operational up state.

If all ASICs are in the up state, proceed to Step 7.

If all the ASICs are not in the up state, contact Cisco Technical Support. For Cisco Technical Support contact information, see the "Obtaining Documentation, Obtaining Support, and Security Guidelines" section in the Preface.

Step 7 

show controllers fabric bundle port all

Example:

RP/0/RP0/CPU0:router(admin)# show controllers fabric bundle port all

For Cisco CRS-1 Multishelf Systems only.

Displays bundle port information.

Verify that all bundle ports are in the operational up state.

If a bundle port is in the operational down state, contact Cisco Technical Support, see the "Obtaining Documentation, Obtaining Support, and Security Guidelines" section in the Preface.

Note For this release, three of the nine ports will be down because there is no third rack connected to the fabric card chassis. The down ports are 6, 7, and 8.

If a bundle port is in the administrative down state, proceed to Step 3 to bring up the plane.

Examples

The output from the show controllers fabric plane all detail command displays information on the fabric plane:

RP/0/RP0/CPU0:router(admin)# show controllers fabric plane all detail
 
   
Flags: P - plane admin down,       p - plane oper down
        C - card admin down,        c - card  oper down
        L - link port admin down,   l - linkport oper down
        A - asic admin down,        a - asic oper down
        B - bundle port admin Down, b - bundle port oper down
        I - bundle admin down,      i - bundle oper down
        N - node admin down,        n - node down
        o - other end of link down  d - data down
        f - failed component downstream
        m - plane multicast down
 
   
Plane  Admin   Oper       Down       Total     Down
Id     State   State      Flags      Bundles   Bundles
------------------------------------------------------
0      UP      UP                    9         3
1      UP      UP                    9         3
2      UP      DOWN       p          0         0
3      UP      DOWN       p          0         0
4      UP      MCAST_DOWN m          9         3
5      UP      UP                    9         3
6      UP      DOWN       p          0         0
7      UP      DOWN       p          0         0
 
   

The output from the show controllers fabric bundle port all command displays the status of the bundle ports:

 
   
RP/0/RP0/CPU0:router(admin)# show controllers fabric bundle port all
 
   
Flags: P - plane admin down,       p - plane oper down
         C - card admin down,        c - card  oper down
         L - link port admin down,   l - linkport oper down
         A - asic admin down,        a - asic oper down
         B - bundle port admin Down, b - bundle port oper down
         I - bundle admin down,      i - bundle oper down
         N - node admin down,        n - node down
         o - other end of link down  d - data down
         f - failed component downstream
         m - plane multicast down
 
   
Bundle Port    Admin     Oper
  R/S/M/P      State     State
------------------------------
 0/SM0/SP/0    UP        UP
 0/SM0/SP/1    UP        UP
 0/SM0/SP/2    UP        UP
 0/SM1/SP/0    UP        UP
 0/SM1/SP/1    UP        UP
 0/SM1/SP/2    UP        UP
 0/SM4/SP/0    UP        UP
 0/SM4/SP/1    UP        UP
 0/SM4/SP/2    UP        UP
 0/SM5/SP/0    UP        UP
 0/SM5/SP/1    UP        UP
 0/SM5/SP/2    UP        UP
 1/SM0/SP/0    UP        UP
 1/SM0/SP/1    UP        UP
 1/SM0/SP/2    UP        UP
 1/SM1/SP/0    UP        UP
 1/SM1/SP/1    UP        UP
 1/SM1/SP/2    UP        UP
 1/SM4/SP/0    UP        UP
 1/SM4/SP/1    UP        UP
 1/SM4/SP/2    UP        UP
 1/SM5/SP/0    UP        UP
 1/SM5/SP/1    UP        UP
 1/SM5/SP/2    UP        UP
 F0/SM0/FM/0   UP        UP
.
.
.