Cisco BPX/IGX/IPX WAN Software

Revised April 25, 2008

February 11, 2004

NOTICE:

THIS FIELD NOTICE HAS BEEN EXPIRED AND IS NO LONGER MAINTAINED OR UPDATED BY CISCO.

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE, WARRANTY OR SUPPORT. USE OF THE INFORMATION ON THIS FIELD NOTICE OR MATERIALS LINKED FROM THIS FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Products Affected

Problem Description

Broadband Controller Card (BCC) or Network Processor Module (NPM) Redundancy is lost due to the Standby Controller Card going to Cleared state.

Background

The standby BCC or NPM goes to the Cleared state due to a mishandling of the system timers.

Problem Symptoms

There are two symptoms:

1. Use the dspcds command to view state of both BCCs and NPMs. Look for the standby BCC or NPM to be in the Cleared state.

   For the BPX8600:

      FrontCard    BackCard                    FrontCard    BackCard
           Type    Rev  Type  Rev   Status          Type    Rev  Type  Rev   Status
        1  BXM-155 CDE  SM-8  BA    Active       9  Empty                               
        2  BXM-155 CDE  SM-8  BA    Active       10 Empty                               
        3  BXM-155 CJF  SM-8  BA    Active       11 Empty                               
        4  BXM-155 ENF  SM-8  BC    Active       12 Empty                               
        5  BXM-622 CHE  SM-2  BB    Active       13 Empty                               
        6  BXM-622 CHE  SM-2  BB    Active       14 Empty                               
        7  BCC-4   HBM  LM-2  AC    Cleared      15 ASM     ACC  LMASM AC    Active
        8  BCC-4   HBM  LM-2  AC    Active     
   

   For the IGX8400:

   FrontCard  BackCard                      FrontCard  BackCard
           Type  Rev  Type     Rev  Status          Type  Rev  Type     Rev  Status
        1  NPM   EFS                Cleared      9  Empty
        2  NPM   EFS                Active      
   

2. Use the command dspswlog. Look for error number 586.

   Active CC's Software Log
        No.  Type    Number      Data(Hex)  PC(Hex)    PROC   SwRev   Date     Time
         1.  Error   586         000B0001   302F1D84   RSRC   9.1.09  11/19/03 231400
   

If both of these conditions exist, the BPX8600 or IGX8400 will need to be rebuilt. This requires scheduling a maintenance period and resetting both of the BCC cards to reset the affected timer.

Workaround/Solution

Workaround:

The workaround is a preventative measure. Cisco recommends resetting the BCC or NPM cards as part of a regularly scheduled preventative maintenance activity using the following procedure:

1. Reset the standby BCC/NPM using the resetcd standby_slot h command.

2. Wait for the standby BCC/NPM to come to standby state.

3. Use the command dspqs to see if the standby BCC or NPM has updates pending. The command should elicit a response that indicates:

   Standby CC Connection Database Updates Pending: No.

   If there are updates pending, wait until the updates clear before proceeding.

4. Switch BCC or NPM by using the switchcc command.

5. Wait for the previously active BCC or NPM to reach standby.

6. Log into each node in the network and use the dspnds command to verify that there are no unreachable nodes.

7. If there is an unreachable node, toggle the comm break state machine by issuing the command off1 5 followed by on1 5 at the node reporting the unreachable node.

8. Schedule the next BCC or NPM reset activity before 1200 days hence.

Solution:

Monitor bug ID CSCea41956 (registered customers only) for resolution using the Cisco Bug Toolkit (registered customers only) .

Recovery:

If the anomaly manifests itself and the standby BCC/NPM is in the cleared state, the BPX8600/IGX8400 must be rebuilt using the following procedure:

1. Physically unseat the standby BCC/NPM which is in cleared state.

2. Issue the command resetcd BCC/NPM_slot_number r to attempt a hitless rebuild. There will be a short traffic outage, so this activity should be planned for a maintenance window.

3. After the rebuild completes, reinstall the standby BCC/NPM removed in step 1.

4. If this procedure does not resolve the Cleared standby BCC/NPM problem, a full rebuild must be attempted.

5. Physically remove the standby BCC or NPM.

6. Issue the command resetcd BCC or NPM_slot_number h on the active BCC or NPM to attempt a full rebuild. There will be a longer traffic outage, so this activity should be planned for a maintenance window.

7. Wait for the BCC or NPM rebuild to complete.

8. Reinstall the standby BCC or NPM, or issue the command upcd slot# .

9. Go to each node and issue the dspnds command to verify there are no unreachable nodes.

10. If an unreachable node is found, issue the command off1 5 followed by the command on1 5 to toggle the comm break state machine and clear the unreachable node. Note that unreachable nodes may be caused by trunk outages as well, so verify that there are no trunk outages that isolate a node from the network before proceeding with the recovery procedure.

Note that for this anomaly, the reset of each controller card should be scheduled at least every 1200 days to prevent this problem from occuring. See also currently unresolved bugs CSCdx48018 (registered customers only) and CSCea76950 (registered customers only) . Cisco recommends regular exercising of controller card and I/O card redundancy. It is up to each network administrator to determine how often such exercising should take place based on their network's operational needs.

DDTS

To follow the bug ID link below and see detailed bug information, you must be a registered user and you must be logged in.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email
• By telephone

Receive Email Notification For New Field Notices

Product Alert Tool - Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.