Guest

Cisco BPX/IGX/IPX WAN Software

Field Notice: IPX 32 Node Rebuilds During Switch Software Upgrade From Version 8.2.59 To Version 9.1.03


December 28, 1998



Products Affected

Product

Remarks

IPX 32

The IPX 32 is affected only if it is equipped with either of the following:

  • IPX-SCC-6236A Hardware Revision A, B, C, D, E, F, G, H, I, or J

  • IPX-SCC-6236B Hardware Revision A, B, or C

Problem Description

After the first loadrev 9.1.03 nodename command is issued during the switch software upgrade process, the IPX 32 goes into continuous Bus Diagnostic failures until CC redundancy is disabled. Please note that this problem has been observed in the field in a single network and has not been duplicated by the Cisco TAC or Cisco Engineering.

Problem Symptoms

There are one or more symptoms that the IPX network operator can observe. These are listed below:

Use the dsplog command to read the node event log. The user will see the following entries in the log:

Major  A and B Buses Failed (STBY CLOCK)
Info   NPC 17 Starting rebuild due to Bus Diagnostic
Info   NPC 1 Restarted due to a Completed Download
Info   NPC 1 Removed
Info   Standby NPC 1 Update Completed
Info   Standby NPC 1 downloaded Revision 9.1.03
Info   NPC 1 Restarted due to a Reset Request
Info   NPC 1 Removed
Info   Standby NPC 1 Update Completed
Info   Standby NPC 1 downloading Revision 9.1.03

Note:?The event log is in reverse chronological order; that is, the earliest events are at the end, or "bottom," of the log while the latest events are at the beginning, or "top," of the log.

In the extreme case, the node may go into the "automatic bus diagnostic" mode which results in a node rebuild. The node rebuild may last several minutes and will cause severe traffic disruption. In this condition, the user may observe unreachable nodes by using the dspnds command:

NodeName   Alarm
RemoteNode UNRCH

Or, the dsplog command may show the following entries:

Minor  Communication Break with RemoteNode

Please note that the classic "auto bus diagnostic" mode causes all cards, except the processor cards to be logically removed. This is the cause for the node unreachability as well as traffic disruption.

Background

The IPX Node upgrade process requires several steps to accomplish a graceful upgrade and thereby reducing the possibility of affecting customer's traffic. One of the commands to initiate the upgrade is the loadrev 9.1 nodename command which loads the new software image into different memory locations on the standby and active NPC-32s (Node Processor Card-32 Megabyte RAM). Ordinarily this command is not service affecting, however, the manifestation of this problem (CSCdk65393) causes the standby NPC to log Bus Diagnostic events and prevents the 9.1.03 image from downloading to the standby NPC.

Bus Diagnostics are invoked when errors are detected on the IPX MUXBUS. There are several buses in the IPX backplane, including a clock bus, a "C" (control) bus, a databus and a power bus. These buses are redundant and are designated as "A" and "B." Upon detection of an error on the active bus, the controller card switches to the standby bus and logs an event in the node event log (use the dsplog command) as well as declaring a minor alarm for the node. Use the command dspalms to see the alarm state of the IPX node. If another error is than subsequently detected on the newly activated bus, while the recently deactivated bus is still in alarm, the node declares both buses as failed and initiates the automatic bus diagnostic routine, which essentially isolates the node from the network by logically removing all cards, including the trunk cards. In the case of this bug (CSCdk65393) the error is detected on the clock bus. Switch Software 9.1 has introduced improved clock self-testing for each node that is not compatible with the earlier versions of the IPX-SCC-LEC used in the IPX 32.

This problem has been assigned the software defect identification CSCdk65393.

If you are a registered user and you have logged in, you can view bug details.

CSCdk65393

Workaround/Solution

There is no known workaround. However, the recovery procedure and solution are described below:

To complete the recovery procedure, use the following four steps:

  1. Issue the loadrev 8.2.59 nodename command.

  2. Issue the cnfnodeparm 16 n command to disable NPC redundancy.

  3. Issue the diagbus command to clear bus failure.

  4. Issue the cnfnodeparm 16 y command to enable NPC redundancy.

Listed below is the solution to this defect:

  1. Replace the IPX-SCC-6236A or IPX-SCC-6236B with the latest hardware revision. The latest IPX-SCC-6236A hardware revision is "M." The latest IPX-SCC-6236B is also "M."

  2. Upgrade to switch software 9.1.06 instead of 9.1.03 or 9.1.04. Switch software 9.1.06 and later versions do not exhibit this problem.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods: