Guest

Cisco 7400 Series Routers

Field Notice: *Expired* FN - 18164 - Memory Corruption on Cisco 7401ASR


Revised December 11, 2007
March 18, 2002


NOTICE:

THIS FIELD NOTICE HAS BEEN EXPIRED AND IS NO LONGER MAINTAINED OR UPDATED BY CISCO.

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE, WARRANTY OR SUPPORT. USE OF THE INFORMATION ON THIS FIELD NOTICE OR MATERIALS LINKED FROM THIS FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.


Products Affected

Product Top Assembly Comments
Part Number
CISCO7401ASR-BB   800-09092-xx   7401ASR, 256M SDRAM, Broadband Feature License  
CISCO7401ASR-CP   800-09092-xx   7401ASR,128M SDRAM, IP Software  

Problem Description

The Cisco 7401 may experience what appears to be a memory corruption issue. When this occurs, the Cisco 7401 will restart or crash. After examination of the crash information, there is an indication of memory corruption. Register contents cannot be derived from memory display.

Note: This problem occurs on 7401s with a part number version prior to 800-09092-10. 7401s with part number version 800-09092-10 or later are not subject to this problem.

Background

This problem may occur under any operating conditions and is not restricted to a special operating condition or a specific configuration. The problem seems to be more prevalent under high traffic conditions. The issue occurs when an errant bus signal causes the L3 cache controller to match an incorrect address. This causes the L3 cache controller to return the wrong data to the CPU. When this occurs, the system is restarted due to a software forced crash or a bus error.

Problem Symptoms

When a show version command is exectued after the system is restarted, you will see the following output:

System was restarted by error - a Software forced crash

or

System was restarted by bus error

Workaround/Solution

Cisco has resolved this issue in Cisco IOS® Software by using the diagnostic CLI (Command Line Interface) function cache L3 bypass to turn off (or bypass) L3 cache. When the diagnostic CLI is executed, L3 cache is turned off (this action requires a reload for it to take place). With the L3 cache removed there may be a slight drop in process performance. Some customer's benchmark tests have reported this drop off to range from as low as 1% to approximately 10 to 15%. Netflow process appears to be the most affected, but most other processes are not affected by disabling the L3 cache. For ease of deployment and supportability, this is Cisco's recommended fix for the problem. This diagnostic CLI is supported in Cisco IOS Software Releases 12.2(4)B3 and higher and 12.2(9)S and higher. Please refer to the "Additional Information" section for proper use of the L3 cache bypass commands.

Cisco has found the root cause: a signal in the L3 cache controller (which is why bypassing L3 is an effective workaround.) The L3 controller chip has been changed to eliminate the problem. This fix has been implemented in 7401s with a part number version of 800-09092-10 or higher. In order to get this fix, the Cisco 7401 needs to be replaced (accept for the Port Adapter, SDRAM and flash disk).

In summary, there are two acceptable methods to prevent this problem from occurring:

  1. IOS Method: Use the IOS command cache L3 bypass to effectively remove the L3 cache.
  2. Replacement: Replace the 7400 with a unit that has the hardware changed to eliminate the problem.

DDTS

To follow the bug ID link below and see detailed bug information, you must be a registered user and you must be logged in.

DDTS Description
CSCdv90940 Memory corruption causes software forced crash or bus error
A> Memory corruption causes software forced crash or bus error

Additional Information

L3 Cache Bypass Instructions

This new command only supports the L3 bypass diagnostic command. This command is only available on the following platforms:

  • 7200 with processor engine: NPE-300, NPE-400, or NSE-1
  • 7400 with processor engine: NSE-1

To bypass the L3 cache, the user in the configure mode issues:

  (config)# cache L3 bypass
  New cache setting will not take effect until reload.


Note: The command show running-config will display cache L3 bypass as an entry. The running configuration must be saved to the startup configuration. Upon reload the L3 cache will be bypassed. Please see bootup note below.

To use the L3 cache, the user issues:

 (config)# no cache L3 bypass
 New cache setting will not take effect until reload.

Note: The command show running-configuration will not display cache L3 bypass as an entry. The running configuration must be saved to the startup configuration. Upon reload the L3 cache will be used. Please see bootup note below.

Bootup Note: When the router boots up, system information will be displayed. It will show the L3 cache. This is because the startup-config file has not yet been processed by the system. After the startup-config file is processed, the L3 cache will be bypassed if the cache L3 bypass entry exists in the file. Issuing a show version after bootup will not show L3 cache if it is bypassed.

To verify L3 Cache state

To verify the L3 cache setting, the user can issue the show version command. If the L3 cache is bypassed, there will be no reference to the L3 cache in the show version output.

To achieve single reload via configuration file:

Method 1:

  1. Create a file on a tftp host [/directory/filename].
  2. Set the [/directory/filename] file permission to 666.
  3. On the IOS router, issue the copy startup-config tftp:[[[//location]/directory]/filename] command.
  4. Press Enter when the router confirms the host IP and filename.
  5. Add the cache L3 bypass command to [/directory/filename] file on the tftp host. Be sure to put this line somewhere in the startup-config file after the first "!" and before the "end" token. For example, place the command below the hostname routername entry.
  6. Save the changes into the [/directory/filename] file on the tftp host machine.
  7. Copy startup-config file from the tftp host to router with the following command on the router copy tftp:[[[//location]/directory]/filename] startup-config.
  8. Reload the router.

Method 2:

  1. Store the 'new-IOS' to slot0: using tftp.
  2. Change the configuration to restart with 'new-config' config by boot config slot0:new-config.
  3. Change the configuration to restart with 'new-IOS' image by boot system slot0:new-IOS.
  4. Save the configuration by copy running-config startup-config.
  5. Create a file on a tftp host [/directory/new-config].
  6. Set the [/directory/new-config] file permission to 666.
  7. On the IOS router, issue the following command copy startup-config tftp:[[[//location]/directory]/new-config].
  8. Press Enter when router confirms its host IP and filename.
  9. Add the cache L3 bypass command to [/directory/new-config] file on the tftp host. Be sure to put this line somewhere in the startup-config file after the first "!" and before the "end" token.
  10. Make sure that new-config will use the new-IOS image for bootup.
  11. Copy the startup-config file from the tftp host to router with the following command on the router copy tftp:[[[//location]/directory]/new-config] startup-config.
  12. Verify boot parameters with the show bootvar command.
  13. Manually reload the router.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods: