Guest

Cisco 3200 Series Rugged Integrated Services Routers

Field Notice: FN - 63170 - Cookie Corruption on Cisco 3200 Causes Router to Fail to Boot - ROMmon Upgrade Required

Field Notice: FN - 63170 - Cookie Corruption on Cisco 3200 Causes Router to Fail to Boot - ROMmon Upgrade Required

Revised January 2, 2009
December 2, 2008


NOTICE:

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision Date Comment
1.1
02-JAN-2009
Added IOS upgrade instructions to Workaround/Solution.
1.0
02-DEC-2008
Initial Public Release

Products Affected

Products Affected
3200 - C3251MARC-TP 
3200 - CISCO3251MARC 
3220 SRS - C3220-1-WMIC-K9 
3220 SRS - C3231MARC 
3220 SRS - C3231MARC-TP 
3220 SRS - CISCO3220 
3220 SRS - CISCO3251MARC 
3230 - C3230-1W-49-K9 
3230 - C3230-1WMIC-K9 
3230 - C3230-24-49-K9 
3230 - C3230-2W24-49-K9 
3230 - C3230-2WMIC-K9 
3230 - C3230-2WMICE-K9 
3230 - C3230-2WMICJ-K9 
3230 - C3230-2WMICJ10-K9 
3230 - C3230-3W-49-K9 
3230 - C3230-3WMIC-K9 
3230 - C3230-3WMICE-K9 
3230 - C3230-K9 
3230 - C3230ENC-1W-49-K9 
3230 - C3230ENC-1WMIC-K9 
3230 - C3230ENC-24-49-K9 
3230 - C3230ENC-2W24-49K9 
3230 - C3230ENC-2WJ-K9 
3230 - C3230ENC-2WJ10-K9 
3230 - C3230ENC-2WMIC-K9 
3230 - C3230ENC-2WMICE-K9 
3230 - C3230ENC-3W-49-K9 
3230 - C3230ENC-3WMIC-K9 
3230 - C3230ENC-3WMICE-K9 
3230 - C3230ENC-K9 
3230 - C3230TP-1W-49-K9 
3230 - C3230TP-1WMIC-K9 
3230 - C3230TP-1WMICE-K9 
3230 - C3230TP-2WMIC-K9 
3230 - C3230TP-2WMICE-K9 
3230 - C3230TP-3WMIC-K9 
3230 - C3230TP14-3WMIC-K9 
3230 - C3230TP24-49-K9 
3230 - C3231-1WMIC-K9 
3230 - C3231-K9 
3230 - C3231MARC 
3230 - C3231MARC-TP 
3230 - C3231TP-1WMIC-K9 
3230 - C3231TP-2WMIC-K9 
3230 - CISCO3251MARC 

Problem Description

The Cisco 3220, 3230 and 3250 Rugged ISR could lose connectivity. The unit will fail to boot up and a corrupt cookie error message will displayed.

The cookie needs to be reprogrammed to restore service. This is not a process performed in the field, so replacement of the CISCO3251MARC is required.

A new release of 3200 IOS ROMMON Software has been released to prevent this error condition which results in hardware replacement.

Background

Failure Analysis on RMA units indicated that the cookie has corrupted data in the first 28 bytes with zeros. This seems to be a deliberate write rather than the result of any external conditions, given the fact that these units subsequently passed all tests under margin conditions after the cookie value was restored. ROMMON code review and testing revealed a flaw which allows corrupted RAM data to be saved to flash and would cause data corruption as experienced by customer.

Engineering Change Order E095161 will be introduced into manufacturing to install the new ROMMON at a future date to be determined.

All units prior to this are susceptible to this error condition. Upgrading of IOS ROMmon software is recommended to prevent this condition. An IOS software image is available to perform this upgrade.

Problem Symptoms

The Cisco 3220, 3230, and 3250 Rugged ISR will fail to boot and a corrupt cookie error message will be displayed.

System Bootstrap, Version 12.3(2r)XA, RELEASE SOFTWARE (fc1)
TAC Support:
http://www.cisco.com/tac
Copyright (c) 2003 by cisco Systems, Inc.
WARNING: Cookie information is corrupt

C3200 platform with 131072 Kbytes of main memory

rommon 1 > dir flash
unable to stat flash/: permission denied
rommon 2 > meminfo

Main memory size: 128 MB.
Available main memory starts at 0x1b000, size 130964KB
IO (packet) memory size: 25 percent of main memory.
NVRAM size: 128KB
rommon 3 >
rommon 4 > cookie

WARNING: Cookie information is corrupt

cookie:
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
rommon 5 >

Workaround/Solution

New IOS ROMmon software, C3200_RM_ALT.srec.124-15r.T1, has been released for the Cisco 3220, 3230 and 3250 Rugged ISR that will prevent cookie corruption from occuring. It can be down loaded from Cisco.com/Support/Download Software by customers with access to Software Downloads.

Download and install C3200_RM_ALT.srec.124-15r.T1 (registered customers only).

The path for locating the Software is Routers > Cisco 3230 Rugged Integrated Services Router > IOS ROMMON Software > 12.4(15r)T1 > C3200_RM_ALT.srec.124-15r.T1

Cisco 3220, 3230, 3250 Rugged ISR ROMmon Upgrade Procedure

1. Configure IP and route on 3250 that would allow 3250 to reach the tftp server.

c3250router# config t
Enter configuration commands, one per line. End with CNTL/Z.
c3250router(config)#int f0/0
c3250router(config-if)#ip address
c3250router(config-if)#no shut
c3250router(config-if)#end

2. Ping gatway and tftp server from 3250 to verify connectivity

c3250router#ping

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms
c3250router#ping

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds:
!!!!!

3. Download the latest image with IOS rommon upgrade support

c3250router#copy tftp flash
Address or name of remote host []?
Source filename []?
Destination filename []?
Accessing tftp:// /...
Loading from (via FastEthernet0/0): !O!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[OK - 17104468 bytes]

17104468 bytes copied in 229.112 secs (74655 bytes/sec)

4. Reload the router to boot the new IOS image with rommon upgrade support

5. Verify current rommon and IOS version by issuing the show version command:

c3250router#sh ver
Cisco IOS Software, 3200 Software (C3250-ADVENTERPRISEK9-M), Version 12.4(22)T, RELEASE SOFTWARE (fc1) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2008 by Cisco Systems, Inc.
Compiled Fri 10-Oct-08 03:36 by prod_rel_team

ROM: System Bootstrap, Version 12.4(15r)T1, RELEASE SOFTWARE (fc1)

c3250router uptime is 19 hours, 0 minutes
System returned to ROM by power-on
System image file is "flash:c3250-adventerprisek9-mz.124-22.T"

This product contains cryptographic features and is subject to United States and local country laws governing import, export, transfer and use. Delivery of Cisco cryptographic products does not imply third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for compliance with U.S. and local country laws. By using this product you agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to export@cisco.com.

Cisco 3250 (MPC8250) processor (revision 0x400) with 98304K/32768K bytes of memory.
Processor board ID FHH12090052
MPC8250 CPU Rev: Part Number 0x0, Mask Number 0x64
5 FastEthernet interfaces
4 Serial(sync/async) interfaces
128K bytes of non-volatile configuration memory.
32768K bytes of processor board System flash (Intel 28F128J3A)

Configuration register is 0x2102

6. Issue the rom-monitor upgrade command:

c3250router#upgrade rom-monitor file tftp:///

Loading from (via FastEthernet0/0): !!!
[OK - 688693 bytes]

image size: 278452

This command will result in a 'power-on reset' of the router!

Continue? [yes/no]:

7. After the rommon upgrade, the router will ask to reboot. Press "y" for yes

8. After reboot, the new rommon will be active.

c3250router#sh ver
Cisco IOS Software, 3200 Software (C3250-ADVENTERPRISEK9-M), Version 12.4(22)T, RELEASE SOFTWARE (fc1) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2008 by Cisco Systems, Inc.
Compiled Fri 10-Oct-08 03:36 by prod_rel_team

ROM: System Bootstrap, Version 12.4(15r)T1, RELEASE SOFTWARE (fc1)

c3250router uptime is 19 hours, 0 minutes
System returned to ROM by power-on
System image file is "flash:c3250-adventerprisek9-mz.124-22.T"

This product contains cryptographic features and is subject to United States and local country laws governing import, export, transfer and use. Delivery of Cisco cryptographic products does not imply third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for compliance with U.S. and local country laws. By using this product youagree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to export@cisco.com.

Cisco 3250 (MPC8250) processor (revision 0x400) with 98304K/32768K bytes of memory.
Processor board ID FHH12090052
MPC8250 CPU Rev: Part Number 0x0, Mask Number 0x64
5 FastEthernet interfaces
4 Serial(sync/async) interfaces
128K bytes of non-volatile configuration memory.
32768K bytes of processor board System flash (Intel 28F128J3A)

Configuration register is 0x2102

For users currently using images released prior to 12.4(2)T and attempting to upgrade to 12.4.22T, steps 1 through 5 of the procedure above will need to be replaced with the following steps.

1. Before upgrading Cisco IOS to 12.4(22)T, copy the router's running-config to the flash filesystem:

c3250router#copy running-config flash:saved-config

2. Configure IP and route on 3250 that would allow 3250 to reach the tftp server.

c3250router#config t
Enter configuration commands, one per line. End with CNTL/Z.
c3250router(config)#int f0/0
c3250router(config-if)#ip address
c3250router(config-if)#no shut
c3250router(config-if)#end

3. Ping gatway and tftp server from 3250 to verify connectivity

c3250router#ping

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms
c3250router#ping

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to , timeout is 2 seconds:
!!!!!

4. Download the latest image with IOS rommon upgrade support

c3250router#copy tftp flash
Address or name of remote host []?
Source filename []?
Destination filename []?
Accessing tftp:// /...
Loading from (via FastEthernet0/0): !O!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[OK - 17104468 bytes]

17104468 bytes copied in 229.112 secs (74655 bytes/sec)

5. Erase startup-config by CLI:

c3250router#write erase

6. Reload the router to boot the new IOS image with remote rommon upgrade support

7. Upon reloading the router, verify the current rommon and IOS version by issuing the show version command:

c3250router#sh ver
Cisco IOS Software, 3200 Software (C3250-ADVENTERPRISEK9-M), Version 12.4(22)T, RELEASE SOFTWARE (fc1) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2008 by Cisco Systems, Inc.
Compiled Fri 10-Oct-08 03:36 by prod_rel_team

ROM: System Bootstrap, Version 12.4(15r)T1, RELEASE SOFTWARE (fc1)

c3250router uptime is 19 hours, 0 minutes
System returned to ROM by power-on
System image file is "flash:c3250-adventerprisek9-mz.124-22.T"

This product contains cryptographic features and is subject to United States and local country laws governing import, export, transfer and use. Delivery of Cisco cryptographic products does not imply third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for compliance with U.S. and local country laws. By using this product you agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to export@cisco.com.

Cisco 3250 (MPC8250) processor (revision 0x400) with 98304K/32768K bytes of memory.
Processor board ID FHH12090052
MPC8250 CPU Rev: Part Number 0x0, Mask Number 0x64
5 FastEthernet interfaces
4 Serial(sync/async) interfaces
128K bytes of non-volatile configuration memory.
32768K bytes of processor board System flash (Intel 28F128J3A)

Configuration register is 0x2102

8. Restore the previously saved-config.

c3250router#copy flash:saved-config running-config

9. Enable the all previously enabled interfaces:

c3250router#conf t
c3250router(config)#int
c3250router(config-if)#no shut

10. Save the config by CLI

c3250router#copy running-config startup-config

Engineering Change Order E095161 will be introduced into manufacturing to install the new ROMMON at a future date to be determined.

DDTS

To follow the bug ID link below and see detailed bug information, you must be a registered customer and you must be logged in.

DDTS Description
CSCsu51482 (registered customers only) C3250 ROMMON can not recover primary cookie error

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

Receive Email Notification For New Field Notices

Product Alert Tool - Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.