Troubleshoot Bootflash and System Image Recovery for CGOS CGR 1000

Introduction

This document describes troublshoot steps to recover bootflash and system image on CGR 1000.

Background Information

CGR1000 use Cisco  Secure Digital (SD) Card memory to stored kickstart images, system images, configurations, etc.  In rare occassions, the SD card gets corrupted. As a result, the kickstart image and/or the system image gets corrupted. This prevents the CGR from booting up. The sequence for the bootup is outlined in the given figure. The procedure outline in this article is to restore the CGR from a corrupted SD card memory state.

Prerequisites

1. Trivial File Transfer Protocol (TFTP)  server installed on the local PC

2. Setup the TFTP server to have the kickstart image and the system image

Requirements

Cisco recommends that you have knowledge of these topics:

1. Console Cable

2. CAT5

3. The kickstart image and the system images

Components Used

This document is restricted to only CGOS version running on CGR 1120 and CGR1240.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Recovery Steps

1. Setup console cable using putty.

2. Connect the CAT5 cable from the local pc NIC to the CGR ETH 2/2 . This is the only interface that works at the time of the recovery process.

3. Setup the local pc NIC to be in the same subnet as the CGR. 

For example; PC NIC is 192.0.2.1 subnet 255.255.255.0.

For the CGR will be 192.0.2.2 subnet 255.255.255.0.

4. In the Putty session, you see:

"loader>"



 IOFPGA @ 0xd0000000 version=0x30020700, datecode=0xc080d17 CPLD version 0x14
 Reset Reason = 0(0)
 Scratch pad test passed !!!

 BIOS Version: Build # 12 -   Wed 06/27/2012   
 CGR Loader Version: 1.00.01

  Filesystem type is ext2fs, partition type 0x83
  Filesystem type is ext2fs, partition type 0x83


              GNU GRUB  version 0.97



              CGR Loader Version 1.00.01


 loader>

5. Setup the IP address of ETH2/2 with the command "set ip".

set ip 192.0.2.2 255.255.255.0



Correct - ip addr is 192.0.2.2, mask is 255.255.255.0
Found Intel IOH GBE [2:0.1] at 0xe020, ROM address 0x0000
Probing...[Intel IOH GBE]
MAC address 78:da:6e:8:ad:e1
External PHY link UP @ 1000/full
Address: 192.0.2.2
Netmask: 255.255.255.0
Server: 0.0.0.0
Gateway: 0.0.0.0


loader>

6. Setup the Gateway Address to be the local PC NIC with the command "set gw".

set gw 192.0.2.1


Correct gateway addr 192.0.2.1
Address: 192.0.2.2
Netmask: 255.255.255.0
Server: 0.0.0.0
Gateway: 192.0.2.1

7. Boot the kickstart image from the local tftp server with the command "boot tftp://".  

loader> boot tftp://192.0.2.1/cgr1000-uk9-kickstart.5.2.1.CG4.3.SPA.bin      
Address: 192.0.2.2
Netmask: 255.255.255.0
Server: 192.0.2.1
Gateway: 192.0.2.1
 Filesystem type is tftp, using whole disk
Booting: /cgr1000-uk9-kickstart.5.2.1.CG4.3.SPA.bin console=ttyS0,9600n8nn quie
t loader_ver="1.00.01"....
...............................................................................
................Kickstart image verification Successful 
Image verification OK


INIT: Checking all filesystems ....... done.
Warning: switch is starting up with default configuration
Creating boot config file...
/etc/rc.d/rcS.d/S26check-flash: line 528: /mnt/bootloader/boot/grub/menu.lst.local: No such file or directory
cp: cannot stat `/mnt/cfg/0/boot/grub/menu.lst.local': No such file or directory
WARNING: image sync is going to be disabled after a loader netboot
Loading system software
INIT: Sending processes the TERM signal
INIT: Sending processes the TERM signal
INIT: Sending processes the

KILL signal

 8.  Once image boots up, you will see an output that is similar to this output.

Router(boot)# ?
Exec commands:
  clear     Reset functions
  config    Enter configuration mode
  copy      Copy from one file to another
  delete    Delete a file or directory
  dir       Directory listing for files
  exit      Exit from the EXEC
  find      Find a file below the current directory
  format    Format disks
  init      Initialize internal disk
  load      Load system image
  mkdir     Create new directory
  move      Move files
  no        Disable debugging functions
  pwd       View current directory
  reload    Reboot this supervisor module
  rmdir     Remove existing directory
  show      Show running system information
  sleep     Sleep for the specified number of seconds
  ssh       SSH to another system
  tail      Display the last part of a file
  telnet    Telnet to another system

9. Format the corrupted SD card with the command "format bootflash:".

This command is going to erase the contents of your bootflash:.

Do you want to continue? (y/n)  [n] y


Formatting bootflash:
Formatting started at:  Fri Feb 20 23:56:00 UTC 2015
mke2fs 1.35 (28-Feb-2004)
Formatting finished at:  Fri Feb 20 23:56:52 UTC 2015
Formatting completed

10. Run init system (option step: this command formats the SD card and wipes out everything).

Router(boot)# init system
This command is going to erase your startup-config, licenses as well as the contents of your bootflash:.

Do you want to continue? (y/n)  [n] yInitializing the system
mount: /dev/mmcblk0p4 already mounted or /bootflash busy
mount: according to mtab, /dev/mmcblk0p4 is already mounted on /bootflash
ERROR: cannot mount filesystem
cp: omitting directory `/bootflash/'
Partitioning failed

11. Setup the Management Interface (this is required in order to copy the kickstart and system image into the bootflash).

Router(boot)# configure terminal
Router(boot)(config)# interface mgmt0
Router(boot)(config-if)#ip address 192.0.2.2 255.255.255.0
Router(boot)(config-if)#no shut

12. Copy the kickstart and system image into the bootflash.

Router(boot)# copy tftp://192.0.2.1/cgr1000-uk9-kickstart.5.2.1.CG4.3.SPA.bin bootflash:
Trying to connect to tftp server......
Connection to server Established. Copying Started.....

TFTP get operation was successful
Copy complete, now saving to disk (please wait)...



Router(boot)# copy tftp://192.0.2.1/cgr1000-uk9.5.2.1.CG4.3.SPA.bin bootflash:
Trying to connect to tftp server......
Connection to server Established. Copying Started.....

TFTP get operation was successful
Copy complete, now saving to disk (please wait)...

13. Verify that the kickstart and system image is on the CGR with "dir".

Router(boot)#dir
   29167616    Feb 21 00:39:59 2015  cgr1000-uk9-kickstart.5.2.1.CG4.3.SPA.bin
  143332283    Feb 21 00:42:06 2015  cgr1000-uk9.5.2.1.CG4.3.SPA.bin
        372    Feb 23 17:55:52 2015  fpga.log
       1905    Feb 23 18:39:54 2015  mts.log

Usage for bootflash://
691462144 bytes used
8859394048 bytes free
9550856192 bytes total

14. Boot the system image with the "load bootflash" command.

Router(boot)# load bootflash:cgr1000-uk9.5.2.1.CG4.3.SPA.bin



Loading system software
Uncompressing bootflash:/cgr1000-uk9.5.2.1.CG4.3.SPA.bin..........done. (in 37 seconds)
Loading plugin 0: core_plugin...

INIT: Switching to runlevel: 3
INIT: Sending processes the TERM signal
Router(boot)# 
INIT: 
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
System is coming up ... Please wait ...
2015 Feb 21 00:47:56  %$ VDC-1 %$ %COPP-2-COPP_NO_POLICY: Control-plane is unprotected.System is coming up ... Please wait ...
2015 Feb 21 00:47:58  %$ VDC-1 %$ %VDC_MGR-2-VDC_ONLINE: vdc 1 has come online 2015 Feb 21 00:47:58 Router %$ VDC-1 %$ %PLATFORM-2-

INPUT_POWER_SOURCE_TRANSITION: Three Phase and DC Input Status Alert:  L1 Phase ON, L2 Phase OFF, L3 Phase OFF, DC Input Absent

15. Once the CGR boots up into the system image, you need to install the kickstart and system image with "install all bootflash:<kickstart-image> system bootflash:<system image>.

cgr1120# install all kickstart bootflash:cgr1000-uk9-kickstart.5.2.1.CG4.3.SPA.bin system bootflash:cgr1000-uk9.5.2.1.CG4.3.SPA.bin

Verifying image bootflash:/cgr1000-uk9-kickstart.5.2.1.CG4.3.SPA.bin for boot variable "kickstart".
 -- SUCCESS

Verifying image bootflash:/cgr1000-uk9.5.2.1.CG4.3.SPA.bin for boot variable "system".
 -- SUCCESS

Verifying image type.
 -- SUCCESS

Extracting "system" version from image bootflash:/cgr1000-uk9.5.2.1.CG4.3.SPA.bin.
 -- SUCCESS

Extracting "kickstart" version from image bootflash:/cgr1000-uk9-kickstart.5.2.1.CG4.3.SPA.bin.
 -- SUCCESS

Extracting "bios" version from image bootflash:/cgr1000-uk9.5.2.1.CG4.3.SPA.bin.
 -- SUCCESS

Checking for Battery Power Mode.
 -- SUCCESS

Checking for Module Power Status.
 -- SUCCESS

Checking for WPAN upgrade compatibility.
 -- SUCCESS

Performing module support checks.
 -- SUCCESS

Notifying services about system upgrade.
 -- SUCCESS



Compatibility check is done:
Module  bootable          Impact  Install-type  Reason
------  --------  --------------  ------------  ------
     1       yes      disruptive         reset  Hitless upgrade is not supported



Images will be upgraded according to following table:
Module       Image                  Running-Version(pri:alt)           New-Version  Upg-Required
------  ----------  ----------------------------------------  --------------------  ------------
     1      system                              5.2(1)CG4(3)          5.2(1)CG4(3)            no
     1   kickstart                              5.2(1)CG4(3)          5.2(1)CG4(3)            no
     1        bios   v16.1.0(10/15/2013):V12.1.0(06/27/2012)   v16.1.0(10/15/2013)            no
     1        fpga                                   2.07.00               2.07.00            no


Switch will be reloaded for disruptive upgrade.

Do you want to continue with the installation (y/n)?  [n] y