Troubleshoot IOS Hypervisor and System Image Recovery for CGR 1000

Introduction

This document describes recovery steps for hypervisor and system image on  Cisco 1000 Series Connected Grid Router (CGR 1000) that run IOS software. If hypervisor or system image is corrupted, this procedure can help you bring CGR 1000 router back online. IOS configuration is stored in NVRAM. Even if another Cisco Secure Digital (SD) card is used, the configuration will not be deleted unless the running-config was specified to be stored at another location.

Prerequisites

• Trivial File Transfer Protocol (TFTP) server installed on the local computer
• Setup the TFTP server to have the hypervisor and the system image  

Requirements

Cisco recommends that you have knowledge of these topics:

• Console Cable
• CAT5
• Hypervisor image, system image, and bundle image

Components Used

The information in this document is restricted to only IOS version running on CGR 1120 and CGR1240.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Steps to Recover Hypervisor and System Image

Download Hypervisor Images and System Images

1. On Cisco download any images cgr1000-universalk9-bundle.xxx.xxx.xxx.bin.

2. Use 7z to open the bin file.

3. Double click the file to browse the content.

4. Download hypervisor image (cgr1000-hv.srp.SPA.1.1.1) and  system image (cgr1000-universalk9-mz.SPA.155-2.T).

Move these three files onto the TFTP server.

• cgr1000-hv.srp.SPA.1.1.1 (hypervisor image)

• cgr1000-universalk9-mz.SPA.155-2.T (system image)

• cgr1000-bundle-universalk9-bundle.SPA.155-2.T.bin (bundle image)

CGR Boot Sequence for IOS

The order of boot sequence is:

Power on > BIOS > Hypervisor Image > System image > Normal Operation for IOS

If there is no hypervisor image, the user sees this:

IOFPGA @ 0xd0000000 version=0x23020900, datecode=0xd091e17 CPLD version 0x13

Reset reason (0.0):   Unknown

BIOS Version: Build # 14 - Wed 04/30/2014

CGR Loader Stage 1 Version: 1.9.16

Autoboot string bootstrap:cgr1000-hv.srp.SPA.1.1.0

Booting image: bootstrap:cgr1000-hv.srp.SPA.1.1.0....

Autoboot failed with error=1



rommon-1>      

If there is no system image (IOS image), the user will see this:

IOFPGA @ 0xd0000000 version=0x23020900, datecode=0xd091e17 CPLD version 0x13

Reset reason (0.0):   Unknown

CGR Loader Stage 2 Version: 1.9.16

Autoboot string flash:/cgr1000-universalk9-mz.SPA.154-3.M1,12;

rommon-2>

Recovery Steps 

1. Setup console cable using putty.

2. Connect the CAT5 cable from the local pc NIC to the CGR ETH 2/2 . This is the only interface that works during the recovery process.

3. Setup the local PC NIC to be in the same subnet as the CGR. 

For example; PC NIC is 192.0.2.1 subnet 255.255.255.0.

For the CGR will be 192.0.2.2 subnet 255.255.255.0.

4. In the Putty session, you see rommon-1>  if the CGR can not find the hypervisor image.

IOFPGA @ 0xd0000000 version=0x23020900, datecode=0xd091e17 CPLD version 0x13
Reset reason (0.0):   Unknown
BIOS Version: Build # 14 - Wed 04/30/2014 
CGR Loader Stage 1 Version: 1.9.16
Autoboot string bootstrap:cgr1000-hv.srp.SPA.1.1.0
Booting image: bootstrap:cgr1000-hv.srp.SPA.1.1.0....
Autoboot failed with error=1
    
rommon-1>     

5. Setup the IP address of ETH2/2 with the command set ip.
    

set ip 192.0.2.2 255.255.255.0
    
Correct - ip addr is 192.0.2.2, mask is 255.255.255.0
Found Intel IOH GBE [2:0.1] at 0xe020, ROM address 0x0000
Probing...[Intel IOH GBE]
MAC address bc:16:65:31:58:b2
External PHY link UP @ 1000/full
Address: 192.0.2.2
Netmask: 255.255.255.0
Server: 0.0.0.0
Gateway: 0.0.0.0  

6. Setup the Gateway Address to be the local PC NIC with the command set gw.

set gw 192.0.2.1
Correct gateway addr 192.0.2.1
Address: 192.0.2.2
Netmask: 255.255.255.0
Server: 0.0.0.0
Gateway: 192.0.2.1

7. Boot the hypervisor image from the local tftp server with the command boot tftp://.
    

Boot tftp://192.0.2.1/cgr1000-hv.srp.SPA.1.1.1
Booting: /cgr1000-hv.srp.SPA.1.1.1 console=ttyS0,9600n8nn quiet loader_ver="1.9
16"....   [Multiboot-kludge, loadaddr=0x1c100000, text-and-data=0x16d05c2
Signature verification was successful, bss=0x0, entry=0x1c10005c]
    
RIF heap: 1519616 bytes, SKH heap: 2310144 bytes
RIF: used 7691/16384 bytes of stack    

8. The screen should look like this once the CGR loads the hypervisor image and is unable to boot the system image.
    

LynxSecure TRUNK (i386; No Service Packs installed)
Copyright 2005-2014 LynuxWorks, Inc
All rights reserved.
        
LynxSecure (i386) build ENGINEERING created on 03/14/2014 13:21:02
URL:         svn://txx.lynx.com/svn/lynxsecure-svn/engr/psubramaniam/cisco/ohci/lynxsk
Revision(s): 5194M
Built by:    psubramaniam@paricos62.localdomain
Initializing the Internal Timekeeping...
Initializing the System State Manager...
Initializing LynxSecure global data areas.
Number of CPU(s) : 2
Initializing the CPU Support Package.
Initializing LynxSecure page table...
Initializing the Board Support Package.
Starting up the other CPUs...
CPUs online: #0 #1
Initializing Scheduler...
Initializing the VCPU module...
Initializing Device Configuration Virtualization...
Initializing Subject Resources...
Initializing Interrupt Routing...
Initializing Hypercalls...
Heap memory used by LynxSecure: 2240444 (0x222fbc) bytes
Launching Subjects
    
IOFPGA @ 0xd0000000 version=0x23020900, datecode=0xd091e17 CPLD version 0x13
Reset reason (0.0):   Unknown
    
CGR Loader Stage 2 Version: 1.9.16
    
rommon-2>

9. Setup the IP address of ETH2/2 with the command set ip.

set ip 192.0.2.2 255.255.255.0
    
Correct - ip addr is 192.0.2.2, mask is 255.255.255.0
Found Intel IOH GBE [2:0.1] at 0xe020, ROM address 0x0000
Probing...[Intel IOH GBE]
MAC address bc:16:65:31:58:b2
External PHY link UP @ 1000/full
Address: 192.0.2.2
Netmask: 255.255.255.0
Server: 0.0.0.0
Gateway: 0.0.0.0

10. Setup the Gateway Address to be the local PC NIC with the command set gw.

set gw 192.0.2.1
Correct gateway addr 192.0.2.1
Address: 192.0.2.2
Netmask: 255.255.255.0
Server: 0.0.0.0
Gateway: 192.0.2.1

11. Boot the system image from the tftp server with the command boot tftp://.

Boot tftp://192.0.2.1/cgr1000-universalk9-mz.SPA.155-2.T
    
Booting: /cgr1000-universalk9-mz.SPA.155-2.T console=ttyS0,9600n8nn quiet loade
r_ver="1.9.16"....   [Multiboot-elf, <0x110000:0xc599aec:0x6667dc>, shtab=0xcd1
1500
Signature verification was successful, entry=0x110240]
    
Smart Init is enabled
smart init is sizing iomem
                     TYPE      MEMORY_REQ
Onboard devices &
             buffer pools      0x02E44000 
-----------------------------------------------
                   TOTAL:      0x02E44000
    
Rounded IOMEM up to: 47MB.
Using 10 percent iomem. [47MB/448MB]
    
                  Restricted Rights Legend
    
Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.
    
cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706
    
    
    
Cisco IOS Software, cgr1000 Software (cgr1000-UNIVERSALK9-M), Version 15.5(2)T, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Wed 25-Mar-15 17:01 by prod_rel_team
    
    
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
    
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
    
If you require further assistance please contact us by sending email to export@cisco.com.
    
Installed image archive
    
Reading module 3 idprom, please wait......
    
..........
Reading module 4 idprom, please wait......
    
Cisco CGR1240/K9 (revision 1.0) with 373760K/52224K bytes of memory.
Processor board ID JAF1720BBGS
Last reset from Power-on
    
FPGA version: 2.9.0 
    
2 Serial(sync/async) interfaces
4 FastEthernet interfaces
3 Gigabit Ethernet interfaces
6 terminal lines
1 802.11 Radio
1 Cellular interface
DRAM configuration is 72 bits wide with parity disabled.
256K bytes of non-volatile configuration memory.
524320K bytes of ATA System Flash (Read/Write)
262176K bytes of ATA Bootstrap Flash (Read/Write)

12. If the NVRAM is still intact, the running-configuration will load up. The router should still have the old configuration saved.

    
    
13. (Optional) If a new SD card is put into the CGR, partition the new SD card with the command partition flash:. Otherwise, this step can be skipped if the current SD card is confirmed to be good.

format flash: 
Format operation may take a while. Continue? [confirm]
Format operation will destroy all data in "flash:".  Continue? [confirm]
Format: All system sectors written. OK...
    
Format: Total sectors in formatted partition: 1048257
Format: Total bytes in formatted partition: 536707584
Format: Operation completed successfully.
    
Format of flash: complete   

14. In IOS, gigabitethernet2/2 is the port for ETH2/2 on the physical box. Configure gigabitethernet2/2 with IP address 192.0.2.2 so that you can copy the bundle image from the TFTP server.

Configure terminal
Interface gigaethernet2/2
Ip address 192.0.2.2 255.255.255.0
No shut

    14. Copy the bundle image from the tftp to the CGR flash with the command copy tftp: flash:

copy tftp: flash:
Source filename [cgr1000-universalk9-bundle.SPA.155-2.T.bin]? cgr1000-universalk9-bundle.SPA.155-2.T.bin
Destination filename [cgr1000-universalk9-bundle.SPA.155-2.T.bin]? cgr1000-universalk9-bundle.SPA.155-2.T.bin
Accessing tftp://192.0.2.1/cgr1000-universalk9-bundle.SPA.155-2.T.bin...
Loading cgr1000-universalk9-bundle.SPA.155-2.T.bin from 192.0.2.1 (via GigabitEthernet2/2): !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[OK - 163484669 bytes]
    
163484669 bytes copied in 570.760 secs (286433 bytes/sec)

15. To install the hypervisor image, system image, and guest OS use the command bundle install flash:   <bundle image>.
    

CGR1240-AST04#bundle install flash:/cgr1000-universalk9-bundle.SPA.155-2.T.bin
Installing bundle image: /cgr1000-universalk9-bundle.SPA.155-2.T.bin...........................................................................................................................................................................................................................................................................................................................................................................................................................................
     
updating Hypervisor image... 
Sending file modes: C0444 23922114 cgr1000-hv.srp.SPA.1.1.1
    
SRP md5 verification passed!
    
updating IOS image... 
Sending file modes: C0664 79967086 cgr1000-universalk9-mz.SPA.155-2.T
    
IOS md5 verification passed!
Done!
    
CGR1240-AST04#
*Mar 27 18:35:09.805: %SYS-5-CONFIG_I: Configured from console by bundle install command
*Mar 27 18:35:09.805: %CGR1K_INSTALL-6-SUCCESS_BUNDLE_INSTALL: Successfully installed bundle image.