How to Recover IC3000 from Console

Introduction

This document describes how to recover the IC3000 with the use of console.

How to Recover IC3000 from Console

In certain cases, the IC3000 becomes unbootable and is stuck in rommon> whilst not able to boot it's image.

When this is the case, the serial/usb console can show this output:

autoboot: Restarting the system.

Rom image verified correctly

Cisco Systems ROMMON, Version 1.0.0(IC3000), RELEASE SOFTWARE
Copyright (c) 1994-2018  by Cisco Systems, Inc.
Compiled Thu 09/06/2018 11:38:52.09 by builders

Current image running: Boot ROM1
Last reset cause: LocalSoft
DIMM Slot 0 : Present

Platform IC3000-2C2F-K9 with 8192 Mbytes of main memory
MAC Address: 00:00:00:00:00:00

Use BREAK or ESC to interrupt boot.
Use SPACE to begin boot immediately.

Warning: filesystem is not clean
Directory .boot_string not found
Unable to locate .boot_string directory
Unable to load .boot_string
Attempt autoboot: "boot disk0:"
Warning: filesystem is not clean
Warning: filesystem is not clean
File size is 0x01360000
Located IC3000-K9-1.0.1.SPA
Image size 20316160 inode num 12, bks cnt 4960 blk size 8*512
#######################################################################################################################################################################################################

Signature verification failed for key# 1
Failed to validate digital signature
Signature verification failed for key# 1
Signature verification failed for key# 2
Failed to validate digital signature
LFBFF signature verification failed!!
No kernel found to launch.
boot: cannot determine first file name on device "disk0:/"
autoboot: All boot attempts have failed.
autoboot: Restarting the system.

When you interrupt the boot process and try to boot the image, present on disk0:, you might see the same output as above and the device is basically in some kind of bootloop.

Solution

In order to recover the IC3000, you need these prerequisites:

• IP connectivity to the management port
• TFTP-server, reachable by the network on the management port
• IC3000 image, available for download from: https://software.cisco.com/download/home/286321941/type/286322235
• Console connection to the serial or USB connection on the IC3000

These steps allow you to get the IC3000 back in a usable state:

1. Download the IC3000 image from the URL provided above and make it available on the TFTP-server
2. Connect to the IC3000 console and interrupt the startup process by pressing ESC when you see this message:
   

   Use BREAK or ESC to interrupt boot.
   Use SPACE to begin boot immediately.

3. After you break the normal boot sequence, you end up with the rommon> prompt:
   

   rommon 1 >

4. Configure the IP address and gateway for the IC3000 on the management port. Even if no gateway is required, a value has to be set:
   

   rommon 1 > address 192.168.100.2
   rommon 2 > netmask 255.255.255.0
   rommon 3 > gateway 192.168.100.1

5. Test connectivity to the TFTP-server:
   

   rommon 4 >  ping 192.168.100.1
   Sending 10, 32-byte ICMP Echoes to 192.168.100.1 timeout is 4 seconds
   ?!!!!!!!!!
   Success rate is 90 percent (9/10)

6. Set the TFTP-server address and filename to download:
   

   rommon 5 > server 192.168.100.1
   rommon 6 > file IC3000-K9-1.0.1.SPA

7. Start the download and boot the image from the TFTP-server:
   

   rommon 7 > tftpdnld
                ADDRESS: 192.168.100.2
                NETMASK: 255.255.255.0
                GATEWAY: 192.168.100.1
                 SERVER: 192.168.100.1
                  IMAGE: IC3000-K9-1.0.1.SPA
                MACADDR: 00:00:00:00:00:00
              VERBOSITY: Progress
                  RETRY: 40
             PKTTIMEOUT: 7200
                BLKSIZE: 1460
               CHECKSUM: Yes
                   PORT: GbE/0
                PHYMODE: Auto Detect
   
   Receiving IC3000-K9-1.0.1.SPA from 192.168.100.1!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   File reception completed.

 Once the image has been transferred successfully, the IC3000 will immediately boot from the image:

File reception completed.
Boot buffer bigbuf=348bd018
Boot image size = 102729968 (0x61f88f0) bytes
[image size]      102729968
[MD5 signaure]    294a052497277c330d6b2159cf37f1ab
LFBFF signature verified.
[    4.446627] sd 2:0:0:0: [sdb] No Caching mode page found
[    4.510305] sd 2:0:0:0: [sdb] Assuming drive cache: write through
INIT: version 2.88 booting
Starting udev
Populating dev cache
INIT: Entering runlevel: 5postinsts/000-monit...
Configuring network interfaces... Setting bridge MAC address to: 00:b8:b3:80:02:c0
done.
Starting system message bus: dbus.
Checking and Mounting BOOT filesystem...
fsck (busybox 1.24.1, 2018-09-13 06:16:00 UTC)
BOOT was not cleanly unmounted, check forced.
BOOT: Inode 12, i_size is 20316160, should be 20447232.  FIXED.
BOOT: Inode 12, i_blocks is 39728, should be 39984.  FIXED.
BOOT: 12/244320 files (0.0% non-contiguous), 22254/976892 blocks
Checking and Mounting BOOT filesystem...Done
Checking GOLDEN filesystem...
fsck (busybox 1.24.1, 2018-09-13 06:16:00 UTC)
GOLDEN was not cleanly unmounted, check forced.
GOLDEN: 12/122160 files (8.3% non-contiguous), 33504/488448 blocks
Checking GOLDEN filesystem...Done
Checking and Mounting SYSTEM filesystem...
fsck (busybox 1.24.1, 2018-09-13 06:16:00 UTC)
SYSTEM: clean, 11/535392 files, 71084/2139136 [   21.111486] fpga_i2c_init_module: FPGA base address = ffffc90001078000
blocks
Checking and Mounting SYSTEM filesystem...Done
Checking and Mounting IOX filesystem...
fsck (busybox 1.24.1, 2018-09-13 06:16:00 UTC)
IOX: clean, 11/5865472 files, 415148/23442851 blocks
Checking and Mounting IOX filesystem...Done
Checking and Mounting LOG filesystem...
fsck (busybox 1.24.1, 2018-09-13 06:16:00 UTC)
LOG: clean, 11/244800 files, 33670/977949 blocks
Checking and Mounting LOG filesystem...Done
0x0000
Authenticating ACT2...ACT2 success

System initializing...
System Type Fiber.
Mounting cgroups...Done
Checking if cgroup is provided by kernel...Yes.
Checking if cgroup is mounted...Yes.
Checking if lssubsys is available...Yes.
Checking if platform defines cgroup parameters...Yes.
Tweaking base cgroup parameters...Done.
Checking if subsystems needed by IOx exist...
Setting up cpu cgroup parameters...
Setting cpu.shares for apphosting.partition to 921...OK
Setting cpu.shares for host to 100...OK
Setting cpu.shares for host/caf to 100...OK
Setting cpuset values for apphosting.partition...OK
Setting up memory cgroup parameters...
Setting memory.limit_in_bytes for apphosting.partition to 6589061529...OK
Setting memory.limit_in_bytes for host to 1647265382...OK
Setting memory.limit_in_bytes for host/caf to 1317812305...OK
OpenBSD Secure Shell server not in use (/etc/ssh/sshd_not_to_be_run)
Starting atd: OK
starting DNS forwarder and DHCP server: dnsmasq... done.
Starting ntpd: done
Starting system log daemon...0
Starting kernel log daemon...0
Network mgmt starting with factory default configuration
User mgmt starting with factory default configuration
Starting konfd: OK
 * Starting virtualization library daemon: libvirtd
no /usr/bin/dnsmasq found; none killed                                   [ ok ]
 * Starting libvirt log management daemon: virtlogd                      [ ok ]
Starting crond: OK
Starting Monit 5.14 daemon with http interface at /var/run/monit.sock


ic3k>