Perform NCS 6000 Disaster Recovery without using USB

Available Languages

Download Options

PDF (77.9 KB)
View with Adobe Reader on a variety of devices
ePub (76.3 KB)
View in various apps on iPhone, iPad, Android, Sony Reader, or Windows Phone
Mobi (Kindle) (73.6 KB)
View on Kindle device or Kindle app on multiple devices

Updated:December 4, 2015

Document ID:200282

Step by Step Procedure

Introduction

This document describes a way to recover Network Convergence System 6000 (NCS6K) after failed upgrade without using a USB drive. Recovery with USB drive requires physical access to the device which most of the time can be a challenge and time consuming.

The procedure described in this document would use a Linux Machine acting as a TFTP and DHCP server to recover the NCS6K via the RP management ethernet port.

Prerequisites

Requirements

Cisco recommends that you have basic knowledge of Linux, TFTP, DHCP and Cisco XR CLI.

Components Used

This document is restricted to NCS6K Platform.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Step by Step Procedure

Download the required XR USB boot file from Cisco Website.

Upload the downloaded ZIP file to Linux server and unzip it in /tftpboot :

root@xxxr:/tftpboot# unzip ncs6k-usb-boot-5.2.3.zip
Archive:  ncs6k-usb-boot-5.2.3.zip
  inflating: EFI/boot/bootx64.efi
  inflating: EFI/boot/grub.cfg
  inflating: boot/install-image.iso

Find the mac address of the NCS6K Route Processor (RP) management ethernet port. It can be found in the console logs :

Press F12 to go to Boot Manager..

Booting System Host OS..
Verifying Image for Secure Boot failed with status 15

System Host OS boot failed.

Booting Int Network 0 for IPv4 (4C-4E-35-B6-63-33)..
>>Start PXE over IPv4.
  PXE-E18: Server response timeout.

Int Network 0 for IPv4 (4C-4E-35-B6-63-33) boot failed.

Booting Ext Network 0 for IPv4 (4C-4E-35-B6-63-33)..

Add the following to dhcpd.conf file. This will allocate a static IP address to RP mgmt ethernet port when booting ( eg : 10.48.32.160) :

root@xxxr:/tftpboot/0A3020A0# cat /etc/dhcp/dhcpd.conf

allow bootp;
allow booting;

subnet 10.48.32.0 netmask 255.255.255.0 {

       option routers 10.48.32.1;
       next-server 10.48.32.93;
       host pani0-rp {
               hardware ethernet  4c:4e:35:b6:63:33;
               fixed-address 10.48.32.160;
               filename "EFI/boot/bootx64.efi" ;
    }
}

Note: 10.48.32.93 is the TFTP and DHCP server address.

Make a copy of grub.cfg (file generated in step 2). Name the file after the IP address NCS6K will get from DHCP:
```
root@xxxr:/tftpboot# cp /tftpboot/EFI/boot/grub.cfg  /tftpboot/10.48.32.160.cfg
```

Edit the file generated above to ensure ISO is picked from the network (remove leading /):

root@xxxr:/tftpboot# diff /tftpboot/EFI/boot/grub.cfg  /tftpboot/10.48.32.160.cfg
11,12c11,12
<         echo "Booting from USB.."
<         loopback loop /boot/install-image.iso
---
>         echo "Booting from network.."
>         loopback loop boot/install-image.iso

For 5.2.3, it would look something like this :

root@xxxr:/tftpboot# cat /tftpboot/10.48.32.160.cfg
set default=0

serial --unit=0 --speed=115200

terminal_input console
terminal_output serial

set timeout=2

menuentry "System Install OS" {
        echo "Booting from network..."
        loopback loop boot/install-image.iso
        root=loop
        echo "Loading Kernel.."
        linux (loop)/boot/bzImage root=/dev/ram install=/dev/sda console=ttyS0,115200 prod=1 crashkernel=192M@0 bigphysarea=10M quiet pci=assign-busses noissu aer=off pci=hpmemsize=0M,hpiosize=0M
        echo "Loading initrd.."
        initrd (loop)/boot/initrd.img signfile=/boot/signature.initrd.img
}

Configuration on Linux server is complete. At next PXE boot attempt, DHCP will allocate 10.48.32.160 to the NCS6K RP. It will then get the grub .efi and .cfg using TFTP. After this, GRUB would start automatically and load the ISO using TFTP.

Note: ISO file is normally around 700Mb. It will take some time (up to 10 mins) after "Booting from network.." message is displayed.

Complete logs of the activity:

Cisco BIOS version : SB.Panini.0014.00
BIOS Build Date : 07/10/2014 by lchinnad
System Memory Speed : 1600 MHz
Processor Type : Intel(R) Xeon(R) CPU E5-2448L @ 1.80GHz

Press F12 to goto Boot Manager..

Booting System Host OS..
Verifying Image for Secure Boot failed with status 15

System Host OS boot failed.

Booting Int Network 0 for IPv4 (4C-4E-35-B6-63-33)..
 Start PXE over IPv4.
  PXE-E18: Server response timeout.

Int Network 0 for IPv4 (4C-4E-35-B6-63-33) boot failed.

Booting Ext Network 0 for IPv4 (4C-4E-35-B6-63-33)..
  Start PXE over IPv4.
  Station IP address is 10.48.32.160
  Server IP address is 10.48.32.93
  NBP filename is bootx64.efi
 Downloading NBP file...

  Succeed to download NBP file.


GNU GRUB version 2.00
Press F2 to goto grub Menu..
Booting from network..

[    6.338259] i8042: No controller found
Starting udev: [  OK  ]
Actual changes:
large-receive-offload: off [requested on]
ntuple-filters: on
Setting hostname host:  [  OK  ]
Checking filesystems:[  OK  ]
Remounting root filesystem in read-write mode:  [  OK  ]
Entering non-interactive startup
Bringing up loopback interface:  [  OK  ]
Starting system logger: [  OK  ]
Starting kernel logger: [  OK  ]
Starting kdump:[  OK  ]
Starting system message bus: [  OK  ]
Starting smartd: [  OK  ]
Generating SSH1 RSA host key: [  OK  ]
Generating SSH2 RSA host key: [  OK  ]
Generating SSH2 DSA host key: [  OK  ]
Starting sshd: [  OK  ]
Starting xinetd: [  OK  ]
Checking PCI block device /dev/sdb disk space
Thu Jun 25 14:07:13 UTC 2015: Detected /iso/host.iso
mount: block device /iso/host.iso is write-protected, mounting read-only
Thu Jun 25 14:07:13 UTC 2015: Mounted /iso/host.iso to /tmp/isomnt.iV1833
Thu Jun 25 14:07:13 UTC 2015: Found /tmp/isomnt.iV1833/rpm/ncs6k-sysadmin-hostos.all-5.2.3-Default.x86_64.rpm in host.iso
Thu Jun 25 14:07:13 UTC 2015: Installing /tmp/isomnt.iV1833/rpm/ncs6k-sysadmin-hostos.all-5.2.3-Default.x86_64.rpm
Preparing packages for installation...
ncs6k-sysadmin-hostos.all-5.2.3-Default.x86_64
hushd_static: no process killed
hushd restarted
Thu Jun 25 14:07:13 UTC 2015: Did not detect new pxe install script, keep going with old xrnginstall
Thu Jun 25 14:07:13 UTC 2015: Running in  Data LV support model
/etc/rc3.d/S60xrnginstall: line 239: SIMULATION: readonly variable
Thu Jun 25 14:07:13 UTC 2015: Prepping System with calvados.iso
Thu Jun 25 14:07:13 UTC 2015: Installer will install image on sda
Thu Jun 25 14:07:13 UTC 2015: Running in LVM support model
Thu Jun 25 14:07:15 UTC 2015: Partition creation on /dev/sda took 1 seconds
Thu Jun 25 14:07:15 UTC 2015: File system creation on /dev/sda1 took 0 seconds
Thu Jun 25 14:07:15 UTC 2015: Install host image on /dev/sda1
Thu Jun 25 14:07:23 UTC 2015: Installing host image size of 183M took 8 seconds
Thu Jun 25 14:07:33 UTC 2015: File system creation on /dev/sda2 took 4 seconds
Thu Jun 25 14:08:38 UTC 2015: Copying XR iso to repository took 65 seconds
Partitioning PCI block device /dev/sdb
Added VLAN with VID == 513 to IF -:eth-pf1:-
Thu Jun 25 14:08:40 UTC 2015: Copying boot/install-image.iso from tftpserver 10.48.32.93
Thu Jun 25 14:16:58 UTC 2015: Copying Pxeboot files from tftpserver 10.48.32.93 took 498 seconds
Thu Jun 25 14:17:28 UTC 2015: File system creation on /dev/panini_vol_grp/calvados_lv0 took 5 seconds
Thu Jun 25 14:17:28 UTC 2015: Install sysadmin-vm image on /dev/panini_vol_grp/calvados_lv0
mount: block device /iso/ncs6k-sysadmin.iso is write-protected, mounting read-only
Thu Jun 25 14:17:35 UTC 2015: sysadmin-vm: RP based installation
Thu Jun 25 14:18:22 UTC 2015: Installing sysadmin-vm image size of 444M took 54 seconds
Install EFI on /dev/sda4
Thu Jun 25 14:18:24 UTC 2015: Install finished on sda
Resetting BIOS Boot Mode register ...
Automatic rebooting system after installation ...

Cisco BIOS version : SB.Panini.0014.00
BIOS Build Date : 07/10/2014 by lchinnad
System Memory Speed : 1600 MHz
Processor Type : Intel(R) Xeon(R) CPU E5-2448L @ 1.80GHz

Press F12 to goto Boot Manager..

Booting System Host OS..

GNU GRUB version 2.00
Press F2 to goto grub Menu..
Booting from Disk..
Loading Kernel..
Loading initrd..
Starting udev: [  OK  ]
Setting hostname sysadmin-vm:  [  OK  ]
Checking filesystems:[  OK  ]
Mount /dev/vdd at /misc/disk1
Entering non-interactive startup
Bringing up loopback interface:  [  OK  ]
Starting system logger: [  OK  ]
Starting kernel logger: [  OK  ]
Starting system message bus: [  OK  ]
Starting smartd: [FAILED]
Generating SSH1 RSA host key: [  OK  ]
Generating SSH2 RSA host key: [  OK  ]
Generating SSH2 DSA host key: [  OK  ]
Starting sshd: [  OK  ]
Starting xinetd: [  OK  ]
Starting crond: [  OK  ]
Starting libvirtd daemon: [  OK  ]
Starting NCS6k programs for RP on sysadmin-vm: [  OK  ]
starting pm
sysadmin_startup: Starting Cisco Login Program on ttyS0
sysadmin initialized
sysadmin_startup: Starting Cisco Login Program on ttys1
sysadmin initialized

0_0_0Jun 25 14:19:32 : Send To Helper Failed - Msg : aaad[2600]: %MGBL-AAAD-7-DEBUG : AAA Init successful
0_0_0Jun 25 14:19:33 : Send To Helper Failed - Msg : vm_manager[2628]: %INFRA-VM_MANAGER-4-INFO : Info: VM Manager started. arguments  -W
0_0_0Jun 25 14:19:34 : Send To Helper Failed - Msg : sdr_mgr[2619]: %SM-SDR_MANAGER-4-INFO : Info: SDR Manager started.

SYSTEM IS NOT READY FOR LOGIN

!!!NO root-system username is configured. Need to configure root-system username!!!

--- Administrative User Dialog ---


Enter root-system Username: 0_0_0Jun 25 14:20:58 : Send To Helper Failed - Msg : plx_fpd[2616]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : Driver missing fpd obfl log function for fpd PLX-8748, FPD init continues but debugability impacted
0/RP0/ADMIN0:Jun 25 14:20:58.410 : envmon[2609]: %PKT_INFRA-FM-4-FAULT_MINOR : ALARM_MINOR :Unsupported power module detected :DECLARE :0/PT0-PM0:
0/RP0/ADMIN0:Jun 25 14:20:58.417 : envmon[2609]: %PKT_INFRA-FM-4-FAULT_MINOR : ALARM_MINOR :Unsupported power module detected :DECLARE :0/PT0-PM1:
0/RP0/ADMIN0:Jun 25 14:20:58.418 : envmon[2609]: %PKT_INFRA-FM-4-FAULT_MINOR : ALARM_MINOR :Unsupported power module detected :DECLARE :0/PT0-PM2:
0/RP0/ADMIN0:Jun 25 14:20:58.434 : envmon[2609]: %PKT_INFRA-FM-4-FAULT_MINOR : ALARM_MINOR :Unsupported power module detected :DECLARE :0/PT3-PM0:
0/RP0/ADMIN0:Jun 25 14:20:58.445 : envmon[2609]: %PKT_INFRA-FM-4-FAULT_MINOR : ALARM_MINOR :Unsupported power module detected :DECLARE :0/PT3-PM1:
0/RP0/ADMIN0:Jun 25 14:20:58.451 : envmon[2609]: %PKT_INFRA-FM-4-FAULT_MINOR : ALARM_MINOR :Unsupported power module detected :DECLARE :0/PT3-PM2:
0/RP0/ADMIN0:Jun 25 14:20:58.517 : zen[2630]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : Driver missing fpd obfl log function for fpd CPU Complex FPGA, FPD init continues but debugability impacted


Enter root-system Username: root

Enter secret:

Enter secret again:


 Successfully created root-system user


System Admin Username: root

Password:

root connected from 127.0.0.1 using console on sysadmin-vm:0_RP0

sysadmin-vm:0_RP0# show platform
Thu Jun  25 14:21:33.150 UTC
Location  Card Type               HW State      SW State      Config State
----------------------------------------------------------------------------
0/1       NC6-60X10GE-M-S         POWERED_ON    SW_INACTIVE   NSHUT
0/7       NC6-10X100G-M-P         POWERED_ON    SW_INACTIVE   NSHUT
0/RP0     NC6-RP                  OPERATIONAL   OPERATIONAL   NSHUT
0/RP1     NC6-RP                  POWERED_ON    SW_INACTIVE   NSHUT
0/FC0     P-L-FC-S                POWERED_ON    N/A           NSHUT
0/FC1     P-L-FC-S                POWERED_ON    N/A           NSHUT
0/FC2     P-L-FC-S                POWERED_ON    N/A           NSHUT
0/FC3     P-L-FC-S                POWERED_ON    N/A           NSHUT
0/FC4     P-L-FC-S                POWERED_ON    N/A           NSHUT
0/FC5     P-L-FC-S                POWERED_ON    N/A           NSHUT
0/CI0     NCS-CRFT=               OPERATIONAL   N/A           NSHUT
0/FT0     NC6-FANTRAY             OPERATIONAL   N/A           NSHUT
0/FT1     NC6-FANTRAY             OPERATIONAL   N/A           NSHUT
0/PT0     NCS-AC-PWRTRAY          OPERATIONAL   N/A           NSHUT
0/PT1     NCS-AC-PWRTRAY          OPERATIONAL   N/A           NSHUT
0/PT2     NCS-AC-PWRTRAY          OPERATIONAL   N/A           NSHUT
0/PT3     NCS-AC-PWRTRAY          OPERATIONAL   N/A           NSHUT
0/PT4     NCS-AC-PWRTRAY          OPERATIONAL   N/A           NSHUT
0/PT5     NCS-AC-PWRTRAY          OPERATIONAL   N/A           NSHUT