Introduction
This document describes a way to recover Network Convergence System 6000 (NCS6K) after failed upgrade without using a USB drive. Recovery with USB drive requires physical access to the device which most of the time can be a challenge and time consuming.
The procedure described in this document would use a Linux Machine acting as a TFTP and DHCP server to recover the NCS6K via the RP management ethernet port.
Prerequisites
Requirements
Cisco recommends that you have basic knowledge of Linux, TFTP, DHCP and Cisco XR CLI.
Components Used
This document is restricted to NCS6K Platform.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Step by Step Procedure
- Download the required XR USB boot file from Cisco Website.
- Upload the downloaded ZIP file to Linux server and unzip it in /tftpboot :
root@xxxr:/tftpboot# unzip ncs6k-usb-boot-5.2.3.zip
Archive: ncs6k-usb-boot-5.2.3.zip
inflating: EFI/boot/bootx64.efi
inflating: EFI/boot/grub.cfg
inflating: boot/install-image.iso
-
Find the mac address of the NCS6K Route Processor (RP) management ethernet port. It can be found in the console logs :
Press F12 to go to Boot Manager..
Booting System Host OS..
Verifying Image for Secure Boot failed with status 15
System Host OS boot failed.
Booting Int Network 0 for IPv4 (4C-4E-35-B6-63-33)..
>>Start PXE over IPv4.
PXE-E18: Server response timeout.
Int Network 0 for IPv4 (4C-4E-35-B6-63-33) boot failed.
Booting Ext Network 0 for IPv4 (4C-4E-35-B6-63-33)..
-
Add the following to dhcpd.conf file. This will allocate a static IP address to RP mgmt ethernet port when booting ( eg : 10.48.32.160) :
root@xxxr:/tftpboot/0A3020A0# cat /etc/dhcp/dhcpd.conf
allow bootp;
allow booting;
subnet 10.48.32.0 netmask 255.255.255.0 {
option routers 10.48.32.1;
next-server 10.48.32.93;
host pani0-rp {
hardware ethernet 4c:4e:35:b6:63:33;
fixed-address 10.48.32.160;
filename "EFI/boot/bootx64.efi" ;
}
}
Note: 10.48.32.93 is the TFTP and DHCP server address.
-
Make a copy of grub.cfg (file generated in step 2). Name the file after the IP address NCS6K will get from DHCP:
root@xxxr:/tftpboot# cp /tftpboot/EFI/boot/grub.cfg /tftpboot/10.48.32.160.cfg
-
Edit the file generated above to ensure ISO is picked from the network (remove leading /):
root@xxxr:/tftpboot# diff /tftpboot/EFI/boot/grub.cfg /tftpboot/10.48.32.160.cfg
11,12c11,12
< echo "Booting from USB.."
< loopback loop /boot/install-image.iso
---
> echo "Booting from network.."
> loopback loop boot/install-image.iso
For 5.2.3, it would look something like this :
root@xxxr:/tftpboot# cat /tftpboot/10.48.32.160.cfg
set default=0
serial --unit=0 --speed=115200
terminal_input console
terminal_output serial
set timeout=2
menuentry "System Install OS" {
echo "Booting from network..."
loopback loop boot/install-image.iso
root=loop
echo "Loading Kernel.."
linux (loop)/boot/bzImage root=/dev/ram install=/dev/sda console=ttyS0,115200 prod=1 crashkernel=192M@0 bigphysarea=10M quiet pci=assign-busses noissu aer=off pci=hpmemsize=0M,hpiosize=0M
echo "Loading initrd.."
initrd (loop)/boot/initrd.img signfile=/boot/signature.initrd.img
}
-
Configuration on Linux server is complete. At next PXE boot attempt, DHCP will allocate 10.48.32.160 to the NCS6K RP. It will then get the grub .efi and .cfg using TFTP. After this, GRUB would start automatically and load the ISO using TFTP.
Note: ISO file is normally around 700Mb. It will take some time (up to 10 mins) after "Booting from network.." message is displayed.
Complete logs of the activity:
Cisco BIOS version : SB.Panini.0014.00
BIOS Build Date : 07/10/2014 by lchinnad
System Memory Speed : 1600 MHz
Processor Type : Intel(R) Xeon(R) CPU E5-2448L @ 1.80GHz
Press F12 to goto Boot Manager..
Booting System Host OS..
Verifying Image for Secure Boot failed with status 15
System Host OS boot failed.
Booting Int Network 0 for IPv4 (4C-4E-35-B6-63-33)..
Start PXE over IPv4.
PXE-E18: Server response timeout.
Int Network 0 for IPv4 (4C-4E-35-B6-63-33) boot failed.
Booting Ext Network 0 for IPv4 (4C-4E-35-B6-63-33)..
Start PXE over IPv4.
Station IP address is 10.48.32.160
Server IP address is 10.48.32.93
NBP filename is bootx64.efi
Downloading NBP file...
Succeed to download NBP file.
GNU GRUB version 2.00
Press F2 to goto grub Menu..
Booting from network..
[ 6.338259] i8042: No controller found
Starting udev: [ OK ]
Actual changes:
large-receive-offload: off [requested on]
ntuple-filters: on
Setting hostname host: [ OK ]
Checking filesystems:[ OK ]
Remounting root filesystem in read-write mode: [ OK ]
Entering non-interactive startup
Bringing up loopback interface: [ OK ]
Starting system logger: [ OK ]
Starting kernel logger: [ OK ]
Starting kdump:[ OK ]
Starting system message bus: [ OK ]
Starting smartd: [ OK ]
Generating SSH1 RSA host key: [ OK ]
Generating SSH2 RSA host key: [ OK ]
Generating SSH2 DSA host key: [ OK ]
Starting sshd: [ OK ]
Starting xinetd: [ OK ]
Checking PCI block device /dev/sdb disk space
Thu Jun 25 14:07:13 UTC 2015: Detected /iso/host.iso
mount: block device /iso/host.iso is write-protected, mounting read-only
Thu Jun 25 14:07:13 UTC 2015: Mounted /iso/host.iso to /tmp/isomnt.iV1833
Thu Jun 25 14:07:13 UTC 2015: Found /tmp/isomnt.iV1833/rpm/ncs6k-sysadmin-hostos.all-5.2.3-Default.x86_64.rpm in host.iso
Thu Jun 25 14:07:13 UTC 2015: Installing /tmp/isomnt.iV1833/rpm/ncs6k-sysadmin-hostos.all-5.2.3-Default.x86_64.rpm
Preparing packages for installation...
ncs6k-sysadmin-hostos.all-5.2.3-Default.x86_64
hushd_static: no process killed
hushd restarted
Thu Jun 25 14:07:13 UTC 2015: Did not detect new pxe install script, keep going with old xrnginstall
Thu Jun 25 14:07:13 UTC 2015: Running in Data LV support model
/etc/rc3.d/S60xrnginstall: line 239: SIMULATION: readonly variable
Thu Jun 25 14:07:13 UTC 2015: Prepping System with calvados.iso
Thu Jun 25 14:07:13 UTC 2015: Installer will install image on sda
Thu Jun 25 14:07:13 UTC 2015: Running in LVM support model
Thu Jun 25 14:07:15 UTC 2015: Partition creation on /dev/sda took 1 seconds
Thu Jun 25 14:07:15 UTC 2015: File system creation on /dev/sda1 took 0 seconds
Thu Jun 25 14:07:15 UTC 2015: Install host image on /dev/sda1
Thu Jun 25 14:07:23 UTC 2015: Installing host image size of 183M took 8 seconds
Thu Jun 25 14:07:33 UTC 2015: File system creation on /dev/sda2 took 4 seconds
Thu Jun 25 14:08:38 UTC 2015: Copying XR iso to repository took 65 seconds
Partitioning PCI block device /dev/sdb
Added VLAN with VID == 513 to IF -:eth-pf1:-
Thu Jun 25 14:08:40 UTC 2015: Copying boot/install-image.iso from tftpserver 10.48.32.93
Thu Jun 25 14:16:58 UTC 2015: Copying Pxeboot files from tftpserver 10.48.32.93 took 498 seconds
Thu Jun 25 14:17:28 UTC 2015: File system creation on /dev/panini_vol_grp/calvados_lv0 took 5 seconds
Thu Jun 25 14:17:28 UTC 2015: Install sysadmin-vm image on /dev/panini_vol_grp/calvados_lv0
mount: block device /iso/ncs6k-sysadmin.iso is write-protected, mounting read-only
Thu Jun 25 14:17:35 UTC 2015: sysadmin-vm: RP based installation
Thu Jun 25 14:18:22 UTC 2015: Installing sysadmin-vm image size of 444M took 54 seconds
Install EFI on /dev/sda4
Thu Jun 25 14:18:24 UTC 2015: Install finished on sda
Resetting BIOS Boot Mode register ...
Automatic rebooting system after installation ...
Cisco BIOS version : SB.Panini.0014.00
BIOS Build Date : 07/10/2014 by lchinnad
System Memory Speed : 1600 MHz
Processor Type : Intel(R) Xeon(R) CPU E5-2448L @ 1.80GHz
Press F12 to goto Boot Manager..
Booting System Host OS..
GNU GRUB version 2.00
Press F2 to goto grub Menu..
Booting from Disk..
Loading Kernel..
Loading initrd..
Starting udev: [ OK ]
Setting hostname sysadmin-vm: [ OK ]
Checking filesystems:[ OK ]
Mount /dev/vdd at /misc/disk1
Entering non-interactive startup
Bringing up loopback interface: [ OK ]
Starting system logger: [ OK ]
Starting kernel logger: [ OK ]
Starting system message bus: [ OK ]
Starting smartd: [FAILED]
Generating SSH1 RSA host key: [ OK ]
Generating SSH2 RSA host key: [ OK ]
Generating SSH2 DSA host key: [ OK ]
Starting sshd: [ OK ]
Starting xinetd: [ OK ]
Starting crond: [ OK ]
Starting libvirtd daemon: [ OK ]
Starting NCS6k programs for RP on sysadmin-vm: [ OK ]
starting pm
sysadmin_startup: Starting Cisco Login Program on ttyS0
sysadmin initialized
sysadmin_startup: Starting Cisco Login Program on ttys1
sysadmin initialized
0_0_0Jun 25 14:19:32 : Send To Helper Failed - Msg : aaad[2600]: %MGBL-AAAD-7-DEBUG : AAA Init successful
0_0_0Jun 25 14:19:33 : Send To Helper Failed - Msg : vm_manager[2628]: %INFRA-VM_MANAGER-4-INFO : Info: VM Manager started. arguments -W
0_0_0Jun 25 14:19:34 : Send To Helper Failed - Msg : sdr_mgr[2619]: %SM-SDR_MANAGER-4-INFO : Info: SDR Manager started.
SYSTEM IS NOT READY FOR LOGIN
!!!NO root-system username is configured. Need to configure root-system username!!!
--- Administrative User Dialog ---
Enter root-system Username: 0_0_0Jun 25 14:20:58 : Send To Helper Failed - Msg : plx_fpd[2616]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : Driver missing fpd obfl log function for fpd PLX-8748, FPD init continues but debugability impacted
0/RP0/ADMIN0:Jun 25 14:20:58.410 : envmon[2609]: %PKT_INFRA-FM-4-FAULT_MINOR : ALARM_MINOR :Unsupported power module detected :DECLARE :0/PT0-PM0:
0/RP0/ADMIN0:Jun 25 14:20:58.417 : envmon[2609]: %PKT_INFRA-FM-4-FAULT_MINOR : ALARM_MINOR :Unsupported power module detected :DECLARE :0/PT0-PM1:
0/RP0/ADMIN0:Jun 25 14:20:58.418 : envmon[2609]: %PKT_INFRA-FM-4-FAULT_MINOR : ALARM_MINOR :Unsupported power module detected :DECLARE :0/PT0-PM2:
0/RP0/ADMIN0:Jun 25 14:20:58.434 : envmon[2609]: %PKT_INFRA-FM-4-FAULT_MINOR : ALARM_MINOR :Unsupported power module detected :DECLARE :0/PT3-PM0:
0/RP0/ADMIN0:Jun 25 14:20:58.445 : envmon[2609]: %PKT_INFRA-FM-4-FAULT_MINOR : ALARM_MINOR :Unsupported power module detected :DECLARE :0/PT3-PM1:
0/RP0/ADMIN0:Jun 25 14:20:58.451 : envmon[2609]: %PKT_INFRA-FM-4-FAULT_MINOR : ALARM_MINOR :Unsupported power module detected :DECLARE :0/PT3-PM2:
0/RP0/ADMIN0:Jun 25 14:20:58.517 : zen[2630]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : Driver missing fpd obfl log function for fpd CPU Complex FPGA, FPD init continues but debugability impacted
Enter root-system Username: root
Enter secret:
Enter secret again:
Successfully created root-system user
System Admin Username: root
Password:
root connected from 127.0.0.1 using console on sysadmin-vm:0_RP0
sysadmin-vm:0_RP0# show platform
Thu Jun 25 14:21:33.150 UTC
Location Card Type HW State SW State Config State
----------------------------------------------------------------------------
0/1 NC6-60X10GE-M-S POWERED_ON SW_INACTIVE NSHUT
0/7 NC6-10X100G-M-P POWERED_ON SW_INACTIVE NSHUT
0/RP0 NC6-RP OPERATIONAL OPERATIONAL NSHUT
0/RP1 NC6-RP POWERED_ON SW_INACTIVE NSHUT
0/FC0 P-L-FC-S POWERED_ON N/A NSHUT
0/FC1 P-L-FC-S POWERED_ON N/A NSHUT
0/FC2 P-L-FC-S POWERED_ON N/A NSHUT
0/FC3 P-L-FC-S POWERED_ON N/A NSHUT
0/FC4 P-L-FC-S POWERED_ON N/A NSHUT
0/FC5 P-L-FC-S POWERED_ON N/A NSHUT
0/CI0 NCS-CRFT= OPERATIONAL N/A NSHUT
0/FT0 NC6-FANTRAY OPERATIONAL N/A NSHUT
0/FT1 NC6-FANTRAY OPERATIONAL N/A NSHUT
0/PT0 NCS-AC-PWRTRAY OPERATIONAL N/A NSHUT
0/PT1 NCS-AC-PWRTRAY OPERATIONAL N/A NSHUT
0/PT2 NCS-AC-PWRTRAY OPERATIONAL N/A NSHUT
0/PT3 NCS-AC-PWRTRAY OPERATIONAL N/A NSHUT
0/PT4 NCS-AC-PWRTRAY OPERATIONAL N/A NSHUT
0/PT5 NCS-AC-PWRTRAY OPERATIONAL N/A NSHUT
After some time, other RP and Line Cards will also start booting.