Contents

Introduction

Background Information

SMU Summary Table

FPD Upgrade MOP

Use Case. System with Cisco IOS-XR Release 7.1.2. And Necessary SMUS Activated

XR VM

sysadmin

Verify FPD Versions on the RPs

Check TAM Area on Both RPs (Evaluate the Removal of the Section)

RP0

RP1

Sample Output of Corrupted TAM Memory

Perform IOFPGA Upgrade on Both RPs

Verify

Reload the Active and Standby RPs for IOFPGA Upgrade to Complete

Verify ADM Version

Perform Other FPD Upgrade on Both RPs

1. ADM

2. BIOS

3. SATA

Verify the Upgrades

Power Cycle Procedure to Complete Other FPD Upgrade

Verify FPD Versions

Verify ADM Version

Upgrade Other FPDs on Other Locations (If Any)

Introduction

This document describes how to avoid situations in which the Cisco NCS 560 route processor (RP) becomes unresponsive after a reload or a power cycle.

Background Information

In most cases, the NCS 560 deployments are in remote locations, in large numbers, and where human intervention must be kept to a minimum.

In these deployments, it is also important to fulfil the mandatory SMU installation requirement based on the installed release.

Note: From Cisco IOS XR Release 7.2.2 and onwards, you are no longer required to install SMUs separately.

SMU Summary Table

IOS XR Version

SMU

Link

Notes

6.6.3

CSCvt69762 (sysadmin)

CSCvt26907 (sysadmin)

CSCvv88955 (sysadmin)

CSCvw42910 (XR)

CSCvw52338 (XR)

link

7.0.1

CSCvw42910 (XR)

CSCvw52338 (XR)

CSCvt69762 (sysadmin)

link

It is mandatory to move to the extended maintenance release (EMR) of the same minor release. For example, IOS XR 7.0.2.

No solution for TAM area corruption is available in this release.

7.0.2

CSCvt69762 (sysadmin)

CSCvt26907 (sysadmin)

CSCvv48056 (sysadmin)

CSCvw42910 (XR)

CSCvw52338 (XR)

link

7.1.1

CSCvv41516 (sysadmin)

CSCvt69762 (sysadmin)

CSCvt26907 (sysadmin)

CSCvw42910 (XR)

CSCvw52338 (XR)

link

It is advisable to move to the EMR release of the same minor release. For example, IOS XR 7.1.2.

7.1.2

CSCvv41516 (sysadmin)

CSCvt69762 (sysadmin)

CSCvt26907 (sysadmin)

CSCvw42910 (XR)

CSCvw52338 (XR)

link

7.2.1

CSCvv41516 (sysadmin)

CSCvt69762 (sysadmin)

CSCvt26907 (sysadmin)

CSCvw42910 (XR)

CSCvw52338 (XR)

link

FPD Upgrade MOP

FPDs must be upgraded to ensure that FPDs are always at a version equal to or higher than the minimum required version required for that Cisco IOS® XR release.

ADM

1.06

IOFPGA

0.64

BIOS

0.16

This document assumes that:

- RP0 is the Active RP

- RP1 is the Standby RP

the prompt for various locations so that all commands to navigate the various location are omitted.

1. XR - RP0

RP/0/RP0/CPU0:ios#

2. XR - RP1

RP/0/RP1/CPU0:ios#

3. sysadmin - RP0

sysadmin-vm:0_RP0#

4. sysadmin - RP1

sysadmin-vm:0_RP1#

5. sysadmin - RP0 shell

[sysadmin-vm:0_RP0:~]$

6. sysadmin - RP1 shell

[sysadmin-vm:0_RP1:~]$

Use Case. System with Cisco IOS-XR Release 7.1.2. And Necessary SMUS Activated

These SMUs must be active and committed to the system prior to this process.

XR VM

RP/0/RP0/CPU0:ios#show install active summary

Active Packages: 10

ncs560-xr-7.1.2 version=7.1.2 [Boot image]

ncs560-mcast-2.0.0.0-r712

ncs560-mgbl-2.0.0.0-r712

ncs560-mpls-1.0.0.0-r712

ncs560-mpls-te-rsvp-2.0.0.0-r712

ncs560-ospf-2.0.0.0-r712

ncs560-k9sec-2.1.0.0-r712

ncs560-iosxr-fwding-2.1.0.1-r712.CSCvp67225

ncs560-os-support-2.1.0.1-r712.CSCvv41516

cisco-klm-rsp4-0.1.p1-r0.0.r712.CSCvt26907  .xr

sysadmin

sysadmin-vm:0_RP0# show install active summary

Active Packages: 4

ncs560-sysadmin-7.1.2 version=7.1.2 [Boot image]

ncs560-sysadmin-ncs560-7.1.2.1-r712.CSCvv41516

cisco-klm-rsp4-0.1.p1-r0.0.r712.CSCvt26907  .admin

cisco-klm-rsp4-0.1.p1-r0.0.r712.CSCvt26907  .host

Verify FPD Versions on the RPs

RP/0/RP0/CPU0:ios#show hw-module fpd

FPD Versions

=================

Location   Card type             HWver FPD device       ATR Status   Running Programd

-----------------------------------------------------------------------------------

0/5        A900-IMA8Z            0.0   IMFPGA               NEED UPGD 17.02   17.02

0/14       A900-IMA8CS1Z-M       0.0   IMFPGA               CURRENT    1.95    1.95

0/RP0      N560-RSP4-E           0.0   ADM                  NEED UPGD  1.04    1.04

0/RP0      N560-RSP4-E           0.0   IOFPGA               NEED UPGD  0.53    0.53

0/RP0      N560-RSP4-E           0.0   PRIMARY-BIOS         NEED UPGD  0.15    0.15

0/RP0      N560-RSP4-E           0.0   SATA                 NEED UPGD  1.30    1.30

0/RP1      N560-RSP4-E           0.0   ADM                  NEED UPGD  1.04    1.04

0/RP1      N560-RSP4-E           0.0   IOFPGA               NEED UPGD  0.53    0.53

0/RP1      N560-RSP4-E           0.0   PRIMARY-BIOS         NEED UPGD  0.15    0.15

0/RP1      N560-RSP4-E           0.0   SATA                 NEED UPGD  1.30    1.30

0/FT0      A907-FAN-E            1.0   PSOC                 CURRENT    1.65    1.65

Check TAM Area on Both RPs (Evaluate the Removal of the Section)

It is important to ensure that the TAM area is not corrupted; otherwise, an IOFPGA, FPD upgrade, and subsequent reloads might lead to an RMA. With this MOP, since FPD version 0.64 or higher is in use, this IOFPGA FPD upgrade process and RP reload leads to TAM corruption recovery.

RP0

sysadmin-vm:0_RP0# show control-driver info location 0/RP0

...

Card view from local RP:

TAM Memory is not corrupted

Can proceed with IOFPGA upgrade if required

...

RP1

sysadmin-vm:0_RP0# show control-driver info location 0/RP1

...

Card view from local RP:

TAM Memory is not corrupted

Can proceed with IOFPGA upgrade if required

...

Note: If TAM Memory is corrupted, it is not a problem if the target IOFPGA version is 0.64 or higher.

Sample Output of Corrupted TAM Memory

sysadmin-vm:0_RP1# show control-driver info location 0/RP0

<snip>

Card view from local RP:

TAM Memory is corrupted

Do not proceed with IOFPGA upgrade

<snip>

Note: If all prerequisites mentioned in this document are fulfilled, you can proceed with the upgrade. The “Do not proceed with IOFPGA upgrade” refers to the IOFPGA version earlier than 0.64.

Perform IOFPGA Upgrade on Both RPs

RP/0/RP0/CPU0:ios#upgrade hw-module location all fpd IOFPGA

upgrade command issued (use "show hw-module fpd" to check upgrade status)

RP/0/RP0/CPU0:RONALDO-RF2a#0/RP0/ADMIN0:Oct 16 15:19:26.609 CEST: fpdserv[4356]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Upgrade for the following FPDs has been committed:

0/RP0/ADMIN0:Oct 16 15:19:26.610 CEST: fpdserv[4356]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Location           FPD name             Force

0/RP0/ADMIN0:Oct 16 15:19:26.610 CEST: fpdserv[4356]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : ==================================================

0/RP0/ADMIN0:Oct 16 15:19:26.610 CEST: fpdserv[4356]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP1              IOFPGA               FALSE

0/RP0/ADMIN0:Oct 16 15:19:26.610 CEST: fpdserv[4356]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP0              IOFPGA               FALSE

Verify

RP/0/RP0/CPU0:ios#show hw-module fpd

Fri Oct 16 15:22:11.256 CEST

FPD Versions

=================

Location   Card type             HWver FPD device       ATR Status   Running Programd

-----------------------------------------------------------------------------------

0/5        A900-IMA8Z            0.0   IMFPGA               NEED UPGD 17.02   17.02

0/14       A900-IMA8CS1Z-M       0.0   IMFPGA               CURRENT    1.95    1.95

0/RP0      N560-RSP4-E           0.0   ADM                  NEED UPGD  1.04    1.04

0/RP0      N560-RSP4-E           0.0   IOFPGA               RLOAD REQ  0.53    0.64

0/RP0      N560-RSP4-E           0.0   PRIMARY-BIOS         NEED UPGD  0.15    0.15

0/RP0      N560-RSP4-E           0.0   SATA                 NEED UPGD  1.30    1.30

0/RP1      N560-RSP4-E           0.0   ADM                  NEED UPGD  1.04    1.04

0/RP1      N560-RSP4-E           0.0   IOFPGA               RLOAD REQ  0.53    0.64

0/RP1      N560-RSP4-E           0.0   PRIMARY-BIOS         NEED UPGD  0.15    0.15

0/RP1      N560-RSP4-E           0.0   SATA                 NEED UPGD  1.30    1.30

0/FT0      A907-FAN-E            1.0   PSOC                 CURRENT    1.65    1.65

Reload the Active and Standby RPs for IOFPGA Upgrade to Complete

First, reload the standby RP:

sysadmin-vm:0_RP0# hw-module location 0/RP1 reload

Reload hardware module ? [no,yes] yes

After the redundancy is restored, reload the active RP (RP0) (it triggers a failover):

sysadmin-vm:0_RP0# hw-module location 0/RP0 reload

Reload hardware module ? [no,yes] yes

Verify ADM Version

On the new active RP (RP1):

[sysadmin-vm:0_RP1:~]$/opt/cisco/calvados/sbin/show_adm_version.sh

ADM 1266 Version info

Firmware Version : 1.14.2

Bootloader Version : 0.0.9

Config Version : 1.4

On the new standby RP (RP0):

[sysadmin-vm:0_RP0:~]$/opt/cisco/calvados/sbin/show_adm_version.sh

ADM 1266 Version info

Firmware Version : 1.14.2

Bootloader Version : 0.0.9

Config Version : 1.4

Perform Other FPD Upgrade on Both RPs

1. ADM

RP/0/RP1/CPU0:ios#upgrade hw-module location all FPd ADM

upgrade command issued (use "show hw-module fpd" to check upgrade status)

RP/0/RP1/CPU0:RONALDO-RF2a#0/RP1/ADMIN0:Oct 16 15:57:14.332 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Upgrade for the following FPDs has been committed:

0/RP1/ADMIN0:Oct 16 15:57:14.333 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Location           FPD name             Force

0/RP1/ADMIN0:Oct 16 15:57:14.333 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : ==================================================

0/RP1/ADMIN0:Oct 16 15:57:14.333 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP1              ADM                  FALSE

0/RP1/ADMIN0:Oct 16 15:57:14.333 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP0              ADM                  FALSE

2. BIOS

RP/0/RP1/CPU0:ios#upgrade hw-module location all FPD PRIMARY-BIOS

upgrade command issued (use "show hw-module fpd" to check upgrade status)

RP/0/RP1/CPU0:RONALDO-RF2a#0/RP1/ADMIN0:Oct 16 16:00:29.257 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Upgrade for the following FPDs has been committed:

0/RP1/ADMIN0:Oct 16 16:00:29.257 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Location           FPD name             Force

0/RP1/ADMIN0:Oct 16 16:00:29.257 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : ==================================================

0/RP1/ADMIN0:Oct 16 16:00:29.257 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP1              PRIMARY-BIOS         FALSE

0/RP1/ADMIN0:Oct 16 16:00:29.257 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP0              PRIMARY-BIOS         FALSE

3. SATA

RP/0/RP1/CPU0:ios#upgrade hw-module location all FPD SATA

upgrade command issued (use "show hw-module fpd" to check upgrade status)

RP/0/RP1/CPU0:RONALDO-RF2a#0/RP1/ADMIN0:Oct 16 16:02:06.472 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Upgrade for the following FPDs has been committed:

0/RP1/ADMIN0:Oct 16 16:02:06.472 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Location           FPD name             Force

0/RP1/ADMIN0:Oct 16 16:02:06.472 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : ==================================================

0/RP1/ADMIN0:Oct 16 16:02:06.472 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP1              SATA                 FALSE

0/RP1/ADMIN0:Oct 16 16:02:06.472 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP0              SATA                 FALSE

Verify the Upgrades

RP/0/RP1/CPU0:ios#show hw-module fpd

FPD Versions

=================

Location   Card type             HWver FPD device       ATR Status   Running Programd

-----------------------------------------------------------------------------------

0/5        A900-IMA8Z            0.0   IMFPGA               NEED UPGD 17.02   17.02

0/14       A900-IMA8CS1Z-M       0.0   IMFPGA               CURRENT    1.95    1.95

0/RP0      N560-RSP4-E           0.0   ADM                  RLOAD REQ  1.04    1.04

0/RP0      N560-RSP4-E           0.0   IOFPGA               CURRENT    0.64    0.64

0/RP0      N560-RSP4-E           0.0   PRIMARY-BIOS         RLOAD REQ  0.15    0.16

0/RP0      N560-RSP4-E           0.0   SATA                 RLOAD REQ  1.30    2.10

0/RP1      N560-RSP4-E           0.0   ADM                  RLOAD REQ  1.04    1.04

0/RP1      N560-RSP4-E           0.0   IOFPGA               CURRENT    0.64    0.64

0/RP1      N560-RSP4-E           0.0   PRIMARY-BIOS         RLOAD REQ  0.15    0.16

0/RP1      N560-RSP4-E           0.0   SATA                 RLOAD REQ  1.30    2.10

0/FT0      A907-FAN-E            1.0   PSOC                 CURRENT    1.65    1.65

Power Cycle Procedure to Complete Other FPD Upgrade

Note: If the ADM firmware version is 1.14.2, you must synchronize all filesystems on all VMs/host before you use the reload script.

[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.0.1 sync

[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.0.4 sync

[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.4.4 sync

[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.4.1 sync

[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.4.1 chvrf 0 ssh 10.0.2.2 sync

[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.0.1 chvrf 0 ssh 10.0.2.2 sync

Note: There is no output for these commands.

In order to reload the chassis:

[sysadmin-vm:0_RP1:~]$/opt/cisco/calvados/sbin/reboot_system.sh

Initiating power-cycle of 0/RP0

Initiating power-cycle of 0/RP1

Note: If the ADM firmware version is 1.14.3, FPDs on the RP require a reload.

First, reload the standby RP (RP0):

sysadmin-vm:0_RP1# hw-module location 0/RP0 reload

Reload hardware module ? [no,yes] yes

After redundancy is restored, reload the active RP (RP1) (it triggers a failover):

sysadmin-vm:0_RP1# hw-module location 0/RP1 reload

Reload hardware module ? [no,yes] yes

Verify FPD Versions

RP/0/RP0/CPU0:ios#show hw-module fpd

Fri Oct 16 16:24:18.849 CEST

FPD Versions

=================

Location   Card type             HWver FPD device       ATR Status   Running Programd

-----------------------------------------------------------------------------------

0/5        A900-IMA8Z            0.0   IMFPGA               NEED UPGD 17.02   17.02

0/14       A900-IMA8CS1Z-M       0.0   IMFPGA               CURRENT    1.95    1.95

0/RP0      N560-RSP4-E           0.0   ADM                  CURRENT    1.06    1.06

0/RP0      N560-RSP4-E           0.0   IOFPGA               CURRENT    0.64    0.64

0/RP0      N560-RSP4-E           0.0   PRIMARY-BIOS         CURRENT    0.16    0.16

0/RP0      N560-RSP4-E           0.0   SATA                 CURRENT    2.10    2.10

0/RP1      N560-RSP4-E           0.0   ADM                  CURRENT    1.06    1.06

0/RP1      N560-RSP4-E           0.0   IOFPGA               CURRENT    0.64    0.64

0/RP1      N560-RSP4-E           0.0   PRIMARY-BIOS         CURRENT    0.16    0.16

0/RP1      N560-RSP4-E           0.0   SATA                 CURRENT    2.10    2.10

0/FT0      A907-FAN-E            1.0   PSOC                 CURRENT    1.65    1.65

Verify ADM Version

[sysadmin-vm:0_RP0:~]$/opt/cisco/calvados/sbin/show_adm_version.sh

ADM 1266 Version info

Firmware Version : 1.14.3

Bootloader Version : 0.0.9

Config Version : 1.6

[sysadmin-vm:0_RP1:~]$/opt/cisco/calvados/sbin/show_adm_version.sh

ADM 1266 Version info

Firmware Version : 1.14.3

Bootloader Version : 0.0.9

Config Version : 1.6

Upgrade Other FPDs on Other Locations (If Any)

RP/0/RP0/CPU0:ios#upgrade hw-module location 0/5 FPD IMFPGA

Fri Oct 16 16:28:15.006 CEST

upgrade command issued (use "show hw-module fpd" to check upgrade status)