Field Notice: FN - 64169 - Cisco A9K Virtual Service Module A9K-VSM-500 Line Card Might Become Unresponsive - BIOS/Firmware Upgrade Recommended

Available Languages

Updated:October 13, 2017
Document ID:FN64169

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision Publish Date Comments
1.0
09-Aug-16
Initial Release
10.0
13-Oct-17
Migration to new field notice system

Products Affected

Affected Product ID Comments
A9K-VSM-500
A9K-VSM-500=

Defect Information

Defect ID Headline
CSCuy67366 backing out CSCuy38364 and addressing service infra.pie dependency
CSCux74411 Unresolved BGP next hops in CEF upon network churn with scale/stress
CSCuz22086 "Mandatory for CGN service alone" serviceapp sometimes not getting programmed for Forge NP, traffic drops
CSCuz53821 VSM Reloads with Persistent Hog detected syslog msg
CSCuy52809 PLATFORM-TEMPERATURE_ALARM-2-DIE_OVERTEMP_CRITICAL error on CGN triggers

Problem Description

Intermittent Aggregation Services Router 9000 (ASR9K) Virtual Service Module (VSM) line card becomes unresponsive due to CPU hog or "CBC" watchdog reset.

Background

When QNX runs in multi-processor mode, CPU cores communicate to each other with inter-processor interrupt (IPI). When an IPI message gets lost or delayed due to the Kernel-based Virtual Machine (KVM) environment, IPI manifests into different symptoms such as race condition in the Linux KVM layer.

Problem Symptom

The symptoms manifest in one of these ways:

  1. The VSM reloads with persistent CPU hog which leads to a CBC watchdog reset:

    LC/0/0/CPU0:Apr 17 15:04:54.983 : wdsysmon[371]: Process devb-ahci pid 32787 prio 21 using 25 percent is the top user of CPU
    LC/0/0/CPU0:Apr 17 15:04:54.983 : wdsysmon[371]: Process devb-ahci pid 32787 tid 3 prio 21 using 25 percent is a CPU Hog and terminated
    LC/0/0/CPU0:Apr 17 15:05:25.073 : wdsysmon[371]: Persistent hog (lasting more than 30 seconds) detected by wdsysmon on CPU2. Resetting node soon
    LC/0/0/CPU0:Apr 17 15:05:25.073 : wdsysmon[371]: Process: devb-ahci, Pid 32787, Tid 3, Priority 21, Util 25 % is a HOG
    RP/0/RSP0/CPU0:Apr 17 15:07:55.565 : canb-server[153]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/0/CPU0 , WDOG SReset (0x06000000)
     
  2. The VSM reloads with persistent CPU hog which leads to a VSM power cycle:

    LC/0/0/CPU0:Apr 18 11:31:45.856 : wdsysmon[371]: Persistent Hog detected for more than 20 seconds
    LC/0/0/CPU0:Apr 18 11:31:47.662 : wdsysmon[371]: Persistent hog (lasting more than 30 seconds) detected by wdsysmon on CPU2. Resetting node soon
    RP/0/RSP0/CPU0:Apr 18 11:31:55.154 : canb-server[153]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/0/CPU0 , Power Cycle (0x05000000)
     
  3. HD0_NOT_RESPONDING might be seen in conjunction with symptoms [1] and [2]:

    LC/0/0/CPU0:Apr 25 07:26:24.343 : pfm_node_lc[289]: %PLATFORM-AHCI-2-DEV_HD0_NOT_RESPONDING : Set|ahci-mon[32789]|0x1000000|HD0 device not responding
     
  4. The HD0 input/output error might be seen in conjunction with symptoms [1] and [2]:

    LC/0/0/CPU0:Mar 14 06:25:55.532 : ahci-mon[65557]: AHCI: cannot read from /dev/hd0: Input/output error
    LC/0/0/CPU0:Mar 14 06:25:56.127 : ahci-mon[65557]: AHCI: failure /dev/hd0: npass=53030, sector=0x7a9e800, fail count=1/1

Workaround/Solution

The workaround requires two steps, a mandatory SMU installation and a manual firmware upgrade. Each DDTS represents a specific SMU that must be installed.

5.3.3 DDTS Headline
CSCuz22086 "Mandatory for CGN service alone" Serviceapp sometimes not getting programmed for Forge NP, traffic drops
CSCuy52809  PLATFORM-TEMPERATURE_ALARM-2-DIE_OVERTEMP_CRITICAL error on CGN triggers
CSCuy67366 backing out CSCuy38364 and addressing service infra.pie dependency
CSCuz53821 VSM Reloads with Persistent Hog detected syslog msg
5.2.4 DDTS Headline 
CSCuz22086 "Mandatory for CGN service alone" Serviceapp sometimes not getting programmed for Forge NP, traffic drops
CSCuy67366  backing out CSCuy38364 and addressing service infra.pie dependency
CSCuz53821 VSM Reloads with Persistent Hog detected syslog msg

Important Note: Read the instructions completely before you begin. You MUST install the Software Maintenance Upgrades (SMUs) BEFORE the firmware upgrade.

Recommendations

  • Perform the upgrade during a maintenance window.
  • Stop traffic during the upgrade process.
  • Install and activate SMUs identified in the SMU tables, along with their prerequisites.
  • After the SMUs are installed and activated, remove the superseded SMU. This helps to release disk and memory space.
  • Proceed with the firmware upgrade after the SMUs are installed.

Manually Upgrade the Firmware on the Unigen eUSB on the A9K-VSM-500 Card

In order to upgrade the firmware, complete these steps:

  1. Confirm the eUSB manufacturer.
  2. Share the upgrade script and firmware binary with your customer.
  3. Upload the upgrade script and firware binary to the Route Processor (RP).
  4. Copy the two files from the Route Switch Processor (RSP) harddisk: to /tmp.
  5. Copy the two files from the RSP /tmp to the VSM Host OS.
  6. Verify the storage devices are properly mounted.
  7. Execute the firmware upgrade.
  8. Reload the A9K-VSM-500 line card.
  9. Verify the firmware is upgraded post VSM reload.

1. Confirm the eUSB manufacturer.

The manufacturer of "usb 2-1.5" should be Unigen.

RP/0/RSP1/CPU0:a9k#run attachCon 0/0/cpu0(do this for VSM slot)
host login: root
Password: lab

[host:~]$ dmesg | grep Manufacturer
usb usb1: Manufacturer: Linux 2.6.34.10-131 ehci_hcd
usb usb2: Manufacturer: Linux 2.6.34.10-131 ehci_hcd
usb 2-1.4: Manufacturer: Avocent
usb 2-1.5: Manufacturer: Unigen Corporation

2. Share the upgrade script and firmware binary with your customer.

The upgrade script usb_fw_upgrd and the firmware binary SM325ACISP.BIN files need to be provided to the customer by the Cisco Support Engineer.

3. Upload the upgrade script and firmware binary to the RP.

Use FTP or TFTP in order to upload the two files to the RP harddisk.

copy tftp://10.1.1.10/SM325ACISP.BIN harddisk:/
copy tftp://10.1.1.10/usb_fw_upgrd harddisk:/harddisk:/

Check the MD5 of both files.

RP/0/RSP0/CPU0:CORE#sh md5 file /harddisk:/SM325ACISP.BIN
Wed Jun 15 06:16:34.919 IST
9b60671ae00d87f36e6acc7c29289734

RP/0/RSP0/CPU0:CORE#sh md5 file /harddisk:/usb_fw_upgrd
Wed Jun 15 06:20:48.818 IST
10d45db8d53cd3a2452a2ad3502771a2

4. Copy the two files from the RSP harddisk: to /tmp.

RP/0/RSP1/CPU0:a9k#run cp /harddisk:/usb_fw_upgrd /tmp

RP/0/RSP1/CPU0:a9k#run cp /harddisk:/SM325ACISP.BIN/tmp

Modify the file permissions in order to allow TFTP transfer to the VSM host.

RP/0/RSP1/CPU0:a9k#run /sbin/chmod +r /tmp/usb_fw_upgrd

RP/0/RSP1/CPU0:a9k#run /sbin/chmod +r /tmp/SM325ACISP.BIN


5. Copy the two files from the RSP /tmp to the VSM Host OS.

The IP address for the TFTP in the VSM Host OS depends on the RSP slot. Enter the ifconfig host-eth0 command on Linux in order to check the subnet number and then use the same subnet address with the last octet "1" in order to access the RP.

RP/0/RSP1/CPU0:a9k#run attachCon 0/0/cpu0
host login: root
Password: lab

[host:~]$ ifconfig host-eth0
host-eth0 Link encap:Ethernet HWaddr 56:4B:47:00:08:43
inet addr:192.0.132.3 Bcast:192.255.255.255 Mask:255.0.0.0
<. . .>

[host:~]$ cd /tmp
[host:~]$ tftp 192.0.132.1 -c get /tmp/usb_fw_upgrd
usb_fw_upgrd [host:~]$ tftp 192.0.132.1 -c get /tmp/SM325ACISP.BIN SM325ACISP.BIN
[host:~]$ chmod +x usb_fw_upgrd
 

6. Verify the storage devices are properly mounted.

Enter the blkid command in order to verify that the SATA disks are mounted as sda and sdb. Only when these two are properly mounted will the eUSB will be mounted at /dev/sg2.

[host:~]$ blkid

/dev/sda1: LABEL="Panini_USB" UUID="08EA-5A5C" TYPE="vfat"
/dev/sda2: UUID="I0dy8o-hv7i-TLJi-w32Y-GNb0-Whfo-9w9ENY" TYPE="LVM2_member"
/dev/sdb1: UUID="b34d9a1c-3ed7-4a27-85e3-a42b7bbc058a" TYPE="ext2"
/dev/mapper/panini_vol_grp-service_vm_iso_repo: UUID="2e0e8ea2-0d2a-4664-882d-cdde10538a56" TYPE="ext3"
/dev/sdc1: SEC_TYPE="msdos" LABEL="CeUSB" UUID="5300-E7B1" TYPE="vfat"

[host:~]$ ls -la /dev/sg?
crw------- 1 root root 21, 0 Jun 4 05:16 /dev/sg0 >>> indicates sda
crw------- 1 root root 21, 1 Jun 4 05:16 /dev/sg1 >>> indicates sdb
crw------- 1 root root 21, 2 Jun 4 05:16 /dev/sg2 >>> eUSB
 

7. Execute the firmware upgrade.

host:~]$ ./usb_fw_upgrd /dev/sg2 /tmp/SM325ACISP.BIN 0 Unigen 1 1 140109 /tmp/usb.bin

The script syntax explanation is:
./usb_fw_upgrd < sg_device > < fw_file > < reset/no-reset > < num-repeat > < num-retry > < fw_ver > [< dump_file >]

If the eUSB firmware is already version 140109, the script reports this and exits without making any changes.

[host:/tmp]$ cat /var/log/eusb_fw_upgrade_log
<...>
Wed Jun 8 17:49:08 2016: CURRENT FIRMWARE VERSION : 140109
Wed Jun 8 17:49:08 2016: CONTROLLER VERSION : SM325AB
Wed Jun 8 17:49:08 2016: UPGRADE IMAGE FIRMWARE VERSION : 140109
Wed Jun 8 17:49:08 2016: CURRENT FIRMWARE VERSION=SM325A MATCHES UPGRADE IMAGE VERSION=140109
Wed Jun 8 17:49:08 2016: UPGRADE NOT REQUIRED RETRY=0
Wed Jun 8 17:49:08 2016: eUSB FLASH FIRMWARE UPGRDAE LOG END
Wed Jun 8 17:49:08 2016:

The upgrade itself takes around two seconds to complete. You can check the upgrade logs in /var/log/eusb_fw_upgrade_log and the upgraded version is 140109. The programmed firmware image is read back and one copy is kept in /tmp/usb.bin.

host:/tmp]$ cat /var/log/eusb_fw_upgrade_log

....

Wed Jun 15 01:00:42 2016: eUSB FLASH FIRMWARE UPGRDAE LOG START
Wed Jun 15 01:00:42 2016: SG DEVICE : /dev/sg2
Wed Jun 15 01:00:42 2016: FIRMWARE IMAGE FILE : /tmp/SM325ACISP.BIN
....
Wed Jun 15 01:00:42 2016: FIRMWARE IMAGE VERSION : 140109
....
Wed Jun 15 01:00:42 2016: CURRENT FIRMWARE VERSION : 101214
Wed Jun 15 01:00:42 2016: CONTROLLER VERSION : SM325AB
Wed Jun 15 01:00:42 2016: UPGRADE IMAGE FIRMWARE VERSION : 140109 >>>>>>>
....
Wed Jun 15 01:00:43 2016: eUSB FLASH FIRMWARE UPGRDAE LOG END

8. Reload the A9K-VSM-500 line card.

Detach from the VSM Host OS console and reload the VSM in order to complete the firmware upgrade. The upgrade log in /var/log/eusb_fw_upgrade_log is purged during the VSM reload.

RP/0/RSP1/CPU0:a9k#hw-module location //CPU0 reload

9. Verify the firmware is upgraded post VSM reload.

If the router reloads, recopy the file from harddisk to /tmp.

RP/0/RSP1/CPU0:a9k#run cp /harddisk:/usb_fw_upgrd /tmp
RP/0/RSP1/CPU0:a9k#run cp /harddisk:/SM325ACISP.BIN /tmp

The IP address for the TFTP in the VSM Host OS depends on the RSP slot. Enter the ifconfig host-eth0 command on Linux in order to check the subnet number and then use the same subnet address with the last octet "1" in order to access the RP. Recopy the upgrade and binary scripts to the host OS.

Execute the upgrade command.

RP/0/RSP1/CPU0:a9k#run attachCon 0/0/cpu0
host login: root
Password: lab

[host:~]$ ifconfig host-eth0
host-eth0 Link encap:Ethernet HWaddr 56:4B:47:00:08:43
inet addr:192.0.132.3 Bcast:192.255.255.255 Mask:255.0.0.0
<. . .>

[host:~]$ cd /tmp
[host:~]$ tftp 192.0.132.1 -c get /tmp/usb_fw_upgrd
usb_fw_upgrd [host:~]$ tftp 192.0.132.1 -c get /tmp/SM325ACISP.BIN SM325ACISP.BIN
[host:~]$ chmod +x usb_fw_upgrd

[host:~]$ ./usb_fw_upgrd /dev/sg2 /tmp/SM325ACISP.BIN 0 Unigen 1 1 140109 /tmp/usb.bin

If the eUSB firmware is already Version 140109, the script reports it and exits without making any changes. The upgrade log in /var/log/eusb_fw_upgrade_log indicates an upgrade not required.

[host:/tmp]$ grep 140109 /var/log/eusb_fw_upgrade_log

Wed Jun 8 17:49:08 2016: CURRENT FIRMWARE VERSION : 140109
Wed Jun 8 17:49:08 2016: CONTROLLER VERSION : SM325AB
Wed Jun 8 17:49:08 2016: UPGRADE IMAGE FIRMWARE VERSION : 140109
Wed Jun 8 17:49:08 2016: CURRENT FIRMWARE VERSION=SM325A MATCHES UPGRADE IMAGE VERSION=140109
Wed Jun 8 17:49:08 2016: UPGRADE NOT REQUIRED RETRY=0
Wed Jun 8 17:49:08 2016: eUSB FLASH FIRMWARE UPGRDAE LOG END
Wed Jun 8 17:49:08 2016:

Additional Information

Question: Will the Fw upgrade be (or is already) implemented in IOS XR, OVA file or something else?

Answer: No, The FW upgrade is not addressed in XR OVA. The existing FPD framework on asr9k does not contain a facility to perform a firmware upgrade of eUSB. The upgrade must be performed manually.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

Receive Email Notification For New Field Notices

Cisco Notification Service—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.

Was this Document Helpful?

FeedbackFeedback

Contact Cisco