Troubleshoot Modules in "SW_INACTIVE" State for IOS XR

Introduction

This document describes "SW_INACTIVE" in CIsco IOS® XR and how to troubleshoot it.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

This document is not restricted to specific hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

"SW_INACTIVE" state in Linecards and Router Switch Processor/ Router Processor (RSP/RPs) for Cisco IOS XR platforms, for example, the next output displays the error.

RP/0/RSP0/CPU0:XR#admin show platform
Location  Card Type               HW State      SW State      Config State
----------------------------------------------------------------------------
0/1       A9K-MOD400-TR           OPERATIONAL   OPERATIONAL   NSHUT
0/0       A9K-MOD400-TR           OPERATIONAL   SW_INACTIVE   NSHUT  >>>
0/RSP0    A9K-RSP880-SE           OPERATIONAL   OPERATIONAL   NSHUT
0/RSP1    A9K-RSP880-SE           OPERATIONAL   OPERATIONAL   NSHUT
0/FT0     ASR-9010-FAN-V2         OPERATIONAL   N/A           NSHUT
0/FT1     ASR-9010-FAN-V2         OPERATIONAL   N/A           NSHUT
0/PT0     A9K-DC-PEM-V3           OPERATIONAL   N/A           NSHUT
0/PT1     A9K-DC-PEM-V3           OPERATIONAL   N/A           NSHUT

In Cisco IOS XR platforms, 64 bits (also know as eXR), the next set of states compose the Finite State Machine (FSM) for a linecard/RSP/RP to boot.

Said FSM is valid for versions higher than 7.1.1 and other NCS5500 versions.

The states are the next:

• "SW_INACTIVE"
• Image Baking
• Booting

The "SW_INACTIVE" (software Inactive) state, means that the software is not ready to be run on the device. The node does not have enough information to continue with the next state and is requesting information to the active RSP/RP to bake the image.

In other words, "SW_INACTIVE" is the state where the node (linecard/RSP/RP) synchronizes with the active (primary) node.

Why does the linecard needs to synchronize with the active node?

• Each node that is not an RSP/RP does not have the primary capabilities to store Optical Disc Image (ISO) images, grub files, and so on.
• Cisco IOS XR archives the distributed system having one main node and clients.
• In Client-Server analogy, the RSP/RP is the Server while the other nodes are the Clients.

Information Downloaded from the Primary Node (Active Route Processor)

The next output shows some of the files that the linecard downloads to start the FSM and boot to IOS_XR_RUN state.

The main files are:

• Common file
• Mini .x64 file
• Boot
• Grub.cfg
• Grub.efi
• System image pointing to the mini file

When a Node is Inserted

• Perspective from the RSP:
• Whenever an RSP is inserted into the chassis, PCIE messages are exchanged in the fabric to ask if there is any other RSP/RP inserted in the chassis. In case the RSP/RP does not detect any other Route Processor, it assumes the active role; if the recently inserted Route Processor detects any other Route Processor already inserted, it forwards all the necessary information to become the standby node. The main goal for each Route processor is to take the role necessary to achieve redundancy in the system.
• Once the Route Processor is identified with a role, process manager initiates all the processes and performs the entries in sysDB for other processes to have a main Database tree.

• Perspective from the Linecard:
• When a linecard is inserted, it sends PCIE messages and asks who is the active. Once it identifies this node, it initializes the data synchronization. Before any booting, the linecard requests an IP via DHCP (for internal communication) and pulls the booting information, if the ISO stored in the linecard is the same as the Active Route Processor then it continues with the next FSM state: Baking.
• In case the linecard does not have the same ISO information, it removes the ISO and other files, and requests them to the Active Route Processor. Once it has the files, it proceeds with the FSM.

Note: Linecards and Standby Route Processors use the same process checking ISO and all the required information to boot. Peripheral Component Interconnect Express (PCIE) protocol was built to handle point-to-point connections for non-core components. One of the first FPGA that can start the communication and establish control path for the linecards is IOFPGA and MIFPGA. These FPGAs are focused in PCIE communications.

To archive the mentioned client-server model, Cisco IOS XR needs to:

• Provide a physical connection to the primary node
• Provide an IP Address and Gateway Address
• Tell the client node what information needs to be provisioned

If a Module Remains in the Same State for a Long Time

1. If the linecard shows this boot loop, proceed to the Information to Collect section.

0/RSP1/ADMIN0:canbus_driver[3903]: %PLATFORM-CANB_SERVER-7-CBC_POST_RESET_NOTIFICATION : Node 0/1 CBC-0, reset reason CPU_RESET_POR (0x05000000)
0/RSP0/ADMIN0:shelf_mgr[3945]: %INFRA-SHELF_MGR-6-HW_EVENT : Rcvd HW event HW_EVENT_POWERED_OFF, event_reason_str 'HW Event Powered OFF' for card 0/1
0/RSP0/ADMIN0:shelf_mgr[3945]: %INFRA-SHELF_MGR-6-HW_EVENT : Rcvd HW event HW_EVENT_POWERED_ON, event_reason_str 'HW Event Powered ON' for card 0/1
0/RSP0/ADMIN0:shelf_mgr[3945]: %INFRA-SHELF_MGR-6-HW_EVENT : Rcvd HW event HW_EVENT_OK, event_reason_str 'HW Event OK' for card 0/1
0/RSP0/ADMIN0:shelf_mgr[3945]: %INFRA-SHELF_MGR-6-CARD_HW_OPERATIONAL : Card: 0/1 hardware state going to Operational
0/RSP1/ADMIN0:canbus_driver[3903]: %PLATFORM-CANB_SERVER-7-CBC_PRE_RESET_NOTIFICATION : Node 0/1 CBC-0, reset reason CPU_RESET_POR (0x05000000)
0/RSP0/ADMIN0:shelf_mgr[3945]: %INFRA-SHELF_MGR-6-HW_EVENT : Rcvd HW event HW_EVENT_RESET, event_reason_str 'HW Event RESET' for card 0/1 
0/RSP1/ADMIN0:canbus_driver[3903]: %PLATFORM-CANB_SERVER-7-CBC_POST_RESET_NOTIFICATION : Node 0/1 CBC-0, reset reason CPU_RESET_POR (0x05000000)
0/RSP0/ADMIN0:shelf_mgr[3945]: %INFRA-SHELF_MGR-6-HW_EVENT : Rcvd HW event HW_EVENT_POWERED_OFF, event_reason_str 'HW Event Powered OFF' for card 0/1
0/RSP0/ADMIN0:shelf_mgr[3945]: %INFRA-SHELF_MGR-6-HW_EVENT : Rcvd HW event HW_EVENT_POWERED_ON, event_reason_str 'HW Event Powered ON' for card 0/1

Note: This state does not mean that the linecard has a hardware failure. Note that "HW_EVENT_OK" means the linecard hardware is working as expected, FPGA and all PCIE show no issue.

Warning: Interrupting boot process generates boot process to restart and modules take time to finish. Be aware of the logs displayed in the CLI. If the logs do not show any error, let the linecard/module continues with the process.

In case the router detects an error in the boot process, the router automatically performs recovery actions.

2. If the linecard shows this logs, do not remove it from the chassis. The linecard is in the downloading process and needs to finish.

0/0/ADMIN0: inst_agent[3930]: %INFRA-INSTAGENT-4-XR_PART_PREP_REQ : Received SDR/XR partition request. Looking for available matching partition. If not found, new one will be created after copying relevant image and RPMs
0/0/ADMIN0: fpd_agent[3927]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :Golden FPGA is unprotected :DECLARE :0/0: FPD IPU-FP
Golden FPGA is below minimum version, Perform force fpd upgrade for IPU-FPGA
0/0/ADMIN0: fpd_agent[3927]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :Golden FPGA is unprotected :DECLARE :0/0: FPD IPU-FS
Golden FPGA is below minimum version, Perform force fpd upgrade for IPU-FSBL
0/0/ADMIN0: fpd_agent[3927]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :Golden FPGA is unprotected :DECLARE :0/0: F
IPU-Linux Golden FPGA is below minimum version, Perform force fpd upgrade for IPU-Linux
0/0/ADMIN0: inst_agent[3930]: %INFRA-INSTAGENT-4-XR_PART_PREP_IMG : SDR/XR image baking in progress

Note: After this, FPD gets upgraded and the linecard must boot with no problem. In case an issue is observed, check the Action Items and Information to open a case with TAC sections.

3. In case there is no log from the recently inserted linecard please go to the section Information to Collect.

Information to Collect

For "SW_INACTIVE" focus on the XR VM and Admin VM.

Consider the next, when issuing the show platform command:

• If the previous command is issued in XR VM and you see "SW_INACTIVE", but "OPERATIONAL" using command admin show platform, XR VM point of view needs to be analyzed.

This example displays the output for both show platform commands:

RP/0/RSP0/CPU0:XR# show platform
Node              Type                       State             Config state
--------------------------------------------------------------------------------
0/RSP0            A9K-RSP5-SE                SW_INACTIVE       NSHUT
0/RSP1/CPU0       A9K-RSP5-SE(Active)        IOS XR RUN        NSHUT
0/FT0             ASR-9006-FAN-V2            OPERATIONAL       NSHUT
0/FT1             ASR-9006-FAN-V2            OPERATIONAL       NSHUT
0/0/CPU0          A9K-24X10GE-1G-SE          IOS XR RUN        NSHUT
0/1/CPU0          A9K-24X10GE-1G-SE          IOS XR RUN        NSHUT
0/2               A9K-20HG-FLEX-SE           SW_INACTIVE       NSHUT
0/3               A9K-20HG-FLEX-SE           SW_INACTIVE       NSHUT
0/PT0             A9K-DC-PEM-V2              OPERATIONAL       NSHUT

RP/0/RSP0/CPU0:XR# admin show platform
Location  Card Type               HW State      SW State      Config State 
----------------------------------------------------------------------------
0/0       A9K-24X10GE-1G-SE       OPERATIONAL   OPERATIONAL   NSHUT        
0/1       A9K-24X10GE-1G-SE       OPERATIONAL   OPERATIONAL   NSHUT        
0/2       A9K-20HG-FLEX-SE        OPERATIONAL   OPERATIONAL   NSHUT        
0/3       A9K-20HG-FLEX-SE        OPERATIONAL   OPERATIONAL   NSHUT        
0/RSP0    A9K-RSP5-SE             OPERATIONAL   OPERATIONAL   NSHUT        
0/RSP1    A9K-RSP5-SE             OPERATIONAL   OPERATIONAL   NSHUT        
0/FT0     ASR-9006-FAN-V2         OPERATIONAL   N/A           NSHUT        
0/FT1     ASR-9006-FAN-V2         OPERATIONAL   N/A           NSHUT        
0/PT0     A9K-DC-PEM-V2           OPERATIONAL   N/A           NSHUT 

Note: The shown output displays information from a different router than the one used in further show commands.

• If the command admin show platform displays "SW_INACTIVE", but "OPERATIONAL" using command show plarform the problem needs to be analyzed from Host OS and Calvados perspective.

This example displays the output for both show platform commands:

RP/0/RSP0/CPU0:XR# show platform
Node              Type                       State             Config state
--------------------------------------------------------------------------------
0/RSP0            A9K-RSP5-SE                IOS XR RUN       NSHUT
0/RSP1/CPU0       A9K-RSP5-SE(Active)        IOS XR RUN        NSHUT
0/FT0             ASR-9006-FAN-V2            OPERATIONAL       NSHUT
0/FT1             ASR-9006-FAN-V2            OPERATIONAL       NSHUT
0/0/CPU0          A9K-24X10GE-1G-SE          IOS XR RUN        NSHUT
0/1/CPU0          A9K-24X10GE-1G-SE          IOS XR RUN        NSHUT
0/2               A9K-20HG-FLEX-SE           IOS XR RUN       NSHUT
0/3               A9K-20HG-FLEX-SE           IOS XR RUN       NSHUT
0/PT0             A9K-DC-PEM-V2              OPERATIONAL       NSHUT

RP/0/RSP0/CPU0:XR# admin show platform
Location  Card Type               HW State      SW State      Config State 
----------------------------------------------------------------------------
0/0       A9K-24X10GE-1G-SE       OPERATIONAL   SW_INACTIVE   NSHUT        
0/1       A9K-24X10GE-1G-SE       OPERATIONAL   OPERATIONAL   NSHUT        
0/2       A9K-20HG-FLEX-SE        OPERATIONAL   OPERATIONAL   NSHUT        
0/3       A9K-20HG-FLEX-SE        OPERATIONAL   OPERATIONAL   NSHUT        
0/RSP0    A9K-RSP5-SE             OPERATIONAL   OPERATIONAL   NSHUT        
0/RSP1    A9K-RSP5-SE             OPERATIONAL   OPERATIONAL   NSHUT        
0/FT0     ASR-9006-FAN-V2         OPERATIONAL   N/A           NSHUT        
0/FT1     ASR-9006-FAN-V2         OPERATIONAL   N/A           NSHUT        
0/PT0     A9K-DC-PEM-V2           OPERATIONAL   N/A           NSHUT  

Continuing with Troubleshooting

Once a boot loop is identified or the linecard is not going to Baking, check:

As mentioned, the linecard needs to download the file and boot. For this there are 2 options:

First Option: Connect to the console of the linecard:

For this, issue the next commands:

RP/0/RP0/CPU0:XR#admin
sysadmin-vm:0_RP0#run chvrf 0 bash -l
[sysadmin-vm:0_RSP0:~]$ attachCon 0/X --> X is the linecard slot number

Example shows how to connect to linecard 0/0/CPU0:

RP/0/RP0/CPU0:XR#admin

sysadmin-vm:0_RP0# run chvrf 0 bash -l

[sysadmin-vm:0_RSP0:~]$ attachCon 0/0
===============================================
====       Connecting to Line Card        =====
===============================================
Line Card: No 0
Press <Ctrl-W> to disconnect
Enabling 16550 on uart 0 baud rate 115200

host login:

Note: Select enter and insert the host log in information, information used when provisioning the router for the first time.

Second Option: Check PCIE logs in the Admin VM:

In admin issue:

sysadmin-vm:0_RSP0# run chvrf 0 bash -l
[sysadmin-vm:0_RSP0:~]$ ls /misc/scratch/pcie/loggerd/

This command displays the linecards that are inserted in the chassis.

Example shows the command in a chassis with only one linecard inserted:

[sysadmin-vm:0_RSP0:~]$ ls /misc/scratch/pcie/loggerd
LC0

Once the "SW_INACTIVE" linecard is identified and displayed in the output, proceed to issue the next command:

[sysadmin-vm:0_RSP0:/]$ ls /misc/scratch/pcie/loggerd/LC0 

Note: When the commands are issued, consider that it displays different linecards. In such case, issue the command with the linecard of interest. For this example LC0.

The command  displays the files in the linecard. Note any file with pslot.

Warning: When issuing the command, consider users can lose access to the CLI. In case the CLI gets frozen, select Ctrl-W.

Example outputs for the pslot found in linecard 0/0/CPU0:

[sysadmin-vm:0_RSP0:~]$ ls /misc/scratch/pcie/loggerd/LC0

[sysadmin-vm:0_RSP0:~]$ ls /misc/scratch/pcie/loggerd/LC0
first  last  pslot_2_uart_0_w0

[sysadmin-vm:0_RSP0:~]$ ls /misc/scratch/pcie/loggerd/LC0/pslot_2_uart_0_w0

The next information is displayed after selecting enter.

===============================================
====       Connecting to Line Card        =====
===============================================
Line Card: No 0
Press <Ctrl-W> to disconnect
Enabling 16550 on uart 0 baud rate 115200
TriggerCpRmonInit 

ASR9K Init Starting ASR9k initialization ...
Reading both MB and DB cookie
Board Type:0x3d1013
Starlord LC setting i2c block 7

 The base address of i2c_mux4to1 is at dc30e000
zl init starlord lc
Initializing Zl clock to 322MHz
Missing Parameter SERVER_URL

##########################################################
System Bootstrap, Version 23.278 [ASR9K x86 ROMMON],
Copyright (c) 1994-2020 by Cisco Systems, Inc.
Compiled on Sat 03/14/2020 14:37:06.76 

BOARD_TYPE             : 0x3d1013 
Rommon                 : 23.278 (Primary) 
IPU FPGA(PL)           : 1.10.1 (Primary)
IPU INIT(HW.FPD)       : 1.10.1
IPU FSBL(BOOT.BIN)     : 1.104.0
IPU LINUX(IMAGE.FPD)   : 1.104.0
GAMORA FPGA            : 0.36.1
CBC0                   : Part 1=55.7, Part 2=55.7, Act Part=1 
Product Number         : ASR-9901-LC
Slot Number            : 2

##########################################################
Got EMT Mode as Disk Boot
<snip>

Note: In some cases, with the linecard in "SW_INACTIVE", the PCIE displays the boot loop. Proceed to log the session and collect these outputs for further troubleshooting.

• This information needs to be collected:

Issue the next commands:

RP/0/RP0/CPU0:PE2#admin
sysadmin-vm:0_RP0# run chvrf 0 bash -l
[sysadmin-vm:0_RSP0:~]$ls -lah /misc/disk1/tftpboot

The next output is the output of this command:

sysadmin-vm:0_RSP0# run ls -lah /misc/disk1/tftpboot
total 3.2G
drwxrwxrwx.  3 root root 4.0K Jan 31 02:14 .
drwxrwxrwx. 12 root root 4.0K Jan 31 01:49 ..
-rw-r--r--.  1 root root 119M Jan 31 01:46 asr9k-common-7.3.2
-rw-r--r--.  1 root root 120M Dec 18 15:32 asr9k-common-7.5.2
-rw-r--r--.  1 root root 1.4G Jan 31 01:36 asr9k-mini-x64-7.3.2
-rw-r--r--.  1 root root 1.5G Dec 18 15:25 asr9k-mini-x64-7.5.2
drwxr-xr-x.  3 root root 4.0K Mar 16  2022 boot
-rwxr-xr-x.  1 root root  637 Jan 31 01:57 grub.cfg
-rw-r--r--.  1 root root 906K Jan 31 01:48 grub.efi
lrwxrwxrwx.  1 root root   22 Jan 31 01:57 system_image.iso -> ./asr9k-mini-x64-7.3.2
-rwxrwxrwx.  1 root root 1007 Jan 31 01:58 zapdisk.log

From the output, consider the next:

• If no file is seen under this directory, continue with the Information to open a case with TAC section.
• If files are seen, it means that either the files are corrupted or the router cannot bake them as expected. Continue with the Action Items section.

• Check DHCP configuration in the linecard/Route Processor:

To review the DHCP configuration, issue the next commands:

RP/0/RP0/CPU0:XR#admin
sysadmin-vm:0_RP0# run chvrf 0 bash -l
[sysadmin-vm:0_RSP0:~]$cat /etc/dhcp/dhcpd.conf

The example for this command outputs:

RP/0/RP0/CPU0:XR#admin

sysadmin-vm:0_RP0# run chvrf 0 bash -l
[sysadmin-vm:0_RSP0:~]$cat /etc/dhcp/dhcpd.conf
ddns-update-style none;

default-lease-time 86400;
max-lease-time 604800;
authoritative;

subnet 192.168.0.0 netmask 255.0.0.0 {
        pool {
           max-lease-time 600;
           range 192.168.0.128 192.168.0.191;
           allow unknown-clients;
        }

        filename "grub.efi";
        next-server 192.168.0.1;
        option subnet-mask 255.0.0.0;
        option broadcast-address 192.168.0.255;
        option ip-forwarding off;
}

Like in the previous scenario, check if this file displays information, if so, please proceed to Information to Open a Case with TAC section.

If no output is seen, please proceed to the Action Item.

Tip: In some cases, and depending on the scenario fpd auto-upgrade enable needs to be configured to both Admin VM and XR VM.

Solution

Action Items

Problem Statement 1

"I see the files in /misc/disk1/tftpboot and /etc/dhcp/dhcpd.conf displays information": Add the fpd auto-upgrade in the XR VM (for more information click: System Management Configuration Guide)

Once this is configured, Online Insertion and Removal (OIR) the Linecard and check the logs that this one displays.

Problem Statement 2

"I do not see any file in /misc/disk1/tftpboot but/etc/dhcp/dhcpd.conf displays information": This can be an issue with the Active Route Processor not being able to share the files.

• Check if this is the only linecard showing this state. If you insert a different linecard and see the same issue, then, the active route processor is not sharing the right files. Consider re-downloading the Software version in the Route Processor. If linecards are not continuing with the FSM as expected, the Active Route processor can have a corrupted file.
• If any other linecard is inserted and boots as expected, issue this command in admin mode while affected module is inserted:

sysadmin-vm:0_RSP0# hw-module location <Linecard in SW_INACTIVE state> bootmedia network reload 

In this example, the command is issued for linecard 0/0/CPU0:

sysadmin-vm:0_RSP0# hw-module location 0/0 bootmedia network reload 

Check the logs with the same troubleshooting steps. If the logs are similar, continue to Information required to open a case with TAC section:

Problem Statement 3

"I do not see files in misc/disk1/tftpboot nor information is displayed in /etc/dhcp/dhcpd.conf": This scenario needs further checking from TAC. Proceed to Information required to open a case with TAC section.

Information Required to Open a Case with TAC

Tip: When opening a case, collect all the information and attach it to the case (when opened). This avoids any delay in the investigation

Logs to be collected:

In XR VM:

show logging

Route Processor/Linecard console logs; in case the issue is presented in the standby Route Processor, consider connecting the console cable to this node and log the output that is being displayed.

show hw-module fpd location all

In XR VM:

show controllers switch statistics location <Route Processor/Linecard>

show controllers switch statistics detail location <active Route Processor> <switch port no>

show alarms

Files to attach:

 In Admin VM:

Show tech-support OS

Show tech-support canbus

Show tech-support control-ethernet

Show tech-support ctrace

Note: Admin show techs: To copy the admin show tech to the default XR_PLANE, issue the next command in admin mode: copy harddisk:/showtech/ harddisk:/showtech/ location 0/RSP/VM1. In case there are issues copying the file, click this link: Show Tech Support files to XR VM.