Troubleshoot Power Supply Module Failure in ASR 9000

Introduction

This document describes how to troubleshoot power supply module failure in ASR9K.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Cisco IOS® XR
• Familiarity with ASR 9000 hardware architecture

Components Used

The information in this document is based on the ASR 9000 series that encompasses a range of models, including the ASR 9001, ASR 9006, ASR 9010, ASR 9901, ASR 9906, ASR 9910, ASR 9912, and ASR 9922, among others.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

The Cisco ASR 9000 Series Aggregation Services Routers (ASR9k) are high-performance routers designed for service provider networks, offers scalability, reliability, and advanced features to support network environments demands. The ASR9k routers provide modular hardware architecture and allow flexible configuration and expansion to meet diverse network requirements.

ASR9k router family includes:

• Modular Design:ASR9k routers feature modular components such as route processors, line cards, fan trays and power tray/modules that enable easy upgrades and maintenance without network operations disruption.
• Redundant Power for High Availability: ASR9k routers support N+1 or N+N power redundancy configurations, ensuring continuous system operation even if one or more power modules fail. Redundant power setups help prevent service interruptions and maintain uptime in mission-critical deployments.
• Serviceability and Hot-Swap Support: Power supply modules in modular chassis (like ASR 9001, 9006, 9010) are hot-swappable, allowing for online insertion and removal (OIR) without powering down the system. This capability enhances serviceability and minimizes planned downtime during maintenance or upgrades.
• Performance and Scalability:Designed to handle large-scale aggregation and edge routing, ASR9k routers support high throughput and advanced routing protocols suitable for service provider core and edge networks.
• Software Features:The routers run Cisco IOS®XR software, which provides carrier-grade reliability, modularity, and programmability to support evolving network demands.

Problem

The Power Supply Module (PSM) in Cisco ASR9K and other Cisco network devices is a critical hardware component responsible for converting and providing stable electrical power to the system. Power supply modules are often hot-swappable and support redundancy and load sharing. Multiple power supply modules can be installed to provide backup power in case one module fails, thereby increasing system availability and minimizing downtime.

A failed or undetected power supply module can cause system errors, delay driver initialization, and prevent line cards from booting properly, which can severely impact the router’s operation and network service continuity.

Procedure to Resolve Power Supply Module Failure in ASR9K

The procedure to troubleshoot power supply module failures in ASR 9000 Series routers generally outlines a consistent approach across models, with specific physical actions differing based on whether the model uses a fixed or a modular PSM.

Step 1. Initial CLI Verification

Login to the router in Cisco IOSXR CLI and execute the these commands in order to identify the status of power supply modules. These commands are common across all ASR 9000 platforms running Cisco IOSXR.

Step 1.1.Check Plarform Status and Identify Failed Power Supply Module.

Run this command to identify power supply module failure.

Sample Command Output:

RP/0/RSP0/CPU0:ASR-9901-A#show platform
Thu Dec 25 15:32:34.625 CST
Node              Type                       State             Config state
--------------------------------------------------------------------------------
0/RSP0/CPU0       ASR9901-RP(Active)         IOS XR RUN        NSHUT
0/FT0             ASR-9901-FAN               OPERATIONAL       NSHUT
0/FT1             ASR-9901-FAN               OPERATIONAL       NSHUT
0/FT2             ASR-9901-FAN               OPERATIONAL       NSHUT
0/0/CPU0          ASR9901-LC                 IOS XR RUN        NSHUT
0/PT0             A9K-AC-PEM                 OPERATIONAL       NSHUT
RP/0/RSP0/CPU0:ASR-9901-A#

Step 1.2. Identify the details of the failed power supply module in detail.

Run this command to verify the actual power supply and power draw by each modules of the chassis.

sysadmin-vm:0_RSP0# show environment power
Thu Dec  25 07:10:42.486 UTC+00:00
================================================================================
CHASSIS LEVEL POWER INFO: 0
================================================================================
   Total output power capacity (N + 1)             :    1600W +       0W
   Total output power required                     :     930W
   Total power input                               :     518W
   Total power output                              :     457W

Power Shelf 0: 
================================================================================
   Power       Supply     ------Input----   ------Output---      Status
   Module      Type        Volts     Amps    Volts     Amps    
================================================================================
   0/PT0-PM0   1k6W-AC     216.0      2.4     12.0     38.1    OK 
   0/PT0-PM1   1k6W-AC       0.0      0.0      0.0      0.0    FAILED or NO PWR 

Total of Power Shelf 0:        518W/  2.4A       457W/ 38.1A
================================================================================
   Location     Card Type            Power       Power       Status
                                     Allocated   Used
                                     Watts       Watts
================================================================================
   0/0          ASR-9901-LC            600         362       ON 
   0/RSP0       ASR-9901-RP            180          60       ON 
   0/FT0        ASR-9901-FAN            50           -       ON 
   0/FT1        ASR-9901-FAN            50           -       ON 
   0/FT2        ASR-9901-FAN            50           -       ON 
sysadmin-vm:0_RSP0# 

Step 1.3. Identify the details of the installed FPD version of the power supply module. This is to ensure the modules are properly installed with required hardware FPD version.

RP/0/RSP0/CPU0:ASR-9901-A#show hw-module fpd
Thu Dec 25 15:26:13.495 CST
Auto-upgrade:Enabled
                                                                         FPD Versions
                                                                        ==============
Location   Card type             HWver FPD device       ATR Status    Running   Programd 
-----------------------------------------------------------------------------------------
0/RSP0     ASR-9901-RP           1.0   CBC                  CURRENT    54.11     54.11    
0/RSP0     ASR-9901-RP           1.0   Drax-FPGA            CURRENT     0.38      0.38    
0/RSP0     ASR-9901-RP           1.0   IPU-FPGA             CURRENT     2.05      2.05    
0/RSP0     ASR-9901-RP           1.0   IPU-FSBL             CURRENT     1.104     1.104   
0/RSP0     ASR-9901-RP           1.0   IPU-Linux            CURRENT     1.104     1.104   
0/RSP0     ASR-9901-RP           1.0   Primary-BIOS         CURRENT    22.28     22.28    
0/RSP0     ASR-9901-RP           1.0   SSDa-MICRON          N/A         7.05      7.05    
0/RSP0     ASR-9901-RP           1.0   SSDb-MICRON          N/A         7.05      7.05    
0/0        ASR-9901-LC           1.0   CBC                  CURRENT    55.07     55.07    
0/0        ASR-9901-LC           1.0   Gamora-FPGA          CURRENT     0.36      0.36    
0/0        ASR-9901-LC           1.0   IPU-FPGA             CURRENT     1.10      1.10    
0/0        ASR-9901-LC           1.0   IPU-FSBL             CURRENT     1.104     1.104   
0/0        ASR-9901-LC           1.0   IPU-Linux            CURRENT     1.104     1.104   
0/0        ASR-9901-LC           1.0   Primary-BIOS         CURRENT    23.23     23.23    
0/0        ASR-9901-LC           1.0   SSDa-MICRON          N/A         7.05      7.05    
0/PT0      A9K-1600W-AC          0.0   PM0-PO-PriMCU        CURRENT    17.137    17.137   
0/PT0      A9K-1600W-AC          0.0   PM1-PO-PriMCU        CURRENT    17.137    17.137  

Step 2. Environmental and Physical Inspection for Power Supply Module Failure

Environmental factors can significantly impact power supply operation and overall system stability.

Ambient conditions:

• Verify ambient temperature and airflow around the router to ensure it is within operational limits. High temperatures can cause power supplies to overheat, reduce their efficiency, and lead to premature failure.
• Check for any obstructions to airflow around the power supply modules and the chassis vents. Ensure proper ventilation and heat dissipation pathways are clear.
• Confirm that the power source (for example, AC outlet, DC power feed) is stable and within the specified voltage and current ranges for the ASR 9000 series router.

Physical inspection for obstructions/damage:

• Inspect the power supply module(s) for any visible debris, loose wiring, or obstructions that can impede cooling or connectivity.
• Carefully check all power cables connected to the power supply modules. Ensure they are securely seated at both the router and the power source ends. Look for any signs of damage to the cables (for example, frayed wires, cuts, burnt insulation).
• Inspect the power supply module itself for any external signs of damage, such as cracks, burnt marks, or unusual odors.
• If safe to do so and within operational guidelines, carefully pull out the suspected power supply module. Visually inspect the module for any internal damage, burnt components, or discolored areas. While the module is out, check the chassis slot for debris or damaged connectors.
• There can be chances that the Power Entry Modules (PEM) or the Power Tray (PT) of the chassis can be faulty. In this case, pullout the problamatic power module and insert into other slot or other device to ensure the fault follows with power module or it stays with PT or PEM.
• Observe the LED indicators on each power supply module. These LEDs typically provide status information (for example, OK, Fault, Input Power, Output Power). Refer to the specific ASR 9000 model documentation for the meaning of these indicators.

Step 3. Check for Known Issues and Bugs

Before proceeding with hardware replacement, it is advisable to check if the observed power module failure aligns with any known software or hardware bugs.

• Cisco Bug Search Tool:Search the Cisco Bug Search Tool (BST) using keywords such as 'ASR 9000 power module failure', 'ASR (model number) power', and the specific Cisco IOS XR version running on your device. Look for known issues that can cause power misreporting or actual failures.
• Cisco Support Documentation:Review Cisco support documentation and community forums for similar reported issues and recommended workarounds or fixes.

Step 4. Remedial Actions and Replacement

1. Reseating (JACK-OUT and JACK-IN - JOJI):
   • Carefully perform a JACK-OUT and JACK-IN (JOJI) procedure on the power module that is experiencing issues. This involves physically removing the power tray or power module and then re-inserting it based on the fault identification during the phycial inspection.
   • While the power tray or the module is pulled out, conduct a thorough visual inspection for any debris or loose wiring.
   • After reseating, verify the status again using admin show environment power.
   • If any one of the power modules are faulty in a tray, swap the power module across the slots to isolate if the module is faulty or the power tray is faulty.
2. Replacement (RMA): If the issue is isolated to the power tray or power module, and reseating does not resolve the problem, it likely indicates a hardware failure. In such cases, customers can raise a case with Cisco TAC for verification. Upon confirmation, Cisco TAC will assess the situation and verify the logs to initiate an RMA for the affected power tray or power module. Alternatively, if your service level agreement includes direct or automated hardware replacement, the RMA process can proceed automatically without additional verification.
   • Collect Evidence Logs:Run show logging | include Power again to capture logs related to the power module JOJI for documentation purposes.

RP/0/RSP0/CPU0:ASR-9901-A#show logging | include Power
0/RP0/ADMIN0:2024 Jul 24 00:29:21.051 IST: envmon[4804]: %PKT_INFRA-FM-6-FAULT_INFO : Power Module removal :INFO :0/PT0-PM1:
0/RP0/ADMIN0:2024 Jul 24 00:31:26.404 IST: envmon[4804]: %PKT_INFRA-FM-6-FAULT_INFO : Power Module insertion :INFO :0/PT0-PM1:

• Collect Product ID (PID) and Serial Number (SN): Obtain the PID and SN of the faulty power tray or power module, which are required for the RMA process.

Sample Command Output:

Command Syntax:
RP/0/RSP0/CPU0:ASR-9901-A#show inventory location <Localtion of the Power tray or Power module>

Sample Command:
RP/0/RSP0/CPU0:ASR-9901-A#show inventory location 0/PT0 

Thu Dec 25 15:15:36.308 CST

NAME: "0/PT0", DESCR: "Simulated Power Tray IDPROM"
PID: A9K-AC-PEM        , VID: V03, SN: FOTXXXXXXX

NAME: "0/PT0-PM0", DESCR: "1600W AC Power Module"
PID: A9K-1600W-AC      , VID: V01, SN: PORXXXXXX

NAME: "0/PT0-PM1", DESCR: "1600W AC Power Module"
PID: A9K-1600W-AC      , VID: V01, SN: POGXXXXXXX
RP/0/RSP0/CPU0:ASR-9901-A#

Revision History

Revision	Publish Date	Comments
1.0	27-Apr-2026	Initial Release