Troubleshoot Power Supply Unit Failure in NCS XR Platform

Contents

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

The Cisco NCS XR router series includes several platforms designed for different use cases and performance levels, each with distinct power supply architectures:

Cisco NCS 540 Series: This is a small-density XR router aimed at sub-100G bandwidth applications such as 5G NR backhaul, FTTx, and enterprise branch deployments. Some models in this series utilize fixed power supplies with 1+1 AC/DC redundancy, meaning the power supply units are integrated into the chassis and are not field-replaceable. Other NCS 540 models can feature modular power supplies.

Cisco NCS 560 Series: This modular system includes modular power supplies with AC and DC options, supporting load-sharing and protection schemes. These power supplies are typically field-serviceable and hot-swappable, allowing for replacement without system shutdown and ensuring high availability.

Cisco NCS 5500 Series: This high fault-resilient modular router platform is designed for data center and high-performance networking environments. It features modular, field-replaceable PSUs that support serviceability and redundancy. The platform supports Cisco IOS XR software with modular packages and resiliency features.

Cisco NCS 5700 Series: Building on the NCS 5500 platform, this series includes enhanced forwarding ASIC design and runs Cisco IOS XR7 OS. The system is modular with field-replaceable PSUs and supports high availability and fault resilience. PSUs are designed for redundancy and hot-swapping. Cisco IOS XR7 OS provides advanced software features that monitor system and fault management.

Problem

The PSU or Power Tray (PT) that consists of PMs in Cisco NCS XR routers is a critical hardware component responsible for converting and providing stable electrical power to the system. PSUs/PTs are often hot-swappable and support redundancy and load sharing. Multiple PSUs can be installed to provide backup power in case one module fails, thereby increasing system availability and minimizing downtime.

A failed or undetected PSU can cause system errors, prevent line cards from booting properly, and lead to system instability or complete shutdown. This can severely impact the operation and network service continuity of the router. The nature and severity of problems vary by platform due to differences in PSU design and serviceability. For models with fixed PSUs (for example, some NCS 540 series), a failure typically requires service or replacement of the entire unit, leading to longer downtime. Modular systems (for example, NCS 560, 5500, 5700, and some 540 models) allow for continued operation during single PSU failures and enable easier maintenance without system shutdown.

Procedure to Resolve PSU Failure in NCS XR Platform

The troubleshooting procedure for PSU failures in NCS XR platforms generally outlines a consistent approach, with specific physical actions differing based on whether the model uses a fixed PSU or a modular PSU.

Step 1. Initial CLI Verification

Login to the router in Cisco IOS XR CLI and execute these commands to identify the status of PSUs. These commands are common across all NCS XR platforms running Cisco IOS XR.

Step 1.1. Check Platform Status: Run this command to identify if it is a PSU failure.

Sample Command Output:

RP/0/RP0/CPU0:NCS-540-B-LNT#show platform
Thu Dec 11 10:06:59.917 +0530
Node         Type                         State            Config state
--------------------------------------------------------------------------------
0/RP0/CPU0   N540X-16Z4G8Q2C-D(Active)   IOS XR RUN          NSHUT
0/PM0        N540-PSU-FIXED-D            OPERATIONAL         NSHUT
0/PM1        N540-PSU-FIXED-D            OFFLINE             NSHUT
0/FT0        N540-X-BB-FAN               OPERATIONAL         NSHUT

Note: If all the Power Modules (for example, `0/PM0`, `0/PM1`) are in 'OPERATIONAL' state, then you can conclude the power supply works fine. Else, if any Power Module is Non-Operational or in a failed state, it implies a PSU failure.

RP/0/RP0/CPU0:NCS-540-B-LNT#show environment power

Thu Dec 11 12:50:16.275 +0530
================================================================================
CHASSIS LEVEL POWER INFO: 0
================================================================================
Total output power capacity : 300W
Total output power required : 175W
Total power input : N/A
Total power output : 97W

================================================================================
Power Supply Status
Module Type 
================================================================================
0/PM1 N540-PSU-FIXED-D OFFLINE
0/PM0 N540-PSU-FIXED-D OK
RP/0/RP0/CPU0:KOL_ISK_901_1AC_M_CNCS540R543#

Note: A status of 'FAILED' or 'NO POWER' for a power module, or significantly low/zero input/output values compared to other modules, indicates a failed or failing power supply.

Step 1.3. Verify Power Module Failure from Alarms: Run this command to check system alarms for power-related alarms.

RP/0/RP0/CPU0:NCS-540-B-LNT#show alarms brief
Thu Dec 11 12:50:02.667 +0530
show alarms brief system active 
--------------------------------------------------------------------------------
Active Alarms for 0/RP0
--------------------------------------------------------------------------------
Location Severity Group Set Time Description 
--------------------------------------------------------------------------------
0/PM1 Major Environ 10/19/2025 12:30:42 +0530 Power Module Generic Fault (PM_GENERIC_FAULT) 
0/PM1 Major Environ 10/19/2025 12:30:42 +0530 Power Module Error (PM_I2C_ACCESS_ERROR) 
0 Major Environ 10/19/2025 12:30:42 +0530 Power Group redundancy lost

--------------------------------------------------------------------------------

Note: Alarm messages indicating 'Power Group Redundancy Lost' or 'Power Module Error' confirm fan failures.

Step 2. Environmental and Physical Inspection

Environmental factors can significantly impact power supply operation and overall system stability.

1. Ambient Conditions:

• Verify ambient temperature and airflow around the router to ensure it is within operational limits. High temperatures can cause power supplies to overheat, reduce their efficiency, and lead to premature failure.

• Check for any obstructions to airflow around the PSUs and the chassis vents. Ensure proper ventilation and heat dissipation pathways are clear.

• Confirm that the power source (for example, AC outlet, DC power feed) is stable and within the specified voltage and current ranges for the NCS series router.

2. Physical Inspection for Obstructions/Damage:

• Inspect the PSUs for any visible debris, loose wiring, or obstructions that can impede connectivity.

• Carefully check all power cables connected to the PSUs. Ensure they are securely seated at both the router and the power source ends. Look for any signs of damage to the cables (for example, frayed wires, cuts, burnt insulation).
• Inspect the PSU itself for any external signs of damage, such as cracks, burnt marks, or unusual odors.
• For platforms with modular PSUs (for example, NCS 560, NCS 5500, NCS 5700, and some NCS 540 models), if safe to do so and within operational guidelines, carefully pull out the suspected PSU. Visually inspect the module for any internal damage, burnt components, or discolored areas. While the module is out, check the chassis slot for debris or damaged connectors.
• For platforms with fixed PSUs (for example, some NCS 540 models), a physical inspection of the PSU and its connectors is limited but must still be performed for any external signs of damage or obstruction. Ensure all power input connections are firm and intact.
• Observe the LED indicators on each PSU. These LEDs typically provide status information (for example, OK, Fault, Input Power, Output Power). Refer to the specific NCS model documentation for the meaning of these indicators.

Step 3. Check for Known Issues and Bugs

Before proceeding with hardware replacement, it is advisable to check if the observed power module failure aligns with any known software or hardware bugs.

1. Cisco Bug Search Tool (BST): Search the Cisco BST using keywords such as 'NCS XR power module failure', 'NCS (model number) power', and the specific Cisco IOS XR version running on your device. Look for known issues that can cause power misreporting or actual failures.
2. Cisco Support Documentation: Review Cisco support documentation and community forums for similar reported issues and recommended workarounds or fixes.

Step 4. Remedial Actions and Replacement

The next steps depend on the type of PSU in your NCS XR Series router.

For NCS XR Platforms with Fixed PS Modules (for Example, some NCS 540 models)

Models with fixed PSUs are typically not hot-swappable.

1. Power Cycle: If the initial checks and environmental adjustments do not resolve the issue, a power cycle of the router can be necessary. This can sometimes clear transient issues and allow the PSU to re-initialize correctly.
2. Replacement RMA: If the fixed PSU is confirmed failed after a power cycle, it typically requires a Return Merchandise Authorization (RMA) for the entire unit or chassis.

Note: Replacement of a fixed PS requires planned downtime as the router must be powered down.

For NCS XR Platforms with Modular PSU Modules (For Example, NCS 560, NCS 5500, NCS 5700, and Some NCS 540 Models)

These platforms feature hot-swappable modular PSUs.

1. Reseating (JACK-OUT and JACK-IN (JOJI)):

• Carefully perform a JOJI procedure on the power module that is experiencing issues. This involves physically removing the power module and then re-inserting it.

• While the module is pulled out, conduct a thorough visual inspection for any debris or loose wiring.
• After reseating, verify the status again using show environment power.
• If any one of the power modules is faulty in a tray, swap the power module across the slots to isolate if the module is faulty or the PT is faulty (if applicable).

2. Replacement RMA: If the issue is isolated to the PT or power module, and reseating does not resolve the problem, it likely indicates a hardware failure. In such cases, customers can raise a case with Cisco TAC for verification. Upon confirmation, Cisco TAC will assess the situation and verify the logs in order to initiate an RMA for the affected PT or power module. Alternatively, if your service level agreement includes direct or automated hardware replacement, the RMA process can proceed automatically without additional verification.

• Collect Evidence Logs: Run 'show logging | include Power` again to capture logs related to the power module JOJI for documentation purposes.

Sample logs:

0/RP0/ADMIN0:Nov 26 06:20:32.269 UTC: shelf_mgr[3081]: %INFRA-SHELF_MGR-5-CARD_REMOVAL : Location: 0/PM0, Serial#: DTMXXXXXX 
0/RP0/ADMIN0:Nov 26 06:20:32.269 UTC: envmon[3021]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :Power Module Output Disabled :CLEAR :0/PM0: Power module is under HW_OUTPUT_DISABLED condition. 
0/RP0/ADMIN0:Nov 26 06:20:32.269 UTC: envmon[3021]: %PKT_INFRA-FM-6-FAULT_INFO : Power Module removal :INFO :0/PM0: 
0/RP0/ADMIN0:Nov 26 06:20:59.052 UTC: envmon[3021]: %PKT_INFRA-FM-6-FAULT_INFO : Power Module insertion :INFO :0/PM0: 
0/RP0/ADMIN0:Nov 26 06:20:59.053 UTC: shelf_mgr[3081]: %INFRA-SHELF_MGR-5-CARD_INSERTION : Location: 0/PM0, Serial #:DTMXXXXXX 
0/RP0/ADMIN0:Nov 26 06:20:59.053 UTC: envmon[3021]: %PKT_INFRA-FM-3-FAULT_MAJOR : ALARM_MAJOR :Power Module Output Disabled :DECLARE :0/PM0: Power module is under HW_OUTPUT_DISABLED condition. 
0/RP0/ADMIN0:Nov 26 06:20:59.053 UTC: shelf_mgr[3081]: %INFRA-SHELF_MGR-6-HW_EVENT : Rcvd HW event HW_EVENT_FAILURE, event_reason_str 'No Input or HW Power Failure' for card 0/PM0 

• Collect Product ID (PID) and Serial Number (SN): Obtain the PID and SN of the faulty power module, which are required for the RMA process.

Sample Command Output:

Command Syntax:

RP/0/RP0/CPU0:NCS-560-B#show inventory location <location of the failed power module>

Sample command:
RP/0/RP0/CPU0:NCS-560-B#show inventory location 0/PM0 
Thu Dec 25 20:41:18.031 KST
NAME: "0/PM0", DESCR: "ASR 900 1200W AC Power Supply"
PID: A900-PWR1200-A    , VID: V03 , SN: DCAXXXXXX

RP/0/RP0/CPU0:NCS-560-B#