Introduction
This document describes the NCS1K-PSM modules exhibiting higher than expected switching time during commissioning.
Background Information
During the commissioning of NCS1K-PSM modules (OPM29151332 and OPM29151335), the customer observed high switching-test return times. The expected switching time was below 50 ms, However, the observed switching time exceeded 300 ms, impacting commissioning and validation activities within the optical network.
The devices were operating on Cisco IOS® XR Software version 7.10.1, and QDD-2X100-LR4-S client optics were used in the deployment.
Problem Description
While performing switching validation tests during commissioning, it is reported that the PSM switching return time was significantly higher than the expected threshold. Instead of completing switching operations within the expected 50 ms, switching events were observed to take greater than 300 ms, resulting in delays during commissioning and operational verification.
In order to investigate the issue, multiple diagnostic artifacts were shared including:
-
show tech-support outputs
-
switching test logs
-
configuration files
-
test result spreadsheets
These logs were analyzed.
Root Cause
After detailed analysis of the collected diagnostics and switching test results, determined that the issue was related to a known Cisco software defect: Cisco Bug ID CSCwr67622.
This defect affects CIM-8 trunk modules during PSM switching, causing abnormally high switching latency under certain operational conditions. The behavior observed in the customer environment matched the known symptoms documented in the bug.
The defect resulted in performance degradation in the PSM switching process, leading to switching times exceeding the expected threshold.
Troubleshooting and Diagnostic Actions
Cisco TAC performed these steps in order to isolate the issue:
-
Log Analysis
-
Reviewed detailed show tech-support files and switching test logs provided by the customer.
-
Identified switching delay patterns consistent with the behavior described in Cisco Bug ID CSCwr67622.
-
-
Engineering Validation
-
Cisco engineering team confirmed that the symptoms matched the defect scenario affecting PSM switching on CIM-8 modules.
-
-
Software Fix Deployment
-
Cisco engineering provided a targeted software fix addressing the switching delay issue.
-
The fix was deployed on the NCS 1000 devices of the customer.
-
Resolution
After applying the software fix:
-
Extensive switching tests were conducted.
-
Multiple switching cycles were executed to verify performance.
-
Test results confirmed that switching times were restored to below 50 ms, meeting expected operational thresholds.
The fix successfully resolved the performance issue in the PSM switching mechanism.
Conclusion
The issue was caused by a Cisco software defect Cisco Bug ID CSCwr67622 affecting PSM switching behavior on CIM-8 trunk modules, resulting in switching times exceeding the expected threshold.
Feedback