Troubleshoot ISSU issues | Troubleshooting Device Upgrades | Upgrade Guide for Cisco Catalyst 9000 Series Switches and Smart Switches

Guides you through troubleshooting ISSU issues by verifying readiness, resolving rejection due to configuration synchronization or software mismatch, and investigating stalled ISSU operations, including log collection, commit, and abort procedures.

Use this procedure to troubleshoot ISSU issues by verifying readiness, resolving rejection due to configuration synchronization or software mismatch, and analyse stalled ISSU operations, including log collection, commit, and abort procedures.

The three-step ISSU workflow has an automatic rollback timer. The install commit operation makes the new software permanent and closes the rollback window. The one-step ISSU workflow commits automatically and therefore has no ISSU pre-commit rollback window.

If install abort issu does not clear the state, preserve the diagnostics data and contact Cisco Support. Do not self-run service internal or clear install state. Do not run install remove inactive while rollback is still required.

Before you begin

Verify ISSU compatibility. Refer the ISSU Compatibility Matrix and In-Service Software Upgrade (ISSU) Release Matrix.

Procedure

Confirm ISSU readiness.

Confirm that no ISSU operation is active.
Example:
```
switch# show issu state detail
Current ISSU Status: Enabled
Previous ISSU Operation: N/A
...
No ISSU operation is in progress
```
This is the expected starting state. The ISSU status indicating Enabled alone is not sufficient; confirm that no ISSU operation is active.

Verify STANDBY HOT and SSO state.

Example:

switch# show redundancy states
my state = 13 -ACTIVE
peer state = 8 -STANDBY HOT
Redundancy Mode (Operational) = sso
Redundancy Mode (Configured)  = sso
Communications = Up

An ISSU-capable redundant system requires a synchronized STANDBY HOT peer, operational SSO, and communications Up.

Verify configuration synchronization.

Example:

switch# show redundancy config-sync failures mcl
Mismatched Command List
-----------------------
The list is Empty

Verify parser return-code synchronization.

Example:

switch# show redundancy config-sync failures prc
The list is Empty

Verify best-effort synchronization.
Example:
```
switch# show redundancy config-sync failures bem
The list is Empty
```
Empty lists mean no reported active-to-standby synchronization failures.

Verify ISSU compatibility inputs.

Example:

switch# show version
switch# show boot
switch# show module
switch# show install summary
switch# show issu comp-matrix

Resolve ISSU rejection by configuration synchronization.
1. Identify MCL synchronization failures and remove or modify the incompatible commands.
  Example:
  
  If the device displays this ERROR log, run the show redundancy config-sync failures historic mcl command.
```
ERROR: MCL-Terminating the current ISSU Operation.
ISSU Abort operation is initiated

ERROR: Once ISSU abort is done please check full list of
mismatched commands via:
show redundancy config-sync failures historic mcl
```
```
switch# show redundancy config-sync failures historic mcl
...
<INCOMPATIBLE_COMMANDS>
```
  The output indicates that at least one configured command cannot synchronize to the target release.
  
  Review each command and its service impact. Remove or modify it only when it is operationally safe. If it cannot be removed, use a normal INSTALL-mode upgrade; do not force the ISSU operation.
2. Identify software or ROMMON mismatch.
  Example:
```
%REDUNDANCY-2-IPC: IOS versions do not match.
```
  The error log indicates that there is a mismatch of software and ROMMON versions.
  
  Compare the software and ROMMON on both supervisors or members. ISSU must not continue until the exact platform requirements are met.
Analyse an ISSU operation that appears stalled.
1. Check the current ISSU stage.
  Example:
```
switch# show issu state detail
Current ISSU Status: In Progress
Previous ISSU Operation: N/A
```
  In Progress status can be normal during a transition. Compare the current stage and timestamps with the expected change-window timing.
2. Recognize a new ISSU operation that is blocked.
  Example:
```
switch# install add file flash:<IMAGE.bin> activate issu commit
FAILED: ISSU in progress; please use install abort issu to abort
the ISSU operation OR use install commit to commit the ISSU operation.
```
  This error log indicates that an earlier three-step ISSU transaction is still active. Do not start another add or activation step in between.
3. Collect ISSU, install, redundancy, and log state.
  Example:
```
switch# show clock
switch# show issu state detail
switch# show issu sessions
switch# show issu entities
switch# show issu negotiated
switch# show install summary
switch# show install log
switch# show redundancy
switch# show redundancy states
switch# show logging | include ISSU|INSTALL|REDUNDANCY
```
4. Commit an intentionally uncommitted ISSU target if post-upgrade validation has passed and the active three-step ISSU is awaiting commit.
  Example:
```
switch# install commit
...
SUCCESS: install_commit
```
  The commit operation makes the ISSU target persistent and closes its rollback window.
5. If the active three-step ISSU is uncommitted and must be abandoned, then abort that active uncommitted ISSU operation.
  Example:
```
switch# install abort issu
...rollback messages...
SUCCESS: ISSU abort completed
```
  This operation returns the system to the previously installed software and can reload supervisors or members. Use it only after confirming that ISSU is active and not committed.