Troubleshoot Identity Services Engine (ISE) Upgrade Failures

Introduction

This document describes the actions that you can take to troubleshoot upgrade failures with Cisco Identity Services Engine.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Basic knowledge of Cisco Identity Service Engine

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

It is common practice to reimage as a last resource however, the purpose is to enable you with the knowledge to find the root cause along with Cisco TAC.

• Full Upgrade: Full upgrade is a multi-step process that enables a complete upgrade of all the nodes in your Cisco ISE deployment at the same time. This method upgrades the deployment in lesser time when compared to the split upgrade process. The application services are down during this upgrade process because all nodes are upgraded parallelly.

• Legacy Split Upgrade: Split upgrade is a multi-step process that enables the upgrade of your Cisco ISE deployment while it allows services to remain available during the upgrade process. This upgrade method allows you to choose the Cisco ISE nodes to be upgraded on your deployment.

• Split Upgrade: Split upgrade is a multi-step process that enables the upgrade of your Cisco ISE deployment to remain available during the upgrade process. This upgrade method allows you to choose the Cisco ISE nodes to be upgraded on your deployment. In the new split upgrade workflow, the prechecks and data upgrade happens when the system is up, it reduces the downtime considerably and leads to a reliable upgrade.

Note: The Full Upgrade method is supported for Cisco ISE 2.6 patch 10 and above, Cisco ISE 2.7 patch 4 and above, and Cisco ISE 3.0 patch 3 and above. The Legacy Split Upgrade method can be done on any supported Cisco ISE version and patch. The new Split Upgrade method is a feature in the rodmap.

Split Upgrade

Split upgrade flow

Full Upgrade

Full upgrade flow

Gather Information

Pre-Checks Failed

Refer to the ADE.log and ise-psc.log files.

show logging system ade/ADE.log
show logging application ise-psc.log

Configuration Data Upgrade Check

Refer to ADE.log, configdb-upgrade-[timestamp].log and dbupgrade-data-global-[timestamp].log on secondary admin node.

show logging system ade/ADE.log
show logging application configdb-upgrade-[timestamp].log
show logging application dbupgrade-data-global-[timestamp].log

Note: When you collect the Support Bundle, make sure to enable full configuration database check to include configdb-upgrade logs.

Upgrade Issues

Full Upgrade

If the upgrade fails on the PAN or any of the secondary nodes.

Refer to ADE.log and ise-psc.log

show logging system ade/ADE.log
show logging application ise-psc.log

Additional logs:

monit.log

Note: Remember to always collect the Support Bundle before you perform any workaround.

Workaround

If the primary admin node upgrade fails, promote the secondary admin to the primary admin and then re-try the upgrade.

Split Upgrade

Upgrade failed in one of the nodes and cannot continue with rest of the deployment.

Refer to ADE.log and ise-psc.log

show logging system ade/ADE.log
show logging application ise-psc.log

Additional logs:

monit.log

Workaround

If the upgrade fails on any other node apart from primary admin, the node would have to be deregistered from the deployment. This node has to be upgraded individually or reimaged directly to the upgraded version and can be joined back to the deployment.

Known Scenarios

Upgrade Gets Stuck on One of the Nodes

There are scenarios where upgrade gets stuck for more than 5-6 hours.

None of the initial steps where services need to be stopped has a timeout configured, hence it would be stuck indefinitely if something fails. On later stages, DB schema and schema upgrade do have timeout configured.

Proceed with Support Bundle collection. ADE logs shows at which step its blocked, more specific debugs are collected based on this information.

Workaround

The only option to take off the node from this state is a manual reload.

Pre-Checks Time Out Before Configuration Data Upgrade is Completed

Pre-checks failure

Workaround

Hit refresh failed checks.

Known Upgrade Defects

Cisco bug ID CSCwa04370 - ISE 3.1 Defalt route removed or tied to wrong interface after upgrading.

Cisco bug ID CSCwa82553 - ISE 3.1 Default route is on the incorrect interface if bonding is configured. 

Cisco bug ID CSCwa08018 - ISE 3.1 GUI does not work when IPV6 is disabled globally.

Related Information