Upgrade Identity Services Engine (ISE)

Introduction

This document describes how to upgrade an existing Identity Services Engine (ISE) version 2.4. to 2.6. set up on the Cisco ISE appliance and Virtual Machine (VM). It also includes how to use the Upgrade Readiness Tool (URT) to detect and fix any configuration data upgrade issues before the upgrade process begins. 

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Identity Services Engine (ISE)
• Understanding of terminology used to describe different types of ISE deployments

Components Used

The information in this document is based on these software and hardware versions:

• ISE, Release 2.4
• ISE, Release 2.6

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Note: The procedure is similar or identical for other ISE versions. These steps can be used on all 2.x ISE Software Releases unless stated otherwise.

Background Information

A Cisco ISE deployment upgrade is a multistep process and must be performed in the order that is specified in this document. Use the time estimates provided in this document to plan for an upgrade with minimum downtime. For a deployment with multiple Policy Service Nodes (PSNs) that are part of a PSN group, there is no downtime. If there are endpoints that are authenticated through a PSN that is upgraded, the request is processed by another PSN in the node group. The endpoint is re-authenticated and is granted network access after the authentication is successful.

Upgrade Readiness Tool

Use the URT in order to detect and fix any configuration data upgrade issues before you start the upgrade process. Most of the upgrade failures occur because of configuration data upgrade issues. The URT validates the data before the upgrade in order to identify, and report or fix the issue, wherever possible. The URT is available as a separate downloadable bundle that can be run on a Secondary Policy Administration node or standalone node. There is no downtime to run this tool.

This video link explains How to use URT.

URT should be run on Secondary Administration Node or a standalone node.

Caution: The URT tool does not simulate Monitoring node (MnT) operational data upgrades.

Here is an example that demonstrates how to run URT on a standalone node (the same process can be followed on a Secondary Administration Node).

Step 1. Download the URT bundle.

Since the plan is to upgrade to version 2.6., download the URT published on Cisco.com for ISE 2.6 (ise-urtbundle-2.6.0.156-1.0.0.SPA.x86_64.tar.gz) as shown in the image.

URT for 2.6

Step 2. Create a repository and copy the URT bundle.

It is recommended to use File Transfer Protocol (FTP) for better performance and reliability. Do not use repositories that are located across slow WAN links. It is suggested to use a local repository that is closer to the nodes.

From ISE GUI, navigate to Administration > System > Maintenance > Repository > Add as shown in the image. 

Repository

Optionally, in order to save time, copy the URT bundle to the local disk on the Cisco ISE node with the use of this command:

copy repository_url/path/ise-urtbundle-2.6.0.xxx-1.0.0.SPA.x86_64.tar.gz disk:/

(Add the host key if it does not exist) crypto host_key add host mySftpserver
copy sftp://aaa.bbb.ccc.ddd/ ise-urtbundle-2.6.0.xxx-1.0.0.SPA.x86_64.tar.gz disk:/

Step 3. Run the URT Bundle.

Enter the application install command in order to install the URT:

application install ise-urtbundle-2.6.0.x.SPA.x86_64.tar.gz reponame

The warning talks about the services that are running on the node and if the user would still like to continue running the URT on this node. Type Y in order to proceed as shown in the image.

You might notice that sometimes the URT age is old. As long as the bundle is the latest one downloaded from Cisco's website, it is ok to proceed further. The latest URT bundles can also be more than 45 days old. Type Y in order to proceed.

Case 1. URT Run is Successful

1. If the URT runs successfully, the output is similar to this:

2. The URT provides an estimated time for each node's upgrade based on the size of the configuration and MNT data.

3. After you have successfully completed the URT run, proceed to upgrade.

4. The URT:

• Checks if the URT is run on a supported version of Cisco ISE. The supported versions are Releases 2.1, 2.2, 2.3, and 2.4 (for upgrading to version 2.6).
• Verifies that the URT is run on a standalone Cisco ISE node or a Secondary Policy Administration Node (secondary PAN).
• Checks if the URT bundle is less than 45 days old - This check is done to ensure that you use the most recent URT bundle.

Checks if all the prerequisites are met.

These are the prerequisites that are checked by the URT:

• Version compatibility
• Persona checks
• Disk space

Note: Verify the available disk size with Disk Requirement Size. If you are required to increase the disk size, reinstall ISE and restore a config backup.

• NTP server
• Memory
• System and trusted certificate validation

5. Clones the configuration database.

6. Copies latest upgrade files to the upgrade bundle.

Note: If there are no patches in URT bundle then the output will return:N/A.This is expected behaviour when you install a hot patch.

7. Performs a schema and data upgrade on the cloned database.

• If the upgrade on the cloned database is successful, it provides an estimate of the time it should take for the upgrade to end.

• If the upgrade is successful, it removes the cloned database.

• If the upgrade on cloned database fails, it collects the required logs, prompts for an encryption password, generates a log bundle, and stores it in the local disk.

Case 2. URT Run is Unsuccessful

1. The URT can fail due to a reason that can cause issues with the upgrade. If that happens, URT returns the cause of failure.

2. Here is an example run of URT failure:

3. The URT failed with the reason: Trust certificate with the friendly name 'VeriSign Class 3 Secure Server CA - G3' is invalid: The certificate has expired.

4. As explained in the upgrade pre-checks, if ISE system has any expired certificates, the upgrade will fail. It is required that all expired certificates are either renewed or replaced.

5. URT saves the failure logs which can be shared with Cisco TAC if the user is unsure of the failure reason.

6. It will prompt to enter a password to encrypt the logs. These URT logs are saved in the local disk.

7. Copy them from the local disk to a repository and share it with Cisco TAC for resolution.

Upgrade ISE

Before the upgrade begins, ensure that these tasks are completed:

1. Obtain a backup of the ISE configuration and operational data.

2. Obtain a backup of the system logs.

3. Disable scheduled backups. Reconfigure the backup schedules after deployment upgrade is complete.

4. Export the certificates and private keys.

5. Configure a repository. Download the upgrade bundle and place it in the repository.

6. Make a note of Active Directory (AD) join credentials and RSA SecurID node secret, if applicable. This information is needed to connect to Active Directory or RSA SecurID server after the upgrade.

7. Purge the operational data to improve upgrade performance.

8. Ensure that the Internet connection to the repository is good.

Note: The download of upgrade bundle from a repository to a node times out if it takes more than 35 minutes to complete.

Prepare for Upgrade:

Upgrade ISE from the GUI

In case this warning message is seen: "The node has been reverted to its pre-upgrade state", navigate to the Upgrade window, click the Details link. Address the issues that are listed in the Upgrade Failure Details window. After all those issues are fixed, click Upgrade to reinitiate the upgrade.

Step 1. Click the Upgrade tab in the Admin portal.

Step 2. Click Proceed.

Step 3. Check the 'I have reviewed the checklist' checkbox, and click Continue.

Step 4. Download the upgrade bundle from the repository to the nodes:

1. Check the checkbox next to the nodes to which the bundle should be downloaded.

2. Click Download. The Select Repository and Bundle window appears as shown in the image.

3. Select the repository.

4. Check the checkbox next to the bundle that will be used for the upgrade.

5. Click Confirm. Once the bundle is downloaded to the node, the node status changes to Ready for Upgrade.

Step 5. Click Continue. Then, the Upgrade Nodes window appears. Click Upgrade in order to begin.

Upgrade ISE from the CLI

Ensure to read the chapter titled 'Before you begin' before you proceed. Here is an example of a standalone ISE node upgrade from CLI.

Step 1. Check if the repository to be used has the upgrade file present with the use of command show repository reponame as shown in the image.

Step 2. From the Cisco ISE Command Line Interface (CLI), enter application upgrade prepare command with the bundle file name and repository name where the file is stored.

Note: If the upgrade fails for some reason. Before you attempt an upgrade again, use the command application upgrade cleanup in order to clear out the old files.

Common Issues

1. If the Bundle takes too long to download from the repository or times out when downloaded via GUI:

• Ensure there is enough bandwidth to handle the bundle download.
• When an upgrade bundle is downloaded from a repository to a node, the download times out if it takes more than 35 minutes to complete. Ensure that the Internet connection to the repository is good. 

2. In a distributed deployment upgrade, the error "No Secondary Administration Node in the Deployment" can be seen when:

• There is no Secondary Administration node in the deployment.

• The Secondary Administration node is down.

• The Secondary Administration node is upgraded and moved to the upgraded deployment. Typically, this occurs if the Refresh Deployment Details option is used after the Secondary Administration node is upgraded.

In order to resolve this issue, perform one of these tasks, as applicable:

• If the deployment does not have a Secondary Administration node, configure a Secondary Administration node and retry the upgrade.

• If the Secondary Administration node is down, bring up the node and retry the upgrade.

• If the Secondary Administration node is upgraded and moved to the upgraded deployment, use the CLI to manually upgrade the other nodes in the deployment.

3. Upgrade status for a node has not changed: 

• If the upgrade status does not change for a long time (and remains at 80%) in the GUI, check the upgrade logs from the CLI or the status of the upgrade from the console.
• Log in to the CLI or view the console of the Cisco ISE node in order to view the progress of the upgrade. Use the show logging application command in order to view the upgrade-uibackend-cliconsole.log and upgrade-postosupgrade-yyyymmdd-xxxxxx.log.
• View these upgrade logs from the CLI with the show logging application command: DB Data Upgrade Log, DB Schema Log and Post OS Upgrade Log.

4. Roll back to the previous version of ISO image: