Issue
Attempts to install the Upgrade Readiness Tool (URT) bundle ise-urtbundle-3.4.0.608b-1.0.0.SPA.x86_64.tar.gz on Cisco Identity Services Engine (ISE) version 3.4 failed. After the failed installation, subsequent attempts to uninstall or reinstall the URT bundle were unsuccessful, and ISE services could not be stopped due to a persistent application process lock (APP_INSTALL). The CLI returned error messages indicating that another application install or upgrade was in progress, effectively locking the ISE database and preventing further management operations.
Environment
- Product: Cisco Identity Services Engine (ISE)
- Version: 3.4.0
- Subtechnology: ISE Upgrade/Patch/Licensing
- URT bundle: ise-urtbundle-3.4.0.608b-1.0.0.SPA.x86_64.tar.gz (version 1.0.0, 52 days old)
- Deployment: Multi-node ISE on Azure VM (Secondary Admin, Primary Monitoring Node)
- CLI installation attempted via:
application install ise-urtbundle-3.4.0.608b-1.0.0.SPA.x86_64.tar.gz NETFTP
- Azure cloud VM environments do not support native ISE upgrade processes
- No recent successful upgrades or installations prior to the URT bundle attempt
Resolution
This detailed workflow outlines the steps required to identify and resolve the database lock resulting from a failed URT bundle installation in Cisco ISE 3.4.
Step 1: Attempt Standard Application Removal and Service Stop
Begin by attempting to remove the URT application and stop the Cisco ISE services using the standard CLI commands. This step confirms the lock state and produces error messages necessary for diagnosis.
Command to remove URT application:
application remove urt
Continue with application removal? (y/n) [n] ? y
% An existing application install, remove, or upgrade is in progress. Try again shortly.
Command to stop ISE services:
application stop ise
Waiting up to 20 seconds for lock: APP_INSTALL
APP_INSTALL to complete
Database is still locked by lock: APP_INSTALL
APP_INSTALL. Aborting. Please try it later
% Error: Another ISE DB process (APP_INSTALL APP_INSTALL) is in progress, cannot perform Application Stop at this time
If you receive messages indicating that another install, remove, or upgrade is in progress, proceed to the next steps for advanced troubleshooting.
Step 2: Identify the Database Lock File
Access the node with root-level privileges via CLI. Navigate to the temporary directory where process locks are stored.
Command to list contents of /temp/ directory:
ls /temp/
Look for a file named ise_db_lock or similar. This file is responsible for maintaining the database lock and preventing service operations.
Step 3: Remove the Stale Database Lock File
Once the lock file is identified, remove it to clear the persistent lock condition.
Command to remove the lock file:
rm /temp/ise_db_lock
This action releases the database and allow further management operations.
Step 4: Stop and Restart ISE Services
After removing the lock file, proceed to stop and then restart the Cisco ISE services to ensure all processes are reset and running as expected.
Command to stop Cisco ISE services:
application stop ise
Command to start Cisco ISE services:
application start ise
Verify that no error messages related to APP_INSTALL are displayed and that services stop and start successfully.
Step 5: Validate Running Status of ISE Services
Check the operational status of all Cisco ISE processes to confirm that they are running normally and no locks persist.
Command to check running status:
show application status ise
Example output:
ISE PROCESS NAME STATE PROCESS ID
--------------------------------------------------------------------
Database Listener running 4056
Database Server running 132 PROCESSES
Application Server running 9481
Profiler Database running 9774
ISE Elasticsearch running 24973
AD Connector running 35580
M&T Session Database running 7838
M&T Log Processor running 38134
ISE Messaging Service running 10373
ISE API Gateway Database Service running 10825
ISE API Gateway Service running 23058
ISE pxGrid Direct Service running 67962
ISE pxGrid Direct Pusher running 68973
Segmentation Policy Service running 39231
REST Auth Service running 42849
SSE Connector disabled
Hermes (pxGrid Cloud Agent) disabled
MFA (Duo Sync Service) running 44767
McTrust (Meraki Sync Service) disabled
aciconn (ACI Connection Service) disabled
Workload Connector Service disabled
ISE Prometheus Service running 62697
ISE Prometheus Exporter running 59234
ISE Grafana Service running 32873
ISE MNT LogAnalytics Elasticsearch disabled
ISE Logstash Service disabled
ISE Kibana Service disabled
ISE Native IPSec Service running 10210
MFC Profiler running 46329
ISE Prometheus Alertmanager Service running 48962
Protocols Engine running 60381
All main Cisco ISE services need to report as "Running."
Step 6: Validate Node Synchronization in GUI
- Log into the Cisco ISE graphical user interface (GUI).
- Navigate to: Administration > System > Deployment.
- Ensure that the node synchronization status is correct and that all nodes in the multi-node deployment are healthy. This confirms that the lock condition has not impacted node communications.
Cause
The root cause of the issue was the presence of a stale database lock file (ise_db_lock) in the /temp/ directory. This lock file was created during the failed installation of the URT bundle and was not automatically removed. The persistent lock prevented the execution of further application install, remove, or upgrade operations and blocked the ability to stop Cisco ISE services via CLI. Additionally, attempting native upgrade operations in an Azure VM environment is unsupported, and Cisco recommends redeployment for upgrades in cloud deployments.
Related Content
Feedback