Upgrade Failed from UBA 5.3.0 to 5.4.0 - Sha256DbMigration Error in User behavior Analytics(UBA).

Applicability

Description

When attempting to upgrade Splunk UBA from version 5.3.0 to version 5.4.0, the upgrade process fails at the database migration step with the below error 

Error message:

Upgrade failed in step: Sha256DbMigration, aborting.

Due to this failure the UBA environment in a non-operational state causing a complete service outage.

Investigation

During investigation, it was found that the upgrade failed specifically at the "Sha256DbMigration" step, which is part of the database schema migration process when upgrading to UBA 5.4.0. This migration step is designed to update the Output Connector configuration in the PostgreSQL database to use SHA256 hashing for improved security.

The root cause is not definitively established, but the issue appears to be related to the Output Connector's SSL configuration conflicting with the migration process. When "connectors.output.splunkes.ssl= true" is set in the uba-site.properties file, the database migration encounters issues that cause it to fail.

Review of the outputconnector.log may provide additional insights into the specific failure mechanism, but the consistent resolution involves temporarily disabling the Output Connector's SSL and ES integration settings during the upgrade process.

This issue is specific to environments mentioned below:

1. UBA is integrated with Splunk ES via Output Connector
2. SSL is enabled for the Splunk ES connection
3. The Sha256DbMigration step needs to modify Output Connector database records

Resolution

To resolve this upgrade failure and successfully complete the upgrade from UBA 5.3.0 to 5.4.0, follow these steps:

Step 1: Identify Current Output Connector Configuration

Before making changes, verify the current configuration:

grep -E "connectors.output.splunkes.ssl|uba.splunkes.integration.enabled" /etc/caspida/local/conf/uba-site.properties

Expected current state (before workaround):

 
Step 2: Disable Output Connector SSL and ES Integration

Edit the UBA configuration file:

sudo vi /etc/caspida/local/conf/uba-site.properties

Modify the below two lines:

connectors.output.splunkes.ssl=false
uba.splunkes.integration.enabled=false

Step 3: Execute the UBA upgrade script:

/opt/caspida/upgrade/utils/upgrade_uba.sh -p /var/vcap/release_archives/caspida-<old-uba-version> -e /home/caspida/uba-ext-pkgs-5.3.0.tgz

After executing the upgrade script the upgrade will be succesfull.

Expected output:
 

After executing the upgrade script, the upgrade will be successful.

After the UBA upgrade has been successfully completed, Re-enable Output Connector SSL and ES Integration

Once UBA 5.4.0 upgrade is confirmed successful, restore the original Output Connector configuration:

sudo vi /etc/caspida/local/conf/uba-site.properties

Modify the settings back to:

connectors.output.splunkes.ssl=true
uba.splunkes.integration.enabled=true

Step 4: Restart Output Connector

/opt/caspida/bin/Caspida stop outputconnector
/opt/caspida/bin/Caspida start outputconnector

Please note: This issue has been identified in UBA version 5.3.0 and has been fixed in UBA version 5.4.0 and later. The issue will not occur when upgrading from UBA 5.4.0 to future versions.

Related Links

Document Link: Reference