ITSI Rules Engine is failing to connect to NATS due to a post-upgrade SSL/TLS certificate validation failure causing up to 20 minute of delay in Episode creation.

Available Languages

Updated:October 1, 2026

Document ID:SPLK055116

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Applicability

Product: Splunk IT Service Intelligence (ITSI) Affected Versions: ITSI 4.20.0, 4.21.0, 4.21.1, 4.21.2 Deployment Type: Splunk Cloud and On-Premises (Standalone Search Head or Search Head Cluster) Component: Event Analytics (EA) / Rules Engine in Queue Mode (NATS) Configuration: TLS/mTLS enabled for NATS (require_tls_client_cert_cloud = 1 or require_tls_client_cert_on_prem = 1)

Description

In ITSI versions 4.20.x through 4.21.2, the internal certificate generation script (itsi_nats_tls_helper.py) self-signs the local root CA (ca-cert.pem) using openssl x509 -req -signkey without passing the required X.509 v3 extension flags.

Missing Basic Constraints: The generated ca-cert.pem is created as an X.509 v1 certificate lacking basicConstraints=critical,CA:TRUE and keyUsage=critical,keyCertSign,cRLSign.
Validation Failure: Go's TLS stack (used by the NATS server) and Java 17 PKIX validation (used by the Rules Engine) strictly reject certificates signed by an authority missing the CA:TRUE constraint (OpenSSL Error 79: invalid CA certificate).
Regeneration Ineffective on 4.21.x: Manually deleting the certificates on 4.21.x versions causes the helper script to re-create the exact same non-compliant CA certificate.

Observed Symptoms

Delayed Episode Creation: Notable events are delayed by 15 to 25+ minutes before appearing as episodes in ITSI Episode Review.
Delayed Incident Generation: ServiceNow ticket creation and operational incident response workflows are blocked or delayed.
Delayed KV Store Access: Entity and service information from the KV Store is slow to update and is not reflected in real time.
Fallback Processing: Notable events are only processed during periodic backfill rather than in real time via the NATS message queue.

Investigation

Run the following checks on the Search Head (or each node in a Search Head Cluster):

Step 1: Check the CA Certificate Basic Constraints

Run the following OpenSSL command:

openssl x509 -in $SPLUNK_HOME/etc/auth/nats/ca-cert.pem -noout -text | grep -A1 "Basic Constraints"

Defective Result: Command returns no output, or CA:TRUE is missing.

Expected Result: Output displays:

X509v3 Basic Constraints: critical
    CA:TRUE

Step 2: Validate Certificate Chain Verification

Verify the local client certificate against the CA:

openssl verify -CAfile $SPLUNK_HOME/etc/auth/nats/ca-cert.pem $SPLUNK_HOME/etc/auth/nats/client-cert.pem

Failure Output: error 79 at 1 depth lookup: invalid CA certificate

Step 3: Check NATS Queue Ingestion Metrics

In Splunk Web, navigate to Dashboards → Event Analytics Monitoring → NATS Monitoring.
Check if Messages In is incrementing while Messages Out remains flat, and Pending Messages is continuously climbing.

Error Messages in Logs

Java Rules Engine Consumer Logs

index="_internal" source="*rules*" EventQueueProcessingActor

ERROR [rulesengine] [pool-454-thread-1] EventQueueProcessingActor:354 - Status=Failed, ErrorMessage="NATS exception: javax.net.ssl.SSLHandshakeException: (certificate_unknown) PKIX path validation failed: sun.security.validator.ValidatorException: TrustAnchor with subject \"CN=Splunk CA, OU=Splunk, O=Splunk, L=NY, ST=CA, C=US\" is not a CA certificate"

NATS Server Logs

$SPLUNK_HOME/var/log/splunk/itsi-nats-server.log

[ERR] TLS handshake error: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "x509: invalid signature: parent certificate cannot sign this kind of certificate" while trying to verify candidate authority certificate "Splunk CA")
[ERR] TLS handshake error: tls: received record with version 301 when expecting version 303
[ERR] TLS route handshake error: tls: client didn't provide a certificate

Resolution

Permanent Resolution: Upgrade ITSI

Upgrade Splunk IT Service Intelligence to ITSI 4.21.3, ITSI 5.0.0, or higher.

These releases include bug fix and introduce the NatsLegacyCACertExtensionMigration upgrade handler.
The migration handler automatically detects legacy, non-compliant CA certificates, cleans them up, and regenerates compliant X.509 v3 CA certificates during the upgrade process without manual intervention.

Workaround 1: Temporarily Switch Rules Engine to Real-Time Search Mode

If certificate replacement cannot be performed immediately, bypass the NATS queue by falling back to Real-Time (RT) search mode:

Via Splunk Web UI:
1. Navigate to Configuration → Configuration Assistant → Advanced ITSI configurations → Rules engine queue mode.
2. Toggle Queue Mode to Disabled. This re-enables the real-time indexed search (itsi_event_grouping).

SPL Search Command:

| itsichangerulesengineprocess is_disable_all=false is_use_queue_mode=false is_use_adhoc_search=false is_use_rt_search=true

Workaround 2: Temporarily Disable TLS for NATS

If operating in an isolated internal/test environment where unencrypted traffic is permissible:

In $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_nats.conf:

[nats_settings]
require_tls_client_cert_cloud = 0
require_tls_client_cert_on_prem = 0

In $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_rules_engine.properties:

nats_require_tls_client_cert_cloud = false
nats_require_tls_client_cert_on_prem = false

Remove $SPLUNK_HOME/etc/apps/SA-ITOA/bin/nats/nats-js.conf and restart Splunk.

Workaround 3: Manually Regenerate and Distribute Valid Certificates (Without Upgrading)

If an immediate upgrade is not feasible, generate compliant certificates manually and distribute them across all Search Heads:

1. Stop Splunk and Backup Existing NATS Certificates

On all Search Heads:

$SPLUNK_HOME/bin/splunk stop

cp -r $SPLUNK_HOME/etc/auth/nats/ /tmp/nats-backup/

2. Generate a Valid CA Certificate (Run on ONE Search Head only)

cd $SPLUNK_HOME/etc/auth/nats/

openssl req -x509 -new -nodes -newkey rsa:2048 -keyout ca-cert.key -out ca-cert.pem -days 365 -subj "/C=US/ST=CA/L=NY/O=Splunk/OU=Splunk/CN=Splunk CA" -addext "basicConstraints=critical,CA:TRUE" -addext "keyUsage=critical,keyCertSign,cRLSign"

3. Distribute the CA Certificate

Copy the generated ca-cert.key and ca-cert.pem to $SPLUNK_HOME/etc/auth/nats/ on all other Search Heads in the cluster to maintain a shared trust anchor.

4. Generate a New Client Certificate on EACH Search Head

Run these commands on every Search Head individually:

cd $SPLUNK_HOME/etc/auth/nats/

openssl req -new -nodes -newkey rsa:2048 -keyout client-cert.key -out client-cert.csr -subj "/C=US/ST=CA/L=NY/O=Splunk/OU=Splunk/CN=$(hostname)"

openssl x509 -req -in client-cert.csr -CA ca-cert.pem -CAkey ca-cert.key -CAcreateserial -out client-cert.pem -days 365 -extfile <(printf "extendedKeyUsage=serverAuth,clientAuth\nsubjectAltName=IP:127.0.0.1,DNS:localhost,DNS:$(hostname)")

# Convert the private key to unencrypted PKCS#8 format for NATS

openssl pkcs8 -topk8 -nocrypt -in client-cert.key -out client-key.pem

# Clean up intermediate files

rm -f client-cert.csr ca-cert.srl client-cert.key

5. Remove Stale Configuration and Start Splunk

On all Search Heads:

rm -f $SPLUNK_HOME/etc/apps/SA-ITOA/bin/nats/nats-js.conf

$SPLUNK_HOME/bin/splunk start

Post-Resolution Verification

Verify Log Stream: Check $SPLUNK_HOME/var/log/splunk/itsi-nats-server.log and confirm there are no TLS handshake error or PKIX path validation entries.
Verify Dashboard Metrics: Navigate to Dashboards → Event Analytics Monitoring → NATS Monitoring and confirm Messages In matches Messages Out with Pending Messages returning to 0.
Verify Episode Review: Check Episode Review to confirm notable events group into episodes within real-time thresholds (< 1 minute).