AI Threshold Recommendation Fails with External Search Command Error - ITSI 4.21.*

Applicability

Description

Under the ITSI KPI, when using the Threshold AI recommendations, 
If the page returns the following error, this may be because of an app version conflict.

"Error in 'recommendthresholdtemplate' command: External search command exited unexpectedly with non-zero error code 1."


The AI threshold feature in ITSI requires additional apps like : "Splunk AI Toolkit" app (also named "Machine learning Toolkit", MLTK), and Python for Scientific Computing (also named SA for Scientific Python, or PSC) apps.

ITSI 4.21.* supports the following major versions :

• Splunk AI Toolkit / Splunk_ML_Toolkit version 5.6.*  - not the latest 5.7.* (as of 2026-04-27)
• Splunk SA Scientific Python (one for linux, macos, windows) version 4.2.*   (as of 2026-04-27)

See documentation on version compatibility matrix.

Investigation

Check what versions of the apps on the ITSI search-head. Verify if you have the apps, and their versions. If you see too recent version.

- in the UI go to apps > manage.
- in the ITSI search-head in the apps folder under default/app.conf

Resolution

Main solution : 
Wait for ITSI 5.* to be released, and upgrade ITSI.
Check ITSI versions on SplunkBase.

Possible workaround for ITSI 4.21.* . If you want to use the feature on ITSI 4.21.*, the option is to downgrade the 2 apps

• Splunkl AI toolkit / Splunk_ML_Toolkit to version 5.6.* 
• Splunk SA for Scientific Python (several apps, for each  server OS) to version 4.2.*
  
  Exceptions : That solution may not be available in SplunkCloud if the stack has both "Enterprise Security' (ES) and the "IT Service Intelligence" (ITSI) premium apps. In this case the MLTK and PSC apps need to stay on the same versions on all SpunkCloud Search-Heads. Depending of versions, it may not be authorized to downgrade apps.

If you are allowed to downgrade. As the UI will not allow you to downgrade, instead :

1. On Splunk search-head on premises 
   1. Go to splunkbase, or use the links above, locate the apps
   2. Go to "version history tab", find the older installers. Preferably the latest patch of the supported version.
   3. Uninstall the apps form the ITSI search-head
   4. Install the older apps, in the apps folder and restart.
2. in SplunkCloud
   1. Go to splunk.com support portal
   2. Open a SplunkCloud case, for change request, and ask for the app's downgrade, provide a change maintenance window
   3. Wait for the change to be scheduled.
3. Login and validate that ITSI KPI threshold are working correctly.