MLTK Searches Fail With: "Model does not exist message".

Applicability

Splunk Enterprise

Description

Splunk version 8.0.3 (build a6754d8441bf) 
Splunk Enterprise Security 6.1.1
Splunk Machine Learning Toolkit, version = 5.2.0

 

Issue:
Correlation searches that reference MLTK model fail with the following error:

"Failed to load model"

 

sample search:

| tstats `summariesonly` sum(Web.bytes) as bytes from datamodel=Web.Web 
where (Web.http_method="POST" OR Web.http_method="PUT") NOT 
(`cim_corporate_web_domain_search("Web.url")`) by Web.user, Web.user_bunit 
| `drop_dm_object_name("Web")` 
| `mltk_apply_upper("app:web_upload_to_non_corporate_by_user_1h", "medium", "bytes")`

 

sample log .../dispatch/1593011576.10160/search.log

INFO  SearchParser - PARSING: | tstats `summariesonly` sum(Web.bytes) as bytes from 
datamodel=Web.Web where (Web.http_method="POST" OR Web.http_method="PUT")  NOT 
(`cim_corporate_web_domain_search("Web.url")`) by Web.user, Web.user_bunit | 
`drop_dm_object_name("Web")` | `mltk_apply_upper("app:web_upload_to_non_corporate_by_user_1h"
"medium", "bytes")`...INFO  SearchPhaseGenerator - Failed to create phases using AST:Failed to create the AST for 
search | tstats `summariesonly` sum(Web.bytes) as bytes from datamodel=Web.Web where 
(Web.http_method="POST" OR Web.http_method="PUT")  NOT 
(`cim_corporate_web_domain_search("Web.url")`) by Web.user, Web.user_bunit | 
`drop_dm_object_name("Web")` | `mltk_apply_upper("app:web_upload_to_non_corporate_by_user_1h"
"medium", "bytes")`. Falling back to 2 phase mode.INFO ChunkedExternProcessor - Running process: 
"E:\Program Files\Splunk\SplunkEnterprise\bin\Python3.exe" 
"E:\Program Files\Splunk\SplunkEnterprise\etc\apps\Splunk_ML_Toolkit\bin\apply.py"ERROR ChunkedExternProcessor - Error in 'apply' command: Failed to load model 
"web_upload_to_non_corporate_by_user_1h": Model does not exist.ERROR SearchPhaseGenerator - Fallback to two phase search failed:Error in 'apply' 
command: Failed to load model "web_upload_to_non_corporate_by_user_1h": Model does not 
exist...

 

– search logs ens with the following error message —

ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: Error in 'apply' 
command: Failed to load model "web_upload_to_non_corporate_by_user_1h": Model does not exis

Investigation

The problem is in "related correlation search" - It was not producing result.
 

[Identity - Web Uploads to Non-corporate Domains by Users Per 1d - Model Gen]
| tstats `summariesonly` sum(Web.bytes) as bytes from datamodel=Web.Web where 
NOT(`cim_corporate_web_domain_search("Web.url")`) (Web.http_method="POST" OR 
Web.http_method="PUT") 
by _time, Web.user, Web.user_bunit span=1h 
| `drop_dm_object_name("Web")` 
| fit DensityFunction bytes by user_bunit dist=norm into 
app:web_upload_to_non_corporate_by_user_1h

 

What causes search failure?

Search fails due to missing data in "bytes" field (Web.bytes). It is specific to their Palo Alto Networks integration.

PAN logs do not include bytes for web traffic, hence Web datamodel has no bytes data in it.
 

Resolution

Contact app vendor.

Possible solutions are non Splunk related.

Solution documented here. 
https://splunk.paloaltonetworks.com/log-correlation.html Each example will correlate traffic logs and url logs to determine how many bytes have been transferred between each FQDN in the time period. The URL log has a dest_name field with the FQDN and the Traffic log has a bytes_out field, so we need to correlate them to know how many bytes went out for each FQDN. The goal is to visualize possible data exfiltration by showing the total bytes_out for each FQDN1. 
Correlation technique 1: Use a 'join' or 'transaction'
sourcetype="pan:threat" log_subtype="url"
  | join type=left session_id [ search sourcetype="pan:traffic" log_subtype="end" ]
  | stats count, sum(bytes_in), sum(bytes_out) BY dest_name
  | sort -sum(bytes_out)
 
2.
Correlation technique 2: Stats correlation(sourcetype="pan:threat" AND log_subtype="url") OR (sourcetype="pan:traffic" AND log_subtype="end")
  | stats values(sourcetype), values(dest_name) AS dest_name, count, sum(bytes_in) AS bytes_in, sum(bytes_out) AS bytes_out BY session_id
  | stats sum(count) AS count, sum(bytes_in), sum(bytes_out) BY dest_name
  | sort -sum(bytes_out)

 

Related Links

External Documentation:
Log Correlation
https://splunk.paloaltonetworks.com/log-correlation.html

 

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release