MC incidents are not generated from Splunk ES Correlation searches

Available Languages

Updated:September 29, 2026
Document ID:SPLK046731

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Applicability

Splunk Enterprise Security

Description

In the Adaptive Response Action section, Correlation searches are configured with the notables and MC incident generation action.

The Correlation search is executed every 30 minutes. However, the notables are generated, but the MC incidents are not.

 


Investigation

In the splunkd.log, it was found the below error regarding the "sendalert" command.

ERROR SearchProcessRunner [2047265 PreforkedSearchesManager-0] - preforked process=0/1017 with search=0/2807 and cmd=splunkd\x00search\x00--id=1715296823.5427\x00--maxbuckets=0\x00--ttl=600\x00--maxout=500000\x00--maxtime=8640000\x00--lookups=1\x00--reduce_freq=10\x00--user=testuser\x00--pro\x00--roles=can_delete:ess_admin:ess_analyst:ess_user:mc_analyst:mc_analyst_edit_default:mc_analyst_read_default:mc_analyst_reader:mc_soar_observer:power:sc_admin:tokens_auth:user\x00--sslclientsession=\x00\x00\x00 died on exception (exit_code=111): Error in 'sendalert' command: Alert script returned error code 5.

 

In the mc_utils.log, the below error was found.

message: Exception occurred in getting files in the file queue path, exc - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/missioncontrol/local/im_indicators'

Based on the above message, some exceptions occurred in the queue path.

In the hot and warm buckets in the below-mentioned directory,

Hot bucket directory - /opt/splunk/var/run/splunk/mc_corr_srch_results_metadata/hot Warm bucket directory - /opt/splunk/var/run/splunk/mc_corr_srch_results_metadata/warm

    Files in hot bucket - 96k files.
    Files in warm bucket - 194680 files.

    The large number of files in the buckets blocks the latest files and the generated notables.


    Resolution

    These files must be removed to resolve this issue as they blocked the queue to process the latest data.

    To remove those files from that directory, the following steps must be followed.

    Navigate to the instance using the ssh.

    To delete the hot bucket files, To remove the files:

    • cd /opt/splunk/var/run/splunk/mc_corr_srch_results_metadata/hot
       
      rm -r *
    • To verify the files use:
       
      ls -ltr

    To delete the warm bucket files,

    • cd /opt/splunk/var/run/splunk/mc_corr_srch_results_metadata/warm
       
      rm -r *
    • To verify the files use:
       
      ls -ltr

    After the removal of the files, restart the Splunk service.

    After this action, the MC incidents will get triggered and generated on the Incident Review page as shown below,

    Revision History

    Revision Publish Date Comments
    1.0
    29-Sep-2026
    Initial Release

    Was this Document Helpful?

    FeedbackFeedback

    Contact Cisco

    Related Articles