ERROR : sendemail:1668 - Splunkd daemon is not responding

Applicability

Description

The alerts are scheduled for every hour with alert action as emails and the alerts are being triggered but it is not getting sent as emails.

08-30-2024 14:39:11.596 +0800 INFO  SavedSplunker - savedsearch_id="", search_type=<"">, search_streaming=0, user=<"">, app="analog_silicon_analysis", savedsearch_name=<>, priority=default, status=success, digest_mode=1, durable_cursor=0, scheduled_time=1724999940, window_time=0, dispatch_time=1724999941, run_time=9.824, result_count=0, alert_actions="", sid=<>, suppressed=0, thread_id="AlertNotifierWorker-0", workload_pool=""

Investigation

Upon investigation, observed the below errors in the splunkd.log:

ERROR ScriptRunner [271749 AlertNotifierWorker-1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/search/bin/sendemail.py "results_link=<> sid=scheduler__admin_cHVibGljX3Jlc291cmNl__RMD54f5021f61fb80dd3_at_1724997000_14446_2D86E62A-9636-41C8-88D9-DB416CDB1A22" "ssname=[RA] getResourceUsageByProjHostgroups" "graceful=True" "trigger_time=1724997016" results_file="/opt/splunk/var/run/splunk/dispatch/scheduler__admin_cHVibGljX3Jlc291cmNl__RMD54f5021f61fb80dd3_at_1724997000_14446_2D86E62A-9636-41C8-88D9-DB416CDB1A22/results.csv.gz" "is_stream_malert=False"': ERROR:root:(552, b'5.3.4 Message size exceeds fixed maximum message size', 'splunk@abcsplksh23') while sending mail to: abc@gmail.com

- This indicates that there is a limitation in sending alerts as emails with higher number of events. In this case, exported the result as a CSV instead of email to avoid this problem. 

- However, observed that the alerts with less number of events were still not getting triggered as email.

- In /opt/splunk/var/log/splunk/python.log, the below error is populating :

ERROR sendemail:1668 - Splunkd daemon is not responding: ('Error connecting to /servicesNS/abcsplksh23/arftool/saved/searches/%5BMMSIM%5D%20Max%20Job%20Violation%20Alert: The read operation timed out',)

Resolution

This error message indicates a timeout issue. 

To resolve the issue, follow the below steps :

i. Navigate to $SPLUNK_HOME/etc/system/local/web.conf

ii. Increase the timeout value 

splunkdConnectionTimeout = <integer> (default=30)

* The amount of time, in seconds, to wait before timing out when communicating with
splunkd.
* Must be at least 30.
* Values smaller than 30 will be ignored, resulting in the use of the
default value
* Default: 30

After increasing the timeout value, the emails are getting triggered as expected.

Related Links

web.conf