Scheduled report with large PDF file is not being sent out

Applicability

Description

Scheduled report which includes PDF file is not sending out when there are large number of search results (more than 20K). Scheduled report can be sent out if CSV is selected to be included instead.

The email sent by PDF delivery indicates that there was an issue generating the PDF file.

The body of the message states: "An error occurred while generating the PDF. Please check the python.log for more details." 

Investigation

To effectively address the issus related to scheduled email deliveries, it is essential to analyze the logs corresponding to the time the email was sent.

1. Examine the scheduler.log to verify if the alert is being generated. Look for entries similar to the following:

index=_internal source=*scheduler.log "Report_Name"
xxxx-xx-xx xx:xx:xx,xxx +xxxx +0000 INFO SavedSplunker - savedsearch_id="xxxxxxxx;search;_ScheduledView__scheduleReportNamexxxxxxxx", search_type="scheduled", search_streaming=0, user="xxxxxxxxxxxxxxxxx", app="search", savedsearch_name="_ScheduledView__xxxxxxxxxxxx, priority=default, status=success, digest_mode=1, durable_cursor=0, scheduled_time=1754031600, window_time=0, dispatch_time=1754031600, run_time=0.053, result_count=0, alert_actions="email", sid="scheduler_dmFuZXNzYS53YWduZXJAcWJleW9uZC5kZQ__search__xxxxxxxxxxxxx", suppressed=0, action_time_ms=1014, thread_id="AlertNotifierWorker-187", workload_pool="standard_perf" 

2. Check the python.log for errors related to PDF generation. You may find entries like: 

xxxx-xx-xx xx:xx:xx,xxx +xxxx ERROR sendemail:1501 - An error occurred while generating a PDF: Failed to fetch PDF (status = '400'): b'<?xml version="1.0" encoding="UTF-8"?>\n<response>\n <messages>\n <msg type="WARN">Could not parse xml reply (no reply from script). See splunkd.log for more info.</msg>\n </messages>\n</response>\n'

3. Connection to splunkd time out while waiting for the PDF to be rendered: 

xxxx-xx-xx xx:xx:xx,xxx +xxxx ERROR     __init__:522 - Socket error communicating with splunkd 
(error=[Errno 111] Connection refused), path = /services/pdfgen/renderxxxx-xx-xx xx:xx:xx,xxx +xxxx ERROR     sendemail:1170 - An error occurred while generating a 
PDF: Failed to fetch PDF (SplunkdConnectionException): Splunkd daemon is not responding: ('Error
connecting to /services/pdfgen/render: [Errno 111] Connection refused',)

Or Warnings regarding script execution timeouts can be found in the logs, indicating that the script has exceeded its maximum execution time:   

xxxx-xx-xx xx:xx:xx,xxx +xxxx WARN ScriptRunner [945949 AlertNotifierWorker-0] - Killing script, probably timed out, grace=5sec, script="/opt/splunk/bin/python3.9 /opt/splunk/etc/apps/search/bin/sendemail.py "results _link=https://xxxxx.splunkcloud.com:443/app/search/@go?sid=scheduler_xxxxxxxxxxxxxxxxxxxx" "ssname=_ScheduledView__reportNamexxxxxx" "graceful =True" "trigger_time=1754474100" results_file="/opt/splunk/var/run/splunk/dispatch/scheduler_xxxxxxxxxxxxxxx/results.srs.zst" "is_stream_malert=False""xxxx-xx-xx xx:xx:xx,xxx +xxxx WARN script [945949 AlertNotifierWorker-0] - Script has timed out and will be killed, maxtime=3600sec, script=/opt/splunk/bin/python3.9 /opt/splunk/etc/apps/search/bin/sendemail.py '"results_link=https://xxx.splunkcloud.com: 443/app/search/@go?sid=scheduler_xxxxxxxxxxxxx"' "ssname=_ScheduledView__reportNamexxxxxx" '"graceful=True"' '"trigger_time=1754474100" 'results_file="/opt/splunk/var/run/splunk/dispatch/scheduler_xxxxxxxx/results.srs.zst"' '"is_stream_malert=False"'

Resolution

1. Increase the max_rows_per_table value for the PDF file in limits.conf:
(Default 1000)

[pdf]
max_rows_per_table = 5000

2. Increase the PDF rendering timeout values in limits.conf:
(Default 3600)

[pdf]
render_endpoint_timeout = 7200

3. Increate the splunkd connection in web.conf:
(Default 30)

[settings]
splunkdConnectionTimeout = 900

4. Restart Splunk service.

Note: To generate PDF with a large number of results, the timeout settings need to be increased to allow rendering to complete. The timeout values depend on the server performance and can be reduced for high performance server that can finish rendering faster.

Refer the following documents for more information : Manage limits.conf configurations using the ACS API