Exporting the PDF for the Dashboard is Getting Failed with "504 Gateway time-out" Error.

Applicability

Description

Environment:
-OS: Linux.
-Search head cluster - Splunk version - 7.3.0.
-Standalone search head - 7.3.0.

Issue:
The pdf export of the dashboard results with more results(>5 K) is getting errored out with the message - "504 Gateway time-out".

The following configuration was tested in limits.conf, it didn't work:

max_rows_per_table in [pdf] to 220000
max_mem_usage_mb = 2048
render_endpoint_timeout = 3600

Every time, the pdf export doesn't work after 1 minute like the observation.
 

Investigation

The below errors were located  in pdfgen.log:
 

WARN HttpListener - Socket error from xxx.xx.xxx.xxx:53820 while accessing 
/en-US/splunkd/__raw/services/pdfgen/render: Broken pipe
WARN HttpListener - Socket error from xxx.xx.xxx.xxx:44638 while accessing 
/en-US/splunkd/__raw/services/pdfgen/render: Broken pipe

From the  above errors, the following changes are applied:

1. In web.conf, increase the below timeout parameters:

[settings]
SplunkdConnectionTimeout = <integer> (Increase to 5 min)
server.socket_timeout = <integer> ( Increase the value to 1-5 min)
busyKeepAliveIdleTimeout = <integer> (Increase the value to 1-5 min)

Restart the splunkd process.

2. As the above step didn't work, requested the below changes -

In limits.conf set:

[pdf]
render_endpoint_timeout = 7200

Restart the splunkd process.

And in the etc/system/bin/pdfgen_ endpoint.py change:

DEFAULT_TIMEOUT = 7200

Restart the splunk service. Even it didn't work.

3. As per the observation, noticed that only 1000-1100 records per minute are getting exported. So, updated the LB timeout setting accordingly i.e. greater than 10 min based on the results exporting which resolved the issue.
 

Resolution

Able to export the desired or large results after updating the Load balancer timeout setting (>10 min)