ES Threat intelligence feeds are failing to download and showing timed out errors

Applicability

Splunk Enterprise Security

Description

The users is observing that a threat intel source configured to download its feed data only once per day is failing and also timing out its subsequent attempts to download the feed based on the retry settings. They have also attempted to run a curl command on the URL to be downloaded and it has worked without any issues. Following errors can also be seen in splunkd logs:

2019-05-14 12:05:17,192+0000 ERROR pid=87151 tid=MainThread file=protocols.py:run:210 | Caught URLError when querying 
https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt: reason=timed out exc=<urlopen error timed out> 2019-05-14 12:05:17,192+0000 ERROR pid=87151 tid=MainThread file=protocols.py:run:220 | 
No content returned when querying https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt

Investigation

1) A search similar to following can also be run to look at threat intel download logs:

index=_internal host={ES_SH_HOST_NAME} source=*threatlist.log* sourcetype=threatintel:download stanza={THREAT_INTEL_SOURCE_NAME}

2) This search may show logs similar to following that provide further insights on download failure(s):

2021-08-11 21:39:39,070+0000 INFO pid=33223 tid=MainThread file=threatlist.py:download_threatlist_file:545 | stanza="{THREAT_SOURCE_NAME}" retries_remaining="0" status="retrying download" retry_interval="60" url="https://raw.githubusercontent.com/mitre/cti/master/enterprise-attack/enterprise-attack.json"2021-08-11 21:40:39,130+0000 ERROR pid=33223 tid=MainThread file=threatlist.py:download_threatlist_file:556 | stanza="{THREAT_SOURCE_NAME}" retries_remaining="-1" status="threat list download failed after multiple retries" url="https://raw.githubusercontent.com/mitre/cti/master/enterprise-attack/enterprise-attack.json"

3) Note the retry_interval="60" in the logs above, which is the currently configured retry interval of 60 seconds. This retry interval could be shorter for larger downloads. Hence, download failures can occur due to timeouts.


 

Resolution

1) Navigate to ES App and then go to Configure > Data Enrichment > Threat Intelligence Management > Sources tab

2) Then click on the source name failing downloads, switch to Advanced tab and change the following field values:
Retries > 6
Retry Interval > 600

3) These values can also be set in the configuration at following: 
$SPLUNK_HOME/etc/apps/SA-ThreatIntelligence/local/inputs.conf 

[threatlist://{THREAT_SOURCE_NAME}] 
retry_interval = 600 
retries = 6

4) This change will allow more time for the threat intelligence downloads to complete.

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release