diag/rapiddiag upload failing with "socket.timeout: The read operation timed out"

Applicability

Description

When trying to upload a diag or rapiddiag from the command line the upload may fail at some point with a read timeout.
Although this may also be caused by other external factors such as network issues etc, this may also be due to the hardcoded timeout within Splunk being set too low for larger file uploads and slower/busier networks.


 

Investigation

The following error is reported from the command line during the upload :

Uploading chunk '1' of '1'.
----------------------------------------------> |81%Unexpected exception while making http requestTraceback (most recent call last):
  File "/opt/splunk/lib/python3.7/site-packages/splunk/clilib/info_gather.py", line 5593, in make_splunkcom_request
    rest_f = urllib_request.urlopen(req, timeout=15)
  File "/opt/splunk/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/opt/splunk/lib/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/opt/splunk/lib/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/opt/splunk/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/opt/splunk/lib/python3.7/urllib/request.py", line 1393, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/opt/splunk/lib/python3.7/urllib/request.py", line 1353, in do_open
    r = h.getresponse()
  File "/opt/splunk/lib/python3.7/http/client.py", line 1373, in getresponse
    response.begin()
  File "/opt/splunk/lib/python3.7/http/client.py", line 319, in begin
    version, status, reason = self._read_status()
  File "/opt/splunk/lib/python3.7/http/client.py", line 280, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/opt/splunk/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/opt/splunk/lib/python3.7/ssl.py", line 1071, in recv_into
    return self.read(nbytes, buffer)
  File "/opt/splunk/lib/python3.7/ssl.py", line 929, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
ERROR: Unexpected exception while making http request
Traceback (most recent call last):
  File "/opt/splunk/lib/python3.7/site-packages/splunk/clilib/info_gather.py", line 5593, in make_splunkcom_request
    rest_f = urllib_request.urlopen(req, timeout=15)
  File "/opt/splunk/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/opt/splunk/lib/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/opt/splunk/lib/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/opt/splunk/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/opt/splunk/lib/python3.7/urllib/request.py", line 1393, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/opt/splunk/lib/python3.7/urllib/request.py", line 1353, in do_open
    r = h.getresponse()
  File "/opt/splunk/lib/python3.7/http/client.py", line 1373, in getresponse
    response.begin()
  File "/opt/splunk/lib/python3.7/http/client.py", line 319, in begin
    version, status, reason = self._read_status()
  File "/opt/splunk/lib/python3.7/http/client.py", line 280, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
 File "/opt/splunk/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/opt/splunk/lib/python3.7/ssl.py", line 1071, in recv_into
    return self.read(nbytes, buffer)
  File "/opt/splunk/lib/python3.7/ssl.py", line 929, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

Resolution

This is a known issue and a fix for this will be implemented in a future Splunk release.
For now the following workaround can be used :

Edit the file info_gather.py (make a backup first)

File "/opt/splunk/lib/python3.7/site-packages/splunk/clilib/info_gather.py", line 5593, in make_splunkcom_request 66 67 rest_f = urllib_request.urlopen(req, timeout=15)
default timeout=15 - seconds

Try changing this to 30 or more if needed