Microsoft O365 getting Error "certificate verify failed: unable to get local issuer certificate (_ssl.c:1161)".

Applicability

Description

Following a Splunk upgrade, the Splunk instance running the Splunk Add-on for Microsoft Office 365 (splunk_ta_o365) fails to execute configured inputs. The modular input management_activity throws certificate-related errors, and data collection from Microsoft services halts.

Investigation

The internal logs show the following error:

2025-07-09 07:44:42,580 level=ERROR pid=649434 tid=MainThread logger=splunk_ta_o365.modinputs.management_activity pos=utils.py:wrapper:72 | datainput=b'O365_Sharepoint_events' start_time=1752061481 | message="Data input was interrupted by an unhandled exception." 
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/urllib3/connectionpool.py", line 700, in urlopen
    self._prepare_proxy(conn)
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/urllib3/connectionpool.py", line 996, in _prepare_proxy
    conn.connect()
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/opt/splunk/lib/python3.9/ssl.py", line 506, in wrap_socket
    return self.sslsocket_class._create(
  File "/opt/splunk/lib/python3.9/ssl.py", line 1082, in _create
    self.do_handshake()
  File "/opt/splunk/lib/python3.9/ssl.py", line 1351, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1161)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='manage.office.com', port=443): Max retries exceeded with url: /api/v1.0/4cb791f1-02d1-4a4e-8610-911ee57bb08c/activity/feed/subscriptions/list?PublisherIdentifier=4cb791f1-02d1-4a4e-8610-911ee57bb08c (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1161)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/splunksdc/utils.py", line 70, in wrapper
    return func(*args, **kwargs)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 201, in run
    executor.run(adapter)
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/splunksdc/batch.py", line 54, in run
    for jobs in delegate.discover():
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 690, in discover
    session = self._get_session()
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 361, in _get_session
    self._enable_subscription(session)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 371, in _enable_subscription
    if not self._subscription.is_enabled(session):
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 368, in is_enabled
    response = self._perform(session, "GET", "/subscriptions/list")
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 419, in _perform
    return self._request(session, method, url, kwargs)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 448, in _request
    response = session.request(
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/requests/adapters.py", line 563, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='manage.office.com', port=443): Max retries exceeded with url: /api/v1.0/4cb791f1-02d1-4a4e-8610-911ee57bb08c/activity/feed/subscriptions/list?PublisherIdentifier=4cb791f1-02d1-4a4e-8610-911ee57bb08c (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1161)')))

The SSL environment is likely reset or altered during the upgrade process, removing or invalidating the local certificate configuration. As a result, the Python-based libraries fail to validate the remote server’s certificate chain, causing the input to terminate unexpectedly.

Resolution

Merging system/custom CA with the CA bundle used by the O365 TA resolves the issue.

1. Identify the CA bundle required to validate the Microsoft Office 365 endpoint certificate.

2. Merge the necessary root and intermediate certificates with the CA bundle used by the Splunk Add-on for Microsoft Office 365.

3. (For version 4.6.0 and above):
   Define the path to the custom CA bundle by creating a configuration file named splunk_ta_o365_ssl_cert.conf in the $SPLUNK_HOME/etc/apps/splunk_ta_o365/local/ directory.

   Add the following configuration:

   [sslCertSettings]
   ca_certs_path = <path_to_ca_certs>
   

4. Ensure that this configuration is placed only in the local directory, not in default.

5. Restart Splunk to apply the changes.