UBA - Failure sending threats alert email, invalid credential
Available Languages
Applicability
UBADescription
After having configured email threats alert sending, recipients don't receive them and this kind of error is traced in the "/var/log/caspida/outputconnector.log" file:2023-01-23 12:13:16,937 INFO OutputConnectorTopic-consumer c.c.c.output.EmailOutputConnector: sendThreatEmail config: EmailConnectorConfig(host=smtp.mail.com, port=587, username=splunk-notification@mail.com, password=****, atomicMails=true, autoProcess=false, maskPii=false) 2023-01-23 12:13:16,967 INFO OutputConnectorTopic-consumer c.c.c.output.EmailOutputConnector: Failed to send email...: java.lang.Exception: Failure to send email. Invalid credentials for account splunk-notification@mail.com (mailto:splunk-notification@mail.com)
Investigation
NOTE: For security reasons all the examples given in this article don't use any information or data coming from the users environment or Splunk Labs.
STEP 1 - Configuration check
Using UBA Console UI to double-check the email configuration settings actually follow what recommended in the Installation Manual.
When TLS port is used to communicate with the SMTP server, storing a certificate in the UBA keystore is not needed.
Nevertheless, if a certificate is stored there with alias "splunk smtp", it must be the same used by the SMTP server to avoid any potential connection issue.
This command returns the SHA256 hash of the certificate used by the SMTP server, it should be compared with the SHA256 of the certificate stored in the UBA key store with alias "splunk smtp" to be sure they are the same:
sudo openssl s_client -starttls smtp -connect smtp.mail.com:587 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -sha256 -in /dev/stdin
Below command returns the info about the certificate stored in the UBA keystore with alias "splunk smtp":
### Setting the variables for the environment . /opt/caspida/bin/CaspidaCommonEnv.sh ### Retrieving the SHA256 hash of the "splunk smtp" certificate stored in the UBA keystore sudo keytool -list -alias "splunk smtp" -keystore $JAVA_HOME/lib/security/cacerts
STEP 2 - Connection test
To test whether a connection between the UBA master cluster node and the SMTP server can be established, execute --> openssl s_client -starttls smtp -connect smtp.mail.com:587 -tls1_2 -crlf
Below you can see what is displayed when a connection has been successfully established:
uba-master-node$ openssl s_client -starttls smtp -connect smtp.mail.com:587 -tls1_2 -crlf
CONNECTED(00000003)
Can't use SSL_get_servername
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
verify return:1
depth=0 C = UK, L = London, O = ACME Ltd, CN = mail.com
verify return:1
---
Certificate chain
0 s:C = UK, L = London, O = ACME Ltd, CN = mail.com
i:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
1 s:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
2 s:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIHvTCCBqWgAwIBAgIQDbA9oWG35agseVa3PnXDijANBgkqhkiG9w0BAQsFADBP
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMSkwJwYDVQQDEyBE
aWdpQ2VydCBUTFMgUlNBIFNIQTI1NiAyMDIwIENBMTAeFw0yMjAyMjgwMDAwMDBa*** SEVERAL OTHER LINES OMITTED BY AUTHOR FOR SECURITY REASONS ***Zf7YTXyrnTfr4xqVvmB9kCsv1YxxTqU5xNdeC+nfdrdVKBbKYddiabqs/WzIN9a2
YuxmmqWKyIjl+SpftI8NuaajrsYvv+7xCXivdSUDzvplpb4lJs7oHSuo72ag1+Y4
/a5xv+NpvSAWWhjNAh+m3E8=
-----END CERTIFICATE-----
subject=C = UK, L =London , O = ACME Ltd, CN = mail.com
issuer=C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5287 bytes and written 318 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 193504EC860C060901C2C7A1CA63C49000AF3B0356329E5B1F6DF16C484B90E3
Session-ID-ctx:
Master-Key: 5A1846FD014871A6E127163B9A4930B85C3234F0C9DCF59A43A609F99749B610E88566BF84065918E1C7B3A032AB7DA7
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
0000 - 2c 7e bc c0 e5 75 99 f0-38 56 66 8e 81 7d be 43 ,~...u..8Vf..}.C
0010 - 4d 18 a1 b5 ff 09 94 78-3c c0 74 80 c0 cf 3d 39 M......x<.t...=9
0020 - 1c c0 16 ef 42 72 ca db-f3 22 e7 e4 6e 63 9e cd ....Br..."..nc..
0030 - bb 14 7a bb 11 9c 81 6a-33 6b 4c 4f 57 41 ad 19 ..z....j3kLOWA..
0040 - 35 2b b3 88 19 75 68 a4-b2 87 73 0e a6 4d e4 49 5+...uh...s..M.I
0050 - 17 83 c5 b4 69 1e 6c a7-7b fa 91 95 84 14 9b a7 ....i.l.{.......
0060 - 17 50 ae a8 af 10 0d 35-8f 6a 4b c8 c9 d3 20 ac .P.....5.jK... .
0070 - 28 19 3e a9 9e 84 e7 fc-7f a7 cf 5e 69 17 81 ff (.>........^i...
0080 - 0e b3 5a 57 7e 60 8d 84-4b 2f a6 52 08 92 2c 6c ..ZW~`..K/.R..,l
0090 - 45 7c c7 14 43 68 78 c5-bb 08 aa 76 fb 4e 7a fc E|..Chx....v.Nz.
00a0 - 1b bd 97 81 63 e0 3c 47-1f d0 7d 11 83 f6 81 e6 ....c.<G..}..... Start Time: 1678096567
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: yes
---
250 SMTPUTF8
STEP 3 - Authentication test
When the command executed in STEP 2 completes, the SMTP server waits for next command.
Executing EHLO UBA_TEST command, a list of SMTP extended features is displayed:
EHLO UBA_TEST 250-smtpserverexample.mail.com 250-PIPELINING 250-SIZE 102400000 250-VRFY 250-ETRN 250-AUTH LOGIN PLAIN 250-AUTH=LOGIN PLAIN 250-ENHANCEDSTATUSCODES 250-8BITMIME 250-DSN 250 CHUNKING
Encrypted credentials must be provided to connect to the SMTP server.
The "AUTH LOGIN" SMTP command requires user and password being encrypted separately (two strings generated)
The "AUTH PLAIN" SMTP command requires user and password being preceded by an ASCII "null" character (\0 meaning \zero) and encrypted together in a single string.
Below you can see an example of how to generate the encrypted strings for both the STMP commands:
### GENERATING TWO ENCRYPTED STRINGS, ONE FOR THE USER AND ONE FOR THE PASSWORD ### uba-master-node$ echo -en "splunk-notification@mail.com" | base64 c3BsdW5rLW5vdGlmaWNhdGlvbkBtYWlsLmNvbQ== uba-master-node$ echo -en "user_strong_password" | base64 dXNlcl9zdHJvbmdfcGFzc3dvcmQ=### GENERATING A SINGLE ENCRYPTED STRING WITH BOTH USER AND PASSWORD ### uba-master-node$ echo -en "\0splunk-notification@mail.com\0user_strong_password" | base64 AHNwbHVuay1ub3RpZmljYXRpb25AbWFpbC5jb20AdXNlcl9zdHJvbmdfcGFzc3dvcmQ=
Now the user is ready to login to the SMTP server by using the credentials.
Below the examples of both the "AUTH LOGIN" and the "AUTH PLAIN" commands:
AUTH LOGIN 334 VXNlcm5hbWU6 <-- The SMTP server is asking for the username (base64 format) c3BsdW5rLW5vdGlmaWNhdGlvbkBtYWlsLmNvbQ== 334 UGFzc3dvcmQ6 <-- The SMTP server is asking for the password (base64 format) dXNlcl9zdHJvbmdfcGFzc3dvcmQ= 235 2.7.0 Authentication successful=================================================================================AUTH PLAIN AHNwbHVuay1ub3RpZmljYXRpb25AbWFpbC5jb20AdXNlcl9zdHJvbmdfcGFzc3dvcmQ= 235 2.7.0 Authentication successful
When a connection attempt is unsuccessful this is what is returned by both the above commands:
535 Authentication credentials invalid
STEP 4 - Connection and Authentication test with a Python script
Below script tests the credentials used to authenticate with an SMTP server.
It asks for user (usually an email address), password and the FQDN of the smtp server.
It connects to port 587 (TLS), sends the EHLO command and tests all the authentication methods available.
Debug level is 2 to show the commands sent and what is returned.
The python3 interpreter is needed to run the script.
########################################################################
# Script: smtp_tls_test.py
# Python Modules: smtplib, getpass
# Python supported version: 3.x
# Description: The script tests the credentials used to authenticate
# with an SMTP server. It asks for user (usually an
# email address), password and the FQDN of the smtp server.
# It connects to port 587 (TLS), sends the EHLO command
# and tests all the authentication methods available.
# Debug level is 2 to show the commands sent and what
# is returned.
# Author: Paolo Massara - TSE EMEA Security Team
########################################################################### MODULES ###
import smtplib
import getpass### VARIABLES ###
user = input("Username: ")
password = getpass.getpass("Password: ")
smtp_server_fqdn = input("SMTP server FQDN: ")### MAIN ###
with smtplib.SMTP(smtp_server_fqdn, 587) as server:
server.set_debuglevel(2)
server.starttls(keyfile=None, certfile=None, context=None)
server.ehlo(name='uba_test')
server.login(user, password)
server.quit()
Below an example of what the script shows when executed (and the authentication fails):
UBA % /usr/bin/python3 ./smtp_tls_test.py
Username: splunk-notification@mail.com
Password:
SMTP server FQDN: smtp.mail.com
15:08:56.004900 send: 'ehlo [127.0.0.1]\r\n'
15:08:56.128467 reply: b'250-mail.com Hello [127.0.0.1] [165.225.196.197]\r\n'
15:08:56.128573 reply: b'250-8BITMIME\r\n'
15:08:56.128595 reply: b'250-SIZE 141557760\r\n'
15:08:56.128612 reply: b'250 STARTTLS\r\n'
15:08:56.128632 reply: retcode (250); Msg: b'mail.com Hello [127.0.0.1] [165.225.196.197]\n8BITMIME\nSIZE 141557760\nSTARTTLS'
15:08:56.129045 send: 'STARTTLS\r\n'
15:08:56.252535 reply: b'220 OK\r\n'
15:08:56.252655 reply: retcode (220); Msg: b'OK'
15:08:56.574774 send: 'ehlo uba_test\r\n'
15:08:56.697067 reply: b'250-mail.com Hello uba_test [165.225.196.197]\r\n'
15:08:56.697179 reply: b'250-8BITMIME\r\n'
15:08:56.697201 reply: b'250-AUTH LOGIN PLAIN\r\n'
15:08:56.697219 reply: b'250 SIZE 141557760\r\n'
15:08:56.697239 reply: retcode (250); Msg: b'mail.com Hello uba_test [165.225.196.197]\n8BITMIME\nAUTH LOGIN PLAIN\nSIZE 141557760'
15:08:56.697320 send: 'AUTH PLAIN AHNwbHVuay1ub3RpZmljYXRpb25AbWFpbC5jb20AdXNlcl9zdHJvbmdfcGFzc3dvcmQ=\r\n'
15:08:56.825779 reply: b'535 Authentication credentials invalid\r\n'
15:08:56.825900 reply: retcode (535); Msg: b'Authentication credentials invalid'
15:08:56.825968 send: 'AUTH LOGIN c3BsdW5rLW5vdGlmaWNhdGlvbkBtYWlsLmNvbQ==\r\n'
15:08:56.948957 reply: b'334 UGFzc3dvcmQ6\r\n'
15:08:56.949083 reply: retcode (334); Msg: b'UGFzc3dvcmQ6'
15:08:56.949139 send: 'dXNlcl9zdHJvbmdfcGFzc3dvcmQ=\r\n'
15:08:57.076810 reply: b'535 Authentication credentials invalid\r\n'
15:08:57.076929 reply: retcode (535); Msg: b'Authentication credentials invalid'
15:08:57.076970 send: 'QUIT\r\n'
15:08:57.199641 reply: b'221 mail.com Service closing transmission channel\r\n'
15:08:57.199733 reply: retcode (221); Msg: b'mail.com Service closing transmission channel'
Traceback (most recent call last):
File "/Users/pmassara/UBA/./smtp_tls_test.py", line 29, in <module>
server.login(user, password)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/smtplib.py", line 745, in login
raise last_exception
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/smtplib.py", line 734, in login
(code, resp) = self.auth(
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/smtplib.py", line 657, in auth
raise SMTPAuthenticationError(code, resp)
smtplib.SMTPAuthenticationError: (535, b'Authentication credentials invalid')
UBA %
STEP 5 - Capturing the packets exchange
Capturing the packets stream between the UBA master node and the SMTP server, is the way to investigate how they negotiate the secure connection.
To do so use the "tcpdump" command.
Below there is an example of the command to execute connected via SSH to the UBA master node as caspida user:
sudo tcpdump -e -i any -tttt -vvv --immediate-mode --dont-verify-checksums -w /tmp/uba_smtp_test.pcap host smtp.mail.com
Related Links
UBA Administration Manual section describing how to configure threats email sending Send_Splunk_UBA_threats_to_analysts_using_emailRevision History
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
30-Sep-2026
|
Initial Release |
Feedback