Some LDAP users are unable to login, splunkd logs show "LDAP Login failed, could not find a valid user"

Applicability

Description

After setting the LDAP configuration, some users are unable to login to Splunk. This article describes some troubleshooting steps to identify the root cause of the issue and how to solve it.

Investigation

To narrow down the issue, perform a search on the splunkd logs using the following components :
 

index=_internal host=<hostname of SH> sourcetype=splunkd component=ScopedLDAPConnection OR 
component=AuthenticationManagerLDAP OR component=UserManagerPro OR component=UiAuth

Messages in the splunkd logs :

ERROR UiAuth - user=homer action=login status=failure reason=user-initiated 
useragent="Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) 
Chrome/x.x.x.x Safari/537.36" clientip=x.x.x.xERROR ScopedLDAPConnection - strategy="uses_ldap_auth" Error binding to LDAP. 
reason="Invalid credentials"WARN ScopedLDAPConnection - strategy="uses_ldap_auth" LDAP Server returned warning in 
search for DN="DC=dev,DC=splunk,DC=com". reason="Operations error"DEBUG AuthenticationManagerLDAP - Attempting to get user information for user="homer" 
from strategy="uses_ldap_auth"ERROR UserManagerPro - Failed to find ldapuser=homer ldapsearchtime=20308 milliseconds 
after searching max_strategies=1.

Other messages found:"Timed out" instead of "Operations error":

ERROR ScopedLDAPConnection - strategy="uses_ldap_auth" Error binding to LDAP. 
reason="Invalid credentials"WARN ScopedLDAPConnection - strategy="uses_ldap_auth" LDAP Server returned warning 
in search for DN="DC=dev,DC=splunk,DC=com". reason="Timed out"INFO AuthenticationManagerLDAP - Could not find user="homer" with strategy="uses_ldap_auth"ERROR UserManagerPro - Failed to find ldapuser=homer ldapsearchtime=20319 milliseconds 
after searching max_strategies=1.ERROR UserManagerPro - LDAP Login failed, could not find a valid user="homer" on any 
configured serversERROR UiAuth - user=homer action=login status=failure reason=user-initiated 
useragent="Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) 
Chrome/x.x.x.x Safari/537.36" clientip=x.x.x.x

When using the Splunk GUI for the initial LDAP configuration, the settings are only saved if the first bind to the LDAP server succeeds. Because anonymous_referrals is set to 1 by default, the first bind attempt might fail if the LDAP server blocks anonymous referrals.

Resolution

The "Operations error" message indicated a problem with anonymous referrals as mentioned in the following document :

Configure LDAP with Splunk Web
 

• Enable referrals with anonymous bind only.
  • This setting is on by default. Turn this off if referrals are not needed.
  • Splunk can chase referrals with anonymous bind only. You must also have anonymous search enabled on your LDAP server.
  • If encountering long LDAP search timeouts (likely in Active Directory) and "Operations error" in splunkd.log for ScopedLDAPConnection, the issues might be related to referrals. 

Checking authentication.conf it was found that anonymous_referrals was enabled (enabled by default)

[uses_ldap_auth]
anonymous_referrals = 1

The problem with following referrals anonymously is that some LDAP servers (notably Active Directory) don’t permit anonymous binding.

By disabling anonymous_referrals it allowed the users to be able to log in.
 

anonymous_referrals = [0|1]
* Set this to 0 to turn off referral chasing
* Set this to 1 to turn on anonymous referral chasing
* NOTE: Splunk software only chases referrals using anonymous bind.
It does not support rebinding using credentials.
* If you do not need referral support, set this to 0.
* If you wish to make referrals work, set this to 1 and confirm your server
allows anonymous searching
* This setting is optional.
* Default: 1

For a more reliable first-time configuration, consider setting up the LDAP connection directly in authentication.conf on the backend rather than relying solely on the GUI. This avoids issues caused by the default anonymous_referrals=1 setting interfering with the initial bind.

Also the other error message above reporting reason="Invalid credentials" indicates a possible password or permissions issue to one of the LDAP servers. For this kind of issue, it is advised to have the LDAP administrators check for any authentication errors in the LDAP logs to see why users are being rejected.
 

Related Links

• Configure LDAP with Splunk Web
• LDAP Settings