LDAP users are not synced across SHC

Applicability

Description

LDAP users are only visible in the UI after successfully logged in to the search head. But there is no sync across the search heads of the LDAP users nor is there a proactive user pull from LDAP for all Users matching the LDAP groups specified in the LDAP settings.
This is giving problems because of bi-directional integrations with the Splunk platform and the users are expected to exist on every node.

User's concerns:

Is there a way so that the user list for LDAP users can be consistent across a search head cluster? 
Can it be automated this reload via an API REST call?
If a user is removed from the LDAP (groups), will the user disappear in the User list in Splunk?
If a user is removed, how can we handle the orphaned objects (if any)?

Investigation

Resolution

Users not showing in the UI unless logged in is expected behavior, also, User Data Preferences should be synced across Search Head Peers (log into the CLI, should see that Users Folder in the etc/users directory after successfully logged into the Splunk Environment).

About "nor is there a proactive user pull from LDAP for all Users matching the LDAP groups specified in the LDAP settings". With the max_user_to_precache setting set to 0 --> Then this would be a "True" Statement. By default, Splunk will "Pre-Cache" up to the first 1000 Users listed in the LDAP Group DN that is configured. So an OOB Splunk Environment that is configured to use LDAP will Cache the first 1000 Users (with modified the max_user_to_precache setting and have that set to 0 then Every Authentication request would be sent to the LDAP Server to validate User first and then Authenticate that user.) 

As for the "Syncing between Peers". Yes, SHC Peers sync User Preferences/Saved Searches/etc and won't see the user listed in the UI until they logged into that specific instance, this is expected behavior.

- If a user is removed from the LDAP (groups), will the user disappear in the User list in Splunk? 

If a user is removed from an external directory, Splunk does not automatically remove the corresponding Splunk user

Not an issue to leave behind the deleted user's KO's

May generate an error if the user KO"s have global permissions.

To remove the user from Splunk:

• Back up the Splunk_Home/etc/users/<user>folder
• Search any reports, views, and other KO'S with global permissions owned by the user under SPLUNK_HOME/etc/apps
• Redirect all global-context objects to a new owner
• Remove the SPLUNK_HOME/etc/users/<user> folder

 If a user is removed, how to handle the orphaned objects (if any)?
Splunk has a few built-in ways to handle orphaned searches and knowledge objects. View the orphaned searches through the "Orphaned scheduled searches" report :

Settings > Searches, reports, and alerts > Set "App" to "All" > Orphaned scheduled searches

To actually reassign or remove the searches, the steps change a bit depending on if the search was privately owned (not shared), or if it was publicly owned (shared to an app or global level).

A. If the search was shared:

• Select Settings > All configurations.
• Click Reassign Knowledge Objects.
• Click Orphaned to filter out non-orphaned objects from the list

At this point, the list should only contain orphaned objects that have been shared. Now determine what to do with the items on that list.

For reference, those steps here:

Use_the_Reassign_Knowledge_Objects_page_in_Settings

B. If the search was not shared, there are only two ways to reassign or remove the search:

1. Temporarily recreate the invalid owner:

The easiest solution for this is to temporarily recreate the invalid owner account, reassign the knowledge object, and then deactivate the invalid owner account.

Remove or modify the group or user on the LDAP provider, tell Splunk to reload the authentication using either this REST call
 

| rest splunk_server=* /services/authentication/providers/services/_reload

or this CLI Splunk command

./splunk _internal call /authentication/providers/services/_reload -auth

This will refresh/reload the LDAP provider information and the removed users/group should be gone.

If the users/group is still visible, check with non-Splunk LDAP tools against the LDAP provider and see what actually get back.

Removed-users-from-LDAP-authentication