Identity Manager Error for KV Store Lookups in Splunk Enterprise Security 8.x

Applicability

Description

Following an upgrade to Splunk Enterprise Security version 8.x, users observe error messages in the Messages dropdown or identity_manager.log similar to: 

ERROR pid=XXXXXX tid=MainThread file=identity_manager.py:lookup_last_update:397 | status="Lookup file error, unknown path or update time" name=<lookup_name>

Where <lookup_name> can be any KV Store-backed identity lookup, such as:

• xyz_kv_user_identities
• xyz_id_identities
• xyz_user_identities

These errors trigger the saved search Identity - Identity Manager Error, which generates alerts visible in the ES interface.

Key Characteristics:

• The error occurs specifically for identity lookups configured as KV Store collections (not traditional CSV file-based lookups)
• Identity data continues to populate correctly in the identity_lookup_expanded collection
• The A&IM search preview functions normally
• Running the identity merge search manually completes without errors
• The logs show the lookup being detected with collection="None", indicating the KV Store collection is not being resolved correctly in the last update time check

Investigation

To confirm this issue affects your environment:

1. Check the identity_manager.log for the error pattern:

Navigate to:

Settings > Data inputs > Files & Directories

Or run the following search:

index=_internal source="*identity_manager.log" ("Lookup file error" OR "unknown path or update time")
| table _time, status, name, file

2. Verify the lookup is a KV Store collection:

Navigate to:

Enterprise Security > Configure > Data Enrichment > Asset and Identity Management

Locate the identity lookup mentioned in the error and confirm it is configured as a KV Store lookup (not a CSV file).

3. Confirm identity data is populating correctly:

Navigate to:

Enterprise Security > Configure > Data Enrichment > Asset and Identity Management > Identity

Click Search Preview and then Open in Search. Verify that the search completes successfully and the KV Store lookup data appears in the results.

4. Check for the alert in the ES interface:

Navigate to the Messages dropdown in the Enterprise Security interface. Look for alerts generated by the saved search Identity - Identity Manager Error.

Resolution

This error is benign and does not impact the functionality of Asset and Identity Management. The Identity Manager's last update time logic evaluates file-based CSV lookups (which have a filesystem path and modified time), but KV Store lookups do not have a file path or modification time to report in the same way.

Recommended Action: Exclude KV Store Lookups from the Alert

To prevent these error messages from triggering alerts, exclude the KV Store lookup from the saved search Identity - Identity Manager Error.

Steps:

1. Navigate to:

Enterprise Security > Configure > Content > Content Management

2. Search for and open the saved search:

Identity - Identity Manager Error

3. Locate the search string. It should resemble:
   

   index=_internal (sourcetype="identity_correlation:modular_input" OR sourcetype="identity_correlation:merge") ERROR status="*"
   | regex _raw="ERROR"
   | eval log_filename=mvindex(split(source, "/"), -1)
   | fields _time, status, error, source, file, log_filename, index, _raw

4. Add an exclusion for the KV Store lookup by name. For example, if the lookup is xyz_kv_user_identities, modify the search as follows:
   

   index=_internal (sourcetype="identity_correlation:modular_input" OR sourcetype="identity_correlation:merge") ERROR status="*" NOT name="xyz_kv_user_identities"
   | regex _raw="ERROR"
   | eval log_filename=mvindex(split(source, "/"), -1)
   | fields _time, status, error, source, file, log_filename, index, _raw

   For multiple KV Store lookups, use the OR operator:
   

   NOT (name="xyz_kv_user_identities" OR name="xyz_id_identities" OR name="xyz_user_identities")

5. Save the search. 

6. Clear the existing message in the ES interface (if present) and monitor for 24-48 hours to confirm the alert no longer triggers for the excluded lookup.

Additional Context

For further background on this behavior, refer to the Splunk Community discussion:

Has anyone encountered this error: Asset and Identity Management

Related Links

• Splunk Enterprise Security: Add Asset and Identity Data
• Splunk Community: Asset and Identity Management Error Discussion