Searches using the outputlookup command are unable to overwrite their own KV Store collection

Applicability

Description

Searches that use both the inputlookup and outputlookup commands to retrieve, modify, and write data into the same KV Store collection do not successfully write to the KV Store. However, the search, input, and output process completes without any errors.

Investigation

Discard the use case

The search in question comes from an app that creates this lookup file and adds it to the KV Store collection

• Once the collection is ready for use, the search in question sources the events from this KV Store lookup using inputlookup.
• Uses a custom sort logic that will assign priority to the events.
• Removes the _key field.
• Once the events are in the desired order, the search writes the events into the same KV lookup using the outputlookup command.

By adding new entries to the KV lookup instead of writing on top of the already existing events, the KV lookup displays the new events which indicates an issue either on the logic in how this was applied or the core process that handles the request to write into the KV collection.

The issue is reproducible by recreating the KV lookup manually and setting the desired fields, while sourcing and writing from it, without any extra processing (like sorting, data transformation, or use of any other functions), this confirms the logic is not what causes this behavior.

Please find the search used for this.

| inputlookup lookupTest
| outputlookup lookupTest

Verify the search process

When running the search and collecting the search.log the job inspector shows INFO messages when it connects to the KV Store collection, Splunk Cloud does has to connect to the KV Store Service, even if the service is held in the same node.
These messages should have these fields filled, _appId, _sid, referrerId, having null values could be a sign of an issue with this connection to the KV collection.

06-13-2024 23:39:55.956 INFO KVServiceClient [337429 phase_1] - EC KVService Client Request : _appId= apps_name, _sid= 1718321995.410774, referrerId= eckvservice-apps_name-1718321995.410774 _userContext= nobody
06-13-2024 23:39:56.223 INFO KVServiceClient [337429 phase_1] - EC KVService Client Request : _appId= null, _sid= null, referrerId= eckvservice-null-null _userContext= nobody
06-13-2024 23:39:56.338 INFO outputcsv [337429 phase_1] - SearchMessage orig_component=outputcsv sid=1718321995.410774 message_key=OUTPUT_CSV:RESULTS_WRITTEN_KVSTORE__%s message=Results written to collection 's_99371BdITkeebpaMsS@+ZKVFzJnnNC__s_broken8uYsLhR7dq+AorW8g42vdiSt__lookupTest'.
06-13-2024 23:39:56.338 INFO outputcsv [337429 phase_1] - 76 events written to s_99371BdITkeebpaMsS@+ZKVFzJnnNC__s_broken8uYsLhR7dq+AorW8g42vdiSt__lookupTest

Resolution

Splunk Cloud is moving from a known KV Store Service to a new kvstore_external as part of an effort to improve the scalability and performance of the environments. in this effort, the team included the possibility of writing into a KV lookup using this new service which is not compatible with the old model of writing on top of the already existing entries of the KV collection.

The team is working to close the gap between the KV Store Service and kvstore_external seamlessly, to avoid disrupting normal usage of the service.

• There is no specific date on when this is going to be changed, but based on previous discussions, this is going to be added to the parent case and an enhancement request to this.

Related Links

Public Links

• #aboutkvstorecollections
• #The-key-ID-field