KVstore fails to write and save the lookup

Applicability

Description

"Could not append to collection 'kvstore_mogroupoutputW10history': An error occurred while saving to the KV Store. Look at search.log for more information.."

Investigation

On-Prem:
 

In the back end, we are able to witness the below errors related to the 'kvstore_mogroupoutputW10history' in search_messages.log

12-14-2022 13:21:43.459 +0000 ERROR SearchMessages - orig_component="SearchOperator:inputcsv" app="MOGroupApp" sid="1671024100.1971" message_key="INPUT_CSV:INVALID_LOOKUP_TABLE_TYPE" message=The lookup table 'mogroupoutputW10history' requires a .csv or KV store lookup definition.12-15-2022 03:30:43.454 +0000 ERROR SearchMessages -
orig_component="outputcsv" app="MOGroupApp" sid="scheduler_cmFqZXNoLmFubmFtYXJlZGR5QG1vZ3JvdXBwYXJ0bmVycy5jb20__MOGroupApp__RMD557790c4228989501_at_1671075000_1468" message_key="INPUT_CSV:INVALID_LOOKUP_TABLE" message=Error in ' An error occurred while saving to the KV Store. Look at search.log for more informatioookup' command: The lookup table 'mogroupoutputW10history' is invalid.12-15-2022 05:17:43.621 +0000 ERROR SearchMessages - orig_component="outputcsv" app="MOGroupApp" sid="1671081458.4707" message_key="OUTPUT_CSV:WRITE_KVSTORE_FAIL" message=Could not append to collection 'kvstore_mogroupoutputW10history': An error occurred while saving to the KV Store. Look at search.log for more information..

The transforms.conf has been set in the below format in the MOGroupApp 
 

Go to /opt/splunk/etc/apps/MoGroupApp/local/transforms.conf
 

[kvstore_mogroupoutputW10history]
collection = kvstore_mogroupoutputW10history
external_type = kvstore
fields_list = date,column,"10.0 1703 (15063)","10.0 1709 (16299)","10.0 1809 (17763)","10.0 1903 (18362)","10.0 1909 (18363)","10.0 2004 (19041)","10.0 20H2 (19042)","10.0 21H1 (19043)","10.0 21H2 (19044)","11.0 21H2 (22000)"

Cloud:

On Search Head GUI,  Go to Settings > Lookups > Lookup definitions > kvstore_mogroupoutputW10history
 

Also, the job inspector shows as below:
 

ERROR KVStoreLookup [2791 phase_1] - KV Store output failed with err: The provided query was invalid. (Document may not contain '$' or '.' in keys.) message: 

Resolution

According to the job inspector, fields_list in transforms.conf for the KVstorelookup should not contain '$' or '.'

Replacing the period('.') with the underscore('_') resolved the issue.

Related Links

https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/kvstore/uselookupswithkvstore/

https://docs.splunk.com/Documentation/AddOns/released/Salesforce/Savedsearch#Migrating_from_CSV_lookups_to_KV_store_lookups

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf#transforms.conf.spec