Field Extraction Issue in Search Head Cluster (SHC) due to Deployer appending configurations.

Applicability

Description

Field extraction using props.conf and transforms.conf was configured for specific event types and worked correctly on the standalone Search Head. However, when deployed to the Search Head Cluster (SHC), the extraction failed. 

Investigation

1. Verified that the standalone Search Head had complete and correct configurations as below in props.conf and transforms.conf

Here, xp_storage_arrays would be taken as an example of an eventtype name and props and transforms names are taken as an example - 

Under eventtypes.conf - 

[xp_storage_arrays] 
search = index=syslog host IN("15.63.224.79","15.63.224.81","15.63.224.104","15.63.224.105")

Under props.conf - 

[eventtype::xp_storage_arrays] 
REPORT-your_extraction = extract_fields_for_xp_storage_arrays

Under transforms.conf - 

[extract_fields_for_xp_storage_arrays] 
REGEX = (?P<IpAddress>[\d\.]+)\s+(?P<IssueReportedDatetime>\w{3}\s{1,2}\d{1,2}\s\d{2}:\d{2}:\d{2})\s+SVP\s+Storage:\s+\d+,(?P<IssuePriority>\w+),(?P<ArraySerialNumber>[^]+Serial:\d+),(?P<FQDN>[^\s,]+),RefCode\s+:\s+(?P<IssueDetails>.+) 
FORMAT = IpAddress::$1 IssueReportedDatetime::$2 IssuePriority::$3 ArraySerialNumber::$4 FQDN::$5 IssueDetails::$6
 

2. Checked the SHC members and found missing or conflicting transforms.conf entries.

3. Observed that the Deployer's bundle push appended the new configurations to the existing /local directory instead of overwriting them.

4. Confirmed this behavior as expected for SHC environments to preserve instance-specific configurations in system local.

Resolution

1. Removed the app (including the /local directory) from all SHC members to eliminate conflicting configurations.

2. Pushed the updated app configurations, including the correct props.conf and transforms.conf, from the Deployer to the SHC members:

-Under props.conf - 

[eventtype::xp_storage_arrays] 
REPORT-your_extraction = extract_fields_for_xp_storage_arrays

-Under transforms.conf - 

[extract_fields_for_xp_storage_arrays] 
REGEX = (?P<IpAddress>[\d\.]+)\s+(?P<IssueReportedDatetime>\w{3}\s{1,2}\d{1,2}\s\d{2}:\d{2}:\d{2})\s+SVP\s+Storage:\s+\d+,(?P<IssuePriority>\w+),(?P<ArraySerialNumber>[^]+Serial:\d+),(?P<FQDN>[^\s,]+),RefCode\s+:\s+(?P<IssueDetails>.+) 
FORMAT = IpAddress::$1 IssueReportedDatetime::$2 IssuePriority::$3 ArraySerialNumber::$4 FQDN::$5 IssueDetails::$6
 

3. Changed the TRANSFORMS-your_extraction to REPORT-your_extraction in props.conf because using the REPORT directive fixed the issue and allowed the field extraction to work correctly. The REPORT directive works at search-time for dynamic field extraction, which resolved the issue.

4. Used the following command to apply the configurations to all SHC members:

splunk apply shcluster-bundle -target <SHC-member>

5. Verified the field extraction functionality across all SHC members after applying the configuration changes. 

Related Links

Conf Page