Data Latency issues with Pub/Sub input configured in the Splunk Add-on for Google Cloud Platform(GCP).

Applicability

Description

The Pub/Sub inputs which are configured in the Splunk Add-on for GCP, are not collecting all logs in real-time from the Google Cloud portal.

Other input types, such as Cloud Monitoring and Cloud Storage Bucket, are successfully fetching logs without latency. The issue appears to be specific to the Pub/Sub, while other data sources remain unaffected.

Investigation

1. Checked the latency in the associated index and the sourcetype with the help of the below SPL to determine the stats of latency.

SPL:

index=<index_name> sourcetype=<sourcetyepe_name> source=<source_name>
| eval diff_in_min= (_indextime - _time)/60
| eval capturetime=strftime(_time,"%Y-%m-%d %H:%M:%S")
| eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S")
| table capturetime indextime diff_in_min

Found that Pub/Sub inputs configured in the Add-on had a latency of 10-12 Hours.

2. Therefore, checked the internal logs, with the below search query to check if there were any error messages in the _internal logs but found that there were no such logs that could explain about the cause of latency.

SPL:

index="_internal" sourcetype=*gcp* datainput="input_name"

No Error logs, as mentioned above.

3. Additionally, checked the queue blockages on the instance where the inputs were configured but no queue blockage were found.

SPL:

index="_internal" host=<hostname> blocked=true

No queue blockage was found.

4. Since data latency was not caused by environmental factors like queue blockages or internal errors, further investigation was conducted. It was found that a large volume of data was present in Pub/Sub on the GCP end. Pub/Sub inputs are inherently scalable, allowing them to be cloned to improve data throughput and performance. These cloned inputs can be created on the same instance or different instances to optimize ingestion.

Resolution

To address the latency caused by the high volume of data in Pub/Sub on GCP, the number of Pub/Sub inputs can be gradually increased to improve data ingestion. These inputs can be cloned either on the same instance or across multiple instances to enhance performance.

Steps to resolve:

1. Scale up inputs gradually by adding more Pub/Sub inputs.
2. Monitor latency after each increase to ensure optimal performance.
3. Avoid excessive scaling unless system resources support it.