Splunk Add-on for Google Cloud Platform(GCP) : Pub/Sub input configured stopped collecting new logs and failing with 403 errors.

Applicability

Description

The Pub/Sub input that utilizes GCP Pub/Sub subscription to fetch the objects at a predefined internal and is configured in the Splunk Add-on for Amazon Web Services (AWS) is encountering a failure, accompanied by the below error message:

requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://pubsub.googleapis.com/v1/projects/bustling-tree-430813-n7/subscriptions/new-sub

Another observation made was that for the same Pub/Sub input, one subscription is working, but another is not.

Investigation

Upon investigating and checking the internal logs, with the below search query to check if there are any error messages in the _internal logs:

SPL:

index="_internal" sourcetype=*gcp* datainput="input_name" level=ERROR

Found the below error messages which are generated repeatedly whenever aPub/Sub input is executed. The error message reads as below:

Error Logs:

2024-07-28 14:59:44,254 level=ERROR pid=1656717 tid=Thread-3 logger=splunk_ta_gcp.modinputs.pubsub pos=pubsub.py:_run_consumer:350 | datainput="Daya_GCP_Test2" start_time=1722059265, subscription="projects/bustling-tree-430813-n7/subscriptions/new-sub" | message="An error occurred when pulling message." 
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/splunk_ta_gcp/modinputs/pubsub.py", line 347, in _run_consumer
    consumer.run()
  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/splunk_ta_gcp/modinputs/pubsub.py", line 195, in run
    parcel = agent.pull()
  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/bin/splunk_ta_gcp/modinputs/pubsub.py", line 149, in pull
    response.raise_for_status()
  File "/opt/splunk/etc/apps/Splunk_TA_google-cloudplatform/lib/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://pubsub.googleapis.com/v1/projects/bustling-tree-430813-n7/subscriptions/new-sub

The error indicates that the input is attempting to pull messages from a Google Cloud Pub/Sub subscription but is facing an HTTP 403 Forbidden error. This typically means that the service account or credentials being used do not have the necessary permissions to access the Pub/Sub subscription.

Resolution

The Pub/Sub Subscription did not have the service account set with access which was causing the issue. The required access was granted on the GCP and the below permissions were provided on the Service Account:

API Permissions:

pubsub.schemas.get
pubsub.schemas.list
pubsub.schemas.listRevisions
pubsub.schemas.validate
pubsub.snapshots.get
pubsub.snapshots.list
pubsub.subscriptions.get
pubsub.subscriptions.list
pubsub.topics.get
pubsub.topics.list
resourcemanager.projects.get
serviceusage.quotas.get
serviceusage.services.get
serviceusage.services.list
pubsub.snapshots.seek
pubsub.subscriptions.consume
pubsub.topics.attachSubscription

Related Links

GCP Pub/Sub 403 Forbidden Troubleshooting Docs

Service Account Prerequisites Permissions