Splunk OTEL Collector for Kubernetes - Required Information to troubleshoot

Applicability

Description

To perform troubleshooting the full diagnostic is needed.

1. Kubernetes Cluster details

• Version: $ kubectl version
• OS (e.g. Ubuntu 20.04)
• Cloud (if applicable, e.g. EKS, AKS, GKE) 

2. OTEL Collector deployment files

• Helm command used to deploy the agent (if applicable)
• Custom values.yaml (if applicable)
• configMap for otel-agent/clusterReceiver

3. OTEL chart version
Please make sure you are not on an old version as there have been improvements in newer versions of the Splunk OTel Collector that can be found here .

4. Enable Debug logs
Configure the logging exporter as per the Splunk OTel Collector Troubleshooting Guide, e.g. set the log level in the config service::telemetry::logs

service:
  telemetry:
    logs:
      level: "debug"

5. Enable log_data_points to log the datapoints sent by the OTel Collector
You can set the log_data_points option in exporters::signalfx to view the metrics that the agent is sending as per the Splunk SignalFx Metrics Exporter documentation, e.g. 

exporters:
  signalfx:
    access_token: "${SPLUNK_ACCESS_TOKEN}"
    api_url: "${SPLUNK_API_URL}"
    ingest_url: "${SPLUNK_INGEST_URL}"
    log_data_points: true

You can overwrite the configuration using the values.yaml "config" option both for clusterReceiver
and otel-agent. Example values.yaml found here .

Note: If log level is set to debug and log_data_points is set to true, all datapoints dispatched to Splunk Observability from the SignalFx Exporter will be logged. Similar to log_data_points, there is also an option to log dimension updates by setting log_dimension_updates to true.

6.0 Example - Retrieving the logs only for Kubernetes 
First, view the overall health and structure of the OTEL pods with $ kubectl get pods. Then, collect logs with the below commands: 

kubectl describe pod my-pod-otel 
kubectl logs my-pod-otel otel-collector >my-pod-otel.log

6.1 Example - Retrieving the configMaps  
First, check OTEL configMaps with $ kubectl get cm. Then, save the otel-agent/clusterReceiver configMaps content:

kubectl get cm my-otel-collector-configmap -o yaml > my-otel-cm.yaml

6.2 Example: Getting the logs and configurations using the Splunk Support Bundle
Get the Splunk support bundle from the release downloads page. For example: 

$ wget https://github.com/signalfx/kubectl-splunk/releases/download/v0.3.0/kubectl-splunk_0.3.0_Linux_x86_64.tar.gz

Run the following command to collect a support bundle:

$ kubectl-splunk support

This command creates a file called signalfx-support.zip. Please attach this to your support case. Learn more about kubectl-splunk here .

kubectl-splunk support (and command options)

Remember to disable debug and log_data_points after you have collected the logs and configs to minimize resource impact on the server.

 

Related Links

How to gather debug and log_data_points on OTEL Cluster Receiver in Kubernetes

Kubernetes Cluster Receiver Metrics Lag - Getting debug logs from the OTel Collectors Kubernetes Receiver

Splunk OpenTelemtry (OTel) for Kubernetes is missing several metrics starting with k8s.pod.* and k8s.node.*

Kubernetes data not being ingested because Otel collector is failing to connect to Splunk Cloud

Note:
Splunk Observability documentation is now the official location for OpenTelemetry documentation and is replacing the Github documentation.
The same information is available, so if any links to Github docs fail with 404, check the Splunk Observability documentation for the equivalent documentation.

See below for current Splunk Observability documentation for the Splunk Opentelemetry Collector for Kubernetes

Get started with the Collector for Kubernetes

Troubleshoot sizing for the Collector for Kubernetes

Troubleshoot missing metrics

Troubleshoot Kubernetes and container runtime compatibility

Collector for Kubernetes: Gather support information

kubectl-splunk support (and command options)