Splunk OpenTelemtry (OTel) for Kubernetes is missing several metrics starting with k8s.pod.* and k8s.node.*

Applicability

Description

After deploying Splunk OTel for Kubernetes version 0.87.0+ the following pod and node metrics are not being collected and are missing when viewing the metrics in Splunk:

• k8s.(pod/node).cpu.time
• k8s.(pod/node).cpu.utilization
• k8s.(pod/node).filesystem.available
• k8s.(pod/node).filesystem.capacity
• k8s.(pod/node).filesystem.usage
• k8s.(pod/node).memory.available
• k8s.(pod/node).memory.major_page_faults
• k8s.(pod/node).memory.page_faults
• k8s.(pod/node).memory.rss
• k8s.(pod/node).memory.usage
• k8s.(pod/node).memory.working_set
• k8s.(pod/node).network.errors
• k8s.(pod/node).network.io

Investigation

1. Using the following SPL in Splunk confirm that the metrics above are missing:

| mstats count(_value) as "Val" where index="otel_metrics_0_93_3" AND metric_name IN (k8s.pod.*, k8s.node.*) by metric_name

2. Check the Splunk Otel Agent pod logs by using the following from the CLI of the Kunernetes node (update namespace and collector-agent-pod-name based on your environment):

kubectl -n {namespace} logs {collector-agent-pod-name} 

You will see the following "tls: failed to verify certificate" error similar to below in the agent pod logs:

2024-02-28T01:11:24.614Z    error   scraperhelper/scrapercontroller.go:200  Error scraping metrics  {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics", "error": "Get \"https://10.202.38.255:10250/stats/summary\": tls: failed to verify certificate: x509: cannot validate certificate for 10.202.38.255 because it doesn't contain any IP SANs", "scraper": "kubeletstats"}
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport
    go.opentelemetry.io/collector/receiver@v0.93.0/scraperhelper/scrapercontroller.go:200
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1
    go.opentelemetry.io/collector/receiver@v0.93.0/scraperhelper/scrapercontroller.go:176

Resolution

The k8s.(pod or node) metrics are collected from the Kubernetes endpoint /stats/summary by the kubeletstats receiver. As of version 0.87.0 of the Splunk Otel Collector the kubelet certificate is verified during this process to confirm it is valid. If you are using a self signed or invalid certificate the kubelestats receiver will fail to collect the metrics.

To resolve this error you can either add valid a certificate to your Kubernetes cluster or disable certificate verification in the OTel agent Kubelet Stats receiver (please consider your security requirements before disabling this)

To add the certificate please refer to the documentation for your Kubernetes cluster.

To disable Certificate verification, set insecure_skip_verify: true for the kubeletstats receiver in the agent.config section of the values.yaml file used to deploy Splunk OTel.

agent:
  config:
    receivers:
      kubeletstats:
        insecure_skip_verify: true

Related Links

kubeletstatsreceiver metric-groups

Breaking Change for 0.87.0

[receiver/kubeletstats] tls verify config not being respected #26319

kubernetes - cannot-use-the-metrics-server-securely-in-a-kubeadm-cluster