Splunk OpenTelemetry Collector not sending metrics from AWS EC2 with aws_tag

Applicability

Description

The issue arises when attempting to monitor metrics in Splunk Observability with default filters like host.name, which works, but applying filters related to AWS tags such as aws_tag_Cluster and aws_tag_Environment results in no metrics displayed on the dashboard. Upon further investigation, it's discovered that the discrepancy in naming conventions between CMS/Cloud Integration and OTEL resourcedetection tags causes this problem.

Investigation

Tag Naming Convention Discrepancy: Users are utilizing the OTEL resourcedetection tag parsing, which follows the naming convention ec2.tag.<tag_name>. However, they expect the CMS/Cloud Integration naming convention aws_tag_<tag_name>.

Configuration Issue: resourcedetection/internal has been defined in the agent.yaml file but is has not been enabled in the pipeline, leading to ineffective tag parsing.

Resolution

To address the issues encountered with Splunk OpenTelemetry Collector configuration and metric visualization, follow these steps:

Enable resourcedetection/internal: Ensure that resourcedetection/internal is enabled in the pipeline configuration of the agent.yaml file. This can be achieved by adding it to the list of enabled extensions under the service section.

Update Tag Filters: Instead of using CMS aws_tag_<tag_name>, adjust the filters to utilize OTEL resourcedetection naming conventions, specifically ec2.tag.<tag_name>. Update the dashboard filters accordingly to reflect this change.

Cloud Integration: If the preference is to use aws_tag_<tag_name>, ensure that cloud integration is enabled. This allows the monitoring of EC2 instances with AWS tags seamlessly.

By implementing these steps, you can effectively configure Splunk-otel-collector on EC2 instances and resolve the issues related to metric visualization in the SignalFX cloud dashboard.

For further assistance or detailed guidance, refer to the Splunk documentation or consult with Splunk support team.

Related Links

AWS Tagging Best Practices

Splunk OpenTelemetry Collector AWS EC2

Splunk Observability Rabbitmq