MTS is not visible in Splunk O11y due to datapoints are silently dropped by OTel Collector because they have more than 36 dimensions and reached O11y backend system limit. This might be case for "heavy" JVM Metrics from Kubernetes or from Cribl.
The OTel Collector silently drops datapoints when there are in excess of 36 dimensions on metric datapoints. This situation can only be identified when debug logging is turned on. This should not be considered as a bug on OTel but expected behavior.
Those drops are silent and will not be displayed in any dashboard in Splunk Observability Cloud.
exporters:
# Debug
debug:
verbosity: detailedservice:
pipelines:
metrics:
exporters: [signalfx, debug]
Restart OTel Collector
sudo systemctl restart splunk-otel-collector.service
journalctl -u splunk-otel-collector.service | grep -A37 -i "Resource attribute"
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: Resource attributes:
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> host.name: Str(ag-lab-otel-1)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> os.type: Str(linux)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag01: Str(ag01)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag02: Str(ag02)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag03: Str(ag03)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag04: Str(ag04)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag05: Str(ag05)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag06: Str(ag06)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag07: Str(ag07)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag08: Str(ag08)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag09: Str(ag09)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag10: Str(ag10)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag11: Str(ag11)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag12: Str(ag12)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag13: Str(ag13)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag14: Str(ag14)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag15: Str(ag15)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag16: Str(ag16)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag17: Str(ag17)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag18: Str(ag18)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag19: Str(ag19)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag20: Str(ag20)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag21: Str(ag21)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag22: Str(ag22)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag23: Str(ag23)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag24: Str(ag24)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag25: Str(ag25)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag26: Str(ag26)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag27: Str(ag27)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag28: Str(ag28)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag29: Str(ag29)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag30: Str(ag30)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag31: Str(ag31)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag32: Str(ag32)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag33: Str(ag33)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag34: Str(ag34)
Sep 30 08:27:59 ag-lab-otel-1 otelcol[2124736]: -> ag35: Str(ag35)
The only workaround would be to cut down attributes with least business and/or technical value for datapoint to not exceed 36-dimensions limitation.
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
29-Sep-2026
|
Initial Release |