It's is crucial to ensure that resource consumption does not exceed recommended throughput limits to prevent performance issues. As per the Sizing and Scaling documentation, with 1 CPU to 2GB of memory, the Collector can ingest the following:
A common troubleshooting step is verifying if a deployment is exceeding these limits. This can be achieved by configuring count connectors within the pipeline configuration. The count connectors help track the number of traces, metrics, and logs processed, allowing you to monitor system load and make adjustments as needed.
To investigate whether data pipelines are exceeding recommended limits:
connectors:
count/logs:
logs:
number.of.logs.count:
count/metrics:
metrics:
number.of.metrics.count:
count/traces:
spans:
number.of.spans.count:
...
service:
pipelines:
logs:
receivers: [...]
processors: [...]
exporters: [splunk_hec/platform_logs, count/logs]
metrics:
receivers: [...]
processors: [...]
exporters: [signalfx, count/metrics]
traces:
receivers: [...]
processors: [...]
exporters: [otlp, signalfx, count/spans]
metrics/count:
receivers: [count/logs, count/metrics, count/traces]
exporters: [signalfx]
Note: Make sure to set up a separate pipeline for count metrics (metrics/count) so as not to disrupt your standard metrics processing.
To effectively monitor and manage your data pipeline throughput:
metrics/count) for the count metrics to prevent interference with your standard metrics pipeline.| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
01-Oct-2026
|
Initial Release |