API throttling issue when collecting high volume AWS CloudWatch data

Applicability

Investigation

Splunk Add-on for AWS is hitting API throttling limitation when collecting high volume of AWS CloudWatch data. 

Error message in splunk_ta_aws_cloudwatch.log: 

level=ERRORpid=8075tid=Thread-625logger=splunk_ta_aws.modinputs.cloudwatch_logs.
aws_cloudwatch_logs_data_loaderpos=aws_cloudwatch_logs_data_loader.py:
describe_cloudwatch_log_streams:73 | | message="Failureindescribingcloudwatchlogs
streamsduetothrottlingexceptionforlog_group=/aws/batch/job, sleep=4.14110814822, 
reason=Traceback (mostrecentcalllast):File "/opt/splunk/etc/apps/Splunk_TA_aws/
bin/splunk_ta_aws/modinputs/cloudwatch_logs/aws_cloudwatch_logs_data_loader.py", 
line63, indescribe_cloudwatch_log_streams...5linesomitted...JSONResponseError:
JSONResponseError:400BadRequest {​u'__type':u'ThrottlingException', u'message':
u'Rateexceeded'}​ "

Resolution

API throttling limitation is known issue imposed by AWS. Refer to How do I avoid throttling when I call PutMetricData in the CloudWatch API?

Here are the options to resolve the AWS API throttling issue:
1. Contact AWS to increase the CloudWatch API rate. 

2. Decrease the number of metrics to collect or increase the granularity of your indexed data in order to make fewer API calls. Refer to API throttling issues. When setting up a AWS CloudWatch inputs for Splunk Add-on for AWS, the unnecessary Metrics can be removed or granularity can be increased by increasing the period (in seconds) the returned data points. Note that larger granularity will affect the accuracy of the returned data points. Refer to Configure CloudWatch inputs for the Splunk Add-on for AWS - Splunk Documentation. 

3. Switch to use Kinesis input to collect the AWS CloudWatch logs. Refer to Configure Kinesis inputs for the Splunk Add-on for AWS - Splunk Documentation