Introduction
This document describes the procedure to troubleshoot how to clean Advance Message Queuing Protocol (AMQP) heartbeats queue
Contributed Aswathi Surendran, Gustavo Bell Cisco TAC Engineers.
Prerequisites
Requirements
Cisco recommends that you have knowledge of these topics:
- Linux Interface
- Virtual Machine Environments
- Rabbit
Components Used
The information in this document is based on these software versions:
- CloudCenter version 4.3.x to 4.6.x
- CloudCenter Message Buss (RabbitMQ_Server)
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Background Information
The active thread size of the pool is 64 where the threads submitted corresponding to the jobs were more than 3000 which will be rejected as the code cannot handle more than 64 threads at a time hence it will wait for previous task to get completed and execute the other task.
Any task which are greater that what a threadpool executed can consume will be rejected.
Problem
the filecliqr-connection.log locate in /usr/local/tomcatgua/logs/ report error:
Caused by: org.springframework.core.task.TaskRejectedException: Executor [java.util.concurrent.ThreadPoolExecutor@6737f4fb[Running, pool size = 64, active threads = 64, queued tasks = 3000, completed tasks = 413]] did not accept task: org.springframework.aop.interceptor.AsyncExecutionInterceptor$1@6b0517b3
at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.submit(ThreadPoolTaskExecutor.java:284)
at org.springframework.aop.interceptor.AsyncExecutionAspectSupport.doSubmit(AsyncExecutionAspectSupport.java:186)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.invoke(AsyncExecutionInterceptor.java:123)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
at com.sun.proxy.$Proxy84.processMessage(Unknown Source)
at com.osmosix.commons.messaging.listeners.impl.DefaultNodeMessageListener.handleMessage(DefaultNodeMessageListener.java:35)
at sun.reflect.GeneratedMethodAccessor78.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:269)
at org.springframework.amqp.rabbit.listener.adapter.MessageListenerAdapter.invokeListenerMethod(MessageListenerAdapter.java:383)
... 12 more
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3911a213 rejected from java.util.concurrent.ThreadPoolExecutor@6737f4fb[Running, pool size = 64, active threads = 64, queued tasks = 3000, completed tasks = 413]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.submit(ThreadPoolTaskExecutor.java:281)
... 23 more
2016-10-19 02:32:32,205 INFO annotation.RequestMappingHandlerMapping [localhost-startStop-1] - Mapped "{[/image/service/dltargetlocal],methods=[POST]}" onto public java.util.Map<java.lang.String, ?> com.osmosix.gateway.image.transform.ImageTransformController.downloadToTargetLocal(com.osmosix.commons.image.transform.DownloadImageFileRequest)
2016-10-19 02:32:32,206 WARN listener.ConditionalRejectingErrorHandler [SimpleAsyncTaskExecutor-1] - Execution of Rabbit message listener failed.
org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener method 'handleMessage' threw exception
at org.springframework.amqp.rabbit.listener.adapter.MessageListenerAdapter.invokeListenerMethod(MessageListenerAdapter.java:391)
at org.springframework.amqp.rabbit.listener.adapter.MessageListenerAdapter.onMessage(MessageListenerAdapter.java:294)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:757)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:680)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:93)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:183)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1352)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:661)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1096)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1080)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$800(SimpleMessageListenerContainer.java:93)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1197)
at java.lang.Thread.run(Thread.java:745)
Resolution
Step 1. Login to RabbitMQ server via SSH.
Step 2. Switch to root user.
# sudo -i
Step 3. Enable Manegement Plugin.
# rabbitmq-plugins enable rabbitmq_management​
Step 4. Enable access to rabbitmqadmin command.
# chmod +x `find /var/lib/rabbitmq/ -name "rabbitmqadmin"`
# ln -s `find /var/lib/rabbitmq/ -name "rabbitmqadmin"` /usr/sbin
Step 5. RabbitMQ can be accessed via GUI.
http://<RabbitMQ_Server_IP>:15672 (username=cliqr password=cliqr)
Note: Replace RabbitMQ_Server_IP in the configuration with your RabbitMQ server address
Navigate queues tab to see the details
or
CLI:
# rabbitmqadmin -V /cliqr -u cliqr -p cliqr list queues
# rabbitmqadmin -V /cliqr -u cliqr -p cliqr list exchanges
# rabbitmqadmin -V /cliqr -u cliqr -p cliqr list channels
Step 6. To delete the gateway heartbeat queue.
#rabbitmqadmin -V /cliqr delete queue name=cliqr.gateway.heartbeat.queue --username=cliqr --password=cliqr
Step 7. Restart tomcat service in RabbitServer.
#/etc/init.d/tomcatgua stop
#/etc/init.d/tomcatgua start
Note: If cliqr.gateway.heartbeat.queue is missing, restart CCO tomcat service by following the below steps:
Step 8. Login to CCO server via SSH.
Step 9. Switch to root user.
# sudo -i
Step 10. Restart Tomcat server.
#/etc/init.d/tomcat stop
#/etc/init.d/tomcat start
Feedback