Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Problem

Resolution

Introduction

This document describes the procedure to troubleshoot how to clean Advance Message Queuing Protocol (AMQP) heartbeats queue

Contributed Aswathi Surendran, Gustavo Bell Cisco TAC Engineers.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- Linux Interface

- Virtual Machine Environments

- Rabbit

Components Used

The information in this document is based on these software versions:

- CloudCenter version 4.3.x to 4.6.x

- CloudCenter Message Buss (RabbitMQ_Server)

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Background Information

The active thread size of the pool is 64 where the threads submitted corresponding to the jobs were more than 3000 which will be rejected as the code cannot handle more than 64 threads at a time hence it will wait for previous task to get completed and execute the other task.

Any task which are greater that what a threadpool executed can consume will be rejected.

Problem

the filecliqr-connection.log locate in /usr/local/tomcatgua/logs/  report error:

Caused by: org.springframework.core.task.TaskRejectedException: Executor [java.util.concurrent.ThreadPoolExecutor@6737f4fb[Running, pool size = 64, active threads = 64, queued tasks = 3000, completed tasks = 413]] did not accept task: org.springframework.aop.interceptor.AsyncExecutionInterceptor$1@6b0517b3

at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.submit(ThreadPoolTaskExecutor.java:284)

at org.springframework.aop.interceptor.AsyncExecutionAspectSupport.doSubmit(AsyncExecutionAspectSupport.java:186)

at org.springframework.aop.interceptor.AsyncExecutionInterceptor.invoke(AsyncExecutionInterceptor.java:123)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)

at com.sun.proxy.$Proxy84.processMessage(Unknown Source)

at com.osmosix.commons.messaging.listeners.impl.DefaultNodeMessageListener.handleMessage(DefaultNodeMessageListener.java:35)

at sun.reflect.GeneratedMethodAccessor78.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:269)

at org.springframework.amqp.rabbit.listener.adapter.MessageListenerAdapter.invokeListenerMethod(MessageListenerAdapter.java:383)

... 12 more

Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3911a213 rejected from java.util.concurrent.ThreadPoolExecutor@6737f4fb[Running, pool size = 64, active threads = 64, queued tasks = 3000, completed tasks = 413]

at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)

at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)

at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)

at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)

at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.submit(ThreadPoolTaskExecutor.java:281)

... 23 more

2016-10-19 02:32:32,205 INFO annotation.RequestMappingHandlerMapping [localhost-startStop-1] - Mapped "{[/image/service/dltargetlocal],methods=[POST]}" onto public java.util.Map<java.lang.String, ?> com.osmosix.gateway.image.transform.ImageTransformController.downloadToTargetLocal(com.osmosix.commons.image.transform.DownloadImageFileRequest)

2016-10-19 02:32:32,206 WARN listener.ConditionalRejectingErrorHandler [SimpleAsyncTaskExecutor-1] - Execution of Rabbit message listener failed.

org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener method 'handleMessage' threw exception

at org.springframework.amqp.rabbit.listener.adapter.MessageListenerAdapter.invokeListenerMethod(MessageListenerAdapter.java:391)

at org.springframework.amqp.rabbit.listener.adapter.MessageListenerAdapter.onMessage(MessageListenerAdapter.java:294)

at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:757)

at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:680)

at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:93)

at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:183)

at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1352)

at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:661)

at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1096)

at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1080)

at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$800(SimpleMessageListenerContainer.java:93)

at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1197)

at java.lang.Thread.run(Thread.java:745)

Resolution

Step 1.  Login to RabbitMQ server via SSH.

Step 2. Switch to root user.

# sudo -i

Step 3. Enable Manegement Plugin.

# rabbitmq-plugins enable rabbitmq_management

Step 4. Enable access to rabbitmqadmin command.

# chmod +x `find /var/lib/rabbitmq/ -name "rabbitmqadmin"`

# ln -s `find /var/lib/rabbitmq/ -name "rabbitmqadmin"` /usr/sbin

Step 5. RabbitMQ can be accessed via GUI.

http://<RabbitMQ_Server_IP>:15672 (username=cliqr password=cliqr)

Note: Replace RabbitMQ_Server_IP in the configuration with your RabbitMQ server address

Navigate queues tab to see the details

or

CLI:

# rabbitmqadmin -V /cliqr -u cliqr -p cliqr list queues

# rabbitmqadmin -V /cliqr -u cliqr -p cliqr list exchanges

# rabbitmqadmin -V /cliqr -u cliqr -p cliqr list channels

Step 6. To delete the gateway heartbeat queue.

#rabbitmqadmin -V /cliqr delete queue name=cliqr.gateway.heartbeat.queue --username=cliqr --password=cliqr

Step 7.  Restart tomcat service in RabbitServer.

#/etc/init.d/tomcatgua stop

#/etc/init.d/tomcatgua start

Note: If cliqr.gateway.heartbeat.queue is missing, restart CCO tomcat service by following the below steps:

Step 8.  Login to CCO server via SSH.

Step 9. Switch to root user.

# sudo -i

Step 10. Restart Tomcat server.

#/etc/init.d/tomcat stop

#/etc/init.d/tomcat start