What does the alert "work queue paused, XX msgs, antispam" or "work queue paused, XX msgs, antivirus" mean?

Introduction

This document describes the meaning of critical alert notification for work queue paused events related to the anti-spam or anti-virus processes, how to attempt to correct these events, and post-event notifications related to the Cisco Email Security Appliance (ESA).

What does the alert "work queue paused, XX msgs, antispam" or "work queue paused, XX msgs, antivirus" mean?

When anti-spam and anti-virus are in the process of updating their respective rules and services, the process will pause the processing of messages in the work queue until completely downloaded and updated. On certain occasions, the update files may be larger than usual, and it takes a bit longer to process and updated - thus causing an alert to be generated.

There are also times if your appliance has had a network interruption, improper reboot, or other service impacting event, that the update process had become stale, or corrupted, and did not complete successfully.

Example, work queue paused on antispam:

The Critical message is:

work queue paused, 98 msgs, antispam

Version: 8.5.6-074
Serial Number: XXYYDBE08931-XXYYKN1
Timestamp: 28 Apr 2014 11:37:39 -0500

Example, work queue paused on antivirus:

The Critical message is:

work queue paused, 134 msgs, antivirus

Version: 8.0.1-023
Serial Number: 848F69E7XXYY-XXYYJ5J
Timestamp: 06 May 2014 13:56:38 +0200

How can you correct this work queue paused event?

On the CLI you can force a complete update via the command antispamupdate ironport force or antivirusupdate force.  A complete update is when the ESA will reach out to the Cisco update servers and pull the complete and most recent IDE, and also will pull the complete and most recent anti-spam or anti-virus engine, and reapply this in the background on your appliance.

Example of antispamupdate ironport force:

> antispamupdate ironport force

Forcing updates for CASE rules.

Example of antivirusupdate force:

> antivirusupdate force

Sophos Anti-Virus updates:
Requesting forced update of Sophos Anti-Virus.
McAfee Anti-Virus updates:
Requesting update of virus definitions

Verification

You can view the process of the anti-virus updates my running tail updater_logs from the CLI on the ESA.  This assure you of the appliance's communication with the Cisco update servers and manifest, and allow you to see the update complete.  

Example of the updater_logs and the anti-spam update success:

Mon Aug 18 11:32:55 2014 Info: case verifying applied files
Mon Aug 18 11:32:55 2014 Info: case updating the client manifest
Mon Aug 18 11:32:55 2014 Info: case update completed
Mon Aug 18 11:32:55 2014 Info: case waiting for new updates

 Example of the updater_logs and the anti-virus update success:

Wed Jul 23 09:41:13 2014 Info: sophos verifying applied files
Wed Jul 23 09:41:13 2014 Info: sophos updating the client manifest
Wed Jul 23 09:41:13 2014 Info: sophos update completed
Wed Jul 23 09:41:13 2014 Info: sophos waiting for new updates

You will want to assure that you see the lines above, which will indicate the successful request and update of the requested associated updates.

What are "work queue resumed" notifications?

Post-successful return to operation of work queue after a paused event, you will receive notification of, "work queue resumed, XX msgs." This means that the work queue's message processing has automatically resumed. These alerts are more of an informational type alert and no further action is required once the work queue resumes processing.

Example:

The Critical message is:

work queue resumed, 0 msgs

Version: 8.0.1-023
Serial Number: 848F69E7XXYY-XXYYJ5J
Timestamp: 06 May 2014 14:00:00 +0200

You may also wish to validate the work queue status directly with workqueue status and workqueue rate 10.  The workqueue status command will validate if the work queue is operational, still paused, or offline, for example.  And the workqueue rate 10 will present the operational output of the actual work queue, in terms of messages pending, message in, and messages out, and provide this updated snapshot each 10 seconds of processing.

Example:

> workqueue status

Status as of: Mon Aug 18 11:49:59 2014 EDT
Status: Operational
Messages: 5

> workqueue rate 10

Type Ctrl-C to return to the main prompt.

Time     Pending In Out
11:50:06       0  5   5
11:50:16       0  0   0
11:50:26       0  0   0

Note: From the CLI, please use the workqueue command for full control and operation of the work queue for your appliance.

What should I do if I do not recieve a "work queue resumed" notification?

If you do not recieve a "work queue resumed" notificaiton after 15 minutes from the original "work queue paused" notification, this can indicate a more serious problem with the ESA's work queue. From the CLI, please run status and check whether the "System status" shows "Online" and mail is indeed getting processed in the work queue. If it still shows "Work Queue Paused", please contact Cisco Technical Assistance further assistance.

Example of status and expected output:

> status

Enter "status detail" for more information.

Status as of: Mon Aug 18 11:44:45 2014 EDT
Up since: Wed Aug 13 17:06:09 2014 EDT (4d 18h 38m 36s)
Last counter reset: Never
System status: Online
Oldest Message: No Messages
Feature - Sophos Anti-Virus: 334 days
Feature - Bounce Verification: 334 days
Feature - IronPort Anti-Spam: 334 days
Feature - IronPort Email Encryption: 334 days
Feature - RSA Email Data Loss Prevention: 334 days
Feature - Incoming Mail Handling: 335 days
Feature - Outbreak Filters: 334 days

Counters: Reset Uptime Lifetime
 Receiving
 Messages Received 132 17 132
 Recipients Received 134 18 134
 Rejection
 Rejected Recipients 1,400 182 1,400
 Dropped Messages 0 0 0
 Queue
 Soft Bounced Events 51 0 51
 Completion
 Completed Recipients 156 20 156
 Current IDs
 Message ID (MID) 1556
 Injection Conn. ID (ICID) 11208
 Delivery Conn. ID (DCID) 178

Gauges: Current
 Connections
 Current Inbound Conn. 0
 Current Outbound Conn. 0
 Queue
 Active Recipients 0
 Messages In Work Queue 0
 Kilobytes Used 0
 Kilobytes Free 8,388,608
 Messages In Quarantine
 Policy, Virus and Outbreak 0
 Kilobytes In Quarantine
 Policy, Virus and Outbreak 0

Related Information

• Cisco Email Security Appliance - End-User Guides
• ESA FAQ: How do I force a download of Sophos or McAfee Anti-Virus updates immediately?
• Technical Support & Documentation - Cisco Systems