Field Notice: FN - 72500 - Prime Infrastructure Server Crashes when /var/log/audit Log Fills and Hits 100% - Workaround Provided

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	23-Jan-23	Initial Release

Products Affected

Affected OS Type	Affected Software Product	Affected Release	Affected Release Number	Comments
NON-IOS	Prime Infrastructure Software	3.8	3.8.0, 3.8.0-FED, Prime3.8MR1EFT	3.8.x, 3.9.x, 3.10, 3.10.1
NON-IOS	Prime Infrastructure Software	3.9	3.9.0	3.8.x, 3.9.x, 3.10, 3.10.1
NON-IOS	Prime Infrastructure Software	3.10	3.10.0	3.8.x, 3.9.x, 3.10, 3.10.1

Defect Information

Defect ID	Headline
CSCwa95488	/var/log/audit log piling and hitting 100% crashes the server
CSCvx62477	Logrotate process not rotating logs in /var/log/audit - Prime Infrastructure 3.9

Problem Description

Prime Infrastructure services stop operating, and a restart of services fails due to no space available in the /var partition. The system crashes because the /var partition fills the /var/log/ directory with audit log file instances and utilizes 100% of the available space.

The workaround is provided in the Workaround/Solution section.

Background

Prime Infrastructure Versions 3.8.X, 3.9.X, 3.10, and 3.10.1 are impacted by the /var partition filling to capacity. Additionally, Prime Infrastructure Version 3.9 has a failure in the logrotate process. A fix for the /var partition filling to capacity was addressed in defect CSCvx62477, but new scenarios were identified where the previous fix did not eliminate the issue. This issue is resolved and available in Version 3.10.2.

In many cases, the audit.log is filled with frequent system level messages that have root causes that need to be individually analyzed. Log files should rotate/clean themselves via routine maintenance, but are not currently successful due to a failure in the logrotate process or because the log grows significantly before logrotate can execute.

Problem Symptom

Prime Infrastructure status (ncs status) shows that the database has stopped or the Network Management System (NMS) Server has stopped.

Workaround/Solution

Customers should contact the Technical Assistance Center (TAC) to have these actions performed on the affected system. TAC will need permission to gain access to the system and approval to delete or relocate files.

1. Identify the larger log files under the /var/log directory.
2. If audit.log file instances are found to occupy too much space, delete the larger log files to recover more space in this partition.
3. Restart Prime Infrastructure services once the /var partition has available free space.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email or telephone

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.