Field Notice: FN - 70441 - Firepower Software - mysql-server.err Log File Might Consume Excessive Disk Space - Workaround Provided

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	03-Sep-19	Initial Release

Products Affected

Affected OS Type	Affected Software Product	Affected Release	Affected Release Number	Comments
NON-IOS	Firepower Management Center Software	6.1	6.1.0, 6.1.0.1, 6.1.0.2, 6.1.0.3, 6.1.0.4, 6.1.0.5, 6.1.0.6, 6.1.0.7
NON-IOS	Firepower Management Center Software	6.2	6.2.0, 6.2.0.1, 6.2.0.2, 6.2.0.3, 6.2.0.4, 6.2.0.5, 6.2.0.6, 6.2.1, 6.2.2, 6.2.2.1, 6.2.2.2, 6.2.2.3, 6.2.2.4, 6.2.2.5, 6.2.3, 6.2.3.1, 6.2.3.10, 6.2.3.11, 6.2.3.12, 6.2.3.13, 6.2.3.14, 6.2.3.2, 6.2.3.3, 6.2.3.4, 6.2.3.5, 6.2.3.6, 6.2.3.7, 6.2.3.8, 6.2.3.9
NON-IOS	Firepower Management Center Software	6.3	6.3.0, 6.3.0.1, 6.3.0.2, 6.3.0.3, 6.3.0.4
NON-IOS	Firepower Management Center Software	6.4	6.4.0, 6.4.0.1, 6.4.0.2, 6.4.0.3, 6.4.0.4

Defect Information

Defect ID	Headline
CSCvn30118	mysql-server.err file is not fully deleted and keeps consuming Firepower disk space
CSCvq71217	High Disk Utilization due to mysql-server.err failing to rotate after CSCvn30118

Problem Description

The mysql-server.err log file might consume excessive disk space on the Firepower security appliance.

Background

Affected Firepower software versions might experience extended high disk utilization due to a failure to properly rotate large mysql-server.err files. This occurs after large mysql-server.err log files are rotated with the logrotate command. Although the file is successfully rotated and compressed, the pre-rotation disk space continues to be consumed since the mysql process maintains an open file handle to the deleted file. This results in continued high disk utilization until either the mysql process or the Firepower platform is restarted to release the disk resources.

Problem Symptom

Use the show disk command in order to view the ngfw disk utilization to determine if the condition exists. The related information is highlighted in this command output.

> show disk
Filesystem Size Used Avail Use% Mounted on
rootfs 114G 437M 114G 1% /
devtmpfs 114G 2.5G 112G 3% /dev
tmpfs 126G 1.8M 126G 1% /run
tmpfs 126G 1.7M 126G 1% /var/volatile
/dev/sda1 7.5G 161M 7.3G 3% /mnt/boot
/dev/sda2 1.9G 32M 1.7G 2% /opt/cisco/config
/dev/sda3 4.5G 22M 4.3G 1% /opt/cisco/platform/logs
/dev/sda5 46G 53M 44G 1% /var/data/cores
/dev/sda6 191G 180G 12G 94% /ngfw     <----- Output shows high disk utilization
cgroup_root 126G 0 126G 0% /dev/cgroups

Complete these steps in order to determine if the high ngfw disk utilization is due to the mysql-server.err log file:

1. Prior to log file rotation, enter this command in order to show the disk space consumed by the mysql-server.err file.

   root@Firepower:# ls -alh /ngfw/var/lib/mysql/mysql-server*

-rw-rw---- 1 mysql mysql 155G Nov 14 13:06 mysql-server.err     <----- Output shows 155G log file

2. After the log file has been rotated, enter this command in order to show the disk space consumed by the rotated and compressed mysql-server.err file.

   root@Firepower:# ls -alh /ngfw/var/lib/mysql/mysql-server*

-rw-rw---- 1 mysql mysql    0 Nov 15 04:43 mysql-server.err
-rw-rw---- 1 mysql mysql   20 Nov 14 13:06 mysql-server.err.1.gz     <----- Rotated and compressed
-rw-rw---- 1 mysql mysql  11G Nov 14 13:06 mysql-server.err.2.gz

3. After the log file has been rotated, enter this command in order to show the disk space consumed by the original mysql-server.err file. The seventh column of the output shows that the mysql-server.err log file continues to consume disk space.

   root@Firepower:# lsof | grep mysql-server.err.*deleted

mysqld    37580            mysql    1w      REG                8,6 165427995057      10344 /ngfw/var/lib/mysql/mysql-server.err.1 (deleted)     <----- Unrotated file continues to consume 155G of disk space.

Workaround/Solution

Enter the flush logs command in order to manually delete the mysql-server.err log file:

root@Firepower:# OmniQuery.pl -db mdb -e "flush logs;"

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email
• By telephone

Receive Email Notification For New Field Notices

Cisco Notification Service—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.