A FireSIGHT Management Center or a FirePOWER appliance can run out of disk space for various reasons. When happens, the high disk utilization triggers health alert or may fail a software update attempt. This article describes the root causes of excessive disk utilization and some troubleshooting steps.
Determine the partition that is highly used. The following command shows the disk utilization:
On a FireSIGHT Management Center,
admin@3DSystem:~# df -TH
On 7000 and 8000 Series appliances and on NGIPS virtual devices,
> show disk
Both commands show an output like below:
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 2.9G 566M 2.2G 21% /
/dev/sda1 99M 16M 79M 17% /boot
/dev/sda7 52G 8.5G 41G 18% /Volume
none 11G 20K 11G 1% /dev/shm
/dev/sdb1 418G 210M 395G 1% /var/storage
On 7000 and 8000 Series appliances and on NGIPS virtual devices, you can run the following command to display detailed disk usage statistics:
> show disk-manager
An example output:
> show disk-manager
Silo Used Minimum Maximum
Temporary Files 143.702 MB 402.541 MB 1.572 GB
Action Queue Results 0 KB 402.541 MB 1.572 GB
Connection Events 17.225 GB 3.931 GB 23.586 GB
User Identity Events 0 KB 402.541 MB 1.572 GB
UI Caches 587 KB 1.179 GB 2.359 GB
Backups 0 KB 3.145 GB 7.862 GB
Updates 13 KB 4.717 GB 11.793 GB
Other Detection Engine 0 KB 2.359 GB 4.717 GB
Performance Statistics 72.442 MB 805.082 MB 9.435 GB
Other Events 669.819 MB 1.572 GB 3.145 GB
IP Reputation & URL Filtering 0 KB 1.966 GB 3.931 GB
Archives & Cores & File Logs 1.381 GB 3.145 GB 15.724 GB
RNA Events 0 KB 3.145 GB 12.579 GB
File Capture 12.089 MB 4.717 GB 14.152 GB
IPS Events 3.389 GB 7.076 GB 15.724 GB
If the /Volume Partition is Full
Old Backup Files
- If you store large volume of old backup files on the system, it can take excessive space on your disk.
- Delete the old backup files using the web user interface. In order to remove backup files, navigate to System > Tools > Backup/Restore.
Older Software Update and Patch Files
- If you always keep the previous software update, upgrade, and patch files (such as, 5.0 or 5.1), the system can run out disk space.
- Delete the older update and patch files that are no longer necessary. In order to delete them, please navigate to System > Updates.
Excessive Event Files Are Stored
- Managed device or sensor might have stopped sending events to the FireSIGHT Management Center.
- A device may be generating more events than a Management Center is designed to receive (per second).
- There might be a communication issue between the managed device and the management center.
- Reapply the policy that are related to the event. For example, if you are not seeing connection events, reapply the Access Controil policy and see if any new events are now being received by the Management Center.
- If a FireSIGHT Management Center is unable to receive new IPS events, please check if there is any communication issues between the managed device and the management center.
Excessive Unknown Files
- The FireSIGHT System stores the unknown Network Discovery data (OS, host and service information).
- If the system cannot determine the operating system on a host on your network, you can use Nmap to actively scan the host. Nmap uses the information it obtains from the scan to rate the possible operating systems. It then uses the operating system that has the highest rating as the host operating system identification.
- Create a correlation rule that triggers when the system detects a host with an unknown operating system.
The rule should trigger when an discovery event occurs and the OS information for a host has changed and it meets the following conditions: OS Name is unknown.
Large Database to Store Events
- If you increase the database event limit beyond the guideline or best practice, the FireSIGHT Management Center can run out of disk space.
- Check the values of the database limit. To improve disk utilization and performance, you should tailor event limits to the number of events you regularly work with. For some event types, you can disable storage.
- In order to change the database limit, please navigate to the System Policy page, click Edit next to the name of the system policy, and then click Database on the left section. To access the System Policy page, please navigate to System > Local > System Policy.
Receive Health Alerts For Over 85% Disk Utilization
- The event rate may be very high. Therefore the device is generating and storing lots of events.
- Communication problems between the managed device and FireSIGHT Management Center.
- Changing the alert threshold level to 87% (Warning) and 92% (Critical) can be a simple solution to frequent health alerts.
- Read the Release Notes to see if there was a known issue with the pruning system. When a solution is available, please update the software version to the latest release to address this issue.
The /var/log/messages files contain data older than 24 hours, or larger than 25MB
- Logrotate daemon may not be working properly.
- If you encounter this issue, please update the software version of your FireSIGHT Systems to the latest release. If you are running the latest version, but still experiencing this issue, please contact Cisco Technical Assistance Center (TAC).
If the Root ( / ) Partition is Full
User Files are Saved on the Root ( / ) Partition
- The root ( / ) partition is a fixed size and is not intended for personal storage.
- The /var/tmp drectory is used manually for temporary storage, instead of the /var/common directory.
- Check for unnecessary files on the /root, /home, and /tmp folder. Since these folders are not created for personal storage, you can delete any personal file with rm command.
Unsupported Processes are Writing to Root ( / ) Partition
- If you install third party software which creates files on root ( / ) partition, you can experience health alert for high disk usage.