This document describes a problem that is encountered with the Cisco Unified Intelligence Center (CUIC) when the local Network Time Protocol (NTP) client is off by more than the acceptable threshold of three seconds from its remote NTP system peer. Solutions to this problem are also described.
You receive this syslog message:
The local NTP client is off by more than the acceptable threshold of 3 seconds from its remote NTP system peer. The normal remedy is for NTP Watch Dog to automatically restart NTP. However, an unusual number of automatic NTP restarts have already occurred on this node. No additional automatic NTP restarts will be done until NTP time synchronization stabilizes. This is likely due to an excessive number of VMware Virtual Machine migrations or Storage VMotions. Please consult your VMware Infrastructure Support Team.
This section describes two possible causes for the problem and the solutions.
The syslog is filled with the previously described NTP messages because you have more than one NTP server configured, and they might not be properly synced.
You must remove the additional NTP servers and generate a new license. Once this is complete, the NTP syslog messages are cleared.
There is no available space in the CUIC.
With CUIC Version 9.0(1), new CLI commands are introduced so that you can delete the files via normal CLI access. Therefore, you do not need root access through the Cisco Technical Assistance Center (TAC).
Enter the file delete activelog /cuic/logs/jmx/*.log noconfirm command on the secondary node in order to delete the jmx.log files. In this example, the disk usage is reduced to 77 percent after the command is entered:
Total Free Used
Disk/active 27087452K 15555208K 11257052K (42%)
Disk/inactive 27087420K 14989416K 11822812K (45%)
Disk/logging 94235980K 20874792K 68574224K (77%)
Enter these commands on the primary node in order to delete the CUIC logs for 2013:
- file delete activelog /cuic/logs/cuic/CCBU-cuic.2013* noconfirm
- file delete activelog /cuic/logs/cuic/ERROR-cuic.2013* noconfirm
For future issues, you can enter the show disk usage common directory sort command in order to identify the directories that have the largest size and delete the log files based on the output. However, the deletion of the files described in this section is usually sufficient.
It is possible that the Real Time Monitor Tool (RTMT) might freeze when you attempt to open the syslog messages. This occurs because the size of the syslog message is 123 MB, which is too large.
With root access, you can navigate to var > log > active > syslog > messages and clean the content of the file. Once this is complete, the RTMT is able to open the file.