This document describes common issues with time synchronization on FireSIGHT Systems and how to troubleshoot them. You can choose to synchronize time between your FireSIGHT Systems in three different ways, such as manually with external Network Time Protocol (NTP) servers, or with FireSIGHT Management Center which serves as an NTP server. You can configure a FireSIGHT Management Center as a time server with NTP and then use it to synchronize time between the FireSIGHT Management Center and managed devices.
In order to configure the time synchronization setting, you need admin level of access on your FireSIGHT Management Center.
This document is not restricted to specific software and hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
FireSIGHT Management Center displays health alerts on the web interface.
The Health Monitor page shows an appliance as critial, because the status of Time Synchronization Module is out-of-sync.
You might see intermittent health alerts if the appliances fail to stay synchronized.
After a system policy is applied you might see health alerts, because a FireSIGHT Management Center and its managed devices could take up to 20 minutes to complete synchronization. This is because a FireSIGHT Management Center must first synchronize with its configured NTP server before it can serve time to a managed device.
The time between a FireSIGHT Management Center and a managed device does not match.
Events generated at the sensor might take minutes or hours to become visible on a FireSIGHT Management Center.
If you run virtual appliances and the Health Monitor page indicates that the clock setup for your virtual appliance is not synchronized, check your system policy time synchronization settings. Cisco recommends that you synchronize your virtual appliances to a physical NTP server. Do not synchronize your managed devices (virtual or physical) to a Virtual Defense Center.
Step 1: Verify NTP Configuration
How to Verify in Versions 5.4 and Earlier
Verify that NTP is enabled on the system policy that is applied on the FireSIGHT Systems. In order to verify that, complete these steps:
Choose System > Local > System Policy.
Edit the system policy applied on your FireSIGHT Systems.
Choose Time Synchronization.
Check if the FireSIGHT Management Center (also known as Defense Center or DC) has the clock set to Via NTP from, and an address of an NTP server is provided. Also confirm that the Managed Device is set to via NTP from Defense Center.
If you specify a remote external NTP server, your appliance must have network access to it. Do not specify an untrusted NTP server. Do not synchronize your managed devices (virtual or physical) to a Virtual FireSIGHT Management Center. Cisco recommends that you synchronize your virtual appliances to a physical NTP server.
How to Verify in Versions 6.0 and Later
In versions 6.0.0 and later, the time synchronization settings are configured in separate places on the Firepower Management Center, though they follow the same logic as the steps for 5.4.
The time synchronization settings for the Firepower Management Center itself are found under System > Configuration > Time Synchronization.
The time synchronization settings for the managed devices are found under Devices > Platform Settings. Click edit next to the Platform Settings policy applied to the device and then choose Time Synchronization.
After you apply the configuration for time synchronization (regardless of version), make sure that the time on your Management Center and managed devices matches. Otherwise, unintended consequences might occur when the managed devices communicate with the Management Center.
Step 2: Identify a Timeserver and It's Status
In order to gather information about the connection to a time server, enter this command on your FireSIGHT Management Center:
admin@FireSIGHT:~$ ntpq -pn
remote refid st t when poll reach delay offset jitter ============================================================================== *198.51.100.2 203.0.113.3 2 u 417 1024 377 76.814 3.458 1.992
An asterisk '*' under the remote indicates the server you are currently synchronized to. If an entry with an asterisk is unavailable, the clock is currently not synchronised with it's timesource.
On a managed device, you can enter this command on shell in order to determine the address of your NTP server:
> show ntp
NTP Server : 127.0.0.2 (Cannot Resolve) Status : Being Used Offset : -8.344 (milliseconds) Last Update : 188 (seconds)
Note: If a managed device is configured to receive time from a FireSIGHT Management Center, the device shows a timesource with loopback address, such as 127.0.0.2. This IP address is an sfipproxy entry and indicates that the Managment Virtual Network is being used to synchronize time.
If an appliance displays that it syncs with 127.127.1.1, it indicates that the appliance syncs with its own clock. It occurs when a timeserver configured on a system policy is not synchronizable. For example:
admin@FirePOWER:~$ ntpq -pn
remote refid st t when poll reach delay offset jitter ============================================================================== 192.0.2.200 .INIT. 16 u - 1024 0 0.000 0.000 0.000 *127.127.1.1 .SFCL. 14 l 3 64 377 0.000 0.000 0.001
On the ntpq command output, if you notice the value of st (stratum) is 16, it indicates that the timeserver is unreachable and the appliance is not able to sychronize with that timeserver.
On the ntpq command output, reach shows an octal number that indicates success or failure to reach source for the most recent eight polling attempts. If you see the value is 377, it means the last 8 attempts were successful. Any other values might indicate that one or more of the last eight attempts were unsuccessful.
Step 3: Verify Connectivity
Check the basic connectivity to the time server.
admin@FireSIGHT:~$ ping <IP_addres_of_NTP_server>
Ensure that port 123 is open on your FireSIGHT System.
admin@FireSIGHT:~$ netstat -an | grep 123
Confirm that port 123 is open on the firewall.
Check the hardware clock:
admin@FireSIGHT:~$ sudo hwclock
If the hardware clock is too far out of date, they might never successfully sync. In order to manually force the clock to be set with a time server, enter this command:
Make sure that the Universally Unique Identifier (UUID) under the peers section matches with the ims.conf file the peer. For example, the UUID found under the peers section of the /etc/sf/sfipproxy.conf file on a FireSIGHT Management Center should match with the UUID found on the /etc/ims.conf file of its managed device. Similarly, the UUID found under the peers section of the /etc/sf/sfipproxy.conf file on a managed device should match with the UUID found on the /etc/ims.conf file of its management appliance.
You can retrieve the UUID of the devices with this command:
These should normally be automatically populated by the system policy, but there have been cases where these stanzas were missing. If they need to be modified or changed you will need to restart sfipproxy and sftunnel as follows:
Verify if the /etc/ntp.conf file is populated correctly. When you apply a system policy, the ntp.conf file is rewritten.
Note: The output of an ntp.conf file shows the timeserver settings configured on a system policy. The time stamp entry should show the time when the last system policy applied to a device. The server entry should should show the specified timeserver address.
admin@FireSIGHT:~$ sudo cat /etc/ntp.conf
# automatically generated by /etc/sysconfig/configure-network ; do not edit # Tue Oct 21 17:44:03 UTC 2014