Contents

Introduction

Prerequisites

Requirements

Components Used

Background information

Before you Begin

Troubleshoot commands

FMC troubleshooting commands.

FTD troubleshooting commands

Verification

FMC - HA validation

Communication from FTD to FMC-HA validation

Introduction

This document describes how to troubleshoot common synchronization issues in a High Availability (HA) set for the Firepower Management Center (FMC).

Prerequisites

Requirements

Cisco recommends that you have knowledge of the following topics:

- FMC - HA configuration requirements

- Basic knowledge of Linux shell.

Components Used

- FMCv for VMware on version 7.2.8.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background information

The initial setup of the laboratory used for this document follows the requirements for the basic FMC-HA initial configuration.

- Two FMCs with the same capacity or hardware version.

- Two FMCs running the same software version, Intrusion Rule Update, Vulnerability Database, and Lightweight Security Package.

- Two FMCs with the corresponding licenses.

Before you Begin

- Ensure administrator has access to both FMCs.

- Ensure administrator has access to the FTD devices managed by the FMC.

Troubleshoot commands

FMC troubleshooting commands.

To validate the connectivity between FMC devices, the user can run these commands.

> expert

admin@firepower:~$ sudo su

root@firepower:/Volume/home/admin# ping xx.xx.18.102

PING xx.xx.18.102 (xx.xx.18.102) 56(84) bytes of data.

64 bytes from xx.xx.18.102: icmp_seq=1 ttl=64 time=0.533 ms

64 bytes from xx.xx.18.102: icmp_seq=2 ttl=64 time=0.563 ms

64 bytes from xx.xx.18.102: icmp_seq=3 ttl=64 time=0.431 ms

^C

--- xx.xx.18.102 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 59ms

rtt min/avg/max/mdev = 0.431/0.509/0.563/0.056 ms

root@firepower:/Volume/home/admin# netstat -an | grep 8305

tcp 0 0 xx.xx.18.101:8305 0.0.0.0:* LISTEN

tcp 0 0 xx.xx.18.101:8305 xx.xx.18.253:48759 ESTABLISHED

tcp 0 0 xx.xx.18.101:8305 xx.xx.18.254:53875 ESTABLISHED

tcp 0 0 xx.xx.18.101:8305 xx.xx.18.254:49205 ESTABLISHED

tcp 0 0 xx.xx.18.101:60871 xx.xx.18.253:8305 ESTABLISHE

ping <peer-ip-address> This command can be used to check the reachability between both devices.

netstat -an | grep 8305 This command displays the devices connected to port 8305.

Note: The port 8305 is the default port configured on the Firepower devices to establish the communication channel with the FMC.

To validate the FMC-HA setup configuration, the user can also run the script troubleshoot_HADC.pl. This is particularly useful in these scenarios:

- When the FMC-HA integration health status is degraded.

- If the access to the FMC Graphic User Interface (GUI) of one of the devices has been missing, but the FMC-CLI access is still working and accessible.

> expert

admin@firepower:~$ sudo su

root@firepower:/Volume/home/admin# troubleshoot_HADC.pl

**************** Troubleshooting Utility **************

1 Show HA Info Of FMC

2 Execute Sybase DBPing

3 Show Arbiter Status

4 Check Peer Connectivity

5 Print Messages of AQ Task

6 Show FMC HA Operations History (ASC order)

7 Dump To File: FMC HA Operations History (ASC order)

8 Last Successful Periodic Sync Time (When it completed)

9 Print HA Status Messages

10 Compare active and standby device list

11 Check manager status of standby missing devices

12 Check critical PM processes details

13 Get Remote Stale Sync AQ Info

14 Help

0 Exit

**************************************************************

Enter choice:

FTD troubleshooting commands

Troubleshooting the connectivity from the FTD to the FMC-HA allows the user to validate connectivity of devices that need to be registered on both FMCs or when HA is degraded, and displays the warning "Degraded – Synchronization incomplete (This Management Center has fewer devices registered)".

From the FTD clish level, the user can run these commands to validate the communication with the FMC.

> ping system xx.xx.18.102

PING xx.xx.18.102 (xx.xx.18.102) 56(84) bytes of data.

64 bytes from xx.xx.18.102: icmp_seq=1 ttl=64 time=0.595 ms

64 bytes from xx.xx.18.102: icmp_seq=2 ttl=64 time=0.683 ms

64 bytes from xx.xx.18.102: icmp_seq=3 ttl=64 time=0.642 ms

64 bytes from xx.xx.18.102: icmp_seq=4 ttl=64 time=24.4 ms

64 bytes from xx.xx.18.102: icmp_seq=5 ttl=64 time=11.4 ms

^C

--- xx.xx.18.102 ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 128ms

rtt min/avg/max/mdev = 0.595/7.545/24.373/9.395 ms

> show managers

Type : Manager

Host : xx.xx..18.101

Display name : xx.xx..18.101

Version : 7.2.8 (Build 25)

Identifier : fc3e3572-xxxx-xxxx-xxxx-39e0098c166c

Registration : Completed

Management type : Configuration and analytics

Type : Manager

Host : xx.xx..18.102

Display name : xx.xx..18.102

Version : 7.2.8 (Build 25)

Identifier : bb333216-xxxx-xxxx-xxxx-c68c0c388b44

Registration : Completed

Management type : Configuration and analytics

> sftunnel-status

SFTUNNEL Start Time: Mon Oct 14 21:29:16 2024

Both IPv4 and IPv6 connectivity is supported

Broadcast count = 5

Reserved SSL connections: 0

Management Interfaces: 2

eth0 (control events) xx.xx..18.254,

tap_nlp (control events) 169.254.1.2,fd00:0:0:1::2

***********************

**RUN STATUS****xx.xx..18.102*************

Key File = /var/sf/peers/bb333216-xxxx-xxxx-xxxx-c68c0c388b44/sftunnel-key.pem

Cert File = /var/sf/peers/bb333216-xxxx-xxxx-xxxx-c68c0c388b44/sftunnel-cert.pem

CA Cert = /var/sf/peers/bb333216-xxxx-xxxx-xxxx-c68c0c388b44/cacert.pem

Cipher used = TLS_AES_256_GCM_SHA384 (strength:256 bits)

ChannelA Connected: Yes, Interface eth0

Cipher used = TLS_AES_256_GCM_SHA384 (strength:256 bits)

ChannelB Connected: Yes, Interface eth0

Registration: Completed.

IPv4 Connection to peer 'xx.xx..18.102' Start Time: Tue Oct 15 00:38:43 2024 UTC

IPv4 Last outbound connection to peer 'xx.xx..18.102' via Primary ip/host 'xx.xx..18.102'

PEER INFO:

sw_version 7.2.8

sw_build 25

Using light registration

Management Interfaces: 1

eth0 (control events) xx.xx..18.102,

Peer channel Channel-A is valid type (CONTROL), using 'eth0', connected to 'xx.xx..18.102' via 'xx.xx..18.254'

Peer channel Channel-B is valid type (EVENT), using 'eth0', connected to 'xx.xx..18.102' via 'xx.xx..18.254'

***********************

**RUN STATUS****xx.xx..18.101*************

Key File = /var/sf/peers/fc3e3572-xxxx-xxxx-xxxx-39e0098c166c/sftunnel-key.pem

Cert File = /var/sf/peers/fc3e3572-xxxx-xxxx-xxxx-39e0098c166c/sftunnel-cert.pem

CA Cert = /var/sf/peers/fc3e3572-xxxx-xxxx-xxxx-39e0098c166c/cacert.pem

Cipher used = TLS_AES_256_GCM_SHA384 (strength:256 bits)

ChannelA Connected: Yes, Interface eth0

Cipher used = TLS_AES_256_GCM_SHA384 (strength:256 bits)

ChannelB Connected: Yes, Interface eth0

Registration: Completed.

IPv4 Connection to peer 'xx.xx..18.101' Start Time: Mon Oct 14 21:29:15 2024 UTC

IPv4 Last outbound connection to peer 'xx.xx..18.101' via Primary ip/host 'xx.xx..18.101'

PEER INFO:

sw_version 7.2.8

sw_build 25

Using light registration

Management Interfaces: 1

eth0 (control events) xx.xx..18.101,

Peer channel Channel-A is valid type (CONTROL), using 'eth0', connected to 'xx.xx..18.101' via 'xx.xx..18.254'

Peer channel Channel-B is valid type (EVENT), using 'eth0', connected to 'xx.xx..18.101' via 'xx.xx..18.254'

***********************

**RPC STATUS****xx.xx..18.102*************

'uuid' => 'bb333216-xxxx-xxxx-xxxx-c68c0c388b44',

'uuid_gw' => '',

'last_changed' => 'Wed Oct 9 07:00:11 2024',

'active' => 1,

'name' => 'xx.xx..18.102',

'ip' => 'xx.xx..18.102',

'ipv6' => 'IPv6 is not configured for management'

**RPC STATUS****xx.xx..18.101*************

'uuid_gw' => '',

'uuid' => 'fc3e3572-xxxx-xxxx-xxxx-39e0098c166c',

'last_changed' => 'Mon Jun 10 18:59:54 2024',

'active' => 1,

'ip' => 'xx.xx..18.101',

'ipv6' => 'IPv6 is not configured for management',

'name' => 'xx.xx..18.101'

Check routes:

No peers to check

ping system <fmc-IP> To generate an ICMP, follow from the FTD management interface.

show managers This command lists the information of the managers where the device is registered.

sftunnel-status This command validates the communication channel established between the devices. This channel receives the name of sftunnel.

The commands to check the connectivity at the root level on the FTD are the same as the FMC. In the case of the FTD, it does not include a script that allows to validation of communication with the FMC, but it is possible to check the information generated during the registration process in the /ngfw/var/log/action.log.

Verification

For the next topology, the communication between the FMC-HA peers and the FTD01 can be validated using the commands previously described.

FMC-HA Topology

FMC - HA validation

For this validation, the basic guidelines to set an FMC-HA can also be validated using the command show version.

FMC Active

> show version

-------------------[ firepower ]--------------------

Model                     : Secure Firewall Management Center for VMware (66) Version 7.2.8 (Build 25)

UUID                      : fc3e3572-xxxx-xxxx-xxxx-39e0098c166c

Rules update version      : 2023-11-29-001-vrt

LSP version               : lsp-rel-20231129-1200

VDB version               : 395

----------------------------------------------------

> expert

admin@firepower:~$ sudo su

root@firepower:/Volume/home/admin# ping xx.xx.18.102

PING xx.xx.18.102 (xx.xx.18.102) 56(84) bytes of data.

64 bytes from xx.xx.18.102: icmp_seq=1 ttl=64 time=0.385 ms

64 bytes from xx.xx.18.102: icmp_seq=2 ttl=64 time=0.433 ms

64 bytes from xx.xx.18.102: icmp_seq=3 ttl=64 time=0.606 ms

64 bytes from xx.xx.18.102: icmp_seq=4 ttl=64 time=0.480 ms

64 bytes from xx.xx.18.102: icmp_seq=5 ttl=64 time=0.524 ms

^C

--- xx.xx.18.102 ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 84ms

rtt min/avg/max/mdev = 0.385/0.485/0.606/0.079 ms

root@firepower:/Volume/home/admin# netstat -an | grep 8305

tcp        0      0 xx.xx.18.101:8305       xx.xx.18.254:53875      ESTABLISHED

tcp        0      0 xx.xx.18.101:8305       xx.xx.18.102:38239      ESTABLISHED ------ communication establish with FMC-Standby

tcp        0      0 xx.xx.18.101:8305       xx.xx.18.254:49205      ESTABLISHED

tcp        0      0 xx.xx.18.101:8305       xx.xx.18.253:34865      ESTABLISHED

tcp        0      0 xx.xx.18.101:60871      xx.xx.18.253:8305       ESTABLISHED

tcp        0      0 xx.xx.18.101:8305       xx.xx.18.102:42253      ESTABLISHED ------ communication establish with FMC-Standby

root@firepower:/Volume/home/admin# troubleshoot_HADC.pl

****************  Troubleshooting Utility  **************

1   Show HA Info Of FMC

2   Execute Sybase DBPing

3   Show Arbiter Status

4   Check Peer Connectivity

5   Print Messages of AQ Task

6   Show FMC HA Operations History (ASC order)

7   Dump To File: FMC HA Operations History (ASC order)

8   Last Successful Periodic Sync Time (When it completed)

9   Print HA Status Messages

10  Compare active and standby device list

11  Check manager status of standby missing devices

12  Check critical PM processes details

13  Get Remote Stale Sync AQ Info

14  Help

0   Exit

**************************************************************

Enter choice: 1

HA Enabled: Yes

This FMC Role In HA: Active - Primary

Status out put: vmsDbEngine (system,gui) - Running 5093

In vmsDbEngineStatus(): vmsDbEngine process is running at /usr/local/sf/lib/perl/5.24.4/SF/Synchronize/HADC.pm line 3748.

Sybase Process: Running (vmsDbEngine, theSybase PM Process is Running)

Sybase Database Connectivity: Accepting DB Connections.

Sybase Database Name: csm_primary

Sybase Role: Active

****************  Troubleshooting Utility  **************

1   Show HA Info Of FMC

2   Execute Sybase DBPing

3   Show Arbiter Status

4   Check Peer Connectivity

5   Print Messages of AQ Task

6   Show FMC HA Operations History (ASC order)

7   Dump To File: FMC HA Operations History (ASC order)

8   Last Successful Periodic Sync Time (When it completed)

9   Print HA Status Messages

10  Compare active and standby device list

11  Check manager status of standby missing devices

12  Check critical PM processes details

13  Get Remote Stale Sync AQ Info

14  Help

0   Exit

**************************************************************

Enter choice: 4

Peer UUID [Enter 'Return' For HA Peer(no UUID required)]:

Peer Is Connected

$VAR1 = {

'vip' => '',

'model_id' => 'E',

'ip' => 'xx.xx.18.102',

'persistent' => 0,

'sw_version' => '7.2.8',

'last_changed' => 1728457211,

'active' => 1,

'uuid' => 'bb333216-xxxx-xxxx-xxxx-c68c0c388b44',

'upgrade_version' => '',

'reg_state' => 0,

'model_number' => '66',

'primary_mgr' => 0,

'name' => 'xx.xx.18.102',

'uuid_gw' => '',

'ipv6' => undef,

'vip_local' => '',

'priority' => 0,

'reg_key' => '',

'vnet' => undef,

'role' => 0,

'mgmt_mac_address' => '00:50:56:B3:D1:07'

};

**************** Troubleshooting Utility **************

1 Show HA Info Of FMC

2 Execute Sybase DBPing

3 Show Arbiter Status

4 Check Peer Connectivity

5 Print Messages of AQ Task

6 Show FMC HA Operations History (ASC order)

7 Dump To File: FMC HA Operations History (ASC order)

8 Last Successful Periodic Sync Time (When it completed)

9 Print HA Status Messages

10 Compare active and standby device list

11 Check manager status of standby missing devices

12 Check critical PM processes details

13 Get Remote Stale Sync AQ Info

14 Help

0 Exit

**************************************************************

Enter choice: 8

---------------------------- Last periodic sync time details ------------------------------------

Last successful sync completed at: Wed Oct 16 16:44:23 2024 UTC

Current time: Wed Oct 16 16:46:34 2024 UTC

Last successful sync completed '2 minutes 11 seconds' ago.

---------------------------- Last periodic sync time details end ------------------------------------

**************** Troubleshooting Utility **************

1 Show HA Info Of FMC

2 Execute Sybase DBPing

3 Show Arbiter Status

4 Check Peer Connectivity

5 Print Messages of AQ Task

6 Show FMC HA Operations History (ASC order)

7 Dump To File: FMC HA Operations History (ASC order)

8 Last Successful Periodic Sync Time (When it completed)

9 Print HA Status Messages

10 Compare active and standby device list

11 Check manager status of standby missing devices

12 Check critical PM processes details

13 Get Remote Stale Sync AQ Info

14 Help

0 Exit

**************************************************************

Enter choice: 9

SYNC_ACTIVE: 1 at /usr/local/sf/lib/perl/5.24.4/SF/Synchronize.pm line 494, <STDIN> line 5.

Sybase state :  at /usr/local/sf/lib/perl/5.24.4/SF/Synchronize.pm line 701.

Sybase state :  at /usr/local/sf/lib/perl/5.24.4/SF/Synchronize.pm line 801.

Sync status : at /usr/local/sf/lib/perl/5.24.4/SF/Synchronize.pm line 802.

---------------------------- FMC HA status messages start ------------------------------------

Status: Healthy

---------------------------- FMC HA status messages end ------------------------------------

**************** Troubleshooting Utility **************

1 Show HA Info Of FMC

2 Execute Sybase DBPing

3 Show Arbiter Status

4 Check Peer Connectivity

5 Print Messages of AQ Task

6 Show FMC HA Operations History (ASC order)

7 Dump To File: FMC HA Operations History (ASC order)

8 Last Successful Periodic Sync Time (When it completed)

9 Print HA Status Messages

10 Compare active and standby device list

11 Check manager status of standby missing devices

12 Check critical PM processes details

13 Get Remote Stale Sync AQ Info

14 Help

0 Exit

**************************************************************

Enter choice: 10

Fetching standby missing device information...

Devices are in sync.

**************** Troubleshooting Utility **************

1 Show HA Info Of FMC

2 Execute Sybase DBPing

3 Show Arbiter Status

4 Check Peer Connectivity

5 Print Messages of AQ Task

6 Show FMC HA Operations History (ASC order)

7 Dump To File: FMC HA Operations History (ASC order)

8 Last Successful Periodic Sync Time (When it completed)

9 Print HA Status Messages

10 Compare active and standby device list

11 Check manager status of standby missing devices

12 Check critical PM processes details

13 Get Remote Stale Sync AQ Info

14 Help

0 Exit

**************************************************************

Enter choice: 0

Thank you

FMC Standby

> show version

-------------------[ firepower ]--------------------

Model                     : Secure Firewall Management Center for VMware (66) Version 7.2.8 (Build 25)

UUID                      : bb333216-xxxx-xxxx-xxxx-c68c0c388b44

Rules update version      : 2023-11-29-001-vrt

LSP version               : lsp-rel-20231129-1200

VDB version               : 395

----------------------------------------------------

> expert

admin@firepower:~$ sudo su

root@firepower:/Volume/home/admin# ping xx.xx.18.101

PING xx.xx.18.101 (xx.xx.18.101) 56(84) bytes of data.

64 bytes from xx.xx.18.101: icmp_seq=1 ttl=64 time=0.402 ms

64 bytes from xx.xx.18.101: icmp_seq=2 ttl=64 time=0.482 ms

64 bytes from xx.xx.18.101: icmp_seq=3 ttl=64 time=0.452 ms

64 bytes from xx.xx.18.101: icmp_seq=4 ttl=64 time=0.490 ms

64 bytes from xx.xx.18.101: icmp_seq=5 ttl=64 time=0.519 ms

^C

--- xx.xx.18.101 ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 123ms

rtt min/avg/max/mdev = 0.402/0.469/0.519/0.039 ms

root@firepower:/Volume/home/admin# netstat -an | grep 8305

tcp        0      0 xx.xx.18.102:8305       xx.xx.18.254:50373      ESTABLISHED

tcp        0      0 xx.xx.18.102:8305       xx.xx.18.253:42083      ESTABLISHED

tcp        0      0 xx.xx.18.102:59439      xx.xx.18.254:8305       ESTABLISHED

tcp        0      0 xx.xx.18.102:36751      xx.xx.18.253:8305       ESTABLISHED

tcp        0      0 xx.xx.18.102:38239      xx.xx.18.101:8305       ESTABLISHED ------ communication establish with FMC-Active

tcp        0      0 xx.xx.18.102:42253      xx.xx.18.101:8305       ESTABLISHED ------ communication establish with FMC-Active

root@firepower:/Volume/home/admin#

root@firepower:/Volume/home/admin# troubleshoot_HADC.pl

**************** Troubleshooting Utility **************

1 Show HA Info Of FMC

2 Execute Sybase DBPing

3 Show Arbiter Status

4 Check Peer Connectivity

5 Print Messages of AQ Task

6 Show FMC HA Operations History (ASC order)

7 Dump To File: FMC HA Operations History (ASC order)

8 Last Successful Periodic Sync Time (When it completed)

9 Print HA Status Messages

10 Compare active and standby device list

11 Check manager status of standby missing devices

12 Check critical PM processes details

13 Get Remote Stale Sync AQ Info

14 Help

0 Exit

**************************************************************

Enter choice: 1

HA Enabled: Yes

This FMC Role In HA: Standby - Secondary

Status out put: vmsDbEngine (system,gui) - Running 29652

In vmsDbEngineStatus(): vmsDbEngine process is running at /usr/local/sf/lib/perl/5.24.4/SF/Synchronize/HADC.pm line 3748.

Sybase Process: Running (vmsDbEngine, theSybase PM Process is Running)

Sybase Database Connectivity: Accepting DB Connections.

Sybase Database Name: csm_secondary

Sybase Role: Standby

**************** Troubleshooting Utility **************

1 Show HA Info Of FMC

2 Execute Sybase DBPing

3 Show Arbiter Status

4 Check Peer Connectivity

5 Print Messages of AQ Task

6 Show FMC HA Operations History (ASC order)

7 Dump To File: FMC HA Operations History (ASC order)

8 Last Successful Periodic Sync Time (When it completed)

9 Print HA Status Messages

10 Compare active and standby device list

11 Check manager status of standby missing devices

12 Check critical PM processes details

13 Get Remote Stale Sync AQ Info

14 Help

0 Exit

**************************************************************

Enter choice: 4

Peer UUID [Enter 'Return' For HA Peer(no UUID required)]:

Peer Is Connected

$VAR1 = {

'vnet' => undef,

'upgrade_version' => '',

'uuid_gw' => '',

'name' => 'xx.xx.18.101',

'primary_mgr' => 1,

'sw_version' => '7.2.8',

'persistent' => 0,

'model_number' => '66',

'last_changed' => 1718045994,

'reg_key' => '',

'active' => 1,

'reg_state' => 0,

'model_id' => 'E',

'vip' => '',

'mgmt_mac_address' => '00:50:56:B3:E1:57',

'vip_local' => '',

'ip' => 'xx.xx.18.101',

'priority' => 0,

'uuid' => 'fc3e3572-xxxx-xxxx-xxxx-39e0098c166c',

'role' => 0,

'ipv6' => undef

};

**************** Troubleshooting Utility **************

1 Show HA Info Of FMC

2 Execute Sybase DBPing

3 Show Arbiter Status

4 Check Peer Connectivity

5 Print Messages of AQ Task

6 Show FMC HA Operations History (ASC order)

7 Dump To File: FMC HA Operations History (ASC order)

8 Last Successful Periodic Sync Time (When it completed)

9 Print HA Status Messages

10 Compare active and standby device list

11 Check manager status of standby missing devices

12 Check critical PM processes details

13 Get Remote Stale Sync AQ Info

14 Help

0 Exit

**************************************************************

Enter choice: 8

---------------------------- Last periodic sync time details ------------------------------------

Last successful sync completed at: Wed Oct 16 16:46:06 2024 UTC

Current time: Wed Oct 16 16:47:35 2024 UTC

Last successful sync completed '1 minute 29 seconds' ago.

---------------------------- Last periodic sync time details end ------------------------------------

**************** Troubleshooting Utility **************

1 Show HA Info Of FMC

2 Execute Sybase DBPing

3 Show Arbiter Status

4 Check Peer Connectivity

5 Print Messages of AQ Task

6 Show FMC HA Operations History (ASC order)

7 Dump To File: FMC HA Operations History (ASC order)

8 Last Successful Periodic Sync Time (When it completed)

9 Print HA Status Messages

10 Compare active and standby device list

11 Check manager status of standby missing devices

12 Check critical PM processes details

13 Get Remote Stale Sync AQ Info

14 Help

0 Exit

**************************************************************

Enter choice: 9

SYNC_ACTIVE: 1 at /usr/local/sf/lib/perl/5.24.4/SF/Synchronize.pm line 494, <STDIN> line 5.

Found running Synchronization task: Initializing at /usr/local/sf/lib/perl/5.24.4/SF/Transaction/HADC.pm line 241.

Sybase state :  at /usr/local/sf/lib/perl/5.24.4/SF/Synchronize.pm line 701.

Sybase state :  at /usr/local/sf/lib/perl/5.24.4/SF/Synchronize.pm line 801.

Sync status :Synchronization Task In-progress  at /usr/local/sf/lib/perl/5.24.4/SF/Synchronize.pm line 802.

Found running Synchronization task: Initializing at /usr/local/sf/lib/perl/5.24.4/SF/Transaction/HADC.pm line 241.

---------------------------- FMC HA status messages start ------------------------------------

Status: Synchronization Task In-progress

---------------------------- FMC HA status messages end ------------------------------------

**************** Troubleshooting Utility **************

1 Show HA Info Of FMC

2 Execute Sybase DBPing

3 Show Arbiter Status

4 Check Peer Connectivity

5 Print Messages of AQ Task

6 Show FMC HA Operations History (ASC order)

7 Dump To File: FMC HA Operations History (ASC order)

8 Last Successful Periodic Sync Time (When it completed)

9 Print HA Status Messages

10 Compare active and standby device list

11 Check manager status of standby missing devices

12 Check critical PM processes details

13 Get Remote Stale Sync AQ Info

14 Help

0 Exit

**************************************************************

Enter choice: 10

Fetching standby missing device information...

Devices are in sync.

**************** Troubleshooting Utility **************

1 Show HA Info Of FMC

2 Execute Sybase DBPing

3 Show Arbiter Status

4 Check Peer Connectivity

5 Print Messages of AQ Task

6 Show FMC HA Operations History (ASC order)

7 Dump To File: FMC HA Operations History (ASC order)

8 Last Successful Periodic Sync Time (When it completed)

9 Print HA Status Messages

10 Compare active and standby device list

11 Check manager status of standby missing devices

12 Check critical PM processes details

13 Get Remote Stale Sync AQ Info

14 Help

0 Exit

**************************************************************

Enter choice: 0

Thank you

Communication from FTD to FMC-HA validation

> show version

-------------------[ firepower ]--------------------

Model                     : Cisco Firepower Threat Defense for VMware (75) Version 7.2.4 (Build 165)

UUID                      : 7064913a-xxxx-xxxx-xxxx-803aefd05d2c

LSP version               : lsp-rel-20231129-1200

VDB version               : 395

----------------------------------------------------

> ping system xx.xx.18.101 ------ ping to FMC-Active

PING xx.xx.18.101 (xx.xx.18.101) 56(84) bytes of data.

64 bytes from xx.xx.18.101: icmp_seq=1 ttl=64 time=14.1 ms

64 bytes from xx.xx.18.101: icmp_seq=2 ttl=64 time=27.8 ms

64 bytes from xx.xx.18.101: icmp_seq=3 ttl=64 time=26.1 ms

64 bytes from xx.xx.18.101: icmp_seq=6 ttl=64 time=55.7 ms

64 bytes from xx.xx.18.101: icmp_seq=7 ttl=64 time=39.9 ms

64 bytes from xx.xx.18.101: icmp_seq=8 ttl=64 time=38.9 ms

^C

--- xx.xx.18.101 ping statistics ---

8 packets transmitted, 6 received, 25% packet loss, time 76ms

rtt min/avg/max/mdev = 14.081/33.733/55.658/13.069 ms

> ping system xx.xx.18.102 ------ ping to FMC-Active

PING xx.xx.18.102 (xx.xx.18.102) 56(84) bytes of data.

64 bytes from xx.xx.18.102: icmp_seq=1 ttl=64 time=23.9 ms

64 bytes from xx.xx.18.102: icmp_seq=2 ttl=64 time=23.10 ms

64 bytes from xx.xx.18.102: icmp_seq=3 ttl=64 time=0.425 ms

64 bytes from xx.xx.18.102: icmp_seq=4 ttl=64 time=6.88 ms

64 bytes from xx.xx.18.102: icmp_seq=5 ttl=64 time=10.5 ms

^C

--- xx.xx.18.102 ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 70ms

rtt min/avg/max/mdev = 0.425/13.131/23.969/9.380 ms

> show managers

Type                      : Manager

Host                      : xx.xx.18.101

Display name              : xx.xx.18.101

Version                   : 7.2.8 (Build 25)

Identifier                : fc3e3572-xxxx-xxxx-xxxx-39e0098c166c

Registration              : Completed

Management type           : Configuration and analytics

Type                      : Manager

Host                      : xx.xx.18.102

Display name              : xx.xx.18.102

Version                   : 7.2.8 (Build 25)

Identifier                : bb333216-xxxx-xxxx-xxxx-c68c0c388b44

Registration              : Completed

Management type           : Configuration and analytics

> sftunnel-status

SFTUNNEL Start Time: Mon Oct 14 21:29:16 2024

Both IPv4 and IPv6 connectivity is supported

Broadcast count = 17

Reserved SSL connections: 0

Management Interfaces: 2

eth0 (control events) xx.xx.18.254,

tap_nlp (control events) 169.254.1.2,fd00:0:0:1::2

***********************

**RUN STATUS****xx.xx.18.102************* --------------------------------------- sftunnel information of the FMC-Standby

Key File    = /var/sf/peers/bb333216-xxxx-xxxx-xxxx-c68c0c388b44/sftunnel-key.pem

Cert File   = /var/sf/peers/bb333216-xxxx-xxxx-xxxx-c68c0c388b44/sftunnel-cert.pem

CA Cert     = /var/sf/peers/bb333216-xxxx-xxxx-xxxx-c68c0c388b44/cacert.pem

Cipher used = TLS_AES_256_GCM_SHA384 (strength:256 bits)

ChannelA Connected: Yes, Interface eth0

Cipher used = TLS_AES_256_GCM_SHA384 (strength:256 bits)

ChannelB Connected: Yes, Interface eth0

Registration: Completed.

IPv4 Connection to peer 'xx.xx.18.102' Start Time: Wed Oct 16 15:06:23 2024 UTC

IPv4 Last outbound connection to peer 'xx.xx.18.102' via Primary ip/host 'xx.xx.18.102'

PEER INFO:

sw_version 7.2.8

sw_build 25

Using light registration

Management Interfaces: 1

eth0 (control events) xx.xx.18.102,

Peer channel Channel-A is valid  type (CONTROL), using 'eth0', connected to 'xx.xx.18.102' via 'xx.xx.18.254'

Peer channel Channel-B is valid  type (EVENT), using 'eth0', connected to 'xx.xx.18.102' via 'xx.xx.18.254'

***********************

**RUN STATUS****xx.xx.18.101************* -------------------------------------- sftunnel information of the FMC-Active

Key File    = /var/sf/peers/fc3e3572-xxxx-xxxx-xxxx-39e0098c166c/sftunnel-key.pem

Cert File   = /var/sf/peers/fc3e3572-xxxx-xxxx-xxxx-39e0098c166c/sftunnel-cert.pem

CA Cert     = /var/sf/peers/fc3e3572-xxxx-xxxx-xxxx-39e0098c166c/cacert.pem

Cipher used = TLS_AES_256_GCM_SHA384 (strength:256 bits)

ChannelA Connected: Yes, Interface eth0

Cipher used = TLS_AES_256_GCM_SHA384 (strength:256 bits)

ChannelB Connected: Yes, Interface eth0

Registration: Completed.

IPv4 Connection to peer 'xx.xx.18.101' Start Time: Mon Oct 14 21:29:15 2024 UTC

PEER INFO:

sw_version 7.2.8

sw_build 25

Using light registration

Management Interfaces: 1

eth0 (control events) xx.xx.18.101,

Peer channel Channel-A is valid  type (CONTROL), using 'eth0', connected to 'xx.xx.18.101' via 'xx.xx.18.254'

Peer channel Channel-B is valid  type (EVENT), using 'eth0', connected to 'xx.xx.18.101' via 'xx.xx.18.254'

***********************

**RPC STATUS****xx.xx.18.101*************

'name' => 'xx.xx.18.101',

'last_changed' => 'Mon Jun 10 18:59:54 2024',

'uuid_gw' => '',

'ip' => 'xx.xx.18.101',

'ipv6' => 'IPv6 is not configured for management',

'active' => 1,

'uuid' => 'fc3e3572-xxxx-xxxx-xxxx-39e0098c166c'

**RPC STATUS****xx.xx.18.102*************

'name' => 'xx.xx.18.102',

'last_changed' => 'Wed Oct  9 07:00:11 2024',

'uuid_gw' => '',

'ip' => 'xx.xx.18.102',

'ipv6' => 'IPv6 is not configured for management',

'active' => 1,

'uuid' => 'bb333216-xxxx-xxxx-xxxx-c68c0c388b44'

Check routes:

No peers to check

Note: If the sftunnel information of one of the FMC is missing, this can indicate that the communication with the manager is compromised

FTD root level troubleshoot

> expert

admin@firepower:~$ sudo su

root@firepower:/home/admin# netstat -an | grep 8305

tcp        0      0 xx.xx.18.254:8305       xx.xx.18.102:59439      ESTABLISHED ------ communication establish with FMC-Standby

tcp        0      0 xx.xx.18.254:49205      xx.xx.18.101:8305       ESTABLISHED ------ communication establish with FMC-Active

tcp        0      0 xx.xx.18.254:50373      xx.xx.18.102:8305       ESTABLISHED ------ communication establish with FMC-Standby

tcp        0      0 xx.xx.18.254:53875      xx.xx.18.101:8305       ESTABLISHED ------ communication establish with FMC-Active

root@firepower:/home/admin# cat /ngfw/var/log/action_queue.log | less

Oct 16 15:06:50 firepower ActionQueueScrape.pl[4166]: Waiting for light registration to complete on device:  try 10 times... at /ngfw/usr/local/sf/lib/p

erl/5.24.4/SF/PeerManager/RegistrationCL.pm line 1805.

Oct 16 15:06:50 firepower ActionQueueScrape.pl[4166]: Found Registered peer with name xx.xx.18.102 (bb333216-xxxx-xxxx-xxxx-c68c0c388b44) at /ngfw/usr/local/sf/lib/perl/5.24.4/SF/PeerManager/RegistrationCL.pm line 1821.

Oct 16 15:06:50 firepower ActionQueueScrape.pl[4166]: Found peer with name xx.xx.18.102 - update DB at /ngfw/usr/local/sf/lib/perl/5.24.4/SF/PeerManager/RegistrationCL.pm line 1827.

Oct 16 15:06:50 firepower ActionQueueScrape.pl[4166]: Found Registered peer with name xx.xx.18.101 (fc3e3572-xxxx-xxxx-xxxx-39e0098c166c) at /ngfw/usr/local/sf/lib/perl/5.24.4/SF/PeerManager/RegistrationCL.pm line 1821.