Contents

Introduction

Pre-Requisites

Background Information

Requirements

Components Required

Ports Required

Debugs Required

Node details

Topology

Time Stamps Involved

Registration Process

PCAP Analysis

CLI Utilities

Log Analysis

Alarms related to node sync issues

Known Defects

Introduction

This document describes Identity Service Engine (ISE) node registration pre-requisites, step by step process, PCAP analysis, log analysis.

Pre-Requisites

You must have the basic knowledge of the Cisco Identity Service Engine (ISE).

Background Information

- Standalone Node: a deployment that has a single Cisco ISE node. This node runs the Administration, Policy Service, and Monitoring personas.

- Distributed deployment: a deployment that has more than one Cisco ISE node. To support failover and to improve performance, you can set up a deployment with multiple Cisco ISE nodes in a distributed fashion. In a Cisco ISE distributed deployment, administration and monitoring activities are centralized, and processing is distributed across the Policy Service nodes. Depending on your performance needs, you can scale your deployment.

- An ISE node can be in one of the state primary, secondary or stand alone node.

- An ISE node can assume any of these personas: Administration, Policy Service, and Monitoring.

- You need to perform registration process to add more nodes to a primary node to form a cluster.

Requirements

- After Installing ISE nodes, you must choose one of the nodes and change its state from standalone to primary. Navigate to Administration > Deployment, click the ISE node > Make Primary .

- You can have a single primary node in a distributed deployment.

- FQDN of the secondary node to be registered must be resolvable from primary node. Forward and reverse DNS lookup must work.

- Version/patch details of both nodes are the same.

- You must have super admin credentials of the secondary node.

- Ensure the admin certificate of both nodes are trusted, If third party CA signed certificate, add the root CA to the trusted certificates store.

- Latency between nodes is less than 300ms.

Components Required

The information in this document is based on these software and hardware versions

- Cisco Identity service Engine 3.3

- ISE VM

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Ports Required

- HTTPS (SOAP): TCP/443

- Data Synchronization/ Replication (JGroups): TCP/12001

- ISE Messaging Service: SSL: TCP/8671

- ISE internal communication: TCP/15672

- Profiler Endpoint ownership Synchronization/ Replication: TCP/6379

Debugs Required

Attributes to be set to debug level:

- Replication-Deployment (replication.log and ise-psc.log)

- Replication-JGroup (replication.log and ise-psc.log)

- Replication Tracker (tracking.log)

- hibernate (hibernate.log)

- JMS (replication.log)

- ca-service (caservice.log)

- admin-ca (ise-psc.log)

Node details

- ise330.tac.local (10.127.197.174) is the PAN node

- ise331.tac.local (10.127.197.175) is the SAN node

Topology

Time Stamps Involved

Node ise331.tac.local is registered on ise330.tac.local

Register Click- 2025-01-31, 16:00:01

Next Click (After filling secondary node FQDN details) -2025-01-31, 16:00:13

Submit Click (After selecting personas)-2025-01-31, 16:00:21

Replication-2025-01-31, 16:02:22

Completed Sync-2025-01-31 16:30:54

Registration Process

Node registration: a process of authenticated configuration database replication from primary admin node to the secondary node

Step 1: Make the ISE node as primary

Initially, every node is in standalone mode. To register other nodes, make one of the nodes the primary node.

Navigate to Administration > Deployment > Click the ISE node/Edit the node .

Click Make Primary.

Click Edit the node.

Click Make Primary and Save it.

The role now becomes primary.

Step 2: Register the secondary node

The node to be registered must be in standalone mode.

Go to the secondary node and get the FQDN.

Navigate to Administration > Deployment. Click on the node name.

Go to the PAN GUI.

Navigate to Administration > Deployment.

Click Register.

It asks for the FQDN and credentials of the node to be registered.

Enter the FQDN and super admin user credentials of the secondary node.

When you click Next, it imports the certificate of the secondary node.

Click Import certificate and proceed.

Now select the personas required on the secondary node, click Submit.

The secondary node is then registered to primary node.Click OK.

Verify that the secondary node added to the PAN and amber color indicates that registration/sync is in progress.

Step 4: When you selected secondary admin node (SAN), you need to renew the license. It is not applicable for PSNs.

Navigate to Administration>Licensing.

Click Renew Registration.

Step 5: Registration is successful.

PCAP Analysis

In PCAP, you see DNS, certificate exchanges over 443.

DNS traffic: When you click register and provided the FQDN, the primary node tries to resolve the FQDN of the secondary node.

You can see the certificate exchange between both the nodes as well.

Client Hello

Server Hello, Certificate, Server Key Exchange, Server Hello

We can see ise331.tac.local shares it certificate to PAN node.

Client Key Exchange has the public key.

Both nodes fix on the the cipher suites and create the encrypted tunnel to share the details.

CLI Utilities

- ise-psc.log- DNS query / cert validation / cert replication / sync status control

Path from root: /opt/CSCOcpm/logs/ise-psc.log

From CLI,

#show logg application ise-psc.log tail

- ADE.log- DB import/export, service start/stop

Path From root: /var/log/ade/ADE.log

From CLI,

#show logg system ade/ADE.log tail

- Replication.log- Logs for issue after DB import or DMP file transfer

Path from root: /opt/CSCOcpm/logs/replication.log

From CLI,

#show application replication.log tail

- Deployment.log- Logs related to certs, response pages

Path From root: /opt/CSCOcpm/logs/deployment.log

From CLI,

#show logging application deployment.log

Log Analysis

Step 1: Click on Register ISE Node

ise330-ise-psc.log:

2025-01-31 16:00:01,992 INFO  [admin-http-pool2][[]] cpm.admin.infra.spring.ISEAdminControllerUtils -::admin:::- mapping path found in action-forwards,forwarding to: /pages/modules/infra/deployment/deploymentRegister.jsp

Step 2: Fill the details of secondary node.

ise330-deployment.log:

2025-01-31 16:00:11,313 INFO   [admin-http-pool5][[]] cpm.infrastructure.deployment.client.DeploymentRegistrationClient -::admin:ise331.tac.local:registerNode:-Calling register for host FQDNise331.tac.local

2025-01-31 16:00:11,371 DEBUG  [admin-http-pool5][[]] deployment.client.cert.validator.HttpsCertPathValidatorImpl -::admin:ise331.tac.local:registerNode:-Is cert path valid?true

ise331-deployment.log

2025-01-31 16:00:11,708 INFO [admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::-Received request for doPost /register

2025-01-31 16:00:11,710 INFO [admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::-Received request for doGet/register

2025-01-31 16:00:11,716 INFO [admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::-Primary hostname retrieved is ise330.tac.local

Step 3: DNS Resolution

ise330- ise-psc.log:

2025-01-31 16:00:13,208 INFO  [admin-http-pool5][[]] cpm.admin.infra.action.DeploymentEditAction -::admin:ise331.tac.local:registerNode:-Node being registered has FQDN 'ise331.tac.local'.This name resolves to IP address10.127.197.175

2025-01-31 16:00:20,638 INFO [admin-http-pool3][[]] cpm.admin.infra.action.DeploymentEditAction -::admin:::- Node being added is Secondary PAP :ise331

ise331-ise-psc.log

2025-01-31 16:00:11,716 INFO [admin-http-pool14][[]] epm.pap.api.handler.CommonUtil -:::::-Checking if the host ise330.tac.local is dns resolvable

2025-01-31 16:00:11,721 INFO [admin-http-pool14][[]] epm.pap.api.handler.CommonUtil -:::::-Host ise330.tac.local is dns resolvable within 5 ms

Step 4:Certificates

ise330-ise.psc.log

2025-01-31 16:00:20,887 INFO [admin-http-pool5][[]] cpm.admin.infra.action.DeploymentEditAction -::admin:ise331:registerNode:-preparing to call getServCert

ise330-deployment.log

2025-01-31 16:00:20,888 DEBUG [admin-http-pool5][[]] cpm.infrastructure.deployment.client.DeploymentRegistrationClient -::admin:ise331:registerNode:- hostAlias ::ise331.tac.local ipAddrString :: 10.127.197.175 urlStr ::https://ise331.tac.local/admin/API/mnt/api/ServCert

2025-01-31 16:00:21,419 DEBUG [admin-http-pool5][[]] cpm.infrastructure.deployment.client.DeploymentRegistrationClient -::admin:ise331:registerNode:- GET: request tohttps://ise331.tac.local/admin/API/mnt/api/ServCert completed.

Step 5: Adding secondary node with personas submitted

ise330-ise-psc.log

2025-01-31 16:00:21,905 INFO [admin-http-pool5][[]] cisco.mnt.dbms.handler.DataSourceReInitializingHandler -::admin:ise331:registerNode:- Received NG event:Add

2025-01-31 16:00:21,905 INFO [admin-http-pool5][[]] cisco.mnt.dbms.handler.DataSourceReInitializingHandler -::admin:ise331:registerNode:-Current : Node masterStatus: STANDBY

2025-01-31 16:00:21,905 INFO [admin-http-pool5][[]] cisco.mnt.dbms.handler.DataSourceReInitializingHandler -::admin:ise331:registerNode:-Current : NodeRoleStatus: SECONDARY

2025-01-31 16:00:21,905 INFO [admin-http-pool5][[]] cisco.mnt.dbms.handler.DataSourceReInitializingHandler -::admin:ise331:registerNode:- Current : HostConfig event hostAlias: ise331.tac.local

Node details created and save successfully

ise330-deployment.log

2025-01-31 16:01:22,247 DEBUG [admin-http-pool5][[]] cisco.cpm.infrastructure.deployment.DeploymentManager -::admin:ise331:registerNode:- HostConfig 'ise331' was created successfully

2025-01-31 16:01:22,251 DEBUG [admin-http-pool5][[]] cisco.cpm.infrastructure.deployment.DeploymentManager -::admin:ise331:registerNode:- Retrieved HostConfig [hostName=ise331, dispayName=ise331, hostId=5ff70820-dfbe-11ef-bb78-12467b70bb3d, gateWay=10.127.197.1,masterStatus=STANDBY, nodeRoleStatus=SECONDARY, nodeTypes=PAP+MNT+PDP, nodeServiceType=SESSION,PROFILER, userName=null, smtpPort=null, smtpHost=null, hostAlias=ise331.tac.local, udiPid=ISE-VM-K9, udiVid=V01, udiSN=S7LEAGKMIHM, udiPT=VM, installType=null, vmInfo=16179780|12|LARGE||2025-01-30 21:44:33 IST|0 MB|0 MHz|4294967295 MB|5000 MHz, isApiNode=false] DBInterface [id=5ff70823-dfbe-11ef-bb78-12467b70bb3d, userName=<not shown>, passWd=<not shown>, portNum=1521, sId=cpm10] MasterInterface [id=5ff70822-dfbe-11ef-bb78-12467b70bb3d, userName=<not shown>, passWd=<not shown>, portNum=80, domainName=Default Domain, contextRoot=admin, serviceType=WEBSERVICE, isSSLEnable=false] NICInterface [id=5ff70821-dfbe-11ef-bb78-12467b70bb3d, ipAddress=10.127.197.175, subNetMask=255.255.255.0, macAddress=00-50-56-8b-23-2e, nicCards=eth0]

Step 6: Certificates

ise330-ise-psc.log

2025-01-31 16:01:22,251 INFO [admin-http-pool5][[]] cpm.admin.infra.action.DeploymentEditAction -::admin:ise331:registerNode:- HostConfig hostId of registering node ise331 at time of local cert save: 5ff70820-dfbe-11ef-bb78-12467b70bb3d. All local certs and CSRs reference the HostConfig using this hostId.

ise330-deployment.log

2025-01-31 16:01:22,841 DEBUG [admin-http-pool5][[]] cpm.infrastructure.deployment.client.DeploymentRegistrationClient -::admin:ise331:registerNode:-Importing deployment shared certificates

2025-01-31 16:01:22,841 DEBUG [admin-http-pool5][[]] cpm.infrastructure.deployment.client.DeploymentRegistrationClient -::admin:ise331:registerNode:- Preparing connection

2025-01-31 16:01:23,804 DEBUG [admin-http-pool5][[]] cpm.infrastructure.deployment.client.DeploymentRegistrationClient -::admin:ise331:registerNode:- connected to registering node

2025-01-31 16:01:23,804 DEBUG [admin-http-pool5][[]] cpm.infrastructure.deployment.client.DeploymentRegistrationClient -::admin:ise331:registerNode:-Sending import CertParams to registering node

2025-01-31 16:01:23,822 DEBUG [admin-http-pool5][[]] cpm.infrastructure.deployment.client.DeploymentRegistrationClient -::admin:ise331:registerNode:- Importing CertParams sent to registering node

2025-01-31 16:01:23,823 DEBUG [admin-http-pool5][[]] cpm.infrastructure.deployment.client.DeploymentRegistrationClient -::admin:ise331:registerNode:- Receiveing imported certs ids

2025-01-31 16:01:24,147 DEBUG [admin-http-pool5][[]] cpm.infrastructure.deployment.client.DeploymentRegistrationClient -::admin:ise331:registerNode:-Imported certs ids received:

2025-01-31 16:01:24,148 DEBUG [admin-http-pool5][[]] cpm.infrastructure.deployment.client.DeploymentRegistrationClient -::admin:ise331:registerNode:-Importing deployment shared certificates - done

2025-01-31 16:01:24,382 DEBUG [admin-http-pool5][[]] cisco.cpm.infrastructure.deployment.DeploymentManager -::admin:ise331:registerNode:- Inside getHostConfigForLocalNode to fetch localhostconfig

2025-01-31 16:01:24,382 DEBUG [admin-http-pool5][[]] cisco.cpm.infrastructure.deployment.DeploymentManager -::admin:ise331:registerNode:- hostName = ise330

2025-01-31 16:01:24,385 DEBUG [admin-http-pool5][[]] cisco.cpm.infrastructure.deployment.DeploymentManager -::admin:ise331:registerNode:- Retrieved HostConfig [hostName=ise330, dispayName=ise330, hostId=50451540-dee0-11ef-90b1-0050568bb121, gateWay=10.127.197.1, masterStatus=ACTIVE, nodeRoleStatus=PRIMARY, nodeTypes=PAP+MNT+PDP, nodeServiceType=SESSION,PROFILER, userName=null, smtpPort=null, smtpHost=null, hostAlias=ise330.tac.local, udiPid=ISE-VM-K9, udiVid=V01, udiSN=IRJN9DCFIG9, udiPT=VM, installType=null, vmInfo=16179788|12|LARGE|300 GiB|2025-01-31 01:44:51 IST|0 MB|0 MHz|4294967295 MB|5000 MHz, isApiNode=false] DBInterface [id=50451543-dee0-11ef-90b1-0050568bb121, userName=<not shown>, passWd=<not shown>, portNum=1521, sId=cpm10] MasterInterface [id=50451542-dee0-11ef-90b1-0050568bb121, userName=<not shown>, passWd=<not shown>, portNum=80, domainName=Default Domain, contextRoot=admin, serviceType=WEBSERVICE, isSSLEnable=true] NICInterface [id=50451541-dee0-11ef-90b1-0050568bb121, ipAddress=10.127.197.174, subNetMask=255.255.255.0, macAddress=00-50-56-8b-b1-21, nicCards=eth0]

Wildcard check

ise-psc.log <==

2025-01-31 16:01:25,304 INFO [admin-http-pool5][[]] cpm.infrastructure.certreqmgmt.notifications.IseEapCertNotificationHandler -::admin:ise331:registerNode:- Received ISE Local Cert notification.

2025-01-31 16:01:25,304 INFO [admin-http-pool5][[]] cpm.infrastructure.certreqmgmt.notifications.IseEapCertNotificationHandler -::admin:ise331:registerNode:- Local Cert is an EAP cert. Updating KeyAndCertStoreUtil

2025-01-31 16:01:25,304 INFO [admin-http-pool5][[]] cpm.infrastructure.certreqmgmt.util.KeyAndCertStoreUtil -::admin:ise331:registerNode:- Received notification to update this node's EAP cert chain.

2025-01-31 16:01:25,304 INFO [admin-http-pool5][[]] cpm.infrastructure.certreqmgmt.notifications.IseEapCertNotificationHandler -::admin:ise331:registerNode:- Sent update requests.

2025-01-31 16:01:25,305 INFO [admin-http-pool5][[]] cpm.infrastructure.certreqmgmt.notifications.WildcardCertNotificationHandler -::admin:ise331:registerNode:- Received ISE Local Cert notification.

2025-01-31 16:01:25,305 INFO [admin-http-pool5][[]] cpm.infrastructure.certreqmgmt.notifications.WildcardCertNotificationHandler -::admin:ise331:registerNode:- Checking certificate type

2025-01-31 16:01:25,305 INFO [admin-http-pool5][[]] cpm.infrastructure.certreqmgmt.notifications.WildcardCertNotificationHandler -::admin:ise331:registerNode:- Default self-signed server certificate is a wildcard cert: false

2025-01-31 16:01:25,305 INFO [admin-http-pool5][[]] cpm.infrastructure.certreqmgmt.notifications.WildcardCertNotificationHandler -::admin:ise331:registerNode:- Checking for SAML,APNS, CMCS or ProfileSigning certificate

2025-01-31 16:01:25,306 INFO [admin-http-pool5][[]] cpm.infrastructure.certreqmgmt.notifications.WildcardCertNotificationHandler -::admin:ise331:registerNode:- Default self-signed server certificate is it SAML Cert: false

2025-01-31 16:01:25,306 INFO [admin-http-pool5][[]] cisco.cpm.provisioning.cert.CertProvisioningFactory -::admin:ise331:registerNode:- CertChangeHandler.handle(): building EAP Cert chain on insert event of Local Certificate and Trust Certificate

ise330-ise-psc.log

2025-01-31 16:01:25,830 INFO [admin-http-pool5][[]] cpm.admin.infra.action.DeploymentEditAction -::admin:ise331:registerNode:- Saved LocalCertificate: LocalCertificate [basicCertificateFields=X509BasicCertificateFields[version=0,serialNumber=17382241548535177602714336375,issuer=<null>,notBefore=Thu Jan 30 13:32:34 IST 2025,notAfter=Sat Jan 30 13:32:34 IST 2027,subject=CN=ise331.tac.local,issuedTo=ise331.tac.local,issuedBy=ise331.tac.local,certificateFilename=<null>,digestHash=<null>], id=f0463881-4c64-4ff5-96a2-4f0089f50459, name=ise331_tac_local#382a40d27abf53ac42387c77#PID$_ISE-VM-K9$_VID$_V01$_SN$_S7LEAGKMIHM, friendlyName=Default self-signed server certificate, x509Certificate=null, iseRoles=[EAP, Admin, Portal, DTLS], hostConfigId=7ec82060-dee0-11ef-898d-0050568b232e, hostUDI=PID: ISE-VM-K9, VID: V01, SN: S7LEAGKMIHM, keySize=0, fipsCompliant=false, privateKeyGuid=[-2, -92, -78, 78, 73, -87, 65, 60, -71, 0, 7, -127, -104, -124, 56, -83], description=null]

2025-01-31 16:01:25,830 INFO [admin-http-pool5][[]] cpm.admin.infra.action.DeploymentEditAction -::admin:ise331:registerNode:- Preparing to save LocalCertificate: LocalCertificate [basicCertificateFields=X509BasicCertificateFields[version=0,serialNumber=17382241611749295087684098633,issuer=<null>,notBefore=Thu Jan 30 13:32:41 IST 2025,notAfter=Tue Jan 29 13:32:41 IST 2030,subject=CN=SAML_ise331.tac.local,issuedTo=SAML_ise331.tac.local,issuedBy=SAML_ise331.tac.local,certificateFilename=<null>,digestHash=<null>], id=26fd28f0-6346-4fd5-8699-312dfdf97a8e, name=SAML_ise331_tac_local#382a40d5e804fc9a4ba71a49#PID$_ISE-VM-K9$_VID$_V01$_SN$_S7LEAGKMIHM, friendlyName=Default self-signed saml server certificate - CN=SAML_ise331.tac.local, x509Certificate=null, iseRoles=[NO_SSOCert], hostConfigId=7ec82060-dee0-11ef-898d-0050568b232e, hostUDI=PID: ISE-VM-K9, VID: V01, SN: S7LEAGKMIHM, keySize=0, fipsCompliant=false, privateKeyGuid=[-18, 47, -17, -88, 79, -30, 69, -12, -73, 120, -111, -36, 40, 27, 43, 68], description=null] Host Name: ise331 Host ConfigId: 5ff70820-dfbe-11ef-bb78-12467b70bb3d

ise330-replication.log

2025-01-31 16:01:34,051 DEBUG [admin-http-pool5][[]] cisco.cpm.deployment.replication.PublisherImpl -::admin:ise331:registerNode:- Add to publish queue Id: 8afb25b0-dfbe-11ef-bb78-12467b70bb3d, seqNo: 0, data: 3058 bytes

ise330-replication.log

2025-01-31 16:02:14,488 INFO [admin-http-pool5][[]] api.services.server.distribution.RegistrationAndSyncHelper -::admin:ise331:registerNode:- Tracking RegistrationAndSync Entering addNodeToDistributionSetup for host: ise331.tac.local

2025-01-31 16:02:14,488 INFO [admin-http-pool5][[]] api.services.server.distribution.RegistrationAndSyncHelper -::admin:ise331:registerNode:- Tracking RegistrationAndSync - entering addNodeToDistributionSetupUsingJgroups

2025-01-31 16:02:14,501 INFO [admin-http-pool5][[]] api.services.persistance.dao.ReplicationStatusDAOImpl -::admin:ise331:registerNode:- Host Name: ise331, DB 'SEC_REPLICATIONSTATUS' = IN PROGRESS, Node Persona: SECONDARY, ReplicationStatus obj status: SYNC_INPROGRESS

2025-01-31 16:02:14,502 INFO [admin-http-pool5][[]] api.services.server.distribution.RegistrationAndSyncHelper -::admin:ise331:registerNode:- Tracking RegistrationAndSync - exiting addNodeToDistributionSetupUsingJgroups

2025-01-31 16:02:14,502 INFO [admin-http-pool5][[]] api.services.server.distribution.RegistrationAndSyncHelper -::admin:ise331:registerNode:- Tracking RegistrationAndSync Done addNodeToDistributionSetup for host: ise331.tac.local

2025-01-31 16:02:14,508 INFO [admin-http-pool5][[]] api.services.server.distribution.RegistrationAndSyncHelper -::admin:ise331:registerNode:- Tracking RegistrationAndSync - entering offerSyncRequestToQueue

2025-01-31 16:02:14,524 INFO [admin-http-pool5][[]] class com.cisco.epm.fullsync.FullSyncInfoHelper -::admin:ise331:registerNode:- Created FullSyncInfo[ a31cab00-dfbe-11ef-bb78-12467b70bb3d ]

2025-01-31 16:02:14,524 INFO [admin-http-pool5][[]] api.services.server.distribution.RegistrationAndSyncHelper -::admin:ise331:registerNode:- Tracking RegistrationAndSync The sync request is hostFQDN[ise331.tac.local] exportFilePath[null] syncRequestIdentifier[null] syncRequestTime[null] syncStatus[SYNC_INPROGRESS] transientSyncStatus[null] syncRequestType[REGISTER]

2025-01-31 16:02:14,524 INFO [admin-http-pool5][[]] api.services.server.distribution.RegistrationAndSyncHelper -::admin:ise331:registerNode:- Tracking RegistrationAndSync - exiting offerSyncRequestToQueue

Step 7: Registering the node in the cluster

ise330-ise-psc.log

2025-01-31 16:02:14,524 INFO [admin-http-pool5][[]] api.services.server.distribution.DistributionImpl -::admin:ise331:registerNode:- Tracking RegistrationAndSync -New Reg Code completed for node ise331

2025-01-31 16:02:14,524 INFO [admin-http-pool5][[]] cpm.admin.infra.action.DeploymentEditAction -::admin:ise331:registerNode:- Node ise331 was registered successfully with persona(s)Administration, Monitoring, Policy Service

Step 8: Firewall Rules

ise330-ise-psc.log

2025-01-31 16:00:39,945 INFO [admin-http-pool5][[]] cisco.cpm.infrastructure.utils.FwUtil-::admin:ise331:registerNode:- Deployment Hosts IPs: [10.127.197.174, 10.127.197.175]iptable Hosts : [10.127.197.174]

2025-01-31 16:00:43,526 INFO [admin-http-pool5][[]] cisco.cpm.infrastructure.utils.FwUtil-::admin:ise331:registerNode:- syncUpfwList : DB access allowed for ips: [10.127.197.175]

2025-01-31 16:00:46,617 INFO [admin-http-pool5][[]] cisco.cpm.infrastructure.utils.FwUtil-::admin:ise331:registerNode:- syncUpfwList : DB access disallowed for ips: []

2025-01-31 16:00:46,617 INFO [admin-http-pool5][[]] cisco.cpm.infrastructure.utils.JGroupsFWUtil-::admin:ise331:registerNode:-Applying Firewall rules for jGroup

2025-01-31 16:00:53,678 INFO [admin-http-pool5][[]] cisco.cpm.infrastructure.utils.JGroupsFWUtil-::admin:ise331:registerNode:-Applied Firewall rules for jGroup.

2025-01-31 16:14:32,420 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.FwUtil-:::::- Deployment Hosts IPs: [10.127.197.174, 10.127.197.175]iptable Hosts : [10.127.197.175, 10.127.197.174]

2025-01-31 16:14:33,605 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.FwUtil-:::::- syncUpfwList : DB access allowed for ips: []

2025-01-31 16:14:34,785 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.FwUtil-:::::- syncUpfwList : DB access disallowed for ips: []

2025-01-31 16:14:34,785 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.JGroupsFWUtil-:::::- Applying Firewall rules for jGroup

2025-01-31 16:14:36,173 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.JGroupsFWUtil-:::::- Applied Firewall rules for jGroup.

2025-01-31 16:14:36,184 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.NodeGroupFWUtil-:::::- Applying Firewall rules for node group.

2025-01-31 16:14:38,964 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.NodeGroupFWUtil-:::::- Applied Firewall rules for node group.

2025-01-31 16:29:33,320 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.FwUtil-:::::- Deployment Hosts IPs: [10.127.197.174, 10.127.197.175]iptable Hosts : [10.127.197.175, 10.127.197.174]

2025-01-31 16:29:35,505 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.FwUtil-:::::- syncUpfwList : DB access allowed for ips: []

2025-01-31 16:29:37,792 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.FwUtil-:::::- syncUpfwList : DB access disallowed for ips: []

2025-01-31 16:29:37,793 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.JGroupsFWUtil-:::::-Applying Firewall rules for jGroup

2025-01-31 16:29:40,551 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.JGroupsFWUtil-:::::-Applied Firewall rules for jGroup.

2025-01-31 16:29:40,558 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.NodeGroupFWUtil-:::::-Applying Firewall rules for node group.

2025-01-31 16:29:45,254 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.NodeGroupFWUtil-:::::-Applied Firewall rules for node group.

ise331-ise-psc.lohg

2025-01-31 16:00:39,945 INFO [admin-http-pool5][[]] cisco.cpm.infrastructure.utils.FwUtil-::admin:ise331:registerNode:- Deployment Hosts IPs: [10.127.197.174, 10.127.197.175]iptable Hosts : [10.127.197.174]

2025-01-31 16:00:43,526 INFO [admin-http-pool5][[]] cisco.cpm.infrastructure.utils.FwUtil-::admin:ise331:registerNode:- syncUpfwList :DB access allowed for ips: [10.127.197.175]

2025-01-31 16:00:46,617 INFO [admin-http-pool5][[]] cisco.cpm.infrastructure.utils.FwUtil-::admin:ise331:registerNode:- syncUpfwList : DB access disallowed for ips: []

2025-01-31 16:00:46,617 INFO [admin-http-pool5][[]] cisco.cpm.infrastructure.utils.JGroupsFWUtil-::admin:ise331:registerNode:-Applying Firewall rules for jGroup

2025-01-31 16:00:53,678 INFO [admin-http-pool5][[]] cisco.cpm.infrastructure.utils.JGroupsFWUtil-::admin:ise331:registerNode:-Applied Firewall rules for jGroup.

2025-01-31 16:14:32,420 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.FwUtil-:::::- Deployment Hosts IPs: [10.127.197.174, 10.127.197.175]iptable Hosts : [10.127.197.175, 10.127.197.174]

2025-01-31 16:14:33,605 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.FwUtil-:::::- syncUpfwList : DB access allowed for ips: []

2025-01-31 16:14:34,785 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.FwUtil-:::::- syncUpfwList : DB access disallowed for ips: []

2025-01-31 16:14:34,785 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.JGroupsFWUtil-:::::- Applying Firewall rules for jGroup

2025-01-31 16:14:36,173 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.JGroupsFWUtil-:::::- Applied Firewall rules for jGroup.

2025-01-31 16:14:36,184 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.NodeGroupFWUtil-:::::- Applying Firewall rules for node group.

2025-01-31 16:14:38,964 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.NodeGroupFWUtil-:::::- Applied Firewall rules for node group.

2025-01-31 16:29:33,320 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.FwUtil-:::::- Deployment Hosts IPs: [10.127.197.174, 10.127.197.175]iptable Hosts : [10.127.197.175, 10.127.197.174]

2025-01-31 16:29:35,505 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.FwUtil-:::::- syncUpfwList : DB access allowed for ips: []

2025-01-31 16:29:37,792 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.FwUtil-:::::- syncUpfwList : DB access disallowed for ips: []

2025-01-31 16:29:37,793 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.JGroupsFWUtil-:::::-Applying Firewall rules for jGroup

2025-01-31 16:29:40,551 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.JGroupsFWUtil-:::::-Applied Firewall rules for jGroup.

2025-01-31 16:29:40,558 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.NodeGroupFWUtil-:::::-Applying Firewall rules for node group.

2025-01-31 16:29:45,254 INFO [DefaultQuartzScheduler_Worker-5][[]] cisco.cpm.infrastructure.utils.NodeGroupFWUtil-:::::-Applied Firewall rules for node group.

Step 9: Datadump export until the current system change number (SCN) and transferring files to ise331

ise330-ade.log

2025-01-31T16:02:41.682824+05:30 ise330 root: info:[sync_export.sh]Starting the datadump export for sync identifier b33db240-dfbe-11ef-bb78-12467b70bb3d...

2025-01-31T16:02:43.545110+05:30 ise330 root: info:[sync_export.sh] SCN retrieved for sync identifier b33db240-dfbe-11ef-bb78-12467b70bb3d... now starting export for CSCN 6022419 and host ise330025-01-31T16:09:35.095186+05:30 ise330 root: info:[sync_export.sh]Export success for sync identifier b33db240-dfbe-11ef-bb78-12467b70bb3d...

ise330-replication.log

2025-01-31 16:02:41,563 INFO [Deployment-Sync-Manager-Thread-0][[]] cisco.epm.fullsync.primary.PrimarySyncManager -::::FullSync:-Processing Sync Requests

2025-01-31 16:02:41,563 INFO [Deployment-Sync-Manager-Thread-0][[]] class com.cisco.epm.fullsync.FullSyncInfoHelper -::::FullSync:- updateFullSyncStatus:a31cab00-dfbe-11ef-bb78-12467b70bb3d,null

2025-01-31 16:02:41,572 INFO [Deployment-Sync-Manager-Thread-0][[]] cisco.epm.fullsync.primary.PrimarySyncManager -::::FullSync:-Found 1 sync requests.

2025-01-31 16:02:41,572 INFO [Deployment-Sync-Manager-Thread-0][[]] class com.cisco.epm.fullsync.FullSyncInfoHelper -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:-updateFullSyncStatus:a31cab00-dfbe-11ef-bb78-12467b70bb3d,b33db240-dfbe-11ef-bb78-12467b70bb3d2025-01-31 16:02:41,584 INFO [Deployment-Sync-Manager-Thread-0][[]] class com.cisco.epm.fullsync.ExportImportExecutor -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:-Tracking RegistrationAndSync - export script started for identifier [b33db240-dfbe-11ef-bb78-12467b70bb3d]

2025-01-31 16:02:41,586 INFO [Deployment-Sync-Manager-Thread-0][[]] class com.cisco.epm.fullsync.ExportImportExecutor -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:

2025-01-31 16:09:40,602 INFO [Deployment-Sync-Manager-Thread-0][[]] class com.cisco.epm.fullsync.ExportImportExecutor -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:- Tracking RegistrationAndSync -export script finished for identifier [b33db240-dfbe-11ef-bb78-12467b70bb3d] returnVal + 0

2025-01-31 16:09:40,602 INFO [Deployment-Sync-Manager-Thread-0][[]] cisco.epm.fullsync.primary.PrimarySyncManager -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:-DB Export Completed: returnValue : 0

2025-01-31 16:09:40,603 INFO [Deployment-Sync-Manager-Thread-0][[]] cisco.epm.fullsync.primary.PrimarySyncManager -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:-Exported the DB Dump:/opt/oracle/base/admin/cpm10/dpdump/cpmsync_b33db240-dfbe-11ef-bb78-12467b70bb3d.dmp Size(bytes): 131614567

2025-01-31 16:09:40,606 INFO [Deployment-Sync-Manager-Thread-0][[]] class com.cisco.epm.fullsync.FileUtil -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:- Tracking RegistrationAndSync -Creating file /opt/oracle/base/admin/cpm10/dpdump/config_b33db240-dfbe-11ef-bb78-12467b70bb3d.properties

2025-01-31 16:09:40,606 INFO [Deployment-Sync-Manager-Thread-0][[]] class com.cisco.epm.fullsync.FileUtil -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:- Tracking RegistrationAndSync -Created file /opt/oracle/base/admin/cpm10/dpdump/config_b33db240-dfbe-11ef-bb78-12467b70bb3d.properties

2025-01-31 16:09:40,606 INFO [Deployment-Sync-Manager-Thread-0][[]] cisco.epm.fullsync.primary.PrimarySyncManager -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:- Created properties file

2025-01-31 16:09:40,606 INFO [Deployment-Sync-Manager-Thread-0][[]] class com.cisco.epm.fullsync.FileUtil -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:- Creating file /opt/oracle/base/admin/cpm10/dpdump/primaryCert_b33db240-dfbe-11ef-bb78-12467b70bb3d.jks

2025-01-31 16:09:40,688 INFO [Deployment-Sync-Manager-Thread-0][[]] class com.cisco.epm.fullsync.FileUtil -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:- Created file /opt/oracle/base/admin/cpm10/dpdump/primaryCert_b33db240-dfbe-11ef-bb78-12467b70bb3d.jks

2025-01-31 16:09:40,688 INFO [Deployment-Sync-Manager-Thread-0][[]] cisco.epm.fullsync.primary.PrimarySyncManager -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:- Created cert

2025-01-31 16:09:40,688 INFO [Deployment-Sync-Manager-Thread-0][[]] class com.cisco.epm.fullsync.FullSyncInfoHelper -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:- updateFullSyncStatus:a31cab00-dfbe-11ef-bb78-12467b70bb3d,b33db240-dfbe-11ef-bb78-12467b70bb3d

2025-01-31 16:09:40,696 INFO [Deployment-Sync-Manager-Thread-0][[]] cisco.epm.fullsync.primary.PrimarySyncManager -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:- Sync request hostFQDN[ise331.tac.local] exportFilePath[/opt/oracle/base/admin/cpm10/dpdump/cpmsync_b33db240-dfbe-11ef-bb78-12467b70bb3d.dmp] syncRequestIdentifier[b33db240-dfbe-11ef-bb78-12467b70bb3d] syncRequestTime[Fri Jan 31 16:14:40 IST 2025] syncStatus[SYNC_INPROGRESS] transientSyncStatus[DBEXPORT_SUCCEEDED] syncRequestType[REGISTER]

2025-01-31 16:09:40,697 INFO [Deployment-Sync-Manager-Thread-0][[]] cisco.epm.fullsync.primary.PrimarySyncManager -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:- Added future object to the Map with key :ise331b33db240-dfbe-11ef-bb78-12467b70bb3d, the syncThreadsFutureMap: {ise331b33db240-dfbe-11ef-bb78-12467b70bb3d=java.util.concurrent.FutureTask@25d5b491}

2025-01-31 16:09:40,697 INFO [Deployment-Sync-Manager-Thread-0][[]] cisco.epm.fullsync.primary.PrimarySyncManager -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:- the timeout set for the Executor thread pool is :300 (minutes)

2025-01-31 16:09:40,697 INFO [Deployment-SyncRequest-Thread-0][[]] class com.cisco.epm.fullsync.AbstractSyncRequest -:::::- Sync thread Deployment-SyncRequest-Thread-0 execution started for the host :ise331, sync identifier :b33db240-dfbe-11ef-bb78-12467b70bb3d

2025-01-31 16:09:40,697 INFO [Deployment-SyncRequest-Thread-0][[]] class com.cisco.epm.fullsync.AbstractSyncRequest -:::::- overriding the sync start time from Fri Jan 31 16:14:40 IST 2025 to Fri Jan 31 16:09:40 IST 2025

Transferring Sync files

2025-01-31 16:09:40,697 INFO [Deployment-SyncRequest-Thread-0][[]] class com.cisco.epm.fullsync.AbstractSyncRequest -:::::-Transferring sync files...

2025-01-31 16:09:40,697 INFO [Deployment-SyncRequest-Thread-0][[]] class com.cisco.epm.fullsync.FullSyncInfoHelper -:::::- updateFullSyncStatus:a31cab00-dfbe-11ef-bb78-12467b70bb3d,b33db240-dfbe-11ef-bb78-12467b70bb3d

2025-01-31 16:09:40,705 INFO [Deployment-SyncRequest-Thread-0][[]] class com.cisco.epm.fullsync.HttpClientHelper -:::::- Tracking RegistrationAndSync - File Name : [ cpmsync_b33db240-dfbe-11ef-bb78-12467b70bb3d.dmp], Size in bytes : [ 131614567 ]

2025-01-31 16:09:40,705 INFO [Deployment-SyncRequest-Thread-0][[]] class com.cisco.epm.fullsync.HttpClientHelper -:::::- Tracking RegistrationAndSync - File Name : [ primaryCert_b33db240-dfbe-11ef-bb78-12467b70bb3d.jks], Size in bytes : [ 25683 ]

2025-01-31 16:09:40,705 INFO [Deployment-SyncRequest-Thread-0][[]] class com.cisco.epm.fullsync.HttpClientHelper -:::::- Tracking RegistrationAndSync - File Name : [ config_b33db240-dfbe-11ef-bb78-12467b70bb3d.properties], Size in bytes : [ 263 ]

2025-01-31 16:09:40,706 INFO [Deployment-SyncRequest-Thread-0][[]] class com.cisco.epm.fullsync.HttpClientHelper -:::::- Tracking RegistrationAndSync - File Name : [ repo_random.key], Size in bytes : [ 24 ]

2025-01-31 16:09:41,920 INFO [Deployment-SyncRequest-Thread-0][[]] class com.cisco.epm.fullsync.HttpClientHelper -:::::- Creating http connection manager

2025-01-31 16:09:42,002 INFO [Deployment-SyncRequest-Thread-0][[]] class com.cisco.epm.fullsync.HttpClientHelper -:::::- Setting credentials on http client

2025-01-31 16:09:42,002 INFO [Deployment-SyncRequest-Thread-0][[]] class com.cisco.epm.fullsync.HttpClientHelper -:::::- Tracking RegistrationAndSync -Starting transfering the file to the secondary node ise331.tac.local

2025-01-31 16:09:42,002 INFO [Deployment-SyncRequest-Thread-0][[]] class com.cisco.epm.fullsync.HttpClientHelper -:::::- Tracking RegistrationAndSync -the secondary url ishttps://ise331.tac.local/deployment-rpc/uploadSyncFiles

2025-01-31 16:09:52,567 WARN [Deployment-SyncRequest-Thread-0][[]] class com.cisco.epm.fullsync.HttpClientHelper -:::::-Successfully Transferd the file to the secondary node ise331.tac.local

2025-01-31 16:09:52,568 WARN [Deployment-SyncRequest-Thread-0][[]] class com.cisco.epm.fullsync.HttpClientHelper -:::::- Upload complete, response=

2025-01-31 16:09:52,568 INFO [Deployment-SyncRequest-Thread-0][[]] class com.cisco.epm.fullsync.FullSyncInfoHelper -:::::- updateFullSyncStatus:a31cab00-dfbe-11ef-bb78-12467b70bb3d,b33db240-dfbe-11ef-bb78-12467b70bb3d

2025-01-31 16:09:52,570 INFO [Deployment-SyncRequest-Thread-0][[]] class com.cisco.epm.fullsync.AbstractSyncRequest -:::::-Sync files transferred. Now starting the sync process on secondary node...

Step 10: Receiving files from ise330

ise331-replication.log

2025-01-31 16:09:45,814 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::-Tracking RegistrationAndSync - starting receiveSyncFiles PAP:

2025-01-31 16:09:45,819 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Tracking RegistrationAndSync - Inside downloadSyncFilesAndVerify : Request org.apache.catalina.connector.RequestFacade@524def8e Response :org.apache.catalina.connector.ResponseFacade@51b18399

2025-01-31 16:09:45,821 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- The max upload file count for sync set to : 6

2025-01-31 16:09:48,164 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Number of items downloaded : 6

2025-01-31 16:09:48,190 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- FileItem name : cpmsync_b33db240-dfbe-11ef-bb78-12467b70bb3d.dmp

2025-01-31 16:09:49,228 WARN[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- secondaryCheckSum [9d2a72222b701adc69f8f70a85e83eca3bbeead3fe025fbac94e9d77fc8fb269]

2025-01-31 16:09:49,229 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Field Name = cpmsync_b33db240-dfbe-11ef-bb78-12467b70bb3d.dmp,File Name = null,Content type = null,File Size = 131614567

2025-01-31 16:09:49,229 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Saving file : cpmsync_b33db240-dfbe-11ef-bb78-12467b70bb3d.dmp

2025-01-31 16:09:49,229 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Saved file : cpmsync_b33db240-dfbe-11ef-bb78-12467b70bb3d.dmp

2025-01-31 16:09:49,229 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- FileItem name : primaryCert_b33db240-dfbe-11ef-bb78-12467b70bb3d.jks

2025-01-31 16:09:49,229 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Field Name = primaryCert_b33db240-dfbe-11ef-bb78-12467b70bb3d.jks,File Name = null,Content type = null,File Size = 25683

2025-01-31 16:09:49,229 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Saving file : primaryCert_b33db240-dfbe-11ef-bb78-12467b70bb3d.jks

2025-01-31 16:09:49,229 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Saved file : primaryCert_b33db240-dfbe-11ef-bb78-12467b70bb3d.jks

2025-01-31 16:09:49,229 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- FileItem name : config_b33db240-dfbe-11ef-bb78-12467b70bb3d.properties

2025-01-31 16:09:49,229 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Field Name = config_b33db240-dfbe-11ef-bb78-12467b70bb3d.properties,File Name = null,Content type = null,File Size = 263

2025-01-31 16:09:49,229 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Saving file : config_b33db240-dfbe-11ef-bb78-12467b70bb3d.properties

2025-01-31 16:09:49,229 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Saved file : config_b33db240-dfbe-11ef-bb78-12467b70bb3d.properties

2025-01-31 16:09:49,229 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- FileItem name : repo_random.key

2025-01-31 16:09:49,229 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Field Name = repo_random.key,File Name = null,Content type = null,File Size = 24

2025-01-31 16:09:49,229 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Saving file : repo_random.key

2025-01-31 16:09:49,229 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Saved file : repo_random.key

2025-01-31 16:09:49,229 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- FileItem name : checkSum

2025-01-31 16:09:49,229 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- FileItem name : syncRequestIdentifier

2025-01-31 16:09:49,229 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::-Tracking RegistrationAndSync - about to validateAndPrepareForSync

2025-01-31 16:09:49,243 INFO[admin-http-pool14][[]] cisco.epm.fullsync.secondary.SecondarySyncHelper -:::::-files downloaded )cpmsync_b33db240-dfbe-11ef-bb78-12467b70bb3d.dmpconfig_b33db240-dfbe-11ef-bb78-12467b70bb3d.propertiesprimaryCert_b33db240-dfbe-11ef-bb78-12467b70bb3d.jks

2025-01-31 16:09:50,294 INFO[admin-http-pool14][[]] cisco.epm.fullsync.secondary.SecondarySyncHelper -:::::- secondaryCheckSum [9d2a72222b701adc69f8f70a85e83eca3bbeead3fe025fbac94e9d77fc8fb269] primaryCheckSum [9d2a72222b701adc69f8f70a85e83eca3bbeead3fe025fbac94e9d77fc8fb269]

2025-01-31 16:09:52,544 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Tracking RegistrationAndSync - completed receiveSyncFilesTracking RegistrationAndSync -

2025-01-31 16:09:52,901 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Tracking RegistrationAndSync - about to startSync

2025-01-31 16:09:52,902 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Received syncRequestIdentifier from PAP: b33db240-dfbe-11ef-bb78-12467b70bb3d

2025-01-31 16:09:52,902 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Received secondaryHostName from PAP: ise331.tac.local

2025-01-31 16:09:52,902 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Received primUserDbPwd from PAP

2025-01-31 16:09:52,902 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Received dataDirectDbPwd from PAP

2025-01-31 16:09:52,902 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Received nodeExporterPwd from PAP

2025-01-31 16:09:52,902 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Received piProfilerMetricPwd from PAP

2025-01-31 16:09:52,902 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::- Received primKekKey from PAP

2025-01-31 16:09:52,902 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::-Received papAdminPwd from PAP

2025-01-31 16:09:52,902 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::-Received syncRequestType from PAP: REGISTER

2025-01-31 16:09:52,902 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::-About to invoke startImportSyncPollerScript

2025-01-31 16:09:52,902 INFO[admin-http-pool14][[]] cisco.epm.fullsync.secondary.SecondarySyncManager -:::::- Invoke sync poller to wait for sync completion ise331.tac.local b33db240-dfbe-11ef-bb78-12467b70bb3d [PrimaryDBPSWD] REGISTER

2025-01-31 16:09:53,082 INFO[admin-http-pool14][[]] cisco.epm.fullsync.secondary.SecondarySyncManager -:::::- returning sync poller to wait for sync completion

2025-01-31 16:09:53,083 INFO[admin-http-pool14][[]] cpm.infrastructure.deployment.rpc.DeploymentRegistrationListener -:::::-invoked startImportSyncPollerScript

2025-01-31 16:10:16,217 DEBUG[main][[]] cisco.cpm.cluster.impl.BaseServiceInvoker -:::::-Registering NodeInfoService

2025-01-31 16:11:29,181 INFO[main][[]] cisco.epm.fullsync.secondary.SecondarySyncManager -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:- Sending Transient Sync status:DBIMPORT_INITIATED, Node Sync Status:SYNC_INPROGRESS to PAP. Sync req id : b33db240-dfbe-11ef-bb78-12467b70bb3d

2025-01-31 16:11:29,199 INFO[main][[]] class com.cisco.epm.fullsync.FileUtil -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:- Reading primary node info from : /opt/oracle/base/admin/cpm10/dpdump/config_b33db240-dfbe-11ef-bb78-12467b70bb3d.properties

2025-01-31 16:11:29,199 INFO[main][[]] class com.cisco.epm.fullsync.HttpClientHelper -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:- syncRequestIdentifier..local value : b33db240-dfbe-11ef-bb78-12467b70bb3d & syncRequestIdentifier in config file : b33db240-dfbe-11ef-bb78-12467b70bb3d

2025-01-31 16:11:29,199 INFO[main][[]] class com.cisco.epm.fullsync.HttpClientHelper -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:- Sending sync status to [https://ise330.tac.local/deployment-rpc/updateSyncStatus] for syncRequestIdentifier b33db240-dfbe-11ef-bb78-12467b70bb3d

2025-01-31 16:11:29,199 INFO[main][[]] class com.cisco.epm.fullsync.HttpClientHelper -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:- Creating http connection manager

2025-01-31 16:11:29,212 DEBUG[main][[]] class com.cisco.epm.fullsync.HttpClientHelper -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:- syncRequestIdentifier ---> b33db240-dfbe-11ef-bb78-12467b70bb3d

2025-01-31 16:12:27,369 INFO[main][[]] class com.cisco.epm.fullsync.HttpClientHelper -:::b33db240-dfbe-11ef-bb78-12467b70bb3d::- Setting credentials on http client

2025-01-31 16:12:27,744 INFO[main][[]] class com.cisco.epm.fullsync.HttpClientHelper -:::b33db240-dfbe-11ef-bb78-12467b70bb3d::- Successfully udpated primary with the sync status

Step 11: DB import on ise331

ise331-replication.log

2025-01-31 16:12:27,746 INFO[main][[]] class com.cisco.epm.fullsync.DbImportDaemon -:::b33db240-dfbe-11ef-bb78-12467b70bb3d::-Starting DB import..

2025-01-31 16:12:27,746 INFO[main][[]] class com.cisco.epm.fullsync.DbImportDaemon -:::b33db240-dfbe-11ef-bb78-12467b70bb3d::- Script to run [ /usr/bin/sudo /opt/CSCOcpm/bin/sync_import.sh ise331.tac.local b33db240-dfbe-11ef-bb78-12467b70bb3d REGISTER ]

2025-01-31 16:16:20,351 INFO[main][[]] class com.cisco.epm.fullsync.DbImportDaemon -:::b33db240-dfbe-11ef-bb78-12467b70bb3d::-DB import completed.

2025-01-31 16:16:20,351 INFO[main][[]] class com.cisco.epm.fullsync.DbImportDaemon -:::b33db240-dfbe-11ef-bb78-12467b70bb3d::- Script to run [ /usr/bin/sudo /opt/CSCOcpm/bin/setdbpw.sh primaryUserDbPWD ]

2025-01-31 16:18:32,753 INFO[main][[]] class com.cisco.epm.fullsync.DbImportDaemon -:::b33db240-dfbe-11ef-bb78-12467b70bb3d::- DB password is reset.

2025-01-31 16:18:32,754 INFO[main][[]] cisco.epm.fullsync.secondary.SecondarySyncManager -:::b33db240-dfbe-11ef-bb78-12467b70bb3d::- Sending Transient Sync status:SECONDARY_SYNC_COMPLETED, Node Sync Status: SYNC_COMPLETED to PAP

ise331-ade.log

2025-01-31T16:12:30.546657+05:30 ise331 root: info:[sync_import.sh] b33db240-dfbe-11ef-bb78-12467b70bb3d:Starting the datadump import ...

2025-01-31T16:16:19.734878+05:30 ise331 root: info:[application:operation:sync_import.sh] b33db240-dfbe-11ef-bb78-12467b70bb3d:DB Sync Import successfully completed

Deleting DB after import is completed

2025-01-31T16:16:19.895863+05:30 ise331 root: info:[application:operation:sync_import.sh]Deleting provisioning resources from DB after sync completed ...

2025-01-31T16:16:20.341542+05:30 ise331 root: info:[application:operation:sync_import.sh] b33db240-dfbe-11ef-bb78-12467b70bb3d:Released the DATABASE_SYNCIMPORT Lock

2025-01-31T16:16:20.345768+05:30 ise331 root: info:[application:operation:sync_import.sh] b33db240-dfbe-11ef-bb78-12467b70bb3d:Datadump import process completed

2025-01-31T16:16:20.738342+05:30 ise331 root: info:[application:operation:setdbpw.sh]Resetting internal DB user password to match that of ISE Primary

Step 12: Updating sync completion status to ise330

ise331-replication.log

2025-01-31 16:18:32,754 INFO[main][[]] cisco.epm.fullsync.secondary.SecondarySyncManager -:::b33db240-dfbe-11ef-bb78-12467b70bb3d::- Sending Transient Sync status:SECONDARY_SYNC_COMPLETED,Node Sync Status: SYNC_COMPLETED to PAP. Sync req id : b33db240-dfbe-11ef-bb78-12467b70bb3d

2025-01-31 16:29:59,419 INFO[Psc-SyncStatusUpdaterJob-Thread][[]] Replication-Deployment.PrimaryNodeProxy -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:- Sending ise331's sync statusSYNC_COMPLETEDafter full sync to Primary, objId: a31cab00-dfbe-11ef-bb78-12467b70bb3d, syncReqId: b33db240-dfbe-11ef-bb78-12467b70bb3d, hostConfigMdDate: 2025-01-31 16:16:19.880098

ise330-replication.log

2025-01-31 16:29:59,429 INFO[OOB-8,ISERepCluster-F3ixn,ise330-46234][[]] cisco.cpm.deployment.replication.MasterNodeHandlerImpl -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:-Recieved status:SYNC COMPLETED,objid:a31cab00-dfbe-11ef-bb78-12467b70bb3d,syncReqId:b33db240-dfbe-11ef-bb78-12467b70bb3d,hostconfigModDt:2025-01-31 16:16:19.880098 from ise331

2025-01-31 16:29:59,480 INFO[OOB-8,ISERepCluster-F3ixn,ise330-46234][[]] cisco.cpm.deployment.replication.MasterNodeHandlerImpl -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:-The update is for the latest full sync information found in the database.

2025-01-31 16:29:59,480 INFO[OOB-8,ISERepCluster-F3ixn,ise330-46234][[]] cisco.cpm.deployment.replication.MasterNodeHandlerImpl -:::b33db240-dfbe-11ef-bb78-12467b70bb3d:FullSync:- Updating ise331's status fromIN PROGRESS to SYNC COMPLETED

Alarms related to node sync issues

- Slow replication Info

- Slow replication warning

- Slow replication Error

- Replication failed

- Replication Stopped

- Node out of sync

Known Defects

Bug ID

Description

CSCwj67089

Cisco ISE app server crashes while importing large files to secondary node through local disk management.

CSCwi78722

Azure VM: Not able to register node to deployment.

CSCwh70275

While registering node with left over certificates from deregistration, the certificates that are currently in use get deleted.

CSCwh08408

Cisco ISE Release 3.3 cannot register new nodes to the deployment post upgrade due to the node exporter password not being found.