Integrate Cloud-delivered Firewall Management Center with ISE Via pxGrid Cloud

Introduction

This document describes the procedure to integrate Cisco ISE with cloud-delivered Firewall Management Center (cdFMC) via pxGrid Cloud.

Prerequisites

• A working knowledge of Cisco Identity Service Engine (ISE)
• An account in Cisco Catalyst Cloud Portal
• An account on Cisco Security Cloud Control Portal

Requirements

• Install and activate the Advantage license tier in your Cisco ISE deployment.
• The pxGrid Cloud agent creates an outbound HTTPS connection to Cisco pxGrid Cloud. Therefore, configure Cisco ISE proxy settings if network uses a proxy to reach the internet. To configure proxy settings in Cisco ISE, choose Administration > System > Settings > Proxy.
• The Cisco ISE Trusted Certificates Store must include the root CA certificate required to validate the server certificate presented by Cisco pxGrid Cloud. Ensure that the Trust for Authentication of Cisco Services option is enabled for this root CA certificate. To enable Trust for Authentication of Cisco Services, choose Administration > System > Certificates.
• Ensure that port 443 is open for outbound connection from Cisco ISE to Cisco pxGrid Cloud Portal. If firewall or proxy settings are configured, ensure that these URLs are not blocked.

https://dna.cisco.com

https://dnaservices.cisco.com

https://ciscodnacloud.com

Components Used

ISE software version: 3.4 Patch 1, 4 nodes deployment (PAN, SAN and 2 PSN with pxGrid service enabled)

cdFMC version: 20241127

Cisco Firepower Threat Defense (FTD) for VMware version: 7.2.5

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Cisco pxGrid Cloud Terminology

Here are some of the common terms that are used in the Cisco pxGrid Cloud solution and their meaning in the Cisco pxGrid Cloud environment:

• Offer: A set of capabilities packaged together and offered as a solution
• Subscription: An instance of an offer being consumed by a tenant is a subscription
• App: You can create and register applications for your product based on your requirements.

Background Information

Cisco ISE provides context sharing between multiple security vendors; however, the current architecture does not allow communication between the on-prem ISE and cloud-based solutions through the network perimeter without some sort of bypass/putting holes in the firewall. 

Cisco ISE’s pxCloud is a cloud-based solution that solves this problem and enables context sharing between on-prem and the cloud without additional installation, overhead, and compromising the security of your network. It is secure and customizable, enabling you to share only the data that you want to share and consume only the contextual data that is relevant for your application. 

Cisco ISE release 3.1 patch 3 and later support pxGrid Cloud. Cisco and its partners can develop pxGrid Cloud-based applications and register them with the pxGrid Cloud offer. It relies on cisco DNA-Cloud Portal to onboard and register applications without depending on other on-premise infrastructure. These applications use the External RESTful Services (ERS), Open APIs and pxGrid (APIs and websocket) to exchange information with Cisco ISE to use subscription and user data from ISE in cdFMC.

Configure

Network Diagram

cdFMC and ISE Integration via pxGrid Cloud

Configurations

Four main steps involved are:

• Import the Cisco pxGrid Cloud server certificate in ISE.
• Register ISE with Catalyst Cloud Portal.
• Activate the cdFMC Application on ISE using Integration Catalog
• Create pxGrid Application (cdFMC) Instance

Import the Cisco pxGrid Cloud server certificate in ISE

ISE must establish trust with the Cisco pxGrid Cloud. Even though the cloud website is authenticated with a publicly signed certificate, ISE does not maintain a complete list of trusted root CAs. Therefore administrator must establish a trust relationship. Export the pxGrid Cloud Root and Intermediate certificates by browsing to Catalyst Cloud Portal . Most browsers allow this. Here are the steps to get certificate from Chrome Browser. 

View the Site Information

Export the Certificate Chain

Import the certificates into "Trusted Certificates" store in ISE if the certificate chain is missing (ISE already have IdenTrust Commercial Root CA 1). 

Ensure that the "Trust for Authentication of Cisco Services" option is enabled for this root CA certificate. To enable Trust for Authentication of Cisco Services, choose Administration > System > Certificates.

Enable Trust for authentication of Cisco Services

Flow Diagram

App Activation Workflow

ISE deployment used in this setup

Registration of ISE with Catalyst Cloud Portal

Enable pxGrid Cloud service in Cisco ISE and register your device.

1. In the Cisco ISE GUI, choose Administration > System > Deployment.

2. Click the node on which you want to enable the pxGrid Cloud service ( in this case first PSN node).

3. In the General Settings tab, enable the pxGrid service.

4. Check the pxGrid Cloud check box.

Note: The pxGrid Cloud service can be enabled only on two nodes to enable high availability. You can enable the pxGrid Cloud option only when the pxGrid service is enabled on that node.

5. In the ISE deployment name field, enter a meaningful name. This name is shown in Catalyst Cloud Portal and can be used to distinguish if multiple ISE deployments are registered with cloud. You can verify your registered Cisco ISE deployment on the Cisco Catalyst Cloud Portal using the ISE deployment name.

(Optional) In the Description (optional) field, enter a description for your Cisco ISE deployment.

6. In the Region drop-down list, choose a region to register your Cisco ISE device. Cisco pxGrid Cloud is now supported in Europe, Asia Pacific and Japan in addition to the U.S.. Note that the application that you want to use with pxGrid Cloud must also be available in the same region.

7. Click Register.

Register ISE PSN with pxGrid Cloud

8. In the Activate your device pop-up page, the Activation Code for your device is automatically filled. Click Next.

Activate the device

Device Activated

Note: When you enable "pxGrid Cloud" persona on second node, ISE do not need all these details as the registration of ISE with pxGrid Cloud is at deployment level.

9. Log in to your Cisco Catalyst Cloud Portal account using your login credentials. If you do not have login credentials then create a new account to complete your device registration. For more information, see Create an account on Cisco Catalyst Cloud Portal

Your Cisco ISE device is activated and registered.

ISE node registered with Catalyst Cloud Portal

10. You can find details of your registered Cisco ISE in the pxGrid section (Administration > System> Deployment > pxGrid).

Verify ISE registered with pxGrid Cloud

You can click Deregister to deregister your Cisco ISE device. Deregistering Cisco ISE also automatically deactivates the connected applications.

Activate the cdFMC Application on ISE using Integration Catalog

1. In the ISE GUI, choose Administration > Integration Catalog.

2. Under the Available integrations, select Firewall Management center application.

Note: The list of applications depend on the account. Some application could be exposed to only specific accounts.

Integration catalog

3. In the App configuration section, Select New instance and choose the Data scopes for your app configuration. Choose at least one data scope to proceed.

Note: When you select a data scope, it also enables the same under system level pxGrid Cloud Policy settings.

4. Click Activate to activate the app.

FMC Application Configuration on ISE

5. From the One-Time Password (OTP) pop-up, copy the OTP to redeem it on cdFMC while creating pxGrid Application instance

OTP for cdFMC Application

6. Configure pxGrid Cloud Policy by navigating to Administration > pxgrid Services > Client Management > pxGrid Cloud Policy. Select the pxGrid services, that you wish to share with the Saas Applications and enable External RESTful Services (ERS) APIs and OpenAPIs read only access to Cisco pxGrid Cloud applications.

Configure pxGrid Cloud Policy

Note: Echo service is used to run health checks to determine both pub-sub and API connectivity to ISE.

         By default, the Cisco pxGrid Cloud applications are granted Read Only access to the APIs (only HTTP GET operations can be performed). Enable the Read/Write option in the pxGrid Cloud Policy window if you want to allow POST, PUT, and DELETE operations as well.

Create pxGrid Application (cdFMC) Instance 

1. Login to Security Cloud Control (SCC) Portal as a user with the Super Admin role.

SCC Login Page

2. From the Security Cloud Control menu, click Administration > Integrations > Firewall Management Center and select your cdFMC instance and on the right pane options, select System > Configuration.

Navigating to Firewall Management Center

3. On the Configuration Page, select Integration > Other Integrations > Identity Sources > Choose Service Type Identity Services Engine (pxGrid Cloud). Click on Create pxGrid Application Instance and redeem the OTP copied from Cisco ISE to add an instance.

Create cdFMC Application Instance

4. Verify it on Cisco Catalyst Cloud Portal at Applications and Products > Firewall Management Center > Manage > Products > Select Instance drop down menu. 

Verify the cdFMC on Catalyst Cloud Portal

5. Select newly created cdFMC application and click Add. Select Region and click Activate.  

Select Region

5. Choose your application instance and click Next. Choose your Product (ISE pxGrid node), click Next.

6. Configure Access Control: Choose the functional capabilities to be allowed for cdFMC on your choosen ISE product. Click Next. Configuration summary is displayed. Verify and click Activate.

Configure Access Control for your cdFMC

Product ISE connected to cfFMC

8. Verify it on Cisco Catalyst Cloud Portal at Applications and Products > Firewall Management Center > Manage > Products > Select Instance drop down menu. 

Verify that App is activated

Verify

1.  On the Catalyst Cloud Portal, Navigate to Applications and Products. Select your ISE product name and check the Product Details. Verify that cdFMC is seen under Activated Application.

Verify on Catalyst Cloud Control Portal

2.  On Security Cloud Control, test the configured cdFMC application Instance. The test shows "Success"

Verify on Security Cloud Control Portal

3. Login to Active pxGrid node and verify that Hermes (pxGrid Cloud Agent) is in running state using the command show application status ise. This agent is in disabled status on Standby pxGrid node.

Verify Hermes (pxGrid Cloud Agent) status

Check the pxcloud.log on both pxGrid nodes to confirm the Active and Standby Status:

On Active pxGrid node (pxcloud.log)

2025-03-17 14:35:25,530 DEBUG  [pxCloud-hermesCheck-2768][[]] cpm.pxcloud.ha.statemachine.StateMachine -:::::- RUNNING (HERMES_OK) ------> RUNNING
2025-03-17 14:35:27,438 DEBUG  [pxCloud-heartbeat-2769][[]] cpm.pxcloud.ha.statemachine.HeartBeat -:::::- url - https://ise341-psn2.poongarg.local:8910/pxgrid/pxcloud/statusLookup
2025-03-17 14:35:27,445 DEBUG  [pxCloud-heartbeat-2769][[]] cpm.pxcloud.ha.statemachine.HeartBeat -:::::- HeartBeat response from peer - StatusResponse [role=STANDBY, state=MONITORING, pxGridConnectionStatus=NOT_CONNECTED, cloudConnectionStatus=NOT_CONNECTED, reason=] 
2025-03-17 14:35:27,445 DEBUG  [pxCloud-heartbeat-2769][[]] cpm.pxcloud.ha.statemachine.HeartBeat -:::::- Post PEER_MONITORING to state machine
2025-03-17 14:35:27,445 DEBUG  [pxCloud-heartbeat-2769][[]] cpm.pxcloud.ha.statemachine.StateMachine -:::::- RUNNING (PEER_MONITORING) 
2025-03-17 14:35:35,548 DEBUG  [pxCloud-hermesCheck-2768][[]] cpm.pxcloud.ha.statemachine.HermesCheck -:::::- Sending request to Hermes: POST https://localhost:8913/hermes/cloudConnectionStatus
2025-03-17 14:35:35,572 DEBUG  [pxCloud-hermesCheck-2768][[]] cpm.pxcloud.ha.statemachine.HermesCheck -:::::- Hermes response: 200 OK
2025-03-17 14:35:35,572 DEBUG  [pxCloud-hermesCheck-2768][[]] cpm.pxcloud.ha.statemachine.HermesCheck -:::::- Status - HermesConnectionStatus [pxGridConnectionStatus=CONNECTED, cloudConnectionStatus=CONNECTED, reason=] 

On Standby pxGrid node (pxcloud.log)

2025-03-17 14:34:14,145 DEBUG  [pxCloud-heartbeat-6441][[]] cpm.pxcloud.ha.statemachine.HeartBeat -:::::- url - https://ise341-psn1.poongarg.local:8910/pxgrid/pxcloud/statusLookup
2025-03-17 14:34:14,153 DEBUG  [pxCloud-heartbeat-6441][[]] cpm.pxcloud.ha.statemachine.HeartBeat -:::::- HeartBeat response from peer - StatusResponse [role=ACTIVE, state=RUNNING, pxGridConnectionStatus=CONNECTED, cloudConnectionStatus=CONNECTED, reason=]
2025-03-17 14:34:14,154 DEBUG  [pxCloud-heartbeat-6441][[]] cpm.pxcloud.ha.statemachine.HeartBeat -:::::- Post PEER_RUNNING to state machine
2025-03-17 14:34:14,154 DEBUG  [pxCloud-heartbeat-6441][[]] cpm.pxcloud.ha.statemachine.StateMachine -:::::- MONITORING (PEER_RUNNING)

Additionally check the port 8913, which gets opened only on the ACTIVE pxGrid node:

ise341-psn1/admin#show ports | include 8913                                     
           tcp: 127.0.0.1:8913
ise341-psn1/admin# 

4. Verify pxGrid cloud client by navigating to Administration > pxGrid Services > Client Management > Clients > pxGrid Cloud clients. Also check the subscribed topics.

pxGrid Cloud Client on ISE

5. Verify the subscribed Topics are fetched on cdFMC. On Security Cloud Control portal, Click Policies > Threat Defense > Integration > Other Integrations > Identity Sources. Click Identity Services Engine (pxGrid Cloud). Click Configure Filters. On the page, click the Dynamic Attributes Filter tab. Create a Dynamic Attributes Filter.

Fetched attributes from ISE

Troubleshoot

1. Failure during ISE registration:

Missing Proxy Configuration

Check the internet connectivity and possible proxy misconfiguration.

2.  pxGrid status shows Not connected on ISE Edit node page after enabling pxGrid Cloud service and configuring the name and region parameters.

Check the hermes.log on the node where you are enabling the pxGrid Cloud service:

ise341-psn1/admin#show logging application hermes/hermes.log | begin 8913
2025-03-17T09:19:35.277Z | INFO | hermes/httpserver.go:57 | Starting REST server on :8913
2025-03-17T09:19:35.285Z | INFO | hermes/httpserver.go:78 | REST server is up and running
2025-03-17T09:19:35.307Z | ERROR | hermes/pxgrid.go:194 | Failed to establish pxGrid WebSocket connection: pubsub service lookup failed: service lookup for pubsub service failed: Post "https://ise341-psn1.poongarg.local:8910/pxgrid/control/ServiceLookup": SSL errors: SSL routines:tls_process_server_certificate:certificate verify failed
2025-03-17T09:19:35.307Z | ERROR | hermes/main.go:166 | Failed to open pxGrid WebSocket connection: Failed to establish pxGrid WebSocket connection: pubsub service lookup failed: service lookup for pubsub service failed: Post "https://ise341-psn1.poongarg.local:8910/pxgrid/control/ServiceLookup": SSL errors: SSL routines:tls_process_server_certificate:certificate verify failed
2025-03-17T09:19:35.307Z | INFO | hermes/config.go:279 | Stopping monitoring of configuration file: /opt/hermes/config.yaml
2025-03-17T09:19:35.307Z | INFO | hermes/connectionstatus.go:81 | Resetting connection status to DISCONNECTED with reason 'Failed to establish pxGrid WebSocket connection: pubsub service lookup failed: service lookup for pubsub service failed: Post "https://ise341-psn1.poongarg.local:8910/pxgrid/control/ServiceLookup": SSL errors: SSL routines:tls_process_server_certificate:certificate verify failed'
2025-03-17T09:19:35.308Z | ERROR | hermes/main.go:402 | Error running Hermes: Failed to establish pxGrid WebSocket connection: pubsub service lookup failed: service lookup for pubsub service failed: Post "https://ise341-psn1.poongarg.local:8910/pxgrid/control/ServiceLookup": SSL errors: SSL routines:tls_process_server_certificate:certificate verify failed
2025-03-17T09:19:35.308Z | INFO | hermes/httpserver.go:90 | Stopping REST server on :8913

Hermes Rest server listen on port 8913. Logs clearly shows that Hermes REST server is trying to start but failed to establish pxGrid WebSocket connection due to Certificate verification failure.

Solution: Verify that the pxGrid Certificate is valid and the certificate chain is not broken. View the certificate and verify Certificate status is good. In this case, the ISE host name was incorrect in the pxGrid certificate issued to this node.

Valid pxGrid certificate

3. cdFMC application activation failed on Catalyst Cloud Portal.

Solution: Make sure that on ISE, under the pxGrid Cloud Policy, External RESTful Services (ERS) APIs and OpenAPIs read only access is enabled.

Logs related to pxGrid Cloud feature:

Note: Regardless of the node on which you are enabling the pxGrid Cloud persona, you need to check the logs on the active PAN node. Once device is registered, the Hermes logs is on the specific node where it is enabled.

         Hermes.log supports only the Debug, Info, Warn, and Error log levels. Hence, if you choose Trace, the log level is set as Debug for hermes.log. If you choose Fatal, the log level is set as Error for hermes.log.

ISE Enrollment/Registeration and App. activation Flow

Before the device is registered, ISE uses a device token to fetch list of regions, list of apps from the cloud. This token is provide by the "telemetry" component of ISE. Check sch.log on PAN node:

2025-03-17 09:10:23,361 INFO [openapi-http-pool7][[]] infrastructure.telemetry.sch.api.DNATelemetryClient -:::::- Telemetry Lifecycle configured : PRODUCTION
2025-03-17 09:10:23,361 INFO [openapi-http-pool7][[]] infrastructure.telemetry.sch.api.DNATelemetryClient -:::::- Telemetry lifecycle::PRODUCTION
2025-03-17 09:10:23,463 INFO [openapi-http-pool7][[]] infrastructure.telemetry.sch.api.TetheringStateStorageImpl -:::::- Read tethering state from the db...
2025-03-17 09:10:23,467 INFO [openapi-http-pool7][[]] infrastructure.telemetry.sch.api.TetheringStateStorageImpl -:::::- Get encryption key for the tethering state data...
2025-03-17 09:10:23,480 INFO [openapi-http-pool7][[]] cisco.dna.tethering.client.TetheringClient -:::::- DNA Cloud Tethering Client Initialized, member ID = 67d7c58488c5fd08d085fffd, enrollment = existing
2025-03-17 09:10:23,480 INFO [openapi-http-pool7][[]] infrastructure.telemetry.sch.api.DNATelemetryClient -:::::- Tethering Client Initialized successfully
2025-03-17 09:10:23,483 INFO [openapi-http-pool7][[]] infrastructure.telemetry.sch.api.DNATetheringClient -:::::- Initialization Details :- Member Id: 67d7c58488c5fd08d085fffd
2025-03-17 09:10:24,492 INFO [openapi-http-pool7][[]] infrastructure.telemetry.sch.api.TetheringStateStorageImpl -:::::- Get encryption key for the tethering state data...
2025-03-17 09:10:24,497 INFO [openapi-http-pool7][[]] infrastructure.telemetry.sch.api.TetheringStateStorageImpl -:::::- Write tethering state to the db...
2025-03-17 09:10:24,529 INFO [openapi-http-pool7][[]] cpm.infrastructure.telemetry.api.TelemetryConfigHandler -:::::- Updated Tethering State in the TelemetryConfig table

pxcloud.log (PAN node), once you enable pxGrid Cloud service, Hermes(pxGrid Cloud agent) gets activated and ISE fetch the region information and received a token via telemetry component.

2025-03-17 08:47:00,300 INFO [main][[]] cisco.cpm.pxcloud.api.PxCloudInitializer -:::::- Initializing pxGrid Cloud
2025-03-17 08:47:00,312 INFO [main][[]] cisco.cpm.pxcloud.pxgrid.PxCloudProviderRegistration -:::::- Registering the pxCloud service into pxGrid
2025-03-17 08:47:00,314 INFO [main][[]] cisco.cpm.pxcloud.hermes.ProxyConfigNotificationHandler -:::::- Register pxCloud ProxyConfig notification handler
2025-03-17 08:47:00,376 INFO [main][[]] cisco.cpm.pxcloud.hermes.HermesConfigManager -:::::- Registering listener for Hermes cert file changes
2025-03-17 08:47:00,376 INFO [main][[]] cisco.cpm.pxcloud.hermes.HermesConfigManager -:::::- Registering listener for pxCloud log level changes
2025-03-17 08:50:18,842 INFO [main][[]] cisco.cpm.pxcloud.hermes.HermesConfigManager -:::::- Updating Hermes certificate files
2025-03-17 08:52:46,834 INFO [pool-24-thread-1][[]] cpm.pxcloud.service.ui.IntegrationCatalogScheduler -:::::- validateIfApplicationStatusChanged :: Scheduler Invoked
2025-03-17 08:55:46,877 INFO [pool-24-thread-1][[]] cpm.pxcloud.service.ui.IntegrationCatalogScheduler -:::::- validateIfApplicationStatusChanged :: Scheduler Invoked
2025-03-17 08:58:46,781 INFO [pool-24-thread-1][[]] cpm.pxcloud.service.ui.IntegrationCatalogScheduler -:::::- validateIfApplicationStatusChanged :: Scheduler Invoked
2025-03-17 09:01:46,781 INFO [pool-24-thread-1][[]] cpm.pxcloud.service.ui.IntegrationCatalogScheduler -:::::- validateIfApplicationStatusChanged :: Scheduler Invoked
2025-03-17 09:03:37,136 INFO [pool-225-thread-1][[]] cpm.pxcloud.service.ui.IntegrationCatalogScheduler -:::::- validateIfApplicationStatusChanged :: Scheduler Invoked
2025-03-17 09:04:46,781 INFO [pool-24-thread-1][[]] cpm.pxcloud.service.ui.IntegrationCatalogScheduler -:::::- validateIfApplicationStatusChanged :: Scheduler Invoked
2025-03-17 09:06:37,136 INFO [pool-225-thread-1][[]] cpm.pxcloud.service.ui.IntegrationCatalogScheduler -:::::- validateIfApplicationStatusChanged :: Scheduler Invoked
2025-03-17 09:07:46,781 INFO [pool-24-thread-1][[]] cpm.pxcloud.service.ui.IntegrationCatalogScheduler -:::::- validateIfApplicationStatusChanged :: Scheduler Invoked
2025-03-17 09:09:11,901 DEBUG [hermes-change-monitor-0][[]] cisco.cpm.pxcloud.hermes.PxCloudNodeChangeHandler -:::::- Periodic check of PPAN hostFQDN: ise341-PAN.poongarg.local
2025-03-17 09:09:37,136 INFO [pool-225-thread-1][[]] cpm.pxcloud.service.ui.IntegrationCatalogScheduler -:::::- validateIfApplicationStatusChanged :: Scheduler Invoked
2025-03-17 09:09:37,139 DEBUG [pool-225-thread-1][[]] cpm.pxcloud.service.ui.IntegrationCatalogScheduler -:::::- ISE enrollment status is 
2025-03-17 09:10:22,475 TRACE [openapi-http-pool4][[]] cpm.iseopenapi.pxcloud.impl.PxGridApiDelegateImpl -:::::- Request to get device information.
2025-03-17 09:10:22,485 INFO [openapi-http-pool4][[]] cpm.pxcloud.service.ui.IseEnrollment -:::::- Fetching Device Info...
2025-03-17 09:10:22,739 TRACE [openapi-http-pool7][[]] cpm.iseopenapi.pxcloud.impl.PxGridApiDelegateImpl -:::::- Request to get regions is received
2025-03-17 09:10:22,750 INFO [openapi-http-pool7][[]] cpm.pxcloud.api.impl.DeviceRegistrationApiImpl -:::::- getRegionList :: get regions list api invoked.
2025-03-17 09:10:22,754 DEBUG [openapi-http-pool7][[]] cpm.pxcloud.api.impl.DeviceRegistrationApiImpl -:::::- getRegionList :: fetch device token.
2025-03-17 09:10:24,529 DEBUG [openapi-http-pool7][[]] cpm.pxcloud.api.impl.DeviceRegistrationApiImpl -:::::- getRegionList :: fetched device token.
2025-03-17 09:10:24,537 INFO [openapi-http-pool7][[]] cisco.cpm.pxcloud.utils.PxCloudHttpClient -:::::- Proxy configured. Address=x.x.x.x Port=80
2025-03-17 09:10:26,938 DEBUG [openapi-http-pool7][[]] cpm.pxcloud.api.impl.DeviceRegistrationApiImpl -:::::- getRegionList :: Retrieved region list response from cloud : HttpResponseProxy{HTTP/1.1 200 OK [Date: Mon, 17 Mar 2025 09:10:28 GMT, Content-Type: application/json; charset=utf-8, Content-Length: 452, Connection: keep-alive, X-Tracking-Id: da111708f760551999f8cdfb2bfcc6bb, vary: Origin, Access-Control-Allow-Credentials: true, Access-Control-Expose-Headers: X-Auth-Token, Via: api-gateway,upstream, Content-Security-Policy: default-src 'self'; base-uri 'self'; frame-ancestors 'self'; block-all-mixed-content] ResponseEntityProxy{[Content-Type: application/json; charset=utf-8,Content-Length: 452,Chunked: false]}}
2025-03-17 09:10:26,946 DEBUG [openapi-http-pool7][[]] cpm.pxcloud.api.impl.DeviceRegistrationApiImpl -:::::- getFilteredRegionInfo :: region list before filtering based on application list : [class Region {
id: ap-southeast-1
name: ap-southeast-1
fqdn: neoffers-sg.cisco.com
}, class Region {
id: eu-central-1
name: eu-central-1
fqdn: neoffers-de.cisco.com
}, class Region {
id: us-west-2
name: us-west-2
fqdn: neoffers.cisco.com
}]
2025-03-17 09:10:26,968 DEBUG [openapi-http-pool7][[]] cpm.pxcloud.api.impl.PxcloudApplicationCatalogImpl -:::::- Inside getApplicationCatalog
2025-03-17 09:10:27,051 INFO [openapi-http-pool7][[]] cpm.pxcloud.api.impl.PxcloudApplicationCatalogImpl -:::::- URL to get apps from cloud: https://dnaservices.cisco.com/api/uno/v1/assembler/data/applications
2025-03-17 09:10:27,055 DEBUG [openapi-http-pool7][[]] cisco.cpm.pxcloud.utils.PxCloudUtils -:::::- Anonymous token is used
2025-03-17 09:10:27,533 DEBUG [openapi-http-pool7][[]] cpm.pxcloud.api.impl.PxcloudApplicationCatalogImpl -:::::- Token is present
2025-03-17 09:10:27,533 DEBUG [openapi-http-pool7][[]] cpm.pxcloud.api.impl.PxcloudApplicationCatalogImpl -:::::- Request to getCatalogsUsingAnonymousToken is received
2025-03-17 09:10:27,533 DEBUG [openapi-http-pool7][[]] cpm.pxcloud.api.impl.PxcloudApplicationCatalogImpl -:::::- Inside getCatalog method
2025-03-17 09:10:27,533 DEBUG [openapi-http-pool7][[]] cisco.cpm.pxcloud.utils.PxCloudHttpClient -:::::- Inside httpGet method
!
2025-03-17 09:10:37,338 INFO [openapi-http-pool7][[]] cpm.pxcloud.api.impl.PxcloudApplicationCatalogImpl -:::::- Application Catalog status code 200 

Activation link is received and auto register and enrollment happens.

2025-03-17 09:16:42,536 TRACE [openapi-http-pool2][[]] cpm.iseopenapi.pxcloud.impl.PxGridApiDelegateImpl -:::::- Request to get verification url is received
2025-03-17 09:16:42,537 DEBUG [openapi-http-pool2][[]] cpm.pxcloud.api.impl.DeviceRegistrationApiImpl -:::::- getVerificationUri :: Checking Preconditions.
2025-03-17 09:16:42,569 DEBUG [openapi-http-pool2][[]] cpm.pxcloud.api.impl.DeviceRegistrationApiImpl -:::::- getVerificationUri :: Preconditions check completed.
2025-03-17 09:16:42,569 DEBUG [openapi-http-pool2][[]] cpm.pxcloud.api.impl.DeviceRegistrationApiImpl -:::::- getVerificationUri :: Trying to fetch activation link from platform.
!
2025-03-17 09:16:44,729 DEBUG [openapi-http-pool2][[]] cisco.cpm.pxcloud.api.PxCloudPropertiesNotificationHandler -:::::- onCreate DEVICE_CODE b6b4c9d0-4d07-4eb6-8d5e-b8a446f4a3eb
2025-03-17 09:16:44,735 DEBUG [openapi-http-pool2][[]] cpm.pxcloud.api.impl.DeviceRegistrationApiImpl -:::::- getVerificationUri :: Activation link retrieval completed.
2025-03-17 09:16:45,310 TRACE [openapi-http-pool3][[]] cpm.iseopenapi.pxcloud.impl.PxGridApiDelegateImpl -:::::- Request to verify CCO login and auto register if single tenant.
2025-03-17 09:16:45,345 INFO [openapi-http-pool3][[]] cpm.pxcloud.api.impl.DeviceRegistrationApiImpl -:::::- verifyCCOLoginAndAutoRegister:: Verify CCO login and try to auto-register if there is single tenant.
!
2025-03-17 09:16:45,538 INFO [openapi-http-pool3][[]] cpm.pxcloud.api.impl.DeviceRegistrationApiImpl -:::::- verifyCCOLoginAndAutoRegisterWithoutTenant :: Verify CCO login to check if device is activated.
2025-03-17 09:16:45,589 DEBUG [openapi-http-pool3][[]] cisco.cpm.pxcloud.utils.PxCloudHttpClient -:::::- Inside httpPost method
2025-03-17 09:16:46,805 INFO [pool-24-thread-1][[]] cpm.pxcloud.service.ui.IntegrationCatalogScheduler -:::::- validateIfApplicationStatusChanged :: Scheduler Invoked
2025-03-17 09:16:46,816 DEBUG [pool-24-thread-1][[]] cpm.pxcloud.service.ui.IntegrationCatalogScheduler -:::::- ISE enrollment status is 
2025-03-17 09:16:47,631 DEBUG [openapi-http-pool3][[]] cpm.pxcloud.api.impl.DeviceRegistrationApiImpl -:::::- verifyCCOLoginAndAutoRegisterWithoutTenant :: auth token info for autoregister 
!
2025-03-17 09:19:14,196 INFO [openapi-http-pool9][[]] cpm.pxcloud.api.impl.DeviceRegistrationApiImpl -:::::- verifyCCOLoginAndAutoRegisterWithoutTenant :: device is activated.
2025-03-17 09:19:14,196 INFO [openapi-http-pool9][[]] cpm.pxcloud.api.impl.DeviceRegistrationApiImpl -:::::- executeAutoRegister :: device is activated, going to execute auto register api.
2025-03-17 09:19:14,199 DEBUG [openapi-http-pool9][[]] cisco.cpm.pxcloud.utils.PxCloudHttpClient -:::::- Inside httpPost method
2025-03-17 09:19:16,956 DEBUG [openapi-http-pool9][[]] cpm.pxcloud.api.impl.DeviceRegistrationApiImpl -:::::- executeAutoRegister :: AutoRegister api executed successfully and response is HttpResponseProxy{HTTP/1.1 201 Created [Date: Mon, 17 Mar 2025 09:19:18 GMT, Content-Type: application/json; charset=utf-8, Content-Length: 704, Connection: keep-alive, vary: Origin, Access-Control-Allow-Credentials: true, Access-Control-Expose-Headers: X-Auth-Token, Via: api-gateway,upstream, Content-Security-Policy: default-src 'self'; base-uri 'self'; frame-ancestors 'self'; block-all-mixed-content] ResponseEntityProxy{[Content-Type: application/json; charset=utf-8,Content-Length: 704,Chunked: false]}}
2025-03-17 09:19:16,964 DEBUG [openapi-http-pool9][[]] cpm.pxcloud.api.impl.DeviceRegistrationApiImpl -:::::- executeAutoRegister :: AutoRegister api executed successfully and response is {"data":{"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJDRE5BIiwiYXV0aFNvdXJjZSI6Imh0dHBzOi8vaWQuY2lzY28uY29tL29hdXRoMi9kZWZhdWx0IiwiYXV0aHpUeXBlIjoiREVWSUNFX1JFR0lTVFJBVElPTiIsImV4cCI6MTc0MjIwNjc1OCwiaWF0IjoxNzQyMjAzMTU4LCJpc3MiOiJkbmFzZXJ2aWNlcy5jaXNjby5jb20iLCJqdGkiOiI5NGM3Mzg4NS1iYzkxLTRmNDQtOTg0MC1kYzEwNzdmYTkyYmQiLCJwciI6InVzZXIiLCJyZWdpb25zIjpbIioiXSwicm9sZXMiOlsiVEVMLUFETUlOIl0sInNlc3Npb25JZCI6ImZjYjEyNjkwLWVhZjctNGY2OS04NzBkLWQyNDExYjE1MmE5ZiIsInRlbmFudElkIjoiZTg3YzE5NmMtYzU4Ni00MWFmLTljMjYtMmVhMWUxODExNjRlIiwidGVuYW50TmFtZSI6ImNpc2NvIiwidXNlcm5hbWUiOiJwb29uZ2FyZyJ9.jSkuMLr17Vwoi3XTQC6A0Mnu8dODXAf5JQQddmtgekSMxUcgs3GrqpYzGpjTNXdS2_0TRjcbxDe4sEppStt7jg"},"responseType":"TOKEN"}.
2025-03-17 09:19:16,964 INFO [openapi-http-pool9][[]] cpm.pxcloud.api.impl.DeviceRegistrationApiImpl -:::::- executeAutoRegister :: AutoRegister api executed successfully.
2025-03-17 09:19:16,964 INFO [openapi-http-pool9][[]] cpm.pxcloud.api.impl.DeviceRegistrationApiImpl -:::::- processSuccessVerification :: Received token to enroll the device.
2025-03-17 09:19:17,284 INFO [openapi-http-pool9][[]] cpm.pxcloud.service.ui.IseEnrollment -:::::- ISE enrollment response: 200 OK
2025-03-17 09:19:17,284 DEBUG [openapi-http-pool9][[]] cpm.pxcloud.service.ui.IseEnrollment -:::::- ISE enrollment, processing success response:
2025-03-17 09:19:17,306 DEBUG [openapi-http-pool9][[]] cisco.cpm.pxcloud.api.PxCloudPropertiesNotificationHandler -:::::- onCreate CLIENT_SECRET <redacted>
2025-03-17 09:19:17,325 DEBUG [openapi-http-pool9][[]] cisco.cpm.pxcloud.api.PxCloudPropertiesNotificationHandler -:::::- onCreate TOKEN_URL <redacted>
2025-03-17 09:19:17,342 DEBUG [openapi-http-pool9][[]] cisco.cpm.pxcloud.api.PxCloudPropertiesNotificationHandler -:::::- onCreate FQDN neoffers.cisco.com
2025-03-17 09:19:17,369 DEBUG [openapi-http-pool9][[]] cisco.cpm.pxcloud.api.PxCloudPropertiesNotificationHandler -:::::- onCreate ENROLLMENT_STATUS true
2025-03-17 09:19:17,394 DEBUG [openapi-http-pool9][[]] cisco.cpm.pxcloud.api.PxCloudPropertiesNotificationHandler -:::::- onCreate CLIENT_ID 1466211b-8cc1-4838-a76b-d11057eb0a4a
2025-03-17 09:19:17,418 DEBUG [openapi-http-pool9][[]] cisco.cpm.pxcloud.api.PxCloudPropertiesNotificationHandler -:::::- onCreate TENANT_NAME cisco
2025-03-17 09:19:17,438 DEBUG [openapi-http-pool9][[]] cisco.cpm.pxcloud.api.PxCloudPropertiesNotificationHandler -:::::- onCreate MEMBER_ID 67d7e91688c5fd08d0860039
2025-03-17 09:19:17,463 DEBUG [openapi-http-pool9][[]] cisco.cpm.pxcloud.api.PxCloudPropertiesNotificationHandler -:::::- onCreate USERNAME <redacted>
2025-03-17 09:19:17,485 DEBUG [openapi-http-pool9][[]] cisco.cpm.pxcloud.api.PxCloudPropertiesNotificationHandler -:::::- onCreate TENANT_ID e87c196c-c586-41af-9c26-2ea1e181164e
2025-03-17 09:19:17,489 INFO [openapi-http-pool9][[]] cpm.pxcloud.service.ui.IseEnrollment -:::::- ISE Device enrollment properties data is saved to DB
2025-03-17 09:19:17,495 INFO [openapi-http-pool9][[]] cpm.pxcloud.service.ui.IseEnrollment -:::::- ISE enrollment with cloud is successful and status is set to true
2025-03-17 09:19:17,500 INFO [openapi-http-pool9][[]] cpm.pxcloud.service.ui.IseEnrollment -:::::- Initiated the cloud configuration process
2025-03-17 09:19:17,500 INFO [openapi-http-pool9][[]] cpm.pxcloud.api.impl.DeviceRegistrationApiImpl -:::::- processSuccessVerification :: device is enrolled successfully.
2025-03-17 09:19:17,554 INFO [openapi-http-pool9][[]] cpm.iseopenapi.pxcloud.util.PxGridCloudUtil -:::::- updatePxGridCloud :: pxg added. preparing to retrieve pxg wallet cert
2025-03-17 09:19:17,554 DEBUG [openapi-http-pool9][[]] cpm.iseopenapi.pxcloud.util.PxGridCloudUtil -:::::- Preparing to save HostConfig [hostName=ise341-psn1, dispayName=ise341-psn1, hostId=07374a40-0301-11f0-873b-765072d6d75e, gateWay=10.106.39.1, masterStatus=NONE, nodeRoleStatus=SECONDARY, nodeTypes=PDP+PXG+PXCLOUD, nodeServiceType=SESSION,PROFILER, userName=null, smtpPort=null, smtpHost=null, hostAlias=ise341-psn1.poongarg.local, udiPid=ISE-VM-K9, udiVid=V01, udiSN=IHJDFEDEIKM, udiPT=VM, installType=null, vmInfo=28481208|12|LARGE||2025-03-14 17:45:14 UTC|0 MB|0 MHz|4294967295 MB|5000 MHz, isApiNode=false]
2025-03-17 09:19:18,501 INFO [pxcloud-configuration-1243][[]] cpm.pxcloud.service.ui.CloudConfigurationProcessor -:::::- Enrollment complete, starting cloud configuration process now
2025-03-17 09:19:18,505 DEBUG [pxcloud-configuration-1243][[]] cpm.pxcloud.service.ui.CloudConfigurationProcessor -:::::- Starting to process the cloud configuration for ISE

pxGrid node assume ACTIVE role, however, here we observed pxGridConnectionStatus as NOT_CONNECTED, which got fixed after adding the right pxGrid Certificate (with complete root CA chain) on this pxGrid node)

2025-03-17 09:20:12,301 TRACE [openapi-http-pool8][[]] cpm.iseopenapi.pxcloud.impl.PxGridApiDelegateImpl -:::::- Request to get device information.
2025-03-17 09:20:12,301 INFO [openapi-http-pool8][[]] cpm.pxcloud.service.ui.IseEnrollment -:::::- Fetching Device Info...
2025-03-17 09:20:12,310 INFO [Thread-150][[]] cpm.pxcloud.service.ui.IseEnrollment -:::::- Get pxCloud status information from ise341-psn1.poongarg.local
2025-03-17 09:20:12,311 INFO [Thread-150][[]] cpm.pxcloud.service.ui.IseEnrollment -:::::- pxCloud statusLookup URL: https://ise341-psn1.poongarg.local:8910/pxgrid/pxcloud/statusLookup
2025-03-17 09:20:12,427 INFO [Thread-150][[]] cpm.pxcloud.service.ui.IseEnrollment -:::::- Received response from ise341-psn1.poongarg.local: StatusResponse [role=ACTIVE, state=HERMES_ERROR, pxGridConnectionStatus=NOT_CONNECTED, cloudConnectionStatus=NOT_CONNECTED, reason=]

Now check Hermes.log on ACTIVE pxGrid node for pubsub events:

2025-03-17T09:37:43.906Z | INFO | hermes/config.go:332 | configMgr created successfully: configMgr[path=/opt/hermes/config.yaml]
2025-03-17T09:37:43.907Z | INFO | hermes/config.go:117 | Parsing configuration file: /opt/hermes/config.yaml
2025-03-17T09:37:43.907Z | INFO | hermes/config.go:338 | Config file /opt/hermes/config.yaml parsed successfully: &{PxGrid:{ClientName:~ise-hermes-ise341-psn1.poongarg.local Host:ise341-psn1.poongarg.local Port:8910 CertFile:/opt/xgrid/conf/ise-latest/host_H1742204190264.pem KeyFile:/opt/xgrid/conf/ise-latest/key_H1742204258381.pem CertPassword:aR8LRo9ZUmHnh5au3Gv7NX6VXn58SxehhxFtz2y2FL59PNfWWK9Lt/r+txTeQryR RootFile:/opt/xgrid/conf/ise-latest/ca_H1742204257774.pem} Log:{Level:info Files:[/opt/hermes/logs/hermes.log] Encoding:console} ERS:{Hostname:ise341-PAN.poongarg.local Port:-1 Enabled:false Username: CertFile:/opt/hermes/certs/adminCertFile.pem ConsumerKey: ConsumerSecret:} Proxy:{Host:bgl11-lab-wsa-1.cisco.com Port:80 Username: Password: BypassHosts:} Cloud:{CertFile:/opt/xgrid/conf/ise-latest/cacs_H1742198418638.pem} OpenAPI:{Hostname: Port:-1 Enabled:false Username: Password: CertFile:}}
2025-03-17T09:37:43.907Z | INFO | hermes/main.go:126 | Configuration loaded successfully
2025-03-17T09:37:43.908Z | INFO | trust/trust.go:28 | Custom trust bundle has been set/updated
2025-03-17T09:37:43.908Z | INFO | hermes/pxgrid.go:187 | Creating pxGrid WebSocket connection
2025-03-17T09:37:43.908Z | INFO | hermes/httpserver.go:57 | Starting REST server on :8913
2025-03-17T09:37:43.921Z | INFO | hermes/httpserver.go:78 | REST server is up and running
2025-03-17T09:37:43.983Z | INFO | pxgrid/websocket.go:93 | Got WS URL: wss://ise341-psn1.poongarg.local:8910/pxgrid/ise/pubsub
2025-03-17T09:37:44.066Z | INFO | pxgrid/websocket.go:107 | Connection to wss://ise341-psn1.poongarg.local:8910/pxgrid/ise/pubsub was created successfully
2025-03-17T09:37:44.066Z | INFO | hermes/connectionstatus.go:44 | Setting pxGrid connection status to CONNECTED

Root CA chain of Catalyst Cloud Portal is verified.

2025-03-17T09:37:44.731Z | INFO | hermes/pxgrid.go:267 | Cloud credentials are obtained from ISE
2025-03-17T09:37:45.034Z | INFO | hermes/pxgrid.go:376 | DeviceID: 67d7e91688c5fd08d0860039, TenantID: e87c196c-c586-41af-9c26-2ea1e181164e
2025-03-17T09:37:45.743Z | INFO | rest/ocsp.go:207 | Making OCSP request at http://commercial.ocsp.identrust.com with timeout of 1500000000
2025-03-17T09:37:45.743Z | INFO | rest/ocsp.go:207 | Making OCSP request at http://commercial.ocsp.identrust.com with timeout of 1500000000
2025-03-17T09:37:46.273Z | INFO | rest/ocsp.go:254 | OCSP Validation passed for CN=HydrantID Server CA O1,OU=HydrantID Trusted Certificate Service,O=IdenTrust,C=US
2025-03-17T09:37:46.279Z | INFO | rest/ocsp.go:254 | OCSP Validation passed for CN=dnaservices.cisco.com,O=Cisco Systems Inc.,L=San Jose,ST=California,C=US
2025-03-17T09:37:47.054Z | INFO | rest/ocsp.go:207 | Making OCSP request at http://commercial.ocsp.identrust.com with timeout of 9000000000
2025-03-17T09:37:47.432Z | INFO | hermes/config.go:262 | File /opt/hermes/config.yaml modified. Event: WRITE
2025-03-17T09:37:47.533Z | INFO | hermes/config.go:117 | Parsing configuration file: /opt/hermes/config.yaml
2025-03-17T09:37:47.533Z | INFO | hermes/config.go:305 | New configuration loaded
2025-03-17T09:37:47.533Z | INFO | hermes/config.go:314 | Restarting Hermes due to configuration change

Specific Topics subscription request got created once, FMC application is configured under Integration Catalog:

2025-03-17T12:54:09.975Z | INFO | pxgrid/subscriber.go:40 | Request to create new subscriber: service=com.cisco.ise.session , topicKey=groupTopic and topicFQN=/topic/com.cisco.ise.session.group
2025-03-17T12:54:09.975Z | INFO | pxgrid/subscriber.go:55 | Subscriber[service: com.cisco.ise.session, topic: groupTopic, id: cvc1msd3ct8r98jaf6dg] created successfully
2025-03-17T12:54:10.263Z | INFO | device-manager@v1.1.12/control.go:240 | Completed activate sync ID [session:userGroups--67d7e91688c5fd08d0860039]
2025-03-17T12:54:10.263Z | INFO | device-manager@v1.1.12/control.go:227 | Processing activate sync ID [profiler:profiles--67d7e91688c5fd08d0860039]
2025-03-17T12:54:10.263Z | INFO | hermes/pxgrid.go:117 | Request to add new pxGrid subscriber [com.cisco.ise.config.profiler:topic] for DxHub stream profiler:profiles
2025-03-17T12:54:10.263Z | INFO | pxgrid/subscriber.go:28 | Request to create new subscriber: com.cisco.ise.config.profiler:topic
2025-03-17T12:54:10.270Z | INFO | pxgrid/subscriber.go:40 | Request to create new subscriber: service=com.cisco.ise.config.profiler , topicKey=topic and topicFQN=/topic/com.cisco.ise.config.profiler
2025-03-17T12:54:10.270Z | INFO | pxgrid/subscriber.go:55 | Subscriber[service: com.cisco.ise.config.profiler, topic: topic, id: cvc1msl3ct8r98jaf6e0] created successfully
2025-03-17T12:54:10.559Z | INFO | device-manager@v1.1.12/control.go:240 | Completed activate sync ID [profiler:profiles--67d7e91688c5fd08d0860039]
2025-03-17T12:54:10.559Z | INFO | device-manager@v1.1.12/control.go:227 | Processing activate sync ID [trustsec:securityGroups--67d7e91688c5fd08d0860039]
2025-03-17T12:54:10.559Z | INFO | hermes/pxgrid.go:117 | Request to add new pxGrid subscriber [com.cisco.ise.config.trustsec:securityGroupTopic] for DxHub stream trustsec:securityGroups
2025-03-17T12:54:10.559Z | INFO | pxgrid/subscriber.go:28 | Request to create new subscriber: com.cisco.ise.config.trustsec:securityGroupTopic 
!
2025-03-17T16:17:30.050Z | INFO | api-proxy@v1.0.10/broker.go:114 | API-Proxy: Broker Agent start consuming 
2025-03-17T16:17:30.050Z | INFO | hermes/apiproxy.go:43 | API Proxy connection established
2025-03-17T16:17:30.050Z | INFO | hermes/connectionstatus.go:62 | Setting cloud connection status to CONNECTED
2025-03-17T16:17:30.057Z | INFO | hermes/dxhub.go:94 | Policies are obtained from ISE : &{Pxgrid:{ContextOutTopics:[com.cisco.ise.sxp:bindingTopic com.cisco.ise.config.profiler:topic com.cisco.ise.endpoint:topic com.cisco.ise.radius:failureTopic com.cisco.ise.trustsec:policyDownloadTopic com.cisco.ise.echo:echoTopic com.cisco.ise.mdm:endpointTopic com.cisco.ise.config.trustsec:securityGroupTopic com.cisco.ise.config.trustsec:securityGroupAclTopic com.cisco.ise.config.anc:statusTopic com.cisco.ise.config.upn:statusTopic com.cisco.ise.session:sessionTopic com.cisco.ise.session:groupTopic]}}

Limitations

1. User cannot enable pxGrid Cloud persona on more than 2 nodes.

2. De-registration from Catalyst Cloud Portal to cdFMC is supported but not vice-versa.

References

User Control with the pxGrid Cloud Identity Source

Cisco pxGrid Cloud

Cisco pxGrid Cloud Solution Guide