TrustSec Troubleshooting Tips on Cisco IE and Allen-Bradley Stratix IES and Cisco Catalyst 3850 Switches
The following section describes certain show commands that can be executed to view potential sources of problems related to Cisco TrustSec.
Note An IT engineer should have some expertise in TrustSec in order to troubleshoot any problems that are discovered. For complete information on Cisco TrustSec troubleshooting tips, refer to the following URL: https://community.cisco.com/t5/security-documents/trustsec-troubleshooting-guide/ta-p/3647576
IES is Unable to Register with Cisco ISE and Download the SGT Table Information
Verify whether the IES and Cisco ISE have the Right TrustSec Credentials Matched
This is the first step and it is possible that the IT security administrator might have a typo in the password or the ID information in the IES or the Cisco ISE. Refer to the following sections in Chapter 4, “Configuring the Infrastructure”:
The credentials may be missing on the IES. Issue the following command:
IE4K-25#show cts credentials
CTS password is defined in keystore, device-id = IE4K-25
Verify Whether the PAC Key Between the IES and the Cisco ISE is Configured Correctly
The PAC key must match between the Cisco ISE and the IES. If there is a mismatch in the IES, then re-configure the key, which will force a new PAC provisioning in the IES. Refer to Configuring RADIUS AAA in Chapter4, “Configuring the Infrastructure” To verify that the PAC is installed:
AID: BA6AAD6CB6C10E7045A4CCD0DA18E706
PAC-type = Cisco Trustsec
AID: BA6AAD6CB6C10E7045A4CCD0DA18E706
A-ID-Info: Identity Services Engine
Credential Lifetime: 12:45:25 EST Nov 10 2018
PAC-Opaque: 000200B00003000100040010BA6AAD6CB6C10E7045A4CCD0DA18E7060006009400030100AA913A603C53109269B2EACF49C2DED3000000135B68B9AB00093A804EB1C0FC8CF53471B62A122C4BB434A3BE2D7C13B59FA9D3BA8DF17CB7988B1E8BE7856DDC50C4F5CA6B20FE8E78270AB163FA73897FAFD7010325AEB3D8CD208D92A1B7BBD2C483D01CA4EE6B8FB9B7AFBF9CA8A5AE2274ECDE5BB9C457674376A48865BADF98C43B2CFC9FA8B8D3FD72FC538B
Refresh timer is set for 8w4d
To clear the credentials:
Verify that RADIUS is Operational from the IES
RADIUS: id 1, priority 1, host 10.13.48.184, auth-port 1812, acct-port 1813
State: current UP, duration 2488903s, previous duration 0s
Dead: total time 0s, count 5968
Authen: request 2275, timeouts 0, failover 0, retransmission 0
Response: accept 20, reject 2255, challenge 0
Response: unexpected 0, server error 0, incorrect 0, time 32ms
Transaction: success 2275, failure 0
Throttled: transaction 0, timeout 0, failure 0
Author: request 2, timeouts 0, failover 0, retransmission 0
Response: accept 2, reject 0, challenge 0
Response: unexpected 0, server error 0, incorrect 0, time 50ms
Transaction: success 2, failure 0
Throttled: transaction 0, timeout 0, failure 0
Account: request 38, timeouts 0, failover 0, retransmission 0
Request: start 18, interim 0, stop 18
Response: start 18, interim 0, stop 18
Response: unexpected 0, server error 0, incorrect 0, time 29ms
Transaction: success 38, failure 0
Throttled: transaction 0, timeout 0, failure 0
Elapsed time since counters last cleared: 4w19h26m
Estimated Outstanding Access Transactions: 0
Estimated Outstanding Accounting Transactions: 0
Estimated Throttled Access Transactions: 0
Estimated Throttled Accounting Transactions: 0
Maximum Throttled Transactions: access 0, accounting 0
Requests per minute past 24 hours:
high - 15 hours, 42 minutes ago: 2
low - 0 hours, 0 minutes ago: 0
Verify that the CTS server-list is Pointing to the Right Policy Server Node
The command to verify the cts server-list is shown below:
IE4K-25#show cts server-list
CTS Server Radius Load Balance = DISABLED
Server Group Deadtime = 20 secs (default)
Global Server Liveness Automated Test Deadtime = 20 secs
Global Server Liveness Automated Test Idle Time = 60 mins
Global Server Liveness Automated Test = DISABLED
Installed list: CTSServerList1-000B, 1 server(s):
*Server: 10.13.48.184, port 1812, A-ID 75FD68D130DA33A44480ED005C93FF49
auto-test = FALSE, keywrap-enable = FALSE, idle-time = 60 mins, deadtime = 20 secs
Verify that the IES has Downloaded the Right SGT Table Information
IE4K-25#show cts environment-data
Installed list: CTSServerList1-0001, 1 server(s):
*Server: 10.13.48.184, port 1812, A-ID BA6AAD6CB6C10E7045A4CCD0DA18E706
auto-test = FALSE, keywrap-enable = FALSE, idle-time = 60 mins, deadtime = 20 secs
Multicast Group SGT Table:
Security Group Name Table:
255-fd:Quarantined_Systems
Environment Data Lifetime = 86400 secs
Last update time = 10:18:52 EDT Sun Sep 9 2018
Env-data expires in 0:01:08:23 (dd:hr:mm:sec)
Env-data refreshes in 0:01:08:23 (dd:hr:mm:sec)
Cache data applied = NONE
IACS Asset is Unable to Authenticate to Cisco ISE
This section describes how to troubleshoot when an IACS device is unable to authenticate to Cisco ISE. To demonstrate the flow the IACS asset 10.17.10.65 is used to show the process.
Verify the Authentication and Authorization State of IACS Assets Attached to an IES
IE4K-34# show authentication brief
Interface MAC Address AuthC AuthZ Fg Uptime
-----------------------------------------------------------------------------
Gi1/14 0000.bc3f.d0ef m:OK AZ: SA- 409219s
Gi1/16 0000.bccd.f76a m:OK AZ: SA- 409221s
Gi1/11 0000.bc2d.20ef m:CF UZ: SA- FA- 409221s
Key to Authentication Attributes:
OK - Authentication Success
Key to Authorization Attributes:
AZ - Authorized, UZ - UnAuthorized
SA - Success Attributes, FA - Failed Attributes
D: - DACL, F: - Filterid / InACL, U: - URL ACL
V: - Vlan, I: - Inactivity Timer, O: - Open Dir
Key to Session Events Blocked Status Flags:
A - Applying Policy (multi-line status for details)
F - Final Removal in progress
I - Awaiting IIF ID allocation
N - Waiting for AAA to come up
R - Removing User Profile (multi-line status for details)
U - Applying User Profile (multi-line status for details)
Verify that NMT has Discovered the IACS Asset 10.17.10.65
The first step would be to verify if NMT has discovered the IACS device 10.17.10.65.
Figure 6-1 NMT Discovering IACS Asset 10.17.10.65
Verify that the pxGrid Service is Enabled at Cisco ISE
To verify that go to (ISE admin web) —> Administration —> Deployment and select the PSN (ise24 in this CPwE Network Security CVD):
Figure 6-2 Verifying that the pxGrid Service is Enabled at Cisco ISE
The next step is to verify if Cisco ISE has learned the IACS asset. Figure 6-3 shows that Cisco ISE has learned about the IACS asset.
Figure 6-3 Cisco ISE has Learned the IACS Asset 10.17.10.65
Verify that Profiling Policies are Configured Correctly
ISE profiles the IACS assets based on the profiling policy. If conditions in the profiling policy are not configured correctly, then ISE will not be able to profile the IACS asset. Refer to Profiling in Cisco ISE in Chapter 4, “Configuring the Infrastructure” for information on configuring the profiling policies
Verify that pxGrid probe is enabled at PSN
To verify that, go to (ISE admin web) —> Administration —> Deployment —> Select the psn (ise24 in this CVD) and select the tab profiling configuration.
Figure 6-4 Verifying that pxGrid Probe is Enabled on the PSN
Verify Live Logs at ISE to Understand the Authentication/Authorization Flow
To see live logs, go to (ISE admin web) —> Operation —> Live Logs to get a list of devices that went through the authentication/authorization process.
Figure 6-5 Live Logs at ISE
Selecting the Details option will provide details about the complete exchange.
Figure 6-6 Authentication and Authorization Results of an IACS Asset
3850 Distribution Switch is not Enforcing the Policy Correctly
Different reasons for this problem to happen exist; it can be troubleshooted by going through the following steps:
Verify that SGT is Assigned to the Port on the IES
IE4K-25#show cts role-based sgt-map all
Active IPv4-SGT Bindings Information
============================================
IP-SGT Active Bindings Summary
============================================
Total number of LOCAL bindings = 2
Total number of INTERNAL bindings = 5
Total number of active bindings = 7
Verify that SXP tunnel is up Between the Cisco ISE and the IES Device
IE4K-25#show cts sxp connections
Highest Version Supported: 4
Default Source IP: Not Set
Connection retry open period: 120 secs
Reconcile period: 120 secs
Retry open timer is not running
Peer-Sequence traverse limit for export: Not Set
Peer-Sequence traverse limit for import: Not Set
----------------------------------------------
Conn capability : IPv4-IPv6-Subnet
Conn hold time : 120 seconds
TCP conn password: default SXP password
Keepalive timer is running
Duration since last state change: 6:01:28:42 (dd:hr:mm:sec)
Total num of SXP Connections = 1
Verify that SXP Tunnel is Up at Cisco ISE to the IES (IE4K-25)
Navigate to (ISE admin web) —> Work Centers —> TrustSec —> SXP Devices and verify the SXP status.
Figure 6-7 Verifying the SXP Status of an IES Switch at ISE
Verify that Cisco ISE has Received the SGT-IP Mapping Information through the SXP Tunnel
Figure 6-8 Verifying the SXP Status of an IES Switch at ISE
Verify that 3850 is Receiving the SGT-IP Information through SXP Tunnel
P5-3850-stack-4#show cts sxp sgt-map brief
SXP Node ID(generated):0xC0A80A0B(192.168.10.11)
IP-SGT Mappings as follows:
IPv4,SGT: <10.13.15.25, 4:TrustSec_Device_SGT>
IPv4,SGT: <10.17.10.65, 5:LEVEL_1_CONTROLLER>
IPv4,SGT: <10.17.10.108, 6:LEVEL_0_IO>
IPv4,SGT: <10.17.10.217, 4:TrustSec_Device_SGT>
IPv4,SGT: <10.17.10.218, 4:TrustSec_Device_SGT>
IPv4,SGT: <10.17.10.219, 4:TrustSec_Device_SGT>
IPv4,SGT: <10.17.10.220, 4:TrustSec_Device_SGT>
IPv4,SGT: <10.17.20.217, 4:TrustSec_Device_SGT>
IPv4,SGT: <10.17.20.218, 4:TrustSec_Device_SGT>
IPv4,SGT: <10.17.20.219, 4:TrustSec_Device_SGT>
IPv4,SGT: <10.17.20.220, 4:TrustSec_Device_SGT>
IPv4,SGT: <10.20.10.5, 4:TrustSec_Device_SGT>
IPv4,SGT: <10.20.25.10, 11:LEVEL_1_GENERIC>
Verify that Policy Matrix is Downloaded to the 3850 Distribution Switch
P5-3850-stack-4#show cts role-based permissions
IPv4 Role-based permissions from group 5:LEVEL_1_CONTROLLER to group 5:LEVEL_1_CONTROLLER:
IPv4 Role-based permissions from group 6:LEVEL_0_IO to group 5:LEVEL_1_CONTROLLER:
IPv4 Role-based permissions from group 8:LEVEL_3 to group 5:LEVEL_1_CONTROLLER:
IPv4 Role-based permissions from group 9:Remote_Access to group 5:LEVEL_1_CONTROLLER:
IPv4 Role-based permissions from group 10:Remote_Desktop to group 5:LEVEL_1_CONTROLLER:
IPv4 Role-based permissions from group 5:LEVEL_1_CONTROLLER to group 6:LEVEL_0_IO:
IPv4 Role-based permissions from group 6:LEVEL_0_IO to group 6:LEVEL_0_IO:
IPv4 Role-based permissions from group 8:LEVEL_3 to group 6:LEVEL_0_IO:
IPv4 Role-based permissions from group 9:Remote_Access to group 6:LEVEL_0_IO:
IPv4 Role-based permissions from group 10:Remote_Desktop to group 6:LEVEL_0_IO:
RBACL Monitor All for Dynamic Policies : FALSE
RBACL Monitor All for Configured Policies : FALSE
Cisco ISE Troubleshooting Tips
The following section provides high level troubleshooting information to assist in identifying and resolving problems you may encounter when you use the Cisco Identity Services Engine (ISE).
Note For complete information on Cisco ISE monitoring and troubleshooting tips, refer to the following URL:
Checking the Status of pxGrid
On the PSN, execute the following command to check the status of the pxGrid:
ise24/admin# show application status ise | include pxGrid
pxGrid Infrastructure Service running 5736
pxGrid Publisher Subscriber Service running 5880
pxGrid Connection Manager running 5851
pxGrid Controller running 5902
Verify pxGrid Certificate in ISE PSN
From (ISE admin Web), navigate to Administration —> System —> Certificates —> System Certificates and expand on PSN (ise24 in this CPwE Network Security CVD) to verify that system certificate is used for pxGrid.
Figure 6-10 Verifying pxGrid Certificate in ISE PSN
Verify the NMT pxGrid Status in ISE
From (ISE admin Web), navigate to Administration —> pxGrid Services and verify that NMT is registered as client.
Figure 6-11 Verifying the NMT pxGrid Status in ISE
Enable DEBUG on Profiler and pxGrid
In certain situations, it may be desired to enable debug on ISE and verify the exchange of information between the NMT and ISE via pxGrid. This section describes how to enable the debug.
Step 1 From (ISE admin web), navigate to Administration —> System —> Logging —> Debug Log Configuration.
Step 2 Select the PSN on the right panel (ise24 in this CPwE Network Security CVD DIG).
Step 3 Select profiler and change the logging levels to DEBUG and click Save.
Step 4 Select pxgrid and change the logging levels to TRACE and click Save.
To verify the log information, navigate to PSN (ise24) and issue the following command:
ise24/admin# show logging application profiler.log | include IND
2018-10-23 12:27:22,421 DEBUG [ProfilerINDSubscriberPoller-84-thread-1][] cisco.profiler.infrastructure.probemgr.INDSubscriber -::- Looking for new publishers...
2018-10-23 12:27:22,439 DEBUG [ProfilerINDSubscriberPoller-84-thread-1][] cisco.profiler.infrastructure.probemgr.INDSubscriber -::- Existing services are: [Service [name=com.cisco.endpoint.asset, nodeName=ind-win1
0, properties={wsPubsubService=com.cisco.ise.pubsub, restBaseUrl=https://ind-win10:8910/pxgrid/ind/asset/, assetTopic=/topic/com.cisco.endpoint.asset}]]
2018-10-23 12:27:22,439 INFO [ProfilerINDSubscriberPoller-84-thread-1][] cisco.profiler.infrastructure.probemgr.INDSubscriber -::- New services are: []
2018-10-23 12:27:22,451 INFO [ProfilerINDSubscriberPoller-84-thread-1][] cisco.profiler.infrastructure.probemgr.INDSubscriber -::- NODENAME:ind-win10
2018-10-23 12:27:22,451 INFO [ProfilerINDSubscriberPoller-84-thread-1][] cisco.profiler.infrastructure.probemgr.INDSubscriber -::- REQUEST BODY{"offset":"0","limit":"500"}
2018-10-23 12:27:22,519 INFO [ProfilerINDSubscriberPoller-84-thread-1][] cisco.profiler.infrastructure.probemgr.INDSubscriber -::- Response status={}200
2018-10-23 12:27:22,520 INFO [ProfilerINDSubscriberPoller-84-thread-1][] cisco.profiler.infrastructure.probemgr.INDSubscriber -::- Content: "OUT_OF_SYNC"
2018-10-23 12:27:22,520 INFO [ProfilerINDSubscriberPoller-84-thread-1][] cisco.profiler.infrastructure.probemgr.INDSubscriber -::- Status is :"OUT_OF_SYNC"
2018-10-23 12:27:22,535 INFO [ProfilerINDSubscriberPoller-84-thread-1][] cisco.profiler.infrastructure.probemgr.INDSubscriber -::- NODENAME:ind-win10
2018-10-23 12:27:22,535 INFO [ProfilerINDSubscriberPoller-84-thread-1][] cisco.profiler.infrastructure.probemgr.INDSubscriber -::- REQUEST BODY{"offset":"0","limit":"500"}
2018-10-23 12:27:22,602 INFO [ProfilerINDSubscriberPoller-84-thread-1][] cisco.profiler.infrastructure.probemgr.INDSubscriber -::- Response status={}200
2018-10-23 12:27:22,602 INFO [ProfilerINDSubscriberPoller-84-thread-1][] cisco.profiler.infrastructure.probemgr.INDSubscriber -::- Content: "OUT_OF_SYNC"
2018-10-23 12:27:22,602 INFO [ProfilerINDSubscriberPoller-84-thread-1][] cisco.profiler.infrastructure.probemgr.INDSubscriber -::- Status is :"OUT_OF_SYNC"
2018-10-23 12:27:22,603 DEBUG [ProfilerINDSubscriberPoller-84-thread-1][] cisco.profiler.infrastructure.probemgr.INDSubscriber -::- Static set after adding new services: [Service [name=com.cisco.endpoint.asset, no
deName=ind-win10, properties={wsPubsubService=com.cisco.ise.pubsub, restBaseUrl=https://ind-win10:8910/pxgrid/ind/asset/, assetTopic=/topic/com.cisco.endpoint.asset}]]
2018-10-23 12:27:22,612 INFO [ProfilerINDSubscriberBulkRequestPool-533-thread-1][] cisco.profiler.infrastructure.probemgr.INDSubscriber -::- NODENAME:ind-win10
2018-10-23 12:27:22,612 INFO [ProfilerINDSubscriberBulkRequestPool-533-thread-1][] cisco.profiler.infrastructure.probemgr.INDSubscriber -::- REQUEST BODY{"offset":"0","limit":"500"}
2018-10-23 12:27:24,451 INFO [ProfilerINDSubscriberBulkRequestPool-533-thread-1][] cisco.profiler.infrastructure.probemgr.INDSubscriber -::- Response status={}200
2018-10-23 12:27:24,468 INFO [ProfilerINDSubscriberBulkRequestPool-533-thread-1][] cisco.profiler.infrastructure.probemgr.INDSubscriber -::- Content: {"assets":[{"assetId":"50135","assetName":"192.168.4.31","asse
tIpAddress":"192.168.4.31","assetMacAddress":"","assetVendor":"Unknown","assetProductId":"Unknown","assetSerialNumber":"","assetDeviceType":"Unknown","assetSwRevision":"","assetHwRevision":"","assetProtocol":"PROFI
NET","assetConnectedLinks":[{"assetId":"30189","assetName":"IE2K-21","assetIpAddress":"192.168.2.21","assetPortName":"FastEthernet1/1","assetDeviceType":"Switch"},{"assetId":"30158","assetName":"IE1K-2","assetIpA