Monitoring and Troubleshooting

Monitoring and Troubleshooting Service in Cisco ISE

The Monitoring and Troubleshooting (MnT) service is a comprehensive identity solution for all Cisco ISE run-time services and uses the following components:

Monitoring: Provides real-time presentation of meaningful data representing the state of access activities on a network. This insight allows you to easily interpret and affect operational conditions.
Troubleshooting: Provides contextual guidance for resolving access issues on networks. You can then address user concerns and provide resolution in a timely manner.
Reporting: Provides a catalog of standard reports that you can use to analyze trends and monitor system performance and network activities. You can customize reports in various ways and save them for future use. You can search records using wild cards and multiple values in all the reports for the Identity, Endpoint ID, and ISE Node (except the Health Summary report) fields.

Cisco ISE Dashboard

The Cisco ISE dashboard or home page (Home > Summary) is the landing page that appears after you log in to the Cisco ISE administration console. The dashboard is a centralized management console consisting of metric meters along the top of the window, with dashlets below. The default dashboards are Summary, Endpoints, Guests, Vulnerability, and Threat.

The dashboard’s real-time data provides an at-a-glance status of the devices and users accessing your network as well as an overview of the system's health.

Note

You must have Adobe Flash Player installed to be able to view the dashlets and all the corresponding drill-down windows.

Note

You cannot rename or delete a default dashboard.

Detach: To view a dashlet in a separate window.
Refresh: To refresh a dashlet.
Remove: To remove a dashlet from the dashboard.

Network Privilege Framework

The dashboard shows the activity on the Network Privilege Framework (NPF), and provides detailed information on the various components.

The NPF is composed of the three tiers outlined in the following table:

Table 1. NPF Tiers
Tier	Specifications
1	Access control based on identity using 802.1x, MAC authentication bypass (MAB), the Cisco ISE Profiler service
2	Access control based on identity using 802.1x, MAB, Profiler, guest provisioning of the Network Admission Control (NAC) manager, central web authentication
3	Access control based on identity and posture using 802.1x, MAB, Profiler, guest provisioning of the NAC manager, central web authentication

NPF authentication and authorization generates a flow of events. The events from the different sources are then collected by Cisco ISE monitoring and troubleshooting tools and summarized. You can view the authentication and authorization results on the dashboard or choose to run any number of reports.

Network Privilege Framework Event Flow Process

The Network Privilege Framework (NPF) authentication and authorization event flow uses the process described in the following table:


Process Stage	Description
1	Network Access Device (NAD) performs either a normal authorization or a flex authorization.
2	An unknown agentless identity is profiled with web authorization.
3	A RADIUS server authenticates and authorizes the identity.
4	Authorization is provisioned for the identity at the port.
5	Unauthorized endpoint traffic is dropped.

User Roles and Permissions for Monitoring and Troubleshooting Capabilities

Monitoring and troubleshooting capabilities are associated with default user roles. The tasks you are allowed to perform are directly related to your assigned user role.

See Cisco ISE Administrator Groups for information on the permissions and restrictions set for each user role.

Note

Accessing Cisco ISE using the root shell without Cisco TAC supervision is not supported, and Cisco is not responsible for any service disruption that might be caused as a result.

Data Stored in the Monitoring Database

The Cisco ISE monitoring service collects and stores data in a specialized monitoring database. The rate and amount of data utilized to monitor network functions may require a node dedicated solely to monitoring. If your Cisco ISE network collects logging data at a high rate from policy service nodes or network devices, we recommend a Cisco ISE node dedicated to monitoring.

To manage the information stored in the monitoring database, perform full and incremental backups of the database. This includes purging unwanted data and then restoring the database.

Device Configuration for Monitoring

The monitoring node receives and uses data from the devices on a network to populate the dashboard display. To enable communication between the monitoring node and the network devices, the switches and NADs must be configured properly.

Network Process Status

You can view process status for the network from the Cisco ISE dashboard using the System Summary dashlet. For example, when processes like the application server or database fail, an alarm is generated and you can view the results using the System Summary dashlet.

The color of the system status icon indicates the health of your system:

Green = Healthy
Yellow = Warning
Red = Critical
Gray = No information

Monitor Network Process Status

Procedure

Step 1

Go to the Cisco ISE Dashboard.

Step 2

Expand the System Summary dashlet. A detailed real-time report appears.

Step 3

Review the following information for the processes that are running on the network:

Name of the process
CPU and memory utilization
Time since process started running

Network Authentications

You can view the passed and failed network authentications from the Authentications dashlet. It provides data on the user or type of device, location, and the identity group to which the user or device belongs. The sparklines along the top of the dashlet represent distribution over the last 24 hours and the last 60 minutes.

Monitor Network Authentications

Procedure

Step 1	Go to the Cisco ISE Dashboard.
Step 2	Expand the Authentications dashlet. A detailed real-time report appears.
Step 3	Review the information for the users or devices that are authenticated on the network.
Step 4	Expand the data categories for more information.

Profiler Activity and Profiled Endpoints

The Profiled Endpoint dashlet focuses on the endpoints on the network that have matched profiles, providing profile data for each endpoint. For example, the statistics allow you to determine the type of device, its location, and its IP address. The sparklines along the top of the dashlet represent endpoint activity over the last 24 hours and last 60 minutes.

The Profiled Endpoint dashlet represents the total number of endpoints that have been profiled on the network for the last 24 hours, including those that are unknown. It is not a representation of how many endpoints are currently active on the network. Sparkline metrics at the top of the dashlet show time specific values for the last 24 hours and 60 minutes.

Determine Profiler Activity and Profiled Endpoints

Procedure

Step 1	Go to the Cisco ISE Dashboard.
Step 2	In the Profiler Activity dashlet, hover your cursor over a stack bar or sparkline. A tooltip provides detailed information.
Step 3	Expand the data categories for more information.
Step 4	Expand the Profiler Activity dashlet. A detailed real-time report appears.

Troubleshooting the Profiler Feed

The Test Feed Service Connection button on Administration > Feed Service > Profiler verifies that there is a connection to the Cisco feed server, and that the certificate is valid.

If the test is able to connect to the Cisco feed server, a pop-up window is displayed with a message stating that the test connection is successful.

If the connection fails, the area adjacent to the Test Feed Service Connection button displays a response from the server, which is similar to the following example in which the important part of the message is in bold:

Test result: Failure: FeedService test connection failed : Feed Service unavailable : SocketTimeoutException invoking https://ise.cisco.com:8443/feedserver/feed/serverinfo: sun.security.validator.ValidatorException:PKIX path building failed: Sun.security.provider.certpath.SunCertPathBuilderException Unable to find valid certification path to requested target

Here are some possible error messages, and the corresponding actions to be taken:

Unable to find valid certification path to requested target: This message indicates that the certificate that the feed server used is not valid. Verify that you have enabled the Verisign certificates.
No route to host: Verify that you have a working connection to an outside network from the Cisco ISE server.
UnknownHostException (at the beginning of an error message): Verify that you have a working connection to an outside network from the Cisco ISE server.

Posture Compliance

The Posture Compliance dashlet provides information about the users who are accessing the network, and whether they are posture compliant. The data that is displayed is for devices that are currently connected to the network. The stack bars show noncompliance statistics, and are arranged according to the operating system and other criteria. Spark lines represent the percentage of compliant versus noncompliant posture attempts.

Check Posture Compliance

Procedure

Step 1	Log in to Cisco ISE and access the dashboard.
Step 2	In the Posture Compliance dashlet, hover your cursor over a stack bar or sparkline. A tooltip provides detailed information.
Step 3	Expand the data categories for more information.
Step 4	Expand the Posture Compliance dashlet. A detailed real-time report appearsis displayed.

Cisco ISE Alarms

Alarms notify you of critical conditions on a network and are displayed in the Alarms dashlet. They also provide information on system activities, such as data purge events. You can configure how you want to be notified about system activities, or disable them entirely. You can also configure the threshold for certain alarms.

Most alarms do not have an associated schedule and are sent immediately after an event occurs. At any given point in time, only the latest 15,000 alarms are retained.

If the event re-occurs, then the same alarms are suppressed for a minimum duration of two hours. During the time that the event re-occurs, depending up on the trigger, it may take up to three hours for the alarms to re-appear.

The following table lists all the Cisco ISE alarms, descriptions and their resolution.

Table 2. Cisco ISE Alarms
Alarm Name	Alarm Description	Alarm Resolution
Administrative and Operational Audit Management
Deployment Upgrade Failure	An upgrade has failed on an ISE node.	Check the ADE.log on the failed node for upgrade failure reason and corrective actions.
Upgrade Bundle Download failure	An upgrade bundle download has failed on an ISE node.	Check the ADE.log on the failed node for upgrade failure reason and corrective actions.
SXP Connection Failure	SXP Connection has failed.	Verify that the SXP service is running. Check peer for compatibility.
Cisco profile applied to all devices	Network device profiles define the capabilities of network access devices, such as MAB, Dot1X, CoA, Web Redirect. As part of the ISE 2.0 upgrade, the default Cisco network device profile was applied to all network devices.	Consider editing the configuration of non-Cisco network devices to assign the appropriate profile.
Secure LDAP connection reconnect due to CRL found revoked certificate	CRL check result is that the certificate used for LDAP connection is revoked.	Check the CRL configuration and verify that it is valid. Check that the LDAP server certificate and its issuer certificates are not revoked. If revoked issue new certificate and install it on LDAP server.
Secure LDAP connection reconnect due to OCSP found revoked certificate	OCSP check result is that the certificate used for LDAP connection is revoked.	Check the OCSP configuration and verify that it is valid. Check that the LDAP server certificate and its issuer certificates are not revoked. If revoked issue new certificate and install it on LDAP server.
Secure syslog connection reconnect due to CRL found revoked certificate	CRL check result is that the certificate used for syslog connection is revoked.	Check the CRL configuration and verify that it is valid. Check that the syslog server certificate and its issuer certificates are not revoked. If revoked issue new certificate and install it on syslog server.
Secure syslog connection reconnect due to OCSP found revoked certificate	OCSP check result is that the certificate used for syslog connection is revoked.	Check the OCSP configuration and verify that it is valid. Check that the syslog server certificate and its issuer certificates are not revoked. If revoked issue new certificate and install it on syslog server.
Administrator account Locked/Disabled	Administrator account is locked or disabled due to password expiration or incorrect login attempts. For more details, refer to the administrator password policy.	Administrator password can be reset by another administrator using the GUI or CLI.
ERS identified deprecated URL	ERS identified deprecated URL	The request URL is deprecated and it is recommended to avoid using it.
ERS identified out-dated URL	ERS identified out-dated URL	The requested URL is outdated and it is recommended to use a newer one. This URL will not be removed in future releases.
ERS request content-type header is outdated	ERS request content-type header is out-dated.	The request resource version stated in the request content-type header is outdated. That means that the resource schema has been modified. One or more attributes may have been added or removed. To overcome that with the outdated schema, the ERS Engine will use default values.
ERS XML input is a suspect for XSS or Injection attack	ERS XML input is a suspect for XSS or Injection attack.	Please review your xml input.
Backup Failed	The ISE backup operation failed.	Check the network connectivity between Cisco ISE and the repository. Ensure that: The credentials used for the repository is correct. There is sufficient disk space in the repository. The repository user has write privileges.
CA Server is down	CA server is down.	Check to make sure that the CA services are up and running on the CA server.
CA Server is Up	CA server is up.	A notification to inform the administrator that the CA server is up.
Certificate Expiration	This certificate will expire soon. When it expires, Cisco ISE may fail to establish secure communication with clients.	Replace the certificate. For a trust certificate, contact the issuing Certificate Authority (CA). For a CA-signed local certificate, generate a CSR and have the CA create a new certificate. For a self-signed local certificate, use Cisco ISE to extend the expiration date. You can delete the certificate if it is no longer used.
Certificate Revoked	Administrator has revoked the certificate issued to an Endpoint by the Internal CA.	Go through the BYOD flow from the beginning to be provisioned with a new certificate.
Certificate Provisioning Initialization Error	Certificate provisioning initialization failed	More than one certificate found with the same value of CN (CommonName) attribute in the subject, cannot build certificate chain. Check all the certificates in the system including those from the SCEP server.
Certificate Replication Failed	Certificate replication to secondary node failed	The certificate is not valid on the secondary node, or there is some other permanent error condition. Check the secondary node for a pre-existing, conflicting certificate. If found, delete the pre-existing certificate on the secondary node, and export the new certificate on the primary, delete it, and import it in order to reattempt replication.
Certificate Replication Temporarily Failed	Certificate replication to secondary node temporarily failed	The certificate was not replicated to a secondary node due to a temporary condition such as a network outage. The replication will be retried until it succeeds.
Certificate Expired	This certificate has expired. Cisco ISE may fail to establish secure communication with clients. Node-to-node communication may also be affected.	Replace the certificate. For a trust certificate, contact the issuing Certificate Authority (CA). For a CA-signed local certificate, generate a CSR and have the CA create a new certificate. For a self-signed local certificate, use Cisco ISE to extend the expiration date. You can delete the certificate if it is no longer used.
Certificate Request Forwarding Failed	Certificate request forwarding failed.	Make sure that the certification request coming in matches with attributes from the sender.
Configuration Changed	Cisco ISE configuration is updated. This alarm is not triggered for any configuration change in users and endpoints.	Check if the configuration change is expected.
CRL Retrieval Failed	Unable to retrieve CRL from the server. This could occur if the specified CRL is unavailable.	Ensure that the download URL is correct and is available for the service.
DNS Resolution Failure	DNS resolution failed on the node.	Check if the DNS server configured by the command ip name-server is reachable. If you get the alarm as 'DNS Resolution failed for CNAME <hostname of the node>', then ensure that you create CNAME RR along with the A record for each Cisco ISE node.
Firmware Update Required	A firmware update is required on this host.	Contact Cisco Technical Assistance Center to obtain firmware update
Insufficient Virtual Machine Resources	Virtual Machine (VM) resources such as CPU, RAM, Disk Space, or IOPS are insufficient on this host.	Ensure that a minimum requirements for the VM host, as specified in the Cisco ISE Hardware Installation Guide.
NTP Service Failure	The NTP service is down on this node.	This could be because there is a large time difference between NTP server and Cisco ISE node( more than 1000s). Ensure that your NTP server is working properly and use the ntp server <servername> CLI command to restart the NTP service and fix the time gap.
NTP Sync Failure	All the NTP servers configured on this node are unreachable.	Execute show ntp command from the CLI for troubleshooting. Ensure that the NTP servers are reachable from Cisco ISE. If NTP authentication is configured, ensure that the key ID and value matches with that of the server.
No Configuration Backup Scheduled	No Cisco ISE configuration backup is scheduled.	Create a schedule for configuration backup.
Operations DB Purge Failed	Unable to purge older data from the operations database. This could occur if M&T nodes are busy.	Check the Data Purging Audit report and ensure that the used_space is lesser than the threshold_space. Login to M&T nodes using CLI and perform the purge operation manually.
Profiler SNMP Request Failure	Either the SNMP request timed out or the SNMP community or user authentication data is incorrect.	Ensure that SNMP is running on the NAD and verify that SNMP configuration on Cisco ISE matches with NAD.
Replication Failed	The secondary node failed to consume the replicated message.	Login to the Cisco ISE GUI and perform a manual syncup from the deployment page. De-register and register back the affected Cisco ISE node.
Restore Failed	Cisco ISE restore operation failed.	Ensure the network connectivity between Cisco ISE and the repository. Ensure that the credentials used for the repository is correct. Ensure that the backup file is not corrupted. Execute the reset-config command from the CLI and restore the last known good backup.
Patch Failure	A patch process has failed on the server.	Re-install the patch process on the server.
Patch Success	A patch process has succeeded on the server.	-
External MDM Server API Version Mismatch	External MDM server API version does not match with what is configured in Cisco ISE.	Ensure that the MDM server API version is the same as what is configured in Cisco ISE. Update Cisco ISE MDM server configuration if needed.
External MDM Server Connection Failure	Connection to the external MDM server failed.	Ensure that the MDM server is up and Cisco ISE-MDM API service is running on the MDM server.
External MDM Server Response Error	External MDM Server response error.	Ensure that the Cisco ISE-MDM API service is properly running on the MDM server.
Replication Stopped	ISE node could not replicate configuration data from the PAN.	Login to the Cisco ISE GUI to perform a manual syncup from the deployment page or de-register and register back the affected ISE node with required field.
Endpoint certificates expired	Endpoint certificates were marked expired by daily scheduled job.	Please re-enroll the endpoint device to get a new endpoint certificate.
Endpoint certificates purged	Expired endpoint certificates were purged by daily scheduled job.	No action needed - this was an administrator-initiated cleanup operation.
Endpoints Purge Activities	Purge activities on endpoints for the past 24 hours. This alarm is triggered at mid-night.	Review the purge activities under Operations > Reports > Endpoints and Users > Endpoint Purge Activities
Slow Replication Error	Slow or a stuck replication is detected .	Please verify that the node is reachable and part of the deployment.
Slow Replication Info	Slow or a stuck replication is detected .	Please verify that the node is reachable and part of the deployment.
Slow Replication Warning	Slow or a stuck replication is detected .	Please verify that the node is reachable and part of the deployment.
PAN Auto Failover - Failover Failed	Promotion request to the Secondary administration node failed.	Please refer to the alarm details for further action.
PAN Auto Failover - Failover Triggered	Successfully triggered the failover of the Secondary Administration node to Primary role.	Wait for promotion of secondary PAN to complete and bring up the old primary PAN.
PAN Auto Failover - Health Check Inactivity	PAN did not receive the health check monitoring request from the designated monitoring node.	Please verify if the reported monitoring node is down or out-of-sync and trigger a manual sync if needed.
PAN Auto Failover - Invalid Health Check	Invalid health check monitoring request received for auto-failover.	Please verify if the health check monitoring node is out-of-sync and trigger a manual sync if needed.
PAN Auto Failover - Primary Administration Node Down	Primary Admin node is down or is not reachable from the monitoring node.	Bring up the PAN or wait for failover to happen.
PAN Auto Failover - Rejected Failover Attempt	Secondary administration node rejected the promotion request made by the health check monitor node.	Please refer to the alarm details for further action.
ISE Services
Excessive TACACS Authentication Attempts	The ISE Policy Service nodes are experiencing higher than expected rate of TACACS Authentications.	Check the re-auth timer in the network devices. Check the network connectivity of the ISE infrastructure.
Excessive TACACS Authentication Failed Attempts	The ISE Policy Service nodes are experiencing higher than expected rate of Failed TACACS Authentications.	Check the authentication steps to identify the root cause. Check the ISE/NAD configuration for Identity and Secret mismatch.
MSE Location Server accessible again	MSE Location Server is accessible again.	None.
MSE Location Server not accessible.	MSE Location Server is not accessible or is down.	Please check if MSE Location Server is up and running and is accessible from ISE node(s).
AD Connector had to be restarted	AD Connector stopped unexpectedly and had to be restarted.	If this issue persists, contact the Cisco TAC for assistance.
Active Directory forest is unavailable	Active Directory forest GC (Global Catalog) is unavailable, and cannot be used for authentication, authorization and group and attribute retrieval.	Check DNS configuration, Kerberos configuration, error conditions, and network connectivity.
Authentication domain is unavailable	Authentication domain is unavailable, and cannot be used for authentication, authorization and group and attribute retrieval.	Check DNS configuration, Kerberos configuration, error conditions, and network connectivity.
ISE Authentication Inactivity	Cisco ISE policy service nodes are not receiving authentication requests from the network devices.	Check the ISE/NAD configuration. Check the network connectivity of the ISE/NAD infrastructure.
ID Map. Authentication Inactivity	No User Authentication events were collected by the Identity Mapping service in the last 15 minutes.	If this is a time when User Authentications are expected (e.g. work hours), then check the connection to Active Directory domain controllers.
COA Failed	Network device has denied the Change of Authorization (CoA) request issued by Cisco ISE policy service nodes.	Ensure that the network device is configured to accept Change of Authorization (CoA) from Cisco ISE. Ensure if CoA is issued on a valid session.
Configured nameserver is down	Configured nameserver is down or unavailable.	Check DNS configuration and network connectivity.
Supplicant Stopped Responding	Cisco ISE sent last message to the client 120 seconds ago but there is no response from the client.	Verify that the supplicant is configured properly to conduct a full EAP conversation with Cisco ISE. Verify that NAS is configured properly to transfer EAP messages to/from the supplicant. Verify that the supplicant or NAS does not have a short timeout for EAP conversation.
Excessive Authentication Attempts	Cisco ISE policy service nodes are experiencing higher than expected rate of authentications.	Check the re-auth timer in the network devices. Check the network connectivity of the Cisco ISE infrastructure. Once the threshold is met, the Excessive Authentication Attempts and Excessive Failed Attempts alarms are triggered. The numbers displayed next to the Description column are the total number of authentications that are authenticated or failed against Cisco ISE in last 15 minutes.
Excessive Failed Attempts	Cisco ISE policy service nodes are experiencing higher than expected rate of failed authentications.	Check the authentication steps to identify the root cause. Check the Cisco ISE/NAD configuration for identity and secret mismatch. Once the threshold is met, the Excessive Authentication Attempts and Excessive Failed Attempts alarms are triggered. The numbers displayed next to the Description column are the total number of authentications that are authenticated or failed against Cisco ISE in last 15 minutes.
AD: Machine TGT refresh failed	ISE server TGT (Ticket Granting Ticket) refresh has failed; it is used for AD connectivity and services.	Check that the ISE machine account exists and is valid. Also check for possible clock skew, replication, Kerberos configuration and/or network errors.
AD: ISE account password update failed	ISE server has failed to update it's AD machine account password.	Check that the ISE machine account password is not changed and that the machine account is not disabled or restricted. Check the connectivity to KDC.
Joined domain is unavailable	Joined domain is unavailable, and cannot be used for authentication, authorization and group and attribute retrieval.	Check DNS configuration, Kerberos configuration, error conditions, and network connectivity.
Identity Store Unavailable	Cisco ISE policy service nodes are unable to reach the configured identity stores.	Check the network connectivity between Cisco ISE and identity store.
Misconfigured Network Device Detected	Cisco ISE has detected too many RADIUS accounting information from NAS	Too many duplicate RADIUS accounting information has been sent to ISE from NAS. Configure NAS with accurate accounting frequency.
Misconfigured Supplicant Detected	Cisco ISE has detected mis-configured supplicant on the network	Ensure that the configuration on Supplicant is correct.
No Accounting Start	Cisco ISE policy service nodes have authorized a session but did not receive accounting start from the network device.	Ensure that RADIUS accounting is configured on the network device. Check the network device configuration for local authorization.
Unknown NAD	Cisco ISE policy service nodes are receiving authentication requests from a network device that is not configured in Cisco ISE.	Check if the network device is a genuine request and add it to the configuration. Ensure that the secret matches.
SGACL Drops	Secure Group Access (SGACL) drops occurred. This occurs if a Trustsec capable device drops packets due to SGACL policy violations.	Run the RBACL drop summary report and review the source causing the SGACL drops. Issue a CoA to the offending source to reauthorize or disconnect the session.
RADIUS Request Dropped	The authentication/accounting request from a NAD is silently discarded. This may occur due to unknown NAD, mismatched shared secrets, or invalid packet content per RFC.	Check that the NAD/AAA client has a valid configuration in Cisco ISE. Check whether the shared secrets on the NAD/AAA client and Cisco ISE matches. Ensure that the AAA client and the network device, have no hardware problems or problems with RADIUS compatibility. Also ensure that the network that connects the device to Cisco ISE has no hardware problems.
EAP Session Allocation Failed	A RADIUS request was dropped due to reaching EAP sessions limit. This condition can be caused by too many parallel EAP authentication requests.	Wait for a few seconds before invoking another RADIUS request with new EAP session. If system overload continues to occur, try restarting the ISE Server.
RADIUS Context Allocation Failed	A RADIUS request was dropped due to system overload. This condition can be caused by too many parallel authentication requests.	Wait for a few seconds before invoking a new RADIUS request. If system overload continues to occur, try restarting the ISE Server.
System Health
High Disk I/O Utilization	Cisco ISE system is experiencing high disk I/O utilization.	Check if the system has sufficient resources. Check the actual amount of work on the system for example, number of authentications, profiler activity etc. Add additional server to distribute the load.
High Disk Space Utilization	Cisco ISE system is experiencing high disk space utilization.	Check if the system has sufficient resources. Check the actual amount of work on the system for example, number of authentications, profiler activity etc. Add additional server to distribute the load.
High Load Average	Cisco ISE system is experiencing high load average.	Check if the system has sufficient resources. Check the actual amount of work on the system for example, number of authentications, profiler activity etc. Add additional server to distribute the load.
High Memory Utilization	Cisco ISE system is experiencing high memory utilization.	Check if the system has sufficient resources. Check the actual amount of work on the system for example, number of authentications, profiler activity etc. Add additional server to distribute the load.
High Operations DB Usage	Cisco ISE monitoring nodes are experiencing higher volume of syslog data than expected.	Check and reduce the purge configuration window for the operations data.
High Authentication Latency	Cisco ISE system is experiencing high authentication latency.	Check if the system has sufficient resources. Check the actual amount of work on the system for example, number of authentications, profiler activity etc. Add additional server to distribute the load.
Health Status Unavailable	The monitoring node has not received health status from the Cisco ISE node.	Ensure that Cisco ISE nodes are up and running. Ensure that Cisco ISE nodes are able to communicate with the monitoring nodes.
Process Down	One of the Cisco ISE processes is not running.	Restart the Cisco ISE application.
Profiler Queue Size Limit Reached	The ISE Profiler queue size limit has been reached. Events received after reaching the queue size limit will be dropped.	Check if the system has sufficient resources, and ensure EndPoint attribute filter is enabled.
OCSP Transaction Threshold Reached	The OCSP transaction threshold has been reached. This alarm is triggered when internal OCSP service reach high volume traffic.	Please check if the system has sufficient resources.
Licensing
License About to Expire	License installed on the Cisco ISE nodes are about to expire.	View the Licencing page in Cisco ISE to view the license usage.
License Expired	License installed on the Cisco ISE nodes has expired.	Contact Cisco Accounts team to purchase new licenses.
License Violation	Cisco ISE nodes have detected that you are exceeding or about to exceed the allowed license count.	Contact Cisco Accounts team to purchase additional licenses.
System Error
Log Collection Error	Cisco ISE monitoring collector process is unable to persist the audit logs generated from the policy service nodes.	This will not impact the actual functionality of the Policy Service nodes. Contact TAC for further resolution.
Scheduled Report Export Failure	Unable to copy the exported report (CSV file) to configured repository.	Verify the configured repository. If it has been deleted, add it back. If it is not available or not reachable, reconfigure the repository to a valid one.
Trustsec
Unknown SGT was provisioned	Unknown SGT was provisioned.	ISE provisioned the Unknown SGT as part of the authorization flow. Unknown SGT should not be assigned as part of a known flow.
Some TrustSec network devices do not have the latest ISE IP-SGT mapping configuration	Some TrustSec network devices do not have the latest ISE IP-SGT mapping configuration.	ISE identified some network devices that have a different IP-SGT mapping sets. Use the IP-SGT mapping Deploy option to update the devices.
TrustSec SSH connection failed	TrustSec SSH connection failed	ISE failed to establish SSH connection to a network device. Verify if the network device SSH credentials in the Network Device page are similar to the credentials configured on the network device. Check the network device enabled ssh connections from ISE (ip address).
TrustSec identified ISE was set to work with TLS versions other then 1.0	TrustSec identified ISE was set to work with TLS versions other then 1.0.	TrustSec supports only TLS version 1.0.
Trustsec PAC validation failed	Trustsec PAC validation failed	ISE could not validate a PAC which was sent by the network device. Check the Trustsec device credentials in the Network Device page and in the device CLI. Make sure the device uses a valid pac which was provisioned by the ISE server.

Alarms are not triggered when you add users or endpoints to Cisco ISE.

Add Custom Alarms

Cisco ISE contains 12 default alarm types, such as High Memory Utilization and Configuration Changes. Cisco-defined system alarms are listed in the Alarms Settings window (Administration > System > Settings > Alarm Settings). You can only edit the system alarms.

In addition to existing system alarms, you can add, edit, or delete custom alarms under the existing alarm types.

In the Alarm Configuration tab of the Alarm Settings window, the Conditions column displays details for four alarms: High Authentication Latency, High Disk I/O Utilization, High Disk Space Utilization, and High Memory Utilization. Each of these alarms has a configurable threshold value. However, the Conditions column may not display details even when the threshold values are configured. In this case, re-edit the relevant threshold field for the alarm to view the details in the Conditions column.

To add an alarm:

Procedure

Step 1	Choose Administration > System > Settings > Alarm Settings.
Step 2	In the Alarm Configuration tab, click Add.
Step 3	Enter the required details. Based on the alarm type (High Memory Utilization, Excessive RADIUS Authentication Attempts, Excessive TACACS Authentication Attempts, and so on), additional attributes are displayed in the Alarm Configuration window. For example, Object Name, Object Type, and Admin Name fields are displayed for Configuration Change alarms. You can add multiple instances of same alarm with different criteria.
Step 4	Click Submit.

Cisco ISE Alarm Notifications and Thresholds

You can enable or disable Cisco ISE alarms and configure alarm notification behavior to notify you of critical conditions. For certain alarms, you can configure thresholds such as maximum failed attempts for the Excessive Failed Attempts alarm or maximum disk utilization for the High Disk Utilization alarm.

Enable and Configure Alarms

Procedure

Step 1	Choose Administration > System > Settings > Alarm Settings.
Step 2	Select an alarm from the list of default alarms and click Edit.
Step 3	Select Enable or Disable.
Step 4	Configure alarm threshold, if applicable.
Step 5	Click Submit.

Cisco ISE Alarms for Monitoring

Cisco ISE provides system alarms which notify you whenever any critical system condition occurs. Alarms that are generated by Cisco ISE are displayed in the Alarm dashlet. These notifications automatically appear in the alarm dashlet.

The Alarm dashlet displays a list of recent alarms, which you can select from to view the alarm details. You can also receive notification of alarms through e-mail and syslog messages.

View Monitoring Alarms

Procedure

Step 1

Go to the Cisco ISE Dashboard.

Step 2

Click on an alarm in the Alarms dashlet. A new window opens with the alarm details and a suggested action.

Step 3

Click Refresh to refresh the alarms.

Step 4

Acknowledge alarms to reduce the alarm counters (number of times an alarm is raised) by marking them as read. You can select alarms to acknowledge by checking the check boxes next to the timestamps.

Choose Acknowledge Selected from the Acknowledge drop-down list to mark as read all the alarms currently displayed in the window. By default, 100 rows are displayed in the window. You can choose a different number of rows to be displayed, by choosing the desired value from the Rows/Page drop-down list.

Choose Acknowledge All from the Acknowledge drop-down list to mark as read all the alarms in the list, whether or not they are currently displayed in the window.

Note

When you check the check box near Time Stamp in the title row, all the alarms displayed in the window are selected. However, if you then uncheck a check box for one or more of the selected alarms, the select all function lapses. You will see that the check box near Time Stamp is unchecked at this point.

Step 5

Click the Details link corresponding to the alarm that you select. A new window opens with the details corresponding to the selected alarm.

Note

The Details link corresponding to the alarms that were generated prior to persona change shows no data.

Log Collection

Monitoring services collect log and configuration data, store the data, and then process it to generate reports and alarms. You can view the details of the logs that are collected from any of the servers in your deployment.

Alarm Syslog Collection Location

If you configure monitoring functions to send alarm notifications as syslog messages, you need a syslog target to receive the notification. Alarm syslog targets are the destinations where alarm syslog messages are sent.

Note

Cisco ISE monitoring requires that the logging-source interface configuration use the network access server (NAS) IP address. You must configure a switch for Cisco ISE monitoring.

You must also have a system that is configured as a syslog server to be able to receive syslog messages. You can create, edit, and delete alarm syslog targets.

To configure a remote logging target as an alarm target, perform this procedure.

Procedure

Step 1	Choose Administration > System > Logging > Remote Logging Targets.
Step 2	Click Add.
Step 3	In the New Logging Target section displayed, submit the required details for the logging target, and check the Include Alarms for this Target check box.

Live Authentications

You can monitor recent RADIUS authentications as they happen from the Live Authentications window. The window displays the top 10 RADIUS authentications in the last 24 hours. This section explains the functions of the Live Authentications window.

The Live Authentications window shows the live authentication entries corresponding to the authentication events as they happen. In addition to authentication entries, this window also shows the live session entries corresponding to the events. You can also drill-down a session to view a detailed report corresponding to that session.

The Live Authentications window provides a tabular account of recent RADIUS authentications, in the order in which they occur. The last update shown at the bottom of the Live Authentications window shows the date of the server, time, and timezone.

Note

If the password attribute in an Access-Request packet is empty, an error message is triggered and the access request will fail.

When a single endpoint authenticates successfully, two entries appear in the Live Authentications window: one corresponding to the authentication record and another corresponding to the session record (pulled from the session live view). Subsequently, when the device performs another successful authentication, the repeat counter corresponding to the session record is incremented. The Repeat Counter that appears in the Live Authentications window shows the number of duplicate RADIUS authentication success messages that are suppressed.

See the Live Authentication data categories that are shown by default. These are described in the Recent RADIUS Authentications section.

You can choose to view all the columns, or only selected data columns. After selecting the columns that you want to be displayed, you can save your selections.

Monitor Live Authentications

Procedure

Step 1	Choose Operations > RADIUS > Live logs.
Step 2	From the Refresh drop-down list, select a time interval to change the data refresh rate.
Step 3	Click the Refresh icon to manually update the data.
Step 4	From the Show drop-down list, choose an option to change the number of records that appear.
Step 5	From the Within drop-down list, choose an option to specify a time interval.
Step 6	Click Add or Remove Columns and choose the options from the drop-down list to change the columns that are displayed.
Step 7	Click Save at the bottom of the drop-down list to save your modifications.
Step 8	Click Show Live Sessions to view the live RADIUS sessions. You can use the dynamic Change of Authorization (CoA) feature for the Live Sessions that allows you to dynamically control active RADIUS sessions. You can send reauthenticate or disconnect requests to a Network Access Device (NAD).

Filter Data in the Live Authentications Page

Using the filters in the Live Authentications page, you can filter the information that you need and troubleshoot network authentication issues quickly. You can filter records in the Authentication (live logs) page and view only those records that you are interested in. The authentication logs contain many details, and filtering the authentications from a particular user or location helps you scan the data quickly. You can use several operators that are available in the fields in the Live Authentications page to filter out records based on your search criteria.:

'abc' : Contains 'abc'
'!abc' : Does not contain 'abc'
'{}' : Is empty
'!{}' : Is not empty
'abc*' : Starts with 'abc'
'*abc' : Ends with 'abc'
'\!', '\*', '\{', '\\' : Escape

The Escape option allows you to filter text with special characters (including the special characters used as filters). You must prefix the special character with a backward slash (\). For example, if you want to view the authentication records of users with identity "Employee!," enter "Employee\!" in the Identity Filter field. In this example, Cisco ISE considers the exclamation mark (!) as a literal character and not as a special character.

In addition, the Status field allows you to filter only passed authentication records, failed authentications, live sessions, and so on. The green check mark filters all the passed authentications that occurred in the past. The red cross mark filters all failed authentications. The blue i icon filters all live sessions. You can also choose to view a combination of these options.

Procedure

Step 1

Choose Operations > RADIUS > Live Logs.

Step 2

Filter data based on any of the fields in the Show Live Authentications page.

You can filter the results based on passed or failed authentications, or live sessions.

Global Search for Endpoints

You can use the global search box available at the top of the Cisco ISE home page to search for endpoints. You can use any of the following criteria to search for an endpoint:

User name
MAC Address
IP Address
Authorization Profile
Endpoint Profile
Failure Reason
Identity Group
Identity Store
Network Device name
Network Device Type
Operating System
Posture Status
Location
Security Group
User Type

You should enter at least three characters for any of the search criteria in the Search field to display data.

Note

If an endpoint has been authenticated by Cisco ISE, or its accounting update has been received, it can be found through the global search. Endpoints that have been manually added and are not authenticated by or accounted for in Cisco ISE will not show up in the search results.

The search result provides a detailed and at-a-glance information about the current status of the endpoint, which you can use for troubleshooting. Search results display only the top 25 entries. It is recommended to use filters to narrow down the results.

The following figure shows an example of the search result.

You can use any of the properties in the left panel to filter the results. You can also click on any endpoint to see more detailed information about the endpoint, such as:

Session trace
Authentication details
Accounting details
Posture details
Profiler details
Client Provisioning details
Guest accounting and activity

Session Trace for an Endpoint

You can use the global search box available at the top of the Cisco ISE home page to get session information for a particular endpoint. When you search with a criteria, you get a list of endpoints. Click on any of these endpoints to see the session trace information for that endpoint. The following figure shows an example of the session trace information displayed for an endpoint.

Note

The dataset used for search is based on Endpoint ID as indexes. Therefore, when authentication occurs, it is mandatory to have Endpoint IDs for the endpoints for those authentications to include them in the search result set.

You can use the clickable timeline at the top to see major authorization transitions. You can also export the results in .csv format by clicking the Export Results button. The report gets downloaded to your browser.

You can click on the Endpoint Details link to see more authentication, accounting, and profiler information for a particular endpoint. The following figure shows an example of endpoint details information displayed for an endpoint.

Session Removal from the Directory

Sessions are cleaned from the session directory on the Monitoring and Troubleshooting node as follows:

Terminated sessions are cleaned 15 minutes after termination.
If there is authentication but no accounting, then such sessions are cleared after one hour.
All inactive sessions are cleared after five days.

Authentication Summary Report

You can troubleshoot network access for a specific user, device, or search criteria based on attributes that are related to the authentication requests. You can do this by running an Authentication Summary report.

Troubleshoot Network Access Issues

Procedure

Step 1

Choose Operations > Reports > Authentication Summary Report.

Step 2

Filter the report for Failure Reasons.

Step 3

Review the data in the Authentication by Failure Reasons section of the report to troubleshoot your network access problem.

Note

Since the Authentication Summary report collects and displays the latest data corresponding to failed or passed authentications, the contents of the report appear after a delay of a few minutes.

Diagnostic Troubleshooting Tools

Diagnostic tools help you diagnose and troubleshoot problems on a Cisco ISE network and provide detailed instructions on how to resolve problems. You can use these tools to troubleshoot authentications and evaluate the configuration of any network device on your network, including Trustsec devices.

The RADIUS Authentication Troubleshooting Tool

This tool allows you to search and select a RADIUS authentication or an Active Directory-related RADIUS authentication for troubleshooting when there is an unexpected authentication result. Use this tool if you expected an authentication to pass, but it failed, or if you expected a user or machine to have a certain level of privileges, and the user or machine did not have those privileges.

Searching RADIUS authentications based on Username, Endpoint ID, Network Access Service (NAS) IP address, and reasons for authentication failure for troubleshooting, Cisco ISE displays authentications only for the system (current) date.

Searching RADIUS authentications based on NAS Port for troubleshooting, Cisco ISE displays all NAS Port values since the beginning of the previous month to the current date.

Note

When searching RADIUS authentications based on NAS IP address and Endpoint ID fields, a search is first performed in the operational database, and then in the configuration database.

Troubleshoot Unexpected RADIUS Authentication Results

Procedure

Step 1	Choose Operations > Troubleshoot > Diagnostic Tools > > General Tools > RADIUS Authentication Troubleshooting.
Step 2	Specify the search criteria in the fields, as needed.
Step 3	Click Search to display the RADIUS authentications that match your search criteria. If you are searching for AD-related authentication, and an Active Directory server is not configured in your deployment, a message saying 'AD not configured' is displayed.
Step 4	Select a RADIUS authentication record from the table, and click Troubleshoot. If you need to troubleshoot AD-related authentication, access the Diagnostics Tool under Administration > Identity Management > External Identity Sources > Active Directory > AD node.
Step 5	Click User Input Required, modify the fields, as needed, and then click Submit.
Step 6	Click Done.
Step 7	Click Show Results Summary after the troubleshooting is complete.
Step 8	To view the diagnosis, the steps taken to resolve the problem, and the troubleshooting summary, click Done.

The Execute Network Device Command Diagnostic Tool

The Execute Network Device Command diagnostic tool allows you to run the show command on any network device. The results are exactly the same as what you would see on a console, and can be used to identify problems in the configuration of the device. You can use it when you suspect that the configuration is wrong, you want to validate it, or if you are just curious about how it is configured.

Execute Cisco IOS show Commands to Check Configuration

Procedure

Step 1	Choose Operations > Troubleshoot > Diagnostic Tools > General Tools > Execute Network Device Command.
Step 2	In the Cisco ISE GUI, click the Menu icon () and choose Operations > Troubleshoot > Diagnostic Tools > General Tools > Execute Network Device Command.
Step 3	Enter the information in the appropriate fields.
Step 4	Click Run to execute the command on the specified network device.
Step 5	Click User Input Required, and modify the fields, as necessary.
Step 6	Click Submit to run the command on the network device, and view the output.

The Evaluate Configuration Validator Tool

You can use this diagnostic tool to evaluate the configuration of a network device and identify configuration problems, if any. The Expert Troubleshooter compares the configuration of the device with the standard configuration.

Troubleshoot Network Device Configuration Issues

Procedure

Step 1	Choose Operations > Troubleshoot > Diagnostic Tools > General Tools > Evaluate Configuration Validator.
Step 2	Enter the Network Device IP address of the device whose configuration you want to evaluate, and specify other fields, as necessary.
Step 3	Select the configuration options to compare against the recommended template.
Step 4	Click Run.
Step 5	Click User Input Required, and modify the fields, as necessary.
Step 6	Check the check boxes next to the interfaces that you want to analyze, and click Submit.
Step 7	Click Show Results Summary.

Posture Troubleshooting Tool

The Posture Troubleshooting tool helps you find the cause of a posture-check failure to identify the following:

Which endpoints were successful in posture and which were not.
If an endpoint failed in posture, what steps failed in the posture process.
Which mandatory and optional checks passed and failed.

You determine this information by filtering requests based on parameters, such as username, MAC address, and posture status.

Troubleshoot Endpoint Posture Failure

Procedure

Step 1	Choose Operations > Troubleshoot > Diagnostic Tools > General Tools > Posture Troubleshooting.
Step 2	Enter the information in the appropriate fields.
Step 3	Click Search.
Step 4	To find an explanation and determine a resolution for an event, select the event in the list and click Troubleshoot.

Technical Support Tunnel for Advanced Troubleshooting

Cisco ISE uses the Cisco IronPort Tunnel infrastructure to create a secure tunnel for Cisco technical support engineers to connect to an ISE server and troubleshoot issues with the system. Cisco ISE uses SSH to create the secure connection through the tunnel.

As an administrator, you can control the tunnel access; you can choose when and how long to grant access to a support engineer. Cisco Customer Support cannot establish the tunnel without your intervention. You will receive notifications about the service logins. You can disable the tunnel connection at any point of time. By default, the technical support tunnel remains open for 72 hours. However, we recommend that you or the support engineer close the tunnel when all the troubleshooting work is complete. You can choose to extend the tunnel beyond 72 hours, if needed.

Use tech support-tunnel enable command to initiate a tunnel connection.

The tech support-tunnel status command displays the status of the connection. This command provides information on whether the connection is established or not, if there is an authentication failure, or if the servers are unreachable. If the tunnel server is reachable, but ISE is unable to authenticate, ISE tries to authenticate again every 5 minutes for a period of 30 minutes, after which the tunnel is disabled.

You can disable the tunnel connection using the tech support-tunnel disable command. This command disconnects an existing tunnel even if a support engineer is currently logged in.

If you have already established a tunnel connection from an ISE server, the SSH keys that are generated are available on the ISE server. When you try to enable the support tunnel at a later point of time, the system prompts you about reusing the SSH keys generated earlier. You can choose to use the same keys or generate new keys. You can also manually reset the keys using the tech support-tunnel resetkey command. If you execute this command when a tunnel connection is enabled, the system prompts you to disable the connection first. If you choose to continue with the existing connection and not disable t,he connection the keys are reset after the existing connection is disabled. If you choose to disable the connection, the tunnel connection is dropped and the keys are reset immediately.

After you establish a tunnel connection, you can extend it using the tech support-tunnel extend command.

See the Cisco Identity Services Engine CLI Reference Guide for usage guidelines about the tech support-tunnel command.

Establish a Technical Support Tunnel

You can establish a secure tunnel through the Cisco ISE Command Line Interface (CLI).

Procedure

Step 1	Enter the following command from the Cisco ISE CLI: tech support-tunnel enable The system prompts you for a password and a nickname for the tunnel.
Step 2	Enter the password.
Step 3	(Optional) Enter a nickname for the tunnel. The system generates an SSH key and displays the password, device serial number, and the SSH key. You must pass this information to Cisco Customer Support for the support engineer to connect to your system.
Step 4	Copy the password, device serial number, and SSH key and send it to Cisco Customer Support. The support engineer can now securely connect to your ISE server. You will receive periodic notifications about service logins.

TCP Dump Utility to Validate the Incoming Traffic

TCP Dump Utility sniffs packets, which you can use to verify that the expected packet has reached a node. For example, when there is no incoming authentication or log indicated in the report, you may suspect that there is no incoming traffic, or that the incoming traffic cannot reach Cisco ISE. In such cases, you can run this tool to validate.

You can configure the TCP Dump options and then collect data from the network traffic to help you troubleshoot a network issue.

Caution

Starting a TCP Dump automatically deletes a previous dump file. To save a previous dump file, perform the task, as described in the Saving a TCP Dump File section before you begin a new TCP Dump session.

Use TCP Dump to Monitor Network Traffic

Before you begin

The Network Interface drop-down list in the TCP Dump page displays only the network interface cards (NICs) that have an IPv4 or IPv6 address configured. By default in VMware, all NICs are connected, which means that all NICs have an IPv6 address and are displayed in the Network Interface drop-down list.

Procedure

Step 1	Choose Operations > Troubleshoot > Diagnostic Tools > General Tools > TCP Dump.
Step 2	In the Cisco ISE GUI, click the Menu icon () and choose Operations > Troubleshoot > Diagnostic Tools > General Tools > TCP Dump.
Step 3	Choose a Host Name as the source for the TCP Dump utility.
Step 4	Choose Network Interface to monitor from the drop-down list.
Step 5	Click the Promiscuous Mode toggle button to On or Off. The default is On. Promiscuous mode is the default packet sniffing mode in which the network interface passes all the traffic to the system’s CPU. We recommend that you leave it On.
Step 6	In the Filter field, enter a boolean expression on which to filter. The following are supported standard tcpdump filter expressions: ip host 10.77.122.123 ip host ISE123 ip host 10.77.122.123 and not 10.77.122.119
Step 7	Click Start to begin monitoring the network.
Step 8	Click Stop after you have collected a sufficient amount of data, or wait for the process to conclude automatically after accumulating the maximum number of packets which is 500,000.

Note

Cisco ISE does not support frames greater than 1500 MTU (jumbo frames).

Save a TCP Dump File

Before you begin

You should have successfully completed the task, as described in the Using TCP Dump to Monitor network Traffic section.

Note

You can also access TCP Dump through the Cisco ISE CLI. For more information, see the Cisco Identity Services Engine CLI Reference Guide.

Procedure

Step 1	Choose Operations > Troubleshoot > Diagnostic Tools > General Tools > TCP Dump.
Step 2	From the Format drop-down list, choose an option. Human Readable is the default.
Step 3	Click Download, navigate to the desired location, and then click Save.
Step 4	To get rid of the previous dump file without saving it first, click Delete.

Compare Unexpected SGACL for an Endpoint or User

Procedure

Step 1	Choose Operations > Troubleshoot > Diagnostic Tools > Trustsec Tools > Egress (SGACL) Policy.
Step 2	In the Cisco ISE GUI, click the Menu icon () and choose Operations > Troubleshoot > Diagnostic Tools > Trustsec Tools > Egress (SGACL) Policy.
Step 3	Enter the Network Device IP address of the Trustsec device whose SGACL policy you want to compare.
Step 4	Click Run.
Step 5	Click User Input Required and modify the fields as necessary.
Step 6	Click Submit.
Step 7	Click Show Results Summary to view the diagnosis and suggested resolution steps.

Egress Policy Diagnostic Flow

The Egress Policy diagnostic tool uses the process described in the following table for its comparison:


Process Stage	Description
1	Connects a the device with the IP address that you provided, and obtains the access control lists (ACLs) for each source and destination SGT pair.
2	Checks the egress policy that is configured in Cisco ISE and obtains the ACLs for each source and destination SGT pair.
3	Compares the SGACL policy that is obtained from the network device with the SGACL policy that is obtained from Cisco ISE.
4	Displays the source and destination SGT pair if there is a mismatch. Also, displays the matching entries as additional information.

Troubleshoot Connectivity Issues in a Trustsec-Enabled Network with SXP-IP Mappings

Procedure

Step 1	Choose Operations > Troubleshoot > Diagnostic Tools > Trustsec Tools > SXP-IP Mappings .
Step 2	Enter the IP address of the network device.
Step 3	Click Select.
Step 4	Click Run, and then click User Input Required and modify the necessary fields. The Expert Troubleshooter retrieves Trustsec SXP connections from the network device and again prompts you to select the peer SXP devices.
Step 5	Click User Input Required, and enter the necessary information.
Step 6	Check the check box of the peer SXP devices for which you want to compare SXP mappings, and enter the common connection parameters.
Step 7	Click Submit.
Step 8	Click Show Results Summary to view the diagnosis and resolution steps.

Troubleshoot Connectivity Issues in a Trustsec-Enabled Network with IP-SGT Mappings

Procedure

Step 1	Choose Operations > Troubleshoot > Diagnostic Tools > Trustsec Tools > IP User SGT.
Step 2	Enter the information in the fields, as needed.
Step 3	Click Run. You are prompted for additional input.
Step 4	Click User Input Required, modify the fields as necessary.
Step 5	Click Submit.
Step 6	Click Show Results Summary to view the diagnosis and resolution steps.

Device SGT Tool

For devices that are enabled with the Trustsec solution, each network device is assigned an SGT value through RADIUS authentication. The Device SGT diagnostic tool connects to the network device (with the IP address that you provide) and obtains the network device SGT value. It then checks the RADIUS authentication records to determine the SGT value that was assigned most recently. Finally, it displays the Device-SGT pairs in a tabular format, and identifies whether the SGT values are the same or different.

Troubleshoot Connectivity Issues in a Trustsec-Enabled Network by Comparing Device SGT Mappings

Procedure

Step 1	Choose Operations > Troubleshoot > Diagnostic Tools > Trustsec Tools > Device SGT.
Step 2	Enter the information in the fields, as needed. The default port number for Telnet is 23 and SSH is 22.
Step 3	Click Run.
Step 4	Click Show Results Summary to view the results of the device SGT comparison.

Download Endpoint Statistical Data From Monitoring Nodes

You can download statistical data about endpoints that connect to your network from the Monitoring nodes. Key Performance Metrics (KPM), which include the load, CPU usage, authentication traffic data are available that you can use to monitor and troubleshoot issues in your network. From the Cisco ISE Command-Line Interface (CLI), use the application configure ise command and choose options 12 or 13 to download the daily KPM statistics or KPM statistics for the last eight weeks, respectively.

The output of this command provides the following data about endpoints:

Total endpoints on your network
Number of endpoints that established a successful connection
Number of endpoints that failed authentication
Total number of new endpoints that have connected each day
Total number of endpoints onboarded each day

The output also includes time stamp details, the total number of endpoints that connected through each of the Policy Service Nodes (PSNs) in the deployment, total number of endpoints, active endpoints, load, and authentication traffic details.

Refer to the Cisco Identity Services Engine CLI Reference Guide for more information on this command.

Obtaining Additional Troubleshooting Information

Cisco allows you to download support and troubleshooting information from the Admin portal. You can use the support bundle to prepare diagnostic information for the Cisco Technical Assistance Center (TAC) to troubleshoot problems with Cisco .

Note

The support bundles and debug logs provide advanced troubleshooting information for TAC and are difficult to interpret. You can use the various reports and troubleshooting tools that Cisco provides to diagnose and troubleshoot issues that you are facing in your network.

Cisco Support Bundle

You can configure the logs that you want to be a part of your support bundle. For example, you can configure logs from a particular service to be part of your debug logs. You can also filter the logs based on dates.

The logs that you can download are categorized as follows:

Full configuration database: Contains the Cisco configuration database in a human-readable XML format. When you troubleshoot issues, you can import this database configuration into another Cisco ISE node to re-create the scenario.
Debug logs: Captures bootstrap, application configuration, run-time, deployment, public key infrastructure (PKI) information and monitoring and reporting.

Debug logs provide troubleshooting information for specific Cisco ISE components. To enable debug logs, see Chapter 11, “Logging”. If you do not enable the debug logs, all the informational messages (INFO) will be included in the support bundle. For more information, see Cisco Debug Logs.
Local logs: Contains syslog messages from the various processes that run on Cisco ISE.
Core files: Contains critical information that helps identify the cause of a crash. These logs are created when the application crashes, and includes heap dumps.
Monitoring and reporting logs: Contains information about alerts and reports.
System logs: Contains Cisco Application Deployment Engine-related (ADE-related) information.
Policy configuration: Contains policies configured in Cisco ISE in human-readable format.

You can download these logs from the Cisco ISE CLI by using the backup-logs command. For more information, See the Cisco Identity Services Engine CLI Reference Guide.

Note

For Inline Posture nodes, you cannot download the support bundle from the Admin portal. You must use the backup-logs command from the Cisco ISE CLI.

If you choose to download these logs from the Admin portal, you can do the following:

Download only a subset of logs based on the log type, such as debug logs or system logs.
Download only the latest n number of files for the selected log type. This option allows you to control the size of the support bundle and the time taken for download.

Monitoring logs provide information about the monitoring, reporting, and troubleshooting features. For more information about downloading logs, see Download Cisco Log Files.

Support Bundle

You can download the support bundle to your local computer as a simple tar.gpg file. The support bundle will be named with the date and time stamps in the format ise-support-bundle_ise-support-bundle-mm-dd-yyyy--hh-mm.tar..gpg. The browser prompts you to save the support bundle to an appropriate location. You can extract the content of the support bundle and view the README.TXT file, which describes the contents of the support bundle, as well as how to import the contents of the ISE database if it is included in the support bundle.

Download Cisco Log Files

You can download the Cisco log files to look for more information while troubleshooting issues in your network.

Before you begin

You must have Super Admin or System Admin privileges to perform the following task.
You should have configured the debug logs and debug log levels.

Procedure

Step 1	Choose Operations > Troubleshoot > Download Logs > Appliance Node List.
Step 2	Click the node from which you want to download the support bundles.
Step 3	In the Support Bundle tab, choose the parameters that you want to be populated in your support bundle. If you include all the logs, your support bundle will be excessively large and the download will take a long time. To optimize the download process, choose to download only the most recent n number of files.
Step 4	Enter the From and To dates for which you want to generate the support bundle.
Step 5	Enter and re-enter the encryption key for the support bundle.
Step 6	Click Create Support Bundle.
Step 7	Click Download to download the newly-created support bundle. The support bundle is a tar.gpg file that is downloaded to the client system that is running your application browser.

What to do next

Download debug logs for specific components.

Cisco Debug Logs

Debug logs provide troubleshooting information for various Cisco components. Debug logs contain critical and warning alarms generated over the last 30 days, and information alarms generated over the last 7 days. While reporting problems, you might be asked to enable these debug logs and send them for diagnosis and resolution of your problems.

Note

Enabling debug logs with heavy load (such as monitoring debug logs) may generate alarms about high load.

Obtain Debug Logs

Procedure

Step 1	Configure the components for which you want to obtain debug logs.
Step 2	Download the debug logs.

Cisco Components and Corresponding Debug Logs

Table 3. Components and Corresponding Debug Logs
Component	Debug Log
Active Directory	ad_agent.log
Cache Tracker	tracking.log
Entity Definition Framework (EDF)	edf.log
JMS	ise-psc.log
License	ise-psc.log
Notification Tracker	tracking.log
Replication-Deployment	replication.log
Replication-JGroup	replication.log
Replication Tracker	tracking.log
RuleEngine-Attributes	ise-psc.log
RuleEngine-Policy-IDGroups	ise-psc.log
accessfilter	ise-psc.log
admin-infra	ise-psc.log
boot-strap wizard	ise-psc.log
cisco-mnt	ise-psc.log
client	ise-psc.log
cpm-clustering	ise-psc.log
cpm-mnt	ise-psc.log
epm-pdp	ise-psc.log
epm-pip	ise-psc.log
anc	ise-psc.log
anc	ise-psc.log
ers	ise-psc.log
guest	ise-psc.log
Guest Access Admin	guest.log
Guest Access	guest.log
MyDevices	guest.log
Portal	guest.log
Portal-Session-Manager	guest.log
Portal-web-action	guest.log
guestauth	ise-psc.log
guestportal	ise-psc.log
identitystore-AD	ise-psc.log
infrastructure	ise-psc.log
mdm	ise-psc.log
mdm-pip	ise-psc.log
mnt-report	reports.log
mydevices	ise-psc.log
nsf	ise-psc.log
nsf-session	ise-psc.log
org-apache	ise-psc.log
org-apache-cxf	ise-psc.log
org-apache-digester	ise-psc.log
posture	ise-psc.log
profiler	profiler.log
provisioning	ise-psc.log
prrt-JNI	prrt-management.log
runtime-AAA	prrt-management.log
runtime-config	prrt-management.log
runtime-logging	prrt-management.log
sponsorportal	ise-psc.log
swiss	ise-psc.log

Download Debug Logs

Before you begin

To perform the following task, you must be a Super Admin or System Admin.

Procedure

Step 1	Choose Operations > Troubleshoot > Download Logs > Appliance Node List.
Step 2	From the Appliance node list, click the node from which you want to download the debug logs.
Step 3	Click the Debug Logs tab. A list of debug log types and debug logs is displayed. This list is based on your debug log configuration.
Step 4	Click the log file that you want to download and save it to the system that is running your client browser. You can repeat this process to download other log files as needed. The following are additional debug logs that you can download from the Debug Logs window: isebootstrap.log: Provides bootstrapping log messages monit.log: Provides watchdog messages pki.log: Provides third-party crypto library logs iseLocalStore.log: Provides logs about the local store files ad_agent.log: Provides Microsoft Active Directory third-party library logs catalina.log: Provides third-party logs

Monitoring Database

The rate and amount of data that is utilized by Monitoring functions requires a separate database on a dedicated node that is used for these purposes.

Like Policy Service, Monitoring has a dedicated database that requires you to perform maintenance tasks, such as the topics covered in this section:

Back Up and Restore of the Monitoring Database

Monitoring database handles large volumes of data. Over time, the performance and efficiency of the monitoring node depends on how well you manage that data. To increase efficiency, we recommend that you back up the data and transfer it to a remote repository on a regular basis. You can automate this task by scheduling automatic backups.

Note

You should not perform a backup when a purge operation is in progress. If you start a backup during a purge operation, the purge operation stops or fails.

If you register a secondary Monitoring node, we recommend that you first back up the primary Monitoring node and then restore the data to the new secondary Monitoring node. This ensures that the history of the primary Monitoring node is in sync with the new secondary node as new changes are replicated.

Monitoring Database Purge

The purging process allows you to manage the size of the Monitoring database by specifying the number of months to retain data during a purge. The default is three months. This value is utilized when the disk space usage threshold for purging (percentage of disk space) is met. For this option, each month consists of 30 days. A default of three months equals 90 days.

Guidelines for Purging the Monitoring Database

The following are some guidelines to follow relating to Monitoring database disk usage:

If the Monitoring database disk usage is greater than 80 percent of the threshold setting, critical alarm is generated indicating that the database size has exceeded the allocated disk size. If the disk usage is greater than 90 percent another alarm is generated.

A purge process runs, creating a status history report that you can view in the Data Purging Audit window. The navigation path to this window is Operations > Reports > Deployment Status > Data Purging Audit. An information (INFO) alarm is generated when the purge completes.
Purging is also based on the percentage of consumed disk space for the database. When the consumed disk space for the Monitoring database is equal to or exceeds the threshold (the default is 80 percent), the purge process starts. This process deletes only the last seven days of monitoring data, irrespective of what is configured in the Admin portal. It will continue this process in a loop until the disk space is below 80 percent. Purging always checks the Monitoring database disk space limit before proceeding.

Purge Older Monitoring Data

Before you begin

To perform the following task, you must be a Super Admin or System Admin.

Procedure

Step 1

Choose Administration > System > Maintenance > Data Purging.

Step 2

Specify the time period in months, for which the data will be retained. All the data prior to the specified time period will be purged. For this option, each month consists of 30 days. The default of three months equals 90 days.

Note

If the configured retention period is less than the existing retention thresholds corresponding to the diagnostics data, then the configured value overrides the existing threshold values. For example, if you configure the retention period as 3 days and this value is less than the existing thresholds in the diagnostics tables (for example, a default of 5 days), then data is purged according to the value that you configure (3 days) in this page.

Step 3

Click Submit.

Step 4

Verify the success of the data purge by viewing the Data Purging Audit report.

What to do next

Cisco ISE Log Collection

Perform an On-demand Backup

Database Crash or File Corruption Issues

Cisco ISE may crash if the oracle database files are corrupted due to power outage or other reasons resulting in data loss. Based on the incident, follow the steps below to recover from data loss.

In case of PAN corruption in deployment, you should promote the Secondary PAN to Primary PAN.
If SPAN is promotion is not possible due to small deployment or any other reason, restore the most recent available backup.
In case of PSN corruption, follow the steps to de-register, reset config and register the node again.

In case of Standalone box, restore most recent available backup.

Note

Obtain backup from the standalone box regularaly to avoid loss in the latest configuration changes.

Cisco Identity Services Engine Administrator Guide, Release 2.0

Bias-Free Language

Results

Chapter: Monitoring and Troubleshooting

Monitoring and Troubleshooting