Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Grafana and Prometheus Stack

Grafana Stack for Advanced ISE Monitoring

Enable or Disable Monitoring

Navigation Menu

Built-in Dashboards

Create Your Own Dashboard

Step 1. Enter the New Dashboard Menu

Step 2. Add a Panel

Step 3. Create Panels by Using Queries

Troubleshooting

Related Information

Introduction

This document describes the Grafana Stack components built-in Identity Services Engine (ISE) 3.3 through System 360 Advanced Monitoring.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- Cisco Identity Service Engine

- Grafana Stack

Components Used

The information in this document is based on these software and hardware versions:

- ISE 3.3

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

System 360 includesMonitoringandLog Analytics features.

The Monitoring feature enables you to monitor a wide range of application and system statistics, and the key performance indicators (KPI) of all the nodes in a deployment from a centralized console. KPIs are useful to gain insight into the overall health of the node environment. Statistics offer a simplified representation of the system configurations and utilization-specific data.

Log Analytics provides a flexible analytics system for in-depth analysis of endpoint authentication, authorization, and accounting (AAA), and profiling syslog data. You can also analyze the Cisco ISE health summary and process statuses. You can generate reports that are similar to the Cisco ISE Counters and Health Summary report.

Grafana and Prometheus Stack

The Grafana stack is a third-party open-source software stack used to provide a graphical or text-based representation of statistics and counters collected within a given environment or software solution. It is conformed by Grafana, Prometheus and Node Exporter components:

- Grafana: Grafana is a visualization and analytics software that works with Prometheus. It allows you to query, visualize, alert on, and explore on a friendly way system metrics, logs, and traces stored in Prometheus database.

- Prometheus: Prometheus pulls, collects and stores time series data cached by Node Exporter.

- Node Exporter: Constantly measures various machine resources metrics such as memory, disk and CPU utilization and caches them.
Grafana Stack Flow Chart

These components form a powerful stack for collecting, managing and analyzing diverse types of system metrics. This allows system administrators to have a real-time, and friendly visualization of the status and performance of their network solutions.

Grafana Stack for Advanced ISE Monitoring

- ISE do not require to have separate instances of Grafana stack on each node to monitor the whole deployment. The stack components running on each node depend on the roles that each ISE node has.

- Each ISE node in a deployment has its own Node exporter instance.

- Policy Administration Nodes (PAN) have independent Grafana and Prometheus instances.

- Prometheus can store up to 5GB or 7 day-old data. Once either of these thresholds is reached, the oldest data is purged first.

- The data collection, storage and processing is not handled by MnT collector. This means that enabling this feature does not have a significant impact on ISE Resource consumption.

- Monitoring feature is enabled by default.

Grafana Flow for ISE Monitoring

Enable or Disable Monitoring

Monitoring is a feature that is enabled by default on ISE. However, you can enable or disable this feature at any time.

Navigate to Operations > System 360 > Settings and click the Monitoring button to Enable or Disable the feature.

Finally, click the Save button.

Enable or Disable Monitoring

ISE takes about a minute to initialize or shut down the Grafana stack, you can check the services status using show app stat ise.

vimontes-ise-33-1/admin#show application status ise

ISE PROCESS NAME                       STATE            PROCESS ID

--------------------------------------------------------------------

Database Listener                      running          81008

Database Server                        running          134 PROCESSES

Application Server                     running          518925

Profiler Database                      running          86939

ISE Indexing Engine                    running          486865

AD Connector                           running          90383

M&T Session Database                   running          486437

M&T Log Processor                      running          2564857

Certificate Authority Service          running          245113

EST Service                            running          583881

SXP Engine Service                     disabled

TC-NAC Service                         disabled

PassiveID WMI Service                  disabled

PassiveID Syslog Service               disabled

PassiveID API Service                  disabled

PassiveID Agent Service                disabled

PassiveID Endpoint Service             disabled

PassiveID SPAN Service                 disabled

DHCP Server (dhcpd)                    disabled

DNS Server (named)                     disabled

ISE Messaging Service                  running          247148

ISE API Gateway Database Service       running          488895

ISE API Gateway Service                running          501344

ISE pxGrid Direct Service              running          559099

Segmentation Policy Service            disabled

REST Auth Service                      disabled

SSE Connector                          disabled

Hermes (pxGrid Cloud Agent)            disabled

McTrust (Meraki Sync Service)          disabled

ISE Node Exporter                      running          91058

ISE Prometheus Service                 running          357191

ISE Grafana Service                    running          504738

ISE MNT LogAnalytics Elasticsearch     running          359800

ISE Logstash Service                   running          362762

ISE Kibana Service                     running          365658

ISE Native IPSec Service               running          507795

MFC Profiler                           running          574221

Note: Depending on the personas running on each ISE node, it can be expected to see some of the Grafana Stack services in not running status even when Monitoring is enabled.

Navigation Menu

Navigate to Operations > System 360 > Monitoring to have access to Grafana Navigaton Menu. Navigation Menu is located to the left of the dashboard that ISE displays.

Grafana Navigation Menu

Built-in Dashboards

ISE by default has two built-in dashboard called ISE-Dashboard and MFC Profiler. These dashboard displays the most common Key Performance Indicators (KPIs),such as Memory, CPU and Disk statistics, separately for each ISE node in the deployment. These dashboards can also display process consumption metrics.

In order to access these dashboard, navigate to Operations > System 360 > Monitoring menu. By default, ISE displays ISE-Dashboard.

Note: Monitoring menu does not show in the GUI if the feature is not enabled.

You can select the ISE node, modify the time span of the displayed information and the dashboard refresh rate.

Monitoring Built-in Dashboard

To switch between dashboards, click on the four-square icon . This opens the Manage Dashboard window. From this window, you can select between the different existing dashboards.

Switching Between Dashboards

Note: ISE server and client machine must have the same time to avoid data inconsistencies. If a time mismatch is detected, ISE shows this warning after accessing to the built-in dashboard: "A time mismatch is detected between ISE server and client machine which can lead to inconsistent Grafana behaviour, kindly sync time on both machines."

Create Your Own Dashboard

Additional to the built-in dashboards provided, you can create your own Dashboards from zero.

Step 1. Enter the New Dashboard Menu

Navigate under Operations > System 360 > Monitoring.

Click the plus (+) icon on the Grafana Navigation Menu and click Dashboard.

Create New Dashboard

Step 2. Add a Panel

Select Add a new panel option. This displays the Edit Panel window.

Add a New Panel

This is how the Edit Panel window looks:

Dashboard Creation Area

a. Visualization area: Shows the graphic representation of the data pulled from the Prometheus database.

b. Data Queries area: You can select the queries to pull specific metrics and data stored in Prometheus database.

c. Panel Options area: Provides an extensive amount of options to modify the graphic panel that displays the data.

Step 3. Create Panels by Using Queries

Troubleshooting

- Verify the Monitoring feature is enabled.

- Verify the Grafana stack services are running on the ISE nodes, depending on the enabled personas on each ISE node.

- Each Grafana stack component has a separate log, you can access these log files using these commands on ISE CLI:

vimontes-ise-33-1/admin#show logging application ise-prometheus/prometheus.log

vimontes-ise-33-1/admin#show logging application ise-node-exporter/node-exporter.log

vimontes-ise-33-1/admin#show logging application ise-grafana/grafana.log

Note: There are no specific components to set to debug level to troubleshoot this feature. Gathering these log files is enough.

Related Information

- Release Notes for Cisco Identity Services Engine, Release 3.2

- Cisco Identity Services Engine Administrator Guide, Release 3.3

- Grafana Documentation

- Cisco Technical Support & Downloads