Verify the Health of a Secure Workload Cluster (Tetration)
Available Languages
Introduction
This document describes steps to verify the health of a Secure Workload cluster and highlights key aspects to review during the health check process.
Background Information
Its primary focus is on health verification; however, if you notice any issues or abnormal behavior, you must collect a snapshot and contact the Cisco Tetration Solution Support TAC team for assistance. The secure workload cluster is made up of hundreds of processes distributed across multiple virtual machines on several UCS C220 servers.
The two main tools for assessing cluster health are the Cluster Status and Service Status pages, both of which are explained in this document. Using these pages is generally the most effective way to confirm the overall health of the cluster.
When to Check the Health of the Cluster
Most of the time, it is not necessary to verify the health of your cluster. However, there are certain situations when it is a good idea:
• If you notice anything unusual or unexpected in the user interface (UI), based on your experience with how things normally work. Some common examples are listed in the Operational Display Parameters section.
• If you expect to see certain data (like flow data from software or hardware sensors) in the UI, but it is missing even though you have selected the right scope and time range.
• Before and after any scheduled maintenance, upgrades, or major changes to the cluster. It is a best practice to take a snapshot of the cluster's state both before and after these activities. If you ever need to contact TAC support, having these snapshots can help to quickly pinpoint what changed.
Different Options You have to Verify the Health of the Secure Workload Cluster
Cluster Status
A secure workload cluster consists of either 6 servers (8RU) or 36 servers (39RU), depending on the cluster type. The Cluster Status page provides the state of the servers as well as bare metal server information.
Note: The Cluster Status page is accessible to users with Site Admin or Customer Support roles for physical clusters. Both roles are able to view and perform actions on the Cluster Status page.
From the navigation pane, choose Troubleshoot > Cluster Status.
Caution: If you notice any node marked as inactive on the cluster status page, generate a CIMC snapshot and raise a TAC case, including the snapshot. If the status appears as Inactive, it usually means the server is either turned off or can be down due to a hardware, cable, or connectivity issue. When you click a server in the list, you see more details, such as • The virtual machines (instances) running on that physical server • The server's private IP address within the cluster • The CIMC (management) IP address • The current firmware versions for the BIOS, CIMC, VIC Card, LOM card, and RAID controller
Service StatusThe Service Status page is located in the left navigation pane under page displays the health of all services that are used in your CiscoSecure Workloadcluster along with their dependencies. The graph view shows the health of the service, each node in the graph shows the health of the service, and an edge represents dependency on other services. Unhealthy services are marked either red when the service is unavailable, and orange when the service is degraded but available. A green color or sky blue color indicates that the service is healthy. For more debug information on these nodes, use tree view which has theExpand Allbutton to show all child nodes in the dependency tree. Down, indicates that the service is not functional, and Unhealthy, indicates that the service is not fully functional.
Note: Starting with patch version 3.10.2.11, the service status page appears in sky blue. A green color or sky blue color indicates that the service is healthy.
By default, the Service Status page shows the cluster functions and dependencies in a graphical view. If the icons are all green or sky blue, no error is detected.
Caution: If you see that any of the services are unhealthy and showing red in color, contact the Technical Assistance Center (TAC) for support in resolving these issues. Quick engagement with TAC can help restore full functionality.
Hawkeye (Charts)Hawkeye dashboards offer visibility into the health of the secure workload cluster, as well as metrics and insights to assist with troubleshooting The Hawkeye (Charts) page is located in the left navigation pane under When you click Hawkeye (Charts), a new browser tab automatically opens, displaying the Hawkeye dashboard as shown here.
From the Hawkeye dashboard, click the Spark Pipeline Current tab to monitor the secure workload cluster's health. On the Spark Pipeline Current page, verify that the End-to-End Lag, Serving Layer Lag, Main Pipeline Latency, and Active Flow Latency values are all under 10 minutes. Also, confirm that the runtime values are less than 1 minute and are presented in seconds and the HDFS state is Good, as illustrated next.
Caution: If you observe any latency values, including end-to-end lag or service layer lag, exceeding 6 hours without showing a gradual decrease, please reach out to the Technical Assistance Center (TAC). Upgrade PrechecksPrior to and after maintenance tasks, use the upgrade precheck to run cluster health checks; this process ensures that services, configurations, and hardware components are all in proper working order
Wait a few minutes for the output of the upgrade prechecks. If everything is successful as shown in this image, then you can proceed with the next actions of the cluster maintenance activities.
Caution: If any upgrade prechecks are unsuccessful, please contact the Technical Assistance Center (TAC) for assistance. |
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
10-Sep-2025
|
Initial Release |
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)