Alarms and events | Monitor System and Application Health | Cisco Crosswork Network Controller 7.2.x Administration Guide

Provides an entry point to Crosswork Network Controller alarms, events, and trap handling information.

This section provides references for the Alarms and Events window, system event categories, Day 0, Day 1, and Day 2 event examples, and SNMP traps for events and alarms.

Alarms and events window controls

Use this topic to look up the main controls and behavior of the Crosswork Network Controller Alarms and Events window.

Alarms and events window

The Alarms and Events window in Crosswork Network Controller provides a centralized interface to view, filter, and manage system alarms and events.

Window controls

You can view the Alarms and Events by navigating to one of these locations:

From the main Crosswork Network Controller window, click .
From the main menu, choose Alerts > Alarms and Events.

Note

For information on Network or Device alarms, refer to Set Up and Monitor Alarms and Events section in the Cisco Crosswork Network Controller 7.2.x Device Lifecycle Management.

By default, Crosswork Network Controller displays the Alarms and Events window with the Show selection set to Alarms and the Category selection set to System, as shown in the figure.

The table describes the main controls and features of the Alarms and Events window:

Table 1. Alarms and events window controls
Item	Description	Details
1	Select alerts	Click the selection box next to the Alarm ID or Event ID column to select one or more alerts. Click the blue ID link in the Alarm ID or Event ID column to view details for that alert. On the Alarms window only: When you have one or more alarms selected, Crosswork Network Controller enables the Actions menu, so you can acknowledge, clear, or annotate the selected alarms.
2	Export alerts	Click the icon to export a PDF or CSV file listing full information for all the alerts shown in the window. If you select one or more alerts when you click the icon, the file contains information only for the selected alerts.
3	Actions menu	In the Alarms window, click the Actions drop-down menu to perform one or more of these actions on the currently selected alarms: Acknowledge: Marks the currently selected alarms as acknowledged. Unacknowledge: If any of the currently selected alarms have been acknowledged, restores them to the unacknowledged state. Clear: Removes all currently selected alarms from the Alarms window. Clear all of this condition: Removes all currently selected alarms that share the same condition. Notes: Lets you add a text note to all of the currently selected alarms. Crosswork Network Controller enables the Actions menu only until you select one or more alarms using the selection box next to the Alarm ID column.
4	Toggle Alarms/Events	Toggles between the Alarms and Events windows.
5	Active Alarms only	In the Alarms window, select the Active Alarms only checkbox to display all active alarms.
6	Category selection	Click the Category drop-down list to select the alarm category (System, Network, or Devices). The default selection is System.
7	More Options	Click More Options to specify whether you want to view all alerts or only the latest, and how often to sync the alerts display with the Crosswork Network Controller database. If you uncheck the Alarm History or Event History checkbox, the list shows all alerts. If you uncheck the Auto Sync checkbox, Crosswork Network Controller pauses synchronization. Note In a geo HA deployment configured with dual stack, a loss of peer connectivity may cause discrepancies in the Events display flow on the standby cluster. To address the peer connectivity issue, complete these steps: Complete the application installation on the active cluster before proceeding with the installation on the standby cluster. In the Events window on the standby cluster, click More options and uncheck the View latest events option.
8	Saved Views	Click in the Saved Views field to manage the previously saved views created using the Save View button. In the Manage Saved Views window, you can view and sort all views or only those you have saved.
9	Save View	Click the Save View button to save the current view. Crosswork Network Controller prompts you to enter and save the view under a unique name.
10	Column Settings	Click the to select which columns to display in the alerts list.
11	Filter	Click the to toggle display of the floating filter fields at the top of the alerts list. You can use these fields to set filter criteria on one or more columns in the list. Click the Filters Applied link, shown next to the icon, to clear any filter criteria you have set.

System events

To help an operator troubleshoot issues, Crosswork Network Controller infrastructure provides a syslog feature that forwards system-related events to an external server.

All the events related to the Crosswork Network Controller platform are classified broadly into three categories: Day 0, Day 1, and Day 2.

System event categories and examples

Crosswork Network Controller infrastructure system events are grouped into three main categories, each with typical actions and events.

This table lists the event categories and sample events or actions within each category:

Table 2. Event classification and sample events
Event Classification	Sample Events and Actions
Day 0 – Events related only to Crosswork Network Controller infrastructure installation.	Checking the status of the cluster Adding a worker node Slow disk or latency issues
Day 1 – Events related to Crosswork Network Controller application installation.	Restarting a microservice Restarting a microservice fails Installing an application successfully Activating an application successfully Application is still not healthy within 3 minutes of activation Node drain fails Activating an application fails Removing a worker node
Day 2 – Events related to system operations and maintenance.	Node eviction Node eviction clean up fails Deactivating an application fails/successfully Uninstallation of an application fails/successfully Slow disk or network Node insertion Node drain fails K8s ETCD clean up Node removal and Node removal fails

Note

Refer to the Cisco Crosswork Network Controller Supported Alarms and Events document for the complete list of supported alarms and events.

Day 0 event examples

Use this topic to look up Day 0 Crosswork Network Controller event examples and their associated alarm, syslog, and recommendation details.

Day 0 event examples

These checks can help determine whether the system is healthy.

Table 3. Adding a worker node
Severity	Major
Description	A VM node has been added. This event occurs when the K8 cluster detects a node.
Sample Alarm	None
Sample Syslog Message	`<time_stamp> <hosting_hybrid_node> <time_stamp> <crosswork_VIP> orchestrator-capp-infra - b54ec903-9e0f-49b8-aaf3-1d72cf644c28 vm4wkr-0 'Successfully added new VM into Inventory: vm4wkr'`
Recommendation	Monitor and confirm that the VM node appears in the UI with a healthy status.

Table 4. Slow disk or latency in network issues
Severity	Critical
Description	This event occurs when the Infrastructure Capp untar takes more than 1.5 minutes or if the Docker push takes more than 2 minutes to complete. This message can be found in the firstboot.log file.
Sample Alarm	Not applicable
Sample Syslog Message	Not applicable
Recommendation	This issue must be addressed before further operations can be made on the system. Complete these actions: Check that disk storage and network SLA requirements are met. Confirm that the observed bandwidth is the same as what is provisioned between the nodes. If using RAID, confirm it is RAID 0.

Day 1 event examples

Use this topic to look up Day 1 Crosswork Network Controller event examples and their associated alarm, syslog, and recommendation details.

Day 1 event examples

These checks can help determine whether the system is healthy.

Table 5. Removing a worker node
Severity	Major
Description	This event occurs when a VM node is erased.
Sample Alarm	None
Sample Syslog Message	`<time_stamp> <hosting_hybrid_node> <time_stamp> <crosswork_VIP> CLUSTER-CLUSTER - 33a5ce0d-6cd0-4e4d-8438-85cfa8fb4ae9 CLUSTER-99 'user=admin,policyId=admin,backend=local,loginTime=2021-02- 28T01:38:48Z,Category=VM Manager,RequestId=vm4wkr [Erase VM []]'`
Recommendation	Monitor and confirm that the VM node is no longer seen in the UI. If the erase operation fails, attempt to erase the node again.

Table 6. Adding an application—success
Severity	Information
Description	This event occurs when an application is added successfully.
Alarm
Syslog Message	`<time_stamp> <hosting_hybrid_node> <time_stamp> <crosswork_VIP> CLUSTER-CLUSTER - 627b2140-a906-4a96-b59b-1af22f2af9f6 CLUSTER-99 'job_type=INSTALL_AND_ACTIVATE_APPLICATION,manager=app_manager: ,user=admin,policyId=admin,backend=local,loginTime=2021-02- 28T09:34:54Z,payload={"package_identifier":{"id":"cappztp"," version":"1.1.0-prerelease.259+build.260"}} [accepted]'`
Recommendation	None

Table 7. Adding an application—failure
Severity	Information
Description	This event occurs when an application cannot be added.
Sample Alarm
Sample Syslog Message	None
Recommendation	After fixing the error, try adding the application again.

Table 8. Activating an application—success
Severity	Information
Description	This event occurs after an application is activated successfully.
Sample Alarm	None
Syslog Message	`<time_stamp> <hosting_hybrid_node> <time_stamp> <crosswork_VIP> orchestrator-Crosswork Health Manager - 010689d1-8842-43c2-8ebd- 5d91ded9d2d7 cw-ztp-service-0-0 ' cw-ztp-service-0 is healthy.'`
Recommendation	Activate the application and license.

Table 9. Activating an application—failure
Severity	Critical
Description	This event occurs if an application cannot be activated. The activation may fail because microservices or pods do not come up in time.
Sample Alarm	None
Syslog Message	None
Recommendation	Complete these actions: Look at the job history and identify where in the activation process it failed. If it fails at the start of one of the pods coming up, restart the pods. Uninstall the application and then try installing the application again.

Table 10. Application remains unhealthy after 3 minutes
Severity	Major
Description	This event occurs if the application was activated successfully but the components remain unhealthy after 3 minutes after application activation.
Sample Alarm	None
Sample Syslog Message	None
Recommendation	You can wait longer and if it becomes healthy, clear the alarm. Contact Cisco TAC if it still appears unhealthy after some time.

Day 2 event examples

Use this topic to look up Day 2 Crosswork Network Controller event examples and their associated alarm, syslog, and recommendation details.

Day 2 event examples

Table 11. Node drain—cleanup
Severity	Information
Description	A node drain occurs if you erase a VM node or if the node has been unresponsive for more than 5 minutes. During the drain operation, pods running on the node are moved (clustered pods may move or go pending, and single instance pods move to another node).
Sample Alarms	Node Drain Failed K8s ETCD Cleanup Failed on Node Removal Node Delete
Syslog Message	`<time_stamp> <hosting_hybrid_node> <time_stamp> <crosswork_VIP> orchestrator-Crosswork Health Manager - b062232f-54dc-49b2-8283- 506b7bf672a6 astackserver-0-0 ' astackserver-0 health is degraded.'`
Recommendation	Monitor the operation. If the drain is a result of eviction, erase the respective node and insert a new one.

Table 12. Node drain—failure
Severity	Major
Description	A node drain occurs if you erase a VM node or if the node has been unresponsive for more than 5 minutes. This event occurs if the node drain operation fails.
Sample Alarm	None
Sample Syslog Message	`<time_stamp> <hosting_hybrid_node> <time_stamp> <crosswork_VIP> orchestrator-Crosswork Health Manager - b062232f-54dc-49b2-8283- 506b7bf672a6 astackserver-0-0 ' astackserver-0 health is degraded.'`
Recommendation	Try erasing the node again.

Table 13. Node eviction—failure
Severity	Critical
Description	In this scenario we assume that one of the hybrid nodes fails. This event occurs if the node has been down for more than 5 minutes and it is automatically taken out of service. This event can be triggered if someone stopped or deleted a VM without using Crosswork Network Controller or if there is a network outage to that node. K8s automatically start evicting pods on that node (drain eviction operation). The VM node is marked down during a successful cleanup.
Sample Alarm	Node Eviction Cleanup Failure K8s ETCD Cleanup Failed on Node Removal
Syslog Message	None
Recommendation	Erase the faulty node and insert a new VM.

Table 14. Node eviction—cleanup failure
Severity	Critical
Description	This event occurs when the drain eviction fails. The node has been down for more than 5 minutes and K8s automatically start evicting pods on that node.
Sample Alarm	None
Sample Syslog Message	None
Recommendation	Erase the node and attempt another cleanup operation.

Table 15. Resource footprint shortage
Severity	Critical
Description	This event occurs when cluster node resources are being highly utilized and there is a lack of a resource footprint.
Sample Alarm	None
Sample Syslog Message	None
Recommendation	Add a new worker node.

Table 16. Deactivating an application—success
Severity	Minor
Description	This event occurs when an application is deactivated.
Sample Alarm	None
Sample Syslog Message	`<time_stamp> <hosting_hybrid_node> <time_stamp> <crosswork_VIP> CLUSTER-CLUSTER - ade982ea-7f60-4d6b-b7e0-ebafc789edee CLUSTER-99 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential – DRAFT version 1 'user=admin,policyId=admin,backend=local,loginTime=2021-02- 28T09:34:54Z,job_type=UNINSTALL_APPLICATION,manager=app_manager: ,payload={"application_id":"capp-ztp"} [accepted]'`
Recommendation	None

Table 17. Deactivating an application—failure
Severity	Critical
Description	This event occurs when an application cannot be deactivated. This can occur if microservices or pods are still running.
Sample Alarm	None
Syslog Message	None
Recommendation	Complete these actions: Look at the job history and identify where in the activation process it failed. If it fails at the start of one of the pods coming up, restart the pods. Uninstall the application and then try installing the application again.

Table 18. Slow disk or latency in network issues
Severity	Critical
Description	This event occurs when the Infrastructure Capp untar takes more than 1.5 minutes or if the Docker push takes more than 2 minutes to complete. This message can be found in the firstboot.log file.
Sample Alarm	Not applicable
Sample Syslog Message	Not applicable
Recommendation	This issue must be addressed before further operations can be made on the system. Complete these actions: Check that disk storage and network SLA requirements are met. Confirm that the observed bandwidth is the same as what is provisioned between the nodes. If using RAID, confirm it is RAID 0.

Note

There a one-time check performed to ensure the hardware attempts to meet the Disk SLA. If this fails, a critical alarm is issued. User can address the alarm as needed and manually clear the alarm.

Table 19. ETCD cleanup
Severity	Information
Description	This event occurs if someone erases a VM node and the ETCD clean membership cleanup operation begins.
Sample Alarms	If ETCD cleanup fails: K8s ETCD Cleanup Failed on Node Removal Alarm Node Delete
Syslog Message	None
Recommendation	Monitor operation.

Table 20. K8s ETCD cleanup failed on node removal
Severity	Major
Description	This event occurs if the ETCD cleanup operation fails.
Sample Alarm	None
Sample Syslog Message	None
Recommendation	Try erasing the node again.

Table 21. Restart microservices—failure
Severity	Warning
Description	This event occurs when someone restarts a microservice or pod and the operation fails.
Sample Alarm	None
Sample Syslog Message	None
Recommendation	Restart the microservices or pods. You may have to do this a few times to see if it recovers.

SNMP traps for events and alarms

In addition to UI options, REST APIs, and Syslogs, Crosswork Network Controller allows users to generate SNMP traps for events and alarms to notify the application and cluster health.

Supports SNMPv2 and SNMPv3 protocols for sending traps.
Alarms and events are filtered based on user-defined criteria before being converted to traps.
Traps are sent to the trap server using the alarm model in CISCO-EPM-NOTIFICATION-MIB.

For configuration details, refer to Configure a Trap Server.

For more information on the alarm model, refer to Cisco EPM Notification MIB.