Application Based Alert

Feature Description

When the system detects an anomaly, it generates an alert notification. The system statistics are the cause for these alert notifications. You can set an expression to trigger an alert when the expression becomes true.

How it Works

This section describes how this feature works.

The Common Execution Environment (CEE) uses the Prometheus Alert Manager for alerting operations. The CEE YANG model, accessible through CLI or API, allows you to view the active alerts, silenced alerts, and alert history. During the application installation or upgradation, the system adds a set of preset alerting rules. Also, the applications can call the alert API directly to add or clear alerts. The Prometheus Alert Manager API (v2) is the standard API used.

The Prometheus Alerts Manager includes the following options:

DefiningAlert Rules: This option defines the types of alerts that the Alert Manager should trigger. Use the Prometheus Query Language (PromQL) to define the alerts.
Defining Alert Routing: This option defines the action the Alert Manager should take after receiving the alerts. At present, the SNMP Trapper is supported as the outbound alerting. Also, the CEE provides an Alert Logger for storing the generated alerts.

Configure alert rules

This section describes how to configure the alert rules.

To configure the alert rules, use this configuration:

config 
   alerts rules group alert_group_name 
   rule rule_name 
      expression promql_expression 
      duration duration 
      severity severity_level 
      type alert-type 
      annotation annotation_name 
      value annotation_value 
      end

NOTES:

alerts rules —Specify the Prometheus alerting rules.
group alert_group_name —Specify the Prometheus alerting rule group. One alert group can have multiple lists of rules. alert-group-name is the name of the alert group. alert_group_name must be a string in the range of 0–64 characters.
rule rule_name —Specify the alerting rule definition. rule_name is the name of the rule.
expression promql_expression —Specify the PromQL alerting rule expression. promql_expression is the alert rule query expressed in PromQL syntax. The promql_expression must be a string in the range of 0–64 characters.
duration duration —Specify the duration of a true condition before it is considered true. duration is the time interval before the alert is triggered.
severity severity_level —Specify the severity of the alert. severity-level is the severity level of the alert. The severity levels are critical, major, minor, and warning.
type alert_type —Specify the type of the alert. alert_type is the user-defined alert type. For example, Communications Alarm, Environmental Alarm, Equipment Alarm, Indeterminate Integrity Violation Alarm, Operational Violation Alarm, Physical Violation Alarm, Processing Error Alarm, Quality of Service Alarm, Security Service Alarm, Mechanism Violation Alarm, or Time Domain Violation Alarm.
annotation annotation_name —Specify the annotation to attach to the alerts. annotation_name is the name of the annotation.
value annotation_value —Specify the annotation value. annotation_value is the value of the annotation.

The following example configures an alert, which is triggered when the percentage of N7 responses is less than the specified threshold limit.

configure terminal
  alerts rules group PCFN7chk_incr
  interval-seconds 300
  rule PCFN7chk_incr
  expression "sum(increase(inbound_request_total{interface_name=\"N7\", result_code=~\"2..\"}[3m])) / sum(increase(inbound_request_total{interface_name=\"N7\"}[3m]))< 0.95"
    severity major
    type "N7 Communications Alarm"
    annotation summary
    value "This alert is fired when the percentage of N7 responses is less than threshold“
    exit
  exit
exit

View alert logger

The alert logger stores the alerts that CPC generates by default. View these alerts using the this command:

show alert history [ filtering ]

Narrow down the result using these filtering options:

annotations—Specify the annotations of the alert.
endsAt—Specify the end time of the alert.
labels—Specify the additional labels of the alert.
severity—Specify the severity of the alert.
source—Specify the source of the alert.
startsAt—Specify the start time of the alert.
type—Specify the type of the alert.

You can view the active and silenced alerts with the show alerts active and show alerts active commands.

show running-config alerts
  interval-seconds 300
  rule PCFN7chk_incr
  expression "sum(increase(inbound_request_total{interface_name=\"N7\", result_code=~\"2..\"}[3m])) / sum(increase(inbound_request_total{interface_name=\"N7\"}[3m]))< 0.95"
    severity major
    type "N7 Communications Alarm"
    annotation summary
    value "This alert is fired when the percentage of N7 responses is less than threshold“
    exit 
  exit
exit

This section provides sample outputs for active and historical alerts in the CPC system.

show alerts history
alerts active PCFN7chk_incr ac2a970ab621
state active
severity major
type "N7 Communications Alarm"
startsAt 2019-11-15T08:26:48.283Z
source System
annotations [ "summary:This alert is fired when the percentage of N7 responses is less than threshold.” ]

The following example displays the active alerts. The alerts remain active as long as the evaluated expression is true.

show alerts active
alerts active PCFN7chk_incr ac2a970ab621
state active
severity major
type "N7 Communications Alarm"
startsAt 2019-11-15T08:26:48.283Z
source System
annotations [ "summary:This alert is fired when the percentage of N7 responses is less than threshold.” ]

Sample alerts configuration

This section provides sample configurations that are defined in CPC.

Interface-Specific Alerts

N7 Interface Inbound

Use these commands to configure alerts related to an inbound N7 interface.

alerts rules group PCFSvcStatus 
   interval-seconds300 
   rulePCFN7Inbound 
   expression sum(increase(inbound_request_total{interface_name=\”N7\”, result_code=~\”2..\”}[5m])) /sum(increase(inbound_request_total{interface_name =\”N7\”}[5m])) <0.90  
   severity major 
   type Communications Alarm 
   annotation summary 
   value This alert is fired when the percentage of Success N7 responses sent is lesser threshold. 
   exit 
exit

N7 Interface Outbound

Use the following commands to configure alerts related to an outbound N7 interface.

alerts rules group PCFSvcStatus 
   interval-seconds 300 
   rule PCFN27outbound 
   expression sum(increase(outgoing_request_total{interface_name =\”N7\”,response_status=~\”2..\”}[5m])) /sum(increase(outgoing_request_total{interface_name =\”N7\”}[5m])) <0.90 
   severity major 
   type Communications Alarm 
   annotation summary 
   value This alert is fired when the percentage of Success N7 responses received is lesser threshold. 
   exit 
exit

N28 Interface Inbound

Use the following commands to configure alerts related to an inbound N28 interface.

alerts rules group PCFSvcStatus 
   interval-seconds 300 
   rule PCFN28Inbound 
   expression sum(increase(inbound_request_total{interface_name=\”N28\”,response_status=~\”2..\”}[5m])) /sum(increase(inbound_request_total{interface_name =\”N28\”}[5m])) <0.90 
   severity major 
   type Communications Alarm 
   annotation summary 
   value This alert is fired when the percentage of Success N28 responses sent is lesser threshold. 
   exit 
exit

N28 Interface Outbound

Use the following commands to configure alerts related to an outbound N28 interface.

alerts rules group PCFSvcStatus 
   interval-seconds 300 
   rule PCFN28outbound 
   expression sum(increase(outgoing_request_total{interface_name =\”N28\”,response_status=~\”2..\”}[5m])) /sum(increase(outgoing_request_total{interface_name =\”N28\”}[5m])) <0.90 
   severity major 
   type Communications Alarm 
   annotation summary 
   value This alert is fired when the percentage of Success N28 responses received is lesser threshold. 
   exit 
exit

Diameter Rx Interface Inbound

Use the following commands to configure alerts related to an inbound Diameter Rx interface.

alerts rules group PCFSvcStatus 
   interval-seconds 300 
   rule PCFNRxInbound 
   expression sum(increase(diameter_responses_total{command_code=\”AAA|STA\”,response_status=~\”2001\”}[5m])) /sum(diameter_responses_total(outgoing_request_total{command_code=\”A AA|STA\”}[5m])) < 0.90 
   severity major 
   type Communications Alarm 
   annotation summary 
   value This alert is fired when the percentage of Success Rx responses Send is lesser threshold. 
   exit 
exit

Diameter Rx Interface Outbound

Use the following commands to configure alerts related to an outbound Diameter Rx interface.

alerts rules group PCFSvcStatus 
   interval-seconds 300 
   rule PCFNRxOutbound 
   expression sum(increase(diameter_responses_total{command_code=\”RAA|ASA\”,response_status=~\”2001\”}[5m])) /sum(diameter_responses_total(outgoing_request_total{command_code=\”AAA|STA\”}[5m])) < 0.90 
   severity major 
   type Communications Alarm 
   annotation summary 
   value This alert is fired when the percentage of Success Rx responses received is lesser threshold. 
   exit 
exit

Process level alerts

CDL Endpoint Down

Use these commands to configure alerts related to CDL endpoint down.

alerts rules group cdl-ep-change 
   rule pod-down 
   expression up{pod=~'cdl-ep.*'} == 0 
   duration 1m 
   severity major 
   type Equipment Alarm 
   annotation description 
   value CDL EP Pod Down 
   exit 
exit

CDL Slot State Change

Use these commands to configure alerts related to CDL slot state change.

alerts rules group cdl-slot-change 
   rule pod-down 
   expression up{pod=\"cdl-slot-session-c1-m1-0\"} == 0 
   severity major 
   type Equipment Alarm 
   annotation description 
   value CDL Pod Slot Change 
   exit 
exit

Diameter Endpoint State Change

Use these commands to configure alerts related to Diameter endpoint state change.

alerts rules group diamter-ep-change 
   rule pod-down 
   expression up{pod=~'diameter-ep.*'} == 0 
   duration 1m 
   severity major 
   type Equipment Alarm 
   annotation description 
   value Diameter EP Change 
   exit 
exit

ETCD State Change

Use these commands to configure alerts related to etcd state change.

alerts rules group ep-mapping-change 
   rule pod-down 
   expression up{pod=~'etcd-pcf.*'} == 0 
   duration 1m 
   severity major 
   type Equipment Alarm 
   annotation description 
   value EP Mapping Change 
   exit 
exit

Grafana Dashboard State Change

Use these commands to configure alerts related to Grafana dashboard state change.

alerts rules group grafana-dashboard-change 
   rule pod-down 
   expression up{pod=~'grafana-dashboard.*'} == 0 
   duration 1m 
   severity major 
   type Equipment Alarm 
   annotation description 
   value Grafana Dashboard Change 
   exit 
exit

Kafka State Change

Use these commands to configure alerts related to Kafka state change.

alerts rules group kafka-change 
   rule pod-down 
   expression up{pod=~'kafka.*'} == 0 
   duration 1m 
   severity major 
   type Equipment Alarm 
   annotation description 
   value Kafka Changed 
   exit 
exit

LDAP Endpoint State Change

Use these commands to configure alerts related to LDAP endpoint state change.

alerts rules group ldap-change 
   rule pod-down 
   expression up{pod=~'ldap-pcf.*'} == 0 
   duration 1m 
   severity major 
   type Equipment Alarm 
   annotation description 
   value LDAP Pod Changed 
   exit 
exit

CPC Engine State Change

Use these commands to configure alerts related to CPC Engine state change.

alerts rules group pcf-engine-change 
   rule pod-down 
   expression up{pod=~'pcf-engine-pcf.*'} == 0 
   duration 1m 
   severity major 
   type Equipment Alarm 
   annotation description 
   value CPC Engine Changed 
   exit 
exit

REST Endpoint State Change

Use these commands to configure alerts related to REST endpoint state change.

alerts rules group pcf-rest-ep-change 
   rule pod-down 
   expression up{pod=~'pcf-rest-ep.*'} == 0 
   duration 1m 
   severity major 
   type Equipment Alarm 
   annotation description 
   value CPC Rest EP Change 
   exit 
exit

Call Flow procedure alerts

LDAP Query Request

Use these commands to configure alerts related to LDAP Query Request.

alerts rules group PCFProcStatus 
   interval-seconds 300 
   rule LDAPQuery 
   expression sum(increase(message_total{type=~\”.*_ldap_query\”, status=\”success\”}[5m])) /sum(increase(message_total{type=~\”.*_ldap_query\”}[5m])) < 0.90 
   severity major 
   type Communications Alarm 
   annotation summary 
   value This alert is fired when the success percentage of ldap query request is lesser threshold. 
   exit 
exit

LDAP Modify Request

Use these commands to configure alerts related to LDAP Modify Request.

alerts rules group PCFProcStatus 
   interval-seconds 300 
   rule LDAPModify 
   expression sum(increase(message_total{component=\”ldap-ep\”, type=~\”.*_ldap_modify\”, status=\”success\”}[5m])) / sum(increase(message_total{component=\”ldap-ep\”, type=~\”.*_ldap_modify\”}[5m])) < 0.90 
   severity major 
   type Communications Alarm 
   annotation summary 
   value This alert is fired when the success percentage of ldap modify request is lesser threshold. 
   exit 
exit

PLF Query Request

Use these commands to configure alerts related to PLF Query Request.

alerts rules group PCFProcStatus 
   interval-seconds 300 
   rule PLFRequest 
   expression sum(increase(message_total{type=~\”ldap_search-res_success\”,status=\”success\”}[5m])) /sum(increase(message_total{type=~\”ldap_search-res_.*\”}[5m])) <0.90 
   severity major 
   type Communications Alarm 
   annotation summary 
   value This alert is fired when the success percentage of PLF request is lesser threshold. 
   exit 
exit

NAP Notification Request

Use these commands to configure alerts related to NAP Notification Request.

alerts rules group PCFProcStatus 
   interval-seconds 300 
   rule NAPNotification 
   expression sum(increase(message_total{type=~\”ldap_change-res_success\”, status=\”success\”}[5m])) /sum(increase(message_total{type=~\”ldap_change-res_.*\”}[5m])) <0.90 
   severity major 
   type Communications Alarm 
   annotation summary 
   value This alert is fired when the success percentage of NAP request is lesser threshold. 
   exit 
exit

System alerts

Disk full alert

Use the following commands to configure alerts related to disk full alert.

alerts rules group 
   rule node-disk-running-full 
   expression node_filesystem_usage > 0.0001 
   duration 5m 
   severity critical 
   type Processing Error Alarm 
   annotation disk_full 
   value test 
   exit 
exit

VM down alert

Use the following commands to configure alerts related to virtual machine down alert.

alerts rules group vm-state-change 
   rule vm-down 
   expression up{pod=~\"node-expo.*\"} == 0 
   duration 1m 
   severity major 
   type Equipment Alarm 
   annotation summary 
   value VM Down 
   exit 
exit

High memory usage

Use the following commands to configure alerts related to high memory usage.

alerts rules group memory-util-high 
   rule mem-util-high 
   expression avg(node_memory_MemAvailable_bytes /node_memory_MemTotal_bytes * 100) by (hostname) < 20 
   duration 1m 
   severity critical 
   type Processing Error Alarm 
   annotation mem_util_high 
   value Hig Memory Usage 
   exit 
exit

High disk usage

Use the following commands to configure alerts related to high disk usage alert.

alerts rules group disk-util-high 
   duration 1m 
   rule disk-util-high 
   expression avg (node_filesystem_avail_bytes{mountpoint =\"/\"} /node_filesystem_size_bytes{mountpoint =\"/\"} *100) by (hostname) <20 
   severity critical 
   type Processing Error Alarm 
   annotation description 
   value Hig Memory Usage 
   exit 
exit

High CPU usage

Use the following commands to configure alerts related to high CPU usage alert.

alerts rules group cpu-util-high 
   rule cpu-util-idle 
   duration 1m 
   expression avg(rate(node_cpu_seconds_total{mode='idle'}[1m])) by (hostname) *100 < 50 
   severity critical 
   type Processing Error Alarm 
   annotation description 
   value Hig CPU 
   exit 
exit

Cisco Ultra Cloud Core CPC Policy- Configuration and Administration Guide, Release 2026.03

Bias-Free Language

Results

Chapter: Application Based Alert